lava: Install vlttng inside a venv

A number of the lava jobs were failing as the python tests couldn't
import the `bt2` module. While it may be possible to improve the
behaviour of vlttng to better address the situation two elements
combine to make the process work more systematically:

* Install `vlttng` inside a venv instead of using the
`--break-system-packages` option: once activated, this will set an
initial value for the `PYTHONPATH` environment variable
* When `vlttng` is invoked, since `PYTHONPATH` is no longer empty, the
value will be updated correctly.

In the case of the new rootfs, it also appears to be necessary to
ensure that `setuptools` is using `distutils` from Python stdlib,
although the module is deprecated and will be removed in Python 3.12

C.f. https://peps.python.org/pep-0632/

Signed-off-by: Kienan Stewart <kstewart@efficios.com>
Change-Id: If0aefb72f5c37721f88b49bd4a841b5ce61f46a0

diff --git a/jobs/system-tests.yaml b/jobs/system-tests.yaml
index eb6434263c49b66ab9dd0b153a8471bee40f34cb..746cbbc0191ef31e40b1c5cdbe20f1746d80ea90 100644 (file)
--- a/jobs/system-tests.yaml
+++ b/jobs/system-tests.yaml
@@ -37,7 +37,7 @@
           description: 'LTTng-UST git repo to checkout the UST id'
       - string:
           name: 'ROOTFS_URL'
-          default: 'https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-12.tar.gz'
+          default: 'https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-15.tar.gz'
           description: 'The URL at which the system root FS can be downloaded'
       - string:
           name: 'LTTNG_CI_REPO'

diff --git a/lava/rootfs/vmdeboostrap/generate-root.py b/lava/rootfs/vmdeboostrap/generate-root.py
index f68678d84cc5ab7ff070348456d97d2445a65896..9635b975b7fa9150b59f767e9a9127c172720934 100755 (executable)
--- a/lava/rootfs/vmdeboostrap/generate-root.py
+++ b/lava/rootfs/vmdeboostrap/generate-root.py
@@ -64,6 +64,7 @@ packages = [
     'libtool',
     'libxml2',
     'libxml2-dev',
+    'locales',
     'netcat-traditional',
     'openssh-server',
     'psmisc',
@@ -75,6 +76,7 @@ packages = [
     'python3-pip',
     'python3-setuptools',
     'python3-sphinx',
+    'python3-venv',
     'rsync',
     'stress',
     'swig',

diff --git a/lava/system-tests/kernel-tests.yml b/lava/system-tests/kernel-tests.yml
index 94dbac0d1a5550e3be33024a15d9651cb2d2cb02..7c33eec5d6fdc44b73c422f04523001bac3b4c36 100644 (file)
--- a/lava/system-tests/kernel-tests.yml
+++ b/lava/system-tests/kernel-tests.yml
@@ -12,6 +12,7 @@ run:
                 - mkdir -p /tmp/coredump
                 - echo "/tmp/coredump/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
                 - ulimit -c unlimited
+                - source /root/python-venv/bin/activate
                 - source /root/lttngvenv/activate
                 - pushd /root/lttngvenv/src/lttng-tools
                 - lava-test-case build-test-suite --shell "make"

diff --git a/lava/system-tests/perf-tests.yml b/lava/system-tests/perf-tests.yml
index f869ff00286e52e146d4876bd264705c39e36b56..ea6251e52bef549ecc86d6e5e655ed590f05d006 100644 (file)
--- a/lava/system-tests/perf-tests.yml
+++ b/lava/system-tests/perf-tests.yml
@@ -12,6 +12,7 @@ run:
                 - mkdir -p /tmp/coredump
                 - echo "/tmp/coredump/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
                 - ulimit -c unlimited
+                - source /root/python-venv/bin/activate
                 - source /root/lttngvenv/activate
                 - pushd /root/lttngvenv/src/lttng-tools
                 - lava-test-case build-test-suite --shell "make"

diff --git a/lava/system-tests/setup.yml b/lava/system-tests/setup.yml
index ac57d9fe5ce8619cc6d0e3f2babf67f28f728e0a..d9fcd4d035dad0139a1a75a3ec42494223d0529f 100644 (file)
--- a/lava/system-tests/setup.yml
+++ b/lava/system-tests/setup.yml
@@ -16,7 +16,12 @@ run:
     - depmod -a
     - apt-get update
     - apt-get upgrade
-    - pip3 install vlttng --break-system-packages
+    # The locale generation is required for the python venv
+    # to be correctgly created.
+    - locale-gen en_US.UTF-8
+    - python3 -m venv /root/python-venv
+    - source /root/python-venv/bin/activate
+    - pip3 install vlttng
     - hash -r
     - git clone "${CI_REPO}" --branch="${CI_BRANCH}" ci
     - mkdir -p /root/.ssh

diff --git a/scripts/system-tests/lava2-submit.py b/scripts/system-tests/lava2-submit.py
index 7ddcbd1a77d585f80af7b4827a869ef0b176e38e..4fa51d402b746a561a8430c6d59b36604c9d3534 100644 (file)
--- a/scripts/system-tests/lava2-submit.py
+++ b/scripts/system-tests/lava2-submit.py
@@ -198,7 +198,7 @@ def main():
     parser.add_argument('-d', '--debug', required=False, action='store_true')
     parser.add_argument(
         '-r', '--rootfs-url', required=False,
-        default="https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-12.tar.gz"
+        default="https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-15.tar.gz"
     )
     parser.add_argument('--ci-repo', required=False, default='https://github.com/lttng/lttng-ci.git')
     parser.add_argument('--ci-branch', required=False, default='master')

diff --git a/scripts/system-tests/template_lava_job.jinja2 b/scripts/system-tests/template_lava_job.jinja2
index 72a3c6eba959c4478990737bc91878a48c0e973e..06ec835992391506d0a665131e4aa809901191e9 100644 (file)
--- a/scripts/system-tests/template_lava_job.jinja2
+++ b/scripts/system-tests/template_lava_job.jinja2
@@ -91,6 +91,12 @@ actions:
                     - mount /dev/sda /tmp
                   {% endif %}
                     - rm -rf /tmp/*
+                    # With the Debian bookwrom rootfs, without
+                    # this being set, the babeltrace2 'bt2' python
+                    # bindings are not correctly installed in the
+                    # vlttng directory.
+                    - export SETUPTOOLS_USE_DISTUTILS=stdlib
+                    - source /root/python-venv/bin/activate
                     - {{ vlttng_cmd }}
                     - ln -s {{ vlttng_path }} /root/lttngvenv
                     - sync