From: Kienan Stewart <kstewart@efficios.com>
Date: Mon, 15 Jan 2024 19:00:11 +0000 (-0500)
Subject: lava: Install vlttng inside a venv
X-Git-Url: https://git.lttng.org./?a=commitdiff_plain;h=db5b14958e79d156b950a2ae79baa9543cb0a91f;p=lttng-ci.git

lava: Install vlttng inside a venv

A number of the lava jobs were failing as the python tests couldn't
import the `bt2` module. While it may be possible to improve the
behaviour of vlttng to better address the situation two elements
combine to make the process work more systematically:

* Install `vlttng` inside a venv instead of using the
`--break-system-packages` option: once activated, this will set an
initial value for the `PYTHONPATH` environment variable
* When `vlttng` is invoked, since `PYTHONPATH` is no longer empty, the
value will be updated correctly.

In the case of the new rootfs, it also appears to be necessary to
ensure that `setuptools` is using `distutils` from Python stdlib,
although the module is deprecated and will be removed in Python 3.12

C.f. https://peps.python.org/pep-0632/

Signed-off-by: Kienan Stewart <kstewart@efficios.com>
Change-Id: If0aefb72f5c37721f88b49bd4a841b5ce61f46a0
---

diff --git a/jobs/system-tests.yaml b/jobs/system-tests.yaml
index eb64342..746cbbc 100644
--- a/jobs/system-tests.yaml
+++ b/jobs/system-tests.yaml
@@ -37,7 +37,7 @@
           description: 'LTTng-UST git repo to checkout the UST id'
       - string:
           name: 'ROOTFS_URL'
-          default: 'https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-12.tar.gz'
+          default: 'https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-15.tar.gz'
           description: 'The URL at which the system root FS can be downloaded'
       - string:
           name: 'LTTNG_CI_REPO'
diff --git a/lava/rootfs/vmdeboostrap/generate-root.py b/lava/rootfs/vmdeboostrap/generate-root.py
index f68678d..9635b97 100755
--- a/lava/rootfs/vmdeboostrap/generate-root.py
+++ b/lava/rootfs/vmdeboostrap/generate-root.py
@@ -64,6 +64,7 @@ packages = [
     'libtool',
     'libxml2',
     'libxml2-dev',
+    'locales',
     'netcat-traditional',
     'openssh-server',
     'psmisc',
@@ -75,6 +76,7 @@ packages = [
     'python3-pip',
     'python3-setuptools',
     'python3-sphinx',
+    'python3-venv',
     'rsync',
     'stress',
     'swig',
diff --git a/lava/system-tests/kernel-tests.yml b/lava/system-tests/kernel-tests.yml
index 94dbac0..7c33eec 100644
--- a/lava/system-tests/kernel-tests.yml
+++ b/lava/system-tests/kernel-tests.yml
@@ -12,6 +12,7 @@ run:
                 - mkdir -p /tmp/coredump
                 - echo "/tmp/coredump/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
                 - ulimit -c unlimited
+                - source /root/python-venv/bin/activate
                 - source /root/lttngvenv/activate
                 - pushd /root/lttngvenv/src/lttng-tools
                 - lava-test-case build-test-suite --shell "make"
diff --git a/lava/system-tests/perf-tests.yml b/lava/system-tests/perf-tests.yml
index f869ff0..ea6251e 100644
--- a/lava/system-tests/perf-tests.yml
+++ b/lava/system-tests/perf-tests.yml
@@ -12,6 +12,7 @@ run:
                 - mkdir -p /tmp/coredump
                 - echo "/tmp/coredump/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
                 - ulimit -c unlimited
+                - source /root/python-venv/bin/activate
                 - source /root/lttngvenv/activate
                 - pushd /root/lttngvenv/src/lttng-tools
                 - lava-test-case build-test-suite --shell "make"
diff --git a/lava/system-tests/setup.yml b/lava/system-tests/setup.yml
index ac57d9f..d9fcd4d 100644
--- a/lava/system-tests/setup.yml
+++ b/lava/system-tests/setup.yml
@@ -16,7 +16,12 @@ run:
     - depmod -a
     - apt-get update
     - apt-get upgrade
-    - pip3 install vlttng --break-system-packages
+    # The locale generation is required for the python venv
+    # to be correctgly created.
+    - locale-gen en_US.UTF-8
+    - python3 -m venv /root/python-venv
+    - source /root/python-venv/bin/activate
+    - pip3 install vlttng
     - hash -r
     - git clone "${CI_REPO}" --branch="${CI_BRANCH}" ci
     - mkdir -p /root/.ssh
diff --git a/scripts/system-tests/lava2-submit.py b/scripts/system-tests/lava2-submit.py
index 7ddcbd1..4fa51d4 100644
--- a/scripts/system-tests/lava2-submit.py
+++ b/scripts/system-tests/lava2-submit.py
@@ -198,7 +198,7 @@ def main():
     parser.add_argument('-d', '--debug', required=False, action='store_true')
     parser.add_argument(
         '-r', '--rootfs-url', required=False,
-        default="https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-12.tar.gz"
+        default="https://obj.internal.efficios.com/lava/rootfs_amd64_bookworm_2024-01-15.tar.gz"
     )
     parser.add_argument('--ci-repo', required=False, default='https://github.com/lttng/lttng-ci.git')
     parser.add_argument('--ci-branch', required=False, default='master')
diff --git a/scripts/system-tests/template_lava_job.jinja2 b/scripts/system-tests/template_lava_job.jinja2
index 72a3c6e..06ec835 100644
--- a/scripts/system-tests/template_lava_job.jinja2
+++ b/scripts/system-tests/template_lava_job.jinja2
@@ -91,6 +91,12 @@ actions:
                     - mount /dev/sda /tmp
                   {% endif %}
                     - rm -rf /tmp/*
+                    # With the Debian bookwrom rootfs, without
+                    # this being set, the babeltrace2 'bt2' python
+                    # bindings are not correctly installed in the
+                    # vlttng directory.
+                    - export SETUPTOOLS_USE_DISTUTILS=stdlib
+                    - source /root/python-venv/bin/activate
                     - {{ vlttng_cmd }}
                     - ln -s {{ vlttng_path }} /root/lttngvenv
                     - sync