Skip to content

Commit 2ac14fc

Browse files
sdimitrosdimitro
committed
DLPX-87572 sdb: want live kernel tests to find kernel regressions early
= Problem With our switch to the new v5.15 kernel a subset of SDB commands broke without us realizing until we actually needed them. Our regression dumps helps us to ensure we don't introduce regressions for older kernels when developing new features but they can't help us in detecting changes in the upstream kernel or ZFS that break our commands. = This Patch This patch attempts to provide a rudimentary mechanism for catching regression introduced by the upstream Ubuntu kernels by running a few basic SDB commands in a Github action that's run nightly and for every PR. Specifically this patch makes it so we have such a test for each Ubuntu LTS kernel starting from 20.04 (currently the `ubuntu-latest` Github runner tag points to 22.04 so we'd test that twice but in the future that tag will point to 24.04, etc...). We also change for all the available Python versions for each Ubuntu version to further ensure SDB's compatibility with future Python versions. = Misc Notes In order to use SDB in the Github runner I had to introduce an extra script that downloads the kernel's debug info. See the `install-live-kernel-dbg.sh` script for more info. I also made sure to decouple the apt-install of the python-dev files to its own shell script too as different Ubuntu versions ship with different Python versions. See `install-python-dev.sh` for more info. = Potential Future Items In the future we may want to detect whenever our ZFS commands are not getting out of date. `test_live_kernel.sh` has a way of detecting whether the ZFS module is installed and running a few ZFS commands on the live kernel. The idea is that we can either introduce Github Actions like the upstream openzfs that install our kernel module to the runner and run the commands there OR we can create a BlackBox test that clones the repo and runs this script.
1 parent 3a1fadc commit 2ac14fc

File tree

5 files changed

+192
-24
lines changed

5 files changed

+192
-24
lines changed

‎.github/scripts/install-libkdumpfile.sh‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@
88
#
99
sudo apt update
1010
sudo apt install autoconf automake liblzo2-dev libsnappy1v5 libtool pkg-config zlib1g-dev
11-
sudo apt install python3.8-dev python3.9-dev
1211

1312
git clone https://github.com/ptesarik/libkdumpfile.git
1413

‎.github/scripts/install-live-kernel-dbg.sh‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash -eux
2+
3+
# uname -a
4+
# uname -r
5+
# cat /etc/apt/sources.list
6+
#
7+
# sudo apt-get clean
8+
# sudo apt-get update
9+
# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
10+
# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
11+
# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
12+
#
13+
# sudo apt install ubuntu-dbgsym-keyring
14+
#
15+
# sudo apt-get clean
16+
# sudo apt-get update
17+
#
18+
# sudo apt-get install -y linux-image-$(uname -r)-dbgsym
19+
20+
kvers=$(uname -r)
21+
ddeb_file=$(curl http://ddebs.ubuntu.com/pool/main/l/linux-azure/ | \
22+
grep -Eo ">linux-image-(unsigned-)?$kvers(.*)amd64\.ddeb" | \
23+
cut -c2-)
24+
25+
wget http://ddebs.ubuntu.com/pool/main/l/linux-azure/$ddeb_file
26+
sudo dpkg -i $ddeb_file
27+
rm $ddeb_file

‎.github/scripts/install-python-dev.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash -eux
2+
3+
sudo apt update
4+
sudo apt install python3.$(python3 --version | cut -d . -f 2)-dev
5+
6+
#
7+
# Debug statements
8+
#
9+
echo $(which python3)

‎.github/workflows/main.yml‎

Lines changed: 72 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -6,24 +6,6 @@ on:
66

77
jobs:
88
#
9-
# Verify the build and installation of SDB.
10-
#
11-
install:
12-
runs-on: ubuntu-20.04
13-
strategy:
14-
matrix:
15-
python-version: [3.8, 3.9]
16-
steps:
17-
- uses: actions/checkout@v2
18-
- uses: actions/setup-python@v1
19-
with:
20-
python-version: ${{ matrix.python-version }}
21-
- run: python3 setup.py install
22-
#
23-
# The statement below is used for debugging the Github job.
24-
#
25-
- run: python3 --version
26-
#
279
# Verify "pylint" runs successfully.
2810
#
2911
# Note, we need to have "drgn" installed in order to run "pylint".
@@ -34,7 +16,7 @@ jobs:
3416
runs-on: ubuntu-20.04
3517
steps:
3618
- uses: actions/checkout@v2
37-
- uses: actions/setup-python@v1
19+
- uses: actions/setup-python@v4
3820
with:
3921
python-version: '3.8'
4022
- run: ./.github/scripts/install-drgn.sh
@@ -55,16 +37,17 @@ jobs:
5537
runs-on: ubuntu-20.04
5638
strategy:
5739
matrix:
58-
python-version: [3.8, 3.9]
40+
python-version: ['3.8', '3.9']
5941
dump: [dump.201912060006.tar.lzma, dump.202303131823.tar.gz]
6042
env:
6143
AWS_DEFAULT_REGION: 'us-west-2'
6244
steps:
6345
- uses: actions/checkout@v2
64-
- uses: actions/setup-python@v1
46+
- uses: actions/setup-python@v4
6547
with:
6648
python-version: ${{ matrix.python-version }}
6749
- run: python3 -m pip install aws python-config pytest pytest-cov
50+
- run: ./.github/scripts/install-python-dev.sh
6851
- run: ./.github/scripts/install-libkdumpfile.sh
6952
- run: ./.github/scripts/install-drgn.sh
7053
- run: ./.github/scripts/download-dump-from-s3.sh ${{ matrix.dump }}
@@ -73,13 +56,79 @@ jobs:
7356
with:
7457
token: ${{ secrets.CODECOV_TOKEN }}
7558
#
59+
# Verify common linux SDB commands can run on vanilla ubuntu kernels: 20.04
60+
#
61+
live_ubuntu_20_04:
62+
runs-on: ubuntu-20.04
63+
strategy:
64+
matrix:
65+
python-version: ['3.8', '3.9']
66+
env:
67+
AWS_DEFAULT_REGION: 'us-west-2'
68+
steps:
69+
- uses: actions/checkout@v2
70+
- uses: actions/setup-python@v4
71+
with:
72+
python-version: ${{ matrix.python-version }}
73+
- run: python3 -m pip install aws python-config pytest pytest-cov
74+
- run: ./.github/scripts/install-python-dev.sh
75+
- run: sudo ./.github/scripts/install-libkdumpfile.sh
76+
- run: sudo ./.github/scripts/install-drgn.sh
77+
- run: sudo ./.github/scripts/install-live-kernel-dbg.sh
78+
- run: sudo python3 ./setup.py install
79+
- run: sudo ./tests/scripts/test_live_kernel.sh
80+
#
81+
# Verify common linux SDB commands can run on vanilla ubuntu kernels: 22.04
82+
#
83+
live_ubuntu_22_04:
84+
runs-on: ubuntu-22.04
85+
strategy:
86+
matrix:
87+
python-version: ['3.10' , '3.11']
88+
env:
89+
AWS_DEFAULT_REGION: 'us-west-2'
90+
steps:
91+
- uses: actions/checkout@v2
92+
- uses: actions/setup-python@v4
93+
with:
94+
python-version: ${{ matrix.python-version }}
95+
- run: python3 -m pip install aws python-config pytest pytest-cov
96+
- run: ./.github/scripts/install-python-dev.sh
97+
- run: sudo ./.github/scripts/install-libkdumpfile.sh
98+
- run: sudo ./.github/scripts/install-drgn.sh
99+
- run: sudo ./.github/scripts/install-live-kernel-dbg.sh
100+
- run: sudo python3 ./setup.py install
101+
- run: sudo ./tests/scripts/test_live_kernel.sh
102+
#
103+
# Verify common linux SDB commands can run on the latest Github Ubuntu kernel
104+
#
105+
live_ubuntu_latest:
106+
runs-on: ubuntu-latest
107+
strategy:
108+
matrix:
109+
python-version: ['3.10' , '3.11']
110+
env:
111+
AWS_DEFAULT_REGION: 'us-west-2'
112+
steps:
113+
- uses: actions/checkout@v2
114+
- uses: actions/setup-python@v4
115+
with:
116+
python-version: ${{ matrix.python-version }}
117+
- run: python3 -m pip install aws python-config pytest pytest-cov
118+
- run: ./.github/scripts/install-python-dev.sh
119+
- run: sudo ./.github/scripts/install-libkdumpfile.sh
120+
- run: sudo ./.github/scripts/install-drgn.sh
121+
- run: sudo ./.github/scripts/install-live-kernel-dbg.sh
122+
- run: sudo python3 ./setup.py install
123+
- run: sudo ./tests/scripts/test_live_kernel.sh
124+
#
76125
# Verify "yapf" runs successfully.
77126
#
78127
yapf:
79128
runs-on: ubuntu-20.04
80129
steps:
81130
- uses: actions/checkout@v2
82-
- uses: actions/setup-python@v1
131+
- uses: actions/setup-python@v4
83132
with:
84133
python-version: '3.8'
85134
- run: python3 -m pip install yapf
@@ -104,7 +153,7 @@ jobs:
104153
runs-on: ubuntu-20.04
105154
steps:
106155
- uses: actions/checkout@v2
107-
- uses: actions/setup-python@v1
156+
- uses: actions/setup-python@v4
108157
with:
109158
python-version: '3.8'
110159
- run: ./.github/scripts/install-drgn.sh

‎tests/scripts/test_live_kernel.sh‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
#!/bin/bash -eu
2+
3+
scmds=(
4+
# Test `stacks`
5+
"stacks"
6+
7+
# Test `stacka`
8+
"stacka"
9+
10+
# Test `dmesg`
11+
"dmesg"
12+
13+
# Test `lxlist`
14+
"addr modules | lxlist module list | member name"
15+
16+
# Test `slabs` and `percpu`
17+
"slabs | filter \"obj.name == 'kmalloc-8'\" | member cpu_slab | percpu 0 1"
18+
19+
# Test `pid`
20+
"pid 1"
21+
22+
# Test `find_task`
23+
"find_task 1 2"
24+
25+
# Test `threads`
26+
"threads"
27+
28+
# Test `walk` and `slub_cache` walker
29+
"slabs | filter \"obj.name == 'TCP'\" | walk"
30+
31+
# Test `rbtree` walker
32+
"addr vmap_area_root | rbtree vmap_area rb_node"
33+
34+
# Test `fget`
35+
"find_task 1 | fget 1 4"
36+
)
37+
38+
for ((i = 0; i < ${#scmds[@]}; i++)); do
39+
sudo /usr/local/bin/sdb -e "${scmds[$i]}"
40+
done
41+
42+
zfs_scmds=(
43+
# Test `arc`
44+
"arc"
45+
46+
# Test `zfs_dbgmsg`
47+
"zfs_dbgmsg"
48+
49+
# Test `zio`
50+
"zio"
51+
52+
# Test `spa`
53+
"spa -vmH"
54+
55+
# Test `vdev` and `metaslab`
56+
"spa | vdev | metaslab"
57+
58+
# Test `vdev` and `metaslab` and `range_tree`
59+
"spa | vdev | metaslab | head 1 | member ms_allocatable | range_tree"
60+
61+
# Test `dbuf`
62+
"dbuf"
63+
64+
# Test `dbuf` and `blkptr`
65+
"dbuf | head 1 | member db_blkptr | blkptr"
66+
67+
# Test `spa` and `zhist`
68+
"spa | member spa_normal_class.mc_histogram | zhist"
69+
70+
# Test `avl`
71+
"address spa_namespace_avl | avl"
72+
73+
# Test `spl_kmem_caches`
74+
"spl_kmem_caches"
75+
)
76+
77+
if $(lsmod | grep -q zfs); then
78+
echo "Detected ZFS kernel module... testing ZFS commands:"
79+
for ((i = 0; i < ${#zfs_scmds[@]}; i++)); do
80+
sudo /usr/local/bin/sdb -e "${zfs_scmds[$i]}"
81+
done
82+
else
83+
echo "Can't find ZFS kernel module... skipping ZFS commands"
84+
fi

0 commit comments

Comments
 (0)