Skip to content

Update KCIDB tests grouping #1453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 8, 2025
+59 −29
Merged

Update KCIDB tests grouping #1453

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ed57f5c
Fetch arch and compiler for KCIDB tests
Sep 1, 2025
597d81f
Update KCIDB tests grouping
Sep 1, 2025
c7c0a2d
Update `summary` jinja template
Sep 3, 2025
d63a5fc
Accommodate `arch/compiler` to `RegressionData`
Sep 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 37 additions & 20 deletions backend/kernelCI_app/management/commands/notifications.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -414,32 +414,49 @@ def evaluate_test_results(
group_size=group_size,
)

# Group by platform, then by config_name, then by path
grouped = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
new_issues = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
fixed_issues = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
unstable_tests = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
# Group by platform, then by config_name, then by arch/compiler, then by path
grouped = defaultdict(
lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
)
new_issues = defaultdict(
lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
)
fixed_issues = defaultdict(
lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
)
unstable_tests = defaultdict(
lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
)

for test in tests:
platform = test.get("platform", "unknown platform")
config_name = test.get("config_name", "unknown config")
grouped[platform][config_name][test["path"]].append(test)
arch = test.get("architecture", "unknown architecture")
compiler = test.get("compiler", "unknown compiler")
grouped[platform][config_name][f"{arch}/{compiler}"][test["path"]].append(test)

for platform, configs in grouped.items():
for config_name, paths in configs.items():
for path, test_group in paths.items():
unique_statuses = {t["status"] for t in test_group}
if len(unique_statuses) == 1:
continue

category = categorize_test_history(test_group)

if category == "regression":
new_issues[platform][config_name][path] = test_group
elif category == "fixed":
fixed_issues[platform][config_name][path] = test_group
else:
unstable_tests[platform][config_name][path] = test_group
for config_name, arch_compilers in configs.items():
for arch_compiler, paths in arch_compilers.items():
for path, test_group in paths.items():
unique_statuses = {t["status"] for t in test_group}
if len(unique_statuses) == 1:
continue

category = categorize_test_history(test_group)

if category == "regression":
new_issues[platform][config_name][arch_compiler][
path
] = test_group
elif category == "fixed":
fixed_issues[platform][config_name][arch_compiler][
path
] = test_group
else:
unstable_tests[platform][config_name][arch_compiler][
path
] = test_group

return new_issues, fixed_issues, unstable_tests

Expand Down
14 changes: 9 additions & 5 deletions backend/kernelCI_app/management/commands/templates/summary.txt.j2
View file
Open in desktop
Copy link
Collaborator

@MarceloRobert MarceloRobert Sep 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is working, but I think that the final result is not the best looking. It shows the config for every test, and seems to group tests by architecture even though the first parameter is the config.

IMO we could place the config_name above arch/compiler and add more spaces to group arch/compiler under configs, and then tests under arch/compiler

Current structure:
image

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the example above, it would mean:

Hardware: mt8395-genio-1200-evk
Config: defconfig+lab-setup+kselftest
- Architecture/compiler: arm64/gcc-12
  - boot
    last run: https://d.kernelci.org/test/maestro:68b7e1f7b9811ea53d01b575
    history:  > ✅  > ✅  > ❌  > ✅  >

Config: defconfig+preempt_rt
- Architecture/compiler: arm64/gcc-12
  - rt-tests.cyclicdeadline
    last run: https://d.kernelci.org/test/maestro:68b79c4eb9811ea53d016421
    history:  > ✅  > ⚠️  
            
  - rt-tests.cyclictest
    last run: https://d.kernelci.org/test/maestro:68b79c53b9811ea53d016440
    history:  > ✅  > ⚠️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for the testing. I fixed the summary template.
Tested OK locally. Thanks for helping me with the local setup @MarceloRobert :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @padovan
Does this structure look OK or do you need any changes? #1453 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about add one more level of indent

Hardware:
  > Config:
      -  Arch/compiler:

So it reads better.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about add one more level of indent

Hardware:
  > Config:
      -  Arch/compiler:

So it reads better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. Updated it.

Hardware: sun50i-h5-libretech-all-h3-cc
  > Config: defconfig+arm64-chromebook+kselftest
    - Architecture/compiler: arm64/gcc-12
      - kselftest.uevent
      last run: https://d.kernelci.org/test/maestro:68be1e3f3273d911562c70a2
      history:  > ✅  > ❌  > ✅  > ✅  > ✅  
            
      - kselftest.uevent.uevent_uevent_filtering
      last run: https://d.kernelci.org/test/maestro:68be330a3273d911562cd016
      history:  > ✅  > ❌  > ✅  > ✅  > ✅  
            
      - kselftest.uevent.uevent_uevent_filtering_global_uevent_filtering
      last run: https://d.kernelci.org/test/maestro:68be330a3273d911562cd017
      history:  > ✅  > ❌  > ✅  > ✅  > ✅

Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,14 @@
{% macro render_issues(issues) -%}
{%- for platform, configs in issues.items() %}
Hardware: {{ platform }}
{%- for config_name, paths in configs.items() %}
{%- for path, test_group in paths.items() %}
- {{path}} ({{config_name}})
last run: https://d.kernelci.org/test/{{test_group[0]["id"]}}
history: {% for t in test_group | reverse -%}
{%- for config_name, arch_compilers in configs.items() %}
> Config: {{config_name}}
{%- for arch_compiler, paths in arch_compilers.items() %}
- Architecture/compiler: {{arch_compiler}}
{%- for path, test_group in paths.items() %}
- {{path}}
last run: https://d.kernelci.org/test/{{test_group[0]["id"]}}
history: {% for t in test_group | reverse -%}
{%- if t["status"] == "PASS" -%}
{{- "> ✅ " -}}
{%- elif t["status"] == "FAIL" -%}
Expand All @@ -18,6 +21,7 @@ Hardware: {{ platform }}
{%- endif -%}
{%- endfor %}
{% endfor %}
{%- endfor %}
{%- endfor -%}
{%- endfor -%}
{%- endmacro %}
Expand Down
10 changes: 8 additions & 2 deletions backend/kernelCI_app/queries/notifications.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -551,15 +551,19 @@ def kcidb_tests_results(
PARTITION BY
path,
platform,
config_name
config_name,
architecture,
compiler
ORDER BY
start_time DESC NULLS LAST
) AS rn,
FIRST_VALUE(git_commit_hash) OVER (
PARTITION BY
path,
platform,
config_name
config_name,
architecture,
compiler
ORDER BY
start_time DESC NULLS LAST
) AS first_hash_by_group
Expand All @@ -586,6 +590,8 @@ def kcidb_tests_results(
path,
platform,
config_name,
architecture,
compiler,
start_time DESC NULLS LAST;
"""

Expand Down
7 changes: 5 additions & 2 deletions backend/kernelCI_app/typeModels/treeReport.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,11 @@ class RegressionHistoryItem(TypedDict):
status: Test__Status


type RegressionData = dict[str, dict[str, dict[str, list[RegressionHistoryItem]]]]
"""The history of tests is grouped by hardware, then config, then path."""
type RegressionData = dict[
str, dict[str, dict[str, dict[str, list[RegressionHistoryItem]]]]
]
"""The history of tests is grouped by hardware, then architecture/compiler string,
then config, then path."""


class TreeReportIssues(BaseModel):
Expand Down