Alternatives to adding a PR comment with the output of coverage-report.yml workflow

[zcorpan]

#1788 removes the regression-coverage-report check's code that comments on the PR with the output of the regression test coverage check, because the way it's implemented is unsafe.

Relaying a discussion over email with myself, @jesdaigle, @mcking65, @nschonni:

I wanted to learn why the check is posting a comment to the PR:

In Travis-CI, this was a perma-failing job through “Allowed Failures”, and GitHub does not have an equivalent actions/runner#2347. I think it was also just an ignored, since it was only there if you clicked through

the checks tab in a PR is kind of ugly with screen readers. It is tricky to work. Seems to have some kind of scroll-related issues.

So there are several problems here:

1. The regression test coverage report has errors on main.
2. The regression test coverage report is being ignored.
3. GitHub Actions doesn't allow non-blocking failing checks. (Though, allowing them makes the check easier to ignore.)

I think we should address the first point by fixing the errors.

I think we should address the second point by enforcing the check, so the main branch remains in an evergreen no-error state.

Then the third point is a non-issue, since we want it to be a blocking check to not fall back into the state of having errors and the check being ignored.

So then where and how should we present the output of the check? Here are some options:

1. Show the output formatted in the same way as the comment but in the Checks tab. When the check fails, the "Details" link should show the right page.
2. Annotate the "Files changed" tab with review comments on the relevant lines in the source.
3. Have an external page, and link to it from the Checks page.

These aren't necessarily mutually exclusive. A combination of (1) and (2) could be good, if we can get accessibility issues fixed or if we can work around them somehow.

Possible changes we could propose to GitHub:

• Change GitHub UI to fix accessibility issues with the Checks page. (@mcking65 mentioned that it doesn't work well with screen readers. I'd like to learn more about what the issues are specifically.)
• Change GitHub API to allow bots to comment with a comment-only access token, without full write access.
• Change GitHub Actions to allow non-blocking failing checks. Please support something like "allow-failure" for a given job actions/runner#2347

[nschonni]

#1976 restricts the token, to resolve the "unsafe" aspect

[zcorpan]

The workflow is now enabled again, see #2068

However, it fails for PRs that are from a fork, e.g. #1869

Could the workflow silently skip the commenting part when it doesn't have access to the secret, but still run the check?

[zcorpan]

Actually I've now disabled the workflow again, since it's blocking some PRs. We should fix the above comment before enabling.

https://github.com/w3c/aria-practices/actions/workflows/coverage-report.yml

[nschonni]

I think it could probably just spit out the results if it can't comment. The other option might be to move it to a "problem matcher" like the VNU ones, and have it put the comment at the top of the affected files

[mcking65]

@nschonni wrote:

The other option might be to move it to a "problem matcher" like the VNU ones, and have it put the comment at the top of the affected files

I particularly like that idea if feasible.

[nschonni]

I took a kick at it in nschonni#282 to test out.
It looks like it will only show the "top 10" when they match files that aren't touched by the PR. Alternate would be to go back to a single file to attach the report to, and lump all the details in it.

[howard-e]

I took a kick at it in nschonni#282 to test out.

Thanks @nschonni. Looks pretty good to me and I think this is something that could work.

[nschonni]

Thanks @howard-e, I'll clean it up a bit and submit a PR