Failed Accessibility Tests

[balthisar]

(Unverified, reported on mailing list.)

Revision: 2e383c6
Date: 28.02.2015 20:32:38

3 accessibility tests still fail: 5-2-1-2, 5-5-1-3, and 5-6-1-3.

Is this intentional or an error on my side?

See output of acctest.cmd below for detailed error messages.

Ralf

---

Testing 1-1-1-1 1.1.1.1 1
Testing 1-1-1-2 1.1.1.2 1
Testing 1-1-1-3 1.1.1.3 1
Testing 1-1-1-4 1.1.1.4 1
Testing 1-1-1-10 1.1.1.10 1
Testing 1-1-2-1 1.1.2.1 1
Testing 1-1-2-2 1.1.2.2 1
Testing 1-1-2-3 1.1.2.3 1
Testing 1-1-3-1 1.1.3.1 1
Testing 1-1-4-1 1.1.4.1 1
Testing 1-1-5-1 1.1.5.1 1
Testing 1-1-6-1 1.1.6.1 1
Testing 1-1-6-2 1.1.6.2 1
Testing 1-1-6-3 1.1.6.3 1
Testing 1-1-6-4 1.1.6.4 1
Testing 1-1-6-5 1.1.6.5 1
Testing 1-1-6-6 1.1.6.6 1
Testing 1-1-8-1 1.1.8.1 1
Testing 1-1-9-1 1.1.9.1 1
Testing 1-1-10-1 1.1.10.1 1
Testing 1-1-12-1 1.1.12.1 2
Testing 1-2-1-1 1.2.1.1 1
Testing 1-4-1-1 1.4.1.1 1
Testing 1-5-1-1 1.5.1.1 3
Testing 2-1-1-1 2.1.1.1 1
Testing 2-1-1-2 2.1.1.2 1
Testing 2-1-1-3 2.1.1.3 1
Testing 2-1-1-4 2.1.1.4 1
Testing 2-1-1-5 2.1.1.5 1
Testing 2-2-1-1 2.2.1.1 3
Testing 2-2-1-2 2.2.1.2 3
Testing 2-2-1-3 2.2.1.3 3
Testing 2-2-1-4 2.2.1.4 3
Testing 3-2-1-1 3.2.1.1 2
Testing 3-3-1-1 3.3.1.1 2
Testing 3-5-1-1 3.5.1.1 2
Testing 3-5-2-1 3.5.2.1 2
Testing 3-5-2-2 3.5.2.2 2
Testing 3-5-2-3 3.5.2.3 2
Testing 3-6-1-1 3.6.1.1 2
Testing 3-6-1-2 3.6.1.2 2
Testing 3-6-1-4 3.6.1.4 2
Testing 4-3-1-1 4.3.1.1 3
Testing 4-3-1-2 4.3.1.2 3
Testing 5-1-2-1 5.1.2.1 1
Testing 5-1-2-2 5.1.2.2 1
Testing 5-1-2-3 5.1.2.3 1
Testing 5-2-1-1 5.2.1.1 1
Testing 5-2-1-2 5.2.1.2 1
FAILED --- test '5.2.1.2' not detected in file 'accessTest\5-2-1-2.html'

Accessibility Checks: Version 0.1

accessTest\5-2-1-2.html:8:1: Access: [5.1.2.3]: data

missing row/column headers (1 row).
Info: Doctype given is "-//W3C//DTD HTML 4.0 Transitional//EN"
Info: Document content looks like HTML 4.01 Strict
No warnings or errors were found.
FAILED --- test '5.2.1.2' not detected in above
Testing 5-3-1-1 5.3.1.1 2
Testing 5-4-1-1 5.4.1.1 2
Testing 5-5-1-1 5.5.1.1 3
Testing 5-5-1-2 5.5.1.2 3
Testing 5-5-1-3 5.5.1.3 3
FAILED --- test '5.5.1.3' not detected in file 'accessTest\5-5-1-3.html'

Accessibility Checks: Version 0.1

accessTest\5-5-1-3.html:1:1: Access: [3.3.1.1]: use style sheets to control presentation.
accessTest\5-5-1-3.html:3:1: Access: [13.2.1.1]: Metadata missing.
accessTest\5-5-1-3.html:7:1: Access: [5.5.1.2]:

summary invalid (null).
accessTest\5-5-1-3.html:7:1: Access: [5.1.2.1]: data missing row/column headers (all).
Info: Doctype given is "-//W3C//DTD HTML 4.0 Transitional//EN"
Info: Document content looks like HTML 4.01 Strict
No warnings or errors were found.
FAILED --- test '5.5.1.3' not detected in above
Testing 5-5-1-6 5.5.1.6 3
Testing 5-5-2-1 5.5.2.1 2
Testing 5-6-1-1 5.6.1.1 3
Testing 5-6-1-2 5.6.1.2 3
Testing 5-6-1-3 5.6.1.3 3
FAILED --- test '5.6.1.3' not detected in file 'accessTest\5-6-1-3.html'

Accessibility Checks: Version 0.1

accessTest\5-6-1-3.html:1:1: Access: [3.3.1.1]: use style sheets to control presentation.
accessTest\5-6-1-3.html:3:1: Access: [13.2.1.1]: Metadata missing.
accessTest\5-6-1-3.html:8:1: Access: [5.4.1.1]: invalid markup used in layout

.
accessTest\5-6-1-3.html:11:13: Access: [5.6.1.2]: header abbreviations invalid (null).
accessTest\5-6-1-3.html:11:72: Access: [5.6.1.2]: header abbreviations invalid (null).
Info: Doctype given is "-//W3C//DTD HTML 4.0 Transitional//EN"
Info: Document content looks like HTML 4.01 Strict
No warnings or errors were found.
FAILED --- test '5.6.1.3' not detected in above
Testing 6-1-1-1 6.1.1.1 1
Testing 6-1-1-2 6.1.1.2 1
Testing 6-1-1-3 6.1.1.3 1
Testing 6-2-1-1 6.2.1.1 1
Testing 6-2-2-1 6.2.2.1 1
Testing 6-2-2-2 6.2.2.2 1
Testing 6-2-2-3 6.2.2.3 1
Testing 6-3-1-1 6.3.1.1 1
Testing 6-3-1-2 6.3.1.2 1
Testing 6-3-1-3 6.3.1.3 1
Testing 6-3-1-4 6.3.1.4 1
Testing 6-5-1-1 6.5.1.1 2
Testing 6-5-1-2 6.5.1.2 2
Testing 6-5-1-3 6.5.1.3 2
Testing 6-5-1-4 6.5.1.4 2
Testing 7-1-1-1 7.1.1.1 1
Testing 7-1-1-2 7.1.1.2 1
Testing 7-1-1-3 7.1.1.3 1
Testing 7-1-1-4 7.1.1.4 1
Testing 7-1-1-5 7.1.1.5 1
Testing 7-2-1-1 7.2.1.1 2
Testing 7-4-1-1 7.4.1.1 2
Testing 7-5-1-1 7.5.1.1 2
Testing 8-1-1-1 8.1.1.1 1
Testing 8-1-1-2 8.1.1.2 1
Testing 8-1-1-3 8.1.1.3 1
Testing 8-1-1-4 8.1.1.4 1
Testing 9-1-1-1 9.1.1.1 1
Testing 9-3-1-1 9.3.1.1 2
Testing 9-3-1-2 9.3.1.2 2
Testing 9-3-1-3 9.3.1.3 2
Testing 9-3-1-4 9.3.1.4 2
Testing 9-3-1-5 9.3.1.5 2
Testing 9-3-1-6 9.3.1.6 2
Testing 10-1-1-1 10.1.1.1 2
Testing 10-1-1-2 10.1.1.2 2
Testing 11-2-1-1 11.2.1.1 2
Testing 11-2-1-2 11.2.1.2 2
Testing 11-2-1-3 11.2.1.3 2
Testing 11-2-1-4 11.2.1.4 2
Testing 11-2-1-5 11.2.1.5 2
Testing 11-2-1-6 11.2.1.6 2
Testing 11-2-1-7 11.2.1.7 2
Testing 11-2-1-8 11.2.1.8 2
Testing 11-2-1-9 11.2.1.9 2
Testing 11-2-1-10 11.2.1.10 2
Testing 12-1-1-1 12.1.1.1 1
Testing 12-1-1-2 12.1.1.2 1
Testing 12-1-1-3 12.1.1.3 1
Testing 12-4-1-1 12.4.1.1 2
Testing 12-4-1-2 12.4.1.2 2
Testing 12-4-1-3 12.4.1.3 2
Testing 13-1-1-1 13.1.1.1 2
Testing 13-1-1-2 13.1.1.2 2
Testing 13-1-1-3 13.1.1.3 2
Testing 13-1-1-4 13.1.1.4 2
Testing 13-2-1-1 13.2.1.1 2
Testing 13-2-1-3 13.2.1.3 2
Testing 13-10-1-1 13.10.1.1 3
FAILED [ 5-2-1-2 5-5-1-3 5-6-1-3] ...

[geoffmcl]

@balthisar It should be noted that these 3 tests also fail in the last 2009 release of tidy, so they are not 'new' failures.

On carefully examining each test...

Test 5-2-1-2:

Hmmmm, as with the other tests that fail, this has been like this since before 2009, so is not due to any change in tidy5 code.

I have found a small fix that outputs this access message, but not sure I understand exactly what the CheckMultiHeaders(...) service is trying to do. It was coded to ONLY test for a rowspan/column span in the first row, but the test file uses rowspan in the second row...

Test 5-5-1-3:

This has a table summary that is all spaces, like -
<table summary=" ">

Tidy has two error messages about this -
{ TABLE_SUMMARY_INVALID_NULL, "[5.5.1.2]: <table> summary invalid null)."... AND
{ TABLE_SUMMARY_INVALID_SPACES, "[5.5.1.3]: <table> summary invalid spaces)."

Sometime BEFORE 2009 the table parser was coded to remove space from the beginning and end of the attribute value, thus tidy now reports 5.5.1.2, and could never report 5.5.1.3. At the time of this test tidy no longer has access to the space data that was there. Since there is already a test 5.5.1.2, with <table summary=""> I propose removing this 5.5.1.3 test, since they address the same issue, a null or all space summary value.

Test 5-6-1-3:

Like the above, since before 2009, tidy started cleaning away the spaces, so can not separate a true blank "", and only spaces " ". In this case can not distinguish between <th abbr=""> in test 5-6-1-2 and <th abbr=" "> in test 5-6-1-3.

Again propose removing test 5-6-1-3...

Perhaps tidy could be coxed into keeping this white space, but it would only ever keep one... in some pretty old code all white space after the first is NOT even added to the lexer... and the comments in the code make it clear that white space should be removed -

    /* ignore leading and trailing white space for all but title, alt, value */
    /* and prompts attributes unless --literal-attributes is set to yes      */
    /* #994841 - Whitespace is removed from value attributes                 */

So at present the attribute value pointer is set to NULL after the white space is removed, and no other characters remain...

I see no alternative but to remove these two tests... but seek comments...

[ralfjunker]

I confirm that 606ffeb fixes test 5-2-1-2 for me.

Regarding tests 5-5-1-3 and 5-6-1-3:

Following your rationale, I suggest to change the tests' results instead of removing them. Even though the results have changed over time, they still serve a (albeit different) purpose now: Test that spaces are indeed cleaned away. Ideally add a comment explaining this and a bit of history as you have above.

IMHO, there can hardly be enough tests ...

[geoffmcl]

@ralfjunker as suggested have altered the expected output, and added a big note in the commit explaining the rational... thanks...

[balthisar]

@ralfjunker, When 5.0.0 is official, one of the next big pushes will be for the testing system, including establishing a new baseline. If we're okay with the altered results today then I don't see a reason to honor the past 100%.

Edit: what @geoffmcl said.