Fixing semantics borders on mobile web

[nturgut]

Changing the css matrix transformation to only add 2d styling values (top, left, width, height). Calculating bounding rectangle for the transform so that if the semantics boundary is not an ordinary 2d translate, still focus is around the semantics area. This is similar to the Flutter native behaviour.

Fixes: flutter/flutter#68225

[nturgut]

@goderbauer can you confirm that the semantics focus node's border can be represented with 2d translation only? Or is it possible to receive a matrix we can't represent with top/left.

I tested this transform on a native Flutter app:
transform: Matrix4.skewY(0.3)..rotateZ(-pi / 12.0),

The screen reader focus node's border was a rectangle (not a parallelogram).

[flutter-dashboard]

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

[goderbauer]

@goderbauer can you confirm that the semantics focus node's border can be represented with 2d translation only? Or is it possible to receive a matrix we can't represent with top/left.

It's really up to the embedder to decide how to and where to draw the focus rect. The flutter framework provides you with a Rect and an arbitrary transform Matrix4 for each semantics node. It's upto the embedder to translate that into a focus rect. As far as I know, most (all?) platforms we support just draw the focus rect as the bounding box around the transformed rect, see Android for example:

engine/shell/platform/android/io/flutter/view/AccessibilityBridge.java

Line 2266 in 4f89ff7

globalRect.set(

However, in theory, if the platform supports it, you could draw a parallelogram as a focus rect. Flutter provides you all the data for this. Example:

Align(
      alignment: Alignment.topLeft,
      child: Transform(
        transform: Matrix4.skewY(0.3)..rotateZ(-pi / 12.0),
        child: Semantics(
          container: true,
          label: 'hello',
          child: Container(
            height: 100,
            width: 100,
            color: Colors.green,
          ),
        ),
      )
    );

This creates the following semantics node:

I/flutter (18226):        └─SemanticsNode#4
I/flutter (18226):            Rect.fromLTRB(0.0, 0.0, 100.0, 100.0) with transform
I/flutter (18226):            [0.9659258262890683,0.25881904510252074,0.0,0.0;
I/flutter (18226):            0.039976827402816106,1.045987939028626,0.0,0.0; 0.0,0.0,1.0,0.0;
I/flutter (18226):            0.0,0.0,0.0,1.0]
I/flutter (18226):            label: "hello"
I/flutter (18226):            textDirection: ltr

and if you apply that matrix to the rect you should get a parallelogram.

I believe just reading the translation values of the transform matrix is not enough, though. In the example above, if you replace the transform parameter with something like Matrix4.identity()..scale(2.0) wouldn't you draw a focus rect that's too small since it's missing the scale? Or is that applied elsewhere in web land?

[nturgut]

@goderbauer can you confirm that the semantics focus node's border can be represented with 2d translation only? Or is it possible to receive a matrix we can't represent with top/left.

It's really up to the embedder to decide how to and where to draw the focus rect. The flutter framework provides you with a Rect and an arbitrary transform Matrix4 for each semantics node. It's upto the embedder to translate that into a focus rect. As far as I know, most (all?) platforms we support just draw the focus rect as the bounding box around the transformed rect, see Android for example:

engine/shell/platform/android/io/flutter/view/AccessibilityBridge.java

Line 2266 in 4f89ff7

globalRect.set(

However, in theory, if the platform supports it, you could draw a parallelogram as a focus rect. Flutter provides you all the data for this. Example:

Align(
      alignment: Alignment.topLeft,
      child: Transform(
        transform: Matrix4.skewY(0.3)..rotateZ(-pi / 12.0),
        child: Semantics(
          container: true,
          label: 'hello',
          child: Container(
            height: 100,
            width: 100,
            color: Colors.green,
          ),
        ),
      )
    );

This creates the following semantics node:

I/flutter (18226):        └─SemanticsNode#4
I/flutter (18226):            Rect.fromLTRB(0.0, 0.0, 100.0, 100.0) with transform
I/flutter (18226):            [0.9659258262890683,0.25881904510252074,0.0,0.0;
I/flutter (18226):            0.039976827402816106,1.045987939028626,0.0,0.0; 0.0,0.0,1.0,0.0;
I/flutter (18226):            0.0,0.0,0.0,1.0]
I/flutter (18226):            label: "hello"
I/flutter (18226):            textDirection: ltr

and if you apply that matrix to the rect you should get a parallelogram.

I believe just reading the translation values of the transform matrix is not enough, though. In the example above, if you replace the transform parameter with something like Matrix4.identity()..scale(2.0) wouldn't you draw a focus rect that's too small since it's missing the scale? Or is that applied elsewhere in web land?

Thanks a lot for the insight. I can see that we are loosing the scale information here, so that's important to address.

Let me rephrase what I understand from the parallelogram example. Other embedder's still does not have a parallelogram shaped focus node. The focus node is a rectangle bounding the parallelogram. I think this is also another area to address on the PR.