Diffing AOT size breakdown for two snapshot profiles

[mehmetf]

Today, Dart tooling can generate snapshot profiles for AOT and use this to create a Treemap representation in HTML format. The HTML format is very useful for visualizing a single snapshot profile. However, it is not suitable for automatically tracking size increases.

FR: We would like to be able to diff two profiles and see where the size delta is coming from.

One way would be to have a structured representation of the Treemap but it would also be OK to have a tool that generates a diff report.

[mehmetf]

@mraleph

[mraleph]

We already have a script for comparing --print-instructions-sizes-to=... dumps for two different builds and producing the diff like this:

+-------------------+------------------------------------------------+--------------+
| Library           | Method                                         | Diff (Bytes) |
+-------------------+------------------------------------------------+--------------+
| dart:_http        | _HttpParser._doParse                           |       +13724 |
| dart:typed_data   | _TypedIntListMixin.setRange                    |        +7344 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| dart:core         | _List.setRange                                 |        -3512 |
+-------------------+------------------------------------------------+--------------+
Comparing /tmp/sizes/release-stripped.json (old) to /tmp/sizes/profile-stripped.json (new)
Old   : 2865780 bytes.
New   : 3493980 bytes.
Change: +628200 bytes.

Would this work for you?

This script only covers instruction portion of the snapshot.

In general comparing diffing snapshots is a hard problem - snapshots are just graphs of objects, and not a structured representation of the original program (like Kernel blobs or Java class files). This means that for an arbitrary object in a snapshot you can't really say which library it originated from. You can compute an approximation - by tracing a path to an object from the roots and looking at which libraries it would pass through, but this approximation has multiple issues.

For example if you have library A which declares a huge string literal constant C and library B which uses it - there is no way by looking at the snapshot to attribute the size of C to A. It would just be attributed to B.

Another issue would be around sharing: in snapshots we try to share bitwise equivalent objects (where identity of the object does not matter), for example if two functions compile to the same machine code - we would make them both use the same copy, rather than producing two versions of the code in the resulting binary. Now: how do you surface the size of this code? Do you attribute it to the package which contains the first function or do you attribute it to the package that contains the second? Or half-half? etc.

[mehmetf]

How often do those issues appear in practice?

Even if we did not do diffing, being able to attribute a blob in AOT to a particular library is a P1 feature request for large 1p apps. E.g. they would like to know how much package:http is affecting their AOT. As long as the skew in real life is under ~5%, I think we can live with it.

I also would assume these size attribution problems affect the web representation of the treemap:

If we could create a simple str(path), size table and diff it, do you think it will be useful in real life scenarios?

Using example screenshot above:

+-------------------------------+--------------+
| Path                          | Diff (Bytes) |
+-------------------------------+--------------+
| package:flutter/src           |       +13724 |
| package:flutter/src/widgets   |        +7344 |
| package:flutter/src/rendering |        +7344 |
| dart:core                     |       +22932 |
| dart:core/int                 |        +3393 |
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[xster]

+1 to @mehmetf's point above. For modularity, I think the real feature request is to be able to dump in a human readable format the AOT size per library. Diff'ing could be another modular feature on top of it but being able to statically break down an AOT is a helpful feature for our users already.

[mraleph]

For modularity, I think the real feature request is to be able to dump in a human readable format the AOT size per library

Unfortunately for an arbitrary object in a snapshot we don't have a way to know, where this object originally came from. We do know this for libraries, classes, code - because they have clear origin. The rest of snapshot is like a soup, we don't store any information that would allow us with 100% certainty attribute an object to a specific library - and introducing this information is prohibitively complicated.

The best we can implement is a heuristic which splits snapshot graph based on reachability, i.e. look at an object in snapshot and see if it is only reachable through specific library - in which case attribute it to that library.

I have done an experiment on an internal application using this heuristic. I observe that 7.33% of bytes in snapshot can't be attributed to a specific package:

• 3.75% are shared - meaning that there are paths through multiple different packages to these objects, so we can't identify unique owner.
• 3.62% are unowned - paths from snapshot roots to these objects never cross any "owning objects". (most of these are not really unowned but actually shared in complicated ways - so each package contributes a bit - but this information can't be recovered from snapshot profile)

To make things "worse" here is how top of the breakdown by packages looks like:

package:flutter: 15.14%
package:some-user-package: 1.87%
dart:core: 1.81%
package:some-user-package: 1.59%
... tons and tons of packages...

So the contributions of individual packages are below the shared / unowned sizes.

I also would assume these size attribution problems affect the web representation of the treemap:

Sharing affects this tree-map but significantly less: if we look at shared code then it only accounts for around 0.6% of total code size.

I will just proceed to make the best tooling we can and document caveats - then we can try applying that internally and see what it catches and how noisy it would be and go from there.

[mehmetf]

Sharing affects this tree-map but significantly less

I am not getting what the data representation has to do with the percentage of shared code. Are you saying that the particular Treemap I pasted does not happen to have much shared code? or are you saying the Treemap represents not AOT bytes but some other format (Kernel?) and in that format, bytes are easier to attribute to a specific package?

If it is the latter, can we grossly assume that if an app is looking at their size increase and see +X% AOT and +Y% Kernel, the distribution of the package-specific increases in Kernel would "roughly" correlate to the size increase in AOT? I am asking this because, for apps, the precise number of bytes often does not matter. What they want to know is "Ah, I depended on this library and it increased my size a ton; let me see if there's a way for me to not depend on it."

I will just proceed to make the best tooling we can and document caveats - then we can try applying that internally and see what it catches and how noisy it would be and go from there.

That sounds great. We should also print "shared" and "unowned" in the output so we can track these explicitly. Thanks!

[xster]

+1 to what @mehmetf said.

Do you have a concrete example for how the shared ownership case happens in an app?

Also scope creep / feel free to branch off to another FR, how feasible would it be to bundle tree shaking data into the output? Specifically:

dream case:

package:flutter: 15.14% (3% tree-shaken)

or even

package:flutter: 15.14% (12345 symbols tree shaken)

[mraleph]

I am not getting what the data representation has to do with the percentage of shared code.

Sorry for not being clear - it's not about data representation, it's about what it represents. This particular tree map only represents the size of the code part (also sometimes referred as instructions part or executable/RX part) of the snapshot. Snapshot also has data part.

Things are easier with the code portion of the snapshot because:

• there is less sharing in it compared to the data portion;
• there is clear ownership for most of it.

What they want to know is "Ah, I depended on this library and it increased my size a ton; let me see if there's a way for me to not depend on it."

Yeah, I understand this desire.

Do you have a concrete example for how the shared ownership case happens in an app?

There are many ways. You can naturally refer to the same constant, you can use the same type in two different places (x as Something or even just in a signature of a function), two methods in two different classes can be called the same name. Compiler can also inline one method into another - bringing references from one place to another. Compiler can canonicalise metadata (e.g. stack maps used for GC) associated with two different code objects if it is bitwise equivalent. So on and so forth.

Also scope creep / feel free to branch off to another FR, how feasible would it be to bundle tree shaking data into the output?

We could theoretically provide information of how many methods were never compiled (it requires some significant plumbing through multiple layers though, because we remove things on multiple layers) - but I don't think that actually gives any actionable information. Providing things like size is impossible, because we can't give size estimates without actually including the thing into snapshot.

If things are tree-shaken why should anybody care about it? People should focus on what is not tree-shaken and why.

[mehmetf]

I am not entirely sure where we landed with this issue so I wanted to clarify. Do you intend to experiment with size attribution of data portion of the snapshot? or do you think it would be misleading and thus would like us to use the existing diff tool?

This is not a must-have, but I wanted to ask: Do we have something that works directly on AOT files rather than instruction-sizes output? (like dexdump).

We would like to integrated the diff tool with internal tooling (go/apkdiffsample). This tool looks at APK contents and uses tools that ships with adb to analyze dex and resources. The nice thing is that it can be applied after the fact to release binaries even if they were not built with additional debugging flags. If we have to have instruction sizes output, it would be tougher to integrate with tooling like this since normal Android builds do not produce these side targets. If it helps, it would be enough to simply list the (readable) symbols in the AOT file like Classes/Methods.

(Just to be clear, this does not replace the diff analysis tool discussed in this issue. I mentioned it here because it is related to size tracking)

[mraleph]

I am not entirely sure where we landed with this issue so I wanted to clarify.

I have been slowly working on this issue - I have a reasonable prototype working, and I will start landing pieces this week.

This is not a must-have, but I wanted to ask: Do we have something that works directly on AOT files rather than instruction-sizes output? (like dexdump).

We don't have such tool, and there are few challenges building one:

• snapshot format is not self descriptive and is tied to a specific VM version (and it is less stable than DEX, especially when we are working on code size)
• some of the connections between different objects in the snapshot are only explicit during snapshotting, after that they become implicit and require for example disassembling code to recover.
• we are actively working on droppings metainformation from the snapshot that allows to clearly identify where objects came from e.g. we have recently dropped some Function objects - some for some instructions you can no longer say which Dart function they originate from (at least not without looking at DWARF information)

Overall this sounds like a lot of complexity.

[jacob314]

Related tracking issue in the devtools repo:
flutter/devtools#2043