llvm-profdata should report errors on unreasonable block counts

[aeubanks]

https://crbug.com/1473604 is a bug where a miscompile resulted in instrumented profiles having very unreasonable block counts, e.g.

% ~/src/llvm-project/out/gn/bin/llvm-profdata show  --detailed-summary /tmp/bad/default-6392451879272816540_0.profraw 
Instrumentation level: IR  entry_first = 0
Total functions: 871932
Maximum function count: 49315915
Maximum internal block count: 12297832523381840770
Total number of blocks: 4448106
Total count: 6148943734960307878
Detailed summary:
2 blocks with count >= 12297832523381840770 account for 1 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 10 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 20 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 30 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 40 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 50 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 60 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 70 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 80 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 90 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 95 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 99 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 99.9 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 99.99 percentage of the total counts.
2 blocks with count >= 12297832523381840770 account for 99.999 percentage of the total counts.
4 blocks with count >= 8727880088090 account for 99.9999 percentage of the total counts.

this would have been easier to debug if llvm-profdata reported unreasonable block counts

@huangjd (not sure who else to cc)

[huangjd]

This sounds like a good idea and in fact a lot of data in the profile can be sanity checked for errors (for example total count doesn't add up). However we need to come up with a definition of "unreasonableness", like what if this data is actually intended? What if there has been a known (and won't-fix) systematic issue with the instrumented profile generator that keep feeding problematic data to llvm-profgen, and users don't want to be spammed by such warnings? I think at least this feature should be a separate functionality in llvm-profdata.

As for deciding unreasonable block count, we can simply use a relative or absolute cutoff, or use some statistics to determine outliers, or even throw in ML models to find out any function with suspected erroneous sample data. What do you think is a reasonable level of implementation?

[aeubanks]

I'm not super familiar with what's available in the profiles, but one very easy thing that probably would have helped us is reporting very large block counts. It seems like block counts shouldn't be nearing the max of 2^64, so warning when a block count is > 2^63 seems useful as a first check. If people have legitimate use cases where they're hitting that block count then we should revisit, but I think we should add a check like that and commit it, and revert if people have legitimate use cases for it.

Agreed that this check should be under a flag, but I think the flag should be on by default.

[zmodem]

As for deciding unreasonable block count, we can simply use a relative or absolute cutoff, or use some statistics to determine outliers, or even throw in ML models to find out any function with suspected erroneous sample data. What do you think is a reasonable level of implementation?

I think just an absolute cutoff should be good enough.

It seems like block counts shouldn't be nearing the max of 2^64, so warning when a block count is > 2^63 seems useful as a first check.

I think we can go lower (to increase the chance of detecting a bad value). Let's say a block runs 10^9 times per second for a year; that would still yield a count less than 2^56.

Agreed that this check should be under a flag, but I think the flag should be on by default.

+1

@ZequanWu could you help with this?