Skip to content

Project with cascading files benchmarks - compare effect on signature files on incremental build #14136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Oct 19, 2022
Merged

Project with cascading files benchmarks - compare effect on signature files on incremental build #14136

merged 10 commits into from
Oct 19, 2022

Conversation

@T-Gro
Copy link
Member

@T-Gro T-Gro commented Oct 18, 2022

This set of benchmarks validates that the existence of .fsi files does have multi-fold impact on the speed of incremental build.
That is, especially if the file being modified happens to be between beginning and middle of the project.

The benchmark was running in 2^3= configurations corresponding to arguments of FsCheck, and the fact whether .fsi files were added to the compilation or not.
enablePartialTypeChecking = this.PartialCheck,
enableParallelCheckingWithSignatureFiles = this.ParaChecking

If you focus on a specific example, let's say **ParseProjectWithChangingMiddleFile ** which changes the middle file of a 128-files large project, you can see the differences across these 3 boolean configs.
In this case, the impact on the speed of incremental build can be up to 2-3x in terms of speed gains.

BenchmarkDotNet=v0.13.2, OS=Windows 11 (10.0.22621.674)
11th Gen Intel Core i9-11950H 2.60GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK=7.0.100-rc.2.22477.23
  [Host]   : .NET 7.0.0 (7.0.22.47203), X64 RyuJIT AVX2 DEBUG
  ShortRun : .NET 7.0.0 (7.0.22.47203), X64 RyuJIT AVX2

Job=ShortRun  IterationCount=3  LaunchCount=1  
WarmupCount=3
Method PartialCheck ParaChecking GenerateFSI Mean Error StdDev Ratio RatioSD Rank Gen0 Gen1 Gen2 Allocated Alloc Ratio
ParseAndCheckLastFileProjectAsIs False False False 11.027 ms 10.202 ms 0.5592 ms 0.01 0.00 I - - - 120.92 KB 0.000
ParseProjectWithChangingPenultimateFile False False False 25.373 ms 17.441 ms 0.9560 ms 0.02 0.00 II 656.2500 281.2500 - 8402.02 KB 0.019
ParseProjectWithChangingMiddleFile False False False 364.452 ms 894.278 ms 49.0184 ms 0.35 0.07 III 14000.0000 2000.0000 - 179193.09 KB 0.416
ParseProjectWithChangingFirstFile False False False 642.130 ms 804.596 ms 44.1026 ms 0.62 0.05 IV 28000.0000 3000.0000 - 350922.82 KB 0.814
ParseProjectWithFullCacheClear False False False 1,046.582 ms 1,394.643 ms 76.4451 ms 1.00 0.00 V 35000.0000 9000.0000 2000.0000 430979.57 KB 1.000
ParseAndCheckLastFileProjectAsIs False False True 21.822 ms 38.597 ms 2.1156 ms 0.02 0.00 I - - - 228.33 KB 0.001
ParseProjectWithChangingPenultimateFile False False True 43.916 ms 36.434 ms 1.9971 ms 0.05 0.00 II 692.3077 230.7692 - 9320.81 KB 0.023
ParseProjectWithChangingMiddleFile False False True 408.622 ms 861.017 ms 47.1952 ms 0.44 0.05 III 13000.0000 2000.0000 - 167302.96 KB 0.412
ParseProjectWithChangingFirstFile False False True 735.510 ms 1,964.457 ms 107.6785 ms 0.79 0.12 IV 26000.0000 5000.0000 - 325652.77 KB 0.802
ParseProjectWithFullCacheClear False False True 929.975 ms 81.306 ms 4.4567 ms 1.00 0.00 V 33000.0000 10000.0000 2000.0000 406068.07 KB 1.000
ParseAndCheckLastFileProjectAsIs False True False 9.807 ms 7.528 ms 0.4126 ms 0.01 0.00 I - - - 118.53 KB 0.000
ParseProjectWithChangingPenultimateFile False True False 22.698 ms 3.642 ms 0.1996 ms 0.02 0.00 II 656.2500 281.2500 - 8453.61 KB 0.020
ParseProjectWithChangingMiddleFile False True False 338.556 ms 334.933 ms 18.3588 ms 0.37 0.01 III 14000.0000 2000.0000 - 179077.58 KB 0.415
ParseProjectWithChangingFirstFile False True False 612.335 ms 1,075.664 ms 58.9607 ms 0.67 0.02 IV 28000.0000 3000.0000 - 351421.38 KB 0.815
ParseProjectWithFullCacheClear False True False 915.377 ms 1,150.000 ms 63.0354 ms 1.00 0.00 V 35000.0000 9000.0000 2000.0000 431008.76 KB 1.000
ParseAndCheckLastFileProjectAsIs False True True 19.375 ms 15.346 ms 0.8412 ms 0.02 0.00 I - - - 230.86 KB 0.001
ParseProjectWithChangingPenultimateFile False True True 35.793 ms 12.569 ms 0.6890 ms 0.04 0.00 II 714.2857 214.2857 - 9303.83 KB 0.023
ParseProjectWithChangingMiddleFile False True True 340.513 ms 157.150 ms 8.6139 ms 0.36 0.02 III 13000.0000 2000.0000 - 167654.45 KB 0.413
ParseProjectWithChangingFirstFile False True True 685.282 ms 339.027 ms 18.5832 ms 0.72 0.03 IV 26000.0000 5000.0000 - 325979.35 KB 0.803
ParseProjectWithFullCacheClear False True True 958.442 ms 541.981 ms 29.7078 ms 1.00 0.00 V 33000.0000 10000.0000 2000.0000 406067.13 KB 1.000
ParseAndCheckLastFileProjectAsIs True False False 8.724 ms 1.675 ms 0.0918 ms 0.01 0.00 I - - - 119.38 KB 0.000
ParseProjectWithChangingPenultimateFile True False False 23.033 ms 5.042 ms 0.2764 ms 0.03 0.00 II 656.2500 281.2500 - 8410.91 KB 0.019
ParseProjectWithChangingMiddleFile True False False 312.968 ms 502.321 ms 27.5339 ms 0.41 0.05 III 14000.0000 2000.0000 - 177325.7 KB 0.403
ParseProjectWithChangingFirstFile True False False 508.210 ms 123.797 ms 6.7857 ms 0.67 0.03 IV 28000.0000 3000.0000 - 347377.86 KB 0.789
ParseProjectWithFullCacheClear True False False 764.298 ms 703.481 ms 38.5601 ms 1.00 0.00 V 36000.0000 11000.0000 4000.0000 440120.64 KB 1.000
ParseAndCheckLastFileProjectAsIs True False True 17.545 ms 9.458 ms 0.5184 ms 0.04 0.00 I - - - 133.31 KB 0.001
ParseProjectWithChangingPenultimateFile True False True 28.262 ms 7.366 ms 0.4038 ms 0.07 0.00 II 312.5000 156.2500 - 4177.03 KB 0.028
ParseProjectWithChangingMiddleFile True False True 115.049 ms 224.340 ms 12.2968 ms 0.28 0.03 III 2800.0000 800.0000 - 35153.53 KB 0.234
ParseProjectWithChangingFirstFile True False True 192.335 ms 152.500 ms 8.3590 ms 0.46 0.01 IV 4666.6667 1000.0000 - 60968.41 KB 0.405
ParseProjectWithFullCacheClear True False True 417.720 ms 190.919 ms 10.4649 ms 1.00 0.00 V 12000.0000 7000.0000 3000.0000 150548.34 KB 1.000
ParseAndCheckLastFileProjectAsIs True True False 9.145 ms 9.866 ms 0.5408 ms 0.01 0.00 I - - - 116.28 KB 0.000
ParseProjectWithChangingPenultimateFile True True False 24.712 ms 18.141 ms 0.9944 ms 0.03 0.00 II 656.2500 281.2500 - 8406.78 KB 0.019
ParseProjectWithChangingMiddleFile True True False 307.731 ms 474.400 ms 26.0035 ms 0.34 0.03 III 14000.0000 2000.0000 - 177520.01 KB 0.404
ParseProjectWithChangingFirstFile True True False 524.612 ms 191.084 ms 10.4740 ms 0.58 0.02 IV 28000.0000 3000.0000 - 347610.17 KB 0.790
ParseProjectWithFullCacheClear True True False 905.933 ms 191.779 ms 10.5121 ms 1.00 0.00 V 35000.0000 10000.0000 4000.0000 439927.7 KB 1.000
ParseAndCheckLastFileProjectAsIs True True True 18.328 ms 2.036 ms 0.1116 ms 0.04 0.00 I - - - 136.55 KB 0.001
ParseProjectWithChangingPenultimateFile True True True 33.871 ms 20.096 ms 1.1015 ms 0.08 0.01 II 312.5000 156.2500 - 4195.5 KB 0.028
ParseProjectWithChangingMiddleFile True True True 119.979 ms 96.769 ms 5.3043 ms 0.29 0.02 III 2750.0000 750.0000 - 35245.54 KB 0.234
ParseProjectWithChangingFirstFile True True True 160.949 ms 109.366 ms 5.9947 ms 0.39 0.02 IV 4666.6667 1000.0000 - 60744.59 KB 0.403
ParseProjectWithFullCacheClear True True True 409.830 ms 485.428 ms 26.6080 ms 1.00 0.00 V 12000.0000 7000.0000 3000.0000 150609.31 KB 1.000

0101
0101 previously approved these changes Oct 18, 2022
View reviewed changes
@T-Gro T-Gro requested a review from 0101 October 18, 2022 16:08
psfinaki
psfinaki approved these changes Oct 18, 2022
View reviewed changes
Copy link
Contributor

@psfinaki psfinaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. So I guess generally this means we should first speed up signature files before focusing on their convenient auto-generation?

@T-Gro T-Gro merged commit fbfc406 into main Oct 19, 2022
@allisonchou allisonchou mentioned this pull request Oct 24, 2022
Closed
@T-Gro T-Gro deleted the feature/project-with-cascading-files-benchmarks branch February 2, 2023 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vzarytovskii vzarytovskii Awaiting requested review from vzarytovskii

@abonie abonie Awaiting requested review from abonie

2 more reviewers

@0101 0101 0101 approved these changes

@psfinaki psfinaki psfinaki approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@T-Gro @0101 @psfinaki