Skip to content

Use zlib compression in DIEDistinctData block to reduce size of distinct data in the debug info representation of MCCAS #7847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 19, 2023
Merged

Use zlib compression in DIEDistinctData block to reduce size of distinct data in the debug info representation of MCCAS #7847

merged 1 commit into from
Dec 19, 2023

Conversation

rastogishubham
Copy link

@rastogishubham rastogishubham commented Dec 5, 2023 •
edited
Loading

To reduce the size of DIEDistinctData, which accounts for over 50% of the size of a CAS after a 10 day build, a cheap way to get some space savings is by using zlib compression, which seems to yield some impressive results.

Size of MCCAS before this patch:

Kind                        Count            Parents           Children           Data (B)           Cost (B)        
====                        =====            =======           ========           ========           ========        
builtin:node                   45   0.00%     18,078   0.00%         87   0.00%        651   0.00%      2,787   0.00%
builtin:tree                5,784   0.11%      6,524   0.00%     29,795   0.01%  1,154,032   0.01%  1,577,480   0.01%
mc:addends                  9,454   0.18%     17,400   0.00%          0   0.00% 96,786,014   0.76% 97,088,542   0.59%
mc:align                       30   0.00%     16,685   0.00%          0   0.00%        210   0.00%      1,170   0.00%
mc:assembler               17,991   0.34%     23,271   0.01%     89,955   0.02%    485,757   0.00%  1,781,109   0.01%
mc:atom                   207,275   3.89%  3,693,945   0.84%  1,096,600   0.25%  2,105,126   0.02% 17,510,726   0.11%
mc:cstring                974,531  18.31%  4,731,871   1.08%          0   0.00% 128,249,985   1.01% 159,434,977   0.97%
mc:data                   285,008   5.36%    327,661   0.07%          0   0.00% 2,211,993,525  17.35% 2,221,113,781  13.52%
mc:data_in_code             8,350   0.16%     17,991   0.00%          0   0.00% 218,377,446   1.71% 218,644,646   1.33%
mc:debug_DIE_abbrev           694   0.01%    942,494   0.22%          0   0.00%     10,149   0.00%     32,357   0.00%
mc:debug_DIE_abbrev_set      4,213   0.08%     17,632   0.00%    942,494   0.22%      4,213   0.00%  7,678,981   0.05%
mc:debug_DIE_data         230,233   4.33% 53,139,473  12.13% 53,121,841  12.13% 76,594,454   0.60% 508,936,638   3.10%
mc:debug_DIE_distinct_data     14,729   0.28%     17,632   0.00%          0   0.00% 8,244,307,631  64.66% 8,244,778,959  50.20%
mc:debug_DIE_top_level     17,632   0.33%     17,632   0.00%     52,896   0.01%     17,632   0.00%  1,005,024   0.01%
mc:debug_abbrev_section          1   0.00%     17,990   0.00%          1   0.00%          2   0.00%         42   0.00%
mc:debug_info_section      17,632   0.33%     17,990   0.00%     35,264   0.01% 190,307,338   1.49% 191,153,674   1.16%
mc:debug_line             195,018   3.66%  8,393,070   1.92%          0   0.00% 30,597,691   0.24% 36,838,267   0.22%
mc:debug_line_distinct_data      5,879   0.11%     11,115   0.00%          0   0.00% 143,614,219   1.13% 143,802,347   0.88%
mc:debug_line_section      11,115   0.21%     17,990   0.00%  8,415,300   1.92% 152,616,004   1.20% 220,294,084   1.34%
mc:debug_string         2,444,020  45.92% 365,386,095  83.42%          0   0.00% 302,874,577   2.38% 381,083,217   2.32%
mc:debug_string_section     11,951   0.22%     17,990   0.00% 365,398,046  83.42%     23,902   0.00% 2,923,590,702  17.80%
mc:fill                       206   0.00%      3,120   0.00%          0   0.00%        936   0.00%      7,528   0.00%
mc:group                   17,991   0.34%     17,991   0.00%    292,958   0.07%     35,987   0.00%  2,955,363   0.02%
mc:header                  15,975   0.30%     17,991   0.00%          0   0.00% 25,592,415   0.20% 26,103,615   0.16%
mc:leb                          8   0.00%         23   0.00%          0   0.00%         16   0.00%        272   0.00%
mc:merged_fragment        743,925  13.98%    749,111   0.17%          0   0.00% 650,589,415   5.10% 674,395,015   4.11%
mc:padding                     16   0.00%    119,864   0.03%          0   0.00%         32   0.00%        544   0.00%
mc:section                 79,165   1.49%    220,998   0.05%  3,790,510   0.87% 274,664,521   2.15% 307,521,881   1.87%
mc:symbol_table             3,261   0.06%     17,991   0.00%  4,731,871   1.08%      9,121   0.00% 37,968,441   0.23%
TOTAL                   5,322,132 100.00% 437,997,618 100.00% 437,997,618 100.00% 12,751,013,001 100.00% 16,425,302,169 100.00%

num-tiny-objects           871238
sec-ref-size            274585356
atom-ref-size             1897851

Size of MCCAS after this patch:

Kind                        Count            Parents           Children           Data (B)           Cost (B)        
====                        =====            =======           ========           ========           ========        
builtin:node                   45   0.00%     18,078   0.00%         87   0.00%        651   0.00%      2,787   0.00%
builtin:tree                5,784   0.11%      6,524   0.00%     29,795   0.01%  1,154,032   0.01%  1,577,480   0.01%
mc:addends                  9,454   0.18%     17,400   0.00%          0   0.00% 96,786,014   1.16% 97,088,542   0.81%
mc:align                       30   0.00%     16,685   0.00%          0   0.00%        210   0.00%      1,170   0.00%
mc:assembler               17,991   0.34%     23,271   0.01%     89,955   0.02%    485,757   0.01%  1,781,109   0.01%
mc:atom                   207,328   3.89%  3,693,988   0.84%  1,098,313   0.25%  2,105,232   0.03% 17,526,232   0.15%
mc:cstring                974,531  18.31%  4,731,871   1.08%          0   0.00% 128,249,985   1.53% 159,434,977   1.32%
mc:data                   285,008   5.35%    327,661   0.07%          0   0.00% 2,212,003,788  26.41% 2,221,124,044  18.44%
mc:data_in_code             8,407   0.16%     17,991   0.00%          0   0.00% 219,268,311   2.62% 219,537,335   1.82%
mc:debug_DIE_abbrev           694   0.01%    942,494   0.22%          0   0.00%     10,149   0.00%     32,357   0.00%
mc:debug_DIE_abbrev_set      4,213   0.08%     17,634   0.00%    942,494   0.22%      4,213   0.00%  7,678,981   0.06%
mc:debug_DIE_data         230,240   4.32% 53,132,628  12.13% 53,114,994  12.13% 76,591,667   0.91% 508,879,299   4.22%
mc:debug_DIE_distinct_data     14,729   0.28%     17,634   0.00%          0   0.00% 3,864,425,670  46.15% 3,864,896,998  32.08%
mc:debug_DIE_top_level     17,634   0.33%     17,634   0.00%     52,902   0.01%     17,634   0.00%  1,005,138   0.01%
mc:debug_abbrev_section          1   0.00%     17,990   0.00%          1   0.00%          2   0.00%         42   0.00%
mc:debug_info_section      17,634   0.33%     17,990   0.00%     35,268   0.01% 190,270,138   2.27% 191,116,570   1.59%
mc:debug_line             195,018   3.66%  8,393,070   1.92%          0   0.00% 30,597,691   0.37% 36,838,267   0.31%
mc:debug_line_distinct_data      5,879   0.11%     11,115   0.00%          0   0.00% 143,631,856   1.72% 143,819,984   1.19%
mc:debug_line_section      11,115   0.21%     17,990   0.00%  8,415,300   1.92% 152,616,501   1.82% 220,294,581   1.83%
mc:debug_string         2,444,020  45.91% 365,386,095  83.42%          0   0.00% 303,457,498   3.62% 381,666,138   3.17%
mc:debug_string_section     11,951   0.22%     17,990   0.00% 365,398,046  83.43%     23,902   0.00% 2,923,590,702  24.27%
mc:fill                       206   0.00%      3,120   0.00%          0   0.00%        936   0.00%      7,528   0.00%
mc:group                   17,991   0.34%     17,991   0.00%    292,958   0.07%     35,987   0.00%  2,955,363   0.02%
mc:header                  15,975   0.30%     17,991   0.00%          0   0.00% 25,592,415   0.31% 26,103,615   0.22%
mc:leb                          8   0.00%         23   0.00%          0   0.00%         16   0.00%        272   0.00%
mc:merged_fragment        745,400  14.00%    750,824   0.17%          0   0.00% 652,053,691   7.79% 675,906,491   5.61%
mc:padding                     16   0.00%    119,916   0.03%          0   0.00%         32   0.00%        544   0.00%
mc:section                 79,215   1.49%    220,998   0.05%  3,790,603   0.87% 274,664,611   3.28% 307,524,315   2.55%
mc:symbol_table             3,261   0.06%     17,991   0.00%  4,731,871   1.08%      9,121   0.00% 37,968,441   0.32%
TOTAL                   5,323,778 100.00% 437,992,587 100.00% 437,992,587 100.00% 8,374,057,710 100.00% 12,048,359,302 100.00%

We reduce the size of the debug_DIE_distinct_data block from 8.2 GB to 3.8 GB, or 53.6%!

The patch also doesn't seem to add too much time to replay either:

Before this patch, replay of commit e31d893: 24.907 s
After this patch, replay of commit e31d893: 25.239 s

@rastogishubham rastogishubham marked this pull request as ready for review December 6, 2023 19:34
@rastogishubham rastogishubham changed the title Compression Use zlib compression in DIEDistinctData block to reduce size of distinct data in the debug info representation of MCCAS Dec 6, 2023
felipepiovezan
felipepiovezan reviewed Dec 8, 2023
View reviewed changes
Copy link

@felipepiovezan felipepiovezan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be a win-win situation!

adrian-prantl
adrian-prantl approved these changes Dec 15, 2023
View reviewed changes
felipepiovezan
felipepiovezan approved these changes Dec 18, 2023
View reviewed changes
Copy link

@felipepiovezan felipepiovezan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@rastogishubham
Copy link
Author

@swift-ci please test

@rastogishubham
Copy link
Author

@swift-ci please test llvm

@rastogishubham rastogishubham merged commit 7702cfd into swiftlang:next Dec 19, 2023
rastogishubham added a commit that referenced this pull request Jan 3, 2024
@rastogishubham
Merge pull request #7910 from rastogishubham/CompressionStable
d0c3489 
Cherry pick PR #7847 into stable/20230725
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@felipepiovezan felipepiovezan felipepiovezan approved these changes

@adrian-prantl adrian-prantl Awaiting requested review from adrian-prantl

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rastogishubham @adrian-prantl @felipepiovezan