Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Commit ffacad9

Browse files
jtmcdolenavaronbracke
jtmcdole
and
navaronbracke
authored
La Grande Fusion Notes - History Editing (#55667)
AKA "Dude Where's My Files?" Document the history editing that will occur during the monorepo merge. `flutter/engine` will remain the source of truth of revisions older than the merge. --------- Co-authored-by: Navaron Bracke <[email protected]>
1 parent 683a14c commit ffacad9

File tree

1 file changed

+260
-0
lines changed

1 file changed

+260
-0
lines changed

‎docs/monorepo/history_strategy.md‎

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
# Monorepo History Pruning
2+
3+
These are the steps we will/did follow to prune excessive history from the [flutter/engine](https://github.com/flutter/engine) repository when we merged with [flutter/flutter](https://github.com/flutter/flutter). The idea was to retain as much useful history as possible without blowing up the footprint of the framework's `.git` folder. The history that should get merged should be as relative and useful to currently engine development.
4+
5+
The engine `.git` folder is ~780MB of history.
6+
7+
* Binary files were checked in that are not used anymore.
8+
* Third party librariers were checked in and removed nearly a decade ago.
9+
* Examples were created and later moved elsewhere.
10+
11+
## Step 1: Fresh Clone + Safety
12+
13+
Do not start with your working tree.
14+
Remove the origin so we don't mess with the `flutter/engine`.
15+
16+
```shell
17+
##############################################
18+
## Do some cleanup work on the engine and get
19+
## the folder structure right.
20+
##############################################
21+
22+
# clone the repo to a fresh working folder
23+
git clone [email protected]:flutter/engine.git engine_prep
24+
cd engine_prep
25+
26+
# for saftey - remove the remote - we're going to edit history
27+
git remote remove origin
28+
```
29+
30+
### Optional - Analyze the repo
31+
32+
If you want to analyze the repository, you should intall [git filter-repo](http://github.com/newren/git-filter-repo) on your path and then run:
33+
34+
```shell
35+
# Analyze if you want, just remember to remove .git/filter-repo
36+
git filter-repo --analyze --force
37+
```
38+
39+
The output is stored in `.git/filter-repo`.
40+
41+
## Step 2: Prune the History
42+
43+
The following table is pulled from git-filter-repo's analsis. The `Packed Size` due to cross referencing. In general; we looked at large files that are not referenced any more and folders older than 2016.
44+
45+
| Packed Size | Deleted Date | Path | Notes |
46+
|-------------|:-------------|:----------------------------------------|:-----------------------------|
47+
| 112784745 | 2024-05-13 | ci/licenses_golden/licenses_third_party | |
48+
| 27531902 | ~2021 | *.jar | binary |
49+
| 27379931 | 2016-08-09 | third_party/android_platform | android_platform and webview |
50+
| 27000000 | 2024-07-15 | impeller/docs/assets/*.(png|gif) | moved to another repository |
51+
| 15121375 | 2023-02-13 | *.ttc | font files |
52+
| 10104182 | 2023-02-13 | */SourceHanSerifCN* | |
53+
| 7985682 | 2018-08-08 | travis | old ci |
54+
| 6315637 | 2015-11-07 | examples/game | |
55+
| 3939429 | 2015-07-28 | sky/sdk | |
56+
| 3939429 | 2015-07-28 | sky/packages/sky | |
57+
| 3903787 | 2016-08-09 | mojo | |
58+
| 3686830 | 2022-06-14 | testing/scenario_app/android/reports | |
59+
| 3188930 | 2015-06-30 | tests/fast | |
60+
| 3173966 | 2015-08-07 | */example/game* | |
61+
| 2018961 | 2016-08-09 | third_party/libxml | |
62+
| 1804199 | 2016-08-09 | third_party/tcmalloc | |
63+
| 1393936 | ~2016 | *.dll | binary |
64+
| 1373740 | 2017-07-06 | tests/data | |
65+
| 1100665 | 2015-06-27 | benchmarks/parser/resources/html5.html | |
66+
| 1059673 | 2015-07-20 | third_party/protobuf | |
67+
| 978870 | 2022-04-27 | impeller/third_party | |
68+
| 798852 | 2015-07-20 | third_party/cython | |
69+
| 778560 | 2022-01-24 | lib/web_ui/test/golden_files | |
70+
| 634455 | 2016-08-09 | third_party/libpng | |
71+
| 610751 | 2024-05-13 | .golden | |
72+
| 550475 | 2024-09-17 | impeller/fixtures/flutter_logo_baked.* | |
73+
| 526837 | 2016-08-09 | third_party/libevent | |
74+
| 523436 | 2015-07-20 | third_party/boringssl | |
75+
| 514968 | 2022-04-27 | impeller/fixtures/image.png | |
76+
| 461527 | 2015-12-11 | third_party/re2 | |
77+
| 418122 | 2015-10-12 | examples/demo_launcher | |
78+
| 413787 | 2015-11-07 | .aac | |
79+
| 362787 | 2016-08-09 | third_party/glfw | |
80+
| 349604 | 2016-08-09 | third_party/harfbuzz-ng | |
81+
| 340869 | 2016-08-09 | third_party/okhttp | |
82+
| 321659 | 2016-08-09 | .S | |
83+
| 300824 | 2016-08-09 | .so | |
84+
| 257633 | 2016-08-09 | third_party/libjpeg | |
85+
| 257519 | 2016-08-09 | third_party/jinja2 | |
86+
| 249618 | 2016-08-09 | third_party/zlib | |
87+
| 218643 | 2015-12-11 | third_party/brotli | |
88+
| 188622 | 2021-01-06 | .idl | |
89+
| 184593 | 2015-09-02 | third_party/khronos | |
90+
| 173210 | 2016-08-09 | .gypi | |
91+
| 170484 | 2016-08-09 | third_party/expat | |
92+
| 169578 | 2016-08-09 | .asm | |
93+
| 161360 | 2016-08-09 | .m4 | |
94+
| 142670 | 2018-05-10 | .in | |
95+
| 140364 | 2015-12-11 | third_party/ots | |
96+
| 137270 | 2016-08-09 | .hh | |
97+
| 136787 | 2016-08-09 | .gyp | |
98+
| 99503 | 2016-08-09 | third_party/qcms | |
99+
| 91730 | 2015-08-21 | .pxd | |
100+
| 84850 | 2016-08-09 | third_party/yasm | |
101+
102+
The following command will remove files and foldes from the checkout history. Since this is a destructive edit, the SHA1 git hashes will be changed in the process. At the end, the `.git` history will be 74 MB of object files.
103+
104+
```shell
105+
# Lets do some heavy filtering;
106+
# .git starts out at ~780MB and ends up at ~110MB
107+
git filter-repo --force --invert-paths \
108+
--path-glob 'impeller/docs/assets/*.png' \
109+
--path-glob 'impeller/docs/assets/*.gif' \
110+
--path-glob '*/example/game/*' \
111+
--path-glob 'benchmarks/parser/resources/html5.html' \
112+
--path-glob '*.dll' \
113+
--path-glob '*.jar' \
114+
--path-glob '*/SourceHanSerifCN*' \
115+
--path-glob 'third_party/txt/third_party/fonts/NotoSansCJK-Regular.ttc' \
116+
--path-glob 'impeller/fixtures/flutter_logo_baked.*' \
117+
--path-glob 'impeller/fixtures/image.png' \
118+
--path-glob '*.golden' \
119+
--path-glob '*.aac' \
120+
--path-glob '*.S' \
121+
--path-glob '*.so' \
122+
--path-glob '*.idl' \
123+
--path-glob '*.gpy' \
124+
--path-glob '*.gypi' \
125+
--path-glob '*.asm' \
126+
--path-glob '*.m4' \
127+
--path-glob '*.in' \
128+
--path-glob '*.pxd' \
129+
--path-glob '*.hh' \
130+
--path 'ci/licenses_golden/licenses_third_party' \
131+
--path 'testing/scenario_app/android/reports' \
132+
--path 'impeller/third_party' \
133+
--path 'mojo/public/third_party' \
134+
--path 'tests/data' \
135+
--path 'tests/fast' \
136+
--path 'tests/framework' \
137+
--path 'travis' \
138+
--path 'mojo' \
139+
--path 'sky/sdk' \
140+
--path 'sky/engine' \
141+
--path 'sky/tools/webkitpy' \
142+
--path 'sky/shell' \
143+
--path 'sky/packages/sky' \
144+
--path 'sky/tests' \
145+
--path 'sky/unit' \
146+
--path 'sky/services' \
147+
--path 'sky/compositor' \
148+
--path 'sky/build' \
149+
--path 'sky/specs' \
150+
--path 'skysprites' \
151+
--path 'examples/demo_launcher' \
152+
--path 'examples/game' \
153+
--path 'third_party/qcms' \
154+
--path 'third_party/libevent' \
155+
--path 'third_party/boringssl' \
156+
--path 'third_party/tcmalloc' \
157+
--path 'third_party/cython' \
158+
--path 'third_party/protobuf' \
159+
--path 'third_party/libpng' \
160+
--path 'third_party/re2' \
161+
--path 'third_party/harfbuzz-ng' \
162+
--path 'third_party/jinja2' \
163+
--path 'third_party/libjpeg' \
164+
--path 'third_party/glfw' \
165+
--path 'third_party/zlib' \
166+
--path 'third_party/android_platform' \
167+
--path 'third_party/expat' \
168+
--path 'third_party/brotli' \
169+
--path 'third_party/yasm' \
170+
--path 'third_party/khronos' \
171+
--path 'third_party/okhttp' \
172+
--path 'third_party/libxml' \
173+
--path 'third_party/ots' \
174+
--path 'third_party/libXNVCtrl' \
175+
--path 'lib/web_ui/test/golden_files' \
176+
--path 'apk' \
177+
--path 'flutter' \
178+
--path 'base' \
179+
--path 'sdk' \
180+
--path 'gpu' \
181+
--path 'engine' \
182+
--path 'tools/webkitpy' \
183+
--path 'tools/valgrind' \
184+
--path 'tools/clang' \
185+
--path 'tools/android' \
186+
--path 'build/linux' \
187+
--path 'build/win' \
188+
--path 'build/mac' \
189+
--path 'ui' \
190+
--path 'examples/stocks' \
191+
--path 'examples/stocks2' \
192+
--path 'examples/stocks-fn' \
193+
--path 'examples/data' \
194+
--path 'examples/fitness' \
195+
--path 'examples/city-list' \
196+
--path 'examples/widgets' \
197+
--path 'examples/raw' \
198+
--path 'examples/color' \
199+
--path 'examples/flights' \
200+
--path 'examples/rendering' \
201+
--path 'examples/fn' \
202+
--path 'specs' \
203+
--path 'url' \
204+
--path 'services' \
205+
--path 'framework' \
206+
--path 'crypto' \
207+
--path 'skia/ext' \
208+
--path 'e2etests' \
209+
--path 'tests/resources' \
210+
--path 'viewer' \
211+
--path 'lib/stub_ui' \
212+
--path 'content_handler'
213+
214+
# Garbage collect!
215+
git reflog expire --expire=now --all && git gc --prune=now --aggressive
216+
```
217+
218+
## Step 3 - Rewirte directories
219+
220+
The final destination for the engine source code will be in the directory `engine/src/flutter` *except* for `DEPS` which remains at the root. Using `git mv` only affects HEAD and can have some problems when logging. Instead we'll re-write history so it makes sense in the new world.
221+
222+
```shell
223+
# Move files to engine/src/flutter, update tags so they don't collide, and move DEPS back to root.
224+
git filter-repo --to-subdirectory-filter engine/src/flutter --tag-rename '':'engine-' --force
225+
git filter-repo --path-rename engine/src/flutter/DEPS:DEPS
226+
```
227+
228+
## Step 4 - Rewrite the PR links
229+
230+
The PR link in the first line of the comment message will be wrong; `flutter/flutter` doesn't have the same history. To make history a little bit better, we only want to edit the first line. This must be done before we merge with the `flutter/flutter` repo so as not to step on their commit lines.
231+
232+
```shell
233+
git filter-repo --force --message-callback '
234+
return re.sub(br"^(.*)\((#\d+)\)\n(.*)", br"\1(flutter/engine\2)\n\3", message, 1)
235+
'
236+
```
237+
238+
## Execute Order 42: Merge The Repositories
239+
240+
```shell
241+
##############################################
242+
## Now handle merging into flutter/flutter
243+
##############################################
244+
245+
git clone [email protected]:flutter/flutter.git flutter_merge
246+
cd flutter_merge
247+
248+
# add the other tree as remote
249+
git remote add -f engine-upstream ~/src/engine_prep
250+
251+
# --no-commit is important because we want to look around
252+
git merge --no-commit --allow-unrelated-histories engine-upstream/main
253+
254+
# You're a wizard, Harry
255+
git commit -m "Merge flutter/engine into framework"
256+
257+
# Garbage collect!
258+
# Now at 234MB .git
259+
git reflog expire --expire=now --all && git gc --prune=now --aggressive
260+
```

0 commit comments

Comments
 (0)