Skip to content

Build flatcc for the host #10855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 13, 2025
Merged

Build flatcc for the host #10855

merged 1 commit into from
May 13, 2025

Conversation

@jathu
Copy link
Contributor

@jathu jathu commented May 13, 2025 •
edited
Loading

Summary

  • Similar to what we did with flatc, let's build flatcc_cli for the host using ExternalProject
  • Move the flatcc definitions to be under third-party/
  • Move the etdump and bundled_program definitions to be under their respective folders
  • Clean up the build files and remove redundant things

Test plan

CI

$ ./install_executorch.sh

$ ./scripts/build_apple_frameworks.sh

$ rm -rf cmake-out \
   && cmake --preset macos-arm64 \
   && cmake --build cmake-out --parallel

$ rm -rf cmake-out \
   && cmake -DEXECUTORCH_BUILD_DEVTOOLS=ON -Dprotobuf_BUILD_TESTS=OFF -DEXECUTORCH_ENABLE_EVENT_TRACER=ON --preset macos-arm64 \
   && cmake --build cmake-out --parallel

cc @larryliu0820

@pytorch-bot
Copy link

pytorch-bot bot commented May 13, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10855

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ No Failures

As of commit 1fec681 with merge base f1ef702 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 13, 2025
@jathu jathu added module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch ciflow/trunk ciflow/binaries release notes: build Changes related to build, including dependency upgrades, build flags, optimizations, etc. and removed CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. labels May 13, 2025
@jathu jathu force-pushed the jathu/wip-flatcc branch from 13e88b0 to f5fabb6 Compare May 13, 2025 17:16
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 13, 2025
@jathu jathu force-pushed the jathu/wip-flatcc branch 3 times, most recently from 773a731 to 83850e3 Compare May 13, 2025 18:04
@jathu jathu force-pushed the jathu/wip-flatcc branch from 83850e3 to 1fec681 Compare May 13, 2025 18:19
@jathu jathu marked this pull request as ready for review May 13, 2025 18:19
@jathu
Copy link
Contributor Author

jathu commented May 13, 2025

cc @zingo I see that arm build scripts set FLATCC_EXECUTABLE. We should be able to remove that now. I'll leave the change up to you

@jathu jathu merged commit 518324f into main May 13, 2025
196 checks passed
@jathu jathu deleted the jathu/wip-flatcc branch May 13, 2025 21:01
This was referenced May 13, 2025
Merged
Merged
@zingo
Copy link
Collaborator

zingo commented May 13, 2025

Thanks for this change, we have an extra build and copy of that executable we now can clean up 🙏

jathu added a commit that referenced this pull request May 13, 2025
@jathu
Use the install method for flatc (#10859)
b20419d 
### Summary

I came across this "cleaner" approach when setting up flatcc
(#10855). So, let's do it for
flatc too.

### Test plan

CI

cc @larryliu0820
jathu added a commit that referenced this pull request May 13, 2025
@jathu
Remove EXECUTORCH_SEPARATE_FLATCC_HOST_PROJECT (#10860)
4a89327 
### Summary
I forgot to do this in #10855.
But since we build flatcc for the host now, we don't need this variable.

### Test plan

CI

```
$ rg EXECUTORCH_SEPARATE_FLATCC_HOST_PROJECT --hidden -g '!.git/'
```

cc @larryliu0820
@jathu jathu mentioned this pull request May 16, 2025
Merged
jathu added a commit that referenced this pull request May 16, 2025
@jathu
Fix libflatccrt race (#10918)
d18a52d 
### Summary

Seems like there is a race in building `libflatccrt.a`. This issue has
existed for a while: #7300.
It was temporarily mitigated in
#7570 by just reducing the
parallelism.

In this diff I attempt to fix it. This is just my assumption of what is
wrong. Given flatccrt builds a debug version with a `_d` suffix, if the
target isn't depended on (i.e. some target don't use the conditional
target name) then the order of how the lib is built causes a race. So
for now, always use the non-debug version.

Given it's a race, I was never able to repro the issue locally — I can't
guarantee this is the problem. However, it seems my recent changes in
#10855 has increased the
frequency of the problem in CI.

### Test plan

CI

cc @larryliu0820
hinriksnaer pushed a commit to hinriksnaer/executorch that referenced this pull request May 19, 2025
@jathu @hinriksnaer
Fix libflatccrt race (pytorch#10918)
aba884c 
### Summary

Seems like there is a race in building `libflatccrt.a`. This issue has
existed for a while: pytorch#7300.
It was temporarily mitigated in
pytorch#7570 by just reducing the
parallelism.

In this diff I attempt to fix it. This is just my assumption of what is
wrong. Given flatccrt builds a debug version with a `_d` suffix, if the
target isn't depended on (i.e. some target don't use the conditional
target name) then the order of how the lib is built causes a race. So
for now, always use the non-debug version.

Given it's a race, I was never able to repro the issue locally — I can't
guarantee this is the problem. However, it seems my recent changes in
pytorch#10855 has increased the
frequency of the problem in CI.

### Test plan

CI

cc @larryliu0820
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kirklandsign kirklandsign kirklandsign approved these changes

@swolchok swolchok Awaiting requested review from swolchok

@mergennachin mergennachin Awaiting requested review from mergennachin

@SS-JIA SS-JIA Awaiting requested review from SS-JIA

@Gasoonjia Gasoonjia Awaiting requested review from Gasoonjia

@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820 larryliu0820 is a code owner

Assignees

No one assigned

Labels

ciflow/binaries ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch release notes: build Changes related to build, including dependency upgrades, build flags, optimizations, etc.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jathu @zingo @kirklandsign @facebook-github-bot