Adding Arm Ethos-U NPU support

[kristofer-jonsson-arm]

MPS3 AN547

• Updating mps3_an547 board files adding DTB entries for Ethos-U.
• Adding DTS bindings for the Ethos-U DTB entry.

Modules

• Adding module files for building the Arm Ethos-U Core driver.
• Updating TFLU module file to build Ethos-U operator.

Samples

• Adding TFLU Ethos-U sample program.

Manifest

• Adding hal_ethos_u to west manifest.

Signed-off-by: Kristofer Jonsson [email protected]
Signed-off-by: Fredrik Knutsson [email protected]

[github-actions]

The following west manifest projects have been modified in this Pull Request:

Name	Old Revision	New Revision	Diff
hal_ethos_u	N/A	zephyrproject-rtos/hal_ethos_u@90ada2e (main)	N/A

Note: This message is automatically posted and updated by the Manifest GitHub Action.

[microbuilder]

Just a heads up that the typical way to review changes like this is to generate a PR with the changes in the module, and then reference that PR in the west.yml manifest here via:

revision: pull/54/head

Where 54 is the PR number.

The changes were already pushed directly to hal_ethos_u so it's no longer relevant in this case, but it's useful to go through the PR route in case something also needs to be changed in the module code.

[mbolivar-nordic]

Can you please rebase?

[microbuilder]

@kristofer-jonsson-arm I have to setup the AN547 FVP image to properly test this since QEMU doesn't have U55 support, but looking through the code quickly, is tflu used elsewhere that you've seen? We've been using tflm in our APIs at Linaro, and it probably makes sense to be consistent one way or the other since this will all end up in Zephyr.

I usually see TensorFlow Lite Micro referenced as TFLM in documentation, and I suspect the m might be more recognisable to people?

[kristofer-jonsson-arm]

Sorry for the late response. I was on vacation last week.

I have rebased the patch and also renamed TFLu to TFLM.

Please note as Kevin points out that QEMU will not be able to run the provided unit test, because it is lacking a NPU. For this the Corstone-300 FVP needs to be used.

https://developer.arm.com/tools-and-software/open-source-software/arm-platforms-software/arm-ecosystem-fvps

[microbuilder]

@kristofer-jonsson-arm I've asked @theotherjimmy to have a look at this for testing, if you can just monitor the CI output and fix anything that shows up? Since you're a new committer, I might need to fire up the CI to run after any changes, though. Just ping me if I miss an update. Thanks!

[theotherjimmy]

it looks to me like a step in the process is rebasing onto main, which seems to fail. Perhaps some of the merges are non-trivial?

[theotherjimmy]

Since much of the commit log was trivial fixes or renames, I have squashed them all into a single commit. The branch is here: main...theotherjimmy:ethos-u and you're welcome to push that branch over this one.

[mbolivar-nordic]

@kristofer-jonsson-arm could you take a look at the twister errors in the CI results?

[carlescufi]

@kristofer-jonsson-arm I have had to make quite a few changes to clean up your sample:

• Moved it to samples/modules/tflite-micro
• Added a sample.yaml
• Cleaned up the CMakeLists.txt

Aside from multiple changes in the driver itself.

[carlescufi]

@microbuilder @kristofer-jonsson-arm now that the sample is actually building in CI, there are errors that must be looked at by those with platform knowledge:

INFO    - 215/634 mps3_an547                zephyr/samples/modules/tflite-micro/tflm_ethosu/sample.drivers.tflm_ethosu FAILED Timeout (qemu 59.967s)
INFO    - /__w/zephyr/zephyr/twister-out/mps3_an547/zephyr/samples/modules/tflite-micro/tflm_ethosu/sample.drivers.tflm_ethosu/handler.log
ERROR   - E: Failed to switch security state and privilege level (ethosu_device_u55_u65.c:244)

E: Failed to initialize Ethos-U device (ethosu_driver.c:418)

*** Booting Zephyr OS build zephyr-v3.2.0-1611-g47a6e23aab63 ***
sender 0: Sending inference. job=0x30002a68, name=keyword_spotting_cnn_small_int8
runner 0: Received inference job. job=0x30002a68
W: No NPU driver handle available.

sender 0: Sending inference. job=0x30002ab4, name=keyword_spotting_cnn_small_int8
sender 1: Sending inference. job=0x30003270, name=keyword_spotting_cnn_small_int8
sender 1: Sending inference. job=0x300032bc, name=keyword_spotting_cnn_small_int8
[00:00:00.000,000] <err> ethos_u: Failed to initialize NPU with ethosu_init().

[microbuilder]

@microbuilder @kristofer-jonsson-arm now that the sample is actually building in CI, there are errors that must be looked at by those with platform knwoledge:

I'll have someone here dig into it, thanks for the cleanup.

EDIT: QEMU doesn't support the NPU, so that's the most likely explanation. This has to be run in the FVP. I'll get someone on the team with some experience with TFLM to kick the tires in, though.

main concerns addressed, approvals left for people with more experience in this area.

Stale review

[carlescufi]

EDIT: QEMU doesn't support the NPU, so that's the most likely explanation. This has to be run in the FVP. I'll get someone on the team with some experience with TFLM to kick the tires in, though.

Oh, it doesn't? Then this should be as easy as setting this as "build only" for now. Let me do that then.

[microbuilder]

EDIT: QEMU doesn't support the NPU, so that's the most likely explanation. This has to be run in the FVP. I'll get someone on the team with some experience with TFLM to kick the tires in, though.

Oh, it doesn't? Then this should be as easy as setting this as "build only" for now. Let me do that then.

I've filed an action internally (to Linaro) to get this working with the FVP in CI, but that is likely a new PR entirely, which I'll follow up separately and enable this sample in CI when that's working.

[carlescufi]

EDIT: QEMU doesn't support the NPU, so that's the most likely explanation. This has to be run in the FVP. I'll get someone on the team with some experience with TFLM to kick the tires in, though.

Oh, it doesn't? Then this should be as easy as setting this as "build only" for now. Let me do that then.

I've filed an action internally (to Linaro) to get this working with the FVP in CI, but that is likely a new PR entirely, which I'll follow up separately and enable this sample in CI when that's working.

Great! The sample will be built in CI, it just won't be run for now.

[carlescufi]

This is now good enough to get in, but please @kristofer-jonsson-arm refactor the sample so that it is generic, and uses the Ethos-U if available as a follow-up PR.