Better Approch for Audio Bridge

[Maski0]

I'm using the package to integrate a Unity character into my Flutter app, where I generate TTS audio in Flutter that needs to drive lip sync animations in Unity. What's the best way to transfer audio data between Flutter and Unity through this package - should I

1. stream audio chunks as base64-encoded messages,
2. send complete audio files,
3. or is there a more efficient method for real-time audio transfer?

I need the audio to reach Unity with low latency for smooth lip sync playback, but I'm unsure if the message passing system can
handle frequent audio data streaming efficiently, or if there's a better approach I'm missing. Has anyone successfully implemented audio streaming between Flutter and Unity using this package?

[jamesncl]

Sorry it's taken so long to reply to this.

I don't really have much insight to help you here. My guess is that sending the path to the audio files from Flutter to Unity as a message and loading the file in a Unity script would be by far the best approach if you can get that to work (I've not tried).

Else, I think you're right, the only other method I can think of is to pass the file as a base64 string. Obviously this isn't going to be particularly efficient as it will likely involve duplicating a large amount of data in memory when the message is passed.

There may be other options like FFI or JNI but honestly that's way out of my depth, I barely know what those are or how they work but sounds like it may be useful for passing large amounts of data at a low level.

Honestly it feels like trying to get lip sync to work across the Flutter / Unity barrier is a tough job. This package is really a bit of a hack and only really designed for doing simple stuff like showing a 3D model. You might want to reconsider your approach to do everything entirely in Unity, or experimenting with Flutter's very basic native 3D rendering features to see if you can do the 3D rendering using Flutter GPU instead.