Add AWS Nova Sonic support to Pipecat Flows

[BaneTrajkovic]

Overview

This PR integrates AWS Nova Sonic LLM support into pipecat_flows, allowing Nova Sonic's speech-to-speech capabilities to work seamlessly with our flow-based conversation management system.

What Changed

Core Implementation

• Added AWSNovaSonicAdapter class that handles Nova Sonic's specific formatting requirements
• Updated adapter detection logic to automatically recognize AWSNovaSonicLLMService instances
• Implemented proper conversion between Nova Sonic's toolSpec format and our internal FlowsFunctionSchema

Key Adapter Features

The new adapter handles Nova Sonic's unique requirements:

• Extracts function names from Nova Sonic's nested toolSpec structure
• Converts between Nova Sonic's JSON schema format and our flow schema
• Manages message formatting for speech-to-speech interactions
• Provides text generation capabilities where needed

Technical Details

Nova Sonic Format Handling:

{
  "toolSpec": {
    "name": "function_name",
    "description": "function description", 
    "inputSchema": {
      "json": "{\"type\":\"object\",\"properties\":{...},\"required\":[...]}"
    }
  }
}

Integration Points:

• Import added: from pipecat.adapters.services.aws_nova_sonic_adapter import AWSNovaSonicLLMAdapter
• Adapter detection updated in create_adapter() function
• Leverages existing AWSNovaSonicLLMAdapter as the base provider

Why This Matters

Nova Sonic offers unique speech-to-speech capabilities that weren't previously accessible through our flow system. This integration enables:

• Direct speech-to-speech AI agents with structured conversation flows
• Function calling and tool use within voice-based interactions
• Automatic adapter selection without manual configuration

Testing Approach

• Verified adapter auto-detection works correctly
• Tested schema conversion in both directions
• Confirmed compatibility with existing flow configurations
• Validated error handling for missing dependencies

Compatibility

This is a purely additive change. All existing functionality remains unchanged, and Nova Sonic support only activates when the service is explicitly used. The feature requires the optional pipecat-ai[aws-nova-sonic] package.

Usage

from pipecat.services.aws_nova_sonic import AWSNovaSonicLLMService
from pipecat_flows import FlowManager

llm = AWSNovaSonicLLMService(
    secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
    access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    region=os.getenv("AWS_REGION"),
    voice_id="tiffany"
)

# Adapter is automatically selected
flow_manager = FlowManager(
    task=task,
    llm=llm,
    context_aggregator=context_aggregator,
    flow_config=flow_config,
)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
pipecat-flows	✅ Ready (Inspect)	Visit Preview	Jul 2, 2025 10:04pm

[anmoljhamb]

Hi @BaneTrajkovic

I was also looking into the same issue of adding nova-sonic to flows, and there is an issue here. The provider_adapter that we add in the constructor, needs to be changed to the _provider_adapter which makes all the difference because that is needed to call the return self._provider_adapter.to_provider_tools_format(tools_schema)