The streaming seems to be not working as expected.

[deepyes02]

Hello,
I am trying to stream an ai model served with llama-server, and hitting an error:

llama-server -m ~/llms/mistral-nemo-15.gguf -c 4096

import asyncio
from strands import Agent
from strands.models.openai import OpenAIModel
from strands_tools.calculator import calculator

model = OpenAIModel(
    client_args={
        "api_key": "sk-yourkeyhere",
        "base_url": "http://localhost:8080/v1",
    },
    model_id="mistral-nemo",
    params={
        "temperature": 0.7,
        "top_p": 0.9,
    },
)
agent = Agent(model=model)



async def streaming_response():
    agent_stream = agent.stream_async("write two short sentences about Nepal")
    async for response in agent_stream:
        if 'data' in response:
            print(response['data'], end="", flush=True)


    
asyncio.run(streaming_response())

The expected output should have been:
Nepal is home to the world's highest peak, Mount Everest. The country is known for its ancient temples and rich cultural heritage.

But some letters are repeating:
$ python scripts/stream.py
NNepepalal is is renowned renowned for for its its stunning stunning Himal Himalayasayas,, including including Mount Mount Ever Everestest.. The The country country is is also also rich rich in in cultural cultural diversity diversity,, with with over over 110000 ethnic ethnic groups groups..

I tried debugging the agent_stream object and noticed that some characters appear before each 'data' object, whose value is the same as the value inside the data object, eg:
Notice 'N' appearing just before the {'data': {}} object ? This is basically happening for each new streaming object that is generated, and seems to be causing the repetition in the response.

N{'data': 'N', 'delta': {'text': 'N'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}

Here is the whole output for debugging:

{'init_event_loop': True}
{'start': True}
{'start_event_loop': True}
{'event': {'messageStart': {'role': 'assistant'}}}
{'event': {'contentBlockStart': {'start': {}}}}
{'event': {'contentBlockDelta': {'delta': {'text': 'N'}}}}
N{'data': 'N', 'delta': {'text': 'N'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': 'ep'}}}}
ep{'data': 'ep', 'delta': {'text': 'ep'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': 'al'}}}}
al{'data': 'al', 'delta': {'text': 'al'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ','}}}}
,{'data': ',', 'delta': {'text': ','}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' home'}}}}
 home{'data': ' home', 'delta': {'text': ' home'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' to'}}}}
 to{'data': ' to', 'delta': {'text': ' to'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' the'}}}}
 the{'data': ' the', 'delta': {'text': ' the'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' Himal'}}}}
 Himal{'data': ' Himal', 'delta': {'text': ' Himal'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': 'ayas'}}}}
ayas{'data': 'ayas', 'delta': {'text': 'ayas'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ','}}}}
,{'data': ',', 'delta': {'text': ','}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' is'}}}}
 is{'data': ' is', 'delta': {'text': ' is'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' known'}}}}
 known{'data': ' known', 'delta': {'text': ' known'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' for'}}}}
 for{'data': ' for', 'delta': {'text': ' for'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' its'}}}}
 its{'data': ' its', 'delta': {'text': ' its'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' stunning'}}}}
 stunning{'data': ' stunning', 'delta': {'text': ' stunning'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' mountain'}}}}
 mountain{'data': ' mountain', 'delta': {'text': ' mountain'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' landscapes'}}}}
 landscapes{'data': ' landscapes', 'delta': {'text': ' landscapes'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': '.'}}}}
.{'data': '.', 'delta': {'text': '.'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' The'}}}}
 The{'data': ' The', 'delta': {'text': ' The'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' country'}}}}
 country{'data': ' country', 'delta': {'text': ' country'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' is'}}}}
 is{'data': ' is', 'delta': {'text': ' is'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' rich'}}}}
 rich{'data': ' rich', 'delta': {'text': ' rich'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' in'}}}}
 in{'data': ' in', 'delta': {'text': ' in'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' culture'}}}}
 culture{'data': ' culture', 'delta': {'text': ' culture'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' with'}}}}
 with{'data': ' with', 'delta': {'text': ' with'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' numerous'}}}}
 numerous{'data': ' numerous', 'delta': {'text': ' numerous'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' temples'}}}}
 temples{'data': ' temples', 'delta': {'text': ' temples'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' and'}}}}
 and{'data': ' and', 'delta': {'text': ' and'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': ' festivals'}}}}
 festivals{'data': ' festivals', 'delta': {'text': ' festivals'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockDelta': {'delta': {'text': '.'}}}}
.{'data': '.', 'delta': {'text': '.'}, 'agent': <strands.agent.agent.Agent object at 0x108df4cd0>, 'event_loop_cycle_id': UUID('b874af09-b7b5-4f9d-871b-e960abcadb13'), 'request_state': {}, 'event_loop_cycle_trace': <strands.telemetry.metrics.Trace object at 0x1096c6e10>, 'event_loop_cycle_span': NonRecordingSpan(SpanContext(trace_id=0x00000000000000000000000000000000, span_id=0x0000000000000000, trace_flags=0x00, trace_state=[], is_remote=False))}
{'event': {'contentBlockStop': {}}}
{'event': {'messageStop': {'stopReason': 'end_turn'}}}
{'event': {'metadata': {'usage': {'inputTokens': 9, 'outputTokens': 31, 'totalTokens': 40}, 'metrics': {'latencyMs': 0}}}}
{'message': {'role': 'assistant', 'content': [{'text': 'Nepal, home to the Himalayas, is known for its stunning mountain landscapes. The country is rich in culture with numerous temples and festivals.'}]}}
{'result': AgentResult(stop_reason='end_turn', message={'role': 'assistant', 'content': [{'text': 'Nepal, home to the Himalayas, is known for its stunning mountain landscapes. The country is rich in culture with numerous temples and festivals.'}]}, metrics=EventLoopMetrics(cycle_count=1, tool_metrics={}, cycle_durations=[2.071805000305176], traces=[<strands.telemetry.metrics.Trace object at 0x1096c6e10>], accumulated_usage={'inputTokens': 9, 'outputTokens': 31, 'totalTokens': 40}, accumulated_metrics={'latencyMs': 0}), state={})}

Can someone please guide me on how to get this right ?

[zastrowm]

I think this is a result of the default callback_handler of strands where it prints the response for you.

https://strandsagents.com/latest/documentation/docs/user-guide/concepts/streaming/callback-handlers/#default-callback-handler

Does the issue go away if you set callback_handler=None when creating the Agent?

[deepyes02]

@zastrowm , thank you,
This solved the issue, however, I want to know why this behavior is being caused. Is it okay to always set callback_handler to false ? Is this a bug or expected behavior ?
Thanks again.

agent = Agent(model=model, callback_handler=None )

[zastrowm]

Yeah that's expected behavior - the callback handler is there as an alternative to stream_async events and if one is not provided the agent uses a default one which prints the output of responses. By explicitly passing None it turns if the printing by default.

#506 is an issue which may clarify this.