stream toolInput cannot be null or empty

[WilliamTan778]

Bug description
I have a tool function that retrieves dates. The interface runs in call mode and returns all correct results, but when I execute it in stream mode, it always reports an error

java.lang.IllegalArgumentException: toolInput cannot be null or empty
at org.springframework.util.Assert.hasText(Assert.java:240)
Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below:
Assembly trace from producer [reactor.core.publisher.FluxFlatMap] :
reactor.core.publisher.Flux.flatMap(Flux.java:5471)
org.springframework.ai.openai.OpenAiChatModel.lambda$internalStream$13(OpenAiChatModel.java:439)
Error has been observed at the following site(s):
*___Flux.flatMap ⇢ at org.springframework.ai.openai.OpenAiChatModel.lambda$internalStream$13(OpenAiChatModel.java:439)
| Flux.doOnError ⇢ at org.springframework.ai.openai.OpenAiChatModel.lambda$internalStream$13(OpenAiChatModel.java:458)
| Flux.doFinally ⇢ at org.springframework.ai.openai.OpenAiChatModel.lambda$internalStream$13(OpenAiChatModel.java:459)
| Flux.contextWrite ⇢ at org.springframework.ai.openai.OpenAiChatModel.lambda$internalStream$13(OpenAiChatModel.java:460)
| Flux.doOnSubscribe ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:94)
| Flux.doOnNext ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:105)
| Flux.doOnComplete ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:144)
| Flux.doOnError ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:171)
*Flux.deferContextual ⇢ at org.springframework.ai.openai.OpenAiChatModel.internalStream(OpenAiChatModel.java:345)
| Flux.map ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultChatClientRequestSpec$2.aroundStream(DefaultChatClient.java:699)
| Flux.publishOn ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultChatClientRequestSpec$2.aroundStream(DefaultChatClient.java:700)
*_________Flux.defer ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:124)
| Flux.doOnError ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:125)
| Flux.doFinally ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:126)
| Flux.contextWrite ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:127)
*__Flux.deferContextual ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundStream(DefaultAroundAdvisorChain.java:103)
*_____Mono.flatMapMany ⇢ at org.springframework.ai.chat.client.advisor.QuestionAnswerAdvisor.aroundStream(QuestionAnswerAdvisor.java:199)
| Flux.map ⇢ at org.springframework.ai.chat.client.advisor.QuestionAnswerAdvisor.aroundStream(QuestionAnswerAdvisor.java:203)
*_____Flux.defer ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:124)
| Flux.doOnError ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:125)
| Flux.doFinally ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:126)
| Flux.contextWrite ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:127)
*Flux.deferContextual ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundStream(DefaultAroundAdvisorChain.java:103)
| Flux.map ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregateAdvisedResponse(MessageAggregator.java:56)
| Flux.doOnSubscribe ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:94)
| Flux.doOnNext ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:105)
| Flux.doOnComplete ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:144)
| Flux.doOnError ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:171)
| Flux.map ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregateAdvisedResponse(MessageAggregator.java:69)
*_________Flux.defer ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:124)
| Flux.doOnError ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:125)
| Flux.doFinally ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:126)
| Flux.contextWrite ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:127)
*__Flux.deferContextual ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundStream(DefaultAroundAdvisorChain.java:103)
*_________Flux.defer ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:124)
| Flux.doOnError ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:125)
| Flux.doFinally ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:126)
| Flux.contextWrite ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:127)
*__Flux.deferContextual ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundStream(DefaultAroundAdvisorChain.java:103)
*Mono.flatMapMany ⇢ at org.springframework.ai.chat.client.advisor.AbstractChatMemoryAdvisor.doNextWithProtectFromBlockingBefore(AbstractChatMemoryAdvisor.java:194)
| Flux.map ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregateAdvisedResponse(MessageAggregator.java:56)
| Flux.doOnSubscribe ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:94)
| Flux.doOnNext ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:105)
| Flux.doOnComplete ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:144)
| Flux.doOnError ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregate(MessageAggregator.java:171)
| Flux.map ⇢ at org.springframework.ai.chat.model.MessageAggregator.aggregateAdvisedResponse(MessageAggregator.java:69)
*______Flux.defer ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:124)
| Flux.doOnError ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:125)
| Flux.doFinally ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:126)
| Flux.contextWrite ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundStream$6(DefaultAroundAdvisorChain.java:127)
*Flux.deferContextual ⇢ at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundStream(DefaultAroundAdvisorChain.java:103)
| Flux.map ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultStreamResponseSpec.lambda$doGetObservableFluxChatResponse$3(DefaultChatClient.java:553)
| Flux.doOnError ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultStreamResponseSpec.lambda$doGetObservableFluxChatResponse$3(DefaultChatClient.java:554)
| Flux.doFinally ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultStreamResponseSpec.lambda$doGetObservableFluxChatResponse$3(DefaultChatClient.java:555)
| Flux.contextWrite ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultStreamResponseSpec.lambda$doGetObservableFluxChatResponse$3(DefaultChatClient.java:556)
*Flux.deferContextual ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultStreamResponseSpec.doGetObservableFluxChatResponse(DefaultChatClient.java:532)
| Flux.map ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultStreamResponseSpec.content(DefaultChatClient.java:568)
| Flux.filter ⇢ at org.springframework.ai.chat.client.DefaultChatClient$DefaultStreamResponseSpec.content(DefaultChatClient.java:574)
| Flux.map ⇢ at com.example.demo.controller.RagController.chatStreamWithDatabase(RagController.java:169)
Original Stack Trace:
at org.springframework.util.Assert.hasText(Assert.java:240)
at org.springframework.ai.tool.function.FunctionToolCallback.call(FunctionToolCallback.java:96)
at org.springframework.ai.model.tool.DefaultToolCallingManager.executeToolCall(DefaultToolCallingManager.java:226)
at org.springframework.ai.model.tool.DefaultToolCallingManager.executeToolCalls(DefaultToolCallingManager.java:138)
at org.springframework.ai.openai.OpenAiChatModel.lambda$internalStream$10(OpenAiChatModel.java:442)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:388)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.DrainUtils.postCompleteDrain(DrainUtils.java:136)
at reactor.core.publisher.DrainUtils.postComplete(DrainUtils.java:187)
at reactor.core.publisher.FluxBuffer$BufferOverlappingSubscriber.onComplete(FluxBuffer.java:551)
at reactor.core.publisher.FluxSwitchMapNoPrefetch$SwitchMapMain.onComplete(FluxSwitchMapNoPrefetch.java:265)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:850)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:612)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:592)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:469)
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drainAsync(FluxFlattenIterable.java:371)
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drain(FluxFlattenIterable.java:724)
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.onComplete(FluxFlattenIterable.java:273)
at reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.drainFused(FluxWindowPredicate.java:459)
at reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.drain(FluxWindowPredicate.java:429)
at reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.onComplete(FluxWindowPredicate.java:312)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onComplete(FluxFilterFuseable.java:171)
at reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber.onComplete(FluxTakeUntil.java:122)
at reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber.onNext(FluxTakeUntil.java:100)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
at reactor.core.publisher.FluxConcatMap$WeakScalarSubscription.request(FluxConcatMap.java:480)
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:202)
at reactor.core.publisher.FluxBufferPredicate$BufferPredicateSubscriber.onNextNewBuffer(FluxBufferPredicate.java:317)
at reactor.core.publisher.FluxBufferPredicate$BufferPredicateSubscriber.tryOnNext(FluxBufferPredicate.java:227)
at reactor.core.publisher.FluxBufferPredicate$BufferPredicateSubscriber.onNext(FluxBufferPredicate.java:200)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:503)
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onNext(FluxConcatArray.java:180)
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drainAsync(FluxFlattenIterable.java:453)
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drain(FluxFlattenIterable.java:724)
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.onNext(FluxFlattenIterable.java:256)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:210)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:294)
at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:403)
at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:425)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:822)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1347)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)

CODE

[junzhiL]

i met this error too, and the reason is when the function has no params
stream call toolCall.arguments field returned "". but non-stream call returned "{}"

org.springframework.ai.tool.method.MethodToolCallback.call() and org.springframework.ai.tool.function.FunctionToolCallback.call() has code:

public String call(String toolInput, @Nullable ToolContext toolContext) {
		Assert.hasText(toolInput, "toolInput cannot be null or empty");

I don't know if this is a problem of LLM or client should compatible this.

[chebyshevvv]

The DefaultToolCallingManager cannot resolve well the stream response of ai.This is my implements that resoved it:

`import io.micrometer.observation.ObservationRegistry;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.ToolResponseMessage;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.chat.model.ToolContext;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.model.function.FunctionCallback;
import org.springframework.ai.model.function.FunctionCallingOptions;
import org.springframework.ai.model.tool.ToolCallingChatOptions;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.model.tool.ToolExecutionResult;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
import org.springframework.ai.tool.execution.DefaultToolExecutionExceptionProcessor;
import org.springframework.ai.tool.execution.ToolExecutionException;
import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor;
import org.springframework.ai.tool.resolution.DelegatingToolCallbackResolver;
import org.springframework.ai.tool.resolution.ToolCallbackResolver;
import org.springframework.util.Assert;
import org.springframework.util.CollectionUtils;

import java.util.*;

@slf4j
public class StreamingToolCallingManager implements ToolCallingManager {
private static final ObservationRegistry DEFAULT_OBSERVATION_REGISTRY
= ObservationRegistry.NOOP;

private static final ToolCallbackResolver DEFAULT_TOOL_CALLBACK_RESOLVER
        = new DelegatingToolCallbackResolver(List.of());

private static final ToolExecutionExceptionProcessor DEFAULT_TOOL_EXECUTION_EXCEPTION_PROCESSOR
        = DefaultToolExecutionExceptionProcessor.builder().build();

// @formatter:on

private final ObservationRegistry observationRegistry;

private final ToolCallbackResolver toolCallbackResolver;

private final ToolExecutionExceptionProcessor toolExecutionExceptionProcessor;
public StreamingToolCallingManager(ObservationRegistry observationRegistry, ToolCallbackResolver toolCallbackResolver, ToolExecutionExceptionProcessor toolExecutionExceptionProcessor) {
    Assert.notNull(observationRegistry, "observationRegistry cannot be null");
    Assert.notNull(toolCallbackResolver, "toolCallbackResolver cannot be null");
    Assert.notNull(toolExecutionExceptionProcessor, "toolCallExceptionConverter cannot be null");

    this.observationRegistry = observationRegistry;
    this.toolCallbackResolver = toolCallbackResolver;
    this.toolExecutionExceptionProcessor = toolExecutionExceptionProcessor;
}

@Override
public List<ToolDefinition> resolveToolDefinitions(ToolCallingChatOptions chatOptions) {
    Assert.notNull(chatOptions, "chatOptions cannot be null");

    List<FunctionCallback> toolCallbacks = new ArrayList<>(chatOptions.getToolCallbacks());
    for (String toolName : chatOptions.getToolNames()) {
        // Skip the tool if it is already present in the request toolCallbacks.
        // That might happen if a tool is defined in the options
        // both as a ToolCallback and as a tool name.
        if (chatOptions.getToolCallbacks().stream().anyMatch(tool -> tool.getName().equals(toolName))) {
            continue;
        }
        FunctionCallback toolCallback = toolCallbackResolver.resolve(toolName);
        if (toolCallback == null) {
            throw new IllegalStateException("No ToolCallback found for tool name: " + toolName);
        }
        toolCallbacks.add(toolCallback);
    }

    return toolCallbacks.stream().map(functionCallback -> {
        if (functionCallback instanceof ToolCallback toolCallback) {
            return toolCallback.getToolDefinition();
        }
        else {
            return ToolDefinition.builder()
                    .name(functionCallback.getName())
                    .description(functionCallback.getDescription())
                    .inputSchema(functionCallback.getInputTypeSchema())
                    .build();
        }
    }).toList();
}

@Override
public ToolExecutionResult executeToolCalls(Prompt prompt, ChatResponse chatResponse) {
    Optional<Generation> first = chatResponse.getResults()
            .stream()
            .filter(g -> !CollectionUtils.isEmpty(g.getOutput().getToolCalls()))
            .findFirst();
    Generation generation = first.orElseThrow();

    String args = generation.getOutput().getToolCalls().stream().map(AssistantMessage.ToolCall::arguments).reduce("", (s, s2) -> s + s2);
    AssistantMessage.ToolCall toolcall = generation.getOutput().getToolCalls().getFirst();
    AssistantMessage.ToolCall newToolCall = new AssistantMessage.ToolCall(toolcall.id(), toolcall.type(), toolcall.name(), args);
    List<AssistantMessage.ToolCall> newToolCalls = List.of(newToolCall);

    AssistantMessage assistantMessage = new AssistantMessage(generation.getOutput().getText(), generation.getOutput().getMetadata(), newToolCalls, generation.getOutput().getMedia());


    ToolContext toolContext = buildToolContext(prompt, assistantMessage);

    InternalToolExecutionResult internalToolExecutionResult = executeToolCall(prompt, assistantMessage,
            toolContext);

    List<Message> conversationHistory = buildConversationHistoryAfterToolExecution(prompt.getInstructions(),
            assistantMessage, internalToolExecutionResult.toolResponseMessage());

    return ToolExecutionResult.builder()
            .conversationHistory(conversationHistory)
            .returnDirect(internalToolExecutionResult.returnDirect())
            .build();
}
private static ToolContext buildToolContext(Prompt prompt, AssistantMessage assistantMessage) {
    Map<String, Object> toolContextMap = Map.of();

    if (prompt.getOptions() instanceof FunctionCallingOptions functionOptions
            && !CollectionUtils.isEmpty(functionOptions.getToolContext())) {
        toolContextMap = new HashMap<>(functionOptions.getToolContext());

        List<Message> messageHistory = new ArrayList<>(prompt.copy().getInstructions());
        messageHistory.add(new AssistantMessage(assistantMessage.getText(), assistantMessage.getMetadata(),
                assistantMessage.getToolCalls()));

        toolContextMap.put(ToolContext.TOOL_CALL_HISTORY,
                buildConversationHistoryBeforeToolExecution(prompt, assistantMessage));
    }

    return new ToolContext(toolContextMap);
}
private static List<Message> buildConversationHistoryBeforeToolExecution(Prompt prompt,
                                                                         AssistantMessage assistantMessage) {
    List<Message> messageHistory = new ArrayList<>(prompt.copy().getInstructions());
    messageHistory.add(new AssistantMessage(assistantMessage.getText(), assistantMessage.getMetadata(),
            assistantMessage.getToolCalls()));
    return messageHistory;
}


private InternalToolExecutionResult executeToolCall(Prompt prompt, AssistantMessage assistantMessage,
                                                                              ToolContext toolContext) {
    List<FunctionCallback> toolCallbacks = List.of();
    if (prompt.getOptions() instanceof ToolCallingChatOptions toolCallingChatOptions) {
        toolCallbacks = toolCallingChatOptions.getToolCallbacks();
    }
    else if (prompt.getOptions() instanceof FunctionCallingOptions functionOptions) {
        toolCallbacks = functionOptions.getFunctionCallbacks();
    }

    List<ToolResponseMessage.ToolResponse> toolResponses = new ArrayList<>();

    Boolean returnDirect = null;

    for (AssistantMessage.ToolCall toolCall : assistantMessage.getToolCalls()) {

        log.debug("Executing tool call: {}", toolCall.name());

        String toolName = toolCall.name();
        String toolInputArguments = toolCall.arguments();

        FunctionCallback toolCallback = toolCallbacks.stream()
                .filter(tool -> toolName.equals(tool.getName()))
                .findFirst()
                .orElseGet(() -> toolCallbackResolver.resolve(toolName));

        if (toolCallback == null) {
            throw new IllegalStateException("No ToolCallback found for tool name: " + toolName);
        }

        if (returnDirect == null && toolCallback instanceof ToolCallback callback) {
            returnDirect = callback.getToolMetadata().returnDirect();
        }
        else if (toolCallback instanceof ToolCallback callback) {
            returnDirect = returnDirect && callback.getToolMetadata().returnDirect();
        }
        else if (returnDirect == null) {
            // This is a temporary solution to ensure backward compatibility with
            // FunctionCallback.
            // TODO: remove this block when FunctionCallback is removed.
            returnDirect = false;
        }

        String toolResult;
        try {
            toolResult = toolCallback.call(toolInputArguments, toolContext);
        }
        catch (ToolExecutionException ex) {
            toolResult = toolExecutionExceptionProcessor.process(ex);
        }

        toolResponses.add(new ToolResponseMessage.ToolResponse(toolCall.id(), toolName, toolResult));
    }

    return new InternalToolExecutionResult(new ToolResponseMessage(toolResponses, Map.of()), returnDirect);
}

private List<Message> buildConversationHistoryAfterToolExecution(List<Message> previousMessages,
                                                                 AssistantMessage assistantMessage, ToolResponseMessage toolResponseMessage) {
    List<Message> messages = new ArrayList<>(previousMessages);
    messages.add(assistantMessage);
    messages.add(toolResponseMessage);
    return messages;
}

private record InternalToolExecutionResult(ToolResponseMessage toolResponseMessage, boolean returnDirect) {
}

}`

config the bean：

`import io.micrometer.observation.ObservationRegistry;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor;
import org.springframework.ai.tool.resolution.ToolCallbackResolver;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@configuration
public class ToolConfig {

@Bean
ToolCallingManager toolCallingManager(ToolCallbackResolver toolCallbackResolver,
                                      ToolExecutionExceptionProcessor toolExecutionExceptionProcessor,
                                      ObjectProvider<ObservationRegistry> observationRegistry) {
    return new StreamingToolCallingManager(observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP),toolCallbackResolver,toolExecutionExceptionProcessor);
}

}`

Then I can use tool calling by stream.
I dont know if this is usefull to you, but you can try or fix the source code.

[WilliamTan778]

The DefaultToolCallingManager cannot resolve well the stream response of ai.This is my implements that resoved it:

`import io.micrometer.observation.ObservationRegistry; import lombok.extern.slf4j.Slf4j; import org.springframework.ai.chat.messages.AssistantMessage; import org.springframework.ai.chat.messages.Message; import org.springframework.ai.chat.messages.ToolResponseMessage; import org.springframework.ai.chat.model.ChatResponse; import org.springframework.ai.chat.model.Generation; import org.springframework.ai.chat.model.ToolContext; import org.springframework.ai.chat.prompt.Prompt; import org.springframework.ai.model.function.FunctionCallback; import org.springframework.ai.model.function.FunctionCallingOptions; import org.springframework.ai.model.tool.ToolCallingChatOptions; import org.springframework.ai.model.tool.ToolCallingManager; import org.springframework.ai.model.tool.ToolExecutionResult; import org.springframework.ai.tool.ToolCallback; import org.springframework.ai.tool.definition.ToolDefinition; import org.springframework.ai.tool.execution.DefaultToolExecutionExceptionProcessor; import org.springframework.ai.tool.execution.ToolExecutionException; import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor; import org.springframework.ai.tool.resolution.DelegatingToolCallbackResolver; import org.springframework.ai.tool.resolution.ToolCallbackResolver; import org.springframework.util.Assert; import org.springframework.util.CollectionUtils;

import java.util.*;

@slf4j public class StreamingToolCallingManager implements ToolCallingManager { private static final ObservationRegistry DEFAULT_OBSERVATION_REGISTRY = ObservationRegistry.NOOP;

private static final ToolCallbackResolver DEFAULT_TOOL_CALLBACK_RESOLVER
        = new DelegatingToolCallbackResolver(List.of());

private static final ToolExecutionExceptionProcessor DEFAULT_TOOL_EXECUTION_EXCEPTION_PROCESSOR
        = DefaultToolExecutionExceptionProcessor.builder().build();

// @formatter:on

private final ObservationRegistry observationRegistry;

private final ToolCallbackResolver toolCallbackResolver;

private final ToolExecutionExceptionProcessor toolExecutionExceptionProcessor;
public StreamingToolCallingManager(ObservationRegistry observationRegistry, ToolCallbackResolver toolCallbackResolver, ToolExecutionExceptionProcessor toolExecutionExceptionProcessor) {
    Assert.notNull(observationRegistry, "observationRegistry cannot be null");
    Assert.notNull(toolCallbackResolver, "toolCallbackResolver cannot be null");
    Assert.notNull(toolExecutionExceptionProcessor, "toolCallExceptionConverter cannot be null");

    this.observationRegistry = observationRegistry;
    this.toolCallbackResolver = toolCallbackResolver;
    this.toolExecutionExceptionProcessor = toolExecutionExceptionProcessor;
}

@Override
public List<ToolDefinition> resolveToolDefinitions(ToolCallingChatOptions chatOptions) {
    Assert.notNull(chatOptions, "chatOptions cannot be null");

    List<FunctionCallback> toolCallbacks = new ArrayList<>(chatOptions.getToolCallbacks());
    for (String toolName : chatOptions.getToolNames()) {
        // Skip the tool if it is already present in the request toolCallbacks.
        // That might happen if a tool is defined in the options
        // both as a ToolCallback and as a tool name.
        if (chatOptions.getToolCallbacks().stream().anyMatch(tool -> tool.getName().equals(toolName))) {
            continue;
        }
        FunctionCallback toolCallback = toolCallbackResolver.resolve(toolName);
        if (toolCallback == null) {
            throw new IllegalStateException("No ToolCallback found for tool name: " + toolName);
        }
        toolCallbacks.add(toolCallback);
    }

    return toolCallbacks.stream().map(functionCallback -> {
        if (functionCallback instanceof ToolCallback toolCallback) {
            return toolCallback.getToolDefinition();
        }
        else {
            return ToolDefinition.builder()
                    .name(functionCallback.getName())
                    .description(functionCallback.getDescription())
                    .inputSchema(functionCallback.getInputTypeSchema())
                    .build();
        }
    }).toList();
}

@Override
public ToolExecutionResult executeToolCalls(Prompt prompt, ChatResponse chatResponse) {
    Optional<Generation> first = chatResponse.getResults()
            .stream()
            .filter(g -> !CollectionUtils.isEmpty(g.getOutput().getToolCalls()))
            .findFirst();
    Generation generation = first.orElseThrow();

    String args = generation.getOutput().getToolCalls().stream().map(AssistantMessage.ToolCall::arguments).reduce("", (s, s2) -> s + s2);
    AssistantMessage.ToolCall toolcall = generation.getOutput().getToolCalls().getFirst();
    AssistantMessage.ToolCall newToolCall = new AssistantMessage.ToolCall(toolcall.id(), toolcall.type(), toolcall.name(), args);
    List<AssistantMessage.ToolCall> newToolCalls = List.of(newToolCall);

    AssistantMessage assistantMessage = new AssistantMessage(generation.getOutput().getText(), generation.getOutput().getMetadata(), newToolCalls, generation.getOutput().getMedia());


    ToolContext toolContext = buildToolContext(prompt, assistantMessage);

    InternalToolExecutionResult internalToolExecutionResult = executeToolCall(prompt, assistantMessage,
            toolContext);

    List<Message> conversationHistory = buildConversationHistoryAfterToolExecution(prompt.getInstructions(),
            assistantMessage, internalToolExecutionResult.toolResponseMessage());

    return ToolExecutionResult.builder()
            .conversationHistory(conversationHistory)
            .returnDirect(internalToolExecutionResult.returnDirect())
            .build();
}
private static ToolContext buildToolContext(Prompt prompt, AssistantMessage assistantMessage) {
    Map<String, Object> toolContextMap = Map.of();

    if (prompt.getOptions() instanceof FunctionCallingOptions functionOptions
            && !CollectionUtils.isEmpty(functionOptions.getToolContext())) {
        toolContextMap = new HashMap<>(functionOptions.getToolContext());

        List<Message> messageHistory = new ArrayList<>(prompt.copy().getInstructions());
        messageHistory.add(new AssistantMessage(assistantMessage.getText(), assistantMessage.getMetadata(),
                assistantMessage.getToolCalls()));

        toolContextMap.put(ToolContext.TOOL_CALL_HISTORY,
                buildConversationHistoryBeforeToolExecution(prompt, assistantMessage));
    }

    return new ToolContext(toolContextMap);
}
private static List<Message> buildConversationHistoryBeforeToolExecution(Prompt prompt,
                                                                         AssistantMessage assistantMessage) {
    List<Message> messageHistory = new ArrayList<>(prompt.copy().getInstructions());
    messageHistory.add(new AssistantMessage(assistantMessage.getText(), assistantMessage.getMetadata(),
            assistantMessage.getToolCalls()));
    return messageHistory;
}


private InternalToolExecutionResult executeToolCall(Prompt prompt, AssistantMessage assistantMessage,
                                                                              ToolContext toolContext) {
    List<FunctionCallback> toolCallbacks = List.of();
    if (prompt.getOptions() instanceof ToolCallingChatOptions toolCallingChatOptions) {
        toolCallbacks = toolCallingChatOptions.getToolCallbacks();
    }
    else if (prompt.getOptions() instanceof FunctionCallingOptions functionOptions) {
        toolCallbacks = functionOptions.getFunctionCallbacks();
    }

    List<ToolResponseMessage.ToolResponse> toolResponses = new ArrayList<>();

    Boolean returnDirect = null;

    for (AssistantMessage.ToolCall toolCall : assistantMessage.getToolCalls()) {

        log.debug("Executing tool call: {}", toolCall.name());

        String toolName = toolCall.name();
        String toolInputArguments = toolCall.arguments();

        FunctionCallback toolCallback = toolCallbacks.stream()
                .filter(tool -> toolName.equals(tool.getName()))
                .findFirst()
                .orElseGet(() -> toolCallbackResolver.resolve(toolName));

        if (toolCallback == null) {
            throw new IllegalStateException("No ToolCallback found for tool name: " + toolName);
        }

        if (returnDirect == null && toolCallback instanceof ToolCallback callback) {
            returnDirect = callback.getToolMetadata().returnDirect();
        }
        else if (toolCallback instanceof ToolCallback callback) {
            returnDirect = returnDirect && callback.getToolMetadata().returnDirect();
        }
        else if (returnDirect == null) {
            // This is a temporary solution to ensure backward compatibility with
            // FunctionCallback.
            // TODO: remove this block when FunctionCallback is removed.
            returnDirect = false;
        }

        String toolResult;
        try {
            toolResult = toolCallback.call(toolInputArguments, toolContext);
        }
        catch (ToolExecutionException ex) {
            toolResult = toolExecutionExceptionProcessor.process(ex);
        }

        toolResponses.add(new ToolResponseMessage.ToolResponse(toolCall.id(), toolName, toolResult));
    }

    return new InternalToolExecutionResult(new ToolResponseMessage(toolResponses, Map.of()), returnDirect);
}

private List<Message> buildConversationHistoryAfterToolExecution(List<Message> previousMessages,
                                                                 AssistantMessage assistantMessage, ToolResponseMessage toolResponseMessage) {
    List<Message> messages = new ArrayList<>(previousMessages);
    messages.add(assistantMessage);
    messages.add(toolResponseMessage);
    return messages;
}

private record InternalToolExecutionResult(ToolResponseMessage toolResponseMessage, boolean returnDirect) {
}

}`

config the bean：

`import io.micrometer.observation.ObservationRegistry; import org.springframework.ai.model.tool.ToolCallingManager; import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor; import org.springframework.ai.tool.resolution.ToolCallbackResolver; import org.springframework.beans.factory.ObjectProvider; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration;

@configuration public class ToolConfig {

@Bean
ToolCallingManager toolCallingManager(ToolCallbackResolver toolCallbackResolver,
                                      ToolExecutionExceptionProcessor toolExecutionExceptionProcessor,
                                      ObjectProvider<ObservationRegistry> observationRegistry) {
    return new StreamingToolCallingManager(observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP),toolCallbackResolver,toolExecutionExceptionProcessor);
}

}`

Then I can use tool calling by stream. I dont know if this is usefull to you, but you can try or fix the source code.

Thank you, I'll give it a try