litellm - bug in v1.73.0

[pgrayy]

Description

We are seeing litellm failures in the integration tests executed in our PR workflows (example). To understand the issue, pip install litellm v1.73.0 and send the following request:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": "What is the time in New York?",
          "type": "text"
        }
      ]
    }
  ],
  "model": "bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0",
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "tool_time",
        "description": "tool_time",
        "parameters": {
          "properties": {},
          "type": "object",
          "required": []
        }
      }
    }
  ]
}

litellm will stream payloads similar to the following:

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content="I'll", role='assistant', function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True}, citations=None)

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' check the current time in New', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True}, citations=None)

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' York for you. Let', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True}, citations=None)

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=' me fetch that information.', role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True}, citations=None)

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='', role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id='tooluse_iZJCYpXyQ_ejPQQ1PZRzOg', function=Function(arguments='', name='tool_time'), type='function', index=0)], audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True})

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='', role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id=None, function=Function(arguments='', name=None), type='function', index=0)], audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True})

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content='', role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id=None, function=Function(arguments='{}', name=None), type='function', index=1)], audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True})

ModelResponseStream(id='chatcmpl-b6984fa2-5f78-4b53-8e2b-b4429769eb9e', created=1750699986, model='us.anthropic.claude-3-7-sonnet-20250219-v1:0', object='chat.completion.chunk', system_fingerprint=None, choices=[StreamingChoices(finish_reason='tool_calls', index=0, delta=Delta(provider_specific_fields=None, content=None, role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields={}, stream_options={'include_usage': True})

Notice that we have only 1 tool call but there are ChatCompletionDeltaToolCall payloads with mismatching indices. All related tool call payloads should have the same index otherwise we cannot correctly piece together the response.

This issue appears to be a bug introduced while addressing BerriAI/litellm#11580. As a temporary workaround, we are setting our litellm dependency version to be <1.73.0.

Related Issues

• [Bug]: tool_calls index starts from 1 BerriAI/litellm#11580
• fix aws bedrock claude tool call index BerriAI/litellm#11842
• [BUG] LiteLLM 1.73.0 - Tool Call Index Mismatch #271 (for tracking revert after litellm releases fix)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.