Letta:Latest does not work with OLLAMA (letta/letta@sha256:858f11fbbbe4392c4ce7ae76d49f406ec614793dce93cd1f030cc7f23004b955)

[racingninja]

Describe the bug
Letta latest version does not work with OLLAMA. I cannot connect to openAI as there are limitations on my side.
It has become a show stopper.

Please describe your setup

• How are you running Letta?

  • Docker

• Describe your setup

  • What's your OS (Windows/MacOS/Linux)?
  • MAC
  • What is your docker run ... command (if applicable)
  • docker run
    -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data
    -v /Users/rmuraly/ttk:/tiktoken_cache
    -p 8283:8283
    -e OLLAMA_BASE_URL="http://host.docker.internal:11434"
    -e TIKTOKEN_CACHE_DIR="/tiktoken_cache"
    letta/letta:latest

ERRORS
"/projects/letta/create_agent.py", line 6, in
genie = client.agents.create(
^^^^^^^^^^^^^^^^^^^^^
File "/.pyenv/versions/3.11.9/lib/python3.11/site-packages/letta_client/agents/client.py", line 458, in create
raise ApiError(status_code=_response.status_code, body=_response_json)
letta_client.core.api_error.ApiError: status_code: 500, body: {'detail': "Client error '404 Not Found' for url 'http://host.docker.internal:11434/models'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404"}

Additional context
Add any other context about the problem here.

• LLM Model : Mistral:latest
• Embedding Model : nomic-embed-text:latest

Agent File (optional)
Please attach your .af file, as this helps with reproducing issues.

---

If you're not using OpenAI, please provide additional information on your local LLM setup:

Local LLM details

If you are trying to run Letta with local LLMs, please provide the following information:

• The exact model you're trying to use (e.g. dolphin-2.1-mistral-7b.Q6_K.gguf)
• The local LLM backend you are using (web UI? LM Studio?)
• Your hardware for the local LLM backend (local computer? operating system? remote RunPod?)

[racingninja]

Attaching the complete client code for reference :

`from letta_client import Letta, EmbeddingConfig
import time

client = Letta(base_url="http://localhost:8283")

dreamboat = client.agents.create(
name="DreamBoat",
model="ollama/mistral:latest",
embedding="ollama/nomic-embed-text:latest",
# optional configuration
context_window_limit=8192,
memory_blocks=[
{
"label": "human",
"value": "The human's name is Dreamboat the Visionary, innovator.",
"limit": 5000
},
{
"label": "persona",
"value": "My name is Genie, the all-knowing AI. I can answer any question from the employee handbook which I can search and retrieve from the archival",
"limit": 5000
}
]
)
print("+"*50)
print(dreamboat.id)

embedding_config = EmbeddingConfig(
embedding_model="nomic-embed-text",
embedding_endpoint_type="ollama",
embedding_endpoint="http://localhost:11434",
embedding_dim=256,
embedding_chunk_size=512,
)

source = client.sources.create(name="bla",
embedding_config=embedding_config)

job = client.sources.files.upload(
source_id=source.id,
file=open("project_chimera.md", "rb")
)

wait until the job is completed

while True:
job = client.jobs.retrieve(job.id)
if job.status == "completed":
break
elif job.status == "failed":
raise ValueError(f"Job failed: {job.metadata}")
print(f"Job status: {job.status}")
time.sleep(1)

list files in the source

files = client.sources.files.list(source_id=source.id)
print(f"Files in source: {files}")

list passages in the source

passages = client.sources.passages.list(source_id=source.id)
print(f"Passages in source: {passages}")

client.agents.sources.attach(agent_id=dreamboat.id, source_id=source.id)`

[cliandy]

Hi @racingninja thanks for reporting this issue. We fixed the ollama embedding model listing for the next release. If using any non-ollama models, do you see any issues?

[racingninja]

@cliandy I am trying to verify using open AI models via Azure. I will let you know if I find any issues doing that.

[racingninja]

@cliandy , I tried to connect to Azure supported Open AI and I do not see any models listed from the list of models allocated to me.

The same code works well outside LETTA

docker run \ -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \ -v /Users/rmuraly/ttk:/tiktoken_cache \ -v /Users/rmuraly/Documents/projects/letta/certify.crt:/etc/ssl/certs/certify.crt \ -p 8283:8283 \ -e AZURE_API_KEY="MY_KEY" \ -e AZURE_BASE_URL="https://MY_HOST/v1/proxy/azure-openai" \ -e TIKTOKEN_CACHE_DIR="/tiktoken_cache" \ letta/letta:latest

[cliandy]

@racingninja I'll take a look at the azure issue. The previous issue should have been resolved with the latest release. Please let me know if that is now working.

[racingninja]

The previous issue still exists..

Looks like you changed the url to http://host.docker.internal:11434/v1/api/tags

But the right url is http://host.docker.internal:11434/api/tags

Error I see :
`File "/app/letta/schemas/providers.py", line 881, in list_llm_models_async
raise Exception(f"Failed to list Ollama models: {response.text}")
Exception: Failed to list Ollama models: <bound method ClientResponse.text of <ClientResponse(http://host.docker.internal:11434/v1/api/tags) [404 Not Found]>
<CIMultiDictProxy('Content-Type': 'text/plain', 'Date': 'Fri, 13 Jun 2025 17:57:41 GMT', 'Content-Length': '18')>

/app/letta/server/server.py:1609: UserWarning: An error occurred while listing LLM models for provider id=None name='ollama' provider_type=<ProviderType.ollama: 'ollama'> provider_category=<ProviderCategory.base: 'base'> api_key=None base_url='http://host.docker.internal:11434/v1' organization_id=None updated_at=None default_prompt_formatter='chatml': Failed to list Ollama models: <bound method ClientResponse.text of <ClientResponse(http://host.docker.internal:11434/v1/api/tags) [404 Not Found]>
`

[sarahwooders]

Does this work with 0.8.4?

[racingninja]

0.8.4 fixes the problem with Ollama models.

[racingninja]

I was looking to attach a simple file and I face this error with file_name being null.

sqlalchemy.exc.IntegrityError: (sqlalchemy.dialects.postgresql.asyncpg.IntegrityError) <class 'asyncpg.exceptions.NotNullViolationError'>: null value in column "file_name" of relation "source_passages" violates not-null constraint DETAIL: Failing row contains (passage-0dd9f78b-d051-4ae3-9a17-45ba4dd10363, hello.. sample text., {"embedding_endpoint_type": "ollama", "embedding_endpoint": "htt..., {}, [0.5046537,-0.85620034,-3.4605947,-0.984015,0.7693641,-0.5242434..., 2025-06-18 19:05:19.603997+00, 2025-06-18 19:05:19.605753+00, f, user-00000000-0000-4000-8000-000000000000, user-00000000-0000-4000-8000-000000000000, org-00000000-0000-4000-8000-000000000000, file-87056ebb-b128-4e5c-b2a5-050d9a850b23, source-6a74300d-df0f-40fd-b0c7-2e553fa9e7b4, null). [SQL: INSERT INTO source_passages (file_name, id, text, embedding_config, metadata_, embedding, created_at, is_deleted, _created_by_id, _last_updated_by_id, organization_id, file_id, source_id) VALUES ($1::VARCHAR, $2::VARCHAR, $3::VARCHAR, $4::JSON, $5::JSON, $6, $7::TIMESTAMP WITH TIME ZONE, $8::BOOLEAN, $9::VARCHAR, $10::VARCHAR, $11::VARCHAR, $12::VARCHAR, $13::VARCHAR) RETURNING source_passages.updated_at] [parameters: (None, 'passage-0dd9f78b-d051-4ae3-9a17-45ba4dd10363', 'hello.. sample text.', '{"embedding_endpoint_type": "ollama", "embedding_endpoint": "http://host.docker.internal:11434/", "embedding_model": "nomic-embed-text:latest", "embed ... (12 characters truncated) ... 68, "embedding_chunk_size": 300, "handle": "ollama/nomic-embed-text:latest", "azure_endpoint": null, "azure_version": null, "azure_deployment": null}', '{}', '[0.5046536922454834,-0.8562003374099731,-3.460594654083252,-0.9840149879455566,0.7693641185760498,-0.5242434144020081,0.6660105586051941,0.2501671016 ... (28171 characters truncated) ... ,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]', datetime.datetime(2025, 6, 18, 19, 5, 19, 603997, tzinfo=datetime.timezone.utc), False, 'user-00000000-0000-4000-8000-000000000000', 'user-00000000-0000-4000-8000-000000000000', 'org-00000000-0000-4000-8000-000000000000', 'file-87056ebb-b128-4e5c-b2a5-050d9a850b23', 'source-6a74300d-df0f-40fd-b0c7-2e553fa9e7b4')].

File is a simple text file with some random text. Attaching the file here for quick reference.

sample.txt

Embedding model used : nomic-embed-text
​

[sarahwooders]

Can you please try with 0.8.5? We just patched an issue related to files.

[racingninja]

I did try with 0.8.5.

But this time, it looks at open AI key. But, I am using Ollama server for all operations and the embedding is configured to nomic-embed-text

Error :
Letta.letta.server.rest_api.routers.v1.sources - ERROR - file_processor.process failed with OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable Traceback (most recent call last): File "/app/letta/utils.py", line 1095, in wrapper await coro File "/app/letta/server/rest_api/routers/v1/sources.py", line 363, in load_file_to_source_cloud embedder = OpenAIEmbedder(embedding_config=embedding_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/letta/services/file_processor/embedder/openai_embedder.py", line 27, in __init__ self.client = openai.AsyncOpenAI(api_key=model_settings.openai_api_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/openai/_client.py", line 427, in __init__ raise OpenAIError( openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

Docker command to start the app :
docker run \ -v /Users/rmuraly/ttk:/tiktoken_cache \ -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \ -p 8283:8283 \ -e OLLAMA_BASE_URL="http://host.docker.internal:11434" \ -e TIKTOKEN_CACHE_DIR="/tiktoken_cache" \ letta/letta:0.8.5