Llm integration POC #1028

timfdev · 2025-08-12T22:23:17Z

pydantic-ai and ag-ui-protocol

need pydantic >= 2.10 and >=2.11.2 respectively, this breaks some of the unit tests

…s allowed token count. Make conflicting libraries pydantic-ai and ag-ui optional; disabling agent route if not installed. Make search routes async and fix small bugs in query building.

codspeed-hq · 2025-08-16T23:26:13Z

CodSpeed Performance Report

Merging #1028 will not alter performance

_{Comparing llm-integration (ed8e3ea) with main (0d6b31c)}

Summary

✅ 13 untouched

…hestrator-core into llm-integration

codecov · 2025-08-18T15:53:58Z

Codecov Report

❌ Patch coverage is 49.34243% with 1040 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.26%. Comparing base (0d6b31c) to head (ed8e3ea).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
orchestrator/search/indexing/indexer.py	22.72%	136 Missing ⚠️
orchestrator/search/retrieval/retriever.py	33.11%	101 Missing ⚠️
orchestrator/api/api_v1/endpoints/search.py	32.03%	70 Missing ⚠️
orchestrator/cli/speedtest.py	26.74%	62 Missing and 1 partial ⚠️
orchestrator/search/filters/base.py	42.05%	62 Missing ⚠️
orchestrator/search/core/types.py	69.19%	56 Missing and 5 partials ⚠️
orchestrator/cli/resize_embedding.py	21.21%	51 Missing and 1 partial ⚠️
orchestrator/search/retrieval/utils.py	22.72%	51 Missing ⚠️
orchestrator/search/retrieval/validation.py	25.00%	45 Missing ⚠️
orchestrator/search/retrieval/engine.py	26.66%	44 Missing ⚠️
... and 22 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1028      +/-   ##
==========================================
- Coverage   85.14%   79.26%   -5.89%     
==========================================
  Files         217      254      +37     
  Lines       10496    12543    +2047     
  Branches     1004     1232     +228     
==========================================
+ Hits         8937     9942    +1005     
- Misses       1305     2330    +1025     
- Partials      254      271      +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

orchestrator/api/api_v1/endpoints/search.py

orchestrator/api/api_v1/api.py

orchestrator/cli/search_explore.py

...migrations/versions/schema/2025-08-12_52b37b5b2714_search_index_model_for_llm_integration.py

orchestrator/search/core/embedding.py

orchestrator/search/docs/running_local_text_embedding_inference.md

orchestrator/search/filters/base.py

orchestrator/search/retrieval/builder.py

…ndpoints for autocompleting paths and UI compatible operators per field type for frontend rendering.

… settings and instructions.

…ption records in response, improve highlighting

…hestrator-core into llm-integration

…d substring highlighting

… data)

…ith just a field name and value type. Support component contains/not contains filters.

tjeerddie · 2025-09-15T07:39:04Z

orchestrator/search/agent/prompts.py

+    except Exception as e:
+        logger.warning(f"Failed to load schema for prompt: {e}")
+        schema_info = "    Schema temporarily unavailable"
+    logger.error(f"Generated schema for agent prompt:\n{schema_info}")


I don't think this is suppose to be an error log?

orchestrator/search/indexing/traverse.py

orchestrator/settings.py

tjeerddie · 2025-09-15T09:31:41Z

orchestrator/search/retrieval/engine.py

+def _extract_matching_field_from_filters(filters: FilterTree) -> MatchingField | None:
+    """Extract the first path filter to use as matching field for structured searches.
+
+    TODO: Should we allow a list of matched fields in the MatchingField model?


what to do with this? new issue?

Mark90

Nice, that's a lot of work 🔥

Overall structure of the code is good, that's why I was able to leave a lot of questions and small remarks. I mean this as a good thing :)

.github/workflows/run-codspeed-tests.yml

orchestrator/api/api_v1/endpoints/search.py

Mark90 · 2025-09-15T07:20:48Z

orchestrator/api/api_v1/endpoints/agent.py

+
+
+def build_agent_app() -> ASGIApp:
+    if not app_settings.AGENT_MODEL or not app_settings.OPENAI_API_KEY:


These settings are strings that can't be None so by default it will be enabled. Since users need to configure the LLM setup, by default it should IMO be disabled with a bool variable like AGENT_ENABLED

Mark90 · 2025-09-15T07:24:38Z

orchestrator/search/agent/agent.py

What kind of failures has this shown?

...migrations/versions/schema/2025-08-12_52b37b5b2714_search_index_model_for_llm_integration.py

Mark90 · 2025-09-16T13:08:11Z

orchestrator/search/retrieval/retriever.py

+                entity_scores.join(entity_highlights, entity_scores.c.entity_id == entity_highlights.c.entity_id)
+            )
+        ).cte("ranked_results")
+


Could we split this function up in one for the DB interaction part which produces an output, and another function that performs the below computations based on the former's output? And preferably also some unittests for the latter

Mark90 · 2025-09-16T13:10:56Z

orchestrator/search/retrieval/retriever.py

@@ -0,0 +1,447 @@
+from abc import ABC, abstractmethod


Maybe split up into a package with a module for each retriever type, it's a lot of scrolling now :)

Mark90 · 2025-09-16T13:27:55Z

orchestrator/search/retrieval/retriever.py

+
+    def _quantize_score_for_pagination(self, score_value: float) -> BindParameter[Decimal]:
+        """Convert score value to properly quantized Decimal parameter for pagination."""
+        pas_dec = Decimal(str(score_value)).quantize(Decimal("0.000000000001"))


Should this change along with the SCORE_PRECISION if that ever changes?

If so maybe do something like f'{1 / 10**precision:.{precision}f}

Mark90 · 2025-09-16T13:38:49Z

orchestrator/search/retrieval/utils.py

+
+        if not matches:
+            substring_pattern = re.escape(word)
+            matches = list(re.finditer(substring_pattern, text, re.IGNORECASE))


If a resulting text has both word and substring matches, wouldn't we want to highlight the substring matches as well?

Mark90 · 2025-09-16T13:43:26Z

orchestrator/search/schemas/results.py

+
+class TypeDefinition(BaseModel):
+    operators: list[FilterOp]
+    valueSchema: dict[FilterOp, ValueSchema]


Is camelCase needed here?

Sharp one, no I think this is left over from something I tried with pydantic aliases to use camelCase in the response, but it was hard to keep that consistent for deep nested data.

#1069) * Refactor traverse.py to use model based traversal with typing introspection. Included with full unittest coverage * some fixes * move type mapping to types file and fix linting errors.

…e to content hash, add test coverage

… check instead of relying on typehints

…subscription for traversal

* Make the LLM module more configurable and do not install all deps straight away * Fix linting problems * Agentic app * Fixes * Simplify start up * lint issue * Added some initial documentation for the LLM module --------- Co-authored-by: Tim Frohlich <[email protected]>

timfdev added 8 commits August 13, 2025 00:20

Vector search and agent mode POC

99da746

l

960ce80

add ag-ui package

d34d467

fix linting

cede6f5

last lint fix

65e963d

Streaming pipeline for indexing, using litellm to track token count v…

90a5d1a

…s allowed token count. Make conflicting libraries pydantic-ai and ag-ui optional; disabling agent route if not installed. Make search routes async and fix small bugs in query building.

fix mypy issues & use pgvector image

d0a23ec

use pgvector for codspeed tests

4fce33d

timfdev and others added 3 commits August 17, 2025 01:30

Merge branch 'main' into llm-integration

b8b4eb8

update docs and cleanup

1ec3625

Merge branch 'llm-integration' of github.com:workfloworchestrator/orc…

5d4c316

…hestrator-core into llm-integration

mrijk reviewed Aug 19, 2025

View reviewed changes

orchestrator/api/api_v1/endpoints/search.py Outdated Show resolved Hide resolved

use python 3.10+ style type hinting

f837226

luc-tielen reviewed Aug 19, 2025

View reviewed changes

timfdev and others added 10 commits August 23, 2025 19:47

refactor from a list of filter conditions to a filter tree; Include e…

9a70722

…ndpoints for autocompleting paths and UI compatible operators per field type for frontend rendering.

small bugfixes

816ead4

Bump pydantic to 2.11

811ebea

CLI command to reshape vector embeddings column, improved local setup…

1348d0a

… settings and instructions.

Update mask_value for masking exposed settings and fix unit tests

ca2a4ff

Bump version to 5.0.0a1

5c8025f

Add keyset pagination, include search metadata, load detailed subscri…

29ae887

…ption records in response, improve highlighting

Merge branch 'llm-integration' of github.com:workfloworchestrator/orc…

c6635dc

…hestrator-core into llm-integration

Speedtest, improved retrieval speed by limiting search space, improve…

6a4cada

…d substring highlighting

Merge main into llm-integration branch (clean merge without sensitive…

e9597e1

… data)

timfdev force-pushed the llm-integration branch from 3840906 to e9597e1 Compare September 3, 2025 12:37

timfdev added 3 commits September 3, 2025 14:54

Normalize all retriever scores to 0-1 range and other small fixes

8830633

fix linting issues

3a37786

Add matchedfields for all endpoints and for structured searches

0e54272

timfdev added 5 commits September 10, 2025 01:58

refactor path endpoint and filters to simplify structured filtering w…

7634d24

…ith just a field name and value type. Support component contains/not contains filters.

negation on group level, not record level

3fdab01

Merge main branch

7a8db9e

Merge remote-tracking branch 'origin/pydantic-2.11' into llm-integration

0a39261

Make agent packages required and remove import safequards

c35a073

tjeerddie reviewed Sep 15, 2025

View reviewed changes

orchestrator/search/indexing/traverse.py Outdated Show resolved Hide resolved

tjeerddie reviewed Sep 15, 2025

View reviewed changes

orchestrator/settings.py Outdated Show resolved Hide resolved

tjeerddie reviewed Sep 15, 2025

View reviewed changes

tjeerddie approved these changes Sep 15, 2025

View reviewed changes

Mark90 self-requested a review September 16, 2025 12:52

Mark90 reviewed Sep 16, 2025

View reviewed changes

timfdev and others added 8 commits September 18, 2025 08:26

Refactor traverse.py to use model based traversal with typing introsp… (

395c820

#1069) * Refactor traverse.py to use model based traversal with typing introspection. Included with full unittest coverage * some fixes * move type mapping to types file and fix linting errors.

Sanitize product name for ltree, batched index deletes, add value_typ…

dad435d

…e to content hash, add test coverage

Sanitize product name for ltree, batched index deletes, add value_typ…

4d9b2eb

…e to content hash, add test coverage

Now that we are introspecting models, is_embeddable should do a value…

0d29a85

… check instead of relying on typehints

Mimick from_product_id without database operations to use a template …

02ee456

…subscription for traversal

Mimick from_product_id without database operations to use a template …

d6adf93

…subscription for traversal

Merge main

e1ac91d

pboers1988 merged commit 89561a4 into main Sep 18, 2025
15 checks passed

pboers1988 deleted the llm-integration branch September 18, 2025 14:08



		def build_agent_app() -> ASGIApp:
		if not app_settings.AGENT_MODEL or not app_settings.OPENAI_API_KEY:

Llm integration POC #1028

Llm integration POC #1028

Uh oh!

Conversation

timfdev commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq bot commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #1028 will not alter performance

Summary

Uh oh!

codecov bot commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mark90 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

timfdev commented Aug 12, 2025 •

edited

Loading

codspeed-hq bot commented Aug 16, 2025 •

edited

Loading

codecov bot commented Aug 18, 2025 •

edited

Loading