Skip to content

Conversation

@codelion
Copy link
Member

@codelion codelion commented Feb 5, 2025

  • Add support to generate JSON from any model using OpenAI compatible pydantic types or jsonschema.
  • Add support for reasoning_effort, you can use it with deepseek r1 models.
  • Simplify local inference and support flash-atten 2 the local inference now works with very little degradation in performance when compared to vllm for requests that are not batched.

@codelion codelion marked this pull request as draft February 5, 2025 13:59
@codelion codelion changed the title Feat add json plugin Feat add json plugin and support low, medium, high reasoning efforts for thinking models Feb 27, 2025
@codelion codelion marked this pull request as ready for review February 27, 2025 00:46
@codelion codelion merged commit 7be1f50 into main Feb 27, 2025
1 check passed
@codelion codelion deleted the feat-add-json-plugin branch February 27, 2025 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants