planning: Cortex Model Compatibility API

[hantran-co]

Goal

• Cortex can generate a model compatibility prediction, based on user's hardware and model.yaml
• This should be an API that Jan can call (potentially as part of GET /models and GET /model/<model_id>)
• Likely linked to planning: Cortex Hardware API #1165
• Model compatibility should "compute" based on Active Hardware

Related Issues

• Update spec with architecture: Cortex.cpp: Model Compatibility Prediction #1130
• bug: Not enough RAM for all models #1143
• bug: "Recommended" labels don't autoupdate when user toggles CPU and GPU modes #1142
• bug: Arch Detector Causing Special Characters in GPU Name in settings.json #1140
• docs: Draft dev docs and Algolia Docsearch integration jan#1089

Original Post

Specs

https://www.notion.so/jan-ai/Hardware-Detection-and-Recommendations-b04bc3109c2846d58572415125e0a9a5?pvs=4

Key user stories

• Migrate Hardware Settings from Advanced Settings to its own settings page.
• Missing Hardware Dependencies → Ask users to download dependencies to turn on GPU acceleration.
• ...

Design

https://www.figma.com/design/DYfpMhf8qiSReKvYooBgDV/Jan-App-(3rd-version)?node-id=5115-60038&t=OgzCw09qXKxZj3DC-4

[dan-menlo]

Note: This should be driven by Cortex team, with Jan UI as one of the task items.

I think this is part of a larger "Hardware Detection, Config and Recommendations"

• Cortex can detect what hardware the user has: CPU, GPU(s)
• Cortex can be instructed to use certain hardware (e.g. CPU-only, GPU, and which GPU)
• Cortex can run models with specific hardware (e.g. GPU 1)
• Cortex can assess hardware compatibility for a given model

[dan-menlo]

This is also being discussed in janhq/jan#1089 - let's link both issues. We will need to scope this to something less ambigious

• e.g. Cortex can provide "recommendations" in GET /models, based on activated/detected hardware