planning: Cortex Hardware API

[dan-menlo]

Goal

• We should have a very clear Eng Spec for Cortex Hardware API for Sprint 23

Key Functionality

• Hardware Detection
  • Cortex can list all available hardware
• Hardware Activation
  • Cortex has a clear CLI and API to select active hardware
  • Cortex can activate specific hardware (e.g. CPU-only, or specific GPU)
• Hardware -> Engines
  • Engines initialize using activated Hardware
  • List of active hardware is passed down to the engine (e.g. llama.cpp or TensorRT-LLM)
  • How does this interact with ngl settings?
• Hardware Usage Detection
  • Cortex can detect free RAM or VRAM
• Hardware Fallback
  • Fallbacks (e.g. if GPU inference fails due to drivers not installed, run inference on CPU?)
  • feat: Automatic fallback to CPU in case of GPU loading failure #805

Tasklist

• Design API (e.g. GET /hardware)
• Design CLI (e.g. cortex hardware list?)

Functionality

• Hardware Detection
• Hardware Activation
• Hardware Usage Detection (e.g. RAM, VRAM)

Cortex & Jan Integration

• Is it possible for us to have a consistent hardware list across engines?
• Update Spec with architecture docs: Cortex Docs for v1.0.0 cortex.so#193

Previous Issues

• bug: "Recommended" labels don't autoupdate when user toggles CPU and GPU modes #1142
• bug: can enable GPU acceleration with cuda not installed - model fails to start jan#3762
• epic: Cortex can detect track free and used VRAM and RAM #1191 (comment)
• bug: BUG on newest jan v5.0, it set the default to gpu instead of cpu jan#3012

Appendix

UX Goal

Cortex.cpp's Hardware API should enable us to do this in Jan

[dan-menlo]

@louis-jan I'm assigning this to you in Sprint 20, as this has a significant CLI and API design component.

• Can discuss later in the week if you want/can take on the implementation of it
• May be a good exercise for you to gain deep understanding of our C++ codebase

EDIT: adding @nguyenhoangthuan99 for implementation

[dan-menlo]

@louis-jan @nguyenhoangthuan99 I am going to move this to Sprint 21, as I think you guys should land the Model Folder and model.yaml first.