Model Naming Convention & Provider Strategy

[crmne]

Thanks for the PRs adding new providers – it's great to see Ruby LLM expanding! As we add support for meta-providers like Bedrock (#50) and OpenRouter (#29), which resell access to models from OpenAI, Anthropic, etc., we need a consistent approach to model names.

The Problem

Right now, we're at risk of inconsistent naming like:

• claude-3-5-sonnet-20241022 (direct from Anthropic)
• anthropic.claude-3-5-sonnet-20241022-v2:0:200k (via Bedrock)
• anthropic/claude-3.5-sonnet (via OpenRouter)

This creates an unnecessary cognitive burden and makes switching providers harder than it needs to be.

The Ruby LLM Way

We'll implement a cleaner approach separating what (model) from where (provider):

# Default way (from the native provider)
chat = RubyLLM.chat(model: "claude-3-5-sonnet")

# Same model via different provider
chat = RubyLLM.chat(model: "claude-3-5-sonnet", provider: :bedrock)

This has several benefits:

1. Clean API - Model names stay simple and consistent
2. Provider flexibility - Switch providers without changing model references
3. Future-proof - New providers can be added without disrupting existing code

Managing models.json

For all providers, including those with large model catalogs like OpenRouter, Ollama, and HuggingFace:

1. Comprehensive models.json - We'll maintain a comprehensive models.json file that includes models from all supported providers
2. Easy refreshing - Users can update their local models catalog with RubyLLM.models.refresh! when needed
3. Regular updates - We'll keep the distributed models.json up-to-date through automated processes

This approach provides the best balance of simplicity, usability, and maintainability.

Model Aliases

To handle provider-specific model identifiers and versions, we'll implement a global model alias system:

1. Global aliases.json - We'll maintain a central aliases.json file mapping friendly model names to provider-specific identifiers
2. Provider mapping - Each provider implementation will use these aliases to map standard names to their specific format

Example aliases.json structure:

{
  "claude-3-5-sonnet": {
    "anthropic": "claude-3-5-sonnet-20241022",
    "bedrock": "anthropic.claude-3-5-sonnet-20241022-v2:0:200k",
    "openrouter": "anthropic/claude-3.5-sonnet"
  },
  "gpt-4o": {
    "openai": "gpt-4o-2024-05-13",
    "bedrock": "anthropic.gpt-4o-2024-05-13",
    "openrouter": "openai/gpt-4o"
  }
}

This allows users to use consistent model names regardless of provider:

# These all use the same logical model through different providers
chat1 = RubyLLM.chat(model: "claude-3-5-sonnet")                      # Uses Anthropic directly
chat2 = RubyLLM.chat(model: "claude-3-5-sonnet", provider: :bedrock)   # Uses via AWS Bedrock
chat3 = RubyLLM.chat(model: "claude-3-5-sonnet", provider: :openrouter) # Uses via OpenRouter

If a model can't be found with the provided ID and provider, a ModelNotFoundError will be raised with an informative message. Your implementation should make this error helpful by suggesting available alternatives.

When the same model has multiple versions and context windows e.g.

anthropic.claude-3-5-sonnet-20240620-v1:0
anthropic.claude-3-5-sonnet-20240620-v1:0:18k
anthropic.claude-3-5-sonnet-20240620-v1:0:200k
anthropic.claude-3-5-sonnet-20240620-v1:0:51k
anthropic.claude-3-5-sonnet-20241022-v2:0
anthropic.claude-3-5-sonnet-20241022-v2:0:18k
anthropic.claude-3-5-sonnet-20241022-v2:0:200k
anthropic.claude-3-5-sonnet-20241022-v2:0:51k

We default all aliases to the biggest context window, and the main alias (without date) to the latest version:

  "claude-3-5-sonnet": {
    "anthropic": "claude-3-5-sonnet-20241022",
    "bedrock": "anthropic.claude-3-5-sonnet-20241022-v2:0:200k",
    "openrouter": "anthropic/claude-3.5-sonnet"
  },
  "claude-3-5-sonnet-20241022": {
    "anthropic": "claude-3-5-sonnet-20241022",
    "bedrock": "anthropic.claude-3-5-sonnet-20241022-v2:0:200k",
    "openrouter": "anthropic/claude-3.5-sonnet"
  },
  "claude-3-5-sonnet-20240620": {
    "anthropic": "claude-3-5-sonnet-20240620",
    "bedrock": "anthropic.claude-3-5-sonnet-20240620-v1:0:200k"
  },

Implementation Guidelines for Contributors

If you're adding a new provider:

1. Use normalized model IDs - Don't include provider prefixes in the model ID itself
2. Add provider mapping - Map the normalized IDs to your provider's specific format internally
3. Preserve capabilities - Ensure models accessed through your provider report the same capabilities as their native counterparts
4. Update models.json - Include your provider's models in models.json
5. Update aliases.json - Add entries to aliases.json for models accessible through your provider
6. Implement refresh mechanism - Ensure your provider supports the list_models method for refreshing

For meta-providers (like Bedrock) that access models from existing providers:

• Models should share the same fundamental ID as their native versions
• Your provider implementation should handle any required translation

This keeps Ruby LLM's API clean and intuitive while giving users the flexibility to choose how they access their models.

[tpaulshippy]

Question - When Bedrock provides four versions of the same model, what should aliases.json and models.json look like? For example, claude-3-5-sonnet-20241022-v2:0 has these model ids available in Bedrock:

anthropic.claude-3-5-sonnet-20241022-v2:0
anthropic.claude-3-5-sonnet-20241022-v2:0:18k
anthropic.claude-3-5-sonnet-20241022-v2:0:200k
anthropic.claude-3-5-sonnet-20241022-v2:0:51k

I ask because the current models.json does not contain these 3 extra models.

[crmne]

Good question @tpaulshippy! Let's default to the version with the biggest context window. I'll update the docs.

[crmne]

Implemented this in eb602d0 and 5be53e6.