fix(ask_sb): Fix long generation times on first message in thread #447
Conversation
WalkthroughThe changes introduce and integrate a new mechanism for generating and updating chat names using AI models. The chat API route is refactored for clarity and modularity, delegating model selection and streaming logic to new helpers. The chat UI now triggers AI-based chat name generation after the first message, and style fixes are made in a utility file. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant ChatThread (UI)
participant API Route (POST)
participant Auth & Org Middleware
participant Model Helper
participant AI Model
participant DB
User->>ChatThread (UI): Send first message
ChatThread (UI)->>API Route (POST): POST /chat (message)
API Route (POST)->>Auth & Org Middleware: Authenticate & check org
Auth & Org Middleware-->>API Route (POST): Auth result
API Route (POST)->>DB: Validate chat, check readonly
DB-->>API Route (POST): Chat info
API Route (POST)->>Model Helper: Get model & options
Model Helper-->>API Route (POST): Model instance/options
API Route (POST)->>AI Model: Stream message response
AI Model-->>API Route (POST): Streamed data
API Route (POST)-->>ChatThread (UI): Stream response
Note over ChatThread (UI): After first message sent
ChatThread (UI)->>actions.ts: generateAndUpdateChatNameFromMessage
actions.ts->>Auth & Org Middleware: Authenticate & check org
actions.ts->>Model Helper: Get AI model
Model Helper-->>actions.ts: Model instance
actions.ts->>AI Model: generateText(prompt)
AI Model-->>actions.ts: Chat name
actions.ts->>DB: updateChatName
DB-->>actions.ts: Success
actions.ts-->>ChatThread (UI): Success/failure
Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Possibly related PRs
Suggested reviewers
Poem
Note 🔌 MCP (Model Context Protocol) integration is now available in Early Access!Pro users can now connect to remote MCP servers under the Integrations page to get reviews and chat conversations that understand additional development context. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
This comment has been minimized.
This comment has been minimized.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
packages/web/src/features/chat/actions.ts (2)
207-221: Consider extracting the prompt and adding validation.The prompt is well-crafted but could be extracted to a constant for better maintainability. Additionally, consider validating the generated text length before updating the chat name.
+const CHAT_NAME_GENERATION_PROMPT = `Convert this question into a short topic title (max 50 characters). + +Rules: +- Do NOT include question words (what, where, how, why, when, which) +- Do NOT end with a question mark +- Capitalize the first letter of the title +- Focus on the subject/topic being discussed +- Make it sound like a file name or category + +Examples: +"Where is the authentication code?" → "Authentication Code" +"How to setup the database?" → "Database Setup" +"What are the API endpoints?" → "API Endpoints" + +User question: `; export const generateAndUpdateChatNameFromMessage = async ({ chatId, languageModelId, message }: { chatId: string, languageModelId: string, message: string }, domain: string) => sew(() => // ... existing code ... - const prompt = `Convert this question into a short topic title (max 50 characters). - -Rules: -- Do NOT include question words (what, where, how, why, when, which) -- Do NOT end with a question mark -- Capitalize the first letter of the title -- Focus on the subject/topic being discussed -- Make it sound like a file name or category - -Examples: -"Where is the authentication code?" → "Authentication Code" -"How to setup the database?" → "Database Setup" -"What are the API endpoints?" → "API Endpoints" - -User question: ${message}`; + const prompt = CHAT_NAME_GENERATION_PROMPT + message;
375-562: Add a default case for defensive programming.While TypeScript should ensure exhaustiveness, adding a default case would provide better runtime safety if an unexpected provider value is encountered.
Add after line 560:
return { model: xai(modelId), }; } + default: { + throw new Error(`Unsupported provider: ${provider}`); + } } }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
packages/web/src/app/api/(server)/chat/route.ts(3 hunks)packages/web/src/features/chat/actions.ts(3 hunks)packages/web/src/features/chat/components/chatThread/chatThread.tsx(2 hunks)packages/web/src/features/chat/utils.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*
📄 CodeRabbit Inference Engine (.cursor/rules/style.mdc)
Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.
Files:
packages/web/src/features/chat/utils.tspackages/web/src/features/chat/components/chatThread/chatThread.tsxpackages/web/src/app/api/(server)/chat/route.tspackages/web/src/features/chat/actions.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (6)
packages/web/src/features/chat/utils.ts (1)
1-3: LGTM! Style improvements for consistency.Adding semicolons to import statements improves code consistency and prevents potential ASI issues.
Also applies to: 17-17
packages/web/src/features/chat/components/chatThread/chatThread.tsx (2)
125-145: Well-implemented async chat name generation!The fire-and-forget pattern for generating chat names is appropriate here, as it prevents blocking the message stream. Good error handling with user-friendly toast notifications.
146-154: Dependency array correctly updated.All necessary dependencies are included for the enhanced sendMessage callback.
packages/web/src/features/chat/actions.ts (1)
418-420: Note: Beta header for Anthropic interleaved thinking.The code uses a beta API feature. Ensure this is monitored for stability and migration when it becomes generally available.
packages/web/src/app/api/(server)/chat/route.ts (2)
54-91: Excellent validation improvements!The added checks for chat existence and readonly status improve the API's robustness. The error messages are clear and helpful.
137-264: Well-structured stream response creation!The extracted
createMessageStreamResponsefunction improves code organization and readability. Good separation of concerns with clear parameter passing.
brendan-kellam
force-pushed
the
bkellam/unblocked_chat_name_gen
branch
from
b95408b to
2706754
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
packages/web/src/features/chat/actions.ts (2)
5-28: Consider dynamic imports for AI SDK providers to reduce bundle size.While the comprehensive provider support is excellent, importing all AI SDK providers upfront may increase the bundle size significantly. Consider using dynamic imports within the switch cases of
_getAISDKLanguageModelAndOptionsto only load the required providers at runtime.For example:
- import { createAmazonBedrock } from '@ai-sdk/amazon-bedrock'; // ... other static imports case 'amazon-bedrock': { + const { createAmazonBedrock } = await import('@ai-sdk/amazon-bedrock'); const aws = createAmazonBedrock({ // ... configuration });
207-221: Consider extracting the prompt template for maintainability.The hardcoded prompt is well-crafted but could benefit from being extracted to a constant or configuration for easier maintenance and potential localization.
+ const CHAT_NAME_PROMPT_TEMPLATE = `Convert this question into a short topic title (max 50 characters). + + Rules: + - Do NOT include question words (what, where, how, why, when, which) + - Do NOT end with a question mark + - Capitalize the first letter of the title + - Focus on the subject/topic being discussed + - Make it sound like a file name or category + + Examples: + "Where is the authentication code?" → "Authentication Code" + "How to setup the database?" → "Database Setup" + "What are the API endpoints?" → "API Endpoints" + + User question: `; - const prompt = `Convert this question into a short topic title (max 50 characters). - - Rules: - - Do NOT include question words (what, where, how, why, when, which) - - Do NOT end with a question mark - - Capitalize the first letter of the title - - Focus on the subject/topic being discussed - - Make it sound like a file name or category - - Examples: - "Where is the authentication code?" → "Authentication Code" - "How to setup the database?" → "Database Setup" - "What are the API endpoints?" → "API Endpoints" - - User question: ${message}`; + const prompt = `${CHAT_NAME_PROMPT_TEMPLATE}${message}`;
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
CHANGELOG.md(1 hunks)packages/web/src/app/api/(server)/chat/route.ts(3 hunks)packages/web/src/features/chat/actions.ts(3 hunks)packages/web/src/features/chat/components/chatThread/chatThread.tsx(2 hunks)packages/web/src/features/chat/utils.ts(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/web/src/features/chat/utils.ts
- CHANGELOG.md
- packages/web/src/features/chat/components/chatThread/chatThread.tsx
🧰 Additional context used
📓 Path-based instructions (1)
**/*
📄 CodeRabbit Inference Engine (.cursor/rules/style.mdc)
Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.
Files:
packages/web/src/features/chat/actions.tspackages/web/src/app/api/(server)/chat/route.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (5)
packages/web/src/features/chat/actions.ts (2)
188-238: LGTM! Well-structured async chat name generation.The implementation properly addresses the PR objective of making title generation asynchronous. The function has excellent error handling, authentication checks, and prompt engineering with clear rules and examples.
375-562: Excellent centralization of AI model instantiation logic.This helper function effectively consolidates provider-specific configuration and credential handling, which aligns perfectly with the DRY principle. The secure token retrieval and provider-specific options handling is well implemented.
packages/web/src/app/api/(server)/chat/route.ts (3)
54-106: Excellent refactoring with improved validation and structure.The integration of validation logic directly into the POST handler improves code clarity and ensures proper checks are performed before processing. The use of the centralized
_getAISDKLanguageModelAndOptionshelper eliminates code duplication and follows the DRY principle.
137-264: Well-implemented streaming response handler.The
createMessageStreamResponsefunction effectively handles the complex streaming workflow with proper message processing, repository scope expansion, and metadata collection. The separation of concerns and error handling with Sentry integration demonstrates good architectural practices.
253-258: Good use of onFinish callback for database updates.The placement of database updates in the
onFinishcallback ensures that messages are persisted only after successful streaming completion, which prevents partial data corruption in case of stream failures.
This PR fixes (or at least reduces) a issue we've seen where the first message in a thread takes longer to generate. I believe this was caused because the "generate title" step was blocking the actual chat stream. I observed ~10-15s generation times for gemini-2.5-pro.
This PR does not address making the title generation faster. Instead, I've made it so the title generation happens asynchronously from the chat stream s.t., it won't block it.
Summary by CodeRabbit
New Features
Bug Fixes
Style
Refactor