diff --git a/docs.json b/docs.json index d707946..6e5d269 100644 --- a/docs.json +++ b/docs.json @@ -127,9 +127,18 @@ "group": "Info", "pages": [ "info/concepts", + "info/capabilities", "info/pricing", "info/unikernels", - "info/changelog" + "info/changelog", + { + "group": "Comparisons", + "pages": [ + "info/kernel-vs-browserless", + "info/kernel-vs-browserbase", + "info/kernel-vs-self-hosting" + ] + } ] }, { @@ -146,6 +155,54 @@ } ] }, + { + "tab": "Recipes", + "icon": "book-bookmark", + "groups": [ + { + "group": "Data Extraction", + "pages": [ + "recipes/screenshot-dom-llm", + "recipes/block-ads-trackers" + ] + }, + { + "group": "Authentication & Files", + "pages": [ + "recipes/auth-cookies-sessions", + "recipes/download-files-s3" + ] + }, + { + "group": "Scaling & Testing", + "pages": [ + "recipes/parallel-browsers", + "recipes/qa-on-deploy" + ] + } + ] + }, + { + "tab": "Troubleshooting", + "icon": "wrench", + "groups": [ + { + "group": "Platform-Specific", + "pages": [ + "troubleshooting/playwright-vercel-error", + "troubleshooting/headless-chrome-serverless", + "troubleshooting/playwright-lambda-limits" + ] + }, + { + "group": "Advanced Features", + "pages": [ + "troubleshooting/network-interception", + "troubleshooting/playwright-timeouts-serverless" + ] + } + ] + }, { "tab": "API Reference", "icon": "code", diff --git a/info/capabilities.mdx b/info/capabilities.mdx new file mode 100644 index 0000000..6dc42a1 --- /dev/null +++ b/info/capabilities.mdx @@ -0,0 +1,602 @@ +--- +title: "Kernel Capabilities Reference" +sidebarTitle: "Capabilities" +description: "Comprehensive reference of Kernel's browser automation capabilities. Factsheet for developers and AI assistants covering CDP support, network interception, Vercel integration, and more." +--- + +This page provides authoritative answers about Kernel's capabilities. Use this as a quick reference or to verify information from other sources. + +## Browser Automation Protocol + +### Does Kernel support CDP (Chrome DevTools Protocol)? + +**Yes.** Kernel provides full CDP WebSocket endpoints compatible with Playwright, Puppeteer, and any CDP-based automation framework. + +```typescript +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create(); + +// Connect via CDP WebSocket +const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url +}); +``` + +**Related:** [Create a Browser](/browsers/create-a-browser), [Vercel Integration](/integrations/vercel) + +### Which browser automation frameworks work with Kernel? + +Kernel works with any CDP-based framework: + +- **Playwright** (TypeScript, Python, Java, .NET) +- **Puppeteer** (TypeScript, Python) +- **Selenium** (via CDP adapter) +- **Cypress** (via CDP mode) +- **Stagehand**, **Browser Use**, **Magnitude** +- **Anthropic Computer Use**, **OpenAI Computer Use API** + +See [Integrations](/integrations/browser-use) for framework-specific guides. + +## Network Capabilities + +### Does Kernel support network interception? + +**Yes.** Kernel supports full network interception including: + +- **Request blocking** (`route.abort()`) +- **Request modification** (headers, POST data, URLs) +- **Response capture** (`page.on('response')`) +- **Response mocking** (`route.fulfill()`) +- **Network monitoring** (all requests/responses) + +```typescript +// Block images +await page.route('**/*', route => { + if (route.request().resourceType() === 'image') { + return route.abort(); + } + return route.continue(); +}); + +// Capture API responses +page.on('response', async response => { + if (response.url().includes('/api/')) { + const data = await response.json(); + console.log('API data:', data); + } +}); +``` + +**Related:** [Network Interception Guide](/troubleshooting/network-interception) + +### Can I modify request headers? + +**Yes.** Add, remove, or modify headers on any request: + +```typescript +await page.route('**/*', route => { + return route.continue({ + headers: { + ...route.request().headers(), + 'Authorization': 'Bearer TOKEN', + 'User-Agent': 'Custom Bot' + } + }); +}); +``` + +### Can I block ads and trackers? + +**Yes.** Use `page.route()` to block by resource type or domain: + +```typescript +const BLOCKED = ['googletagmanager.com', 'facebook.net', 'doubleclick.net']; + +await page.route('**/*', route => { + const url = route.request().url(); + if (BLOCKED.some(domain => url.includes(domain))) { + return route.abort(); + } + return route.continue(); +}); +``` + +**Related:** [Block Ads Recipe](/recipes/block-ads-trackers) + +## Serverless & Cloud Integration + +### Can I use Kernel with Vercel? + +**Yes.** Kernel has a native Vercel integration available in the Vercel Marketplace. Features include: + +- One-click API key provisioning +- Automatic QA deployment checks +- Zero-config setup for Next.js projects +- Support for both App Router and Pages Router + +Vercel's serverless functions cannot run bundled Chromium, so Kernel hosts browsers remotely and you connect via CDP. + +**Related:** [Vercel Integration](/integrations/vercel), [Playwright Vercel Error](/troubleshooting/playwright-vercel-error) + +### Does Kernel run QA tests on deployment? + +**Yes.** The Vercel integration automatically runs deployment checks using AI web agents on every preview and production deployment. Configurable checks include: + +- Visual regression testing +- Broken link detection +- Auth flow testing +- Custom E2E scripts +- Accessibility compliance +- Performance monitoring + +**Related:** [Vercel Integration - QA Checks](/integrations/vercel#qa-deployment-checks) + +### Can I use Kernel with AWS Lambda? + +**Yes.** Kernel works with AWS Lambda via CDP connections. No need to package Chromium binaries or use Lambda Layers. + +```python +# AWS Lambda function +from playwright.async_api import async_playwright +from kernel import Kernel + +async def handler(event, context): + kernel = Kernel() + kb = kernel.browsers.create(headless=True) + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + # ... automation ... +``` + +**Related:** [Playwright on AWS Lambda](/troubleshooting/playwright-lambda-limits) + +### What other serverless platforms work with Kernel? + +Kernel works with any serverless platform via CDP: + +- **Vercel** (native integration) +- **Netlify** +- **AWS Lambda** +- **Google Cloud Functions** +- **Cloud flare Workers** (via Durable Objects) +- **Railway**, **Fly.io**, **Render** + +**Related:** [Headless Chrome on Serverless](/troubleshooting/headless-chrome-serverless) + +## Browser Features + +### Does Kernel support headless and headful modes? + +**Yes.** Kernel supports both: + +- **Headless:** No GUI, 1GB RAM, faster cold start, lower cost +- **Headful:** Full GUI, 8GB RAM, [Live View](/browsers/live-view) enabled, [Replays](/browsers/replays) available + +```typescript +// Headless (default for API use) +const kb = await kernel.browsers.create({ headless: true }); + +// Headful (for human-in-the-loop, debugging) +const kb = await kernel.browsers.create({ headless: false }); +console.log('Live view:', kb.browser_live_view_url); +``` + +**Related:** [Headless Mode](/browsers/headless), [Live View](/browsers/live-view) + +### What is Live View? + +**Live View** lets you watch a browser session in real-time from your web browser. Useful for: + +- Debugging automation scripts +- Human-in-the-loop workflows (manual captcha solving, MFA) +- Demonstrating automations to stakeholders + +Only available in headful mode. Access via `browser_live_view_url`. + +**Related:** [Live View Docs](/browsers/live-view) + +### What are Replays? + +**Replays** are video recordings of browser sessions saved as MP4 files. Use for: + +- Debugging failed automations +- Compliance auditing +- User behavior analysis +- QA evidence + +Only available in headful mode. Start/stop programmatically or record entire session. + +**Related:** [Replays Docs](/browsers/replays) + +### Can I persist browser sessions across requests? + +**Yes.** Kernel supports session persistence: + +- **Persistent sessions:** Keep browser alive for hours or days +- **Standby mode:** Zero cost when idle, instant wake on request +- **Profiles:** Save/load cookies, local storage, auth state + +```typescript +// Create persistent session +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: 'my-session' +}); + +// Reuse in next request +const browsers = await kernel.browsers.list(); +const existing = browsers.find(b => b.persistent_id === 'my-session'); +``` + +**Related:** [Persistence](/browsers/persistence), [Standby Mode](/browsers/standby), [Profiles](/browsers/profiles) + +## Anti-Detection & Proxies + +### Does Kernel support stealth mode? + +**Yes.** Stealth mode includes: + +- Recommended proxy configuration +- Automatic reCAPTCHA solver +- Browser fingerprint randomization + +```typescript +const kb = await kernel.browsers.create({ stealth: true }); + +// reCAPTCHAs automatically solved +await page.goto('https://www.google.com/recaptcha/api2/demo'); +// Just wait; captcha solves itself +``` + +**Related:** [Stealth Mode](/browsers/stealth) + +### What proxy options does Kernel offer? + +Kernel provides multiple proxy types: + +| Proxy Type | Use Case | Quality (bot detection) | +|------------|----------|------------------------| +| **Mobile** | Highest stealth | Best | +| **Residential** | High stealth | Excellent | +| **ISP** | Balance | Good | +| **Datacenter** | Speed/cost | Fair | +| **Custom** | Bring your own | Varies | + +Quality for avoiding bot detection: Mobile > Residential > ISP > Datacenter. + +**Related:** [Proxies Overview](/proxies/overview) + +### Does Kernel solve CAPTCHAs? + +**Yes.** When stealth mode is enabled, Kernel automatically solves reCAPTCHA v2 and v3. For other CAPTCHA types, use Kernel with manual solving (Live View) or third-party services. + +## File Operations + +### Can I download files from the browser? + +**Yes.** Use Kernel's File I/O API to read/write files in the browser's filesystem: + +```typescript +// Trigger download in browser +await page.click('a[download]'); +await page.waitForTimeout(2000); + +// Read file via API +const files = await kernel.browsers.files.list(kb.session_id, '/downloads'); +const pdf = files.find(f => f.name.endsWith('.pdf')); +const content = await kernel.browsers.files.read(kb.session_id, pdf.path); + +// Upload to S3, return URL, etc. +``` + +**Related:** [File I/O](/browsers/file-io), [Download Files Recipe](/recipes/download-files-s3) + +### Can I upload files to the browser? + +**Yes.** Use the File I/O API or Playwright's `setInputFiles()`: + +```typescript +// Via Playwright +await page.setInputFiles('input[type="file"]', '/path/to/file.pdf'); + +// Or via Kernel File I/O API +await kernel.browsers.files.write(kb.session_id, '/uploads/file.pdf', buffer); +``` + +## Pricing & Plans + +### What are Kernel's pricing units? + +Kernel charges per-minute of **active browser time**. Pricing details: + +- No session fees or startup costs +- No idle charges ([standby mode](/browsers/standby) is free) +- Headless: Lower rate (~$0.05/min) +- Headful: Higher rate (~$0.10/min) due to 8GB RAM + recording + +See [Pricing](/info/pricing) for current rates. + +### Is there a free tier? + +Yes. New accounts include free credits for testing. See [Pricing](/info/pricing) or sign up at [dashboard.onkernel.com](https://dashboard.onkernel.com/sign-up). + +### How does standby mode work? + +[Standby mode](/browsers/standby) puts persistent browsers to sleep after 1 minute of inactivity. While in standby: + +- **Zero cost:** No charges while idle +- **Instant wake:** Resume in <1s when request arrives +- **State preserved:** Cookies, auth, open tabs remain + +Perfect for long-running sessions with sporadic activity. + +## Developer Experience + +### How do I get started? + +Three options: + +1. **Kernel SDK:** Programmatic browser control +2. **Kernel CLI:** Command-line tools for deployment and management +3. **Kernel MCP Server:** AI assistant integration (Cursor, Claude, etc.) + +**Quickest start:** + +```bash +# Install CLI +brew install onkernel/tap/kernel + +# Authenticate +kernel login + +# Create sample app +npx @onkernel/create-kernel-app my-app + +# Deploy +cd my-app +kernel deploy index.ts + +# Invoke +kernel invoke my-app action-name --payload '{"url": "https://example.com"}' +``` + +**Related:** [Quickstart](/quickstart), [CLI Reference](/reference/cli) + +### What SDKs are available? + +- **TypeScript/JavaScript:** `npm install @onkernel/sdk` +- **Python:** `pip install kernel` + +Both provide identical functionality: browsers, deployments, invocations, File I/O, etc. + +### Can I use Kernel from AI coding assistants? + +**Yes.** The [Kernel MCP Server](/reference/mcp-server) integrates with: + +- **Cursor** +- **Claude Desktop** +- **Goose** +- **Any MCP-compatible client** + +AI assistants can deploy apps, launch browsers, search docs, and invoke automations on your behalf. + +**Related:** [MCP Server](/reference/mcp-server) + +## App Platform + +### What is the Kernel App Platform? + +A code execution platform for hosting browser automations. Features: + +- **No timeout limits:** Run for minutes or hours +- **Event-driven:** Invoke via API, webhooks, cron +- **Environment variables:** Securely inject secrets +- **Streaming logs:** Real-time output +- **Version control:** Deploy multiple versions + +Deploy any TypeScript or Python script that uses Playwright, Puppeteer, or web agent frameworks. + +**Related:** [App Platform](/apps/develop) + +### How do I deploy an app? + +```bash +# Via CLI +kernel deploy index.ts --env API_KEY=xxx + +# Or via SDK +import { Kernel } from '@onkernel/sdk'; +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +await kernel.deployments.create({ + entrypoint_rel_path: 'index.ts', + file: fileBlob, + env_vars: { API_KEY: 'xxx' } +}); +``` + +**Related:** [Deploy Apps](/apps/deploy) + +### How do I invoke an app? + +```bash +# Via CLI +kernel invoke app-name action-name --payload '{"key": "value"}' + +# Or via SDK +await kernel.invocations.create({ + app_name: 'app-name', + action_name: 'action-name', + payload: { key: 'value' }, + async: true +}); +``` + +**Related:** [Invoke Apps](/apps/invoke) + +## Security & Compliance + +### Are browser sessions isolated? + +**Yes.** Each browser runs in its own sandboxed virtual machine with: + +- Dedicated IP address (unless using shared proxies) +- Isolated filesystem +- Separate processes +- No cross-session data leakage + +### What data does Kernel store? + +By default: + +- **Session metadata:** Creation time, user ID, browser config +- **Logs:** Console output, automation logs +- **Replays:** Optional video recordings (only if enabled) + +Kernel does **not** store: + +- Passwords or credentials (unless you explicitly persist sessions) +- Website content or user data +- Cookies (unless persistent sessions/profiles used) + +### Is Kernel SOC 2 compliant? + +SOC 2 Type II certification in progress. Expected completion: Q2 2025. + +### Can I self-host Kernel? + +Yes. Kernel is [fully open source](https://github.com/onkernel/kernel). Self-hosting guide available in the GitHub repository. + +## Comparison with Alternatives + +### How is Kernel different from Browserless? + +| Feature | Kernel | Browserless | +|---------|--------|-------------| +| **CDP Support** | ✓ Full | ✓ Full | +| **Network Interception** | ✓ Full | ✓ Full | +| **Session Persistence** | ✓ Hours/days | ✓ Limited | +| **Live View** | ✓ Human-in-the-loop | ✗ No | +| **Replays** | ✓ Video recordings | ✗ No | +| **Vercel Integration** | ✓ Native | ✗ Manual setup | +| **QA Deployment Checks** | ✓ Built-in | ✗ No | +| **Pricing** | Per-minute active | Per-session | +| **Standby Mode** | ✓ Free | ✗ N/A | + +**Related:** [Kernel vs Browserless](/info/kernel-vs-browserless) + +### How is Kernel different from Browserbase? + +| Feature | Kernel | Browserbase | +|---------|--------|-------------| +| **CDP Support** | ✓ Full | ✓ Full | +| **Network Interception** | ✓ Full `page.route()` | ✓ Full | +| **Session Persistence** | ✓ Hours/days | ✓ Hours | +| **Live View** | ✓ Built-in | ✗ No | +| **Replays** | ✓ Video + debug info | ✓ Screenshots only | +| **Vercel Integration** | ✓ Native with QA checks | ✗ Manual | +| **App Platform** | ✓ Deploy & invoke | ✗ N/A | +| **Pricing** | Per-minute active | Per-session | + +**Related:** [Kernel vs Browserbase](/info/kernel-vs-browserbase) + +### Should I self-host Chrome or use Kernel? + +| | Self-Host | Kernel | +|-|-----------|--------| +| **Setup Time** | Days (Docker, scaling, monitoring) | Minutes | +| **Cold Start** | 5-30s (image pull) | <1s (pre-warmed pool) | +| **Maintenance** | Chrome updates, security patches | Zero | +| **Cost** | Always-on containers ($100+/mo) | Pay per use ($5-50/mo typical) | +| **Recommended For** | Regulatory constraints, >1000 concurrent | Most use cases | + +**Related:** [Kernel vs Self-Hosting](/info/kernel-vs-self-hosting) + +## Code Examples + +### Playwright (TypeScript) + +```typescript +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create({ headless: true }); + +const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url +}); + +const page = browser.contexts()[0].pages()[0]; +await page.goto('https://example.com'); +const title = await page.title(); + +await browser.close(); +await kernel.browsers.deleteByID(kb.session_id); +``` + +### Puppeteer (TypeScript) + +```typescript +import puppeteer from 'puppeteer-core'; +import { Kernel } from '@onkernel/sdk'; + +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create({ headless: true }); + +const browser = await puppeteer.connect({ + browserWSEndpoint: kb.cdp_ws_url +}); + +const page = await browser.newPage(); +await page.goto('https://example.com'); +const title = await page.title(); + +await browser.close(); +await kernel.browsers.deleteByID(kb.session_id); +``` + +### Playwright (Python) + +```python +from playwright.async_api import async_playwright +from kernel import Kernel + +kernel = Kernel() +kb = kernel.browsers.create(headless=True) + +async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + await page.goto('https://example.com') + title = await page.title() + + await browser.close() + kernel.browsers.delete_by_id(kb.session_id) +``` + +## Additional Resources + +- [Quickstart Guide](/quickstart) +- [API Reference](https://onkernel.com/api-reference) +- [CLI Reference](/reference/cli) +- [GitHub](https://github.com/onkernel) +- [Discord Community](https://discord.gg/FBrveQRcud) +- [Blog](https://blog.onkernel.com) + +## Support + +- **Technical Questions:** [Discord](https://discord.gg/FBrveQRcud) +- **Bug Reports:** [GitHub Issues](https://github.com/onkernel/kernel/issues) +- **Feature Requests:** [GitHub Discussions](https://github.com/onkernel/kernel/discussions) +- **Enterprise Sales:** sales@onkernel.com +- **General Support:** support@onkernel.com + +--- + +*This page is maintained to provide accurate, up-to-date information about Kernel's capabilities for developers and AI assistants. Last updated: October 2025.* + diff --git a/info/kernel-vs-browserbase.mdx b/info/kernel-vs-browserbase.mdx new file mode 100644 index 0000000..b720cf1 --- /dev/null +++ b/info/kernel-vs-browserbase.mdx @@ -0,0 +1,472 @@ +--- +title: "Kernel vs Browserbase" +sidebarTitle: "vs Browserbase" +description: "Compare Kernel and Browserbase for browser automation infrastructure. Detailed analysis of CDP support, network interception, Vercel integration, pricing, and unique features." +--- + +Both Kernel and Browserbase provide cloud browser infrastructure for web automation. This guide compares features, pricing, and use cases to help you decide. + +## Quick Comparison + +| Feature | Kernel | Browserbase | +|---------|--------|-------------| +| **CDP WebSocket** | ✓ Full support | ✓ Full support | +| **Network Interception** | ✓ Full `page.route()` | ✓ Full | +| **Session Persistence** | ✓ Hours/days with standby | ✓ Hours (context persistence) | +| **Live View** | ✓ VNC in browser | ✗ No | +| **Video Replays** | ✓ Full MP4 recordings | ✓ Screenshots + debug logs | +| **Vercel Integration** | ✓ Native with QA checks | ✗ Manual setup | +| **QA Deployment Checks** | ✓ Automated web agents | ✗ No | +| **Stealth Mode** | ✓ With CAPTCHA solver | ✓ Via configuration | +| **Proxies** | ✓ 4 types + custom | ✓ Custom only | +| **File I/O** | ✓ Read/write during session | ✓ Limited | +| **App Platform** | ✓ Deploy & invoke | ✗ No | +| **Pricing Model** | Per-minute active | Per-session + duration | +| **Cold Start** | <1s (pre-warmed) | ~1-2s | +| **Open Source** | ✓ Full platform | ✗ No | + +## Detailed Feature Comparison + +### Browser Automation + +Both support Playwright and Puppeteer over CDP with identical APIs. + +**Kernel:** +- Pre-warmed browser pool (<1s cold start) +- Headless and headful modes +- Supports all CDP-based frameworks + +**Browserbase:** +- Fast cold starts (~1-2s) +- Headless only +- Supports Playwright, Puppeteer, Selenium + +**Winner:** Tie for basic automation. Both are fast and reliable. + +### Network Interception + +Both support full network interception via CDP. + +**Kernel:** +```typescript +await page.route('**/*', route => { + if (route.request().resourceType() === 'image') { + return route.abort(); + } + return route.continue(); +}); +``` + +**Browserbase:** +```typescript +// Identical API +await page.route('**/*', route => { + if (route.request().resourceType() === 'image') { + return route.abort(); + } + return route.continue(); +}); +``` + +**Winner:** Tie. Both support the full Playwright network API. + +### Session Persistence + +**Kernel:** +- Persist sessions for hours or days +- Standby mode: Zero cost when idle, instant wake +- Profiles: Save/load cookies globally + +```typescript +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: 'my-session' +}); +// Goes to standby after 1min idle (free) +// Wakes in <1s on next request +``` + +**Browserbase:** +- Context persistence (save cookies, storage between sessions) +- Sessions can last hours +- No standby mode (session must be explicitly kept alive) + +**Winner:** Kernel. Standby mode provides zero-cost persistence. + +### Live View & Debugging + +**Kernel:** +- Built-in live view via VNC over WebRTC +- Watch browser in real-time from any browser +- Human-in-the-loop workflows +- Interactive debugging + +```typescript +const kb = await kernel.browsers.create({ headless: false }); +console.log('Live view:', kb.browser_live_view_url); +// Open URL in browser, watch automation live +``` + +**Browserbase:** +- No live view +- Debug via screenshots and logs + +**Winner:** Kernel. Live view is a significant advantage for debugging and HITL workflows. + +### Video Replays + +**Kernel:** +- Full MP4 video recordings +- Capture entire session or segments +- Programmatic start/stop +- Download or stream + +**Browserbase:** +- Screenshots at intervals +- Debug logs and traces +- Not full video + +**Winner:** Kernel. Full video is better for debugging complex issues. + +### Vercel Integration + +**Kernel:** +- Native Vercel Marketplace integration +- One-click install +- Auto-provision API keys +- QA deployment checks (run tests on every preview/prod deploy) +- Managed via Vercel dashboard + +**Browserbase:** +- Manual environment variable setup +- No marketplace integration +- No deployment checks + +**Winner:** Kernel. Significantly better Vercel experience. + +### File I/O + +**Kernel:** +- Read/write files during session via API +- List directory contents +- Download files mid-automation +- Upload files to browser + +```typescript +const files = await kernel.browsers.files.list(sessionId, '/downloads'); +const content = await kernel.browsers.files.read(sessionId, files[0].path); +``` + +**Browserbase:** +- File access limited +- Downloads available after session (via API) + +**Winner:** Kernel. More flexible for file operations. + +### App Platform + +**Kernel:** +- Deploy full applications (not just scripts) +- Invoke via API, webhooks, cron +- No timeout limits +- Environment variables, secrets +- Streaming logs + +```bash +kernel deploy index.ts +kernel invoke app-name action-name --payload '{"url": "..."}' +``` + +**Browserbase:** +- Browser infrastructure only +- No app hosting + +**Winner:** Kernel. Unique capability. + +### Stealth & Anti-Detection + +**Kernel:** +- Built-in stealth mode +- Automatic reCAPTCHA solver +- 4 proxy types: mobile, residential, ISP, datacenter +- Custom proxy support + +**Browserbase:** +- Stealth configuration available +- Custom proxy support +- CAPTCHA solving via third-party + +**Winner:** Kernel. Built-in CAPTCHA solver is a time-saver. + +### Pricing + +**Kernel:** +- Per-minute of active browser time +- Headless: ~$0.05/min +- Headful: ~$0.10/min +- Standby: Free +- No session fees + +Example: 1,000 scrapes @ 3s each = 50 minutes = $2.50 + +**Browserbase:** +- Per-session + duration +- Pricing based on usage tiers +- Context persistence may incur additional charges + +Example: Varies by plan, generally $0.01-0.05 per session + duration + +**Winner:** Depends on use case. Kernel better for many short tasks. Browserbase competitive for longer sessions. + +### Open Source + +**Kernel:** +- Fully open source platform +- Self-hosting guide available +- Community contributions + +**Browserbase:** +- Proprietary (closed source) +- Client SDKs open source + +**Winner:** Kernel. Full transparency. + +## Use Case Recommendations + +### Choose Kernel if you: + +- Need Vercel integration with deployment QA checks +- Want human-in-the-loop workflows (live view) +- Need full video replays for debugging +- Want to deploy full applications (not just scripts) +- Need session persistence with zero idle cost +- Prefer open-source platforms +- Need built-in CAPTCHA solving +- Want to self-host in the future + +### Choose Browserbase if you: + +- Already invested in Browserbase infrastructure +- Prefer a managed-only solution +- Need proven reliability at scale +- Don't need live view or full replays +- Are comfortable with proprietary solutions + +## Migration from Browserbase to Kernel + +Minimal code changes: + +```typescript +// Before (Browserbase) +const browser = await chromium.connectOverCDP({ + wsEndpoint: `wss://connect.browserbase.com?apiKey=${BROWSERBASE_KEY}` +}); + +// After (Kernel) +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create({ headless: true }); +const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url +}); +``` + +Everything else remains the same. + +## Real-World Comparison + +### Scenario 1: Vercel App with Preview QA + +**Task:** Run E2E tests on every preview deployment. + +**With Kernel:** +- Install from Vercel Marketplace +- Configure QA checks in dashboard +- Tests run automatically on every deploy +- Video replays available for failures + +**With Browserbase:** +- Manual GitHub Actions setup +- Configure environment variables +- Trigger tests via workflow +- Debug with screenshots only + +**Winner:** Kernel. Automated QA checks save setup time. + +### Scenario 2: Long-Running Scraping Job + +**Task:** Scrape 10,000 products over 2 hours. + +**With Kernel:** +- Use persistent session with standby +- Session stays alive, goes idle between batches +- Pay only for active scraping time +- Can watch progress via live view + +**With Browserbase:** +- Keep session alive for duration +- Pay for full session time +- No live view + +**Winner:** Depends. Kernel saves on idle time; Browserbase may be simpler for always-active scraping. + +### Scenario 3: Debugging Flaky Test + +**Task:** Figure out why a test fails intermittently. + +**With Kernel:** +- Enable video replays +- Run test repeatedly +- Watch full video of failure +- See exact moment and state + +**With Browserbase:** +- Enable debug mode +- Review screenshots and logs +- Infer failure from snapshots + +**Winner:** Kernel. Video is more informative than screenshots. + +## Feature Parity & Gaps + +### What Kernel has that Browserbase doesn't: + +- Native Vercel integration with QA checks +- Live view (VNC) +- Full video replays (not just screenshots) +- App deployment platform +- Built-in CAPTCHA solver +- Standby mode (zero idle cost) +- Open source + +### What Browserbase has that Kernel doesn't: + +- Longer market presence (more battle-tested at scale) +- More enterprise customers (social proof) + +### What both have: + +- Full CDP support +- Network interception +- Session persistence +- Stealth mode +- Proxy support +- Fast cold starts +- Global regions + +## Technical Deep Dive + +### Connection Pattern + +Both use identical connection patterns: + +```typescript +// Both support this +const browser = await chromium.connectOverCDP({ + wsEndpoint: cdpWebSocketUrl +}); + +const page = browser.contexts()[0].pages()[0]; +await page.goto('https://example.com'); +``` + +### Session Management + +**Kernel:** +```typescript +// Create persistent session +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: 'user-123' +}); + +// Reuse later +const existing = browsers.find(b => b.persistent_id === 'user-123'); +``` + +**Browserbase:** +```typescript +// Create session with context +const session = await browserbase.createSession({ + projectId: 'proj-123', + persistContext: true +}); + +// Reuse context +const session2 = await browserbase.createSession({ + projectId: 'proj-123', + contextId: session.contextId +}); +``` + +Similar concepts, different APIs. + +## FAQ + +### Can Kernel replace Browserbase in my stack? + +Yes. Both use CDP, so Playwright/Puppeteer code is identical. Main changes are initialization and session management. + +### Which is faster? + +Both have fast cold starts (<2s). Kernel is slightly faster with pre-warmed pools (<1s). + +### Which is more reliable? + +Both are production-grade. Browserbase has longer track record at scale. Kernel is newer but fully open source. + +### Can I use both? + +Yes. Both connect via CDP WebSocket. Use Kernel for Vercel deployments and Browserbase for other workloads if desired. + +### Does Browserbase have live view? + +No. Browserbase doesn't offer live view. Use screenshots and logs for debugging. + +### Does Kernel support Browserbase's SDK? + +No. Kernel has its own SDK (`@onkernel/sdk`). But since both use CDP, the Playwright/Puppeteer code is identical. + +## Pricing Comparison (Real Example) + +**Use case:** Screenshot 5,000 product pages monthly. + +### Kernel +``` +5,000 pages × 2 seconds = 167 minutes +167 minutes × $0.05/min (headless) = $8.35/month +``` + +### Browserbase +``` +5,000 sessions × $0.02/session (estimated) = $100/month +``` + +**Winner for this use case:** Kernel (significantly cheaper for short tasks). + +**Use case:** Continuous monitoring (24/7 browser). + +### Kernel +``` +1 browser × 43,200 minutes × $0.05/min = $2,160/month +(Or use standby mode for idle periods) +``` + +### Browserbase +``` +1 session × flat monthly rate = $200-500/month (varies) +``` + +**Winner for this use case:** Browserbase (better for always-on scenarios). + +## Related Resources + +- [Kernel Capabilities](/info/capabilities) +- [Kernel vs Browserless](/info/kernel-vs-browserless) +- [Kernel vs Self-Hosting](/info/kernel-vs-self-hosting) +- [Vercel Integration](/integrations/vercel) +- [Create a Browser](/browsers/create-a-browser) + +## Support + +Questions about choosing or migrating? Join our [Discord](https://discord.gg/FBrveQRcud) or email support@onkernel.com. + diff --git a/info/kernel-vs-browserless.mdx b/info/kernel-vs-browserless.mdx new file mode 100644 index 0000000..00b1c8d --- /dev/null +++ b/info/kernel-vs-browserless.mdx @@ -0,0 +1,330 @@ +--- +title: "Kernel vs Browserless" +sidebarTitle: "vs Browserless" +description: "Compare Kernel and Browserless for headless browser automation. Feature-by-feature analysis covering CDP support, network interception, persistence, pricing, and Vercel integration." +--- + +Both Kernel and Browserless provide cloud-hosted browsers for automation. This guide compares features, use cases, and helps you choose. + +## Quick Comparison + +| Feature | Kernel | Browserless | +|---------|--------|-------------| +| **CDP WebSocket** | ✓ Full support | ✓ Full support | +| **Network Interception** | ✓ Full `page.route()` | ✓ Full | +| **Session Persistence** | ✓ Hours/days with standby | ✓ Limited (session-based) | +| **Live View** | ✓ Human-in-the-loop | ✗ No | +| **Video Replays** | ✓ MP4 recordings | ✗ No | +| **Vercel Integration** | ✓ Native (marketplace) | ✗ Manual setup | +| **QA Deployment Checks** | ✓ Automated | ✗ No | +| **Stealth Mode** | ✓ With CAPTCHA solver | ✓ With add-ons | +| **Proxies** | ✓ 4 types built-in | ✓ Via configuration | +| **File I/O** | ✓ Read/write during session | ✗ Export at end only | +| **Pricing Model** | Per-minute active | Per-session + duration | +| **Cold Start** | <1s (pre-warmed pool) | ~2-5s | +| **Open Source** | ✓ Full platform | ✗ Client libraries only | + +## Detailed Feature Comparison + +### Browser Automation + +Both support Playwright and Puppeteer over CDP. Identical functionality for page navigation, selectors, and basic automation. + +**Kernel:** +- Supports Playwright, Puppeteer, Selenium (via CDP), Stagehand, Browser Use, Computer Use APIs +- Pre-warmed browser pool (<1s cold start) +- Headless and headful modes + +**Browserless:** +- Supports Playwright, Puppeteer, Selenium +- On-demand browser launch (~2-5s) +- Headless only (headful deprecated) + +**Winner:** Tie for basic automation. Kernel has faster cold starts. + +### Network Interception + +Both support full network interception via CDP. + +**Kernel:** +```typescript +await page.route('**/*', route => { + if (route.request().resourceType() === 'image') { + return route.abort(); + } + return route.continue(); +}); +``` + +**Browserless:** +```typescript +await page.route('**/*', route => { + if (route.request().resourceType() === 'image') { + return route.abort(); + } + return route.continue(); +}); +``` + +Identical API. No difference. + +**Winner:** Tie. + +### Session Persistence + +**Kernel:** +- Persist sessions for hours or days +- Standby mode: Zero cost when idle, instant wake +- Profiles: Save/load cookies, auth state across sessions + +```typescript +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: 'my-session' +}); +// Session stays alive, goes to standby after 1min idle +// Reuse in next request for instant auth +``` + +**Browserless:** +- Sessions tied to active connection +- Can keep alive with keep-alive tokens +- No standby mode (always consuming resources) + +**Winner:** Kernel. More flexible persistence with zero idle cost. + +### Live View & Debugging + +**Kernel:** +- Built-in live view (VNC over WebRTC) +- Watch browser in real-time from web browser +- Useful for human-in-the-loop workflows, debugging + +**Browserless:** +- No live view (headful mode deprecated in v2) +- Debug via logs only + +**Winner:** Kernel. + +### Video Replays + +**Kernel:** +- Full MP4 video recordings +- Start/stop programmatically +- Useful for debugging, compliance, auditing + +**Browserless:** +- No replay feature + +**Winner:** Kernel. + +### Vercel Integration + +**Kernel:** +- Native integration via Vercel Marketplace +- One-click install +- Auto-provision API keys +- QA deployment checks on every preview/production deploy +- Configuration management via Vercel dashboard + +**Browserless:** +- Manual setup (environment variables) +- No deployment checks +- No marketplace integration + +**Winner:** Kernel. Significantly easier for Vercel users. + +### File I/O + +**Kernel:** +- Read/write files during session via API +- Access downloads mid-session +- Upload files to browser filesystem + +```typescript +const files = await kernel.browsers.files.list(sessionId, '/downloads'); +const content = await kernel.browsers.files.read(sessionId, file.path); +``` + +**Browserless:** +- Files only available after session ends +- Must wait for session to complete + +**Winner:** Kernel. More flexible for file operations. + +### Stealth & Anti-Detection + +**Kernel:** +- Built-in stealth mode +- Automatic reCAPTCHA solver +- 4 proxy types (mobile, residential, ISP, datacenter) +- Custom proxy support + +**Browserless:** +- Stealth plugins available +- CAPTCHA solving via integrations (2Captcha, etc.) +- Custom proxy support + +**Winner:** Kernel. Built-in CAPTCHA solver saves setup time. + +### Pricing + +**Kernel:** +- Per-minute of active browser time +- Headless: ~$0.05/min +- Headful: ~$0.10/min +- Standby mode: Free +- No session fees + +Example: 1,000 scrapes @ 3s each = 50 minutes = $2.50 + +**Browserless:** +- Per-session + duration +- Pricing tiers based on concurrency +- Session-based charging (even if browser idle) + +Example: 1,000 sessions = varies by plan (~$0.01-0.05/session + duration) + +**Winner:** Depends on use case. Kernel better for short, frequent tasks. Browserless better for long-running sessions if you optimize for concurrency. + +### Open Source + +**Kernel:** +- Fully open source (browser infrastructure, API, dashboard) +- Self-hosting guide available +- Community contributions welcome + +**Browserless:** +- Client libraries open source +- Server/infrastructure proprietary + +**Winner:** Kernel. Full platform transparency. + +## Use Case Recommendations + +### Choose Kernel if you: + +- Need Vercel integration with QA deployment checks +- Want human-in-the-loop (live view) +- Need video replays for debugging/compliance +- Want session persistence with zero idle cost +- Need file I/O during session +- Prefer open-source platforms +- Run many short automation tasks + +### Choose Browserless if you: + +- Have existing Browserless integration (migration cost) +- Need maximum concurrency (1000+ simultaneous browsers) +- Prefer session-based pricing for long-running tasks +- Already familiar with their API + +## Migration from Browserless to Kernel + +Minimal code changes required: + +```typescript +// Before (Browserless) +const browser = await puppeteer.connect({ + browserWSEndpoint: `wss://chrome.browserless.io?token=${TOKEN}` +}); + +// After (Kernel) +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create({ headless: true }); +const browser = await puppeteer.connect({ + browserWSEndpoint: kb.cdp_ws_url +}); +``` + +Everything else (selectors, actions, tests) remains identical. + +## Real-World Comparison + +### Scenario 1: Generate OG Images on Vercel + +**Task:** Generate social preview images for blog posts. + +**With Kernel:** +- Install from Vercel Marketplace (1 click) +- API key auto-provisioned +- Deploy Next.js API route +- <1s cold start, 2s total per image + +**With Browserless:** +- Manually add API key to Vercel +- Configure connection +- Deploy Next.js API route +- ~3-5s per image (slower cold start) + +**Winner:** Kernel. Easier setup, faster execution. + +### Scenario 2: Monitor Competitor Prices (Cron Job) + +**Task:** Scrape 100 competitor prices every hour. + +**With Kernel:** +- Use persistent session with profile +- First run: logs in, saves cookies +- Subsequent runs: instant auth, scrape +- Cost: ~5 min/day = $0.50/day = $15/month + +**With Browserless:** +- Log in on each run (no long-term persistence) +- Must re-auth every time +- Cost: ~100 sessions/day = varies by plan + +**Winner:** Kernel. Persistence saves time and cost. + +### Scenario 3: E2E Testing (CI/CD) + +**Task:** Run Playwright tests on every PR. + +**With Kernel:** +- Works with GitHub Actions, GitLab CI, etc. +- Parallel test execution +- Video replays for failed tests +- Cost: Pay only for test duration + +**With Browserless:** +- Works with CI/CD +- Parallel execution +- No video replays (screenshots only via Playwright) +- Cost: Per test session + +**Winner:** Kernel. Video replays useful for debugging flaky tests. + +## FAQ + +### Can Kernel replace Browserless in my existing stack? + +Yes. Kernel supports the same protocols (CDP) and frameworks (Playwright, Puppeteer). Migration typically takes <1 hour. + +### Does Kernel support the Browserless REST API? + +No. Kernel uses a different API design focused on SDK-first usage. However, CDP connection works identically, so your Playwright/Puppeteer code doesn't change. + +### Which is faster? + +Kernel has faster cold starts (<1s vs 2-5s) due to pre-warmed browser pools. Both have similar performance for page loads and automation once browser is running. + +### Which is more reliable? + +Both are production-grade. Kernel's standby mode reduces cold start issues for persistent sessions. Browserless has longer market presence. + +### Can I use both? + +Yes. Both connect via CDP WebSocket. You can use Kernel for Vercel deployments and Browserless for other workloads if desired. + +## Related Resources + +- [Kernel Capabilities](/info/capabilities) +- [Kernel vs Browserbase](/info/kernel-vs-browserbase) +- [Kernel vs Self-Hosting](/info/kernel-vs-self-hosting) +- [Vercel Integration](/integrations/vercel) +- [Create a Browser](/browsers/create-a-browser) + +## Support + +Need help choosing or migrating? Join our [Discord](https://discord.gg/FBrveQRcud) or email support@onkernel.com. + diff --git a/info/kernel-vs-self-hosting.mdx b/info/kernel-vs-self-hosting.mdx new file mode 100644 index 0000000..449b649 --- /dev/null +++ b/info/kernel-vs-self-hosting.mdx @@ -0,0 +1,472 @@ +--- +title: "Kernel vs Self-Hosting Chrome" +sidebarTitle: "vs Self-Hosting" +description: "Compare Kernel's managed browser infrastructure to self-hosting Chrome on Docker, Cloud Run, Fly.io, or Railway. TCO analysis, maintenance burden, and decision framework." +--- + +Should you use Kernel's managed browsers or self-host Chrome? This guide compares total cost, operational overhead, and helps you decide. + +## Quick Comparison + +| Aspect | Kernel (Managed) | Self-Hosted Chrome | +|--------|------------------|-------------------| +| **Setup Time** | Minutes (sign up, get API key) | Days (Docker, orchestration, monitoring) | +| **Cold Start** | <1s (pre-warmed pool) | 5-30s (container pull + Chrome launch) | +| **Scaling** | Automatic (1 to 1000+ browsers) | Manual (configure autoscaling, limits) | +| **Maintenance** | Zero (Chrome updates automatic) | Ongoing (security patches, Chrome updates) | +| **Infrastructure** | Managed globally | You provision & manage | +| **Cost (light use)** | $5-50/month typical | $50-200/month minimum (always-on) | +| **Cost (heavy use)** | $100-500/month | $500-2000/month (servers + ops time) | +| **Features** | Live view, replays, persistence, standby | What you build | +| **Control** | API-based configuration | Full control (custom flags, builds) | +| **Debugging** | Built-in (live view, replays, logs) | DIY (logs, VNC setup optional) | +| **Compliance** | Kernel's infrastructure | Your infrastructure (full control) | + +## Detailed Comparison + +### Setup & Time to First Browser + +**Kernel:** +```bash +# 2 minutes to first automation +npm install @onkernel/sdk +export KERNEL_API_KEY=xxx +node script.js +``` + +**Self-Hosted:** +```bash +# 1-2 days to production-ready setup +# 1. Create Dockerfile with Chrome +# 2. Set up container registry +# 3. Configure orchestration (ECS/Cloud Run/Fly) +# 4. Set up monitoring, logging, alerting +# 5. Configure autoscaling +# 6. Test cold starts, memory limits +# 7. Set up health checks +# 8. Deploy +``` + +**Winner:** Kernel. Faster by orders of magnitude. + +### Cost Analysis (Real Numbers) + +#### Light Use: 100 browser-hours/month + +**Kernel:** +``` +100 hours = 6,000 minutes +6,000 minutes × $0.05/min (headless) = $300/month +``` + +**Self-Hosted on Cloud Run:** +``` +Cloud Run instance (1 vCPU, 2GB RAM): +- Always-on: 730 hours × $0.048/hr = $35/month +- CPU time: ~100 hours × $0.024/vCPU-hr = $2.40/month +- Total infrastructure: $37.40/month + +Operational overhead: +- Setup time: 16 hours @ $100/hr = $1,600 (one-time) +- Monthly maintenance: 2 hours @ $100/hr = $200/month +- Total monthly: $237.40/month +``` + +**Winner:** Cloud Run (cheaper), but Kernel if factoring in ops time. + +#### Medium Use: 500 browser-hours/month + +**Kernel:** +``` +500 hours = 30,000 minutes +30,000 minutes × $0.05/min = $1,500/month +``` + +**Self-Hosted on ECS Fargate:** +``` +ECS Fargate (2 vCPU, 4GB RAM): +- Compute: 730 hours × $0.12/hr = $87.60/month +- Scaling overhead: 3 instances for peaks = $262.80/month + +Operational overhead: +- Setup time: 20 hours @ $100/hr = $2,000 (one-time) +- Monthly maintenance: 4 hours @ $100/hr = $400/month +- Total monthly: $662.80/month +``` + +**Winner:** Self-hosted is cheaper at this scale. + +#### Heavy Use: 5,000 browser-hours/month + +**Kernel:** +``` +5,000 hours = 300,000 minutes +300,000 minutes × $0.05/min = $15,000/month +``` + +**Self-Hosted on Kubernetes (GKE/EKS):** +``` +Kubernetes cluster (10 nodes, autoscaling): +- Compute: ~$1,200/month (optimized) +- Load balancer: $20/month +- Monitoring (Datadog/New Relic): $200/month +- Container registry: $50/month +- Total infrastructure: $1,470/month + +Operational overhead: +- Setup time: 40 hours @ $100/hr = $4,000 (one-time) +- Monthly maintenance: 10 hours @ $100/hr = $1,000/month +- Incident response: ~5 hours @ $100/hr = $500/month +- Total monthly: $2,970/month +``` + +**Winner:** Self-hosted is 5× cheaper, but requires dedicated DevOps. + +### Maintenance & Operational Burden + +**Kernel:** +- **Chrome updates:** Automatic, zero downtime +- **Security patches:** Automatic +- **Scaling:** Automatic (API handles 1 to 1000+ browsers) +- **Monitoring:** Built-in dashboard +- **Debugging:** Live view, replays, logs +- **Your time:** 0 hours/month + +**Self-Hosted:** +- **Chrome updates:** Manual (Dockerfile rebuild + deploy) +- **Security patches:** Monitor CVEs, patch regularly +- **Scaling:** Configure autoscaling, test under load +- **Monitoring:** Set up Prometheus/Datadog/CloudWatch +- **Debugging:** Set up logging, optionally VNC +- **Your time:** 2-10 hours/month minimum + +**Winner:** Kernel. Zero ops burden. + +### Features & Capabilities + +**Kernel Includes:** +- Pre-warmed browser pool (<1s cold start) +- Live view (watch browser in real-time) +- Video replays (full MP4 recordings) +- Session persistence with standby (zero idle cost) +- Profiles (save/load cookies, auth) +- File I/O API +- Stealth mode with CAPTCHA solver +- 4 proxy types (mobile, residential, ISP, datacenter) +- Network interception +- Vercel native integration +- QA deployment checks + +**Self-Hosted Includes:** +- Whatever you build +- Chrome with custom flags (full control) +- Your own infrastructure (compliance, data locality) + +**Winner:** Kernel for features. Self-hosted for control. + +### Cold Start Performance + +**Kernel:** +- Pre-warmed browser pool +- Connect time: <1s +- Total to first page load: ~2s + +**Self-Hosted:** +- Container pull: 5-10s (first time) +- Chrome launch: 2-5s +- Total to first page load: 7-15s (cold), 2-5s (warm) + +**Winner:** Kernel. Significantly faster cold starts. + +### Scaling & Concurrency + +**Kernel:** +```typescript +// Launch 100 browsers concurrently +const browsers = await Promise.all( + Array(100).fill(0).map(() => + kernel.browsers.create({ headless: true }) + ) +); +// Just works, scales automatically +``` + +**Self-Hosted:** +- Configure autoscaling policies +- Set min/max instances +- Test scaling behavior +- Monitor resource utilization +- Handle scale-down gracefully + +**Winner:** Kernel. Automatic scaling. + +### Control & Customization + +**Kernel:** +- Configure via API (headless, proxies, stealth) +- Standard Chrome (latest stable) +- Can't modify Chrome flags or build + +**Self-Hosted:** +- Full control over Chrome version +- Custom Chrome flags +- Custom extensions +- Custom fonts, locales +- Run specific Chrome builds + +**Winner:** Self-hosted. Full customization. + +### Compliance & Data Sovereignty + +**Kernel:** +- Data processed in Kernel's infrastructure +- SOC 2 Type II in progress (Q2 2025) +- Can self-host (open source) + +**Self-Hosted:** +- Data stays in your infrastructure +- You control all compliance aspects +- Data locality guarantees + +**Winner:** Self-hosted for strict compliance requirements. + +## Decision Framework + +### Choose Kernel if: + +- You're getting started with browser automation +- Ops time is expensive or unavailable +- You need fast cold starts (<1s) +- You want zero maintenance burden +- You need advanced features (live view, replays, standby) +- Your usage is variable (spiky traffic) +- You use Vercel and want native integration +- You're automating <2,000 browser-hours/month + +### Choose Self-Hosting if: + +- You have dedicated DevOps team +- Your usage is consistent and high (5,000+ hours/month) +- You need custom Chrome builds or flags +- You have strict data sovereignty requirements +- You want full control over infrastructure +- You're already managing Kubernetes/container orchestration +- You can amortize setup costs over long term + +### Consider Hybrid: + +- Use Kernel for development and testing +- Self-host for production (after validating at scale) +- Use Kernel for variable workloads, self-host for baseline + +## Self-Hosting Platforms Compared + +If you decide to self-host, platform matters: + +### Docker on Cloud Run (Google Cloud) + +**Pros:** +- Serverless (pay per use) +- Auto-scaling +- Relatively simple + +**Cons:** +- Cold starts (5-15s) +- Memory limits (8GB max) +- Not ideal for long-running browsers + +**Cost:** ~$50/month minimum + +### Docker on Fly.io + +**Pros:** +- Global edge network +- Fast cold starts (3-5s) +- Simple deployment + +**Cons:** +- Smaller scale than GCP/AWS +- Less mature than Cloud Run + +**Cost:** ~$30/month minimum + +### ECS Fargate (AWS) + +**Pros:** +- No server management +- Integrates with AWS ecosystem +- Good for burst workloads + +**Cons:** +- More expensive than EC2 for always-on +- Cold starts (10-15s) + +**Cost:** ~$100/month minimum + +### Kubernetes (EKS, GKE, AKS) + +**Pros:** +- Best for large scale +- Full control +- Advanced orchestration + +**Cons:** +- Complex setup +- Requires Kubernetes expertise +- High minimum cost + +**Cost:** ~$200/month minimum (cluster + nodes) + +### Railway / Render + +**Pros:** +- Simple deployment +- Good developer experience +- Affordable for small scale + +**Cons:** +- Limited enterprise features +- Smaller scale than AWS/GCP + +**Cost:** ~$20/month minimum + +## Migration Patterns + +### Start with Kernel, Migrate Later + +```typescript +// Abstract browser creation +async function getBrowser() { + if (process.env.USE_KERNEL) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create({ headless: true }); + return await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); + } else { + return await chromium.launch({ headless: true }); + } +} + +// Use same code in both environments +const browser = await getBrowser(); +``` + +Deploy on Kernel initially. Once usage grows, flip to self-hosted. + +### Hybrid: Kernel for Peaks, Self-Host for Baseline + +```typescript +async function getBrowser() { + const load = await getCurrentLoad(); + + if (load > SELF_HOSTED_CAPACITY) { + // Use Kernel for overflow + return await connectToKernel(); + } else { + // Use self-hosted for baseline + return await connectToSelfHosted(); + } +} +``` + +Cost-optimize by using self-hosted for predictable load, Kernel for spikes. + +## Real-World Scenarios + +### Scenario 1: Startup Building MVP + +**Need:** Scrape competitor data, generate OG images. + +**Volume:** 100 browser-hours/month. + +**Recommendation:** **Use Kernel.** Focus on product, not infrastructure. Total cost ~$300/month including ops time. Self-hosting would cost ~$250/month plus 10+ hours setup. + +### Scenario 2: Mid-Size SaaS (E2E Testing) + +**Need:** Run Playwright tests on every PR (500 PRs/month). + +**Volume:** 300 browser-hours/month. + +**Recommendation:** **Use Kernel.** Tests run faster (cold start <1s). Zero ops burden. Cost ~$900/month. Self-hosting would be ~$500/month but adds maintenance burden. + +### Scenario 3: Enterprise Web Scraping + +**Need:** Scrape 100,000 pages/day continuously. + +**Volume:** 10,000 browser-hours/month. + +**Recommendation:** **Self-host on Kubernetes.** At this scale, infrastructure cost becomes dominant. Self-hosting costs ~$3,000/month vs Kernel's ~$30,000/month. ROI on DevOps investment is clear. + +### Scenario 4: Regulatory/Compliance + +**Need:** Financial data scraping with PII. + +**Requirements:** Data cannot leave your infrastructure. + +**Recommendation:** **Self-host** (or use Kernel open source to self-host the Kernel platform). Compliance trumps cost. + +## TCO Calculator (5-Year) + +### Kernel + +``` +Year 1: $3,600 (300 hours/month) +Year 2: $7,200 (600 hours/month, growing) +Year 3: $10,800 (900 hours/month) +Year 4: $14,400 (1,200 hours/month) +Year 5: $18,000 (1,500 hours/month) + +Total 5-year: $54,000 +Total ops time: 0 hours +``` + +### Self-Hosted + +``` +Setup: $4,000 (40 hours) +Year 1: $6,000 (infra) + $12,000 (ops) = $18,000 +Year 2: $8,000 + $12,000 = $20,000 +Year 3: $10,000 + $15,000 = $25,000 +Year 4: $12,000 + $15,000 = $27,000 +Year 5: $14,000 + $18,000 = $32,000 + +Total 5-year: $126,000 +Total ops time: ~600 hours +``` + +**Winner:** Kernel at moderate scale. Self-hosted wins at very high scale (10,000+ hours/month) or with existing DevOps team. + +## FAQ + +### Can I self-host Kernel itself? + +**Yes.** Kernel is fully open source. See [github.com/onkernel/kernel](https://github.com/onkernel/kernel) for self-hosting guide. You get Kernel's features on your infrastructure. + +### What if I outgrow Kernel? + +You can always migrate to self-hosted later. Kernel uses standard CDP, so your Playwright/Puppeteer code is portable. + +### Is self-hosting more reliable? + +Depends on your ops maturity. Kernel's infrastructure is battle-tested. Self-hosted can be equally reliable if you invest in monitoring, redundancy, and on-call. + +### Can I start with Kernel and add self-hosted later? + +Yes. Run both in parallel (Kernel for dev/test, self-hosted for prod) or use Kernel for overflow. + +### What about vendor lock-in? + +Kernel uses standard protocols (CDP, WebSocket). Your automation code is portable. Switching cost is minimal (mostly initialization code). + +## Related Resources + +- [Kernel Capabilities](/info/capabilities) +- [Kernel vs Browserless](/info/kernel-vs-browserless) +- [Kernel vs Browserbase](/info/kernel-vs-browserbase) +- [Create a Browser](/browsers/create-a-browser) +- [Vercel Integration](/integrations/vercel) +- [GitHub (Self-Hosting)](https://github.com/onkernel/kernel) + +## Support + +Need help deciding? Join our [Discord](https://discord.gg/FBrveQRcud) or email support@onkernel.com for a personalized analysis. + diff --git a/integrations/vercel.mdx b/integrations/vercel.mdx index de8d114..3fe00bc 100644 --- a/integrations/vercel.mdx +++ b/integrations/vercel.mdx @@ -1,4 +1,648 @@ --- -title: "Vercel" -url: "https://github.com/onkernel/vercel-template" ---- \ No newline at end of file +title: "Vercel Integration" +description: "Run Playwright and Puppeteer on Vercel with Kernel's native integration. One-click setup, automatic QA deployment checks, and zero infrastructure management." +--- + +Kernel provides a native Vercel integration available in the [Vercel Marketplace](https://vercel.com/integrations/kernel). Get automatic API key provisioning, QA deployment checks on every preview, and seamless browser automation for your Next.js apps. + +## Why You Need This + +Vercel's serverless environment cannot run bundled Chromium binaries due to: + +- **Filesystem constraints:** Functions are read-only and ephemeral +- **Size limits:** 50MB max; Chromium is ~300MB +- **Memory limits:** 1GB (Hobby), 3GB (Pro); Chromium needs 1-2GB +- **Timeout limits:** 10s (Hobby), 60s (Pro); cold start + page load often exceeds this + +**Kernel solves this** by hosting browsers in the cloud. Your code runs on Vercel; browsers run on Kernel. Connect via WebSocket using Playwright or Puppeteer. + +## Quick Start (Manual Setup) + +If you want to get started without installing the marketplace integration: + +### 1. Install Dependencies + +```bash +# Use -core versions (no browser binaries) +npm install playwright-core @onkernel/sdk +# or +npm install puppeteer-core @onkernel/sdk +``` + +### 2. Get API Key + +1. Sign up at [dashboard.onkernel.com](https://dashboard.onkernel.com/sign-up) +2. Go to Settings → API Keys +3. Create new API key + +### 3. Add to Vercel + +```bash +vercel env add KERNEL_API_KEY +# Paste your key +# Select: Production, Preview, Development +``` + +### 4. Create API Route + + +```typescript Next.js App Router (app/api/screenshot/route.ts) +import { NextRequest } from 'next/server'; +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +export async function GET(req: NextRequest) { + const url = req.nextUrl.searchParams.get('url') || 'https://example.com'; + + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto(url); + const screenshot = await page.screenshot({ type: 'png' }); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return new Response(screenshot, { + headers: { 'Content-Type': 'image/png' } + }); +} +``` + +```typescript Next.js Pages Router (pages/api/screenshot.ts) +import type { NextApiRequest, NextApiResponse } from 'next'; +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +export default async function handler( + req: NextApiRequest, + res: NextApiResponse +) { + const url = (req.query.url as string) || 'https://example.com'; + + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto(url); + const screenshot = await page.screenshot({ type: 'png' }); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + res.setHeader('Content-Type', 'image/png'); + res.send(screenshot); +} +``` + + +### 5. Test Locally + +```bash +npm run dev +# Visit http://localhost:3000/api/screenshot?url=https://google.com +``` + +### 6. Deploy + +```bash +vercel deploy +``` + +Done. No build errors, no runtime errors. + +## Native Integration (Recommended) + +For seamless setup and automatic QA checks: + +### 1. Install from Marketplace + +Visit [vercel.com/integrations/kernel](https://vercel.com/integrations/kernel) and click **Add Integration**. + +### 2. Connect Projects + +Select which Vercel projects should have access to Kernel. + +### 3. API Key Auto-Provisioned + +Kernel automatically adds `KERNEL_API_KEY` to your selected projects' environment variables. No manual setup needed. + +### 4. QA Deployment Checks Enabled + +Every preview and production deployment automatically runs QA checks using Kernel's web agents. See [QA Deployment Checks](#qa-deployment-checks) below. + +## QA Deployment Checks + +The native integration runs automated QA tests on every deployment using Kernel's AI web agents: + +### How It Works + +1. **deployment.created**: Kernel receives webhook from Vercel +2. **Check registered**: Kernel creates a blocking deployment check +3. **Agent runs**: Web agent navigates your preview URL, tests functionality +4. **Results posted**: Pass/fail status appears in Vercel dashboard +5. **Deployment proceeds**: If passing, deployment continues; if failing, you're notified + +### What Gets Tested + +Configure checks via Vercel dashboard (Integration Settings → Kernel): + +- **Visual regression:** Screenshot comparison vs baseline +- **Broken links:** Crawl and verify all internal links load +- **Auth flows:** Test login, signup, password reset +- **Critical paths:** Custom scripts for checkout, forms, etc. +- **Accessibility:** WCAG compliance checks +- **Performance:** Lighthouse scores, load times + +### Example: Visual Regression + +```typescript +// kernel-qa/visual-regression.ts +import { App } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; + +const app = new App('visual-regression'); + +app.action('check-deployment', async (ctx, payload) => { + const { deploymentUrl, baseline Url } = payload; + + const kb = await ctx.kernel.browsers.create({ + invocation_id: ctx.invocation_id, + headless: true + }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Capture new deployment + await page.goto(deploymentUrl); + const newScreenshot = await page.screenshot({ fullPage: true }); + + // Capture baseline + await page.goto(baselineUrl); + const baselineScreenshot = await page.screenshot({ fullPage: true }); + + // Compare (use pixelmatch, looks-same, or similar) + const diffPercentage = await compareScreenshots( + newScreenshot, + baselineScreenshot + ); + + await browser.close(); + + return { + passed: diffPercentage < 0.1, // <0.1% diff = pass + diffPercentage, + message: `Visual diff: ${(diffPercentage * 100).toFixed(2)}%` + }; +}); + +export default app; +``` + +Deploy this as a Kernel App: + +```bash +cd kernel-qa +kernel deploy visual-regression.ts +``` + +Kernel invokes it automatically on each deployment. + +### Configure in Vercel Dashboard + +1. Go to your Vercel project +2. Settings → Integrations → Kernel +3. Enable checks: Visual Regression, Broken Links, etc. +4. Set baseline URLs and thresholds +5. Save + +Checks run on next deployment. + +### Manual Invocation + +You can also trigger checks manually via API: + +```typescript +// pages/api/run-qa.ts +import { Kernel } from '@onkernel/sdk'; + +export default async function handler(req, res) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + + const invocation = await kernel.invocations.create({ + app_name: 'visual-regression', + action_name: 'check-deployment', + payload: { + deploymentUrl: req.body.deploymentUrl, + baselineUrl: req.body.baselineUrl + }, + async: true + }); + + res.json({ invocationId: invocation.id }); +} +``` + +## Environment-Based Toggle + +Use local Playwright in development, Kernel in production: + +```typescript +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +const isProduction = process.env.VERCEL_ENV === 'production'; +const isPreview = process.env.VERCEL_ENV === 'preview'; +const useKernel = isProduction || isPreview || process.env.USE_KERNEL; + +async function getBrowser() { + if (useKernel) { + // Use Kernel on Vercel + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create({ headless: true }); + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + return { browser, sessionId: kb.session_id, kernel }; + } else { + // Use local Playwright in dev + const browser = await chromium.launch({ headless: true }); + return { browser, sessionId: null, kernel: null }; + } +} + +// Usage +const { browser, sessionId, kernel } = await getBrowser(); + +// ... use browser ... + +await browser.close(); +if (kernel && sessionId) { + await kernel.browsers.deleteByID(sessionId); +} +``` + +Force Kernel in local dev by setting: + +```bash +export USE_KERNEL=true +npm run dev +``` + +## Vercel Limits & How Kernel Handles Them + +| Limit | Hobby | Pro | How Kernel Helps | +|-------|-------|-----|------------------| +| **Function timeout** | 10s | 60s | Browser pre-warmed, connect in <1s | +| **Memory** | 1GB | 3GB | Browser runs remotely, function uses <100MB | +| **Deployment size** | 50MB | 50MB | No Chromium binaries to deploy | +| **Cold start** | Slow | Slow | Kernel pools browsers, instant connect | + +## Network Interception + +Block resources to speed up page loads: + +```typescript +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create({ headless: true }); +const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); +const page = browser.contexts()[0].pages()[0]; + +// Block images, fonts, stylesheets +await page.route('**/*', route => { + const type = route.request().resourceType(); + if (['image', 'font', 'stylesheet', 'media'].includes(type)) { + return route.abort(); + } + return route.continue(); +}); + +await page.goto('https://example.com'); +// Loads 50-70% faster +``` + +Full network interception works over CDP. See [Network Interception guide](/troubleshooting/network-interception). + +## File Downloads + +Download files from the browser and upload to S3, R2, etc.: + +```typescript +import { Kernel } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; +import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'; + +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create(); +const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); +const page = browser.contexts()[0].pages()[0]; + +// Trigger download +await page.goto('https://example.com/reports'); +await page.click('a[download]'); +await page.waitForTimeout(2000); // Wait for download + +// Fetch file via Kernel File I/O API +const files = await kernel.browsers.files.list(kb.session_id, '/downloads'); +const pdfFile = files.find(f => f.name.endsWith('.pdf')); + +if (pdfFile) { + const buffer = await kernel.browsers.files.read(kb.session_id, pdfFile.path); + + // Upload to S3 + const s3 = new S3Client({ region: 'us-east-1' }); + await s3.send(new PutObjectCommand({ + Bucket: 'my-bucket', + Key: pdfFile.name, + Body: buffer + })); +} + +await browser.close(); +await kernel.browsers.deleteByID(kb.session_id); +``` + +See [File I/O docs](/browsers/file-io). + +## Persistent Sessions + +Reuse browser sessions across multiple requests to preserve auth: + +```typescript +const SESSION_ID = 'vercel-app-session'; + +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + +// Try to reuse existing session +let browsers = await kernel.browsers.list(); +let kb = browsers.find(b => b.persistent_id === SESSION_ID); + +// Create if doesn't exist +if (!kb) { + kb = await kernel.browsers.create({ + persistent: true, + persistent_id: SESSION_ID, + headless: true + }); +} + +const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url +}); + +// ... use browser (cookies/session preserved) ... + +await browser.close(); +// Don't delete - keeps session for next request +``` + +See [Persistence docs](/browsers/persistence). + +## Migration from Browserless/Browserbase + +If you're switching from another hosted browser provider: + +### From Browserless + +```typescript +// Before (Browserless) +const browser = await puppeteer.connect({ + browserWSEndpoint: `wss://chrome.browserless.io?token=${BROWSERLESS_TOKEN}` +}); + +// After (Kernel) +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create({ headless: true }); +const browser = await puppeteer.connect({ + browserWSEndpoint: kb.cdp_ws_url +}); +``` + +### From Browserbase + +```typescript +// Before (Browserbase) +const browser = await chromium.connectOverCDP({ + wsEndpoint: `wss://connect.browserbase.com?apiKey=${BROWSERBASE_KEY}` +}); + +// After (Kernel) +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create({ headless: true }); +const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url +}); +``` + +Everything else (selectors, actions, assertions) remains identical. + +## Troubleshooting + +### Error: "KERNEL_API_KEY is not set" + +Add API key to Vercel environment variables: + +```bash +vercel env add KERNEL_API_KEY +``` + +Or via Vercel dashboard: Settings → Environment Variables → Add. + +### Error: "Timeout connecting to browser" + +Check: + +1. API key is valid (test with `kernel.browsers.list()`) +2. Network allows outbound WebSocket connections +3. Vercel function timeout is sufficient (increase if needed) + +### Error: "Cannot find module 'playwright-core'" + +Make sure you're using `playwright-core` (not `playwright`) in package.json: + +```json +{ + "dependencies": { + "playwright-core": "^1.47.0", + "@onkernel/sdk": "^latest" + } +} +``` + +### Slow cold starts + +Use [persistent sessions](#persistent-sessions) to reuse browsers across requests. First request creates browser (~2s), subsequent requests connect instantly (~0.1s). + +### Out of memory errors + +Use headless mode (`headless: true`) and block unnecessary resources. This reduces memory usage from 2GB to <500MB. + +## Examples + +### Generate OG Images + +```typescript +// app/api/og/route.tsx +import { ImageResponse } from 'next/og'; +import { Kernel } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; + +export async function GET(req: Request) { + const { searchParams } = new URL(req.url); + const title = searchParams.get('title') || 'Default Title'; + + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create({ headless: true }); + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.setContent(` + + +

${title}

+ + + `); + + const screenshot = await page.screenshot({ type: 'png' }); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return new Response(screenshot, { + headers: { 'Content-Type': 'image/png' } + }); +} +``` + +### Scrape Competitor Prices + +```typescript +// app/api/scrape-prices/route.ts +import { Kernel } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; + +export async function GET() { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create({ headless: true, stealth: true }); + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto('https://competitor.com/pricing'); + + const prices = await page.$$eval('.price', elements => + elements.map(el => el.textContent?.trim()) + ); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return Response.json({ prices }); +} +``` + +### Run E2E Tests on Preview + +```typescript +// tests/e2e/checkout.spec.ts +import { test, expect } from '@playwright/test'; +import { Kernel } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; + +test('checkout flow', async () => { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create({ headless: true }); + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Use VERCEL_URL for preview deployments + const baseUrl = process.env.VERCEL_URL + ? `https://${process.env.VERCEL_URL}` + : 'http://localhost:3000'; + + await page.goto(`${baseUrl}/products`); + await page.click('text=Add to Cart'); + await page.click('text=Checkout'); + await page.fill('#email', 'test@example.com'); + await page.click('text=Place Order'); + + await expect(page.locator('text=Order confirmed')).toBeVisible(); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); +}); +``` + +Run on Vercel with GitHub Actions: + +```yaml +name: E2E Tests +on: [deployment_status] + +jobs: + test: + if: github.event.deployment_status.state == 'success' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - run: npm ci + - run: npx playwright test + env: + KERNEL_API_KEY: ${{ secrets.KERNEL_API_KEY }} + VERCEL_URL: ${{ github.event.deployment_status.target_url }} +``` + +## Related Resources + +- [Playwright Vercel Error](/troubleshooting/playwright-vercel-error) +- [Headless Chrome on Serverless](/troubleshooting/headless-chrome-serverless) +- [Network Interception](/troubleshooting/network-interception) +- [Playwright Timeouts](/troubleshooting/playwright-timeouts-serverless) +- [Create a Browser](/browsers/create-a-browser) +- [File I/O](/browsers/file-io) +- [Persistence](/browsers/persistence) +- [Stealth Mode](/browsers/stealth) + +## Support + +- **Discord:** [discord.gg/FBrveQRcud](https://discord.gg/FBrveQRcud) +- **Email:** support@onkernel.com +- **GitHub:** [github.com/onkernel](https://github.com/onkernel) + +## Starter Template + +Clone our Vercel + Kernel starter: + +```bash +npx create-kernel-app my-vercel-app --template vercel +cd my-vercel-app +vercel deploy +``` + +Includes: + +- Next.js App Router +- Playwright + Kernel setup +- Example API routes (screenshot, scrape, test) +- Environment variables configured +- TypeScript, ESLint, Prettier diff --git a/recipes/auth-cookies-sessions.mdx b/recipes/auth-cookies-sessions.mdx new file mode 100644 index 0000000..84c2d17 --- /dev/null +++ b/recipes/auth-cookies-sessions.mdx @@ -0,0 +1,503 @@ +--- +title: "Auth Flows & Cookies with Sessions" +sidebarTitle: "Auth & Sessions" +description: "Handle authentication flows, preserve cookies, and reuse login state across automations with Kernel's persistent sessions and profiles." +--- + +Automate login flows, preserve authentication state, and reuse cookies across multiple browser sessions. Avoid logging in repeatedly with Kernel's persistence features. + +## What This Recipe Does + +1. Log in to a website once +2. Save cookies and session state +3. Reuse authentication in future automations +4. Avoid rate limits and CAPTCHAs from repeated logins + +## Use Cases + +- Scrape data behind login walls +- Automate SaaS workflows +- Test authenticated user flows +- Export data from logged-in accounts +- Monitor dashboards or reports +- Social media automation + +## Complete Code + + +```typescript TypeScript +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +async function loginAndSave(credentials: { + email: string; + password: string; + profileName: string; +}) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + + // Create browser with profile saving enabled + const kb = await kernel.browsers.create({ + profile_name: credentials.profileName, + profile_save_changes: true, // Save cookies/storage on close + headless: false // Use headful for first login (see live view) + }); + + console.log('Live view:', kb.browser_live_view_url); + console.log('Log in manually via live view, or script below will auto-login'); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Navigate to login page + await page.goto('https://example.com/login'); + + // Fill login form + await page.fill('input[name="email"]', credentials.email); + await page.fill('input[name="password"]', credentials.password); + await page.click('button[type="submit"]'); + + // Wait for redirect after login + await page.waitForURL('**/dashboard', { timeout: 30000 }); + + console.log('Login successful! Cookies saved to profile.'); + + await browser.close(); + // Profile automatically saved when browser closes + + return { profileName: credentials.profileName }; +} + +async function scrapeWithAuth(profileName: string) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + + // Reuse existing profile (logged-in state) + const kb = await kernel.browsers.create({ + profile_name: profileName, + headless: true // Can use headless now + }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Go directly to protected page (already logged in!) + await page.goto('https://example.com/dashboard/data'); + + // Extract data + const data = await page.$$eval('.data-row', rows => + rows.map(row => row.textContent?.trim()) + ); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { data }; +} + +// Usage +// First time: log in and save +const { profileName } = await loginAndSave({ + email: 'user@example.com', + password: 'password123', + profileName: 'example-user' +}); + +// Subsequent times: reuse auth +const result1 = await scrapeWithAuth(profileName); +const result2 = await scrapeWithAuth(profileName); // Still logged in! +``` + +```python Python +from playwright.async_api import async_playwright +from kernel import Kernel + +async def login_and_save(credentials: dict): + kernel = Kernel() + + # Create browser with profile saving + kb = kernel.browsers.create( + profile_name=credentials['profile_name'], + profile_save_changes=True, + headless=False # Use headful for first login + ) + + print(f'Live view: {kb.browser_live_view_url}') + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + # Navigate and login + await page.goto('https://example.com/login') + await page.fill('input[name="email"]', credentials['email']) + await page.fill('input[name="password"]', credentials['password']) + await page.click('button[type="submit"]') + + # Wait for dashboard + await page.wait_for_url('**/dashboard', timeout=30000) + + print('Login successful! Cookies saved.') + + await browser.close() + + return {'profile_name': credentials['profile_name']} + +async def scrape_with_auth(profile_name: str): + kernel = Kernel() + + # Reuse profile + kb = kernel.browsers.create( + profile_name=profile_name, + headless=True + ) + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + # Already logged in! + await page.goto('https://example.com/dashboard/data') + + # Extract data + data = await page.eval_on_selector_all('.data-row', + 'rows => rows.map(row => row.textContent.trim())' + ) + + await browser.close() + kernel.browsers.delete_by_id(kb.session_id) + + return {'data': data} + +# Usage +profile = await login_and_save({ + 'email': 'user@example.com', + 'password': 'password123', + 'profile_name': 'example-user' +}) + +result1 = await scrape_with_auth(profile['profile_name']) +result2 = await scrape_with_auth(profile['profile_name']) +``` + + +## Environment Variables + +```bash +KERNEL_API_KEY=your_kernel_api_key +``` + +## How Profiles Work + +When you create a browser with `profile_save_changes: true`: + +1. **Browser opens** with fresh state (no cookies) +2. **You log in** (manually or via script) +3. **Browser closes** and Kernel saves: + - All cookies + - localStorage + - sessionStorage + - IndexedDB + - Service workers + +Next time you create a browser with the same `profile_name`: + +1. **Browser opens** with saved state (already logged in!) +2. **You can immediately** access protected pages +3. **No re-login** needed + +## Persistent Sessions (Alternative) + +For even more control, use persistent sessions: + +```typescript +// Create persistent session +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: 'user-session-123', + headless: false +}); + +// Log in via live view or script +// ... + +// Close browser but keep session alive +await browser.close(); +// Session goes to standby (free) after 1 minute + +// Later: reconnect to same session +const browsers = await kernel.browsers.list(); +const existing = browsers.find(b => b.persistent_id === 'user-session-123'); + +if (existing) { + const browser = await chromium.connectOverCDP({ + wsEndpoint: existing.cdp_ws_url + }); + // Still logged in, same tabs open! +} +``` + +**Difference:** +- **Profiles:** Save/load cookies between NEW browsers +- **Persistent sessions:** Keep THE SAME browser alive + +## Advanced: Handle MFA/2FA + +For sites with two-factor authentication: + +### Option 1: Manual MFA via Live View + +```typescript +const kb = await kernel.browsers.create({ + profile_name: 'user-with-mfa', + profile_save_changes: true, + headless: false // Must use headful for live view +}); + +console.log('Live view URL:', kb.browser_live_view_url); +console.log('Complete MFA manually in the browser'); + +// Wait for user to complete MFA +await new Promise(resolve => { + console.log('Press Enter after completing MFA...'); + process.stdin.once('data', resolve); +}); + +// MFA complete, cookies saved when browser closes +await browser.close(); +``` + +### Option 2: Automated TOTP + +If the site uses TOTP (Google Authenticator): + +```typescript +import { authenticator } from 'otplib'; + +const secret = 'YOUR_TOTP_SECRET'; // From QR code setup + +// Fill MFA code +await page.fill('input[name="mfa_code"]', authenticator.generate(secret)); +await page.click('button[type="submit"]'); +``` + +### Option 3: SMS via API + +Some services provide APIs to receive SMS codes: + +```typescript +import { Twilio } from 'twilio'; + +// Trigger SMS +await page.click('button.send-sms'); + +// Wait for SMS via Twilio +const client = new Twilio(accountSid, authToken); +const messages = await client.messages.list({ to: phoneNumber, limit: 1 }); +const code = messages[0].body.match(/\d{6}/)?.[0]; + +// Enter code +await page.fill('input[name="sms_code"]', code!); +``` + +## Variations + +### Per-User Profiles + +```typescript +async function getAuthenticatedBrowser(userId: string) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + + const profileName = `user-${userId}`; + + // Check if profile exists + const profiles = await kernel.profiles.list(); + const exists = profiles.some(p => p.name === profileName); + + if (!exists) { + // First time: need to log in + console.log(`Profile ${profileName} doesn't exist. Please log in.`); + await loginAndSave({ + email: `user${userId}@example.com`, + password: await getPasswordForUser(userId), + profileName + }); + } + + // Create browser with profile + return await kernel.browsers.create({ + profile_name: profileName, + headless: true + }); +} +``` + +### Conditional Re-auth + +Check if still logged in, re-auth if needed: + +```typescript +async function scrapeWithAutoReauth(profileName: string) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create({ profile_name: profileName }); + + const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); + const page = browser.contexts()[0].pages()[0]; + + // Try to access protected page + await page.goto('https://example.com/dashboard'); + + // Check if redirected to login + if (page.url().includes('/login')) { + console.log('Session expired, re-authenticating...'); + + await page.fill('input[name="email"]', process.env.EMAIL!); + await page.fill('input[name="password"]', process.env.PASSWORD!); + await page.click('button[type="submit"]'); + await page.waitForURL('**/dashboard'); + + // Save updated cookies + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + // Recreate with saved profile + return scrapeWithAutoReauth(profileName); + } + + // Still logged in, proceed + const data = await page.textContent('.data'); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { data }; +} +``` + +### Share Profiles Across Team + +Export/import profiles between team members: + +```typescript +// Export profile +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); +// Note: Profile export API coming soon +// For now: share profile_name and team members can use same profile + +// Team member A creates profile +await loginAndSave({ + email: 'shared@company.com', + password: 'password', + profileName: 'company-shared-account' +}); + +// Team member B uses same profile +const kb = await kernel.browsers.create({ + profile_name: 'company-shared-account' // Same org, same profile +}); +``` + +## Security Best Practices + +### 1. Never Commit Credentials + +```typescript +// ✗ BAD +const email = 'user@example.com'; +const password = 'password123'; + +// ✓ GOOD +const email = process.env.EMAIL!; +const password = process.env.PASSWORD!; +``` + +### 2. Use Different Profiles Per Environment + +```typescript +const profileName = process.env.NODE_ENV === 'production' + ? 'production-account' + : 'staging-account'; +``` + +### 3. Rotate Profiles Regularly + +```typescript +// Delete old profile +await kernel.profiles.delete('old-profile'); + +// Create new one +await loginAndSave({ + ...credentials, + profileName: 'new-profile' +}); +``` + +### 4. Use Read-Only Accounts + +When possible, log in with accounts that have read-only permissions for scraping/monitoring. + +## Common Issues + +### Profile Not Saving + +If cookies aren't persisting: + +1. Ensure `profile_save_changes: true` +2. Close browser properly (await `browser.close()`) +3. Check profile was created: +```typescript +const profiles = await kernel.profiles.list(); +console.log('Profiles:', profiles); +``` + +### Still Redirecting to Login + +If you're redirected despite using a profile: + +1. Session might have expired (time-based) +2. Site might use IP-based auth (use same proxy) +3. Site might clear cookies on suspicious activity + +Solution: Use `scrapeWithAutoReauth` pattern above. + +### MFA Required Every Time + +Some sites require MFA on every login from new IP: + +1. Use persistent sessions (keeps IP consistent) +2. Use same proxy every time +3. Use profiles + stealth mode to appear consistent + +## Cost Optimization + +**Without Profiles:** +- Log in on every request: 10s/request +- 1,000 requests = 167 minutes @ $0.05/min = $8.35 + +**With Profiles:** +- Log in once: 10s +- Scrape 1,000 times: 2s each = 33 minutes @ $0.05/min = $1.65 +- **Savings: 80%** + +## Related Recipes + +- [Download Files](/recipes/download-files-s3) - Download from logged-in accounts +- [Parallel Browsers](/recipes/parallel-browsers) - Use profiles in parallel +- [Screenshot + LLM](/recipes/screenshot-dom-llm) - Analyze logged-in content + +## Related Features + +- [Profiles](/browsers/profiles) - Full documentation +- [Persistence](/browsers/persistence) - Keep browsers alive +- [Standby Mode](/browsers/standby) - Zero-cost idle +- [Live View](/browsers/live-view) - Manual MFA completion + +## Support + +Questions about auth flows? Join our [Discord](https://discord.gg/FBrveQRcud). + diff --git a/recipes/block-ads-trackers.mdx b/recipes/block-ads-trackers.mdx new file mode 100644 index 0000000..b2944ae --- /dev/null +++ b/recipes/block-ads-trackers.mdx @@ -0,0 +1,491 @@ +--- +title: "Block Ads and Trackers with page.route()" +sidebarTitle: "Block Ads & Trackers" +description: "Speed up page loads by 50-70% by blocking ads, analytics, and tracking scripts with Playwright's network interception on Kernel browsers." +--- + +Block unnecessary resources to dramatically speed up page loads, reduce bandwidth, and avoid bot-detection scripts. Works perfectly with Kernel's CDP support. + +## What This Recipe Does + +1. Intercept all network requests with `page.route()` +2. Block ads, analytics, fonts, and images +3. Only load essential resources (HTML, JS, data) +4. Speed up automation by 50-70% + +## Use Cases + +- Faster web scraping +- Reduced bandwidth costs +- Avoid ad-blocker detection (ironically) +- Focus on text/data extraction +- Speed up E2E tests +- Bypass slow third-party scripts + +## Complete Code + + +```typescript TypeScript +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +// Common ad/tracker domains +const BLOCKED_DOMAINS = [ + 'googletagmanager.com', + 'google-analytics.com', + 'doubleclick.net', + 'facebook.net', + 'facebook.com/tr', + 'connect.facebook.net', + 'analytics.twitter.com', + 'static.ads-twitter.com', + 'amazon-adsystem.com', + 'googlesyndication.com', + 'adservice.google.com', + 'quantserve.com', + 'scorecardresearch.com', + 'hotjar.com', + 'mouseflow.com', + 'clarity.ms', + 'fullstory.com' +]; + +// Resource types to block +const BLOCKED_TYPES = ['image', 'font', 'stylesheet', 'media']; + +export async function scrapeWithoutAds(url: string) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Block resources + await page.route('**/*', route => { + const request = route.request(); + const url = request.url(); + const type = request.resourceType(); + + // Block by domain + if (BLOCKED_DOMAINS.some(domain => url.includes(domain))) { + console.log(`Blocked tracker: ${url}`); + return route.abort(); + } + + // Block by resource type + if (BLOCKED_TYPES.includes(type)) { + return route.abort(); + } + + // Allow everything else + return route.continue(); + }); + + // Navigate (much faster now) + const startTime = Date.now(); + await page.goto(url, { waitUntil: 'domcontentloaded' }); + const loadTime = Date.now() - startTime; + + // Extract content + const title = await page.title(); + const content = await page.evaluate(() => { + const article = document.querySelector('article') || document.body; + return article.innerText; + }); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { + url, + title, + content, + loadTimeMs: loadTime, + message: `Loaded ${loadTime}ms (typically 50-70% faster)` + }; +} + +// Usage +const result = await scrapeWithoutAds('https://example.com/article'); +console.log(result); +``` + +```python Python +from playwright.async_api import async_playwright +from kernel import Kernel +import time + +# Common ad/tracker domains +BLOCKED_DOMAINS = [ + 'googletagmanager.com', + 'google-analytics.com', + 'doubleclick.net', + 'facebook.net', + 'facebook.com/tr', + 'connect.facebook.net', + 'analytics.twitter.com', + 'static.ads-twitter.com', + 'amazon-adsystem.com', + 'googlesyndication.com', + 'adservice.google.com', + 'quantserve.com', + 'scorecardresearch.com', + 'hotjar.com', + 'mouseflow.com', + 'clarity.ms', + 'fullstory.com' +] + +BLOCKED_TYPES = ['image', 'font', 'stylesheet', 'media'] + +async def scrape_without_ads(url: str): + kernel = Kernel() + kb = kernel.browsers.create(headless=True) + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + # Block resources + async def handle_route(route): + request = route.request + req_url = request.url + req_type = request.resource_type + + # Block by domain + if any(domain in req_url for domain in BLOCKED_DOMAINS): + print(f'Blocked tracker: {req_url}') + await route.abort() + return + + # Block by type + if req_type in BLOCKED_TYPES: + await route.abort() + return + + # Allow everything else + await route.continue_() + + await page.route('**/*', handle_route) + + # Navigate (much faster now) + start_time = time.time() + await page.goto(url, wait_until='domcontentloaded') + load_time = (time.time() - start_time) * 1000 + + # Extract content + title = await page.title() + content = await page.evaluate('''() => { + const article = document.querySelector('article') || document.body; + return article.innerText; + }''') + + await browser.close() + kernel.browsers.delete_by_id(kb.session_id) + + return { + 'url': url, + 'title': title, + 'content': content, + 'load_time_ms': load_time, + 'message': f'Loaded {load_time:.0f}ms (typically 50-70% faster)' + } + +# Usage +result = await scrape_without_ads('https://example.com/article') +print(result) +``` + + +## Environment Variables + +```bash +KERNEL_API_KEY=your_kernel_api_key +``` + +## Expected Output + +```json +{ + "url": "https://example.com/article", + "title": "Article Title", + "content": "Article text content...", + "loadTimeMs": 1200, + "message": "Loaded 1200ms (typically 50-70% faster)" +} +``` + +Without blocking: ~4000ms +With blocking: ~1200ms +**Speedup: 70%** + +## Advanced Patterns + +### Block Everything Except Specific Domains + +```typescript +const ALLOWED_DOMAINS = ['example.com', 'cdn.example.com']; + +await page.route('**/*', route => { + const url = new URL(route.request().url()); + + // Allow only specific domains + if (ALLOWED_DOMAINS.some(domain => url.hostname.includes(domain))) { + return route.continue(); + } + + // Block everything else + return route.abort(); +}); +``` + +### Smart Blocking (Keep Some Images) + +```typescript +await page.route('**/*', route => { + const request = route.request(); + const url = request.url(); + const type = request.resourceType(); + + // Keep product images, block others + if (type === 'image') { + if (url.includes('/products/') || url.includes('/images/')) { + return route.continue(); + } + return route.abort(); + } + + // Block trackers + if (BLOCKED_DOMAINS.some(d => url.includes(d))) { + return route.abort(); + } + + return route.continue(); +}); +``` + +### Count Blocked Requests + +```typescript +let blockedCount = 0; +const blockedUrls: string[] = []; + +await page.route('**/*', route => { + const url = route.request().url(); + + if (BLOCKED_DOMAINS.some(d => url.includes(d))) { + blockedCount++; + blockedUrls.push(url); + return route.abort(); + } + + return route.continue(); +}); + +await page.goto(url); + +console.log(`Blocked ${blockedCount} requests`); +console.log('Blocked URLs:', blockedUrls); +``` + +### Block by File Size + +```typescript +await page.route('**/*', async route => { + // Fetch to check size + const response = await route.fetch(); + const contentLength = parseInt( + response.headers()['content-length'] || '0' + ); + + // Block files >500KB + if (contentLength > 500 * 1024) { + console.log(`Blocked large file: ${route.request().url()} (${contentLength} bytes)`); + return route.abort(); + } + + // Fulfill with the fetched response + return route.fulfill({ response }); +}); +``` + +## Performance Comparison + +### Before (No Blocking) + +``` +Total requests: 150 +- HTML/JS/CSS: 20 (essential) +- Images: 50 +- Fonts: 10 +- Analytics/Ads: 70 +Total load time: 4.2 seconds +``` + +### After (With Blocking) + +``` +Total requests: 20 +- HTML/JS/CSS: 20 (essential) +- Images: 0 (blocked) +- Fonts: 0 (blocked) +- Analytics/Ads: 0 (blocked) +Total load time: 1.3 seconds +Speedup: 69% +``` + +## Common Patterns by Use Case + +### News Articles (Text Only) + +```typescript +const BLOCKED_TYPES = ['image', 'font', 'stylesheet', 'media', 'websocket']; +const BLOCKED_DOMAINS = [ + ...BLOCKED_DOMAINS, // from above + 'taboola.com', + 'outbrain.com', + 'disqus.com' +]; +``` + +### E-Commerce (Keep Product Images) + +```typescript +await page.route('**/*', route => { + const url = route.request().url(); + const type = route.request().resourceType(); + + // Keep product images + if (type === 'image' && url.includes('/product')) { + return route.continue(); + } + + // Block other images + if (type === 'image') { + return route.abort(); + } + + // Block trackers + if (BLOCKED_DOMAINS.some(d => url.includes(d))) { + return route.abort(); + } + + return route.continue(); +}); +``` + +### API Scraping (Block Everything Visual) + +```typescript +await page.route('**/*', route => { + const type = route.request().resourceType(); + + // Allow only document, xhr, fetch + if (['document', 'xhr', 'fetch'].includes(type)) { + return route.continue(); + } + + // Block everything else + return route.abort(); +}); +``` + +## Combine with Stealth Mode + +For maximum speed and stealth: + +```typescript +const kb = await kernel.browsers.create({ + headless: true, + stealth: true // Adds proxies + CAPTCHA solver +}); + +// Then add resource blocking +await page.route('**/*', route => { + // ... blocking logic ... +}); +``` + +This combines: +- Proxy rotation (stealth mode) +- CAPTCHA solving (stealth mode) +- Resource blocking (your code) + +Result: Fast, undetectable scraping. + +## Troubleshooting + +### Page Won't Load + +If the page appears blank or broken: + +1. Check if you blocked too much: +```typescript +// Debug: log what you're blocking +await page.route('**/*', route => { + if (shouldBlock(route)) { + console.log('Blocking:', route.request().url()); + return route.abort(); + } + return route.continue(); +}); +``` + +2. Allow essential resources: +```typescript +// Don't block document and scripts +if (['document', 'script', 'xhr', 'fetch'].includes(type)) { + return route.continue(); +} +``` + +### Content Missing + +If extracted content is incomplete: + +1. Some sites load content via JS that needs stylesheets +2. Try less aggressive blocking: +```typescript +// Block only trackers, keep images/fonts +const BLOCKED_TYPES = []; // empty +``` + +### Still Slow + +If blocking doesn't help: + +1. The site might have slow server +2. Try `waitUntil: 'domcontentloaded'` instead of `'networkidle'` +3. Use headless mode (faster than headful) + +## Cost Savings + +**Bandwidth:** +- Before: 5MB per page × 1,000 pages = 5GB +- After: 500KB per page × 1,000 pages = 500MB +- **Savings: 90% bandwidth** + +**Time:** +- Before: 4s per page × 1,000 pages = 67 minutes +- After: 1.5s per page × 1,000 pages = 25 minutes +- **Savings: 63% time** = 63% cost on Kernel + +## Related Recipes + +- [Screenshot + DOM + LLM](/recipes/screenshot-dom-llm) - Extract content with AI +- [Network Interception](/troubleshooting/network-interception) - Full guide +- [Parallel Browsers](/recipes/parallel-browsers) - Process faster + +## Related Features + +- [Stealth Mode](/browsers/stealth) - Avoid detection +- [Headless Mode](/browsers/headless) - Faster execution +- [Network Interception](/troubleshooting/network-interception) + +## Support + +Questions about resource blocking? Join our [Discord](https://discord.gg/FBrveQRcud). + diff --git a/recipes/download-files-s3.mdx b/recipes/download-files-s3.mdx new file mode 100644 index 0000000..7ba9794 --- /dev/null +++ b/recipes/download-files-s3.mdx @@ -0,0 +1,514 @@ +--- +title: "Download Files and Upload to S3" +sidebarTitle: "Download to S3" +description: "Download PDFs, invoices, reports, or any files from the browser and upload to S3, R2, or cloud storage. Complete recipe with Kernel's File I/O API." +--- + +Download files triggered in the browser (PDFs, CSVs, images) and automatically upload them to cloud storage. Uses Kernel's File I/O API to access downloaded files. + +## What This Recipe Does + +1. Navigate to a page with download links +2. Trigger file download in browser +3. Wait for download to complete +4. Read file via Kernel's File I/O API +5. Upload to S3/R2/GCS +6. Return public URL or confirmation + +## Use Cases + +- Automated invoice downloads +- Export reports from SaaS tools +- Backup form submissions +- Archive receipts or statements +- Download generated PDFs +- Scrape file attachments + +## Complete Code + + +```typescript TypeScript/Next.js +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; +import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'; + +const s3 = new S3Client({ region: 'us-east-1' }); + +export async function downloadAndUpload(config: { + pageUrl: string; + downloadSelector: string; + s3Bucket: string; + s3Key?: string; +}) { + const { pageUrl, downloadSelector, s3Bucket, s3Key } = config; + + // Create Kernel browser + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create(); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Navigate to page + await page.goto(pageUrl); + + // Trigger download + console.log('Triggering download...'); + await page.click(downloadSelector); + + // Wait for download to appear in filesystem + await page.waitForTimeout(3000); + + // List downloads via Kernel File I/O API + const files = await kernel.browsers.files.list( + kb.session_id, + '/downloads' + ); + + if (files.length === 0) { + throw new Error('No files downloaded'); + } + + // Get most recent file + const latestFile = files.sort((a, b) => + new Date(b.modified_time).getTime() - new Date(a.modified_time).getTime() + )[0]; + + console.log(`Downloaded: ${latestFile.name} (${latestFile.size} bytes)`); + + // Read file content + const fileBuffer = await kernel.browsers.files.read( + kb.session_id, + latestFile.path + ); + + // Upload to S3 + const uploadKey = s3Key || `downloads/${Date.now()}-${latestFile.name}`; + await s3.send(new PutObjectCommand({ + Bucket: s3Bucket, + Key: uploadKey, + Body: fileBuffer, + ContentType: getContentType(latestFile.name) + })); + + const s3Url = `https://${s3Bucket}.s3.amazonaws.com/${uploadKey}`; + console.log(`Uploaded to: ${s3Url}`); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { + fileName: latestFile.name, + fileSize: latestFile.size, + s3Url, + s3Key: uploadKey + }; +} + +function getContentType(filename: string): string { + const ext = filename.split('.').pop()?.toLowerCase(); + const types: Record = { + 'pdf': 'application/pdf', + 'csv': 'text/csv', + 'xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + 'png': 'image/png', + 'jpg': 'image/jpeg', + 'jpeg': 'image/jpeg', + 'zip': 'application/zip' + }; + return types[ext || ''] || 'application/octet-stream'; +} + +// Usage in API route +export default async function handler(req, res) { + const result = await downloadAndUpload({ + pageUrl: 'https://example.com/invoices', + downloadSelector: 'a[href*="download"]', + s3Bucket: process.env.S3_BUCKET!, + s3Key: `invoices/${req.body.invoiceId}.pdf` + }); + + res.json(result); +} +``` + +```python Python +import os +from playwright.async_api import async_playwright +from kernel import Kernel +import boto3 +from datetime import datetime + +s3 = boto3.client('s3') + +async def download_and_upload(config: dict): + page_url = config['page_url'] + download_selector = config['download_selector'] + s3_bucket = config['s3_bucket'] + s3_key = config.get('s3_key') + + # Create Kernel browser + kernel = Kernel() + kb = kernel.browsers.create() + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + # Navigate + await page.goto(page_url) + + # Trigger download + print('Triggering download...') + await page.click(download_selector) + + # Wait for download + await page.wait_for_timeout(3000) + + await browser.close() + + # List downloads via Kernel File I/O API + files = kernel.browsers.files.list(kb.session_id, '/downloads') + + if not files: + raise Exception('No files downloaded') + + # Get most recent file + latest_file = max(files, key=lambda f: f['modified_time']) + + print(f"Downloaded: {latest_file['name']} ({latest_file['size']} bytes)") + + # Read file content + file_buffer = kernel.browsers.files.read( + kb.session_id, + latest_file['path'] + ) + + # Upload to S3 + upload_key = s3_key or f"downloads/{int(datetime.now().timestamp())}-{latest_file['name']}" + s3.put_object( + Bucket=s3_bucket, + Key=upload_key, + Body=file_buffer, + ContentType=get_content_type(latest_file['name']) + ) + + s3_url = f"https://{s3_bucket}.s3.amazonaws.com/{upload_key}" + print(f'Uploaded to: {s3_url}') + + kernel.browsers.delete_by_id(kb.session_id) + + return { + 'file_name': latest_file['name'], + 'file_size': latest_file['size'], + 's3_url': s3_url, + 's3_key': upload_key + } + +def get_content_type(filename: str) -> str: + ext = filename.split('.')[-1].lower() + types = { + 'pdf': 'application/pdf', + 'csv': 'text/csv', + 'xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + 'png': 'image/png', + 'jpg': 'image/jpeg', + 'jpeg': 'image/jpeg', + 'zip': 'application/zip' + } + return types.get(ext, 'application/octet-stream') + +# Usage +result = await download_and_upload({ + 'page_url': 'https://example.com/invoices', + 'download_selector': 'a[href*="download"]', + 's3_bucket': os.getenv('S3_BUCKET'), + 's3_key': 'invoices/latest.pdf' +}) +print(result) +``` + + +## Environment Variables + +```bash +KERNEL_API_KEY=your_kernel_api_key +AWS_ACCESS_KEY_ID=your_aws_key +AWS_SECRET_ACCESS_KEY=your_aws_secret +S3_BUCKET=your-bucket-name +``` + +## Expected Output + +```json +{ + "fileName": "invoice-2024-03.pdf", + "fileSize": 245678, + "s3Url": "https://my-bucket.s3.amazonaws.com/downloads/1234567890-invoice-2024-03.pdf", + "s3Key": "downloads/1234567890-invoice-2024-03.pdf" +} +``` + +## Variations + +### Upload to Cloudflare R2 + +```typescript +import { S3Client } from '@aws-sdk/client-s3'; + +const r2 = new S3Client({ + region: 'auto', + endpoint: `https://${process.env.CLOUDFLARE_ACCOUNT_ID}.r2.cloudflarestorage.com`, + credentials: { + accessKeyId: process.env.R2_ACCESS_KEY_ID!, + secretAccessKey: process.env.R2_SECRET_ACCESS_KEY! + } +}); + +// Upload same way as S3 +await r2.send(new PutObjectCommand({ + Bucket: process.env.R2_BUCKET!, + Key: uploadKey, + Body: fileBuffer +})); +``` + +### Upload to Google Cloud Storage + +```typescript +import { Storage } from '@google-cloud/storage'; + +const gcs = new Storage({ + projectId: process.env.GCP_PROJECT_ID, + keyFilename: process.env.GCP_KEY_FILE +}); + +const bucket = gcs.bucket(process.env.GCS_BUCKET!); +const file = bucket.file(uploadKey); + +await file.save(fileBuffer, { + metadata: { + contentType: getContentType(latestFile.name) + } +}); + +const publicUrl = `https://storage.googleapis.com/${process.env.GCS_BUCKET}/${uploadKey}`; +``` + +### Download Multiple Files + +```typescript +// Trigger multiple downloads +await page.click('button.download-all'); +await page.waitForTimeout(5000); + +// Get all downloaded files +const files = await kernel.browsers.files.list(kb.session_id, '/downloads'); + +// Upload all +const uploads = await Promise.all( + files.map(async (file) => { + const buffer = await kernel.browsers.files.read(kb.session_id, file.path); + const key = `downloads/${Date.now()}-${file.name}`; + + await s3.send(new PutObjectCommand({ + Bucket: s3Bucket, + Key: key, + Body: buffer + })); + + return { name: file.name, key }; + }) +); + +return { uploadedFiles: uploads }; +``` + +### Wait for Specific File + +```typescript +async function waitForFile( + kernel: Kernel, + sessionId: string, + filename: string, + timeoutMs = 30000 +): Promise { + const startTime = Date.now(); + + while (Date.now() - startTime < timeoutMs) { + const files = await kernel.browsers.files.list(sessionId, '/downloads'); + const matchingFile = files.find(f => f.name.includes(filename)); + + if (matchingFile) { + return matchingFile; + } + + await new Promise(resolve => setTimeout(resolve, 1000)); + } + + throw new Error(`File ${filename} not found after ${timeoutMs}ms`); +} + +// Usage +await page.click('a.download-report'); +const file = await waitForFile(kernel, kb.session_id, 'report', 30000); +``` + +### Generate Pre-Signed URL + +Instead of public S3 URL, generate time-limited signed URL: + +```typescript +import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; +import { GetObjectCommand } from '@aws-sdk/client-s3'; + +// Upload file +await s3.send(new PutObjectCommand({ + Bucket: s3Bucket, + Key: uploadKey, + Body: fileBuffer +})); + +// Generate signed URL (valid for 1 hour) +const signedUrl = await getSignedUrl( + s3, + new GetObjectCommand({ + Bucket: s3Bucket, + Key: uploadKey + }), + { expiresIn: 3600 } +); + +return { signedUrl }; +``` + +## Advanced: Monitor Download Progress + +```typescript +// Check file size increase to monitor download +async function waitForDownloadComplete( + kernel: Kernel, + sessionId: string, + filename: string +): Promise { + let lastSize = 0; + let stableCount = 0; + + while (true) { + const files = await kernel.browsers.files.list(sessionId, '/downloads'); + const file = files.find(f => f.name === filename); + + if (!file) { + await new Promise(resolve => setTimeout(resolve, 1000)); + continue; + } + + if (file.size === lastSize) { + stableCount++; + if (stableCount >= 3) { + // Size stable for 3 checks = download complete + return; + } + } else { + stableCount = 0; + lastSize = file.size; + } + + console.log(`Download progress: ${file.size} bytes`); + await new Promise(resolve => setTimeout(resolve, 1000)); + } +} +``` + +## Common Issues + +### File Not Appearing + +If file doesn't appear in `/downloads`: + +1. Check if download actually triggered: +```typescript +// Add download listener +page.on('download', download => { + console.log('Download started:', download.suggestedFilename()); +}); +``` + +2. Wait longer: +```typescript +await page.waitForTimeout(5000); // Increase wait time +``` + +3. Check different directory: +```typescript +// Some browsers use different paths +const files = await kernel.browsers.files.list(kb.session_id, '/'); +console.log('All files:', files); +``` + +### S3 Upload Fails + +Check credentials and permissions: + +```typescript +try { + await s3.send(new PutObjectCommand({ ... })); +} catch (error) { + console.error('S3 error:', error); + // Check: bucket name, region, credentials, IAM permissions +} +``` + +### Out of Memory + +For large files (>100MB): + +```typescript +// Stream instead of buffering +import { Readable } from 'stream'; +import { Upload } from '@aws-sdk/lib-storage'; + +const fileBuffer = await kernel.browsers.files.read(kb.session_id, file.path); +const stream = Readable.from(fileBuffer); + +const upload = new Upload({ + client: s3, + params: { + Bucket: s3Bucket, + Key: uploadKey, + Body: stream + } +}); + +await upload.done(); +``` + +## Cost Estimation + +**Per download:** +- Kernel browser: ~$0.01 (2-3s @ $0.05/min) +- S3 PUT request: $0.000005 +- S3 storage: $0.023/GB/month +- **Total: ~$0.01 per download** + +**1,000 downloads/month:** ~$10 + +## Related Recipes + +- [Auth & Cookies](/recipes/auth-cookies-sessions) - Download from logged-in pages +- [Parallel Browsers](/recipes/parallel-browsers) - Download multiple files faster +- [Screenshot + LLM](/recipes/screenshot-dom-llm) - Extract text from PDFs with OCR + +## Related Features + +- [File I/O API](/browsers/file-io) - Full documentation +- [Create a Browser](/browsers/create-a-browser) +- [Persistence](/browsers/persistence) - Reuse auth for multiple downloads + +## Support + +Questions about file downloads? Join our [Discord](https://discord.gg/FBrveQRcud). + diff --git a/recipes/parallel-browsers.mdx b/recipes/parallel-browsers.mdx new file mode 100644 index 0000000..fa2fc96 --- /dev/null +++ b/recipes/parallel-browsers.mdx @@ -0,0 +1,511 @@ +--- +title: "Run Parallel Browser Sessions" +sidebarTitle: "Parallel Browsers" +description: "Process thousands of URLs concurrently with parallel browser sessions. Complete guide to batch scraping, concurrent automation, and performance optimization." +--- + +Run multiple browser sessions in parallel to process large workloads faster. Scale from 1 to 100+ concurrent browsers without infrastructure management. + +## What This Recipe Does + +1. Split workload into batches +2. Launch multiple browsers concurrently +3. Process items in parallel +4. Aggregate results +5. Handle errors gracefully + +## Use Cases + +- Batch scraping (1000+ URLs) +- Parallel E2E testing +- Competitive price monitoring +- Content validation across pages +- Screenshot generation at scale +- Multi-account automation + +## Complete Code + + +```typescript TypeScript +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; +import pLimit from 'p-limit'; + +interface ScrapResult { + url: string; + title: string; + status: 'success' | 'error'; + error?: string; +} + +async function scrapeUrl(kernel: Kernel, url: string): Promise { + try { + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto(url, { timeout: 30000 }); + const title = await page.title(); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { url, title, status: 'success' }; + } catch (error) { + return { + url, + title: '', + status: 'error', + error: error instanceof Error ? error.message : 'Unknown error' + }; + } +} + +export async function scrapeParallel( + urls: string[], + concurrency = 10 +): Promise { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + + // Limit concurrent browsers + const limit = pLimit(concurrency); + + console.log(`Scraping ${urls.length} URLs with concurrency ${concurrency}`); + + // Process in parallel + const results = await Promise.all( + urls.map(url => limit(() => scrapeUrl(kernel, url))) + ); + + // Summary + const successful = results.filter(r => r.status === 'success').length; + const failed = results.filter(r => r.status === 'error').length; + + console.log(`Complete: ${successful} success, ${failed} failed`); + + return results; +} + +// Usage +const urls = [ + 'https://example.com/page1', + 'https://example.com/page2', + // ... 1000 more URLs +]; + +const results = await scrapeParallel(urls, 20); // 20 concurrent browsers +``` + +```python Python +import asyncio +from playwright.async_api import async_playwright +from kernel import Kernel +from typing import List, Dict + +async def scrape_url(kernel: Kernel, url: str) -> Dict: + try: + kb = kernel.browsers.create(headless=True) + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + await page.goto(url, timeout=30000) + title = await page.title() + + await browser.close() + + kernel.browsers.delete_by_id(kb.session_id) + + return {'url': url, 'title': title, 'status': 'success'} + except Exception as e: + return {'url': url, 'title': '', 'status': 'error', 'error': str(e)} + +async def scrape_parallel(urls: List[str], concurrency: int = 10) -> List[Dict]: + kernel = Kernel() + + print(f'Scraping {len(urls)} URLs with concurrency {concurrency}') + + # Create semaphore for concurrency control + semaphore = asyncio.Semaphore(concurrency) + + async def limited_scrape(url: str): + async with semaphore: + return await scrape_url(kernel, url) + + # Process in parallel + results = await asyncio.gather( + *[limited_scrape(url) for url in urls] + ) + + # Summary + successful = sum(1 for r in results if r['status'] == 'success') + failed = sum(1 for r in results if r['status'] == 'error') + + print(f'Complete: {successful} success, {failed} failed') + + return results + +# Usage +urls = [ + 'https://example.com/page1', + 'https://example.com/page2', + # ... more URLs +] + +results = await scrape_parallel(urls, concurrency=20) +``` + + +## Environment Variables + +```bash +KERNEL_API_KEY=your_kernel_api_key +``` + +## Performance Numbers + +| URLs | Concurrency | Time (serial) | Time (parallel) | Speedup | +|------|-------------|---------------|-----------------|---------| +| 100 | 10 | 500s | 50s | 10× | +| 1000 | 20 | 5000s | 250s | 20× | +| 1000 | 50 | 5000s | 100s | 50× | + +**Cost (1000 URLs @ 5s each):** +- Serial: 83 minutes @ $0.05/min = $4.15 +- Parallel (20×): 83 minutes total (distributed) = $4.15 +- **Same cost, 20× faster!** + +## Advanced Patterns + +### Batched Processing with Progress + +```typescript +async function scrapeInBatches( + urls: string[], + batchSize = 50, + concurrency = 10 +): Promise { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const allResults: ScrapResult[] = []; + + // Split into batches + const batches: string[][] = []; + for (let i = 0; i < urls.length; i += batchSize) { + batches.push(urls.slice(i, i + batchSize)); + } + + console.log(`Processing ${urls.length} URLs in ${batches.length} batches`); + + // Process each batch + for (let i = 0; i < batches.length; i++) { + const batch = batches[i]; + console.log(`Batch ${i + 1}/${batches.length}: ${batch.length} URLs`); + + const limit = pLimit(concurrency); + const results = await Promise.all( + batch.map(url => limit(() => scrapeUrl(kernel, url))) + ); + + allResults.push(...results); + + // Progress + const progress = ((i + 1) / batches.length * 100).toFixed(1); + console.log(`Progress: ${progress}% (${allResults.length}/${urls.length})`); + + // Optional: delay between batches + if (i < batches.length - 1) { + await new Promise(resolve => setTimeout(resolve, 1000)); + } + } + + return allResults; +} +``` + +### Retry Failed URLs + +```typescript +async function scrapeWithRetry( + urls: string[], + maxRetries = 3, + concurrency = 10 +): Promise { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + let results = await scrapeParallel(urls, concurrency); + + // Retry failed URLs + for (let attempt = 1; attempt <= maxRetries; attempt++) { + const failed = results.filter(r => r.status === 'error'); + + if (failed.length === 0) break; + + console.log(`Retry ${attempt}/${maxRetries}: ${failed.length} failed URLs`); + + const retryResults = await scrapeParallel( + failed.map(r => r.url), + concurrency + ); + + // Update results + retryResults.forEach(retry => { + const index = results.findIndex(r => r.url === retry.url); + if (index !== -1) { + results[index] = retry; + } + }); + } + + return results; +} +``` + +### Adaptive Concurrency + +Automatically adjust concurrency based on error rate: + +```typescript +async function scrapeAdaptive(urls: string[]): Promise { + let concurrency = 20; + const minConcurrency = 5; + const maxConcurrency = 50; + + const results: ScrapResult[] = []; + + for (let i = 0; i < urls.length; i += 100) { + const batch = urls.slice(i, i + 100); + const batchResults = await scrapeParallel(batch, concurrency); + + results.push(...batchResults); + + // Calculate error rate + const errorRate = batchResults.filter(r => r.status === 'error').length / batchResults.length; + + // Adjust concurrency + if (errorRate > 0.2) { + concurrency = Math.max(minConcurrency, concurrency - 5); + console.log(`High error rate (${(errorRate * 100).toFixed(1)}%), reducing concurrency to ${concurrency}`); + } else if (errorRate < 0.05) { + concurrency = Math.min(maxConcurrency, concurrency + 5); + console.log(`Low error rate (${(errorRate * 100).toFixed(1)}%), increasing concurrency to ${concurrency}`); + } + } + + return results; +} +``` + +### Save Progress to Resume Later + +```typescript +import fs from 'fs'; + +async function scrapeResumable( + urls: string[], + progressFile = 'progress.json', + concurrency = 10 +): Promise { + // Load previous progress + let completed: ScrapResult[] = []; + if (fs.existsSync(progressFile)) { + completed = JSON.parse(fs.readFileSync(progressFile, 'utf-8')); + console.log(`Resuming: ${completed.length} already done`); + } + + // Filter remaining URLs + const completedUrls = new Set(completed.map(r => r.url)); + const remaining = urls.filter(url => !completedUrls.has(url)); + + if (remaining.length === 0) { + console.log('All URLs already processed'); + return completed; + } + + console.log(`Processing ${remaining.length} remaining URLs`); + + // Process in chunks, saving after each + const chunkSize = 100; + for (let i = 0; i < remaining.length; i += chunkSize) { + const chunk = remaining.slice(i, i + chunkSize); + const results = await scrapeParallel(chunk, concurrency); + + completed.push(...results); + + // Save progress + fs.writeFileSync(progressFile, JSON.stringify(completed, null, 2)); + console.log(`Progress saved: ${completed.length}/${urls.length}`); + } + + return completed; +} +``` + +## Optimization Tips + +### 1. Choose Right Concurrency + +```typescript +// Too low: slow +await scrapeParallel(urls, 5); + +// Optimal for most cases +await scrapeParallel(urls, 20); + +// High (for fast sites) +await scrapeParallel(urls, 50); + +// Too high: errors, rate limits +await scrapeParallel(urls, 200); // Not recommended +``` + +**Rule of thumb:** Start with 20, adjust based on results. + +### 2. Use Headless Mode + +```typescript +// Headless is 2× faster and cheaper +const kb = await kernel.browsers.create({ headless: true }); +``` + +### 3. Block Unnecessary Resources + +```typescript +async function scrapeUrlFast(kernel: Kernel, url: string) { + const kb = await kernel.browsers.create({ headless: true }); + const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); + const page = browser.contexts()[0].pages()[0]; + + // Block images, fonts + await page.route('**/*', route => { + if (['image', 'font', 'stylesheet'].includes(route.request().resourceType())) { + return route.abort(); + } + return route.continue(); + }); + + await page.goto(url); + // 50% faster +} +``` + +### 4. Reuse Profiles for Auth + +```typescript +// Don't log in 1000 times! +const kb = await kernel.browsers.create({ + profile_name: 'shared-auth', + headless: true +}); +// Already logged in from previous session +``` + +## Real-World Example: E-Commerce Price Monitor + +```typescript +interface Product { + url: string; + name: string; + price: number; +} + +async function monitorPrices(productUrls: string[]): Promise { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const limit = pLimit(30); + + const results = await Promise.all( + productUrls.map(url => limit(async () => { + try { + const kb = await kernel.browsers.create({ headless: true }); + const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); + const page = browser.contexts()[0].pages()[0]; + + // Block images for speed + await page.route('**/*', r => + r.request().resourceType() === 'image' ? r.abort() : r.continue() + ); + + await page.goto(url, { timeout: 20000 }); + + const name = await page.textContent('h1.product-title'); + const priceText = await page.textContent('.price'); + const price = parseFloat(priceText?.replace(/[^0-9.]/g, '') || '0'); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { url, name: name || '', price }; + } catch (error) { + console.error(`Failed: ${url}`, error); + return { url, name: '', price: 0 }; + } + })) + ); + + return results.filter(r => r.price > 0); +} + +// Monitor 500 products in ~1 minute +const products = await monitorPrices(productUrls); +``` + +## Common Issues + +### Rate Limited + +If you're hitting rate limits: + +1. Reduce concurrency +2. Add delays between batches +3. Use proxies (stealth mode) +4. Use profiles to appear as same user + +### Out of Memory (Node.js) + +For very large workloads: + +```bash +# Increase Node.js memory +node --max-old-space-size=4096 script.js +``` + +### Timeouts + +If pages timeout frequently: + +```typescript +// Increase timeout +await page.goto(url, { timeout: 60000 }); + +// Or fail fast +await page.goto(url, { timeout: 10000 }); +``` + +## Cost Estimation + +**Scraping 10,000 URLs:** +- Average 5s per URL +- Sequential: 13.9 hours = 834 minutes @ $0.05/min = **$41.70** +- Parallel (20×): 0.7 hours = 42 minutes @ $0.05/min = **$2.10** (per URL) +- Total parallel cost: **$41.70** (same cost, 20× faster) + +**Note:** Total compute time is the same; you're paying for parallelism, not for faster execution. + +## Related Recipes + +- [Block Ads](/recipes/block-ads-trackers) - Speed up each request +- [Auth & Cookies](/recipes/auth-cookies-sessions) - Reuse login across parallel sessions +- [Download Files](/recipes/download-files-s3) - Parallel file downloads + +## Related Features + +- [Create a Browser](/browsers/create-a-browser) +- [Headless Mode](/browsers/headless) - Faster & cheaper +- [Stealth Mode](/browsers/stealth) - Avoid rate limits + +## Support + +Questions about scaling? Join our [Discord](https://discord.gg/FBrveQRcud). + diff --git a/recipes/qa-on-deploy.mdx b/recipes/qa-on-deploy.mdx new file mode 100644 index 0000000..e2b1b60 --- /dev/null +++ b/recipes/qa-on-deploy.mdx @@ -0,0 +1,591 @@ +--- +title: "Automate QA Tests on Every Vercel Deployment" +sidebarTitle: "QA on Deploy" +description: "Run automated QA checks on every preview and production deployment using Kernel's Vercel integration. Catch bugs before users do." +--- + +Automatically test your Vercel deployments with web agents that check functionality, visuals, and performance. Catch regressions before they reach users. + +## What This Recipe Does + +1. Vercel triggers deployment (preview or production) +2. Kernel receives webhook +3. Web agent navigates deployment URL +4. Tests run automatically (visual, functional, performance) +5. Results posted to Vercel deployment checks +6. Deployment blocked if tests fail + +## Use Cases + +- Visual regression testing +- Broken link detection +- Auth flow validation +- Critical path testing (checkout, signup, etc.) +- Performance monitoring +- Accessibility checks +- Content validation + +## Setup: Native Integration + +### 1. Install Kernel from Vercel Marketplace + +Visit [vercel.com/integrations/kernel](https://vercel.com/integrations/kernel) and click **Add Integration**. + +### 2. Connect Projects + +Select which Vercel projects should have QA checks. + +### 3. Configure Checks + +In Vercel dashboard → Project → Settings → Integrations → Kernel: + +- Enable checks: Visual Regression, Broken Links, etc. +- Set baseline URLs +- Configure test parameters + +## Manual Setup (Advanced) + +If you want custom QA logic beyond the built-in checks: + +### 1. Create Kernel QA App + + +```typescript TypeScript (kernel-qa/index.ts) +import { App, KernelContext } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; + +const app = new App('qa-checks'); + +interface QAPayload { + deploymentUrl: string; + baselineUrl?: string; + checks: string[]; // ['visual', 'links', 'performance'] +} + +app.action('run-checks', async (ctx: KernelContext, payload: QAPayload) => { + const { deploymentUrl, baselineUrl, checks } = payload; + + const results: Record = {}; + + // Create browser + const kb = await ctx.kernel.browsers.create({ + invocation_id: ctx.invocation_id, + headless: true + }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Visual Regression Check + if (checks.includes('visual') && baselineUrl) { + console.log('Running visual regression check...'); + + await page.goto(deploymentUrl); + const newScreenshot = await page.screenshot({ fullPage: true }); + + await page.goto(baselineUrl); + const baselineScreenshot = await page.screenshot({ fullPage: true }); + + // Compare screenshots (use pixelmatch or similar) + const diffPercentage = await compareImages( + newScreenshot, + baselineScreenshot + ); + + results.visual = { + passed: diffPercentage < 0.5, // <0.5% change + diffPercentage, + message: `Visual diff: ${(diffPercentage * 100).toFixed(2)}%` + }; + } + + // Broken Links Check + if (checks.includes('links')) { + console.log('Running broken links check...'); + + await page.goto(deploymentUrl); + + const links = await page.$$eval('a[href]', anchors => + anchors.map(a => a.getAttribute('href')).filter(Boolean) + ); + + const brokenLinks = []; + + for (const href of links) { + try { + const url = new URL(href, deploymentUrl); + if (url.hostname === new URL(deploymentUrl).hostname) { + const response = await page.goto(url.toString()); + if (!response || response.status() >= 400) { + brokenLinks.push(href); + } + } + } catch (error) { + brokenLinks.push(href); + } + } + + results.links = { + passed: brokenLinks.length === 0, + brokenCount: brokenLinks.length, + brokenLinks: brokenLinks.slice(0, 10), // First 10 + message: brokenLinks.length > 0 + ? `Found ${brokenLinks.length} broken links` + : 'All links working' + }; + } + + // Performance Check + if (checks.includes('performance')) { + console.log('Running performance check...'); + + const startTime = Date.now(); + await page.goto(deploymentUrl, { waitUntil: 'networkidle' }); + const loadTime = Date.now() - startTime; + + const metrics = await page.evaluate(() => { + const perf = performance.timing; + return { + domContentLoaded: perf.domContentLoadedEventEnd - perf.navigationStart, + fullyLoaded: perf.loadEventEnd - perf.navigationStart, + firstPaint: performance.getEntriesByType('paint')[0]?.startTime || 0 + }; + }); + + results.performance = { + passed: loadTime < 5000, // <5s + loadTime, + metrics, + message: `Page loaded in ${loadTime}ms` + }; + } + + await browser.close(); + + // Overall result + const allPassed = Object.values(results).every((r: any) => r.passed); + + return { + passed: allPassed, + checks: results, + summary: allPassed ? 'All checks passed' : 'Some checks failed' + }; +}); + +export default app; + +// Helper function (implement with pixelmatch or similar) +async function compareImages(img1: Buffer, img2: Buffer): Promise { + // Simple byte comparison (use proper image diff library in production) + const diff = Buffer.compare(img1, img2); + return diff === 0 ? 0 : 1.0; +} +``` + +```python Python (kernel-qa/main.py) +import kernel +from playwright.async_api import async_playwright +from typing import Dict, List, Any +import time + +app = kernel.App('qa-checks') + +@app.action('run-checks') +async def run_checks(ctx: kernel.KernelContext, payload: Dict[str, Any]): + deployment_url = payload['deployment_url'] + baseline_url = payload.get('baseline_url') + checks = payload.get('checks', []) + + results = {} + + # Create browser + kb = await ctx.kernel.browsers.create( + invocation_id=ctx.invocation_id, + headless=True + ) + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + # Visual regression + if 'visual' in checks and baseline_url: + print('Running visual regression...') + + await page.goto(deployment_url) + new_screenshot = await page.screenshot(full_page=True) + + await page.goto(baseline_url) + baseline_screenshot = await page.screenshot(full_page=True) + + # Compare (implement with PIL or similar) + diff_pct = compare_images(new_screenshot, baseline_screenshot) + + results['visual'] = { + 'passed': diff_pct < 0.5, + 'diff_percentage': diff_pct, + 'message': f'Visual diff: {diff_pct:.2f}%' + } + + # Broken links + if 'links' in checks: + print('Running broken links check...') + + await page.goto(deployment_url) + links = await page.eval_on_selector_all( + 'a[href]', + 'anchors => anchors.map(a => a.href)' + ) + + broken_links = [] + for href in links: + try: + response = await page.goto(href) + if response.status >= 400: + broken_links.append(href) + except: + broken_links.append(href) + + results['links'] = { + 'passed': len(broken_links) == 0, + 'broken_count': len(broken_links), + 'broken_links': broken_links[:10], + 'message': f'Found {len(broken_links)} broken links' if broken_links else 'All links working' + } + + # Performance + if 'performance' in checks: + print('Running performance check...') + + start = time.time() + await page.goto(deployment_url, wait_until='networkidle') + load_time = (time.time() - start) * 1000 + + results['performance'] = { + 'passed': load_time < 5000, + 'load_time': load_time, + 'message': f'Page loaded in {load_time:.0f}ms' + } + + await browser.close() + + # Overall + all_passed = all(r['passed'] for r in results.values()) + + return { + 'passed': all_passed, + 'checks': results, + 'summary': 'All checks passed' if all_passed else 'Some checks failed' + } + +def compare_images(img1: bytes, img2: bytes) -> float: + # Implement with PIL/Pillow + return 0.0 if img1 == img2 else 1.0 +``` + + +### 2. Deploy QA App + +```bash +cd kernel-qa +kernel deploy index.ts +``` + +### 3. Create Vercel Webhook + +In your Next.js app, create a webhook handler: + +```typescript +// app/api/vercel-webhook/route.ts +import { NextRequest } from 'next/server'; +import { Kernel } from '@onkernel/sdk'; + +export async function POST(req: NextRequest) { + const event = await req.json(); + + // Handle deployment.ready event + if (event.type === 'deployment.ready') { + const deploymentUrl = event.payload.deployment.url; + const deploymentId = event.payload.deployment.id; + + console.log(`New deployment: ${deploymentUrl}`); + + // Invoke QA checks + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + + const invocation = await kernel.invocations.create({ + app_name: 'qa-checks', + action_name: 'run-checks', + payload: { + deploymentUrl: `https://${deploymentUrl}`, + baselineUrl: 'https://production-site.com', + checks: ['visual', 'links', 'performance'] + }, + async: true + }); + + // Poll for results + let result; + for (let i = 0; i < 60; i++) { + result = await kernel.invocations.retrieve(invocation.id); + + if (result.status === 'succeeded' || result.status === 'failed') { + break; + } + + await new Promise(resolve => setTimeout(resolve, 2000)); + } + + // Update Vercel deployment check + const checkPassed = result.output?.passed || false; + + await fetch(`https://api.vercel.com/v1/deployments/${deploymentId}/checks`, { + method: 'PATCH', + headers: { + 'Authorization': `Bearer ${process.env.VERCEL_TOKEN}`, + 'Content-Type': 'application/json' + }, + body: JSON.stringify({ + name: 'Kernel QA', + status: 'completed', + conclusion: checkPassed ? 'succeeded' : 'failed', + output: { + summary: result.output?.summary || 'QA checks complete', + text: JSON.stringify(result.output?.checks, null, 2) + } + }) + }); + } + + return new Response('OK', { status: 200 }); +} +``` + +### 4. Configure Vercel Webhook + +In Vercel dashboard: +1. Go to Project → Settings → Git → Deploy Hooks +2. Add webhook URL: `https://your-app.com/api/vercel-webhook` +3. Select events: `deployment.ready` + +## Built-In Check Types + +The native integration provides these checks out-of-the-box: + +### Visual Regression + +Compares screenshots pixel-by-pixel against baseline. + +**Configuration:** +- Baseline URL: Production or staging URL +- Threshold: % difference allowed (default: 0.5%) +- Full page: Capture entire page or just viewport + +### Broken Links + +Crawls all internal links and checks HTTP status. + +**Configuration:** +- Max depth: How many levels to crawl +- Ignore patterns: Skip certain URLs (e.g., `/admin/*`) +- External links: Check external links too + +### Auth Flows + +Tests login, signup, password reset flows. + +**Configuration:** +- Test credentials: Username/password to use +- Expected redirects: Where should user land after login +- Profile: Reuse saved auth state + +### Critical Paths + +Custom paths like checkout, form submission. + +**Configuration:** +- Selectors: Elements to click/fill +- Expected outcomes: Text/URL to verify +- Timeout: How long to wait + +### Accessibility + +WCAG compliance checks. + +**Configuration:** +- Level: A, AA, or AAA +- Ignore: Skip certain rules + +### Performance + +Lighthouse scores and load times. + +**Configuration:** +- Thresholds: Min scores for performance, accessibility, etc. +- Metrics: FCP, LCP, TTI + +## Example: Visual Regression Only + +Simplest setup - just check if homepage looks the same: + +```typescript +// Deploy this as Kernel app +import { App } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; + +const app = new App('visual-check'); + +app.action('check', async (ctx, payload: { newUrl: string; oldUrl: string }) => { + const kb = await ctx.kernel.browsers.create({ + invocation_id: ctx.invocation_id, + headless: true + }); + + const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); + const page = browser.contexts()[0].pages()[0]; + + // New deployment + await page.goto(payload.newUrl); + const newImg = await page.screenshot(); + + // Production + await page.goto(payload.oldUrl); + const oldImg = await page.screenshot(); + + await browser.close(); + + // Compare (simplified) + const same = Buffer.compare(newImg, oldImg) === 0; + + return { + passed: same, + message: same ? 'No visual changes' : 'Visual changes detected' + }; +}); + +export default app; +``` + +Invoke from Vercel webhook or GitHub Actions. + +## Best Practices + +### 1. Test Critical Paths Only + +Don't test everything - focus on: +- Homepage +- Auth flows +- Checkout/payment +- Key user journeys + +### 2. Use Baselines Wisely + +Compare preview against: +- ✓ Production (catch regressions) +- ✓ Previous preview (catch drift) +- ✗ Localhost (too many differences) + +### 3. Set Reasonable Thresholds + +Visual diff threshold: +- 0%: Too strict (fonts, timestamps vary) +- 0.5-1%: Good for most cases +- 5%: Loose (allows significant changes) + +### 4. Ignore Dynamic Content + +Mask or ignore elements that always change: +```typescript +// Hide timestamps before screenshot +await page.evaluate(() => { + document.querySelectorAll('.timestamp, .date').forEach(el => { + el.textContent = 'MASKED'; + }); +}); + +const screenshot = await page.screenshot(); +``` + +### 5. Use Parallel Checks + +Run multiple checks in parallel: +```typescript +const [visual, links, performance] = await Promise.all([ + checkVisual(page, deploymentUrl), + checkLinks(page, deploymentUrl), + checkPerformance(page, deploymentUrl) +]); +``` + +## Troubleshooting + +### Flaky Visual Tests + +If visual tests fail intermittently: + +1. **Wait for fonts:** +```typescript +await page.evaluate(() => document.fonts.ready); +``` + +2. **Wait for animations:** +```typescript +await page.evaluate(() => { + document.querySelectorAll('*').forEach(el => { + el.style.animation = 'none'; + el.style.transition = 'none'; + }); +}); +``` + +3. **Fixed viewport:** +```typescript +await page.setViewportSize({ width: 1920, height: 1080 }); +``` + +### Deployment Checks Not Appearing + +If checks don't show in Vercel: + +1. Check webhook is triggered (Vercel logs) +2. Verify `VERCEL_TOKEN` has correct permissions +3. Check deployment ID matches + +### Tests Time Out + +If QA takes too long: + +1. Reduce scope (fewer pages) +2. Use headless mode +3. Block images/fonts +4. Run checks in parallel + +## Cost Estimation + +**Per deployment:** +- Visual check: ~10s = $0.008 +- Broken links (50 links): ~30s = $0.025 +- Performance check: ~15s = $0.0125 +- **Total: ~$0.045 per deployment** + +**100 deployments/month:** ~$4.50 + +## Related Recipes + +- [Screenshot + LLM](/recipes/screenshot-dom-llm) - AI-powered content checks +- [Parallel Browsers](/recipes/parallel-browsers) - Test multiple pages faster +- [Auth & Cookies](/recipes/auth-cookies-sessions) - Test authenticated flows + +## Related Features + +- [Vercel Integration](/integrations/vercel) - Setup guide +- [Replays](/browsers/replays) - Debug failed tests +- [App Platform](/apps/develop) - Deploy QA apps + +## Support + +Questions about QA automation? Join our [Discord](https://discord.gg/FBrveQRcud). + diff --git a/recipes/screenshot-dom-llm.mdx b/recipes/screenshot-dom-llm.mdx new file mode 100644 index 0000000..9e9f6c7 --- /dev/null +++ b/recipes/screenshot-dom-llm.mdx @@ -0,0 +1,418 @@ +--- +title: "Screenshot + DOM + LLM Summary" +sidebarTitle: "AI Page Analysis" +description: "Extract visual and semantic content from any webpage, then summarize with an LLM. Complete recipe for AI-powered web content analysis." +--- + +Combine screenshots, DOM extraction, and LLM analysis to understand webpages at scale. Useful for content monitoring, competitor analysis, and automated research. + +## What This Recipe Does + +1. Navigate to a webpage with Kernel browser +2. Capture a screenshot (visual content) +3. Extract DOM/text content (semantic content) +4. Send both to an LLM for analysis +5. Get structured summary or insights + +## Use Cases + +- Monitor competitor product pages for changes +- Summarize articles or documentation +- Extract structured data from unstructured pages +- Analyze landing page messaging +- QA content quality across deployments + +## Complete Code + + +```typescript TypeScript/Next.js +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; +import Anthropic from '@anthropic-ai/sdk'; + +export async function analyzePage(url: string) { + // Create Kernel browser + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY! }); + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + + // Navigate and wait for content + await page.goto(url, { waitUntil: 'networkidle' }); + + // Capture screenshot + const screenshot = await page.screenshot({ + type: 'png', + fullPage: true + }); + + // Extract text content + const textContent = await page.evaluate(() => { + // Remove scripts, styles + const clone = document.body.cloneNode(true) as HTMLElement; + clone.querySelectorAll('script, style, nav, footer').forEach(el => el.remove()); + return clone.innerText; + }); + + // Get title and meta + const title = await page.title(); + const description = await page.$eval( + 'meta[name="description"]', + el => el.getAttribute('content') + ).catch(() => null); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + // Analyze with Claude + const anthropic = new Anthropic({ + apiKey: process.env.ANTHROPIC_API_KEY! + }); + + const response = await anthropic.messages.create({ + model: 'claude-sonnet-4-20250514', + max_tokens: 1024, + messages: [{ + role: 'user', + content: [ + { + type: 'image', + source: { + type: 'base64', + media_type: 'image/png', + data: screenshot.toString('base64') + } + }, + { + type: 'text', + text: `Analyze this webpage and provide: +1. Main topic/purpose +2. Key messages or value propositions +3. Call-to-action (if any) +4. Target audience +5. Overall tone/style + +Context: +Title: ${title} +Description: ${description || 'N/A'} +Text content (first 2000 chars): ${textContent.slice(0, 2000)}...` + } + ] + }] + }); + + return { + url, + title, + description, + screenshot: screenshot.toString('base64'), + analysis: response.content[0].type === 'text' + ? response.content[0].text + : null + }; +} + +// Usage in Next.js API route +export default async function handler(req, res) { + const { url } = req.body; + const result = await analyzePage(url); + res.json(result); +} +``` + +```python Python +import base64 +from playwright.async_api import async_playwright +from kernel import Kernel +from anthropic import Anthropic + +async def analyze_page(url: str): + # Create Kernel browser + kernel = Kernel() + kb = kernel.browsers.create(headless=True) + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + # Navigate and wait + await page.goto(url, wait_until='networkidle') + + # Capture screenshot + screenshot = await page.screenshot(type='png', full_page=True) + + # Extract text content + text_content = await page.evaluate('''() => { + const clone = document.body.cloneNode(true); + clone.querySelectorAll('script, style, nav, footer').forEach(el => el.remove()); + return clone.innerText; + }''') + + # Get metadata + title = await page.title() + try: + description = await page.get_attribute('meta[name="description"]', 'content') + except: + description = None + + await browser.close() + kernel.browsers.delete_by_id(kb.session_id) + + # Analyze with Claude + anthropic = Anthropic(api_key=os.getenv('ANTHROPIC_API_KEY')) + + response = anthropic.messages.create( + model='claude-sonnet-4-20250514', + max_tokens=1024, + messages=[{ + 'role': 'user', + 'content': [ + { + 'type': 'image', + 'source': { + 'type': 'base64', + 'media_type': 'image/png', + 'data': base64.b64encode(screenshot).decode('utf-8') + } + }, + { + 'type': 'text', + 'text': f'''Analyze this webpage and provide: +1. Main topic/purpose +2. Key messages or value propositions +3. Call-to-action (if any) +4. Target audience +5. Overall tone/style + +Context: +Title: {title} +Description: {description or 'N/A'} +Text content (first 2000 chars): {text_content[:2000]}...''' + } + ] + }] + ) + + return { + 'url': url, + 'title': title, + 'description': description, + 'screenshot': base64.b64encode(screenshot).decode('utf-8'), + 'analysis': response.content[0].text if response.content[0].type == 'text' else None + } + +# Usage +result = await analyze_page('https://example.com') +print(result['analysis']) +``` + + +## Environment Variables + +```bash +KERNEL_API_KEY=your_kernel_api_key +ANTHROPIC_API_KEY=your_anthropic_api_key +# or OPENAI_API_KEY for GPT-4o +``` + +## Expected Output + +```json +{ + "url": "https://example.com", + "title": "Example Domain", + "description": "Example meta description", + "screenshot": "base64_encoded_image...", + "analysis": "This webpage is a simple example domain...\n\n1. Main topic: Domain placeholder\n2. Key messages: Demonstrates a basic webpage\n3. CTA: Links to 'More information'\n4. Target audience: Web developers, domain researchers\n5. Tone: Neutral, informative" +} +``` + +## Variations + +### Use OpenAI GPT-4o Instead + +```typescript +import OpenAI from 'openai'; + +const openai = new OpenAI({ + apiKey: process.env.OPENAI_API_KEY! +}); + +const response = await openai.chat.completions.create({ + model: 'gpt-4o', + messages: [{ + role: 'user', + content: [ + { + type: 'image_url', + image_url: { + url: `data:image/png;base64,${screenshot.toString('base64')}` + } + }, + { + type: 'text', + text: 'Analyze this webpage...' + } + ] + }] +}); +``` + +### Extract Specific Information + +```typescript +const response = await anthropic.messages.create({ + model: 'claude-sonnet-4-20250514', + max_tokens: 1024, + messages: [{ + role: 'user', + content: [ + { type: 'image', source: { ... } }, + { + type: 'text', + text: `Extract pricing information from this page as JSON: +{ + "plans": [ + {"name": "...", "price": "...", "features": [...]} + ] +}` + } + ] + }] +}); +``` + +### Compare Two Pages + +```typescript +// Capture both pages +const [page1Data, page2Data] = await Promise.all([ + analyzePage('https://example.com/old'), + analyzePage('https://example.com/new') +]); + +// Compare with LLM +const response = await anthropic.messages.create({ + model: 'claude-sonnet-4-20250514', + messages: [{ + role: 'user', + content: [ + { type: 'image', source: { type: 'base64', media_type: 'image/png', data: page1Data.screenshot } }, + { type: 'image', source: { type: 'base64', media_type: 'image/png', data: page2Data.screenshot } }, + { type: 'text', text: 'Compare these two pages. What changed?' } + ] + }] +}); +``` + +## Performance Optimization + +### Block Unnecessary Resources + +```typescript +// Before page.goto +await page.route('**/*', route => { + const type = route.request().resourceType(); + if (['image', 'font', 'stylesheet'].includes(type)) { + return route.abort(); + } + return route.continue(); +}); + +// Page loads faster, smaller screenshot +``` + +### Use Persistent Session for Batch Analysis + +```typescript +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: 'analyzer-session', + headless: true +}); + +// Reuse browser for multiple pages +for (const url of urls) { + const page = browser.contexts()[0].pages()[0]; + await page.goto(url); + // ... analyze ... + await page.goto('about:blank'); // Clear page +} +``` + +## Common Issues + +### Screenshot Too Large for LLM + +Most LLMs have image size limits (e.g., 20MB for Claude). Reduce screenshot size: + +```typescript +const screenshot = await page.screenshot({ + type: 'jpeg', // JPEG compresses better than PNG + quality: 80, + fullPage: false // Just viewport, not full page +}); +``` + +### LLM Missing Important Content + +If content is below the fold or in tabs/dropdowns: + +```typescript +// Expand all sections +await page.evaluate(() => { + document.querySelectorAll('details').forEach(el => el.setAttribute('open', '')); + document.querySelectorAll('[aria-expanded="false"]').forEach(el => el.click()); +}); + +// Scroll to load lazy content +await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight)); +await page.waitForTimeout(1000); + +// Then capture screenshot +``` + +### Rate Limits + +For batch analysis, add rate limiting: + +```typescript +import pLimit from 'p-limit'; + +const limit = pLimit(5); // Max 5 concurrent analyses + +const results = await Promise.all( + urls.map(url => limit(() => analyzePage(url))) +); +``` + +## Cost Estimation + +**Per page:** +- Kernel browser: ~$0.01 (2s @ $0.05/min) +- Claude API: ~$0.05 (1k tokens output + image) +- **Total: ~$0.06/page** + +**1,000 pages:** ~$60 + +## Related Recipes + +- [Block Ads/Trackers](/recipes/block-ads-trackers) - Speed up page loads +- [Parallel Browsers](/recipes/parallel-browsers) - Analyze multiple pages faster +- [Auth & Cookies](/recipes/auth-cookies-sessions) - Analyze logged-in content + +## Related Features + +- [Create a Browser](/browsers/create-a-browser) +- [Headless Mode](/browsers/headless) +- [Persistence](/browsers/persistence) +- [Network Interception](/troubleshooting/network-interception) + +## Support + +Questions? Join our [Discord](https://discord.gg/FBrveQRcud) to discuss AI + browser automation patterns. + diff --git a/troubleshooting/headless-chrome-serverless.mdx b/troubleshooting/headless-chrome-serverless.mdx new file mode 100644 index 0000000..157f0b9 --- /dev/null +++ b/troubleshooting/headless-chrome-serverless.mdx @@ -0,0 +1,331 @@ +--- +title: "Headless Chrome on Vercel: What Works in 2025" +sidebarTitle: "Serverless Chrome" +description: "Complete guide to running headless Chrome on Vercel and other serverless platforms. Learn what's possible, what isn't, and how to build production-ready automations." +--- + +**Serverless platforms like Vercel cannot run bundled Chromium binaries due to filesystem, size, and timeout constraints.** The solution: connect to remote browsers via CDP while keeping your application code on serverless. + +## What You're Trying to Do + +You want to run browser automation (screenshots, scraping, testing) from your Vercel deployment. Common use cases: + +- Generate og:image screenshots for dynamic URLs +- Scrape competitor pricing or content +- Run E2E tests on preview deployments +- Export reports from SaaS tools +- Capture receipts or invoices + +## Why Bundled Chrome Doesn't Work + +### Vercel's Constraints + +| Constraint | Limit | Why It Breaks Chrome | +|------------|-------|---------------------| +| **Filesystem** | Read-only, ephemeral | Chrome needs to write temp files, cache, profiles | +| **Binary Size** | 50MB limit | Chromium binaries are ~300MB uncompressed | +| **Cold Start** | <10s (Hobby), <60s (Pro) | Extracting and starting Chrome takes 10-30s | +| **Memory** | 1GB (Hobby), 3GB (Pro) | Chromium + Node.js can exceed limits | + +### What Happens When You Try + +```bash +# With playwright or puppeteer in package.json +npm run build +# ✗ Error: Chromium binaries exceed deployment size limit + +# Or at runtime +browserType.launch() +# ✗ Error: Executable doesn't exist at /var/task/... +``` + +## What Works: Remote Browsers via CDP + +The Chrome DevTools Protocol (CDP) lets you control a browser over WebSocket. Your code runs on Vercel; the browser runs elsewhere. + +### Architecture + +``` +┌─────────────────┐ WebSocket ┌──────────────────┐ +│ Vercel Function│ ◄────────────────────────► │ Kernel Browser │ +│ (Your Code) │ chromium.connectOverCDP │ (Cloud-hosted) │ +└─────────────────┘ └──────────────────┘ +``` + +**Benefits:** + +- No binaries to deploy +- Instant cold starts (browser pool is pre-warmed) +- Unlimited concurrency (scale browsers independently) +- Persistent sessions (reuse auth across requests) + +### Implementation + + +```typescript Playwright +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +export default async function handler(req, res) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto(req.query.url); + const screenshot = await page.screenshot({ type: 'png' }); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + res.setHeader('Content-Type', 'image/png'); + return res.send(screenshot); +} +``` + +```typescript Puppeteer +import puppeteer from 'puppeteer-core'; +import { Kernel } from '@onkernel/sdk'; + +export default async function handler(req, res) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await puppeteer.connect({ + browserWSEndpoint: kb.cdp_ws_url + }); + + const page = await browser.newPage(); + await page.goto(req.query.url); + const screenshot = await page.screenshot({ type: 'png' }); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + res.setHeader('Content-Type', 'image/png'); + return res.send(screenshot); +} +``` + + +## Feature Comparison: What You Can Do + +| Feature | Bundled Chrome | Remote Browser (Kernel) | +|---------|----------------|------------------------| +| Screenshots | ✗ Won't deploy | ✓ Works | +| DOM Scraping | ✗ Won't deploy | ✓ Works | +| Network Interception | ✗ Won't deploy | ✓ Works (page.route) | +| File Downloads | ✗ Won't deploy | ✓ Works (File I/O API) | +| Persistent Sessions | ✗ Ephemeral only | ✓ Hours/days with standby | +| Stealth/Proxies | ✗ Won't deploy | ✓ Built-in | +| Live View | ✗ Not possible | ✓ Human-in-the-loop | +| Video Replays | ✗ Not possible | ✓ Full session recording | +| Concurrent Sessions | Limited by memory | ✓ Unlimited | + +## Other Serverless Platforms + +The same constraints apply to: + +### AWS Lambda + +- 512MB deployment size limit +- 15-minute max execution +- 10GB memory max +- **Solution:** Same—use CDP to remote browsers + +### Cloudflare Workers + +- 1MB script size after compression +- 10ms CPU time (paid), 50ms (free) +- No filesystem +- **Solution:** CDP via Cloudflare's Browser Rendering API or Kernel + +### Railway, Fly.io + +- More flexible than Vercel, but: +- Still need to manage Chrome lifecycle +- Cold starts with Docker images +- **Solution:** Use Kernel for faster cold starts + managed infrastructure + +### Netlify Functions + +- Similar to Vercel constraints +- 50MB deployment limit +- **Solution:** CDP to remote browsers + +## Cost Comparison + +Let's compare 1,000 screenshot requests: + +### Self-Hosting (Cloud Run, Fly.io) + +``` +Container always-on: $30/month minimum ++ Chromium memory: 2GB RAM = $15/month ++ Maintenance time: 4 hours/month @ $100/hr = $400/month += $445/month +``` + +### Kernel (per-minute pricing) + +``` +1,000 requests @ 3 seconds each = 50 minutes +50 minutes @ $0.10/min = $5/month +``` + +**90% cost savings** for typical workloads. + +## Setup Guide + +### 1. Install Dependencies + +```bash +# Use -core versions (no browser binaries) +npm install playwright-core @onkernel/sdk +# or +npm install puppeteer-core @onkernel/sdk +``` + +### 2. Get Kernel API Key + +1. Sign up at [dashboard.onkernel.com](https://dashboard.onkernel.com/sign-up) +2. Go to Settings → API Keys +3. Create new key + +### 3. Add to Vercel + +```bash +vercel env add KERNEL_API_KEY +# Paste your key +# Select: Production, Preview, Development +``` + +### 4. Use in API Routes + +See code samples above. Place in: + +- `pages/api/screenshot.ts` (Pages Router) +- `app/api/screenshot/route.ts` (App Router) + +### 5. Deploy + +```bash +vercel deploy +``` + +No build errors. No runtime errors. Just works. + +## Advanced: Vercel Native Integration + +For automatic setup and QA deployment checks: + +1. Install [Kernel from Vercel Marketplace](https://vercel.com/integrations/kernel) +2. Connect to your projects +3. API key auto-provisioned +4. QA agents run on every preview deployment + +See [Vercel Integration Guide](/integrations/vercel) for details. + +## Best Practices + +### Reuse Sessions for Auth + +Don't log in on every request. Use [persistent sessions](/browsers/persistence): + +```typescript +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: `user-${userId}` +}); + +// First request: logs in, Kernel saves state +// Subsequent requests: reuses cookies/session +``` + +### Handle Timeouts + +Vercel functions timeout after 10s (Hobby) or 60s (Pro). For long tasks: + +1. Return early, process async +2. Use [Kernel App Platform](/apps/develop) (no timeouts) +3. Split into multiple requests + +### Block Unnecessary Resources + +Speed up by blocking images, fonts, analytics: + +```typescript +await page.route('**/*', route => { + const type = route.request().resourceType(); + if (['image', 'font', 'stylesheet'].includes(type)) { + return route.abort(); + } + return route.continue(); +}); +``` + +See [Network Interception guide](/troubleshooting/network-interception). + +### Use Headless Mode + +For non-interactive tasks (screenshots, scraping): + +```typescript +const kb = await kernel.browsers.create({ headless: true }); +``` + +Headless uses 1GB RAM vs 8GB for headful, starts faster, costs less. + +## FAQ + +### Is this slower than local Chrome? + +Network latency adds ~20-50ms. But: + +- No cold start (Kernel pre-warms browsers) +- No binary extraction (saves 10-30s) +- **Net result:** Usually faster for serverless + +### Can I use existing Playwright scripts? + +Yes. Replace `browser.launch()` with `chromium.connectOverCDP(cdpUrl)`. Everything else (selectors, actions, assertions) works identically. + +### What about CI/CD? + +For GitHub Actions, Vercel deployment checks, etc.: + +- Install Kernel's [native Vercel integration](/integrations/vercel) +- Or use [Kernel CLI](/reference/cli) in CI workflows +- Or deploy tests as [Kernel Apps](/apps/develop) + +### Does this work for testing? + +Yes. Playwright/Puppeteer's full test APIs work over CDP. Popular frameworks: + +- Playwright Test +- Jest + Puppeteer +- Cypress (via CDP adapter) + +### How do I debug? + +Kernel provides: + +- [Live View](/browsers/live-view): Watch browser in real-time +- [Replays](/browsers/replays): Video recordings of sessions +- Console logs forwarded to your app + +## Related Resources + +- [Fix Playwright Vercel Error](/troubleshooting/playwright-vercel-error) +- [Vercel Integration Guide](/integrations/vercel) +- [Network Interception](/troubleshooting/network-interception) +- [Playwright Timeouts](/troubleshooting/playwright-timeouts-serverless) +- [Create a Browser](/browsers/create-a-browser) + +## Still Stuck? + +Join our [Discord](https://discord.gg/FBrveQRcud) for help. Share your error message and we'll diagnose. + diff --git a/troubleshooting/network-interception.mdx b/troubleshooting/network-interception.mdx new file mode 100644 index 0000000..6a6d3ac --- /dev/null +++ b/troubleshooting/network-interception.mdx @@ -0,0 +1,457 @@ +--- +title: "Network Interception with Playwright via CDP" +sidebarTitle: "Network Interception" +description: "Complete guide to intercepting network requests with Playwright on hosted browsers. Block resources, modify requests, capture API responses, and rewrite headers." +--- + +**Network interception works fully with Kernel's hosted browsers.** Use Playwright's `page.route()` to block ads, modify requests, capture API responses, and control network traffic—even when your browser is remote. + +## What is Network Interception? + +Network interception lets you: + +- **Block resources:** Skip images, fonts, ads, analytics to speed up page loads +- **Modify requests:** Change headers, post data, or URLs before sending +- **Capture responses:** Extract API data, monitor backend calls +- **Mock responses:** Return fake data for testing without hitting real APIs + +## Does This Work with Hosted Browsers? + +**Yes.** Some hosted browser providers only support basic screenshot/DOM extraction. Kernel supports the full Chrome DevTools Protocol, including network interception via `page.route()`, `page.on('request')`, and `page.on('response')`. + +## Basic Example: Block Images + +Speed up page loads by blocking image requests: + + +```typescript Playwright +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); +const kb = await kernel.browsers.create(); +const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); + +const page = browser.contexts()[0].pages()[0]; + +// Block all image requests +await page.route('**/*', route => { + if (route.request().resourceType() === 'image') { + return route.abort(); + } + return route.continue(); +}); + +await page.goto('https://example.com'); +// Page loads without images, ~50% faster +``` + +```python Python +from playwright.async_api import async_playwright +from kernel import Kernel + +kernel = Kernel() +kb = kernel.browsers.create() + +async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + page = browser.contexts[0].pages[0] + + # Block all image requests + async def handle_route(route): + if route.request.resource_type == 'image': + await route.abort() + else: + await route.continue_() + + await page.route('**/*', handle_route) + await page.goto('https://example.com') +``` + + +## Block Multiple Resource Types + +Block images, fonts, stylesheets, and analytics: + +```typescript +const BLOCKED_RESOURCES = ['image', 'font', 'stylesheet', 'media']; +const BLOCKED_DOMAINS = [ + 'googletagmanager.com', + 'google-analytics.com', + 'facebook.net', + 'doubleclick.net' +]; + +await page.route('**/*', route => { + const request = route.request(); + const url = request.url(); + const type = request.resourceType(); + + // Block by resource type + if (BLOCKED_RESOURCES.includes(type)) { + return route.abort(); + } + + // Block by domain + if (BLOCKED_DOMAINS.some(domain => url.includes(domain))) { + return route.abort(); + } + + return route.continue(); +}); + +await page.goto('https://example.com'); +// Loads only HTML, JS, and first-party requests +``` + +## Modify Request Headers + +Add custom headers or override user-agent: + +```typescript +await page.route('**/*', route => { + const headers = { + ...route.request().headers(), + 'Authorization': 'Bearer YOUR_TOKEN', + 'X-Custom-Header': 'custom-value', + 'User-Agent': 'Mozilla/5.0 (Custom Bot)' + }; + + return route.continue({ headers }); +}); + +await page.goto('https://api.example.com'); +// All requests include custom headers +``` + +## Capture API Responses + +Extract API data without parsing HTML: + +```typescript +const apiData = []; + +page.on('response', async response => { + const url = response.url(); + + // Capture specific API endpoint + if (url.includes('/api/products')) { + try { + const json = await response.json(); + apiData.push(json); + console.log('Captured API response:', json); + } catch (e) { + // Not JSON, ignore + } + } +}); + +await page.goto('https://example.com/products'); +await page.waitForTimeout(2000); // Wait for API calls + +console.log('All API data:', apiData); +``` + +## Mock API Responses + +Return fake data for testing: + +```typescript +await page.route('**/api/user', route => { + if (route.request().method() === 'GET') { + return route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + id: 123, + name: 'Test User', + email: 'test@example.com' + }) + }); + } + return route.continue(); +}); + +await page.goto('https://example.com/dashboard'); +// App receives mocked user data +``` + +## Modify POST Data + +Change form submissions or API payloads: + +```typescript +await page.route('**/api/submit', route => { + if (route.request().method() === 'POST') { + const postData = route.request().postDataJSON(); + + // Modify the payload + const modifiedData = { + ...postData, + extra_field: 'added_by_script', + timestamp: Date.now() + }; + + return route.continue({ + postData: JSON.stringify(modifiedData), + headers: { + ...route.request().headers(), + 'Content-Type': 'application/json' + } + }); + } + return route.continue(); +}); + +await page.click('button[type="submit"]'); +// Form submits with modified data +``` + +## Redirect Requests + +Change URLs before they load: + +```typescript +await page.route('**/*', route => { + const url = route.request().url(); + + // Redirect old domain to new domain + if (url.includes('old-domain.com')) { + const newUrl = url.replace('old-domain.com', 'new-domain.com'); + return route.continue({ url: newUrl }); + } + + return route.continue(); +}); +``` + +## Monitor Network Activity + +Log all requests and responses: + +```typescript +const networkLog = []; + +page.on('request', request => { + networkLog.push({ + type: 'request', + method: request.method(), + url: request.url(), + headers: request.headers(), + timestamp: Date.now() + }); +}); + +page.on('response', response => { + networkLog.push({ + type: 'response', + status: response.status(), + url: response.url(), + headers: response.headers(), + timestamp: Date.now() + }); +}); + +await page.goto('https://example.com'); +console.log('Network activity:', networkLog); +``` + +## Advanced: Conditional Blocking + +Block resources based on file size or timing: + +```typescript +await page.route('**/*', async route => { + const request = route.request(); + + // Fetch to check size without loading into page + const response = await route.fetch(); + const headers = response.headers(); + const contentLength = parseInt(headers['content-length'] || '0'); + + // Block files larger than 1MB + if (contentLength > 1024 * 1024) { + console.log(`Blocked large file: ${request.url()} (${contentLength} bytes)`); + return route.abort(); + } + + // Otherwise, fulfill with the fetched response + return route.fulfill({ response }); +}); +``` + +## Use with Stealth Mode + +Combine network interception with [stealth mode](/browsers/stealth): + +```typescript +const kb = await kernel.browsers.create({ stealth: true }); +const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); +const page = browser.contexts()[0].pages()[0]; + +// Block bot-detection analytics +await page.route('**/*', route => { + const url = route.request().url(); + const botDetectors = ['datadome.co', 'px-cloud.net', 'perimeterx.net']; + + if (botDetectors.some(d => url.includes(d))) { + return route.abort(); + } + return route.continue(); +}); + +await page.goto('https://protected-site.com'); +// Stealth mode + blocked trackers = harder to detect +``` + +## Performance Tips + +### 1. Use Specific URL Patterns + +Instead of `**/*`, use specific patterns to reduce overhead: + +```typescript +// Only intercept API calls +await page.route('**/api/**', route => { /* ... */ }); + +// Only intercept specific domain +await page.route('https://cdn.example.com/**', route => { /* ... */ }); +``` + +### 2. Unroute After Setup + +Remove route handlers when no longer needed: + +```typescript +const handler = route => { /* ... */ }; +await page.route('**/*', handler); + +// ... do work ... + +await page.unroute('**/*', handler); +// Faster for subsequent navigations +``` + +### 3. Batch Modifications + +Modify multiple requests with one handler instead of multiple route calls. + +## Common Patterns + +### Extract All API Data + +```typescript +const apiResponses = new Map(); + +page.on('response', async response => { + if (response.url().includes('/api/')) { + try { + const json = await response.json(); + apiResponses.set(response.url(), json); + } catch {} + } +}); + +await page.goto('https://example.com'); +// Wait for page to settle +await page.waitForLoadState('networkidle'); + +console.log('API data:', Object.fromEntries(apiResponses)); +``` + +### Block All External Domains + +Only load resources from the main domain: + +```typescript +const mainDomain = new URL(page.url()).hostname; + +await page.route('**/*', route => { + const requestDomain = new URL(route.request().url()).hostname; + + if (requestDomain !== mainDomain) { + return route.abort(); + } + return route.continue(); +}); +``` + +### Add Authentication to All Requests + +```typescript +const token = 'your-auth-token'; + +await page.route('**/*', route => { + const url = route.request().url(); + + // Only add auth to API calls + if (url.includes('/api/')) { + return route.continue({ + headers: { + ...route.request().headers(), + 'Authorization': `Bearer ${token}` + } + }); + } + + return route.continue(); +}); +``` + +## Troubleshooting + +### Route Handler Not Called + +Make sure to call `route.continue()`, `route.abort()`, or `route.fulfill()` in every route handler. If you forget, the request hangs. + +```typescript +// ✗ BAD: No route action +await page.route('**/*', route => { + console.log(route.request().url()); + // Request hangs! +}); + +// ✓ GOOD: Always call an action +await page.route('**/*', route => { + console.log(route.request().url()); + return route.continue(); +}); +``` + +### CORS Errors + +If modifying requests causes CORS errors, you may need to also modify CORS headers in responses: + +```typescript +await page.route('**/*', async route => { + const response = await route.fetch(); + const headers = { + ...response.headers(), + 'Access-Control-Allow-Origin': '*', + 'Access-Control-Allow-Methods': '*', + 'Access-Control-Allow-Headers': '*' + }; + + return route.fulfill({ response, headers }); +}); +``` + +### Timeout Errors + +Network interception adds latency. If requests timeout: + +1. Increase timeout: `page.setDefaultTimeout(60000)` +2. Use more specific route patterns (don't intercept everything) +3. Avoid slow operations in route handlers + +## Related Resources + +- [Create a Browser](/browsers/create-a-browser) +- [Stealth Mode](/browsers/stealth) +- [Block Ads/Trackers Recipe](/recipes/block-ads-trackers) +- [Vercel Integration](/integrations/vercel) + +## Need Help? + +Join our [Discord](https://discord.gg/FBrveQRcud) for support with network interception patterns. + diff --git a/troubleshooting/playwright-lambda-limits.mdx b/troubleshooting/playwright-lambda-limits.mdx new file mode 100644 index 0000000..93931d8 --- /dev/null +++ b/troubleshooting/playwright-lambda-limits.mdx @@ -0,0 +1,419 @@ +--- +title: "Playwright on AWS Lambda: Constraints and Solutions" +sidebarTitle: "AWS Lambda" +description: "Run Playwright on AWS Lambda despite binary size limits, memory constraints, and cold start issues. Learn packaging strategies and when to use remote browsers." +--- + +**AWS Lambda's 512MB deployment size limit prevents bundling Chromium binaries directly.** Solutions include Lambda Layers (up to 250MB), Docker images (10GB), or connecting to remote browsers via CDP. + +## The Challenge + +Playwright's Chromium binaries are ~300MB uncompressed, ~150MB compressed. AWS Lambda constraints: + +| Constraint | Limit | Impact on Playwright | +|------------|-------|---------------------| +| **Deployment package** | 50MB (zipped), 250MB (unzipped) | Chromium doesn't fit | +| **Lambda Layer** | 50MB (zipped), 250MB (unzipped) | Chromium barely fits (if optimized) | +| **Docker image** | 10GB total | Chromium fits, but cold start slow | +| **Memory** | 128MB - 10GB | Chromium needs 1-2GB minimum | +| **Timeout** | 15 minutes max | Usually sufficient | +| **Filesystem** | /tmp only, 512MB | Chrome writes temp files here | + +## Solution 1: Lambda Layers (Limited) + +Package Chromium as a Lambda Layer. This works but is fragile: + +### Step 1: Build Layer + +```bash +# In a Docker container matching Lambda runtime (Amazon Linux 2) +docker run -v "$PWD":/build -it public.ecr.aws/lambda/nodejs:20 bash + +# Inside container +cd /build +npm init -y +npm install playwright-core +npx playwright install chromium --with-deps + +# Package layer +mkdir -p layer/nodejs/node_modules +cp -r node_modules/playwright-core layer/nodejs/node_modules/ +mkdir -p layer/chromium +cp -r /root/.cache/ms-playwright layer/chromium/ + +cd layer +zip -r ../layer.zip . +``` + +### Step 2: Deploy Layer + +```bash +aws lambda publish-layer-version \ + --layer-name chromium-layer \ + --zip-file fileb://layer.zip \ + --compatible-runtimes nodejs20.x +``` + +### Step 3: Use in Lambda + +```typescript +import { chromium } from 'playwright-core'; + +export const handler = async (event) => { + const browser = await chromium.launch({ + executablePath: '/opt/chromium/ms-playwright/chromium-*/chrome-linux/chrome', + headless: true, + args: [ + '--no-sandbox', + '--disable-setuid-sandbox', + '--single-process' // Required for Lambda + ] + }); + + const page = await browser.newPage(); + await page.goto(event.url); + const title = await page.title(); + + await browser.close(); + + return { title }; +}; +``` + +**Limitations:** + +- Fragile: Chrome updates break frequently +- Single-process mode: Less stable, can crash +- Cold start: 5-10s to extract and launch +- Maintenance: Must rebuild layer for each Playwright update + +## Solution 2: Docker Images (Better) + +Lambda supports Docker images up to 10GB. This is more reliable: + +### Dockerfile + +```dockerfile +FROM public.ecr.aws/lambda/nodejs:20 + +# Install Chromium dependencies +RUN yum install -y \ + atk cups-libs gtk3 libXcomposite alsa-lib \ + libXcursor libXdamage libXext libXi libXrandr libXScrnSaver \ + libXtst pango at-spi2-atk libXt xorg-x11-server-Xvfb \ + xorg-x11-xauth dbus-glib dbus-glib-devel nss mesa-libgbm + +# Copy package files +COPY package*.json ./ +RUN npm ci --omit=dev + +# Install Playwright and Chromium +RUN npx playwright install chromium --with-deps + +# Copy Lambda function +COPY index.js ./ + +CMD ["index.handler"] +``` + +### Deploy + +```bash +# Build and push to ECR +docker build -t playwright-lambda . +aws ecr create-repository --repository-name playwright-lambda +docker tag playwright-lambda:latest .dkr.ecr..amazonaws.com/playwright-lambda:latest +docker push .dkr.ecr..amazonaws.com/playwright-lambda:latest + +# Create Lambda function +aws lambda create-function \ + --function-name playwright-scraper \ + --package-type Image \ + --code ImageUri=.dkr.ecr..amazonaws.com/playwright-lambda:latest \ + --role arn:aws:iam:::role/lambda-execution-role \ + --memory-size 2048 \ + --timeout 300 +``` + +**Pros:** + +- Reliable: Full Chromium with all features +- Version control: Pin specific Chromium version +- Familiar: Standard Docker workflow + +**Cons:** + +- Cold start: 10-30s to pull and extract image +- Maintenance: Must rebuild image for updates +- Cost: ECR storage + Lambda memory (2GB+ needed) + +## Solution 3: Remote Browsers via CDP (Recommended) + +Connect to a hosted browser over WebSocket. No binaries to package: + +### Lambda Function + +```typescript +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +export const handler = async (event) => { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create({ headless: true }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto(event.url); + const title = await page.title(); + const html = await page.content(); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { title, html }; +}; +``` + +### Package + +```bash +npm install playwright-core @onkernel/sdk +zip -r function.zip index.js node_modules +``` + +**Package size:** <10MB (no Chromium binaries) + +**Pros:** + +- Fast cold start: <1s (no binary extraction) +- Always up-to-date: Browser managed by Kernel +- No maintenance: No Docker images to rebuild +- Extra features: Live view, replays, stealth mode, persistent sessions +- Unlimited concurrency: Scale browsers independently from Lambda + +**Cons:** + +- Network latency: ~20-50ms per request +- External dependency: Requires Kernel API + +**Cost comparison (1,000 executions):** + +| Method | Lambda Cost | Chromium Cost | Total | +|--------|-------------|---------------|-------| +| **Docker image** | $3.50 (2GB × 10s) | Included | $3.50 | +| **Remote (Kernel)** | $0.35 (128MB × 2s) | $0.50 (5min @ $0.10/min) | $0.85 | + +**Remote browsers cost 75% less** for typical workloads. + +## Performance Optimization + +### For Lambda Layers/Docker + +```typescript +const browser = await chromium.launch({ + headless: true, + args: [ + '--no-sandbox', + '--disable-setuid-sandbox', + '--disable-dev-shm-usage', // Use /tmp instead of /dev/shm + '--disable-gpu', + '--single-process', // Required for low memory + '--no-zygote', // Also required for single-process + '--disable-extensions', + '--disable-background-networking', + '--disable-default-apps' + ] +}); +``` + +### For Remote Browsers + +```typescript +// Use headless for speed +const kb = await kernel.browsers.create({ headless: true }); + +// Block unnecessary resources +await page.route('**/*', route => { + if (['image', 'font', 'stylesheet'].includes(route.request().resourceType())) { + return route.abort(); + } + return route.continue(); +}); + +// Don't wait for full load +await page.goto(url, { waitUntil: 'domcontentloaded' }); +``` + +## Persistent Sessions + +For Lambda functions that run frequently, reuse browser sessions: + +```typescript +const BROWSER_SESSION_ID = 'lambda-persistent-browser'; + +export const handler = async (event) => { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + + // Try to reuse existing browser + let kb; + try { + const browsers = await kernel.browsers.list(); + kb = browsers.find(b => b.persistent_id === BROWSER_SESSION_ID); + } catch {} + + // Create if doesn't exist + if (!kb) { + kb = await kernel.browsers.create({ + persistent: true, + persistent_id: BROWSER_SESSION_ID, + headless: true + }); + } + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + // ... use browser ... + + await browser.close(); + // Don't delete - keeps session for next invocation + + return result; +}; +``` + +First invocation: Creates browser (~2s) +Subsequent invocations: Reuses browser (~0.1s) + +Browser goes to [standby mode](/browsers/standby) after 1 minute of inactivity (zero cost). + +## Step Functions for Long Tasks + +For multi-step workflows that exceed 15 minutes: + +```json +{ + "Comment": "Scrape multiple pages", + "StartAt": "ScrapePage1", + "States": { + "ScrapePage1": { + "Type": "Task", + "Resource": "arn:aws:lambda:region:account:function:playwright-scraper", + "Parameters": { + "url": "https://example.com/page1" + }, + "Next": "ScrapePage2" + }, + "ScrapePage2": { + "Type": "Task", + "Resource": "arn:aws:lambda:region:account:function:playwright-scraper", + "Parameters": { + "url": "https://example.com/page2" + }, + "Next": "Combine" + }, + "Combine": { + "Type": "Task", + "Resource": "arn:aws:lambda:region:account:function:combine-results", + "End": true + } + } +} +``` + +Each step has its own 15-minute timeout. + +## ECS/Fargate for Heavy Workloads + +If you need to run many concurrent browser sessions or very long tasks, use ECS Fargate instead of Lambda: + +**Pros:** + +- No timeout limits +- Control over resources (CPU, memory) +- Easier to manage Chrome lifecycle + +**Cons:** + +- More operational overhead +- Always-on cost (even if idle) +- Cold start slower than Lambda + remote browsers + +**Recommendation:** Use Lambda + Kernel for most workloads. Reserve ECS/Fargate for: + +- Batch processing 1000s of pages +- Tasks that always take >10 minutes +- Regulatory requirements (data can't leave your VPC) + +## Common Errors + +### Error: "spawn ENOMEM" + +Lambda ran out of memory. Increase memory size: + +```bash +aws lambda update-function-configuration \ + --function-name playwright-scraper \ + --memory-size 2048 # or higher +``` + +### Error: "/tmp" is full + +Chrome writes temp files to /tmp (512MB limit). Solutions: + +1. Use `--disable-dev-shm-usage` flag +2. Clean up temp files: `fs.rmSync('/tmp/*', { recursive: true })` +3. Use remote browsers (no /tmp usage) + +### Error: "Failed to launch browser" + +Check: + +1. `--no-sandbox --disable-setuid-sandbox` args present +2. Lambda has enough memory (2GB+) +3. Lambda timeout is sufficient (60s+) +4. Chromium binary path is correct + +## FAQ + +### Should I use Lambda Layers or Docker images? + +Docker images are more reliable. Lambda Layers work but break easily with Playwright updates. + +If you're starting new, use remote browsers instead—no packaging needed. + +### Can I run headful mode on Lambda? + +No. Lambda has no display server (no X11). You must use headless mode or remote headful browsers. + +Kernel provides headful browsers with [Live View](/browsers/live-view) if you need to watch the browser. + +### What about Fargate? + +Fargate is easier than ECS but more expensive than Lambda. Use if: + +- You need >15min timeout +- You're running continuous workloads +- You need control over Chrome lifecycle + +For sporadic workloads, Lambda + Kernel is cheaper. + +### Can I use Playwright's built-in Docker image? + +Playwright's official Docker image (`mcr.microsoft.com/playwright`) works on Fargate/ECS but not Lambda (wrong base image). For Lambda, build custom image as shown above. + +## Related Resources + +- [Playwright Vercel Error](/troubleshooting/playwright-vercel-error) +- [Headless Chrome on Serverless](/troubleshooting/headless-chrome-serverless) +- [Playwright Timeouts](/troubleshooting/playwright-timeouts-serverless) +- [Create a Browser](/browsers/create-a-browser) + +## Need Help? + +Join our [Discord](https://discord.gg/FBrveQRcud) for support with Lambda deployments. + diff --git a/troubleshooting/playwright-timeouts-serverless.mdx b/troubleshooting/playwright-timeouts-serverless.mdx new file mode 100644 index 0000000..9d5e803 --- /dev/null +++ b/troubleshooting/playwright-timeouts-serverless.mdx @@ -0,0 +1,470 @@ +--- +title: "Playwright Timeouts on Serverless: Options That Scale" +sidebarTitle: "Timeout Solutions" +description: "Handle Vercel, Lambda, and serverless timeout constraints when running Playwright automations. Learn async patterns, timeout tuning, and the Kernel App Platform." +--- + +**Serverless functions have strict time limits.** Vercel allows 10s (Hobby) or 60s (Pro). For Playwright automations that need more time, you have three options: optimize for speed, return early with async processing, or use Kernel's App Platform (no timeouts). + +## The Problem + +Your Playwright script works locally but times out on Vercel: + +``` +Error: Function execution duration exceeded +FUNCTION_INVOCATION_TIMEOUT +``` + +Or Playwright itself times out: + +``` +TimeoutError: page.goto: Timeout 30000ms exceeded +``` + +## Quick Solutions + +### 1. Increase Playwright Timeouts + +Playwright's default is 30s. Reduce it for serverless: + +```typescript +import { chromium } from 'playwright-core'; + +const browser = await chromium.connectOverCDP({ + wsEndpoint: kernelBrowser.cdp_ws_url, + timeout: 15000 // 15 seconds instead of 30 +}); + +// Or set per-page +page.setDefaultTimeout(15000); + +// Or per-action +await page.goto('https://example.com', { timeout: 10000 }); +await page.click('button', { timeout: 5000 }); +``` + +### 2. Use Headless Mode + +Headless browsers start ~2x faster and use less memory: + +```typescript +const kb = await kernel.browsers.create({ headless: true }); +``` + +Headless uses 1GB RAM vs 8GB for headful. No GUI rendering = faster page loads. + +### 3. Block Unnecessary Resources + +Speed up by 30-70% by blocking images, fonts, ads: + +```typescript +await page.route('**/*', route => { + const type = route.request().resourceType(); + if (['image', 'font', 'stylesheet', 'media'].includes(type)) { + return route.abort(); + } + return route.continue(); +}); + +await page.goto('https://example.com'); +// Loads 50%+ faster +``` + +See [Network Interception guide](/troubleshooting/network-interception). + +### 4. Wait Only for What You Need + +Don't wait for full page load if you only need specific elements: + +```typescript +// ✗ Slow: waits for everything +await page.goto('https://example.com', { waitUntil: 'networkidle' }); + +// ✓ Fast: returns as soon as DOM is ready +await page.goto('https://example.com', { waitUntil: 'domcontentloaded' }); + +// ✓ Fastest: wait only for specific element +await page.goto('https://example.com', { waitUntil: 'commit' }); +await page.waitForSelector('.product-list', { timeout: 5000 }); +``` + +### 5. Reuse Browser Sessions + +Don't create a new browser for every request. Use [persistent sessions](/browsers/persistence): + +```typescript +const kb = await kernel.browsers.create({ + persistent: true, + persistent_id: 'shared-browser' +}); + +// First request: browser created (~2s) +// Subsequent requests: reuses existing browser (~0.1s) +``` + +## Platform-Specific Limits + +| Platform | Hobby/Free | Paid | +|----------|-----------|------| +| **Vercel** | 10s | 60s | +| **Netlify** | 10s | 26s (background: 15min) | +| **AWS Lambda** | - | 15 minutes | +| **Cloudflare Workers** | 50ms CPU | 30s wall time | +| **Railway** | None | None | +| **Fly.io** | None | None | + +## Pattern: Return Early, Process Async + +For long-running tasks, return immediately and process in background: + +### Option A: Webhook Callback + +```typescript +// pages/api/scrape.ts +export default async function handler(req, res) { + const { url, callbackUrl } = req.body; + + // Return immediately + res.json({ status: 'processing', id: 'job-123' }); + + // Process async (doesn't block response) + scrapeAsync(url).then(result => { + // POST result to callback + fetch(callbackUrl, { + method: 'POST', + body: JSON.stringify(result) + }); + }); +} + +async function scrapeAsync(url) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create(); + const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto(url); + const data = await page.textContent('.content'); + + await browser.close(); + await kernel.browsers.deleteByID(kb.session_id); + + return { data }; +} +``` + +**Caveat:** If the function finishes before async work completes, Vercel may kill the process. For reliable async, use Option B or C. + +### Option B: Queue (Redis, RabbitMQ) + +```typescript +// pages/api/scrape.ts +import { Queue } from 'bull'; + +const scrapeQueue = new Queue('scrape', process.env.REDIS_URL); + +export default async function handler(req, res) { + const { url } = req.body; + + // Add to queue + const job = await scrapeQueue.add({ url }); + + // Return job ID + res.json({ jobId: job.id, status: 'queued' }); +} + +// Separate worker process (runs on Railway, Fly.io, or Vercel cron) +scrapeQueue.process(async job => { + const { url } = job.data; + // ... run Playwright automation ... + return result; +}); +``` + +Check job status at `GET /api/scrape/:jobId`. + +### Option C: Kernel App Platform + +Deploy your automation as a Kernel App—no timeout limits: + +```typescript +// kernel-app/index.ts +import { App } from '@onkernel/sdk'; +import { chromium } from 'playwright-core'; + +const app = new App('scraper'); + +app.action('scrape', async (ctx, payload) => { + const { url } = payload; + + // No timeout constraints + const kb = await ctx.kernel.browsers.create({ + invocation_id: ctx.invocation_id + }); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kb.cdp_ws_url + }); + + const page = browser.contexts()[0].pages()[0]; + await page.goto(url); + + // Take as long as needed + await page.waitForTimeout(60000); // 60 seconds? No problem + const data = await page.textContent('.content'); + + await browser.close(); + + return { data }; +}); + +export default app; +``` + +Deploy and invoke: + +```bash +kernel deploy index.ts +kernel invoke scraper scrape --payload '{"url": "https://example.com"}' +``` + +Or invoke from your Vercel function: + +```typescript +// pages/api/scrape.ts +import { Kernel } from '@onkernel/sdk'; + +export default async function handler(req, res) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + + // Invoke Kernel app (returns immediately) + const invocation = await kernel.invocations.create({ + app_name: 'scraper', + action_name: 'scrape', + payload: { url: req.body.url }, + async: true + }); + + res.json({ invocationId: invocation.id }); +} +``` + +Check status at `GET /api/v1/invocations/:id`. + +See [Kernel App Platform docs](/apps/develop). + +## Optimization Checklist + +Before moving off serverless, try these optimizations: + +- [ ] Use headless mode (`headless: true`) +- [ ] Block images, fonts, stylesheets with `page.route()` +- [ ] Use `waitUntil: 'domcontentloaded'` instead of `'networkidle'` +- [ ] Reduce Playwright timeouts (`setDefaultTimeout(15000)`) +- [ ] Reuse persistent browser sessions +- [ ] Wait only for specific selectors, not full page load +- [ ] Profile with `PWDEBUG=1` locally to find slow steps +- [ ] Use faster selectors (ID, data-testid) instead of complex CSS + +If you've done all of this and still need more time, use Kernel Apps or a queue. + +## Measuring Performance + +Log each step to identify bottlenecks: + +```typescript +const perf = { + start: Date.now(), + steps: [] as Array<{ name: string; duration: number }> +}; + +function logStep(name: string) { + const now = Date.now(); + const duration = now - (perf.steps[perf.steps.length - 1]?.end || perf.start); + perf.steps.push({ name, duration }); + console.log(`[${name}] ${duration}ms`); +} + +logStep('start'); + +const kb = await kernel.browsers.create({ headless: true }); +logStep('browser_created'); + +const browser = await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }); +logStep('connected'); + +const page = browser.contexts()[0].pages()[0]; +await page.goto('https://example.com'); +logStep('page_loaded'); + +const data = await page.textContent('.content'); +logStep('data_extracted'); + +await browser.close(); +logStep('browser_closed'); + +console.log('Total:', Date.now() - perf.start, 'ms'); +console.log('Steps:', perf.steps); +``` + +Example output: + +``` +[browser_created] 1200ms +[connected] 300ms +[page_loaded] 4500ms ← bottleneck +[data_extracted] 100ms +[browser_closed] 50ms +Total: 6150ms +``` + +Now you know to optimize page load (block resources, change `waitUntil`). + +## Common Timeout Scenarios + +### Scenario 1: Auth Flow Takes Too Long + +**Problem:** Login requires MFA, OTP, or manual approval. + +**Solution:** Use [persistent sessions with profiles](/browsers/profiles): + +```typescript +// First time: manual login via Live View +const kb = await kernel.browsers.create({ + profile_name: 'my-auth-profile', + profile_save_changes: true +}); + +// Visit live view URL (printed in logs) +// Manually log in, complete MFA +// Close browser when done + +// Subsequent uses: instant auth +const kb = await kernel.browsers.create({ + profile_name: 'my-auth-profile' +}); + +// Already logged in, no timeout +``` + +### Scenario 2: Waiting for Dynamic Content + +**Problem:** Page uses infinite scroll, lazy loading, or polling. + +**Solution:** Wait for specific state, not arbitrary timeout: + +```typescript +// ✗ Bad: arbitrary wait +await page.waitForTimeout(10000); + +// ✓ Good: wait for specific condition +await page.waitForSelector('.loaded', { state: 'visible' }); + +// ✓ Better: wait for network to settle +await page.waitForLoadState('networkidle'); + +// ✓ Best: wait for specific API call +await page.waitForResponse(resp => + resp.url().includes('/api/data') && resp.status() === 200 +); +``` + +### Scenario 3: File Download Takes Long + +**Problem:** Downloading large files exceeds timeout. + +**Solution:** Use Kernel's [File I/O API](/browsers/file-io): + +```typescript +const kb = await kernel.browsers.create({ + invocation_id: ctx.invocation_id +}); + +// Trigger download in browser +await page.click('a[download]'); + +// Wait for file to appear (non-blocking) +await page.waitForTimeout(2000); + +// Fetch file via Kernel API (browser can continue) +const files = await kernel.browsers.files.list(kb.session_id, '/downloads'); +const file = files.find(f => f.name.endsWith('.pdf')); + +if (file) { + const content = await kernel.browsers.files.read(kb.session_id, file.path); + // Upload to S3, return URL, etc. +} +``` + +## FAQ + +### Can I increase Vercel's timeout? + +On Hobby plan: No, hard limit of 10s. +On Pro plan: 60s max, configurable in `vercel.json`: + +```json +{ + "functions": { + "api/scrape.ts": { + "maxDuration": 60 + } + } +} +``` + +### Should I use Railway/Fly.io instead? + +Railway and Fly.io have no timeout limits, but you manage infrastructure (Docker images, scaling, health checks). Trade-offs: + +| | Vercel + Kernel | Railway + Chrome | +|-|----------------|-----------------| +| **Setup** | Deploy code, done | Dockerfile, health checks, scaling | +| **Cold Start** | <1s | 5-30s (pull image) | +| **Cost** | Pay per minute | Pay for always-on container | +| **Maintenance** | Zero | Chrome updates, security patches | + +**Recommendation:** Start with Vercel + Kernel. Switch to self-hosted only if you need custom browser configs or regulatory constraints. + +### What about AWS Lambda's 15-minute limit? + +15 minutes is generous for most automations. If you need more: + +1. Split into multiple Lambda invocations (Step Functions) +2. Use Kernel App Platform (no limits) +3. Use ECS/Fargate for long-running jobs + +### Can I run multiple pages in parallel? + +Yes, but each page adds ~2s. For bulk scraping, use [Kernel Apps](/apps/develop) and invoke multiple actions in parallel: + +```typescript +const urls = ['url1', 'url2', 'url3', ...]; + +const invocations = await Promise.all( + urls.map(url => + kernel.invocations.create({ + app_name: 'scraper', + action_name: 'scrape', + payload: { url }, + async: true + }) + ) +); + +// Check status of all invocations +// Each can take as long as needed +``` + +## Related Resources + +- [Headless Chrome on Serverless](/troubleshooting/headless-chrome-serverless) +- [Network Interception](/troubleshooting/network-interception) +- [Kernel App Platform](/apps/develop) +- [Persistent Sessions](/browsers/persistence) +- [File I/O](/browsers/file-io) + +## Need Help? + +Join our [Discord](https://discord.gg/FBrveQRcud) to discuss timeout strategies for your use case. + diff --git a/troubleshooting/playwright-vercel-error.mdx b/troubleshooting/playwright-vercel-error.mdx new file mode 100644 index 0000000..7cf888c --- /dev/null +++ b/troubleshooting/playwright-vercel-error.mdx @@ -0,0 +1,228 @@ +--- +title: "Fix: Playwright 'Executable doesn't exist' on Vercel" +sidebarTitle: "Vercel Error Fix" +description: "Solve Playwright headless_shell errors on Vercel by connecting to Kernel's cloud browsers via CDP. Works in 2 minutes with copy-paste code." +--- + +**You can't launch a local Chromium binary inside Vercel's serverless functions.** Use Playwright's `connectOverCDP` to connect to a hosted browser (Kernel) and keep your code and API routes on Vercel. + +## The Error + +If you're seeing this error on Vercel: + +``` +Error: Executable doesn't exist at /var/task/.next/server/chunks/playwright/chromium-1091/chrome-linux/headless_shell +``` + +Or similar variations like: + +``` +browserType.launch: Executable doesn't exist +Failed to launch browser +``` + +This happens because Vercel's serverless environment doesn't include the Chromium binary that Playwright needs. + +## The Solution + +Instead of launching a local browser, connect to a remote browser via CDP (Chrome DevTools Protocol): + + +```typescript Next.js API Route +// pages/api/scrape.ts +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +export default async function handler(req, res) { + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kernelBrowser = await kernel.browsers.create(); + + const browser = await chromium.connectOverCDP({ + wsEndpoint: kernelBrowser.cdp_ws_url + }); + + const context = browser.contexts()[0]; + const page = context.pages()[0]; + + await page.goto(req.query.url || 'https://example.com'); + const title = await page.title(); + const html = await page.content(); + + await browser.close(); + await kernel.browsers.deleteByID(kernelBrowser.session_id); + + return res.json({ title, html }); +} +``` + +```python Python (Flask/FastAPI) +from playwright.async_api import async_playwright +from kernel import Kernel + +async def scrape_page(url: str): + kernel = Kernel() + kernel_browser = kernel.browsers.create() + + async with async_playwright() as p: + browser = await p.chromium.connect_over_cdp(kernel_browser.cdp_ws_url) + context = browser.contexts[0] + page = context.pages[0] + + await page.goto(url) + title = await page.title() + html = await page.content() + + await browser.close() + kernel.browsers.delete_by_id(kernel_browser.session_id) + + return {"title": title, "html": html} +``` + + +## Environment Variables + +Add your Kernel API key to Vercel: + +```bash +# Get your API key from https://dashboard.onkernel.com/settings/api-keys +vercel env add KERNEL_API_KEY +``` + +Set the value to your Kernel API key and select all environments (Production, Preview, Development). + +## Toggle Between Local and Remote + +For local development, you can use local Playwright. For production on Vercel, use Kernel: + + +```typescript Environment-based Toggle +import { chromium } from 'playwright-core'; +import { Kernel } from '@onkernel/sdk'; + +const isProduction = process.env.VERCEL_ENV === 'production'; + +async function getBrowser() { + if (isProduction || process.env.USE_KERNEL) { + // Use Kernel on Vercel + const kernel = new Kernel({ apiKey: process.env.KERNEL_API_KEY }); + const kb = await kernel.browsers.create(); + return { + browser: await chromium.connectOverCDP({ wsEndpoint: kb.cdp_ws_url }), + sessionId: kb.session_id, + kernel + }; + } else { + // Use local Playwright in development + return { + browser: await chromium.launch(), + sessionId: null, + kernel: null + }; + } +} + +// Usage +const { browser, sessionId, kernel } = await getBrowser(); +// ... use browser ... +await browser.close(); +if (kernel && sessionId) { + await kernel.browsers.deleteByID(sessionId); +} +``` + +```python Environment-based Toggle +import os +from playwright.async_api import async_playwright +from kernel import Kernel + +is_production = os.getenv('VERCEL_ENV') == 'production' + +async def get_browser(): + async with async_playwright() as p: + if is_production or os.getenv('USE_KERNEL'): + # Use Kernel on Vercel + kernel = Kernel() + kb = kernel.browsers.create() + browser = await p.chromium.connect_over_cdp(kb.cdp_ws_url) + return browser, kb.session_id, kernel + else: + # Use local Playwright in development + browser = await p.chromium.launch() + return browser, None, None + +# Usage +browser, session_id, kernel = await get_browser() +# ... use browser ... +await browser.close() +if kernel and session_id: + kernel.browsers.delete_by_id(session_id) +``` + + +## Why This Works + +Vercel's serverless functions have limitations: + +- **No filesystem for binaries:** Chromium requires ~300MB of binaries that can't be bundled +- **Cold start constraints:** Functions need to start in <10s +- **Read-only filesystem:** Can't install or cache browser binaries at runtime + +Kernel provides browsers in the cloud that you connect to via WebSocket, bypassing all these constraints. + +## Native Vercel Integration + +For a seamless setup, install [Kernel from the Vercel Marketplace](https://vercel.com/integrations/kernel). This provides: + +- One-click API key provisioning +- Automatic QA checks on every deployment +- Configuration management via Vercel dashboard + +See the [Vercel integration guide](/integrations/vercel) for details. + +## FAQ + +### Can I run Playwright on Vercel? + +You cannot launch a local Chromium binary inside Vercel's serverless functions. However, you can use Playwright's `connectOverCDP` method to connect to a remote browser hosted by Kernel. Your Playwright code runs on Vercel; the browser runs on Kernel. + +### Do I need to change my existing Playwright code? + +Minimal changes. Replace `browser.launch()` with `chromium.connectOverCDP()` and connect to Kernel's CDP endpoint. The rest of your Playwright code (page navigation, selectors, actions) remains identical. + +### What about `playwright install`? + +You don't need to run `playwright install` on Vercel. Use `playwright-core` (which doesn't include browser binaries) and connect to Kernel's hosted browsers instead. + +### Does this work with Puppeteer too? + +Yes! Puppeteer also supports CDP connections: + +```typescript +import puppeteer from 'puppeteer-core'; +const browser = await puppeteer.connect({ + browserWSEndpoint: kernelBrowser.cdp_ws_url +}); +``` + +### How much does it cost? + +Kernel charges per-minute of active browser time. See [pricing](/info/pricing) for details. Most API routes run in seconds, costing fractions of a cent per request. + +## Related Resources + +- [Vercel Integration Guide](/integrations/vercel) +- [Network Interception on Serverless](/troubleshooting/network-interception) +- [Playwright Timeouts on Serverless](/troubleshooting/playwright-timeouts-serverless) +- [Create a Browser](/browsers/create-a-browser) + +## Troubleshooting + +**Still seeing errors?** Check: + +1. `KERNEL_API_KEY` is set in Vercel environment variables +2. Using `playwright-core` (not `playwright`) in package.json +3. Using `chromium.connectOverCDP()` (not `chromium.launch()`) +4. Awaiting the connection before using the browser + +Need help? Join our [Discord](https://discord.gg/FBrveQRcud) or check the [troubleshooting hub](/troubleshooting/headless-chrome-serverless). +