Skip to content

[FIX] Snowflake - Insert Multiple Rows #17722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 23, 2025

Conversation

jcortes
Copy link
Collaborator

@jcortes jcortes commented Jul 21, 2025

WHY

Resolves #17239

Summary by CodeRabbit

  • New Features

    • Added support for batch processing when inserting multiple rows, including configurable batch size, payload size, and batching toggle.
    • Enhanced reporting for batch insert operations, providing detailed success and failure summaries.
    • Improved error handling with clearer diagnostics and suggestions for batch-related issues.
  • Bug Fixes

    • Improved input validation for row insertion, ensuring better error messages for malformed data.
  • Chores

    • Updated version numbers for several actions, sources, and the Snowflake component package.

@jcortes jcortes self-assigned this Jul 21, 2025
Copy link

vercel bot commented Jul 21, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

3 Skipped Deployments
Name Status Preview Comments Updated (UTC)
docs-v2 ⬜️ Ignored (Inspect) Visit Preview Jul 21, 2025 10:27pm
pipedream-docs ⬜️ Ignored (Inspect) Jul 21, 2025 10:27pm
pipedream-docs-redirect-do-not-edit ⬜️ Ignored (Inspect) Jul 21, 2025 10:27pm

Copy link
Contributor

coderabbitai bot commented Jul 21, 2025

Walkthrough

This update introduces robust batch processing for inserting multiple rows into Snowflake. The insertRows method now supports configurable batching, payload size estimation, and detailed error handling. A new utility module provides batching helpers. Input validation and error messaging are improved. Several version numbers are incremented to reflect these enhancements.

Changes

File(s) Change Summary
components/snowflake/actions/execute-sql-query/execute-sql-query.mjs
components/snowflake/actions/insert-row/insert-row.mjs
components/snowflake/sources/*
components/snowflake/package.json
Incremented version numbers only.
components/snowflake/actions/insert-multiple-rows/insert-multiple-rows.mjs Added batch processing options (batchSize, maxPayloadSizeMB, enableBatching). Enhanced input validation, error handling, and result reporting for batch inserts. Logic now supports and manages batch inserts with improved diagnostics.
components/snowflake/common/utils.mjs New utility module with estimatePayloadSize and createBatches functions for batching and payload size estimation.
components/snowflake/snowflake.app.mjs Replaced insertRows with a batch-capable version, added _insertRowsOriginal for single-batch inserts. Implements payload size estimation, batch splitting, sequential batch processing, error aggregation, and summary reporting.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant InsertMultipleRowsAction
    participant SnowflakeApp
    participant Utils

    User->>InsertMultipleRowsAction: Trigger action with table, columns, values, batch options
    InsertMultipleRowsAction->>Utils: Parse and validate input, create batches
    InsertMultipleRowsAction->>SnowflakeApp: insertRows(table, columns, values, options)
    SnowflakeApp->>Utils: estimatePayloadSize(values)
    SnowflakeApp->>Utils: createBatches(values, batchSize)
    loop For each batch
        SnowflakeApp->>SnowflakeApp: _insertRowsOriginal(table, columns, batch)
        SnowflakeApp-->>InsertMultipleRowsAction: Batch result (success/failure)
    end
    InsertMultipleRowsAction->>User: Export summary and results
Loading

Estimated code review effort

3 (~45 minutes)

Poem

In snowy fields of data, rows march in a line,
Now grouped in cozy batches—how efficiently they dine!
With payloads weighed and errors caught,
The bunny hops, no worries fraught.
Version bumps and helpers new,
Snowflake’s ready—hoppity-hoo!
🐇✨

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

components/snowflake/actions/execute-sql-query/execute-sql-query.mjs

Oops! Something went wrong! :(

ESLint: 8.57.1

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs
at Object.getPackageJSONURL (node:internal/modules/package_json_reader:255:9)
at packageResolve (node:internal/modules/esm/resolve:767:81)
at moduleResolve (node:internal/modules/esm/resolve:853:18)
at defaultResolve (node:internal/modules/esm/resolve:983:11)
at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:801:12)
at #cachedDefaultResolve (node:internal/modules/esm/loader:725:25)
at ModuleLoader.resolve (node:internal/modules/esm/loader:708:38)
at ModuleLoader.getModuleJobForImport (node:internal/modules/esm/loader:309:38)
at #link (node:internal/modules/esm/module_job:202:49)

components/snowflake/actions/insert-multiple-rows/insert-multiple-rows.mjs

Oops! Something went wrong! :(

ESLint: 8.57.1

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs
at Object.getPackageJSONURL (node:internal/modules/package_json_reader:255:9)
at packageResolve (node:internal/modules/esm/resolve:767:81)
at moduleResolve (node:internal/modules/esm/resolve:853:18)
at defaultResolve (node:internal/modules/esm/resolve:983:11)
at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:801:12)
at #cachedDefaultResolve (node:internal/modules/esm/loader:725:25)
at ModuleLoader.resolve (node:internal/modules/esm/loader:708:38)
at ModuleLoader.getModuleJobForImport (node:internal/modules/esm/loader:309:38)
at #link (node:internal/modules/esm/module_job:202:49)

components/snowflake/snowflake.app.mjs

Oops! Something went wrong! :(

ESLint: 8.57.1

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs
at Object.getPackageJSONURL (node:internal/modules/package_json_reader:255:9)
at packageResolve (node:internal/modules/esm/resolve:767:81)
at moduleResolve (node:internal/modules/esm/resolve:853:18)
at defaultResolve (node:internal/modules/esm/resolve:983:11)
at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:801:12)
at #cachedDefaultResolve (node:internal/modules/esm/loader:725:25)
at ModuleLoader.resolve (node:internal/modules/esm/loader:708:38)
at ModuleLoader.getModuleJobForImport (node:internal/modules/esm/loader:309:38)
at #link (node:internal/modules/esm/module_job:202:49)

  • 16 others

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4018d2d and 0465cb4.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (20)
  • components/snowflake/actions/execute-sql-query/execute-sql-query.mjs (1 hunks)
  • components/snowflake/actions/insert-multiple-rows/insert-multiple-rows.mjs (2 hunks)
  • components/snowflake/actions/insert-row/insert-row.mjs (1 hunks)
  • components/snowflake/common/utils.mjs (1 hunks)
  • components/snowflake/package.json (1 hunks)
  • components/snowflake/snowflake.app.mjs (2 hunks)
  • components/snowflake/sources/change-to-warehouse/change-to-warehouse.mjs (1 hunks)
  • components/snowflake/sources/deleted-role/deleted-role.mjs (1 hunks)
  • components/snowflake/sources/deleted-user/deleted-user.mjs (1 hunks)
  • components/snowflake/sources/failed-task-in-schema/failed-task-in-schema.mjs (1 hunks)
  • components/snowflake/sources/new-database/new-database.mjs (1 hunks)
  • components/snowflake/sources/new-role/new-role.mjs (1 hunks)
  • components/snowflake/sources/new-row/new-row.mjs (1 hunks)
  • components/snowflake/sources/new-schema/new-schema.mjs (1 hunks)
  • components/snowflake/sources/new-table/new-table.mjs (1 hunks)
  • components/snowflake/sources/new-user/new-user.mjs (1 hunks)
  • components/snowflake/sources/query-results/query-results.mjs (1 hunks)
  • components/snowflake/sources/updated-role/updated-role.mjs (1 hunks)
  • components/snowflake/sources/updated-user/updated-user.mjs (1 hunks)
  • components/snowflake/sources/usage-monitor/usage-monitor.mjs (1 hunks)
✅ Files skipped from review due to trivial changes (15)
  • components/snowflake/sources/query-results/query-results.mjs
  • components/snowflake/sources/new-table/new-table.mjs
  • components/snowflake/sources/new-user/new-user.mjs
  • components/snowflake/sources/updated-role/updated-role.mjs
  • components/snowflake/sources/new-database/new-database.mjs
  • components/snowflake/sources/deleted-role/deleted-role.mjs
  • components/snowflake/sources/updated-user/updated-user.mjs
  • components/snowflake/sources/usage-monitor/usage-monitor.mjs
  • components/snowflake/sources/new-row/new-row.mjs
  • components/snowflake/sources/new-role/new-role.mjs
  • components/snowflake/sources/deleted-user/deleted-user.mjs
  • components/snowflake/sources/change-to-warehouse/change-to-warehouse.mjs
  • components/snowflake/sources/new-schema/new-schema.mjs
  • components/snowflake/sources/failed-task-in-schema/failed-task-in-schema.mjs
  • components/snowflake/common/utils.mjs
🚧 Files skipped from review as they are similar to previous changes (3)
  • components/snowflake/package.json
  • components/snowflake/actions/insert-row/insert-row.mjs
  • components/snowflake/actions/execute-sql-query/execute-sql-query.mjs
🧰 Additional context used
🧬 Code Graph Analysis (2)
components/snowflake/actions/insert-multiple-rows/insert-multiple-rows.mjs (1)
components/snowflake/snowflake.app.mjs (4)
  • rows (176-179)
  • rows (218-218)
  • summary (464-472)
  • error (477-479)
components/snowflake/snowflake.app.mjs (2)
components/snowflake/common/utils.mjs (2)
  • batches (14-14)
  • i (15-15)
components/snowflake/actions/insert-multiple-rows/insert-multiple-rows.mjs (1)
  • error (167-167)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
  • GitHub Check: Verify TypeScript components
  • GitHub Check: Publish TypeScript components
  • GitHub Check: Lint Code Base
🔇 Additional comments (13)
components/snowflake/actions/insert-multiple-rows/insert-multiple-rows.mjs (7)

9-9: Version bump looks appropriate.

The patch version increment from 0.1.2 to 0.1.3 correctly reflects the addition of new batch processing features.


53-77: Well-designed batch configuration properties.

The new batch processing properties are thoughtfully implemented with:

  • Sensible default values (100 batch size, 5MB payload limit)
  • Appropriate constraints and validation ranges
  • Clear descriptions with usage guidance
  • Optional nature ensuring backward compatibility

85-89: Enhanced JSON parsing with proper error handling.

The addition of try-catch around JSON.parse is a crucial improvement that prevents runtime crashes and provides clear, actionable error messages to users when invalid JSON is provided.


95-119: Excellent validation improvements with detailed error reporting.

The enhanced validation logic provides:

  • Detailed logging of problematic rows with indices
  • Column count validation preventing data corruption
  • Clear error messages specifying exactly how many rows are invalid
  • Early validation before any database operations

121-126: Clean batch options configuration.

The encapsulation of batch configuration into a single options object is a clean pattern that makes the method call more readable and maintainable.


128-161: Robust response handling for both batch and single insert modes.

The response handling logic correctly:

  • Differentiates between batched and single insert responses
  • Provides detailed batch statistics and debugging information
  • Maintains backward compatibility for single inserts
  • Exports comprehensive summary data

163-183: Comprehensive error handling for batch processing scenarios.

The enhanced error handling provides:

  • Graceful handling of partial batch failures with detailed reporting
  • Specific guidance for 413 Payload Too Large errors with actionable advice
  • Contextual information about data size for troubleshooting
  • Preservation of existing error behavior for other scenarios
components/snowflake/snowflake.app.mjs (6)

5-7: Appropriate imports for batch processing functionality.

The addition of ConfigurationError and the utils module imports supports the new batch processing features with proper error handling and utility functions.


372-388: Clean refactoring of original insert logic.

Moving the original insert logic to _insertRowsOriginal is a well-executed refactor that:

  • Preserves existing functionality unchanged
  • Uses appropriate private method naming convention
  • Enables reuse by the new batch processing logic
  • Maintains clean separation of concerns

389-399: Smart fallback logic for small datasets.

The decision to use the original approach for small datasets (≤50 rows) or when batching is disabled provides optimal performance and maintains simplicity for common use cases.


401-417: Intelligent dynamic batch sizing with safety margins.

The payload size estimation and dynamic batch sizing logic is well-designed:

  • Uses sample data to estimate average row size
  • Applies 80% safety margin to prevent payload limit issues
  • Enforces reasonable bounds (10-500 rows per batch)
  • Provides helpful logging for troubleshooting

419-462: Robust batch processing with comprehensive error handling.

The sequential batch processing implementation excels at:

  • Processing batches individually to isolate failures
  • Detailed progress logging for each batch
  • Collecting comprehensive result data for each batch
  • Continuing processing after individual batch failures
  • Adding reasonable delays between batches to prevent server overload

464-489: Excellent summary reporting and error handling.

The final summary and error handling provides:

  • Comprehensive statistics about the batch operation
  • Partial failure support with detailed error information
  • Attachment of summary to errors for debugging
  • Clear success/failure distinction in return values
✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@jcortes jcortes force-pushed the fix-snowflake-insert-multiple-rows branch from 4018d2d to 0465cb4 Compare July 21, 2025 22:27
Copy link
Collaborator

@GTFalcao GTFalcao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@vunguyenhung vunguyenhung merged commit 30f72f2 into master Jul 23, 2025
11 checks passed
@vunguyenhung vunguyenhung deleted the fix-snowflake-insert-multiple-rows branch July 23, 2025 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Snowflake - Insert Multiple Rows
3 participants