Skip to content

[SPARK-10040][SQL] Use batch insert for JDBC writing #8273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[SPARK-10040][SQL] Use batch insert for JDBC writing #8273

wants to merge 4 commits into from

Conversation

@viirya
Copy link
Member

@viirya viirya commented Aug 18, 2015

JIRA: https://issues.apache.org/jira/browse/SPARK-10040

We should use batch insert instead of single row in JDBC.

viirya
viirya reviewed Aug 18, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala Outdated
Copy link
Member Author

@viirya viirya Aug 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might not need to have a configuration for it.

Copy link

@rama-mullapudi rama-mullapudi Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should batch size be increased to 10,000 since its big data ;)

Copy link
Member Author

@viirya viirya Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. :-)

@SparkQA
Copy link

SparkQA commented Aug 18, 2015

Test build #41121 has finished for PR 8273 at commit 1062cda.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Member Author

viirya commented Aug 18, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 18, 2015

Test build #41140 has finished for PR 8273 at commit 1062cda.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 19, 2015

Test build #41200 has finished for PR 8273 at commit 1a474e9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Member Author

viirya commented Aug 19, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 19, 2015

Test build #41210 has finished for PR 8273 at commit 1a474e9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

sarutak
sarutak reviewed Aug 19, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala Outdated
Copy link
Member

@sarutak sarutak Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about allowing users to configure this value rather than use fixed value(10000) to avoid OOM?.

Copy link
Member Author

@viirya viirya Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it might not worth adding a configuration for it before. But seems it is hard to decide a proper batch size now. So I add a configuration as you suggested. Thanks.

@sarutak
Copy link
Member

sarutak commented Aug 19, 2015

/CC @rxin

@SparkQA
Copy link

SparkQA commented Aug 19, 2015

Test build #41257 has finished for PR 8273 at commit a63a1b3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zzcclp
Copy link
Contributor

zzcclp commented Aug 20, 2015

will this pr be merged into branch-1.5?I need it.

@rxin
Copy link
Contributor

rxin commented Aug 20, 2015

Sorry this is too late for merging into branch-1.5. We can consider this for master (1.6).

@zzcclp
Copy link
Contributor

zzcclp commented Aug 20, 2015

@rxin , thanks for your reply.

rxin
rxin reviewed Aug 20, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala Outdated
Copy link
Contributor

@rxin rxin Aug 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the other properties called? do they start with "jdbc"?

Copy link
Member Author

@viirya viirya Aug 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think no. Just use "batchsize"?

Copy link
Contributor

@rxin rxin Aug 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds good.

@sarutak
Copy link
Member

sarutak commented Aug 21, 2015

LGTM.

@rxin
Copy link
Contributor

rxin commented Aug 21, 2015

Thanks - I've merged this.

@asfgit asfgit closed this in bb220f6 Aug 21, 2015
@SparkQA
Copy link

SparkQA commented Aug 21, 2015

Test build #41355 has finished for PR 8273 at commit f8b9566.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya viirya deleted the jdbc-insert-batch branch December 27, 2023 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@viirya @SparkQA @sarutak @zzcclp @rxin @rama-mullapudi