Skip to content

SPARK-4267 [YARN] Backport: Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later #4575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

SPARK-4267 [YARN] Backport: Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later #4575

wants to merge 1 commit into from

Conversation

@srowen
Copy link
Member

@srowen srowen commented Feb 12, 2015

Backport of SPARK-4267 (de78060) for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string.

@srowen
Backport of SPARK-4267 for branch-1.2: Before passing to YARN, escape…
7bfc75b 
… arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string.
@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27385 has started for PR 4575 at commit 7bfc75b.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27385 has finished for PR 4575 at commit 7bfc75b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Feb 12, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27385/
Test FAILed.

@srowen
Copy link
Member Author

srowen commented Feb 12, 2015

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27393 has started for PR 4575 at commit 7bfc75b.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27393 has finished for PR 4575 at commit 7bfc75b.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Feb 12, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27393/
Test FAILed.

@srowen
Copy link
Member Author

srowen commented Feb 13, 2015

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27403 has started for PR 4575 at commit 7bfc75b.

  • This patch merges cleanly.

@vanzin
Copy link
Contributor

vanzin commented Feb 13, 2015

LGTM pending tests.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27403 has finished for PR 4575 at commit 7bfc75b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Feb 13, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27403/
Test FAILed.

@srowen
Copy link
Member Author

srowen commented Feb 13, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27439 has started for PR 4575 at commit 7bfc75b.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27439 has finished for PR 4575 at commit 7bfc75b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Feb 13, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27439/
Test PASSed.

asfgit pushed a commit that referenced this pull request Feb 13, 2015
@srowen
SPARK-4267 [YARN] Backport: Failing to launch jobs on Spark on YARN w…
c2b4633 
…ith Hadoop 2.5.0 or later

Backport of SPARK-4267 (de78060) for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string.

Author: Sean Owen <[email protected]>

Closes #4575 from srowen/SPARK-4267-1.2 and squashes the following commits:

7bfc75b [Sean Owen] Backport of SPARK-4267 for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string.
@srowen srowen closed this Feb 13, 2015
@srowen srowen deleted the SPARK-4267-1.2 branch February 13, 2015 14:49
markhamstra pushed a commit to markhamstra/spark that referenced this pull request Feb 24, 2015
@srowen @markhamstra
SPARK-4267 [YARN] Backport: Failing to launch jobs on Spark on YARN w…
39c2c60 
…ith Hadoop 2.5.0 or later

Backport of SPARK-4267 (apache@de78060) for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string.

Author: Sean Owen <[email protected]>

Closes apache#4575 from srowen/SPARK-4267-1.2 and squashes the following commits:

7bfc75b [Sean Owen] Backport of SPARK-4267 for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string.
LuciferYang pushed a commit that referenced this pull request Jul 9, 2024
@wayneguow @LuciferYang
[SPARK-48826][BUILD] Upgrade fasterxml.jackson to 2.17.2
65daff5 
### What changes were proposed in this pull request?

This PR amis to upgrade `fasterxml.jackson` from 2.17.1 to 2.17.2.

### Why are the changes needed?

There are some bug fixes about [Databind](https://github.com/FasterXML/jackson-databind):
[#4561](FasterXML/jackson-databind#4561): Issues using jackson-databind 2.17.1 with Reactor (wrt DeserializerCache and ReentrantLock)
[#4575](FasterXML/jackson-databind#4575): StdDelegatingSerializer does not consider a Converter that may return null for a non-null input
[#4577](FasterXML/jackson-databind#4577): Cannot deserialize value of type java.math.BigDecimal from String "3." (not a valid representation)
[#4595](FasterXML/jackson-databind#4595): No way to explicitly disable wrapping in custom annotation processor
[#4607](FasterXML/jackson-databind#4607): MismatchedInput: No Object Id found for an instance of X to assign to property 'id'
[#4610](FasterXML/jackson-databind#4610): DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS does not work when used with Polymorphic type handling

The full release note of 2.17.2:
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.17.2

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #47241 from wayneguow/upgrade_jackson.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
ericm-db pushed a commit to ericm-db/spark that referenced this pull request Jul 10, 2024
@wayneguow @ericm-db
[SPARK-48826][BUILD] Upgrade fasterxml.jackson to 2.17.2
3d54034 
### What changes were proposed in this pull request?

This PR amis to upgrade `fasterxml.jackson` from 2.17.1 to 2.17.2.

### Why are the changes needed?

There are some bug fixes about [Databind](https://github.com/FasterXML/jackson-databind):
[apache#4561](FasterXML/jackson-databind#4561): Issues using jackson-databind 2.17.1 with Reactor (wrt DeserializerCache and ReentrantLock)
[apache#4575](FasterXML/jackson-databind#4575): StdDelegatingSerializer does not consider a Converter that may return null for a non-null input
[apache#4577](FasterXML/jackson-databind#4577): Cannot deserialize value of type java.math.BigDecimal from String "3." (not a valid representation)
[apache#4595](FasterXML/jackson-databind#4595): No way to explicitly disable wrapping in custom annotation processor
[apache#4607](FasterXML/jackson-databind#4607): MismatchedInput: No Object Id found for an instance of X to assign to property 'id'
[apache#4610](FasterXML/jackson-databind#4610): DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS does not work when used with Polymorphic type handling

The full release note of 2.17.2:
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.17.2

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#47241 from wayneguow/upgrade_jackson.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
jingz-db pushed a commit to jingz-db/spark that referenced this pull request Jul 22, 2024
@wayneguow @jingz-db
[SPARK-48826][BUILD] Upgrade fasterxml.jackson to 2.17.2
ca24fc0 
### What changes were proposed in this pull request?

This PR amis to upgrade `fasterxml.jackson` from 2.17.1 to 2.17.2.

### Why are the changes needed?

There are some bug fixes about [Databind](https://github.com/FasterXML/jackson-databind):
[apache#4561](FasterXML/jackson-databind#4561): Issues using jackson-databind 2.17.1 with Reactor (wrt DeserializerCache and ReentrantLock)
[apache#4575](FasterXML/jackson-databind#4575): StdDelegatingSerializer does not consider a Converter that may return null for a non-null input
[apache#4577](FasterXML/jackson-databind#4577): Cannot deserialize value of type java.math.BigDecimal from String "3." (not a valid representation)
[apache#4595](FasterXML/jackson-databind#4595): No way to explicitly disable wrapping in custom annotation processor
[apache#4607](FasterXML/jackson-databind#4607): MismatchedInput: No Object Id found for an instance of X to assign to property 'id'
[apache#4610](FasterXML/jackson-databind#4610): DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS does not work when used with Polymorphic type handling

The full release note of 2.17.2:
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.17.2

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#47241 from wayneguow/upgrade_jackson.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
attilapiros pushed a commit to attilapiros/spark that referenced this pull request Oct 4, 2024
@wayneguow @attilapiros
[SPARK-48826][BUILD] Upgrade fasterxml.jackson to 2.17.2
5e9b6a4 
### What changes were proposed in this pull request?

This PR amis to upgrade `fasterxml.jackson` from 2.17.1 to 2.17.2.

### Why are the changes needed?

There are some bug fixes about [Databind](https://github.com/FasterXML/jackson-databind):
[apache#4561](FasterXML/jackson-databind#4561): Issues using jackson-databind 2.17.1 with Reactor (wrt DeserializerCache and ReentrantLock)
[apache#4575](FasterXML/jackson-databind#4575): StdDelegatingSerializer does not consider a Converter that may return null for a non-null input
[apache#4577](FasterXML/jackson-databind#4577): Cannot deserialize value of type java.math.BigDecimal from String "3." (not a valid representation)
[apache#4595](FasterXML/jackson-databind#4595): No way to explicitly disable wrapping in custom annotation processor
[apache#4607](FasterXML/jackson-databind#4607): MismatchedInput: No Object Id found for an instance of X to assign to property 'id'
[apache#4610](FasterXML/jackson-databind#4610): DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS does not work when used with Polymorphic type handling

The full release note of 2.17.2:
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.17.2

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#47241 from wayneguow/upgrade_jackson.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
turboFei pushed a commit to turboFei/spark that referenced this pull request Nov 6, 2025
@wangyum @panbingkun @wayneguow
[HADP-54080][SPARK-48826] Upgrade jackson-databind to 2.17.2 (apache#433
1529945 
)

* [SPARK-45967][BUILD] Upgrade jackson to 2.16.0

### What changes were proposed in this pull request?
The pr aims to upgrade FasterXML jackson from 2.15.2 to 2.16.0.

### Why are the changes needed?
New version that fix some bugs, release notes as follows:
- 2.1.6.0 https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.16, eg:
[Databind](https://github.com/FasterXML/jackson-databind) [apache#1770](FasterXML/jackson-databind#1770): Incorrect deserialization for BigDecimal numbers
- 2.15.3 https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15.3, eg:
[Databind](https://github.com/FasterXML/jackson-databind) [apache#3968](FasterXML/jackson-databind#3968): Records with additional constructors failed to deserialize

The last upgrade occurred 6 months ago, apache#41414

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#43859 from panbingkun/SPARK-45967.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

(cherry picked from commit c17b092)

* [SPARK-48826][BUILD] Upgrade `fasterxml.jackson` to 2.17.2

### What changes were proposed in this pull request?

This PR amis to upgrade `fasterxml.jackson` from 2.17.1 to 2.17.2.

### Why are the changes needed?

There are some bug fixes about [Databind](https://github.com/FasterXML/jackson-databind):
[apache#4561](FasterXML/jackson-databind#4561): Issues using jackson-databind 2.17.1 with Reactor (wrt DeserializerCache and ReentrantLock)
[apache#4575](FasterXML/jackson-databind#4575): StdDelegatingSerializer does not consider a Converter that may return null for a non-null input
[apache#4577](FasterXML/jackson-databind#4577): Cannot deserialize value of type java.math.BigDecimal from String "3." (not a valid representation)
[apache#4595](FasterXML/jackson-databind#4595): No way to explicitly disable wrapping in custom annotation processor
[apache#4607](FasterXML/jackson-databind#4607): MismatchedInput: No Object Id found for an instance of X to assign to property 'id'
[apache#4610](FasterXML/jackson-databind#4610): DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS does not work when used with Polymorphic type handling

The full release note of 2.17.2:
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.17.2

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#47241 from wayneguow/upgrade_jackson.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: yangjie01 <[email protected]>

(cherry picked from commit 65daff5)

* fix

---------

Co-authored-by: panbingkun <[email protected]>
Co-authored-by: Wei Guo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@srowen @SparkQA @AmplabJenkins @vanzin