Skip to content

[SPARK-3399][PySpark] Test for PySpark should ignore HADOOP_CONF_DIR and YARN_CONF_DIR #2270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-3399][PySpark] Test for PySpark should ignore HADOOP_CONF_DIR and YARN_CONF_DIR #2270

wants to merge 1 commit into from

Conversation

@sarutak
Copy link
Member

@sarutak sarutak commented Sep 4, 2014

No description provided.

@sarutak sarutak changed the title [SPARK-3399] Test for PySpark should ignore HADOOP_CONF_DIR and YARN_CONF_DIR [SPARK-3399][PySpark] Test for PySpark should ignore HADOOP_CONF_DIR and YARN_CONF_DIR Sep 4, 2014
@sarutak
Copy link
Member Author

sarutak commented Sep 4, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Sep 4, 2014

QA tests have started for PR 2270 at commit 7613be6.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 4, 2014

QA tests have finished for PR 2270 at commit 7613be6.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class BlockManagerMaster(
    • class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])

davies
davies reviewed Sep 5, 2014
View reviewed changes
bin/pyspark
Copy link
Contributor

@davies davies Sep 5, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this problem only happen during testing, could we put these in python/run-tests? pyspark will be often used as python shell.

Copy link
Member Author

@sarutak sarutak Sep 5, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your comment.
As I mentioned in JIRA, YARN_CONF_DIR and HADOOP_CONF_DIR is loaded in pyspark script and some tests like rdd.py are kicked by pyspark in python/run-tests so it's not make sense that putting "unset" on python/run-tests.

Copy link
Contributor

@davies davies Sep 5, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I get it.

@davies
Copy link
Contributor

davies commented Sep 5, 2014

This patch looks good to me.

@JoshRosen could you help to re-visit this?

@JoshRosen
Copy link
Contributor

JoshRosen commented Sep 5, 2014

Looks good to me, too. Thanks for fixing this!

@asfgit asfgit closed this in 7ff8c45 Sep 5, 2014
@sarutak sarutak deleted the SPARK-3399 branch April 11, 2015 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sarutak @SparkQA @davies @JoshRosen