[SPARK-1459] Use local path (and not complete URL) when opening local lo... #375

vanzin · 2014-04-09T23:06:04Z

...g file.

… log file.

AmplabJenkins · 2014-04-09T23:07:23Z

Can one of the admins verify this patch?

pwendell · 2014-04-10T00:42:37Z

/cc @andrewor14

andrewor14 · 2014-04-10T18:40:15Z

Good catch. The existing code fails when the logDir includes file://. This LGTM.

andrewor14 · 2014-04-10T18:40:23Z

Jenkins, test this please.

pwendell · 2014-04-10T18:56:18Z

Jenkins, test this please.

AmplabJenkins · 2014-04-10T18:58:13Z

Merged build triggered.

AmplabJenkins · 2014-04-10T18:58:21Z

Merged build started.

AmplabJenkins · 2014-04-10T19:37:51Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-10T19:37:51Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14009/

vanzin · 2014-04-11T17:28:17Z

FYI: just found another issue in this same code. It does not handle the case where the default fs is not "file:", and will try to use FileOutputStream even when it's something like hdfs. I'll fix that and push the new code.

andrewor14 · 2014-04-11T18:17:41Z

Not sure if I understand. Are you saying that if we somehow configure the default file scheme to be hdfs://, then it will still match the null in the first case statement, even if the path is really for HDFS? Currently the FileLogger requires you to explicitly specify the hdfs:// scheme as it does not have a notion of default file schemes (unlike in say sc.textFile), though this need not be the case.

vanzin · 2014-04-11T18:25:06Z

Ok, maybe it's not a bug per se. But I think it somewhat breaks the user's expectations.

Let's say that given a user's configuration, the default fs is hdfs; I'd expect that if I specify the log directory to be "/user/vanzin/logs", that would be in HDFS, not in the local fs. But with the current FileLogger code, it will assume that is a local path, since there is no scheme defined.

Anyway, the current patch fixes the actual bug; if you think the above should be handled too I have the code to do it working, but it's not strictly necessary.

vanzin · 2014-04-11T20:17:02Z

Commit that implements what I'm suggesting, if you're interested:
vanzin@15c4d1e

I haven't sent a pull request for that one.

andrewor14 · 2014-04-18T01:28:16Z

@vanzin I recently experienced what you mean on a YARN cluster myself. I've looked at your patch and I think it's a good fix for Spark on YARN. Could you submit a PR for it? It will be more convenient for me to leave my comments there.

andrewor14 · 2014-04-18T01:28:40Z

On a separate note, @pwendell this PR is ready for merge.

pwendell · 2014-04-22T06:11:22Z

Thanks, I've merged this.

… lo... ...g file. Author: Marcelo Vanzin <[email protected]> Closes #375 from vanzin/event-file and squashes the following commits: f673029 [Marcelo Vanzin] [SPARK-1459] Use local path (and not complete URL) when opening local log file. (cherry picked from commit ac164b7) Signed-off-by: Patrick Wendell <[email protected]>

This is related to SPARK-1459 / PR apache#375. Without this fix, FileLogger.createLogDir() may try to create the log dir on HDFS, while createWriter() will try to open the log file on the local file system, leading to interesting errors and confusion.

This is related to SPARK-1459 / PR #375. Without this fix, FileLogger.createLogDir() may try to create the log dir on HDFS, while createWriter() will try to open the log file on the local file system, leading to interesting errors and confusion. Author: Marcelo Vanzin <[email protected]> Closes #450 from vanzin/event-file-2 and squashes the following commits: 592cdb3 [Marcelo Vanzin] Honor default fs name when initializing event logger.

This is related to SPARK-1459 / PR #375. Without this fix, FileLogger.createLogDir() may try to create the log dir on HDFS, while createWriter() will try to open the log file on the local file system, leading to interesting errors and confusion. Author: Marcelo Vanzin <[email protected]> Closes #450 from vanzin/event-file-2 and squashes the following commits: 592cdb3 [Marcelo Vanzin] Honor default fs name when initializing event logger. (cherry picked from commit dd1b7a6) Signed-off-by: Patrick Wendell <[email protected]>

… lo... ...g file. Author: Marcelo Vanzin <[email protected]> Closes apache#375 from vanzin/event-file and squashes the following commits: f673029 [Marcelo Vanzin] [SPARK-1459] Use local path (and not complete URL) when opening local log file.

This is related to SPARK-1459 / PR apache#375. Without this fix, FileLogger.createLogDir() may try to create the log dir on HDFS, while createWriter() will try to open the log file on the local file system, leading to interesting errors and confusion. Author: Marcelo Vanzin <[email protected]> Closes apache#450 from vanzin/event-file-2 and squashes the following commits: 592cdb3 [Marcelo Vanzin] Honor default fs name when initializing event logger.

Test with ./dev/scalastyle

…mbol is a token (apache#375) In the case where the offending symbol is a CommonToken, this PR increases the accuracy of the start and stop origin by leveraging the start and stop index information from CommonToken.

…pache#375) This change add the job of terraform-provider-huaweicloud-acceptance-test-fusioncloud Closes-Bug: theopenlab/openlab#130

…quet (apache#375) [SPARK-47368][SQL]][3.5] Remove inferTimestampNTZ config check in ParquetRo… ### What changes were proposed in this pull request? The configuration `spark.sql.parquet.inferTimestampNTZ.enabled` is not related the parquet row converter. This PR is the remove the config check `spark.sql.parquet.inferTimestampNTZ.enabled` in the ParquetRowConverter ### Why are the changes needed? Bug fix. Otherwise reading TimestampNTZ columns may fail when `spark.sql.parquet.inferTimestampNTZ.enabled` is disabled. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New UT ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#45492 from gengliangwang/PR_TOOL_PICK_PR_45480_BRANCH-3.5. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Gengliang Wang <[email protected]> Co-authored-by: Gengliang Wang <[email protected]>

[SPARK-1459] Use local path (and not complete URL) when opening local…

f673029

… log file.

vanzin mentioned this pull request Apr 19, 2014

Honor default fs name when initializing event logger. #450

Closed

asfgit closed this in ac164b7 Apr 22, 2014

vanzin deleted the event-file branch May 23, 2014 22:39

tangzhankun pushed a commit to tangzhankun/spark that referenced this pull request Jul 25, 2017

Fix import order and scalastyle (apache#375)

db5f5be

Test with ./dev/scalastyle

erikerlandson pushed a commit to erikerlandson/spark that referenced this pull request Jul 28, 2017

Fix import order and scalastyle (apache#375)

42f578f

Test with ./dev/scalastyle

arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020

* Bug 32263 - Seek called on unsubscribed partitions (apache#375)

3dbb6b2

[SPARK-1459] Use local path (and not complete URL) when opening local lo... #375

[SPARK-1459] Use local path (and not complete URL) when opening local lo... #375

Uh oh!

Conversation

vanzin commented Apr 9, 2014

Uh oh!

AmplabJenkins commented Apr 9, 2014

Uh oh!

pwendell commented Apr 10, 2014

Uh oh!

andrewor14 commented Apr 10, 2014

Uh oh!

andrewor14 commented Apr 10, 2014

Uh oh!

pwendell commented Apr 10, 2014

Uh oh!

AmplabJenkins commented Apr 10, 2014

Uh oh!

AmplabJenkins commented Apr 10, 2014

Uh oh!

AmplabJenkins commented Apr 10, 2014

Uh oh!

AmplabJenkins commented Apr 10, 2014

Uh oh!

vanzin commented Apr 11, 2014

Uh oh!

andrewor14 commented Apr 11, 2014

Uh oh!

vanzin commented Apr 11, 2014

Uh oh!

vanzin commented Apr 11, 2014

Uh oh!

andrewor14 commented Apr 18, 2014

Uh oh!

andrewor14 commented Apr 18, 2014

Uh oh!

pwendell commented Apr 22, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants