[SPARK-28752][BUILD][DOCS] Documentation build to support Python 3 #26521

HyukjinKwon · 2019-11-14T13:06:46Z

What changes were proposed in this pull request?

This PR proposes to switch pygments.rb, which only support Python 2 and seems inactive for the last few years (https://github.com/tmm1/pygments.rb), to Rouge which is pure Ruby code highlighter that is compatible with Pygments.

I thought it would be pretty difficult to change but thankfully Rouge does a great job as the alternative.

Why are the changes needed?

We're moving to Python 3 and drop Python 2 completely.

Does this PR introduce any user-facing change?

Maybe a little bit of different syntax style but should not have a notable change.

How was this patch tested?

Manually tested the build and checked the documentation.

HyukjinKwon · 2019-11-14T13:07:11Z

I'm going to closely double check tomorrow again. I just roughly checked.

HyukjinKwon · 2019-11-14T13:07:46Z

cc @dongjoon-hyun, @shaneknapp, @BryanCutler, @WeichenXu123, @mengxr, @srowen

HyukjinKwon · 2019-11-14T13:08:35Z

dev/create-release/spark-rm/Dockerfile


 ARG BASE_PIP_PKGS="setuptools wheel virtualenv"
-ARG PIP_PKGS="pyopenssl pypandoc numpy pygments sphinx"
+ARG PIP_PKGS="pyopenssl pypandoc numpy sphinx"


Sphinx still needs pygments but it's in sphinx's dependency. I don't think we should explicitly list since Spark doesn't directly use it (but sphinx uses it).

cc @vanzin too

srowen

If the doc build works, seems fine to me.

srowen · 2019-11-14T14:15:14Z

docs/README.md


 You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
 [Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
 installed. Also install the following libraries:


Do you want to mention Python 3 is required here? or does it still work with Python 2?

I think it still works with Python 2 because it seems not using Python at all. But I guess we should better stick to Python 3 ..

SparkQA · 2019-11-14T15:34:14Z

Test build #113785 has finished for PR 26521 at commit 2b142b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-11-15T04:43:49Z

Right, I double checked and seems fine. should be good to go. I am merging this.

HyukjinKwon · 2019-11-15T04:44:01Z

Merged to master.

dongjoon-hyun · 2019-11-15T04:51:46Z

Sorry for being late. Thank you for finishing this~

dongjoon-hyun · 2019-11-26T22:11:15Z

Hi, @HyukjinKwon . This seems to break our release script because rouge requires Ruby 2.5.0+ and our spark-rm Docker file has Ruby 2.3.

$ ruby -v
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-darwin18]

$ gem install rogue
ERROR:  Error installing rogue:
	sprockets requires Ruby version >= 2.5.0.

dongjoon-hyun · 2019-11-26T22:11:54Z

cc @gatorsmile and @jiangxb1987

srowen · 2019-11-26T22:24:39Z

Oh, I think that's a typo. It's rouge not rogue, right?

jiangxb1987 · 2019-11-26T22:25:32Z

oops

dongjoon-hyun · 2019-11-26T22:28:32Z

Oh... It's weird. Although this PR and https://github.com/apache/spark-website/pull/233/files#diff-04c6e90faac2675aa89e2176d2eec7d8R9 uses rogue. Yes. It should be rouge.

srowen · 2019-11-26T22:36:08Z

If you haven't already opened a PR, I'll fix that.

### What changes were proposed in this pull request? This PR aims to fix a type; `rogue` -> `rouge` . This is a follow-up of #26521. ### Why are the changes needed? To support `Python 3`, we upgraded from `pygments` to `rouge`. ### Does this PR introduce any user-facing change? No. (This is for only document generation.) ### How was this patch tested? Manually. ``` $ docker build -t test dev/create-release/spark-rm/ ... 1 gem installed Successfully installed rouge-3.13.0 Parsing documentation for rouge-3.13.0 Installing ri documentation for rouge-3.13.0 Done installing documentation for rouge after 4 seconds 1 gem installed Removing intermediate container 9bd8707d9e84 ---> a18b2f6b0bb9 ... ``` Closes #26686 from dongjoon-hyun/SPARK-28752. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

HyukjinKwon · 2019-11-26T23:14:20Z

Oops ..

### What changes were proposed in this pull request? This PR aims to fix a type; `rogue` -> `rouge` . This is a follow-up of apache#26521. ### Why are the changes needed? To support `Python 3`, we upgraded from `pygments` to `rouge`. ### Does this PR introduce any user-facing change? No. (This is for only document generation.) ### How was this patch tested? Manually. ``` $ docker build -t test dev/create-release/spark-rm/ ... 1 gem installed Successfully installed rouge-3.13.0 Parsing documentation for rouge-3.13.0 Installing ri documentation for rouge-3.13.0 Done installing documentation for rouge after 4 seconds 1 gem installed Removing intermediate container 9bd8707d9e84 ---> a18b2f6b0bb9 ... ``` Closes apache#26686 from dongjoon-hyun/SPARK-28752. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…ekyll properly via Rouge ### What changes were proposed in this pull request? This PR proposes to use Pygment compatible format by Rouge. As of #26521, we use Rouge instead of Pygment wrapper in Ruby. Rouge claims Pygment compatibility; and we should output as Pygment does. ```ruby Rouge::Formatters::HTMLPygments.new(formatter) ``` wraps codes with `<div class="highlight"><pre>...` properly. ### Why are the changes needed? To keep the documentation pretty and not broken. ### Does this PR introduce any user-facing change? Theoretically, no. This is rather a regression fix in documentation (that happens only by #26521 in master). See the malformed doc in preview - https://spark.apache.org/docs/3.0.0-preview2/sql-pyspark-pandas-with-arrow.html ### How was this patch tested? Manually built the doc. **Before:** ![Screen Shot 2020-01-13 at 10 21 28 AM](https://user-images.githubusercontent.com/6477701/72229159-ba766a80-35ef-11ea-9a5d-9583448e7c1c.png) **After:** ![Screen Shot 2020-01-13 at 10 26 33 AM](https://user-images.githubusercontent.com/6477701/72229157-b34f5c80-35ef-11ea-8b3a-492e8aa0f82a.png) Closes #27182 from HyukjinKwon/SPARK-28752-followup. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

Documentation build to support Python 3

2b142b3

HyukjinKwon commented Nov 14, 2019

View reviewed changes

srowen reviewed Nov 14, 2019

View reviewed changes

dongjoon-hyun added the PYSPARK label Nov 15, 2019

HyukjinKwon added BUILD DOCUMENTATION and removed PYSPARK labels Nov 15, 2019

HyukjinKwon closed this in d1ac25b Nov 15, 2019

HyukjinKwon mentioned this pull request Nov 22, 2019

Use rogue instead of pygments.rb which is not Python 3 compatible apache/spark-website#233

Merged

dongjoon-hyun mentioned this pull request Nov 26, 2019

[SPARK-28752][BUILD][FOLLOWUP] Fix to install rouge instead of rogue #26686

Closed

HyukjinKwon mentioned this pull request Jan 13, 2020

[SPARK-28752][BUILD][DOCS][FOLLOW-UP] Render examples imported from Jekyll properly via Rouge #27182

Closed

HyukjinKwon deleted the SPARK-28752 branch March 3, 2020 01:17

HyukjinKwon mentioned this pull request Dec 30, 2020

[SPARK-33927][BUILD] Fix Dockerfile for Spark release to work #30971

Closed

[SPARK-28752][BUILD][DOCS] Documentation build to support Python 3 #26521

[SPARK-28752][BUILD][DOCS] Documentation build to support Python 3 #26521

Uh oh!

Conversation

HyukjinKwon commented Nov 14, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon commented Nov 14, 2019

Uh oh!

HyukjinKwon commented Nov 14, 2019

Uh oh!

HyukjinKwon Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Nov 14, 2019

Choose a reason for hiding this comment

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

srowen Nov 14, 2019

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Nov 14, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 14, 2019

Uh oh!

HyukjinKwon commented Nov 15, 2019

Uh oh!

HyukjinKwon commented Nov 15, 2019

Uh oh!

dongjoon-hyun commented Nov 15, 2019

Uh oh!

dongjoon-hyun commented Nov 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Nov 26, 2019

Uh oh!

srowen commented Nov 26, 2019

Uh oh!

jiangxb1987 commented Nov 26, 2019

Uh oh!

dongjoon-hyun commented Nov 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Nov 26, 2019

Uh oh!

HyukjinKwon commented Nov 26, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HyukjinKwon Nov 14, 2019 •

edited

Loading

dongjoon-hyun commented Nov 26, 2019 •

edited

Loading

dongjoon-hyun commented Nov 26, 2019 •

edited

Loading