Skip to content

[SPARK-28752][BUILD][DOCS] Documentation build to support Python 3 #26521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-28752][BUILD][DOCS] Documentation build to support Python 3 #26521

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions dev/create-release/spark-rm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ ENV DEBCONF_NONINTERACTIVE_SEEN true
ARG APT_INSTALL="apt-get install --no-install-recommends -y"

ARG BASE_PIP_PKGS="setuptools wheel virtualenv"
ARG PIP_PKGS="pyopenssl pypandoc numpy pygments sphinx"
ARG PIP_PKGS="pyopenssl pypandoc numpy sphinx"
Copy link
Member Author

@HyukjinKwon HyukjinKwon Nov 14, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sphinx still needs pygments but it's in sphinx's dependency. I don't think we should explicitly list since Spark doesn't directly use it (but sphinx uses it).

Copy link
Member Author

@HyukjinKwon HyukjinKwon Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @vanzin too


# Install extra needed repos and refresh.
# - CRAN repo
Expand Down Expand Up @@ -80,7 +80,7 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
$APT_INSTALL ruby2.3 ruby2.3-dev mkdocs && \
gem install jekyll --no-rdoc --no-ri -v 3.8.6 && \
gem install jekyll-redirect-from && \
gem install pygments.rb
gem install rogue

WORKDIR /opt/spark-rm/output

Expand Down
3 changes: 1 addition & 2 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,7 @@ You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/
installed. Also install the following libraries:
Copy link
Member

@srowen srowen Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to mention Python 3 is required here? or does it still work with Python 2?

Copy link
Member Author

@HyukjinKwon HyukjinKwon Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it still works with Python 2 because it seems not using Python at all. But I guess we should better stick to Python 3 ..


```sh
$ sudo gem install jekyll jekyll-redirect-from pygments.rb
$ sudo pip install Pygments
$ sudo gem install jekyll jekyll-redirect-from rouge
# Following is needed only for generating API docs
$ sudo pip install sphinx pypandoc mkdocs
$ sudo Rscript -e 'install.packages(c("knitr", "devtools", "rmarkdown"), repos="https://cloud.r-project.org/")'
Expand Down
2 changes: 1 addition & 1 deletion docs/_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
highlighter: pygments
highlighter: rouge
markdown: kramdown
gems:
- jekyll-redirect-from
Expand Down
6 changes: 4 additions & 2 deletions docs/_plugins/include_example.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#

require 'liquid'
require 'pygments'
require 'rouge'

module Jekyll
class IncludeExampleTag < Liquid::Tag
Expand Down Expand Up @@ -56,7 +56,9 @@ def render(context)
end
code = select_lines(code)

rendered_code = Pygments.highlight(code, :lexer => @lang)
formatter = Rouge::Formatters::HTML.new
lexer = Rouge::Lexer.find(@lang)
rendered_code = formatter.format(lexer.lex(code))

hint = "<div><small>Find full example code at " \
"\"examples/src/main/#{snippet_file}\" in the Spark repo.</small></div>"
Expand Down
6 changes: 5 additions & 1 deletion docs/css/pygments-default.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Also, I was thrown off for a while at first when I was using markdown
code block inside my {% highlight scala %} ... {% endhighlight %} tags
(I was using 4 spaces for this), when it turns out that pygments will
insert the code (or pre?) tags for you.

Note that due to Python 3 compatibility in the project, now we use
Rouge which claims Pygments compatibility, instead of pygments.rb which
does not support Python 3. See SPARK-28752.
*/

.hll { background-color: #ffffcc }
Expand Down Expand Up @@ -73,4 +77,4 @@ insert the code (or pre?) tags for you.
.vc { color: #bb60d5 } /* Name.Variable.Class */
.vg { color: #bb60d5 } /* Name.Variable.Global */
.vi { color: #bb60d5 } /* Name.Variable.Instance */
.il { color: #40a070 } /* Literal.Number.Integer.Long */
.il { color: #40a070 } /* Literal.Number.Integer.Long */