added initial version of document generator #140

armando-fandango · 2019-04-04T18:10:35Z

Adding initial version of Document Generator
modified from tensorflow/tools/docs
The docs can be created and pushed to /docs
Once this pull request is accepted, we can add the command to "generate and push the docs" to build pipeline.

googlebot · 2019-04-04T18:10:37Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

armando-fandango · 2019-04-04T18:21:35Z

Reopening to fix CLA

…tools/docs

googlebot · 2019-04-05T17:02:00Z

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ Googlers: Go here for more info.

ewilderj · 2019-04-05T17:02:21Z

Overrode the CLA status, submitter used wrong email address in error.

seanpmorgan

Thanks @armando-fandango . Generally looks good but some minor things we need to do in order to integrate it with our pipeline

seanpmorgan · 2019-04-08T16:50:13Z

tools/docs/generate2.py

+
+Requires a local installation of:
+  Docs Generator - pip install git+https://github.com/tensorflow/docs
+  tensorflow_addons


So the reason there needed to be a local install of tensorflow (in the tensorflow/docs/generate2.py) was because the script used an installed package to generate docs (and didn't require a heavy TF rebuild).

Thats a bit hacky imo and we should integrate doc generation with our bazel build / testing

something like configure.sh? Sean, note that it's tensorflow-doc, rather than tensorflow package.

Yeah we could automate the install/generation with a shell script, but we should be able to build the docs using bazel without the need to pip install the package first (which would require a nightly build that we do not have yet).

See the "note" in this section for the reason why they used a pip install instead of just using the source code: https://www.tensorflow.org/community/contribute/documentation#python

@seanpmorgan @facaiy : Two questions here:

Should I still rely on doc s generator in tensorflow_docs .. or that is not an option ?

Should the docs be generated from installed version of addons or from source code version of addons ? if Source code version then I assume it would always be from master branch, Or do we want to keep docs for multiple versions like in TF API docs you can select previous version and see docs generated for previous versions too.

To import docs into tensorflow.org, each "subsite" has a config file that specifies the branch for narrative docs and the branch for API docs.
Typically, narrative docs are set to 'master' since it's easier if browsing the GitHub repo looks close to the site (minus changes from last publish date).
For API docs, we typically publish from a release branch (though it could be a tag). That way you can point to older versions of generated docs in GitHub and the Markdown is still readable. The downside is that you need to bump the (internal) config file whenever you make a release and want to publish it. But we can come up with process for that, or just ping me.

tools/docs/generate2.py

seanpmorgan · 2019-04-08T16:52:07Z

tools/docs/generate2.py

+"""A tool to generate api_docs for TensorFlow Addons.
+
+```
+Run from root folder of source code


So I believe this will fail if we run as a python file from root. Throws a .so file not found error because they are not built in the repo. We'll need to incorporate this into the bazel BUILD so that docs are generated after the files are compiled.

There seems at least two solutions:

build and install tf_addons in local machine, and then run generate.py manually.

like build_pip_pkg.sh, we create a shell script which runs generate.py, and write a sh_binary target in bazel BUILD.
pros: automatically compile c++ codes
cons: cannot pass argument, say --output_dir, --code_url_prefix

@MarkDaoust Hi, Mark, can you introduce how tensorflow team use generate.py file?

Yes a best practice from docs team would be best.

The third option would be:
3. Run the doc generator from a bazel build command. Which is what was insinuated as possible from the note linked above (i think)

#46

I would suggest to run it from Bazel build and then check in the docs into git repo from there. One of the points in discussion in #46 was that once we push docs in /docs folder in our git repo, then tensorflow core team would be able to pull them into community docs from there.

build and install tf_addons in local machine, and then run generate.py manually.

Run the doc generator from a bazel build command.

I think either of these work, the generator just wants access to the module as a user would see it.
How you set it up is only a question of whether you want to use bazel to generate your docs or not.
If you don't do anything too out of the ordinary, you may be able to do (1) even if you have the bazel BUILD files setup for (3).

Bazel build and then check in the docs into git repo from there. ... then tensorflow core team would be able to pull them into community docs from there.

Yes, that would be the least-friction path for us. It also gives your users somewhere to read the docs independent of what happens with this tensorflow.org integration.

seanpmorgan · 2019-04-08T16:58:24Z

tools/docs/generate2.py

+from tensorflow.python.util import tf_inspect
+
+# Use tensorflow's `tf_inspect`, which is aware of `tf_decorator`.
+parser.tf_inspect = tf_inspect


Can we confirm if this is required? I'm not aware of us using tf_decorator, but it can probably be empirically checked. If we don't need to use this that's great because it's leveraging a private API

Sean, it seems that tensorflow_docs cannot recognize those imported modules/function, such as keras_utils, absolute_import etc. Not sure whether that's one reason to use tf_export in tensorflow. cc @lamberta

@MarkDaoust

It will be good to find out a way to link them back to the documentation in TF core.

Can we confirm if this is required? I'm not aware of us using tf_decorator, but it can probably be empirically checked.

Correct, this line only does anything for you if you're using tf_decorator.

it seems that tensorflow_docs cannot recognize those imported modules/function, such as keras_utils, absolute_import etc. Not sure whether that's one reason to use tf_export in tensorflow.

I'm not sure I understand the question here what is the problem, what are we trying to acomplish?.
I should block the generator from ever mentioning "absolute_import", "division", "print_function".

Note that one of the functions of the base_dir argument is to allow it to filter out objects which are imported by your package, but not part of your package. No docs are generated for objects defined in files outside of base_dir.

seanpmorgan · 2019-04-08T16:58:52Z

tools/docs/generate2.py

+
+
+def build_docs(output_dir, code_url_prefix, search_hints=True):
+    """Build api docs for tensorflow v2.


for tensorflow_addons

seanpmorgan · 2019-04-08T16:59:22Z

tools/docs/generate2.py

+        "https://github.com/tensorflow/addons/tree/master/tensorflow_addons")
+
+    doc_generator = generate_lib.DocGenerator(
+        root_title="TensorFlow Addons 2.0 Preview",


Just TensorFlow Addons -- no 2.0 preview

thanks, yes will fix it.

seanpmorgan · 2019-04-08T16:59:59Z

tools/docs/generate2_test.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for addons.tools.docs.generate2."""


We'll need to integrate this with bazel so this test actually gets ran

I just moved it as it is and fixed it, but if we are generating docs through CI pipeline then we can get rid of this test file completely. What do you think ?

seanpmorgan · 2019-04-08T17:00:21Z

tools/docs/generate2_test.py

+import shutil
+
+import tensorflow as tf
+from tensorflow.python.platform import googletest


Looks like this is just used for a temp dir? Could we replace this with something that doesn't use a private API

as per above comment, I believe we can eliminate this test file completely.

facaiy · 2019-04-11T06:58:04Z

tools/docs/generate2.py

+    code_url_prefixes = (
+        code_url_prefix,
+        # External packages source repositories
+        "https://github.com/tensorflow/addons/tree/master/tensorflow_addons")


I think we should remove the line, and run the script like: python tools/docs/generate2.py --output_dir=do cs/ --code_url_prefix=https://github.com/tensorflow/addons/tree/master/tensorflow_addons

code URL prefix would be fixed. What would be the benefit of moving it as an arg?

Unless you're trying to document multiple installed pacakages you likely want a single string for both base_dir and code_url_prefix.

I think what you want is:

base_dir = path.dirname(tfa.__file__) code_url_prefix = "https://github.com/tensorflow/addons/tree/master/tensorflow_addons"

Thanks Mark, Yan, Sean, will update with these changes in this week.

Unless you're trying to document multiple installed pacakages you likely want a single string for both base_dir and code_url_prefix.

I think what you want is:

base_dir = path.dirname(tfa.__file__) code_url_prefix = "https://github.com/tensorflow/addons/tree/master/tensorflow_addons"

For 0.1 version, we should redirect link to r0.1 branch, instead of master branch. That's why I think we need code_url_prefix argument. What do you think?

@facaiy As I also asked above, do we want to make the doc generator such that it generates docs for all tagged versions in the github repo, like in TF API you can view docs for all previous versions too, or do we want to just keep the docs for latest master branch ?

facaiy · 2019-04-11T06:58:53Z

tools/docs/generate2.py

+
+Requires a local installation of:
+  Docs Generator - pip install git+https://github.com/tensorflow/docs
+  tensorflow_addons


something like configure.sh? Sean, note that it's tensorflow-doc, rather than tensorflow package.

facaiy · 2019-04-11T08:44:44Z

tools/docs/generate2.py

+      search_hints: Bool. Include meta-data search hints at the top of file.
+    """
+    try:
+        doc_controls.do_not_generate_docs(tfa.tools)


tfa.utils, rather than tfa.tools?

utils we want to generate documentation of.. right? Just not tools package which only has install and doc generator stuff.

And yes we would need to install tensorflow docs from configure.sh as generate2.py relies on code from that repo.

facaiy · 2019-04-11T08:55:44Z

tools/docs/generate2.py

+from tensorflow.python.util import tf_inspect
+
+# Use tensorflow's `tf_inspect`, which is aware of `tf_decorator`.
+parser.tf_inspect = tf_inspect


Sean, it seems that tensorflow_docs cannot recognize those imported modules/function, such as keras_utils, absolute_import etc. Not sure whether that's one reason to use tf_export in tensorflow. cc @lamberta

MarkDaoust

LMK if you have any questions, need clarifications, or fixes to the docs tools.

MarkDaoust · 2019-04-11T22:02:05Z

tools/docs/generate2.py

+"""A tool to generate api_docs for TensorFlow Addons.
+
+```
+Run from root folder of source code


build and install tf_addons in local machine, and then run generate.py manually.

Run the doc generator from a bazel build command.

I think either of these work, the generator just wants access to the module as a user would see it.
How you set it up is only a question of whether you want to use bazel to generate your docs or not.
If you don't do anything too out of the ordinary, you may be able to do (1) even if you have the bazel BUILD files setup for (3).

Bazel build and then check in the docs into git repo from there. ... then tensorflow core team would be able to pull them into community docs from there.

Yes, that would be the least-friction path for us. It also gives your users somewhere to read the docs independent of what happens with this tensorflow.org integration.

MarkDaoust · 2019-04-11T22:22:51Z

tools/docs/generate2.py

+parser.tf_inspect = tf_inspect
+
+# So patch `tfa.__all__` to list everything.
+tfa.__all__ = [item_name for item_name, value in tf_inspect.getmembers(tfa)]


This line is likely not necessary.

The doc generator respects a module's __all__ but will document everything if all is not set. I only added this for tensorflow/tensorflow because there was a miss-match between what was in all and what we wanted to document.

thanks.. will remove it.

MarkDaoust · 2019-04-11T22:22:52Z

tools/docs/generate2.py

+from tensorflow.python.util import tf_inspect
+
+# Use tensorflow's `tf_inspect`, which is aware of `tf_decorator`.
+parser.tf_inspect = tf_inspect


Can we confirm if this is required? I'm not aware of us using tf_decorator, but it can probably be empirically checked.

Correct, this line only does anything for you if you're using tf_decorator.

it seems that tensorflow_docs cannot recognize those imported modules/function, such as keras_utils, absolute_import etc. Not sure whether that's one reason to use tf_export in tensorflow.

I'm not sure I understand the question here what is the problem, what are we trying to acomplish?.
I should block the generator from ever mentioning "absolute_import", "division", "print_function".

Note that one of the functions of the base_dir argument is to allow it to filter out objects which are imported by your package, but not part of your package. No docs are generated for objects defined in files outside of base_dir.

MarkDaoust · 2019-04-11T22:40:06Z

tools/docs/generate2.py

+
+FLAGS = flags.FLAGS
+
+flags.DEFINE_string(


I think it's safe to just hard code the code_url_prefix

yes, I agree, will do that.

MarkDaoust · 2019-04-11T22:47:28Z

tools/docs/generate2.py

+    code_url_prefixes = (
+        code_url_prefix,
+        # External packages source repositories
+        "https://github.com/tensorflow/addons/tree/master/tensorflow_addons")


Unless you're trying to document multiple installed pacakages you likely want a single string for both base_dir and code_url_prefix.

I think what you want is:

base_dir = path.dirname(tfa.__file__) code_url_prefix = "https://github.com/tensorflow/addons/tree/master/tensorflow_addons"

facaiy · 2019-04-16T10:11:11Z

@seanpmorgan Sean, after discussion with @armando-fandango : 1) We plan to address current comments and then make sure doc generator works nicely from command line; 2) and then someone else, who're familiar enough with bazel, can take over from there in a separate PR for bazel integration.

What do you think?

seanpmorgan · 2019-04-16T12:24:22Z

@seanpmorgan Sean, after discussion with @armando-fandango : 1) We plan to address current comments and then make sure doc generator works nicely from command line; 2) and then someone else, who're familiar enough with bazel, can take over from there in a separate PR for bazel integration.

What do you think?

Sounds great. Thanks everyone for the reviews and iteration

lamberta · 2019-04-18T21:55:09Z

FYI, @MarkDaoust added an example build_docs.py script to use the API generator: https://github.com/tensorflow/docs/blob/master/tools/templates/build_docs.py

Teams typically keep this in their tools/ directory (or wherever you want), and then it outputs to docs/api_docs/python/ (for example, see TF Datasets). This is the path we need to add to out importer config to bring it into the site.

googlebot · 2019-04-21T22:41:59Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

armando-fandango · 2019-04-21T22:45:56Z

closing this and creating a fresh pull request as the API generator is changed.

googlebot added the cla: no label Apr 4, 2019

armando-fandango closed this Apr 4, 2019

armando-fandango reopened this Apr 4, 2019

added initial version of document generator modified from tensorflow/…

c016c3e

…tools/docs

ewilderj added cla: yes and removed cla: no labels Apr 5, 2019

seanpmorgan requested changes Apr 8, 2019

View reviewed changes

facaiy reviewed Apr 11, 2019

View reviewed changes

facaiy mentioned this pull request Apr 11, 2019

Remove __init__.py in custom_ops directory #163

Closed

seanpmorgan added the documentation label Apr 11, 2019

MarkDaoust reviewed Apr 11, 2019

View reviewed changes

facaiy self-assigned this Apr 15, 2019

added API docs generator in build_docs.py

a97bc81

armando-fandango requested a review from a team as a code owner April 21, 2019 22:41

googlebot added cla: no and removed cla: yes labels Apr 21, 2019

armando-fandango closed this Apr 21, 2019

armando-fandango deleted the 46-add-docs-generator branch April 21, 2019 22:50

facaiy mentioned this pull request Apr 25, 2019

add bazel command to generate API docs #206

Merged

seanpmorgan mentioned this pull request May 1, 2019

Link API docs with with tensorflow.org #226

Closed



		def build_docs(output_dir, code_url_prefix, search_hints=True):
		"""Build api docs for tensorflow v2.


		FLAGS = flags.FLAGS

		flags.DEFINE_string(

added initial version of document generator #140

added initial version of document generator #140

Uh oh!

Conversation

armando-fandango commented Apr 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

googlebot commented Apr 4, 2019

Uh oh!

armando-fandango commented Apr 4, 2019

Uh oh!

googlebot commented Apr 5, 2019

Uh oh!

ewilderj commented Apr 5, 2019

Uh oh!

seanpmorgan left a comment

Choose a reason for hiding this comment

Uh oh!

seanpmorgan Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seanpmorgan Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facaiy Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facaiy Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facaiy Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seanpmorgan Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

armando-fandango commented Apr 4, 2019 •

edited

Loading

seanpmorgan Apr 8, 2019 •

edited

Loading

seanpmorgan Apr 11, 2019 •

edited

Loading

facaiy Apr 11, 2019 •

edited

Loading

facaiy Apr 11, 2019 •

edited

Loading

facaiy Apr 11, 2019 •

edited

Loading

seanpmorgan Apr 8, 2019 •

edited

Loading

facaiy Apr 11, 2019 •

edited

Loading