Document Jupyter Notebook execution and data validation

Author: JohT

COMMANDS.md

@@ -4,6 +4,7 @@
 
 An analysis is started with the script [analyze.sh](./scripts/analysis/analyze.sh).
 To run all analysis steps simple execute the following command:
+
 ```shell
 ./../../scripts/analysis/analyze.sh
 ```
@@ -55,7 +56,7 @@ Note: Generating a PDF from a Jupyter notebook using [nbconvert](https://nbconve
 ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION=true ./../../scripts/analysis/analyze.sh
 ```
 
-#### Setup everything to explore the graph manually
+#### Only run setup and explore the Graph manually
 
 To prepare everything for analysis including installation, configuration and preparation queries to explore the graph manually
 without report generation use this command:
@@ -200,7 +201,7 @@ Query parameters can be added as arguments after the file name. Here is an examp
 ./scripts/executeQuery.sh ./cypher/Get_Graph_Data_Science_Library_Version.cypher a=1
 ```
 
-### executeQueryFunctions
+### [executeQueryFunctions.sh](./scripts/executeQueryFunctions.sh)
 
 The script [executeQueryFunctions.sh](./scripts/executeQueryFunctions.sh) contains functions to simplify the
 call of [executeQuery.sh](./scripts/executeQuery.sh) for different purposes. For example, `execute_cypher_summarized`
@@ -221,7 +222,35 @@ Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Gra
 
 ## Jupyter Notebook
 
-### Commands
+### Create a report simplified with [executeJupyterNotebookReport.sh](./scripts/executeJupyterNotebookReport.sh)
+
+[executeJupyterNotebookReport.sh](./scripts/executeJupyterNotebookReport.sh) includes everything from creating a directory within the "reports" directory to data availability validation and executing and converting the given Notebook. This is the all in one script that is also used inside the pipeline. Under the hood it uses [executeJupyterNotebook.sh](#execute-a-notebook-simplified-with-executejupyternotebooksh) to execute the Notebook and [executeQueryFunctions.sh](#executequeryfunctionssh) to query the Database for data availability validation.
+
+Here is an example on how to use [executeJupyterNotebookReport.sh](./scripts/executeJupyterNotebookReport.sh) to for example run the report [Wordcloud.ipynb](./jupyter/Wordcloud.ipynb):
+
+```shell
+./scripts/executeJupyterNotebookReport.sh  --jupyterNotebook Wordcloud.ipynb
+```
+
+#### Data Availability Validation
+
+Jupyter notebooks can have additional custom tags within their "metadata" section. Opening these files with a text editor unveils that typically at the end of the file. Some editors also support editing them directly. Here, the optional metadata property `code_graph_analysis_pipeline_data_validation` is used to specify which data validation query in the [cypher/Validation](./cypher/Validation/) directory should be used. Without this property, the data validation step is skipped. If a validation is specified, it will be executed before the Jupyter Notebook is executed. If the query has at least one result, the validation is seen as successful. Otherwise, the Jupyter Notebook will not be executed.
+
+This is helpful for Juypter Notebook reports that are specific to a programming language or other specific data prerequisites. The Notebook will be skipped if there is no data available which would otherwise lead to confusing and distracting reports with empty tables and figures.
+
+### Execute a Notebook simplified with [executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh)
+
+[executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh) contains everything that is needed to execute a Jupyter Notebook in the command line and convert it to different formats like Markdown and PDF (optionally). It takes care of [setting up the environment](#manually-setup-the-environment-using-conda) and [uses nbconvert](#executing-jupyter-notebooks-with-nbconvert) to execute the notebook and convert it to other file formats under the hood.
+
+Here is an example on how to use [executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh) to for example run [Wordcloud.ipynb](./jupyter/Wordcloud.ipynb):
+
+```shell
+./scripts/executeJupyterNotebook.sh ./jupyter/Wordcloud.ipynb
+```
+
+### Manually setup the environment using [Conda](https://conda.io)
+
+[Conda](https://conda.io) provides package, dependency, and environment management for any language. Here, it is used to setup the environment for Juypter Notebooks.
 
 - Setup environment
 
@@ -249,6 +278,10 @@ Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Gra
   conda env export --from-history --name codegraph | grep -v "^prefix: " > codegraph-environment.yml
   ```
 
+### Executing Jupyter Notebooks with [nbconvert](https://nbconvert.readthedocs.io)
+
+[nbconvert](https://nbconvert.readthedocs.io) converts Jupyter Notebooks to other static formats including HTML, LaTeX, PDF, Markdown, reStructuredText, and more.
+
 - Install pandoc used by nbconvert for LaTeX support (Mac)
 
   ```shell
@@ -273,14 +306,6 @@ Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Gra
   jupyter nbconvert --to pdf ./jupyter/first-neo4j-tryout.nbconvert.ipynb
   ```
 
-- Shell script to execute and convert a Jupyter notebook file
-  
-  Use [executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh) like this:
-
-  ```shell
-  ./scripts/executeJupyterNotebook.sh ./jupyter/first-neo4j-tryout.ipynb
-  ```
-
 ## References
 
 - [Managing environments with Conda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)