Document Jupyter Notebook execution and data validation

Author: JohT

COMMANDS.md

@@ -4,6 +4,7 @@
 
 An analysis is started with the script [analyze.sh](./scripts/analysis/analyze.sh).
 To run all analysis steps simple execute the following command:
+
 ```shell
 ./../../scripts/analysis/analyze.sh
 ```
@@ -55,7 +56,7 @@ Note: Generating a PDF from a Jupyter notebook using [nbconvert](https://nbconve
 ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION=true ./../../scripts/analysis/analyze.sh
 ```
 
-#### Setup everything to explore the graph manually
+#### Only run setup and explore the Graph manually
 
 To prepare everything for analysis including installation, configuration and preparation queries to explore the graph manually
 without report generation use this command:
@@ -200,7 +201,7 @@ Query parameters can be added as arguments after the file name. Here is an examp
 ./scripts/executeQuery.sh ./cypher/Get_Graph_Data_Science_Library_Version.cypher a=1
 ```
 
-### executeQueryFunctions
+### [executeQueryFunctions.sh](./scripts/executeQueryFunctions.sh)
 
 The script [executeQueryFunctions.sh](./scripts/executeQueryFunctions.sh) contains functions to simplify the
 call of [executeQuery.sh](./scripts/executeQuery.sh) for different purposes. For example, `execute_cypher_summarized`
@@ -221,7 +222,41 @@ Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Gra
 
 ## Jupyter Notebook
 
-### Commands
+### Create a report simplified with [executeJupyterNotebookReport.sh](./scripts/executeJupyterNotebookReport.sh)
+
+The script [executeJupyterNotebookReport.sh](./scripts/executeJupyterNotebookReport.sh) combines:
+
+- creating a directory within the "reports" directory
+- data availability validation using [executeQueryFunctions.sh](#executequeryfunctionssh)
+- executing and converting the given Notebook using [executeJupyterNotebook.sh](#execute-a-notebook-simplified-with-executejupyternotebooksh)
+
+Here is an example on how to run the report [Wordcloud.ipynb](./jupyter/Wordcloud.ipynb):
+
+```shell
+./scripts/executeJupyterNotebookReport.sh  --jupyterNotebook Wordcloud.ipynb
+```
+
+#### Data Availability Validation
+
+[Jupyter Notebooks](https://jupyter.org) can have additional custom tags within their [metadata section](https://ipython.readthedocs.io/en/3.x/notebook/nbformat.html#metadata). Opening these files with a text editor unveils that typically at the end of the file. Some editors also support editing them directly. Here, the optional metadata property `code_graph_analysis_pipeline_data_validation` is used to specify which data validation query in the [cypher/Validation](./cypher/Validation/) directory should be used. Without this property, the data validation step is skipped. If a validation is specified, it will be executed before the Jupyter Notebook is executed. If the query has at least one result, the validation is seen as successful. Otherwise, the Jupyter Notebook will not be executed.
+
+This is helpful for Jupyter Notebook reports that are specific to a programming language or other specific data prerequisites. The Notebook will be skipped if there is no data available which would otherwise lead to confusing and distracting reports with empty tables and figures.
+
+You can search the messages `Validation succeeded` or `Validation failed` inside the log to get detailed information which Notebook had been skipped for which reason.
+
+### Execute a Notebook simplified with [executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh)
+
+[executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh) contains everything that is needed to execute a Jupyter Notebook in the command line and convert it to different formats like Markdown and PDF (optionally). It takes care of [setting up the environment](#manually-setup-the-environment-using-conda) and [uses nbconvert](#executing-jupyter-notebooks-with-nbconvert) to execute the notebook and convert it to other file formats under the hood.
+
+Here is an example on how to use [executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh) to for example run [Wordcloud.ipynb](./jupyter/Wordcloud.ipynb):
+
+```shell
+./scripts/executeJupyterNotebook.sh ./jupyter/Wordcloud.ipynb
+```
+
+### Manually setup the environment using [Conda](https://conda.io)
+
+[Conda](https://conda.io) provides package, dependency, and environment management for any language. Here, it is used to setup the environment for Juypter Notebooks.
 
 - Setup environment
 
@@ -249,6 +284,10 @@ Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Gra
   conda env export --from-history --name codegraph | grep -v "^prefix: " > codegraph-environment.yml
   ```
 
+### Executing Jupyter Notebooks with [nbconvert](https://nbconvert.readthedocs.io)
+
+[nbconvert](https://nbconvert.readthedocs.io) converts Jupyter Notebooks to other static formats including HTML, LaTeX, PDF, Markdown, reStructuredText, and more.
+
 - Install pandoc used by nbconvert for LaTeX support (Mac)
 
   ```shell
@@ -273,23 +312,21 @@ Use [stopNeo4j.sh](./scripts/stopNeo4j.sh) to stop the locally running Neo4j Gra
   jupyter nbconvert --to pdf ./jupyter/first-neo4j-tryout.nbconvert.ipynb
   ```
 
-- Shell script to execute and convert a Jupyter notebook file
-  
-  Use [executeJupyterNotebook.sh](./scripts/executeJupyterNotebook.sh) like this:
-
-  ```shell
-  ./scripts/executeJupyterNotebook.sh ./jupyter/first-neo4j-tryout.ipynb
-  ```
-
 ## References
 
-- [Managing environments with Conda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)
+- [Conda](https://conda.io)
+- [jQAssistant](https://jqassistant.org/get-started)
+- [Jupyter Notebook](https://jupyter.org)
 - [Jupyter Notebook - Using as a command line tool](https://nbconvert.readthedocs.io/en/latest/usage.html)
 - [Jupyter Notebook - Installing TeX for PDF conversion](https://nbconvert.readthedocs.io/en/latest/install.html#installing-tex)
-- [Integrate Neo4j with Jupyter notebook](https://medium.com/@technologydata25/connect-neo4j-to-jupyter-notebook-c178f716d6d5)
+- [Jupyter Notebook Format - Metadata](https://ipython.readthedocs.io/en/3.x/notebook/nbformat.html#metadata)
+- [Integrate Neo4j with Jupyter Notebook](https://medium.com/@technologydata25/connect-neo4j-to-jupyter-notebook-c178f716d6d5)
 - [Hello World](https://nicolewhite.github.io/neo4j-jupyter/hello-world.html)
+- [Managing environments with Conda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)
+- [Neo4j - Download](https://neo4j.com/download-center)
+- [Neo4j - HTTP API](https://neo4j.com/docs/http-api/current/query)
 - [py2neo](https://pypi.org/project/py2neo/)
-- [The Py2neo Handbook](https://py2neo.org/2021.1/)
+- [The Py2neo Handbook](https://py2neo.org/2021.1)
 - [How to Use Conda With Github Actions](https://autobencoder.com/2020-08-24-conda-actions)
 - [Older database download link (neo4j community)](https://community.neo4j.com/t/older-database-download-link/43334/9)
 

README.md

@@ -179,10 +179,10 @@ The [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.ym
   👉 Create a new artifacts download script in the [scripts/downloader](./scripts/downloader/) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference.  
   👉 Run the script separately before executing [analyze.sh](./scripts/analysis/analyze.sh) also in the [pipeline](./.github/workflows/java-code-analysis.yml).
 
-- How can i trigger a full rescan of all artifacts?  
+- How can i trigger a full re-scan of all artifacts?  
   👉 Delete the file `artifactsChangeDetectionHash.txt` in the `artifacts` directory.
 
-- How can PDF generation for Jupyter Notebooks be enabled (depends on chromium, takes more time)?  
+- How can i enable PDF generation for Jupyter Notebooks (depends on chromium, takes more time)?  
   👉 Set environment variable `ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION` to anything except an empty string. Example:  
 
   ```shell
@@ -195,6 +195,10 @@ The [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.ym
   ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION=true ./../../scripts/analysis/analyze.sh
   ```
 
+- Why are some Jupyter Notebook reports skipped?
+  👉 The custom Jupyter Notebook metadata property `code_graph_analysis_pipeline_data_validation` can be set to choose a query from [cypher/Validation](./cypher/Validation) that will be executed preliminary to the notebook. If the query leads to at least one result, the validation succeeds and the notebook will be run. If the query leads to no result, the notebook will be skipped.
+  For more details see [Data Availability Validation](./COMMANDS.md#data-availability-validation).
+
 ## 🕸 Web References
 
 - [Graph Data Science 101: Understanding Graphs and Graph Data Science](https://techfirst.medium.com/graph-data-science-101-understanding-graphs-and-graph-data-science-c25055a9db01)