diff --git a/.github/workflows/build-sphinx.yml b/.github/workflows/build-sphinx.yml index 1d65411..e62df68 100644 --- a/.github/workflows/build-sphinx.yml +++ b/.github/workflows/build-sphinx.yml @@ -25,7 +25,7 @@ jobs: - name: Install floatCSEP run: | - pip install sphinx sphinx-autoapi sphinx-gallery sphinx-rtd-theme + pip install -r requirements_dev.txt pip install --no-deps -e . python -c "import floatcsep; print('Version: ', floatcsep.__version__)" diff --git a/.github/workflows/release-tutorials.yml b/.github/workflows/release-tutorials.yml new file mode 100644 index 0000000..de87c73 --- /dev/null +++ b/.github/workflows/release-tutorials.yml @@ -0,0 +1,24 @@ +name: Release Tutorials + +on: + release: + types: [ published ] + +jobs: + upload-tutorials: + runs-on: ubuntu-latest + + steps: + - name: Check out the code + uses: actions/checkout@v3 + + - name: Zip the tutorials folder + run: | + zip -r tutorials.zip tutorials/ + + - name: Upload the tutorials.zip to the release + uses: softprops/action-gh-release@v1 + with: + files: tutorials.zip + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/MANIFEST.in b/MANIFEST.in index 39e8be4..3ce7ff8 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -2,6 +2,7 @@ include README.md include CODE_OF_CONDUCT.md include requirements.txt include requirements_dev.txt +include environment.yml include setup.py include setup.cfg include pyproject.toml @@ -14,7 +15,6 @@ graft tests graft docs prune docs/_build prune docs/referenced/generated -prune docs/tutorials/ global-exclude .git* global-exclude *.pyc diff --git a/README.md b/README.md index bb7e30e..94c18b7 100644 --- a/README.md +++ b/README.md @@ -20,9 +20,11 @@ DOI

-* Set up a testing **experiment** for your earthquake forecasts using authoritative data sources and benchmarks. +* Set up a testing **experiment** for your earthquake forecasts using authoritative data sources + and benchmarks. * **Encapsulate** the complete experiment's definition and rules in a couple of lines. -* **Reproduce**, **reuse**, and **share** forecasting experiments from you, other researchers and institutions. +* **Reproduce**, **reuse**, and **share** forecasting experiments from you, other researchers + and institutions. # Table of Contents @@ -34,12 +36,14 @@ * [Contributing](#contributing) * [License](#license) - # Installing floatCSEP -The core of `floatCSEP` is built around the `pyCSEP` package (https://github.com/sceccode/pycsep), which itself contains the core dependencies. +The core of `floatCSEP` is built around the `pyCSEP` +package (https://github.com/sceccode/pycsep), which itself contains the core dependencies. -The simplest way to install `floatCSEP`, is by creating a `conda` environment (https://conda.io - checkout Anaconda or Miniconda) and install `pyCSEP` from `conda-forge` +The simplest way to install `floatCSEP`, is by creating a `conda` +environment (https://conda.io - checkout Anaconda or Miniconda) and install `pyCSEP` +from `conda-forge` ``` conda env create -n $NAME @@ -48,21 +52,29 @@ conda install -c conda-forge pycsep ``` Clone and install the floatCSEP source code using `pip` + ``` git clone https://github.com/cseptesting/floatcsep cd floatcsep pip install . ``` -Please read the [Installation](https://floatcsep.readthedocs.io/en/latest/intro/installation.html) documentation for detailed instructions and additional installation methods. +Please read +the [Installation](https://floatcsep.readthedocs.io/en/latest/intro/installation.html) +documentation for detailed instructions and additional installation methods. # Run an Experiment -Using the command terminal, navigate to an example experiment in `floatcsep/examples/` and type +Using the command terminal, navigate to an example experiment in ``floatcsep/tutorials/`` and +type + ``` floatcsep run config.yml ``` -A runtime directory will be created in a `results` folder. The experiment results can be visualized in `results/report.md`. **Check out the experiment, models and tests definition in the examples**! + +A runtime directory will be created in a `results` folder. The experiment results can be +visualized in `results/report.md`. **Check out the experiment, models and tests definition in +the tutorials**! # Important Links @@ -71,30 +83,35 @@ A runtime directory will be created in a `results` folder. The experiment result * `pyCSEP` [Github](https://github.com/sceccode/pycsep) * `pyCSEP` [Documentation](https://docs.cseptesting.org/) - # Roadmap and Known Issues -* Add report customization * Create tool to check a TD model's interface with ``floatcsep`` * Define a dependency strategy to ensure experiments' reproducibility. * Implement spatial database and HDF5 experiment storage feature -* Set up task paralellization - +* Set up task parallelization +* Document the process of an experiment deployment # Contributing -We encourage all types of contributions, from reporting bugs, suggesting enhancements, adding new features and more. Please refer to the [Contribution Guidelines](https://github.com/cseptesting/floatcsep/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/cseptesting/floatcsep/blob/main/CODE_OF_CONDUCT.md) for more information +We encourage all types of contributions, from reporting bugs, suggesting enhancements, adding +new features and more. Please refer to +the [Contribution Guidelines](https://github.com/cseptesting/floatcsep/blob/main/CONTRIBUTING.md) +and the [Code of Conduct](https://github.com/cseptesting/floatcsep/blob/main/CODE_OF_CONDUCT.md) +for more information # License -The `floatCSEP` (as well as `pyCSEP`) software is distributed under the BSD 3-Clause open-source license. Please see the [license file](https://github.com/cseptesting/floatcsep/blob/main/LICENSE) for more information. +The `floatCSEP` (as well as `pyCSEP`) software is distributed under the BSD 3-Clause open-source +license. Please see +the [license file](https://github.com/cseptesting/floatcsep/blob/main/LICENSE) for more +information. ## Support
-| | | -|:---|:---| +| | | +|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
This research was supported by the Statewide California Earthquake Center. SCEC is funded by NSF Cooperative Agreement EAR-2225216 and USGS Cooperative Agreement G24AC00072-00.
|
The work in this repository has received funding from the European Union’s Horizon research and innovation programme under grant agreements No.s 101058518 and 821115 of the projects GeoInquire and RISE.
|
diff --git a/docs/_static/custom.js b/docs/_static/custom.js new file mode 100644 index 0000000..9def49d --- /dev/null +++ b/docs/_static/custom.js @@ -0,0 +1,31 @@ +/** + * custom.js + * + * This script contains custom JavaScript modifications for the Sphinx documentation. + * It can be expanded to include additional customizations related to behavior, + * style, and functionality of the generated documentation. + * + * + * Usage: + * - Place this script in the _static directory of your Sphinx project. + * - Include it in the html_js_files configuration in conf.py to load it automatically. + * - Expand this file with other JavaScript customizations as needed. + * + * Author: Pablo Iturrieta + * Date: 28.09.2024 + */ + +document.addEventListener("DOMContentLoaded", function () { +// - Ensures that all external links open in a new tab by adding the target="_blank" +// attribute to all links with the 'external' class (automatically applied by Sphinx). +// - Adds rel="noopener noreferrer" for security purposes, ensuring the new page +// does not have access to the originating window context (prevents security risks). + // Select all external links in the documentation + const links = document.querySelectorAll('a.external'); + + // Loop through all the links and set them to open in a new tab + links.forEach(function (link) { + link.setAttribute('target', '_blank'); + link.setAttribute('rel', 'noopener noreferrer'); + }); +}); diff --git a/docs/_static/experiment_classes.png b/docs/_static/experiment_classes.png new file mode 100644 index 0000000..8b76fed Binary files /dev/null and b/docs/_static/experiment_classes.png differ diff --git a/docs/_static/float_scheme.png b/docs/_static/float_scheme.png new file mode 100644 index 0000000..ab527e7 Binary files /dev/null and b/docs/_static/float_scheme.png differ diff --git a/docs/conf.py b/docs/conf.py index 79f8e6c..d2b4e4f 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -14,9 +14,9 @@ # -- Project information ----------------------------------------------------- project = "floatCSEP" -copyright = "2022, Pablo Iturrieta" +copyright = "2024, Pablo Iturrieta" author = "Pablo Iturrieta" -release = "v0.1.0" +release = "v0.2.0" # -- General configuration --------------------------------------------------- @@ -25,13 +25,20 @@ "sphinx.ext.todo", "sphinx.ext.autosummary", "sphinx.ext.coverage", + 'sphinx_toolbox.github', + 'sphinx_toolbox.sidebar_links', "sphinx.ext.mathjax", "sphinx.ext.viewcode", "sphinx.ext.napoleon", "sphinx.ext.intersphinx", + "sphinx_copybutton", + "sphinx_design", ] -# language = 'en' +github_username = 'cseptesting' +github_repository = 'floatcsep' + +language = 'en' autosummary_generate = False autoclass_content = "both" suppress_warnings = [ @@ -51,6 +58,10 @@ "matplotlib": ("https://matplotlib.org/stable", None), "pycsep": ("https://docs.cseptesting.org/", None), } +todo_include_todos = False +copybutton_prompt_text = "$ " # Text to ignore when copying (for shell commands) +copybutton_only_copy_prompt_lines = False + # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output @@ -59,9 +70,51 @@ html_theme_options = { "display_version": True, "prev_next_buttons_location": "both", - "collapse_navigation": False, + "sticky_navigation": True, + "collapse_navigation": True, "style_nav_header_background": "#343131ff", "logo_only": True, } html_logo = "_static/floatcsep_logo.svg" -todo_include_todos = False +html_js_files = [ + "custom.js", +] + +html_context = { + "github_links": [ + ( + 'Getting help', + "https://github.com/cseptesting/floatcsep/issues" + ), + ( + 'Contributing', + "https://github.com/cseptesting/floatcsep/blob/master/CONTRIBUTING.md" + ), + ( + 'Code of Conduct', + "https://github.com/cseptesting/floatcsep/blob/master/CODE_OF_CONDUCT.md" + ), + ( + 'License', + "https://github.com/cseptesting/floatcsep/blob/master/LICENSE" + ), + ( + 'Source Code', + "https://github.com/cseptesting/floatcsep" + ), + ], +} +extlinks = { + 'github_contrib': ('https://github.com/cseptesting/floatcsep/main/blob/%s', ''), +} +rst_epilog = """ +.. raw:: html + +
+
+ GitHub | + CSEP Website | + pyCSEP | + Download PDF +
+""" diff --git a/docs/deployment/intro.rst b/docs/deployment/intro.rst deleted file mode 100644 index 1570e66..0000000 --- a/docs/deployment/intro.rst +++ /dev/null @@ -1,4 +0,0 @@ -Introduction -============ - -In construction \ No newline at end of file diff --git a/docs/examples/case_a.rst b/docs/examples/case_a.rst deleted file mode 100644 index d3b8fea..0000000 --- a/docs/examples/case_a.rst +++ /dev/null @@ -1,165 +0,0 @@ -A - Simple Model and Catalog -============================ - -.. currentmodule:: floatcsep - -.. contents:: - :local: - -.. admonition:: **TL; DR** - - In a terminal, navigate to ``floatcsep/examples/case_a`` and type: - - .. code-block:: console - - $ floatcsep run config.yml - - After the calculation is complete, the results will be summarized in ``results/report.md``. - - -Artifacts ---------- - -The following example shows the definition of a testing experiment of a simple -forecast against a simple catalog. The input structure of the experiment is: - -:: - - case_a - ├── catalog.csep - ├── config.yml - ├── best_model.dat - └── region.txt - - -* The testing region consists of a grid with two 1ºx1º bins, whose bottom-left nodes are defined in the file `region.txt`. The grid spacing is obtained automatically. The nodes are: - - .. literalinclude:: ../../examples/case_a/region.txt - -* The testing catalog ``catalog.csep`` contains only one event and is formatted in the :meth:`~pycsep.utils.readers.csep_ascii` style (see :doc:`pycsep:concepts/catalogs`). Catalog formats are detected automatically - - .. literalinclude:: ../../examples/case_a/catalog.csep - -* The forecast ``best_model.dat`` to be evaluated is written in the ``.dat`` format (see :doc:`pycsep:concepts/forecasts`). Forecast formats are detected automatically (see :class:`floatcsep.readers`) - - .. literalinclude:: ../../examples/case_a/best_model.dat - -.. important:: - - Every file path should be relative to the ``config.yml`` file. - -Configuration -------------- - -The experiment is defined by a time-, region-, model- and test-configurations. In this example, they are written together in the ``config.yml`` file. - - -Time -~~~~ - - The time configuration is manifested in the ``time_config`` inset. The simplest definition is to set only the start and end dates of the experiment. These are always UTC date-times in isoformat (``%Y-%m-%dT%H:%M:%S.%f`` - ISO861): - - .. literalinclude:: ../../examples/case_a/config.yml - :language: yaml - :lines: 3-5 - - .. note:: - - In case the time window are bounded by their midnights, the ``start_date`` and ``end_date`` can be in the format ``%Y-%m-%d``. - - The results of the experiment run will be associated with this time window, whose identifier will be its bounds: ``2020-01-01_2021-01-01`` - -Region -~~~~~~ - - The region - a file path or :class:`csep` function (e.g. :obj:`csep.core.regions.italy_csep_region`) -, the depth limits and magnitude discretization are defined in the ``region_config`` inset. - - .. literalinclude:: ../../examples/case_a/config.yml - :language: yaml - :lines: 7-13 - - -Catalog -~~~~~~~ - - It is defined in the ``catalog`` inset. This should only make reference to a catalog file or a catalog query function (e.g. ``query_comcat``). ``floatcsep`` will automatically filter the catalog to the experiment time, spatial and magnitude frames: - - .. literalinclude:: ../../examples/case_a/config.yml - :language: yaml - :lines: 15-15 - -Models -~~~~~~ - The model configuration is set in the ``models`` inset with a list of model names, which specify their file paths (and other attributes). Here, we just set the path as ``best_model.dat``, whose format is automatically detected. - - .. literalinclude:: ../../examples/case_a/config.yml - :language: yaml - :lines: 17-19 - - .. note:: - - A time-independent forecast model has default units of ``[eq/year]`` per cell. A forecast defined for a different number of years can be specified with the ``forecast_unit: {years}`` attribute. - -Evaluations -~~~~~~~~~~~ - The experiment's evaluations are defined in the ``tests`` inset. It should be a list of test names, making reference to their function and plotting function. These can be either defined in ``pycsep`` (see :doc:`pycsep:concepts/evaluations`) or manually. In this example, we employ the consistency N-test: its function is :func:`csep.core.poisson_evaluations.number_test`, whereas its plotting function correspond to :func:`csep.utils.plots.plot_poisson_consistency_test` - -.. literalinclude:: ../../examples/case_a/config.yml - :language: yaml - :lines: 21-24 - - -Running the experiment ----------------------- - -Run command -~~~~~~~~~~~ - - The experiment can be run by simply navigating to the ``examples/case_a`` folder in the terminal and typing. - - .. code-block:: console - - $ floatcsep run config.yml - - This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. - - .. note:: - - The command ``floatcsep run `` can be called from any working directory, as long as the specified file paths (e.g. region, models) are relative to the ``config.yml`` file. - - -Plot command -~~~~~~~~~~~~ - - If only the result plots are desired, when the calculation was already completed, you can type: - - .. code-block:: console - - $ floatcsep plot config.yml - - This can be used, for example, when an additional plot is desired. Try adding to ``config.yml`` the following lines - - .. code-block:: yaml - - postproc_config: - plot_forecasts: True - - and re-run with the ``plot`` command. A forecast figure will appear in ``results/{window}/forecasts`` - -Results -~~~~~~~ - - The :obj:`~floatcsep.cmd.main.run` command creates the result path tree for each time window analyzed. - - * The testing catalog of the window is stored in ``results/{window}/catalog`` in ``json`` format. This is a subset of the global testing catalog. - * Human-readable results are found in ``results/{window}/evaluations`` - * Catalog and evaluation results figures in ``results/{window}/figures``. - * The complete results are summarized in ``results/report.md`` - - -Advanced --------- - -The experiment run logic can be seen in the file ``case_a.py``, which executes the same example but in python source code. The run logic of the terminal commands ``run``, ``plot`` and ``reproduce`` can be found in :class:`floatcsep.cmd.main` - - diff --git a/docs/examples/case_c.rst b/docs/examples/case_c.rst deleted file mode 100644 index e603646..0000000 --- a/docs/examples/case_c.rst +++ /dev/null @@ -1,82 +0,0 @@ -C - Multiple Time Windows -========================= - -.. currentmodule:: floatcsep - -.. contents:: - :local: - -.. admonition:: **TL; DR** - - In a terminal, navigate to ``floatcsep/examples/case_c`` and type: - - .. code-block:: console - - $ floatcsep run config.yml - - After the calculation is complete, the results will be summarized in ``results/report.md``. - - -Artifacts ---------- - -The following example shows an experiment with multiple time windows. The input structure of the experiment is: - -:: - - case_c - └── models - ├── model_a.csv - ├── model_b.csv - ├── model_c.csv - └── model_d.csv - ├── config.yml - ├── catalog.json - ├── models.yml - ├── tests.yml - └── region.txt - -Configuration -------------- - -Time -~~~~ - - The time configuration now set the time intervals between the start and end dates. - - .. literalinclude:: ../../examples/case_c/config.yml - :language: yaml - :lines: 3-7 - - .. note:: - - The time interval ``growth`` can be either ``cumulative`` (all windows start from ``start_date``) or ``incremental`` (each window starts from the previous window's end). - - The results of the experiment run will be associated with each time window (``2010-01-01_2011-01-01``, ``2010-01-01_2012-01-01``, ``2010-01-01_2013-01-01``, ...). - - - -Evaluations -~~~~~~~~~~~ - The experiment's evaluations are defined in ``tests.yml``, which can now include temporal evaluations (see :func:`~floatcsep.extras.sequential_likelihood`, :func:`~floatcsep.extras.sequential_information_gain`, :func:`~floatcsep.utils.plot_sequential_likelihood`). - - .. literalinclude:: ../../examples/case_c/tests.yml - :language: yaml - - .. note:: - - Plot arguments (title, labels, font sizes, axes limits, etc.) can be passed as a dictionary in ``plot_args`` (see details in :func:`~csep.utils.plots.plot_poisson_consistency_test`) - -Results -------- - -The :obj:`~floatcsep.cmd.main.run` command creates the result path tree for all time windows. - -* The testing catalog of the window is stored in ``results/{window}/catalog`` in ``json`` format. This is a subset of the global testing catalog. -* Human-readable results are found in ``results/{window}/evaluations`` -* Catalog and evaluation results figures in ``results/{window}/figures``. -* The complete results are summarized in ``results/report.md`` - -The report now shows the temporal evaluations for all time-windows, whereas the discrete evaluations are shown only for the last time window. - - diff --git a/docs/examples/case_e.rst b/docs/examples/case_e.rst deleted file mode 100644 index 92d3093..0000000 --- a/docs/examples/case_e.rst +++ /dev/null @@ -1,108 +0,0 @@ -E - A Realistic Time-independent Experiment -=========================================== - -.. currentmodule:: floatcsep - -.. contents:: - :local: - -.. admonition:: **TL; DR** - - In a terminal, navigate to ``floatcsep/examples/case_e`` and type: - - .. code-block:: console - - $ floatcsep run config.yml - - After the calculation is complete, the results will be summarized in ``results/report.md``. - - -Artifacts ---------- - -This example shows how to run a realistic testing experiment in Italy (based on https://doi.org/10.4401/ag-4844), summarizing the concepts of previous examples. The example has only a subset of the original models and evaluations. The input structure of the experiment is: - -:: - - case_e - └── models - ├── gulia-wiemer.ALM.italy.10yr.2010-01-01.xml - ├── meletti.MPS04.italy.10yr.2010-01-01.xml - └── zechar.TripleS-CPTI.italy.10yr.2010-01-01.xml - ├── config.yml - ├── models.yml - └── tests.yml - - -Configuration -------------- - - -Time -~~~~ - - The time configuration is manifested in the ``time-config`` inset. - - .. literalinclude:: ../../examples/case_e/config.yml - :language: yaml - :lines: 3-7 - -Region -~~~~~~ - - The testing region is the official Italy CSEP Region obtained from :obj:`csep.core.regions.italy_csep_region`. - - .. literalinclude:: ../../examples/case_e/config.yml - :language: yaml - :lines: 9-15 - - -Catalog -~~~~~~~ - - The catalog is obtained from an authoritative source, namely the Bollettino Sismico Italiano (http://terremoti.ingv.it/en/bsi ), using the function :func:`csep.query_bsi` - - .. literalinclude:: ../../examples/case_e/config.yml - :language: yaml - :lines: 17-17 - -Models -~~~~~~ - The models are set in ``models.yml``. For simplicity, there are only three of the nineteen models originally submitted to the Italy Experiment. - - .. literalinclude:: ../../examples/case_e/models.yml - :language: yaml - - The ``.xml`` format is automatically detected and parsed by ``floatcsep`` readers. - - .. note:: - - The forecasts are defined in ``[Earthquakes / 10-years]``, specified with the ``forecast_unit`` option. - - .. note:: - - The ``use_db`` flag allows ``floatcsep`` to transform the forecasts into a database (HDF5), which speeds up the calculations. - -Post-Process -~~~~~~~~~~~~ - - Additional options for post-processing can set using the ``postprocess`` option. - - .. literalinclude:: ../../examples/case_e/config.yml - :language: yaml - :lines: 21-34 - - See :func:`~csep.utils.plots.plot_spatial_dataset` for forecast plot options and :func:`~csep.utils.plots.plot_catalog` for the catalog placed on top. - - -Running the experiment ----------------------- - - The experiment can be run by simply navigating to the ``examples/case_a`` folder in the terminal and typing. - - .. code-block:: console - - floatcsep run config.yml - - This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. - diff --git a/docs/examples/case_g.rst b/docs/examples/case_g.rst deleted file mode 100644 index 203f7b8..0000000 --- a/docs/examples/case_g.rst +++ /dev/null @@ -1,130 +0,0 @@ -G - Time-Dependent, Catalog-Based Model (from Source Code) -========================================================== - -.. currentmodule:: floatcsep - -.. contents:: - :local: - -.. admonition:: **TL; DR** - - In a terminal, navigate to ``floatcsep/examples/case_g`` and type: - - .. code-block:: console - - $ floatcsep run config.yml - - After the calculation is complete, the results will be summarized in ``results/report.md``. - - -Artifacts ---------- - -This example shows how a time-dependent model should be set up for a time-dependent experiment -:: - - case_g - └── pymock - ├── input - ├── args.txt (model arguments) - └── catalog.csv (dynamically allocated catalog) - ├── pymock (model source code) - └── forecasts (where forecasts will be stored) - ├── catalog.csv - ├── config.yml - ├── models.yml - └── tests.yml - -* The model to be evaluated (``pymock``) is a source code that generates forecast for multiple time windows. - -* The testing catalog `catalog.csv` works also as the input catalog, by being filtered until the starting test_date and allocated in `pymock/input` dynamically (before the model is run) - - -Model ------ - -The experiment's complexity increases from time-independent to dependent, since we now need a **Model** (source code) that is able to generate forecast that changes every window. The model should be conceptualized as a **black-box**, whose only interface/interaction with the ``floatcsep`` system is to receive an input (i.e. input catalog) and generates an output (i.e. the forecasts). - -* Input: The input consists in input **data** and **arguments**. - - 1. The input data is, at the least, a catalog filtered until the forecast beginning, which is automatically allocated by ``fecsep`` in the `{model}/input` prior to each model's run. It is stored inside the model in ``csep.ascii`` format for simplicity's sake (see :doc:`pycsep:concepts/catalogs`). - - .. literalinclude:: ../../examples/case_g/catalog.csv - :lines: 1-2 - - 2. The input arguments controls how the source code works. The minimum arguments to run a model (which should be modified dynamically during an experiment) are the forecast ``start_date`` and ``end_date``. The experiment will read `{model}/input/args.txt` and change the values of ``start_date = {datetime}`` and ``end_date = {datetime}`' before the model is run. Additional arguments can be set by convenience, such as ``catalog`` (the input catalog name), ``n_sims`` (number of synthetic catalogs) and random ``seed`` for reproducibility. - -* Output: The model's output are the synthetic catalogs, which should be allocated in `{model}/forecasts/{filename}.csv`. The format is identically to ``csep_ascii``, but unlike in an input catalog, the ``catalog_id`` column should be modified for each synthetic catalog starting from 0. The file name follows the convention `{model_name}_{start}_{end}.csv`, where ``start`` and ``end`` folowws the `%Y-%m-%dT%H:%M:%S.%f` - ISO861 FORMAT - -* Model run: The model should be run with a simple command to which only ``arguments`` should be passed. For this example, is - - .. code-block:: console - - $ python pymock/run.py input/args.txt - -or using a binary (see ``pymock/setup.cfg:entry_point``) - - .. code-block:: console - - $ pymock input/args.txt - - - - -Configuration -------------- - - -Time -~~~~ - - The configuration is identical to time-independent models, with the exception that now a ``horizon`` can be defined instead of ``intervals``, which is the forecast time-window length. The experiment's class should now be explicited as ``exp_class: td`` - - .. literalinclude:: ../../examples/case_g/config.yml - :language: yaml - :lines: 3-7 - -Catalog -~~~~~~~ - - The catalog was obtained ``previous to the experiment`` using ``query_bsi``, but it was filtered from 2006 onwards, so it has enough data for the model calibration. - -Models -~~~~~~ - - Additional arguments should be passed to time-independent models. - - .. literalinclude:: ../../examples/case_g/models.yml - :language: yaml - :lines: 3-7 - - Now ``path`` points to the folder where the source is installed. Input and forecasts should be in ``{path}/input`` and ``{path}/forecasts``. The ``func`` option is the shell command with which the model is run (the shell will be run from the model's directory). Note that ``python run.py`` can be changed to ``pymock`` (see entry_point in ``pymock/setup.cfg``). In practice, the model will be run as - - .. code-block:: console - - $ python {path}/run.py {path}/input/args.txt - -Tests -~~~~~ - - With time-dependent models, now catalog evaluations found in :obj:`csep.core.catalog_evaluations` can be used. - - - .. literalinclude:: ../../examples/case_g/tests.yml - :language: yaml - - .. note:: - It is possible to assign two plotting functions to a test, whose ``plot_args`` and ``plot_kwargs`` can be placed indented beneath - - -Running the experiment ----------------------- - - The experiment can be run by simply navigating to the ``examples/case_g`` folder in the terminal and typing. - - .. code-block:: console - - $ floatcsep run config.yml - - This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. - diff --git a/docs/guide/config.rst b/docs/guide/config.rst deleted file mode 100644 index 86a7a6e..0000000 --- a/docs/guide/config.rst +++ /dev/null @@ -1,4 +0,0 @@ -Experiment configuration -======================== - -In progress \ No newline at end of file diff --git a/docs/guide/evaluation_config.rst b/docs/guide/evaluation_config.rst new file mode 100644 index 0000000..e01af13 --- /dev/null +++ b/docs/guide/evaluation_config.rst @@ -0,0 +1,187 @@ +.. _evaluation_config: + +Evaluations Definition +====================== + +**floatCSEP** evaluate forecasts using the routines defined in **pyCSEP** (See `Testing Theory `_). Depending on the forecast types (e.g., **GriddedForecasts** or **CatalogForecasts**), different evaluation functions can be used. + +Each evaluation specifies a ``func`` parameter, representing the evaluation function to be applied, a configuration of the function with ``func_kwargs`` (e.g., number of simulations, confidence intervals) and a ``plot_func`` parameter for visualizing the results. Evaluations for **GriddedForecasts** typically use functions from :mod:`csep.core.poisson_evaluations` or :mod:`csep.core.binomial_evaluations`, while evaluations for **CatalogForecasts** use functions from :mod:`csep.core.catalog_evaluations`. + +.. important:: + + An evaluation in ``test_config`` points to a **pyCSEP** `evaluation function `_, valid for the forecast class. + + +**Example Configuration**: + +.. code-block:: yaml + :caption: test_config.yml + + - S-test: + func: poisson_evaluations.spatial_test + plot_func: plot_poisson_consistency_test + plot_kwargs: + one_sided_lower: True + - T-test: + func: poisson_evaluations.paired_t_test + ref_model: Model A + plot_func: plot_comparison_test + + +Evaluation Parameters +--------------------- + +Each evaluation listed in ``test_config`` accepts the following parameters: + +.. list-table:: + :widths: 30 80 + :header-rows: 1 + + * - **Parameter** + - **Description** + * - **func** (required) + - Specify which evaluation/test function to run. Must be a **pyCSEP** ``{module}.{function}`` suite \ + (e.g., :func:`poisson_evaluations.number_test `) or + **floatCSEP** function. + * - **func_kwargs** + - Any keyword argument to control the specific **func**. For example, :func:`poisson_evaluations.spatial_test ` may be configured with ``num_simulations: 2000``. + * - **plot_func** (required) + - The function to plot the evaluation results, from either the :mod:`csep.utils.plots` module (e.g., :func:`plot_poisson_consistency_test `) or **floatCSEP** :mod:`~floatcsep.utils.helpers` module. + * - **plot_args** + - Arguments passed to customize the plot function. Can be titles, labels, colors, font size, etc. Review the documentation of the respective function. + * - **plot_kwargs** + - Keyword arguments to customize the plot function. Review the documentation of the respective function. + * - **ref_model** + - A reference model against which the current model is compared in comparative tests (e.g., `Model A`). + * - **markdown** + - A description of the test to be used as caption when reporting results + + +Evaluations Functions +--------------------- + +**floatCSEP** supports the following evaluations: + + +.. dropdown:: **Evaluations for GriddedForecasts** + :animate: fade-in-slide-down + :icon: list-unordered + + .. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - **Function** + - **Evaluates:** + * - :func:`poisson_evaluations.number_test ` + - the total number of forecasted events compared to the observed events using a Poisson distribution. + * - :func:`poisson_evaluations.spatial_test ` + - the forecasted spatial distribution relative to the observed events using a Poisson distribution. + * - :func:`poisson_evaluations.magnitude_test ` + - the forecasted magnitude distribution relative to the observed events using a Poisson distribution. + * - :func:`poisson_evaluations.conditional_likelihood_test ` + - the likelihood of the observed events given the forecasted rates, conditioned on the total earthquake occurrences, assuming a Poisson distribution. + * - :func:`poisson_evaluations.paired_t_test ` + - the information gain between one forecast to a reference (``ref_model``), and test for a significant difference by using a paired T-test. + * - :func:`binomial_evaluations.binary_spatial_test ` + - the forecasted spatial distribution relative to the observed events, assuming a Binary/Bernoulli process. + * - :func:`binomial_evaluations.binary_likelihood_test ` + - the likelihood of the observed events given the forecasted rates, assuming a Binary distribution. + * - :func:`binomial_evaluations.negative_binomial_number_test ` + - the total number of forecasted events compared to the observed events using a Negative Binomial distribution. + * - :func:`brier_score ` + - the forecast skill using a quadratic metric rather than logarithmic. Does not penalize false-negatives as much as log-likelihood metrics. + * - :func:`vector_poisson_t_w_test ` + - a forecast skill compared to multiple forecasts, by carrying out the paired_t_test and w_test jointly. + * - :func:`sequential_likelihood ` + - the temporal evolution of log-likelihoods scores. + * - :func:`sequential_information_gain ` + - the temporal evolution of the information gain in time, compared to a ``ref_model``. + + + +.. dropdown:: **Evaluations for CatalogForecasts** + :animate: fade-in-slide-down + :icon: list-unordered + + .. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - **Function** + - **Evaluates:** + * - :func:`catalog_evaluations.number_test ` + - the total number of forecasted events compared to observed events in an earthquake catalog. + * - :func:`catalog_evaluations.spatial_test ` + - the spatial distribution of forecasted vs. observed earthquake events in an earthquake catalog. + * - :func:`catalog_evaluations.magnitude_test ` + - the magnitude distribution of forecasted events to those observed in the earthquake catalog. + * - :func:`catalog_evaluations.pseudolikelihood_test ` + - the pseudolikelihood of the observed events, given the forecasted synthetic catalogs + * - :func:`catalog_evaluations.calibration_test ` + - the consistency of multiple test-quantiles in time with the expected uniform distribution using a Kolmogorov-Smirnov test. + +.. note:: + + Check each function's `docstring` to see which ``func_kwargs`` are compatible with it. + +Plotting Functions +------------------ + +**floatCSEP** supports the following: + +.. dropdown:: Plotting functions + :animate: fade-in-slide-down + :icon: list-unordered + + .. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - **Plotting function** + - **Compatible with:** + * - :obj:`~csep.utils.plots.plot_poisson_consistency_test` + - :func:`poisson_evaluations.number_test `, :func:`poisson_evaluations.spatial_test `, :func:`poisson_evaluations.magnitude_test `, :func:`poisson_evaluations.conditional_likelihood_test `. + * - :obj:`~csep.utils.plots.plot_consistency_test` + - :func:`binomial_evaluations.negative_binomial_number_test `, :func:`binomial_evaluations.binary_likelihood_test `, :func:`binomial_evaluations.binary_spatial_test `, :func:`brier_score `, :func:`catalog_evaluations.number_test `, :func:`catalog_evaluations.magnitude_test `, :func:`catalog_evaluations.spatial_test `, :func:`catalog_evaluations.pseudolikelihood_test ` + * - :obj:`~csep.utils.plots.plot_comparison_test` + - :func:`poisson_evaluations.paired_t_test ` + * - :obj:`~csep.utils.plots.plot_number_test` + - :func:`catalog_evaluations.number_test ` + * - :obj:`~csep.utils.plots.plot_magnitude_test` + - :func:`catalog_evaluations.magnitude_test ` + * - :obj:`~csep.utils.plots.plot_spatial_test` + - :func:`catalog_evaluations.spatial_test ` + * - :obj:`~csep.utils.plots.plot_likelihood_test` + - :func:`catalog_evaluations.pseudolikelihood_test ` + * - :obj:`~csep.utils.plots.plot_calibration_test` + - :func:`catalog_evaluations.calibration_test ` + * - :obj:`~floatcsep.utils.helpers.plot_sequential_likelihood>` + - :func:`sequential_likelihood `, :func:`sequential_information_gain ` + * - :obj:`~floatcsep.utils.helpers.plot_matrix_comparative_test` + - :func:`vector_poisson_t_w_test ` + +.. note:: + + Check each plot functions's `docstring` to see which ``plot_args`` and ``plot_kwargs`` are compatible with it. + + + +It is also possible to assign two or more plotting functions to a test, the ``plot_args`` and ``plot_kwargs`` of which can be placed as dictionaries indented beneath the functions: + +**Example**: + +.. code-block:: yaml + :caption: test_config.yml + + - Number Test: + func: catalog_evaluations.number_test + plot_func: + - plot_number_test: + plot_args: + title: Number test distribution + - plot_consistency_test: + plot_args: + linewidth: 2 + plot_kwargs: + one_sided_lower: True diff --git a/docs/guide/executing_experiment.rst b/docs/guide/executing_experiment.rst new file mode 100644 index 0000000..78ee59c --- /dev/null +++ b/docs/guide/executing_experiment.rst @@ -0,0 +1,125 @@ +.. _running: + +Executing an Experiment +======================= + +In **floatCSEP**, experiments are executed through a set of core functions provided in the command-line interface (CLI). These commands allow you to stage models, run experiments, plot results, and generate reports. The general structure of these commands is: + +.. code-block:: console + + $ floatcsep + + +The `` can be one of the following: +- `run`: Run the experiment. +- `stage`: Prepare the environment and models. +- `plot`: Plot forecasts, catalogs, and results. +- `reproduce`: (see folllowing\ section) Reproduce the results of a previously run experiment. + +Each command requires a configuration file (in `YAML` format), which defines the parameters of the experiment. + +Running an Experiment: ``floatcsep run`` +---------------------------------------- + +The core command to run an experiment is: + +.. code-block:: console + + $ floatcsep run + + +This command initiates the workflow summarized as: + +1. **Experiment Initialization**: The experiment is initialized by reading the configuration file (`config.yml`). This file contains details such as time windows, regions, catalogs, models, and evaluation tests. + +2. **Staging Models**: If the models are not already staged, they are fetched from their respective repositories (e.g., GitHub, Zenodo) or located on the file system. The necessary computational environments are built using tools like **Conda**, **venv**, or **Docker**. + +3. **Task Generation**: Depending on the experiment class, a given acyclic graph of tasks is created (e.g., a collection of tasks with dependencies to one another). These tasks include creating forecasts, filtering catalogs, and evaluating the forecasts using statistical tests. + +4. **Execution**: The task graph is executed on a standardized fashion. Depending on the characteristics of the experiment (e.g., time-dependent, evaluation windows), this step might involve generating forecasts and running evaluations in sequence. + +5. **Postprocessing**: After the core tasks are completed, the postprocessing step involves: + - Plotting forecasts, catalogs, and evaluation results. + - Generating human-readable reports, which summarize the results. + +6. **Reproducibility**: The configuration and results are saved, allowing the experiment to be reproduced or compared in future runs. + +Here is an example of how to run an experiment: + +.. code-block:: sh + + $ floatcsep run config.yml + + +For more information on how to structure the configuration file, refer to the :ref:`experiment_config` section. + + +Staging Models: ``floatcsep stage`` +----------------------------------- + +Before running an experiment, you may need to check if the models are `staged` properly instead. This involves fetching the source code for the models from a repository (e.g., GitHub, Zenodo) and setting up the necessary computational environment. + +.. code-block:: console + + floatcsep stage + + +Staging the models can be done previous to an experiment run when dealing with source-code models that need specific environments and dependencies to run properly. The `staging` process includes: + +- Downloading the source code. +- Building the computational environment (e.g., setting up a Conda environment or Docker container). +- Installing any dependencies required by the models. +- Building the source code. +- Check a correct integration with floatcsep. +- Prepare the structure of the required forecast. +- Self-discovery of existing forecasts in the filesystem. + + it does plot the results of an already completed experiment. + +.. note:: + + This command should be executed to check if everything is present and working correctly before an official ``run`` execution. + +Plotting Results: ``floatcsep plot`` +------------------------------------ + +Once the experiment has been run, you can regenerate plots for the forecasts, catalogs, and evaluation results using the `plot` command: + +.. code-block:: console + + $ floatcsep plot + + +The `plot` command re-loads the experiment configuration, stages the models, identifying the necessary time windows and results to plot. It does not re-run the forecasts or evaluations, but it does plot the results of an already completed experiment. + +.. note:: + + This command can be useful when trying to customize plots or reports after the results have been created. + + +Reproducing Results: ``floatcsep reproduce`` +-------------------------------------------- + +The `reproduce` command in **floatCSEP** allows users to re-run a previously executed experiment using the same configuration, in order to compare the results and assess reproducibility. This feature allows to ensure that experiments yield consistent outputs when re-executed and to validate scientific results. + +The general command structure is: + +.. code-block:: console + + $ floatcsep reproduce + +A ``repr_config.yml`` is always generated once an experiment is run with ``floatcsep run``. The ``reproduce`` command re-runs the experiment based on this configuration and compares the newly generated results with the original results to provide reproducibility metrics: + +- **Statistical Reproducibility**: It analyzes statistical changes of the evaluation results: + + - **Forecast Scores**: The numerical difference between the observed scores of the original and reproduced experiments. + - **Test Statistics**: Statistical metrics like mean, standard deviation, and skewness of the test distributions are compared. + - **Kolmogorov-Smirnov (KS) Test**: The KS-test p-value is computed to assess whether the test distributions from both experiments are significantly different. A p-value below 0.1 indicates a potential difference between distributions. + +- **Data Reproducibility**: A comparison of the result files, checking for discrepancies in file contents or structure. + + - **Hash Comparison (SHA-256)**: Each result file is hashed using the SHA-256 algorithm to check if the content has changed between the original and reproduced experiments. + - **Byte-to-Byte Comparison**: This is a direct comparison of the file contents at the byte level, ensuring that no unintended changes have occurred. + + +The analysis can now be found in the created ``reproducibility_report.md``. diff --git a/docs/guide/experiment_config.rst b/docs/guide/experiment_config.rst new file mode 100644 index 0000000..a5c513a --- /dev/null +++ b/docs/guide/experiment_config.rst @@ -0,0 +1,280 @@ +.. _experiment_config: + +Experiment Configuration +======================== + +**floatCSEP** provides a standardized workflow for forecasting experiments, the instructions of which can be set in a configuration file. Here, we need to define the Experiments' temporal settings, geographic region, seismic catalogs, forecasting models, evaluation tests and any post-process options. + + +Configuration Structure +----------------------- +Configuration files are written in ``YAML`` format and are divided into different aspects of the experiment setup: + +1. **Metadata**: Experiment's information such as its ``name``, ``authors``, ``doi``, ``URL``, etc. +2. **Temporal Configuration** (``time_config``): Temporal characteristics of the experiment, such as the start and end dates, experiment class (time-independent or time-dependent), the testing intervals, etc. +3. **Spatial and Magnitude Configuration** (``region_config``): Describes the testing region, such as its geographic bounds, magnitude ranges, and depth ranges. +4. **Seismic Catalog** (``catalog``): Defines the seismicity data source to test the models. It can either link to a seismic network API, or an existing file in the system. +5. **Models** (``models``): Configuration of forecasting models. It can direct to an additional configuration file with ``model_config`` for readability. See :ref:`model_config`. +6. **Evaluation Tests** (``tests``): Configuration of the statistical tests to evaluate the models. It can direct to an additional configuration file with ``test_config`` for readability. See :ref:`evaluation_config`. +7. **Postprocessing** (``postprocess``): Instructions on how to process and visualize the experiment's results, such as plotting forecasts or generating reports. See :ref:`postprocess`. + +.. note:: + + `YAML` (Yet Another Markup Language) is a human-readable format used for configuration files. It uses **key: value** pairs to define settings, and indentation to represent nested structures. Lists are denoted by hyphens (`-`). + + +**Example Basic Configuration**: + +.. code-block:: yaml + :caption: config.yml + + name: CSEP Experiment + time_config: + start_date: 2010-1-1T00:00:00 + end_date: 2020-1-1T00:00:00 + region_config: + region: region.txt + mag_min: 4.0 + mag_max: 9.0 + mag_bin: 0.1 + depth_min: 0 + depth_max: 70 + catalog: catalog.csv + models: + - Smoothed-Seismicity: + path: ssm.dat + - Uniform: + path: uniform.dat + tests: + - Poisson S-test: + func: poisson_evaluations.spatial_test + plot_func: plot_poisson_consistency_test + postprocess: + plot_forecasts: + cmap: magma + catalog: True + + + +Experiment Metadata +------------------- + +.. list-table:: + :widths: 20 80 + + * - **name** + - Maximum magnitude to be considered. + * - **authors** + - Authors of the experiment + * - **doi** + - identifier associated to the experiment + * - **URL** + - repository of the experiment (e.g., Github) + +Any non-parsed parameter (e.g., not specified in the documentation) will be stored also as metadata. + + +Temporal Definition +------------------- + +Configuring the experiment temporal definition is indicated with the ``time_config`` option, followed by an indented block of admissible parameters. The purpose of this configuration section is to set a testing **time-window**, or a sequence of **time-windows**. Each time-window is defined by two ``datetime`` strings representing its lower and upper edges. + +Time-windows can be defined from different combination of the following parameters: + +.. list-table:: + :widths: 20 80 + + * - **start_date** + - The start date of the experiment in UTC and ISO8601 format (``%Y-%m-%dT%H:%M:%S``) + * - **end_date** + - The end date of the experiment in UTC and ISO8601 format (``%Y-%m-%dT%H:%M:%S``) + * - **intervals** + - An integer amount of testing intervals (time windows). If **horizon** is given, each time-window has a length equal to **horizon**. Else, the range between **start_date** and **end_date** is equally divided into the amount of **intervals**. + * - **horizon** + - Indicates the time windows `length`. It is written as a number, followed by a hyphen (`-`) and a time unit (``days``, ``weeks``, ``months``, ``years``). e.g.: ``1-days``, ``2.5-years``. + * - **growth** + - How to discretize the time-windows between ``start_date`` and ``end_date``. Options are: **incremental** (The end of a time window matches the beginning of the next) or **cumulative** (All time-windows have a start at the experiment ``start_date``). + * - **offset** + - Offset between consecutive time-windows. If none given or ``offset=horizon``, time-windows are non-overlapping. It is written as a number, followed by a hyphen (`-`) and a time unit (``days``, ``weeks``, ``months``, ``years``). e.g.: ``1-days``, ``2.5-years``. + * - **exp_class** + - Experiment temporal class. Options are: + **ti** (default): Time-Independent; **td**: Time-Dependent. + +.. note:: + + For a Time-Independent (``ti``) experiment class, the following argument combinations are possible: + + - (``start_date``, ``end_date``) + - (``start_date``, ``end_date``, ``intervals``) + - (``start_date``, ``end_date``, ``horizon``) + - (``start_date``, ``intervals``, ``horizon``) + + For a Time-Dependent (``td``) experiment class, the following argument combinations are possible: + + - (``start_date``, ``end_date``, ``intervals``) + - (``start_date``, ``end_date``, ``horizon``) + - (``start_date``, ``intervals``, ``horizon``) + - (``start_date``, ``end_date``, ``horizon``, ``offset``) + - (``start_date``, ``intervals``, ``horizon``, ``offset``) + + +Some example of parameter combinations: + ++------------------------------------------------+----------------------------------------------------------+ +| .. code-block:: yaml | Two time-windows of equal size between 2010 and 2020 | +| | | +| time_config: | - ``2010-01-01T00:00:00`` - ``2015-01-01T00:00:00`` | +| start_date: 2010-01-01T00:00:00 | - ``2015-01-01T00:00:00`` - ``2020-01-01T00:00:00`` | +| end_date: 2020-01-01T00:00:00 | | +| intervals: 2 | | ++------------------------------------------------+----------------------------------------------------------+ +| .. code-block:: yaml | Two cummulative time-windows between 2010 and 2020 | +| | | +| time_config: | - ``2010-01-01T00:00:00`` - ``2015-01-01T00:00:00`` | +| start_date: 2010-01-01T00:00:00 | - ``2010-01-01T00:00:00`` - ``2020-01-01T00:00:00`` | +| end_date: 2020-01-01T00:00:00 | | +| intervals: 2 | | +| growth: cumulative | | ++------------------------------------------------+----------------------------------------------------------+ +| .. code-block:: yaml | Time-Dependent experiment with three ``1-day`` windows | +| | | +| time_config: | | +| start_date: 2010-01-01T00:00:00 | - ``2010-01-01T00:00:00`` - ``2010-01-02T00:00:00`` | +| intervals: 3 | - ``2010-01-02T00:00:00`` - ``2010-01-03T00:00:00`` | +| horizon: 1-days | - ``2010-01-03T00:00:00`` - ``2010-01-04T00:00:00`` | +| exp_class: td | | ++------------------------------------------------+----------------------------------------------------------+ +| .. code-block:: yaml | Two overlapping ``7-days`` time-windows | +| | | +| time_config: | - ``2010-01-01T00:00:00`` - ``2010-01-08T00:00:00`` | +| start_date: 2010-01-01T00:00:00 | - ``2010-01-02T00:00:00`` - ``2020-01-09T00:00:00`` | +| intervals: 2 | | +| horizon: 7-days | | +| offset: 1-day | | +| exp_class: td | | ++------------------------------------------------+----------------------------------------------------------+ + +Alternatively, time windows can be defined explicitly as a **list** of time-windows (each of which is a **list** of ``datetimes``): + +.. code-block:: yaml + + time_config: + timewindows: + - - 2010-01-01T00:00:00 + - 2011-01-01T00:00:00 + - - 2011-01-01T00:00:00 + - 2012-01-01T00:00:00 + +Spatial and Magnitude Definition +-------------------------------- + +Configuring the spatial and magnitude definitions is done through the ``region_config`` option, followed by an indented block of admissible parameters. Here, we need to define the spatial region (check the `Region `_ documentation from **pyCSEP**), the magnitude `bins` (i.e., discretization) and the `depth` extent. + +.. list-table:: + :widths: 20 80 + + * - **region** + - The spatial domain where forecasts will be tested. Either a file or a **CSEP** region. + * - **mag_min** + - The minimum magnitude of the experiment. + * - **mag_max** + - The maximum magnitude of the experiment. + * - **mag_bin** + - The size of the magnitude bin. + * - **depth_min** + - The minimum depth (in `km`) of the experiment. + * - **depth_max** + - The maximum depth (in `km`) of the experiment. + + +1. The ``region`` parameter can be defined from: + + + * A **CSEP** region: They correspond to pre-established testing regions for seismic areas. This parameter is linked to **pyCSEP** functions, and can be one of the following values: + + * ``italy_csep_region`` + * ``nz_csep_region`` + * ``california_relm_region`` + * ``global_region``. + + * A text file with the spatial cells collection. Each cell is defined by its origin (e.g., the x (lon) and y (lat) of the lower-left corner). For example, for a region consisting of three cells, their origins can be written as: + + .. code-block:: + + 10.0 40.0 + 10.0 40.1 + 10.1 40.0 + + See the **pyCSEP** `Region documentation `_, the class :class:`~csep.core.regions.CartesianGrid2D` and its method :meth:`~csep.core.regions.CartesianGrid2D.from_origins` for more info. +2. Magnitude definition: We need to define a magnitude discretization or `bins`. The parameters **mag_min**, **mag_max**, **mag_bin** allows to create an uniformly distributed set of bins. For example, the command: + + .. code-block:: yaml + + mag_min: 4.0 + mag_max: 5.0 + mag_bin: 0.5 + + would result in two magnitude bins with ranges ``[4.0, 4.5)`` and ``[4.5, 5.0)``. Alternatively, magnitudes can be written explicitly by their bin `left` edge. For example: + + .. code-block:: yaml + + magnitudes: + - 4.0 + - 4.1 + - 4.2 + + resulting in the ``[4.0, 4.1)``, ``[4.1, 4.2)`` and ``[4.2, 4.3)``. + + +3. Depths: The minimum and maximum depths are just required to filter out seismicity outside those ranges. + + +Some example of region configurations would be: + ++------------------------------------------------+---------------------------------------------------------------------+ +| .. code-block:: yaml | - Uses the **CSEP** Italy region, as defined by the function | +| | :func:`~csep.core.regions.italy_csep_region`. | +| region_config: | - Discretizes the magnitude range into 40 bins between 4.0 and 9.0 | +| region: italy_csep_region | - Test the models against `crustal` seismicity above 30 km. | +| mag_min: 5.0 | The -2 is meant in case of shallow events above sea level | +| mag_max: 9.0 | | +| mag_bin: 0.1 | | +| depth_min: -2 | | +| depth_max: 30 | | ++------------------------------------------------+---------------------------------------------------------------------+ +| .. code-block:: yaml | - Loads a file ``region_file.txt`` which contains the cells' | +| | originsof the region. | +| region_config: | - Contains two magnitude bins: ``[6.0, 7.0)``, ``[7.0, 8.0)`` and | +| region: region_file.txt | | +| depth_min: 70 | | +| depth_max: 150 | | +| magnitudes: | | +| - 6.0 | | +| - 7.0 | | +| - 8.0 | | ++------------------------------------------------+---------------------------------------------------------------------+ + + +Seismicity Catalog +------------------ + +The seismicity catalog can be defined with the ``catalog`` parameter. It represents the **main catalog** of the experiment, and will be used to test the forecasts against, or if required, as input catalog for time-dependent models. It can be obtained from: + +* **Authorative data source** + + **floatCSEP** can retrieve the catalog from a seismic network API. The possible options are: + + - ``query_gcmt``: Global Centroid Moment Tensor Catalog (https://www.globalcmt.org/), obtained via ISC (https://www.isc.ac.uk/) + - ``query_comcat``: ANSS ComCat (https://earthquake.usgs.gov/data/comcat/) + - ``query_bsi``: Bollettino Sismico Italiano (https://bsi.ingv.it/) + - ``query_gns``: GNS GeoNet New Zealand Catalog (https://www.geonet.org.nz/) + +* **Catalog file in pyCSEP format** + + A file can be used as **main catalog**. It must be in a **pyCSEP** format, namely in the :meth:`~pycsep.utils.readers.csep_ascii` style (see :doc:`pycsep:concepts/catalogs`) or ``.json`` format. The latter is the default catalog used by **floatCSEP**, as it allows the storage of metadata. + + .. note:: + A catalog can be stored as ``.json`` with :meth:`CSEPCatalog.write_json() ` using **pyCSEP**. + +.. important:: + The main catalog will be stored, and consecutively filtered to the extent of each testing time-window, as well as to the experiment's spatial domain, and magnitude- and depth- ranges. diff --git a/docs/guide/model_config.rst b/docs/guide/model_config.rst index 165f615..fe47848 100644 --- a/docs/guide/model_config.rst +++ b/docs/guide/model_config.rst @@ -1,4 +1,393 @@ -Models configuration +.. _model_config: + + +Models Configuration ==================== -TBI \ No newline at end of file +**floatCSEP** can integrate **source-code** models or just **forecast files**. Depending on the model type, configuration can be as simple as specifying a file path or as complex as defining the computational environment, run commands and model arguments. In the case of source-codes, the **Model Integration** section covers the environment management, executing the model code, and input/output dataflow. + +In the experiment ``config.yml`` file (See :ref:`experiment_config`), the parameter ``model_config`` can point to a **model configuration** file, also in ``YAML`` format, with the generic structure: + +**Example**: + + .. code-block:: yaml + :caption: model_config.yml + + - MODEL_1 NAME: + parameter_1: value + parameter_2: value + ... + - MODEL_2 NAME: + parameter_1: value + parameter_2: value + ... + ... + +Model names are used to identify models in the system, and spaces are replaced by underscores `_`. + + +Time-Independent Models +----------------------- + +A **Time-Independent** model is usually represented by a single-file forecast, whose statistical description does not change over time. +Thus, the model configuration needs only to point to the **path** of the file relative to the ``model_config`` file. + +**Example**: + +.. code-block:: yaml + + - GEAR: + path: models/gear.xml + forecast_unit: 1 + +``forecast_unit`` represents the time frame upon which the forecast rates are defined (Defaults to 1). In time-independent forecasts, ``forecast_unit`` is in decimal **years**. Forecasts are scaled to the testing time-window if its length is different to the one of the forecast. + + + +Time-Dependent Models +--------------------- + +**Time-Dependent** models are composed by forecasts issued for multiple time windows. These models can be either a **collection** of forecast files or a **source-code** that generate such collection. + + +1. **Forecast Collection**: + + In this case, the ``path`` must point to a model **directory**. To standardize with the directory structure of **source-code** models, forecasts should be contained in a folder named **forecasts** inside the model's ``path``. + + **Example**: + + .. code-block:: yaml + + - ETAS: + path: models/etas + forecast_unit: 3 + n_sims: 10000 + + * Forecasts must be contained in a folder ``models/etas/forecasts``, relative to the ``model_config`` file. + * The ``forecast_unit`` is defined in **days** for Time-Dependent models. + * ``n_sims`` represents the total number of simulations from a catalog-based forecast (usually simulations with no events are not written, so the total amount of catalogs must be explicit). + + .. important:: + + Forecast files are automatically detected. The standard way the model source should name a forecast is : + + .. code-block:: + + {model_name}_{start}_{end}.csv + + where ``start`` and ``end`` follow either the ``%Y-%m-%dT%H:%M:%S`` - ISO8601 format, or the short date version ``%Y-%m-%d`` if the windows are set by UTC midnight. + + See the **pyCSEP** `Documentation `_ to see how forecast files should be written. See the :ref:`model_integration` section for details about how a model source-code should be designed or adapted to be integrated with **floatCSEP** + +1. **Source-Code**: + + **floatCSEP** interacts with a model's source code by (i) creating a running environment, (ii) placing the input data (e.g., training catalog) within the model's directory structure, (iii) executing an specified run command and (iv) retrieving forecasts from the model directory structure. These actions will be detailed in the :ref:`model_integration` section. + + The basic parameters of the configuration are: + + * ``path`` refers to the source-code directory. + * The ``build`` parameter defines the environment type (e.g., ``conda``, ``venv``, or ``docker``) and ensures the model runs in isolation with the necessary dependencies. + * ``func`` is a `shell` command (**entrypoint**) with which the source-code is executed inside the environment. + * The ``forecast_unit`` is defined in **days** for Time-Dependent models. + + **Example**: + + .. code-block:: yaml + + - STEP: + path: models/step + build: docker + func: etas-run + forecast_unit: 1 + +Repository Download +------------------- + +A model file(s) or source code can be accessed from a code or data repository (i.e., `GitHub `_ or `Zenodo `_). + +.. code-block:: yaml + + - etas: + giturl: https://git.gfz-potsdam.de/csep/it_experiment/models/vetas.git + repo_hash: v3.2 + +where ``repo_hash`` refers to a given **release**, **tag** or **branch**. Alternatively, a model can be retrieved from a Zenodo repository by specifying its ID: + +.. code-block:: yaml + + - wheel: + zenodo_id: 6255575 + + + +Configuration Parameters +------------------------ + +Here you can find a comprehensive list of parameters used to configure models + +.. list-table:: + :widths: 20 20 60 + :header-rows: 1 + + * - **Name** + - **Type** + - **Description** + * - **path** (required) + - All + - Path to the model’s (i) **forecast file** for a time-independent class, or (ii) **model's directory** for time-dependent class + * - **build** + - TD + - Specifies the environment type in which the model will be built (e.g., ``conda``, ``venv``, ``docker``). + * - **zenodo_id** + - All + - Zenodo record ID for downloading the model's data. + * - **giturl** + - All + - Git repository URL for the model’s source code. + * - **repo_hash** + - All + - Specifies the commit, branch, or tag to be checked out from the repository. + * - **args_file** (required) + - TD + - Path to the input arguments file for the model, relative to ``path``. In here, the forecast start_date and end_date will be dynamically written before each forecast creation. Defaults to ``input/args.txt``. + * - **func** + - TD + - The command to execute the model (i.e., **entrypoint**) in a terminal. Examples of ``func`` are: ``run``, ``etas-run``, ``python run_script.py``, ``Rscript script.r``. + * - **func_kwargs** (optional) + - TD + - Additional arguments for the model execution, passed via the arguments file. + * - **forecast_unit** (required) + - All + - Specifies the time unit for the forecast. Use **years** for time-independent models and **days** for time-dependent models. + * - **store_db** (optional) + - All + - If the model consists on only files, this is a boolean (true/false) specifying whether to store the forecast in a database (HDF5). + * - **flavours** (optional) + - All + - A set of parameter variations to generate multiple model variants (e.g., different settings for the same model). + * - **prefix** (optional) + - TD + - The prefix used for the model to name its forecast (The default is the Model's name) + * - **input_cat** (optional) + - TD + - Specifies the input catalog path used by the model, relative to the model's ``path``. Defaults to ``input/catalog.csv``. + * - **force_stage** (optional) + - All + - Forces the entire staging of the model (e.g., downloading data, database preparation, environment creation, installation of dependencies and source-code build) + * - **force_build** (optional) + - All + - Forces the build of the model's environment (e.g., creation, dependencies installation and source-code build) + + + +.. _model_integration: + +Model Integration +----------------- + +The integration of external model source-codes into **floatCSEP** requires: + +* Follow (loosely) a directory structure to allow the dataflow (input/output) between the model and **pyCSEP**. +* Define a environment/container manager. +* Provide source-code build instructions. +* Set up an entrypoint (terminal command) to run the model and create a forecast. + +.. note:: + + To integrate a broader range of model classes and code complexities, we opted in **floatCSEP** for a simple interface design rather than specifying a complex model API. Therefore, the integration will have sometimes strict requirements, or customizable options and sometimes undefined aspects. We encourage any feedback from modelers (and hopefully their contributions) through our GitHub, to encompass the majority of model implementations possible. + +Directory Structure +~~~~~~~~~~~~~~~~~~~ + +The repository should contain, at the least, the following structure: + +.. code-block:: none + + model_name/ + ├── /forecasts # Forecast outputs should be stored here (Required) + ├── /input # Input data will be placed here dynamically by **floatCSEP** (Required) + │ ├── {input_catalog} # Input catalog file provided by the testing center + │ └── {args_file} # Contains the input arguments for model execution + ├── /{source} # [optional] Where to store all the source code of the model + │ └── ... + ├── /state # [optional] State files (e.g., data to be persisted throughout consistent simulations) + ├── README.md # [optional] Basic information of the model and instructions to run it. + ├── {run_script} # [optional] Script to generate forecasts. Can be either located here, or in the environment PATH (e.g., a binary entrypoint for python) + ├── Dockerfile # Docker environment setup file + ├── environment.yml # Instructions to build a conda environment. + └── setup.py # Script to build the code with "pip install . ". Can also be `project.toml` or `setup.cfg` + + +* The name of the files ``input_catalog`` (default: `catalog.csv`) and ``args_file`` (default: `args.txt`) can be controlled within ``model_config``. +* It is required (for this integration protocol) that the folders ``input`` and ``forecasts`` exists in the model directory. The latter could be created during the first model run. + +.. important:: + The directory structure should remain unchanged during the experiment run, except for the dynamic modification of the `input/`, `forecasts/` and `state/` contents. All of the source-code file management routines should point to these folders (e.g., routines to read input catalogs, read input arguments, to write forecasts, etc.). + + +Environment Management +~~~~~~~~~~~~~~~~~~~~~~ + +The `build` parameter in the model configuration specifies the environment type (e.g., `conda`, `venv`, `docker`). Models should be defined in an isolated environment to ensure reproducibility and prevent conflicts with system dependencies. + +1. **venv**: A Python virtual environment (`venv`) setup is specified. The source code will be built by running the command ``pip install .`` within the virtual sub-environment (an environment within the one **floatCSEP** is run, but isolated from it), pointing to a ``setup.py``, ``setup.cfg`` or ``project.toml`` (See the `Packaging guide `_) + +2. **conda**: The model sub-environment is managed via a `conda` environment file (``environment.yml``). The model source-code will still be built using ``pip``. + +3. **docker**: A Docker container is created based on a provided `Dockerfile` that contains the instruction to build the source-code within.(`Writing a Dockerfile `_). If python, the model source-code will still be built using ``pip`` inside a virtual environment. + +.. note:: + All the environment names will be handled internally by **floatCSEP**. + +**Example setup.cfg** + + +.. code-block:: cfg + + [metadata] + name = cookie_model + description = Just another model + author = Monster, Cookie + + [options] + packages = + cookie_model + install_requires = + numpy + python_requires = >=3.9 + + [options.entry_points] + console_scripts = + cookie-run = cookie_model.main:run + +This build configuration installs the dependencies (``numpy``), the module ``cookie_model`` (i.e., the ``{source}`` folder) and creates an entrypoint command (see the :ref:`model_execution` section). + + + +**Example Dockerfile** + +.. code-block:: dockerfile + + # Use a specific Python version from a trusted source + FROM python:3.9.20 + + # Set up user and permissions + ARG USERNAME=modeler + ARG USER_UID=1100 + RUN useradd -u $USER_UID -m -s /bin/sh $USERNAME + + # Set work directory + WORKDIR /usr/src/ + + # Copy repository contents to the container + COPY --chown=$USERNAME cookie_model ./cookie_model/ + COPY --chown=$USERNAME setup.cfg ./ + + # Install the Python package and upgrade pip + RUN pip install --no-cache-dir --upgrade pip && pip install . + + # Set the default user + USER $USERNAME + + +This Dockerfile will install the python package inside a container, but the concept can be applied also for other programming languages. The ``func`` parameter will be used identically as done for ``conda`` and ``venv`` options, but now **floatCSEP** will handle the container execution and the entrypoint. + + +.. _model_execution: + +Model Entrypoint +~~~~~~~~~~~~~~~~ + +A model should be executed always with a shell command through a terminal. This provides flexibility to the modeler to abstract their model as convenient. +The **func** parameter in the model configuration defines the shell command used to execute the model. This command is invoked within the environment set up by **floatCSEP**, and will be run from ``model_path`` or the entrypoint defined in the ``Dockerfile``. + +Example ``func`` commands: + +.. code-block:: console + + $ cookie-run + $ python run.py + $ Rscript run.R + $ sh run.sh + +The ``cookie-run`` was a binary python entrypoint defined in the previous **Example setup.cfg**. It allows to execute the command ``cookie-run`` from the terminal, which itself will run the `python` function :func:`cookie_model.main.run` from the file ``cookie_model/main.py``. + +.. note:: + + This entrypoint function should contain the high-level logic of the model workflow (e.g, reading input, parsing arguments, calling core routines, write forecasts, etc.). An example pseudo-code of a model's workflow is: + + .. code-block:: R + + start, end, args = read_input(args_path) + training_catalog = read_catalog(input_cat) + parameters = fit(training_catalog) + forecast = create_forecast(start, end, args, parameters) + write(forecast) + + + +Input/Output Dataflow +~~~~~~~~~~~~~~~~~~~~~ + +The input to run a model will be placed into the ``model_path/input/`` directory dynamically by the testing system before each model execution. The model should be able to read these files from this directory. Similarly, after each model execution, the resulting forecast should be stored in a ``model_path/forecasts/`` directory + +We distinguish **input data** versus **input arguments**. The input data is given to a model without control of the modeler (e.g. authoritative input catalog, region), whereas input arguments (as in *function* arguments) can be the forecast specifications (e.g. time-window, target magnitudes) or hyper-parameters (e.g. declustering algorithm, optimization time-windows, cutoff magnitude) that control the model. + + +1. **Input Arguments**: The input arguments are the forecast specifications (e.g. time-window, target magnitudes) and hyper-parameters (e.g. declustering algorithm, optimization time-windows, cutoff magnitude) that will control the model. The input arguments will be written in the ``args_file`` (default ``args.txt``) always located in the input folder. A model requires at minimum one set of modifiable arguments: ``start_date`` and ``end_date`` (in ISO8601), but it is possible to include additional arguments. + + Example content of ``args.txt``: + + .. code-block:: yaml + + start_date: 2023-01-01T00:00:00 + end_date: 2023-01-02T00:00:00 + seed: 23 + nsims: 1000 + + Therefore, the model source-code should be at least able to dynamically read the obligatory arguments (simply the time window of the issued forecast) + +2. **Input Data**: Correspond to any data source outside the control of the modeler (e.g., authoritative input catalog, testing region). For now, **floatCSEP** just handles an input **catalog**, which are all the events within the **main catalog** until the forecast **start_date**. The catalog is written by default in ``model_path/input/catalog.csv`` in the CSEP ascii format (see :doc:`pycsep:concepts/catalogs`) as: + + .. code-block:: none + + longitude, latitude, magnitude, time_string, depth, event_id + + - **longitude**: Decimal degrees of the forecasted event location. + - **latitude**: Decimal degrees of the forecasted event location. + - **magnitude**: Magnitude of the forecasted event. + - **time_string**: Timestamp in UTC following the ISO8601 format (`%Y-%m-%dT%H:%M:%S`). + - **depth**: Depth of the event in kilometers. + - **event_id**: The event ID in case is necessary to map the event to an additional table. + + +3. **Output Forecasts**: After execution, forecast files should be written to the `forecasts/` folder. The forecast output must follow the filename convention: + + .. code-block:: none + + {model_name}_{start-date}_{end-date}.csv + + ``model_name`` can be replaced in the model configuration with the parameter ``prefix``, such that: + + .. code-block:: none + + {prefix}_{start-date}_{end-date}.csv + + + This ensures that forecast files are easily identified and retrieved by **floatCSEP** for further evaluation. + + + .. important:: + + The forecast files should adhere to the **pyCSEP** format. In summary, each forecast file should be a ``.csv`` file containing rows for each forecasted event, whose columns are: + + .. code-block:: none + + longitude, latitude, magnitude, time_string, depth, catalog_id, event_id + + where catalog_id represents the a single simulation of the stochastic catalog collection. This format ensures compatibility with the **pyCSEP** testing framework (See the `Catalog-based forecasts `_ documentation for further information). + + + + + diff --git a/docs/guide/postprocess_config.rst b/docs/guide/postprocess_config.rst new file mode 100644 index 0000000..9c3a325 --- /dev/null +++ b/docs/guide/postprocess_config.rst @@ -0,0 +1,499 @@ +.. _postprocess: + +Post-Process Options +==================== + +The ``postprocess`` inset can be used within an experiment configuration file to configure the **plotting** and **reporting** functions to be performed after the experiment calculations have been completed. The plotting functions provide a graphic representation of the catalogs, forecasts and evaluation results, whereas the reporting functions assemble these into a human-readable report. + +**Example postprocess configuration**: + +.. code-block:: yaml + :caption: config.yml + :emphasize-lines: 8- + + name: experiment + time_config: ... + region_config: ... + catalog: ... + model_config: ... + test_config: ... + + postprocess: + plot_forecasts: + colormap: magma + basemap: ESRI_terrain + catalog: True + + plot_catalog: + basemap: google-satellite + mag_ticks: [5, 6, 7, 8] + markersize: 7 + + +.. important:: + + By default, **floatCSEP** plots the testing catalogs, forecasts and results, summarizing them into a **Markdown** report. The postprocess configuration aids to customize these options, or to extend them by using custom python scripts. + +Plot Forecasts +-------------- + +**floatCSEP** can quickly plot the spatial rates of used and/or created forecasts. The ``plot_forecast`` command wraps the functionality of the **pyCSEP** function :func:`~csep.utils.plots.plot_spatial_dataset`, used to plot mean rates (in ``log10``) of both :class:`~csep.core.forecasts.GriddedForecast` and :class:`~csep.core.forecasts.CatalogForecast`. +Most arguments of ``plot_forecast`` mimics those of :func:`~csep.utils.plots.plot_spatial_dataset` with some extra additions. These are summarized here: + +.. dropdown:: Forecast plotting arguments + :animate: fade-in-slide-down + :icon: list-unordered + + .. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - **Arguments** + - **Description** + * - ``all_time_windows`` + - Whether all testing time windows are plotted (true or false). By default, only the last time window is plotted. + * - ``figsize`` + - List with the figure proportions. Default is `[6.4, 4.8]` + * - ``title`` + - Title for the plot. Default is None + * - ``title_size`` + - Size of the title text. Default is 10 + * - ``projection`` + - Projection for the map. Default ``cartopy.crs.PlateCarree`` Example: + + .. code-block:: yaml + + plot_forecasts: + projection: Mercator + + or if the projection contains keyword arguments: + + .. code-block:: yaml + + plot_forecasts: + projection: + Mercator: + central_longitude: 50 + + * - ``grid`` + - Whether to show grid lines. Default is True + * - ``grid_labels`` + - Whether to show grid labels. Default is True + * - ``grid_fontsize`` + - Font size for grid labels. Default is 10.0 + * - ``basemap`` + - Basemap option. Possible values are: ``stock_img``, ``google-satellite``, ``ESRI_terrain``, ``ESRI_imagery``, ``ESRI_relief``, ``ESRI_topo``, ``ESRI_terrain``, or a webservice URL. Default is None + * - ``coastline`` + - Flag to plot coastline. Default is True + * - ``borders`` + - Flag to plot country borders. Default is False + * - ``region_border`` + - Flag to plot the forecast region border. Default is True + * - ``tile_scaling`` + - Zoom level (1-12) for basemap tiles or ``auto`` for automatic scaling + * - ``linewidth`` + - Line width of borders and coastlines. Default is 1.5 + * - ``linecolor`` + - Color of borders and coastlines. Default is ``black`` + * - ``cmap`` + - Color map for the plot. Default is ``viridis`` + * - ``clim`` + - Range of the colorbar, in ``log10`` values. Example: ``[-5, 0]`` + * - ``clabel`` + - Label for the colorbar. Default is None + * - ``clabel_fontsize`` + - Font size for the colorbar label. Default is None + * - ``cticks_fontsize`` + - Font size for the colorbar ticks. Default is None + * - ``alpha`` + - Transparency level. Default is 1 + * - ``alpha_exp`` + - Exponent for the alpha function, recommended between 0.4 and 1. Default is 0 + * - ``catalog`` + - Plots the testing catalog on top of the forecast, corresponding to the forecast time window. + + .. code-block:: yaml + + plot_forecasts: + catalog: True + + and if the catalog needs to be customized: + + .. code-block:: yaml + + plot_forecasts: + catalog: + legend_loc: 1 + legend_fontsize: 14 + markercolor: blue + + See :ref:`plot_catalogs` for customization options. + + +.. important:: + + By default, only the forecast corresponding to the last time window of a model is plotted. To plot all time windows, use ``all_time_windows: True`` + + +.. _plot_catalogs: + +Plot Catalogs +------------- + +Test catalogs are automatically plotted when **floatCSEP** calculations are finished. Similar to plotting the forecasts, the ``plot_catalog`` command wraps the functionality of the **pyCSEP** function :func:`~csep.utils.plots.plot_catalog`. + + + +.. dropdown:: Catalog Plotting Arguments + :animate: fade-in-slide-down + :icon: list-unordered + + .. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - **Arguments** + - **Description** + * - ``all_time_windows`` + - If along the main testing catalogs, all sub-testing catalogs from all the time windows are plotted (true or false). Default is False. + * - ``figsize`` + - List or tuple with the figure proportions. Default is [6.4, 4.8]. + * - ``title`` + - Title for the plot. Default is the catalog name. + * - ``title_size`` + - Size of the title text. Default is 10. + * - ``filename`` + - File name to save the figure. Default is None. + * - ``projection`` + - Projection for the map. Default ``cartopy.crs.PlateCarree`` Example: + + .. code-block:: yaml + + plot_forecasts: + projection: Mercator + + or if the projection contains keyword arguments: + + .. code-block:: yaml + + plot_forecasts: + projection: + Mercator: + central_longitude: 50 + + * - ``basemap`` + - Basemap option. Possible values are: ``stock_img``, ``google-satellite``, ``ESRI_terrain``, ``ESRI_imagery``, ``ESRI_relief``, ``ESRI_topo``, ``ESRI_terrain``, or a webservice URL. Default is None + * - ``coastline`` + - Flag to plot coastline. Default is True. + * - ``grid`` + - Whether to display grid lines. Default is True. + * - ``grid_labels`` + - Whether to display grid labels. Default is True. + * - ``grid_fontsize`` + - Font size for grid labels. Default is 10.0. + * - ``marker`` + - Marker type for plotting earthquakes. + * - ``markersize`` + - Constant size for all earthquakes. + * - ``markercolor`` + - Color for all earthquakes. Default is ``blue``. + * - ``borders`` + - Flag to plot country borders. Default is False. + * - ``region_border`` + - Flag to plot the catalog region border. Default is True. + * - ``alpha`` + - Transparency level for the earthquake scatter. Default is 1. + * - ``mag_scale`` + - Scaling factor for the scatter plot based on earthquake magnitudes. + * - ``legend`` + - Flag to display the legend box. Default is True. + * - ``legend_loc`` + - Position of the legend (integer or string). Default is 'best'. + * - ``mag_ticks`` + - List of magnitude ticks to display in the legend. + * - ``labelspacing`` + - Separation between legend ticks. Default is 0.5. + * - ``tile_scaling`` + - Zoom level (1-12) for basemap tiles, or ``auto`` for automatic scaling based on extent. + * - ``linewidth`` + - Line width of borders and coastlines. Default is 1.5. + * - ``linecolor`` + - Color of borders and coastlines. Default is ``black``. + + +.. important:: + + By default, only the main test catalog (containing all events within the experiment frame) is plotted. To also plot the test catalogs from each time window separately, use ``all_time_windows: True`` + + +Custom Plotting +--------------- + +Additional plotting functionality can be injected to an experiment by using a custom **python** script, which is specified within the ``postprocess`` configuration: + +**Example:** + +.. code-block:: yaml + + postprocess: + plot_custom: plot_script.py:main + +where the script path and a function within should be written as: + +.. code-block:: yaml + + plot_custom: {python_script_path}:{function_name} + +This option provides a `hook` for python code to be run after the experiment calculation, giving it read access to attributes from the :class:`floatcsep.experiment.Experiment` class. The `hook` requirements are that the script to be located within the same directory as the configuration file, whereas the function must receive a :class:`floatcsep.experiment.Experiment` as unique argument: + + +**Example custom plot script**: + +.. code-block:: python + + from floatcsep import Experiment + + def main_function(experiment: Experiment): + + timewindows = experiment.timewindows + model = experiment.get_model("pymock") + + rates = [] + start_times = [] + + for timewindow in timewindows: + forecast = model.get_forecast(timewindow) + rates.append(forecast.event_counts) + start_times = timewindow[0] + + fig, ax = plt.subplots(1, 1) + ax.plot(start_times, rates) + pyplot.savefig("results/pymock_rates.png") + + +In this way, the plot function can use all the :class:`~floatcsep.experiment.Experiment` attributes/methods to access catalogs, forecasts and test results. Please check the :ref:`postprocess_api` and the Tutorial :ref:`case_g` for an advanced use. + + + +.. _custom_reporting: + +Custom Reporting +---------------- + +In addition to plotting, **floatCSEP** allows users to generate custom reports in **Markdown** format. The **MarkdownReport** class is designed to support the automatic creation of these reports, allowing users to assemble figures, text, and other results in a well-structured manner. + +The custom report functionality can be invoked by specifying the following in the ``postprocess`` configuration: + +**Example**: + +.. code-block:: yaml + :caption: config.yml + + postprocess: + report: report_script.py:generate_report + +This configuration specifies a custom **python** script with the following format: + +.. code-block:: yaml + + report: {python_script_path}:{function_name} + +The script must be located within the same directory as the configuration file and the function must receive an instance of :class:`floatcsep.experiment.Experiment` instance as its only argument. + +**Example Custom Report Script:**: + +.. code-block:: python + + from floatcsep.utils.reporting import MarkdownReport + + def generate_report(experiment): + # Create a MarkdownReport object + report = MarkdownReport(out_name="custom_report.md") + + # Add an introduction based on the experiment details + intro = { + 'simulation_name': experiment.name, + 'forecast_name': 'ETAS', + 'origin_time': experiment.start_date, + 'evaluation_time': experiment.end_date, + 'catalog_source': 'Observed Catalog', + 'num_simulations': 10000 + } + report.add_introduction(intro) + + # Add some text + report.add_text(['This report contains results from the ETAS model experiment.', 'Additional details below.']) + + + # Add a figure (for example, forecast rates over time) + report.add_figure( + title="Forecast Rates", + relative_filepaths=["results/2020-01-01_2020_01_02/forecasts/etas/forecast_rates.png"], + ncols=1, + caption="Forecasted seismicity rates over time." + ) + + # Save the report + report.save(save_dir="results") + + +The **MarkdownReport** class provides various methods for assembling a report, allowing the user to format the content, insert figures, add tables, and generate text dynamically based on the results of an experiment. + +For more advanced usage of report generation, please review the `default` **floatCSEP** report in the module :mod:`floatcsep.postprocess.reporting.generate_report`, an implementation example in the tutorial :ref:`case_h` and the :ref:`postprocess_api` for an advance use. + + +.. _postprocess_api: + +Postprocess API +--------------- + +Here are some basic functionalities from **floatCSEP** to access catalogs, forecasts and results using **python** code: + +.. dropdown:: Experiment and Catalogs + :animate: fade-in-slide-down + :icon: list-unordered + + .. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - **Method/Attribute** + - **Description** + * - :attr:`Experiment.timewindows ` + - A list of timewindows, where each is a pair of :class:`datetime.datetime` objects representing the window boundaries. + * - :attr:`Experiment.start_date ` + - The starting :class:`datetime.datetime` of the experiment. + * - :attr:`Experiment.end_date ` + - The end :class:`datetime.datetime` of the experiment. + * - :attr:`Experiment.region ` + - A :class:`csep.core.regions.CartesianGrid2D` object representing the spatial extent of the experiment. + * - :attr:`Experiment.mag_min ` + - The minimum magnitude of the experiment. + * - :attr:`Experiment.mag_max ` + - The maximum magnitude of the experiment. + * - :attr:`Experiment.mag_bin ` + - The magnitude bin size. + * - :attr:`Experiment.magnitudes ` + - A list of the magnitude bins of the experiment. + * - :attr:`Experiment.depth_min ` + - The minimum depth of the experiment. + * - :attr:`Experiment.depth_max ` + - The maximum depth of the experiment. + * - :attr:`Experiment.run_dir ` + - Returns the running directory of the experiment, where all evaluation results and figures are stored. Default is ``results/`` unless specified different in the ``config.yml``. + * - :attr:`Experiment.models ` + - Returns a list containing all the experiment's :class:`~floatcsep.model.Model` objects. + * - :meth:`Experiment.get_model(str) ` + - Returns a :class:`~floatcsep.model.Model` from its given name. + * - :attr:`Experiment.tests ` + - Returns a list containing all the experiment's :class:`~floatcsep.evaluation.Evaluation` objects. + * - :meth:`Experiment.get_test(str) ` + - Returns a :class:`~floatcsep.evaluation.Evaluation` from its given name + * - :attr:`Experiment.catalog_repo ` + - A :class:`~floatcsep.infrastructure.repositories.CatalogRepository` which can access the experiments catalogs. + * - :attr:`Experiment.catalog_repo.catalog ` + - The main catalog of the experiment, of :class:`csep.core.catalogs.CSEPCatalog` class. + * - :meth:`Experiment.catalog_repo.get_test_cat(timewindow) ` + - Returns the testing catalog for a given ``timewindow`` formatted as string. Use :func:`floatcsep.utils.helpers.timewindow2str` in case the window is a list of two :class:`datetime.datetime` objects. + + + +.. dropdown:: Models and forecasts + :animate: fade-in-slide-down + :icon: list-unordered + + The experiment models can be accessed by using :attr:`Experiment.models ` or :meth:`Experiment.get_model(str) `. + + .. list-table:: + :widths: 60 40 + :header-rows: 1 + + * - **Method/Attribute** + - **Description** + * - :attr:`Model.name ` + - Name of the model + * - :meth:`Model.get_forecast(timewindow) ` + - Returns the forecast for a given ``timewindow`` (formatted as string. Use :func:`floatcsep.utils.helpers.timewindow2str` in case the window is a list of two :class:`datetime.datetime` objects). Example: + + .. code-block:: python + + model = experiment.get_model('etas') + timewindow = experiment.timewindows[0] + timewindow_str = timewindow2str(timewindow) + model.get_forecast(timewindow_str) + + * - :attr:`Model.registry.path ` + - Directory of the model file or source code. + * - :attr:`Model.registry.database ` + - Database path where forecasts are stored. + * - :attr:`TimeIndependentModel.forecast_unit ` + - The forecast unit for a time independent model. + * - :meth:`TimeDependentModel.func ` + - The function command to execute a time dependent source code. + * - :meth:`TimeDependentModel.func_kwargs` + - The keyword arguments of the model, passed to the arguments file. + * - :meth:`TimeDependentModel.registry.args_file ` + - The path of the arguments file. Default is ``args.txt``. + * - :meth:`TimeDependentModel.registry.input_cat ` + - The path of the input catalog for the model execution. + + +.. dropdown:: Results + :animate: fade-in-slide-down + :icon: list-unordered + + The experiment evaluations can be accessed by using :attr:`Experiment.tests ` or :meth:`Experiment.get_test(str) `. + + .. list-table:: + :widths: 50 50 + :header-rows: 1 + + * - **Method/Attribute** + - **Description** + * - :meth:`Evaluation.read_results(timewindow, models) ` + - Returns the evaluation results for a given time window and models. Example usage: + + .. code-block:: python + + test = experiment.get_test('n_test') # get a test by its name + model = experiment.get_model('etas') # get a model by its name + timewindow = experiment.timewindows[0] # first time window + result = test.read_results(timewindow, model) + + or from all models: + + .. code-block:: python + + test = experiment.get_test('s_test') # get a test by its name + timewindow = experiment.timewindows[-1] # last time window + result = test.read_results(timewindow, experiment.models) + + + +.. dropdown:: MarkdownReport Methods + :animate: fade-in-slide-down + :icon: list-unordered + + .. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - **Method** + - **Description** + * - :meth:`MarkdownReport.add_introduction` + - Adds an introductory section to the report. This typically contains metadata such as the simulation name, forecast model, evaluation time, and other summary information. + * - :meth:`MarkdownReport.add_text` + - Adds text to the report. Each entry corresponds to a paragraph, and the text argument should be provided as a list of strings. + * - :meth:`MarkdownReport.add_figure` + - Inserts one or more figures into the report. You can specify the title, filepaths to the figures, and an optional caption. Figures are arranged in rows and columns as specified by the ``ncols`` argument. + * - :meth:`MarkdownReport.add_table` + - Creates a table in the report. The table data should be provided as a list of rows, where each row is a list of cell contents. + * - :meth:`MarkdownReport.add_list` + - Adds a bulleted list of items to the report. + * - :meth:`MarkdownReport.add_heading` + - Inserts a heading into the report. The ``level`` argument controls the heading level (1 for top-level, 2 for subheading, etc.). + * - :meth:`MarkdownReport.table_of_contents` + - Generates a table of contents based on the headings and sections included in the report so far. It will be automatically placed at the beginning of the report if an introduction is included. + * - :meth:`MarkdownReport.save` + - Saves the Markdown report to a specified directory. + diff --git a/docs/guide/region_config.rst b/docs/guide/region_config.rst deleted file mode 100644 index d9c8cd4..0000000 --- a/docs/guide/region_config.rst +++ /dev/null @@ -1,4 +0,0 @@ -Region definition -================= - -TBI \ No newline at end of file diff --git a/docs/guide/tests_config.rst b/docs/guide/tests_config.rst deleted file mode 100644 index fbca1bc..0000000 --- a/docs/guide/tests_config.rst +++ /dev/null @@ -1,4 +0,0 @@ -Evaluations definition -====================== - -TBI \ No newline at end of file diff --git a/docs/guide/time_config.rst b/docs/guide/time_config.rst deleted file mode 100644 index d0642a8..0000000 --- a/docs/guide/time_config.rst +++ /dev/null @@ -1,4 +0,0 @@ -Temporal definition -=================== - -TBI \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index d4db04a..774dc66 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,15 +1,143 @@ -.. floatCSEP documentation master file, created by - sphinx-quickstart on Wed Nov 16 09:56:55 2022. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - - +=============================== floatCSEP: Floating Experiments =============================== +*Earthquake forecasting experiments made simple.* + +.. image:: https://img.shields.io/badge/GitHub-Repository-blue?logo=github + :target: https://github.com/cseptesting/floatcsep + :alt: GitHub Repository + +.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.7953816.svg + :target: https://doi.org/10.5281/zenodo.7953816 + :alt: Zenodo + +.. image:: https://img.shields.io/github/license/cseptesting/floatcsep.svg + :target: https://github.com/cseptesting/floatcsep/blob/main/LICENSE + :alt: License + +.. image:: https://img.shields.io/pypi/v/floatcsep.svg + :target: https://pypi.org/project/floatcsep/ + :alt: PyPI version + +.. image:: https://img.shields.io/conda/vn/conda-forge/floatcsep.svg + :target: https://anaconda.org/conda-forge/floatcsep + :alt: Conda Version + + + +.. |start| image:: https://img.icons8.com/office/40/rocket.png + :target: intro/installation.html + :height: 48px + +.. |learn1| image:: https://img.icons8.com/nolan/64/literature.png + :target: https://docs.cseptesting.org/getting_started/core_concepts.html + :height: 48px + +.. |learn2| image:: https://i.postimg.cc/wMTtqBGB/icons8-literature-64.png + :target: https://docs.cseptesting.org/getting_started/theory.html + :height: 48px + +.. |experiment| image:: https://img.icons8.com/pulsar-color/48/science-application.png + :target: intro/concepts.html + :height: 48px + +.. |api| image:: https://img.icons8.com/nolan/64/code--v2.png + :target: reference/api_reference.html + :height: 48px + +.. |tutorials| image:: https://img.icons8.com/nolan/64/checklist.png + :target: tutorials/case_a.html + :height: 48px + + +Quickstart +---------- + ++--------------------------------------------------+-------------------------------------+ +| |start| **Get Started** | |tutorials| **Tutorials** | +| | | +| |learn1| **Forecasting Concepts** | - :ref:`case_a` | +| | - :ref:`case_b` | +| |learn2| **Testing Theory** | - :ref:`case_c` | +| | - :ref:`case_d` | +| |experiment| **Experiment Concepts** | - :ref:`case_e` | +| | - :ref:`case_f` | +| |api| **API Reference** | - :ref:`case_g` | +| | - :ref:`case_h` | ++--------------------------------------------------+-------------------------------------+ + +What is floatCSEP +----------------- + +The `Collaboratory for the Study of Earthquake Predictability `_ (CSEP) has organized Earthquake Forecast Testing Experiments during the last decades and is now consolidating its research into open-software initiatives. + +**floatCSEP** is an easy-to-use software application that contains the workflow to deploy Earthquake Forecasting Experiments. It is based on the code python library **pyCSEP** (`Github `_), which itself contains the core routines to test earthquake forecasts. + +Goals +----- + +* Test your forecasts with simple commands. +* Set up a testing experiment for your forecasts using authoritative data sources/benchmarks. +* Encapsulate the complete experiment's definition and rules in a couple of lines. +* Reproduce, reuse, and share forecasting experiments. + +Running +------- + +Start using **floatCSEP** by `installing `_ the latest version and running the ``tutorials`` with simply: + +.. code-block:: console + + $ floatcsep run config.yml + +Useful Links +------------ + ++---------------------------------------------------------+-----------------------------------------+ +| .. image:: https://img.icons8.com/nolan/64/github.png | **GitHub Repository** | +| :height: 48px | | +| :target: https://github.com/cseptesting/floatcsep | | ++---------------------------------------------------------+-----------------------------------------+ +| .. image:: https://i.postimg.cc/HW2Pssx1/logo-csep.png | **CSEP Website** | +| :height: 48px | | +| :target: https://cseptesting.org | | ++---------------------------------------------------------+-----------------------------------------+ +| .. image:: https://img.icons8.com/nolan/64/github.png | **pyCSEP GitHub** | +| :height: 48px | | +| :target: https://github.com/sceccode/pycsep | | ++---------------------------------------------------------+-----------------------------------------+ +| .. image:: https://img.icons8.com/nolan/64/github.png | **hazard2csep GitHub** | +| :height: 48px | | +| :target: https://github.com/cseptesting/hazard2csep | | ++---------------------------------------------------------+-----------------------------------------+ +| .. image:: https://img.icons8.com/nolan/64/europe.png | **European Testing Center** | +| :height: 48px | | +| :target: http://eqstats.efehr.org | | ++---------------------------------------------------------+-----------------------------------------+ + + + + +Collaborators +------------- + + * Pablo Iturrieta, GFZ Potsdam, Germany (pciturri@gfz-potsdam.de) + * William Savran, University of Nevada, Reno, USA + * Jose Bayona, University of Bristol, United Kingdom + * Francesco Serafini, University of Edinburgh, United Kingdom + * Kenny Graham, GNS Science, New Zealand + * Khawaja Asim, GFZ Potsdam, Germany + * Fabio Silva, Southern California Earthquake Center, USA + * Marcus Hermann, University of Naples ‘Frederico II’, Italy + * Max Werner, University of Bristol, United Kingdom + * Danijel Schorlemmner, GFZ Potsdam, Germany + * Philip Maechling, Southern California Earthquake Center, USA + + -Preliminary documentation. .. toctree:: + :hidden: :maxdepth: 1 :caption: Get Started @@ -18,44 +146,37 @@ Preliminary documentation. .. toctree:: + :hidden: :maxdepth: 1 - :caption: Example Experiments + :caption: Tutorial Experiments - examples/case_a.rst - examples/case_b.rst - examples/case_c.rst - examples/case_d.rst - examples/case_e.rst - examples/case_g.rst + tutorials/case_a.rst + tutorials/case_b.rst + tutorials/case_c.rst + tutorials/case_d.rst + tutorials/case_e.rst + tutorials/case_f.rst + tutorials/case_g.rst + tutorials/case_h.rst .. toctree:: :maxdepth: 2 + :hidden: :caption: Defining an Experiment - guide/config.rst - guide/time_config.rst - guide/region_config.rst + guide/experiment_config.rst guide/model_config.rst - guide/tests_config.rst - -.. toctree:: - :maxdepth: 2 - :caption: Deploying an Experiment + guide/evaluation_config.rst + guide/postprocess_config.rst + guide/executing_experiment.rst - deployment/intro.rst -.. toctree:: - :maxdepth: 0 +.. sidebar-links:: :caption: Help & Reference + :github: reference/api_reference - - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` + Getting Help + Contributing + License diff --git a/docs/intro/concepts.rst b/docs/intro/concepts.rst index dde7229..3036290 100644 --- a/docs/intro/concepts.rst +++ b/docs/intro/concepts.rst @@ -1,55 +1,82 @@ -Floating Experiments -==================== +Concepts +======== + +Forecasting Models +----------------------------- +An earthquake **Forecasting Model** is a representation of our understanding of seismicity, consisting of a collection of hypotheses, assumptions, data and methods. It is capable of generating **Forecasts**, i.e., `a probabilistic statement about the future occurrence of seismicity, which may include information about its magnitude and spatial location` (see the `Core Concepts `_ in the **pyCSEP** documentation). For now, we support earthquake forecasts expressed as: -Concept -------- + * **Gridded Forecasts**: Expected occurrence rate in a spatial-magnitude-temporal discretization. + * **Catalog Forecasts**: Families of synthetic earthquake catalogs. -The CSEP experiments consist in the prospective evaluation of probabilistic earthquake forecasting models. In these prospective experiments, the parameters of the experiment (including forecast generation, data sets, and evaluation metrics) must be defined with zero degrees of freedom before any evaluations begin. +From a computational perspective, a **Model** can be conceptualized as a **black-box system**, which receives an **input** (e.g. catalog, time window, target magnitude) to produce an **output** (a forecast). A model may consist of a single or a collection forecast (e.g., no input required and the output is given directly), or a forecast-generating source-code, which could require a training catalog to be calibrated. -A Floating Testing Experiment encapsulates each experiment into its own runnable environment, taking advantage of version control (i.e. ``git``), open-data repositories (e.g. `Zenodo `_) and Infrastructure as Code (e.g. `Docker `_) technologies, making it reproducible, re-usable and shareable during the time scale of the -evaluations. -``floatCSEP`` goals -------------------- +Forecasting Experiments +----------------------- -This is an application to deploy reproducible and prospective experiments of earthquake forecasting, namely a Floating Eperiment, that can operate independent of a particular testing server. With this application, researchers, institutions and users can +A **Forecasting Experiment** is defined here as the complete scientific process that encodes the questions, hypotheses to be addressed by **Forecasting Models**, and the **Evaluation** of such hypotheses and their results. +The purpose of an experiment is to ultimately lead to scientific and methodological improvements in our forecasting capabilities. - * Set up a testing experiment for your earthquake forecasts using authoritative data sources and benchmarks. - * Encapsulate the complete experiment's definition and rules in a couple of lines. - * Produce human-readable results and figures. - * Reproduce, reuse, and share forecasting experiments. +In **Prospective Experiments**, the parameters of the experiment (including forecast generation, data sets, and evaluation metrics) must be defined with zero degrees of freedom before any evaluations begin. Prospective experiments provide the most objective view of a model's forecasting skill, by removing any unconscious (or conscious) bias of the modelers during forecast production. On the other hand, **Retrospective** experiments or **Pseudo-Prospective** experiments, where the testing data is known to the modeler, are also important during model development and should be carried out as standard scientific praxis. -Collaborators -------------- - * Pablo Iturrieta, GFZ Potsdam (pciturri@gfz-potsdam.de) +.. figure:: ../_static/experiment_classes.png + :alt: Floating Experiments + :width: 80% + :align: center - * William Savran, UNR + Different experiment classes depending on the data temporality and its availability to the modeler. Figure from Mizrahi et al., (2024). - * Fabio Silva, Southern California Earthquake Center - * Khawaja Asim, GFZ Potsdam +Examples of past prospective experiments are: - * Jose Bayona, University of Bristol - * Leila Mizrahi, ETH +.. list-table:: + :header-rows: 1 + :widths: 20 80 - * Kirsty Bayliss, GEM + * - Region + - References + * - California + - * `Schorlemmer, D., & Gerstenberger, M. (2007). RELM testing center. Seismological Research Letters, 78(1), 30-36. `_ + * `Schorlemmer, D., et al. (2010). First results of the regional earthquake likelihood models experiment. Seismogenesis and Earthquake Forecasting: The Frank Evison Volume II, 5-22. `_ + * `Strader, A., et al. (2017). Prospective and retrospective evaluation of five-year earthquake forecast models for California. Geophysical Journal International, 211-1, 239–251. `_ + * - Japan + - * `Nanjo, K., et al. (2011). Overview of the first earthquake forecast testing experiment in Japan, Earth Planets Space, 63 (3), 159–169 `_ + * `Tsuruoka, H., et al., (2012). CSEP Testing Center and the first results of the earthquake forecast testing experiment in Japan. Earth, planets and space, 64, 661-671. `_ - * Francesco Serafini, University of Edinburgh + * - New Zealand + - * `Gerstenberger, M., & Rhoades, D. (2010). New Zealand earthquake forecast testing centre. Seismogenesis and Earthquake Forecasting: The Frank Evison Volume II, 23-38. `_ + * `Rhoades, D., et al. (2018). Highlights from the first ten years of the New Zealand earthquake forecast testing center. Seismological Research Letters, 89(4), 1229-1237. `_ + * - Italy + - * `Schorlemmer, D., et al. (2010). Setting up an earthquake forecast experiment in Italy. Annals of Geophysics. `_ + * `Taroni, M., et al. (2018). Prospective CSEP evaluation of 1‐day, 3‐month, and 5‐yr earthquake forecasts for Italy. Seismological Research Letters, 89(4), 1251-1261. `_ + * `Iturrieta, P., et al. (2024). Evaluation of a Decade-Long Prospective Earthquake Forecasting Experiment in Italy. Seismological Research Letters. `_ + +Floating Experiments +-------------------- - * Marcus Hermann, University of Naples ‘Frederico II’ +They are a new conceptual framework for modern prospective experiments, whose operation rely on version control systems (i.e. ``git``), open-data repositories ((e.g. `Zenodo `_) and the containerization of computational environments (e.g., `Docker `_), making experiments reproducible, re-usable and shareable during the time scale of the evaluations. **Floating Experiments** are computational reproducibility packages (e.g., `World Bank `_) expanded to a dynamic implementation, as new earthquake data becomes available in time and new testing results can be continuously released. - * Max Werner, University of Bristol +.. figure:: ../_static/float_scheme.png + :alt: Floating Experiments + :width: 50% + :align: center - * Danijel Schorlemmner, GFZ Potsdam + The forecasting experiment is stored along with the system (**floatCSEP**) and testing routines (**pyCSEP**). It can be cloned to a local machine and run to create results, by using a containerized environment. Results can then be published back into the same repositories, tagging a version/release for each update. - * Philip Maechling, Southern California Earthquake Center +**floatCSEP** assists scientists and institutions in the deployment of forecasting experiments, by standardizing and curating the artifacts and methods required to continuously run and/or reproduce an experiment, without it being coupled to a fixed physical infrastructure. +References +---------- + * Mizrahi, L., Dallo, I., van der Elst, N. J., Christophersen, A., Spassiani, I., Werner, M. J., et al. (2024). Developing, testing, and communicating earthquake forecasts: Current practices and future directions. Reviews of Geophysics, 62, e2023RG000823. https://doi.org/10.1029/2023RG000823 + * Iturrieta, P., Savran, W. H., Khawaja, M. A. M., Bayona, J., Maechling, P. J., Silva, F., et al. (2023). Modernizing earthquake forecasting experiments: The CSEP floating experiments. In AGU Fall Meeting Abstracts (Vol. 2023). + * Savran, W. H., Bayona, J. A., Iturrieta, P., Asim, K. M., Bao, H., et al. (2022). pyCSEP: a Python toolkit for earthquake forecast developers. Seismological Society of America, 93(5), 2858-2870. https://doi.org/10.1785/0220220033 + * Krafczyk, M. S., Shi, A., Bhaskar, A., Marinov, D., Stodden, V., (2021). Learning from reproducing computational results: Introducing three principles and the Reproduction Package. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 379, 20200069. https://doi.org/10.1098/rsta.2020.0069. diff --git a/docs/intro/installation.rst b/docs/intro/installation.rst index 24b6eda..d22693a 100644 --- a/docs/intro/installation.rst +++ b/docs/intro/installation.rst @@ -1,55 +1,104 @@ Installation ============ - .. note:: +.. important:: - This application uses ``python >= 3.8`` + This application uses ``3.9 <= python <= 3.11`` -Installing the latest version ------------------------------ -Using ``conda`` -~~~~~~~~~~~~~~~ +Latest Version +-------------- -The core of `floatCSEP` is built around the `pyCSEP` package (https://github.com/sceccode/pycsep), which itself contains the core dependencies. +Recommended to learn the software, run the tutorials, and drafting **Testing Experiments**. -The simplest way to install `floatCSEP`, is by creating a `conda` environment (https://conda.io - checkout Anaconda or Miniconda) and install `pyCSEP` from `conda-forge` +1. Using ``conda`` +~~~~~~~~~~~~~~~~~~ + +To install **floatCSEP**, first a ``conda`` manager should be installed (https://conda.io). Checkout `Anaconda`, `Miniconda` or `Miniforge` (recommended). Once installed, create an environment with: .. code-block:: console - $ conda env create -n $NAME - $ conda activate $NAME - $ conda install -c conda-forge pycsep + $ conda env create -n csep_env + $ conda activate csep_env Then, clone and install the floatCSEP source code using ``pip`` .. code-block:: console - git clone https://github.com/cseptesting/floatcsep - cd floatcsep - pip install . + $ git clone https://github.com/cseptesting/floatcsep + $ cd floatcsep + $ pip install . + +.. note:: + + Use the ``mamba`` command instead of ``conda`` if `Miniforge` was installed. + + +2. Using ``pip`` only +~~~~~~~~~~~~~~~~~~~~~ + +To install using the ``pip`` manager only, we require to install the binary dependencies of **pyCSEP** (see `Installing pyCSEP `_}. The **floatCSEP** latest version can then be installed as: + + .. code-block:: console + + $ git clone https://github.com/cseptesting/floatcsep + $ cd floatcsep + $ python -m venv venv + $ pip install . + + +Latest Stable Release +--------------------- + +Recommended for deploying live Floating Testing Experiments + +1. From the ``conda-forge`` channel +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Having a ``conda`` manager installed (https://conda.io), type in a console: + + + .. code-block:: console + + $ conda env create -n csep_env + $ conda activate csep_env + $ conda install -c conda-forge floatcsep + + +2. From the ``PyPI`` repository +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Having installed the binary dependencies of **pyCSEP** (see `Installing pyCSEP `_}, install **floatCSEP** by: + + .. code-block:: console + + $ python -m venv venv + $ pip install floatcsep -Using ``apt`` and ``pip`` -~~~~~~~~~~~~~~~~~~~~~~~~~ +.. important:: + If you want to run the tutorials from a **floatCSEP** installation obtained through ``conda-forge`` or ``PyPI``, the tutorials can be downloaded to your current directory as: -To install from ``pip``, we require to install the binary dependencies of ``pyCSEP`` (see `Installing pyCSEP `_} + .. code-block:: console -Then, install the ``pycsep`` latest version + $ latest_version=$(curl --silent "https://api.github.com/repos/cseptesting/floatcsep/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/') && \ + wget "https://github.com/cseptesting/floatcsep/releases/download/$latest_version/tutorials.zip" + $ unzip tutorials.zip -d ./ && rm tutorials.zip - .. code-block:: + Or downloaded manually from the `latest release `_. - git clone https://github.com/SCECcode/pycsep - cd pycsep - python -m virtualenv venv - source venv/bin/activate - pip install -e .[all] -and the ``floatcsep`` latest version - .. code-block:: +For Developers +-------------- + +It is recommended (not obligatory) to use a ``conda`` environment to make sure your contributions do not depend on your system local libraries. For contributions to the **floatCSEP** codebase, please consider using a `fork `_ and creating pull-requests from there. + + .. code-block:: console - cd .. - git clone https://github.com/cseptesting/floatcsep - cd floatcsep - pip install .[all] + $ conda env create -n csep_dev + $ conda activate csep_dev + $ git clone https://github.com/${your_fork}/floatcsep + $ cd floatcsep + $ pip install .[dev] +This will install and configure all the unit-testing, linting and documentation packages. diff --git a/docs/tutorials/case_a.rst b/docs/tutorials/case_a.rst new file mode 100644 index 0000000..a69c15b --- /dev/null +++ b/docs/tutorials/case_a.rst @@ -0,0 +1,162 @@ +.. _case_a: + +A - Testing a Simple Model +========================== + +The following example shows the definition of a testing experiment of a single **time-independent** forecast against a catalog. + +.. currentmodule:: floatcsep + +.. admonition:: **TL; DR** + + In a terminal, navigate to ``floatcsep/tutorials/case_a`` and type: + + .. code-block:: console + + $ floatcsep run config.yml + + After the calculation is complete, the results will be summarized in ``results/report.md``. + +.. contents:: Contents + :local: + :depth: 2 + + +Experiment Components +--------------------- + +The source code can be found in the ``tutorials/case_a`` folder or in `GitHub `_. The directory structure of the experiment is: + +:: + + case_a + ├── region.txt + ├── catalog.csep + ├── best_model.dat + └── config.yml + + +* The testing region ``region.txt`` consists of a grid with two 1ºx1º bins, defined by its bottom-left nodes. The grid spacing is obtained automatically. The nodes are: + + .. literalinclude:: ../../tutorials/case_a/region.txt + :caption: tutorials/case_a/region.txt + +* The testing catalog ``catalog.csep`` contains only one event and is formatted in the :meth:`~pycsep.utils.readers.csep_ascii` style (see :doc:`pycsep:concepts/catalogs`). Catalog formats are detected automatically + + .. literalinclude:: ../../tutorials/case_a/catalog.csep + :caption: tutorials/case_a/catalog.csep + +* The forecast ``best_model.dat`` to be evaluated is written in the ``.dat`` format (see :doc:`pycsep:concepts/forecasts`). Forecast formats are detected automatically (see :mod:`floatcsep.utils.readers.ForecastParsers`) + + .. literalinclude:: ../../tutorials/case_a/best_model.dat + :caption: tutorials/case_a/best_model.dat + + +Configuration +------------- + +The experiment is defined by a time-, region-, model- and test-configurations, as well as a catalog and a region. In this example, they are written together in the ``config.yml`` file. + + +.. important:: + + Every file path (e.g., of a catalog) specified in the ``config.yml`` file should be relative to the directory containing the configuration file. + + + +Time +~~~~ + + The time configuration is manifested in the ``time_config`` inset. The simplest definition is to set only the start and end dates of the experiment. These are always UTC date-times in isoformat (``%Y-%m-%dT%H:%M:%S.%f`` - ISO861): + + .. literalinclude:: ../../tutorials/case_a/config.yml + :caption: tutorials/case_a/config.yml + :language: yaml + :lines: 3-5 + + .. note:: + + In case the time window are bounded by their midnights, the ``start_date`` and ``end_date`` can be in the format ``%Y-%m-%d``. + + The results of the experiment run will be associated with this time window, whose identifier will be its bounds: ``2020-01-01_2021-01-01`` + +Region +~~~~~~ + + The region - a file path or a :mod:`pycsep` function, such as :obj:`~csep.core.regions.italy_csep_region` (check the available regions in :mod:`csep.core.regions`) -, the depth limits and magnitude discretization are defined in the ``region_config`` inset. + + .. literalinclude:: ../../tutorials/case_a/config.yml + :caption: tutorials/case_a/config.yml + :language: yaml + :lines: 7-13 + + +Catalog +~~~~~~~ + + It is defined in the ``catalog`` inset. This should only make reference to a catalog **file** or a catalog **query function** (e.g. :func:`~csep.query_comcat`). **floatCSEP** will automatically filter the catalog to the experiment time, spatial and magnitude frames: + + .. literalinclude:: ../../tutorials/case_a/config.yml + :caption: tutorials/case_a/config.yml + :language: yaml + :lines: 15-15 + +Models +~~~~~~ + The model configuration is set in the ``models`` inset with a list of model names, which specify their file paths (and other attributes). Here, we just set the path as ``best_model.dat``, whose format is automatically detected. + + .. literalinclude:: ../../tutorials/case_a/config.yml + :caption: tutorials/case_a/config.yml + :language: yaml + :lines: 17-19 + + .. note:: + + A time-independent forecast model has default units of ``[eq/year]`` per cell. A forecast defined for a different number of years can be specified with the ``forecast_unit: {years}`` attribute. + +Evaluations +~~~~~~~~~~~ + The experiment's evaluations are defined in the ``tests`` inset. It should be a list of test names making reference to their function and plotting function. These can be either from **pyCSEP** (see :doc:`pycsep:concepts/evaluations`) or defined manually. Here, we use the Poisson consistency N-test: its function is :func:`poisson_evaluations.number_test ` with a plotting function :func:`plot_poisson_consistency_test ` + +.. literalinclude:: ../../tutorials/case_a/config.yml + :caption: tutorials/case_a/config.yml + :language: yaml + :lines: 21-24 + + +Running the experiment +---------------------- + +Run command +~~~~~~~~~~~ + + The experiment can be run by simply navigating to the ``tutorials/case_a`` folder in the terminal and typing. + + .. code-block:: console + + $ floatcsep run config.yml + + This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. + + .. note:: + + The command ``floatcsep run {config_file}`` can be called from any working directory, as long as the specified file paths (e.g. region, models) are relative to the ``config.yml`` file. + + +Results +~~~~~~~ + + The :obj:`~floatcsep.cmd.main.run` command creates the result path tree for each time window analyzed. + + * The testing catalog of the window is stored in ``results/{window}/catalog`` in ``json`` format. This is a subset of the global testing catalog. + * Human-readable results are found in ``results/{window}/evaluations`` + * Catalog, forecasts and evaluation results figures in ``results/{window}/figures``. + * The complete results are summarized in ``results/report.md`` + + +Advanced +~~~~~~~~ + +The experiment run logic can be seen in the file ``case_a.py``, which executes the same example but in python source code. The run logic of the terminal commands ``run``, ``plot`` and ``reproduce`` can be found in :mod:`floatcsep.commands.main`, and can be customized by creating a script similar to ``case_a.py``. + + diff --git a/docs/examples/case_b.rst b/docs/tutorials/case_b.rst similarity index 58% rename from docs/examples/case_b.rst rename to docs/tutorials/case_b.rst index 6e07d7b..4765b12 100644 --- a/docs/examples/case_b.rst +++ b/docs/tutorials/case_b.rst @@ -1,14 +1,15 @@ +.. _case_b: + B - Multiple Models and Tests ============================= -.. currentmodule:: floatcsep +The following example is an experiment including **multiple** time-independent forecasts and **multiple** evaluations. -.. contents:: - :local: +.. currentmodule:: floatcsep .. admonition:: **TL; DR** - In a terminal, navigate to ``floatcsep/examples/case_b`` and type: + In a terminal, navigate to ``floatcsep/tutorials/case_b`` and type: .. code-block:: console @@ -16,16 +17,20 @@ B - Multiple Models and Tests After the calculation is complete, the results will be summarized in ``results/report.md``. +.. contents:: Contents + :local: + :depth: 2 -Artifacts ----------------------- -The following example is an experiment including multiple forecasts and evaluations. The input structure of the experiment is: +Experiment Components +--------------------- + +The source code can be found in the ``tutorials/case_b`` folder or in `GitHub `_. The input structure of the experiment is: :: case_b - └── models + └── models ├── model_a.csv ├── model_b.csv ├── model_c.csv @@ -36,11 +41,11 @@ The following example is an experiment including multiple forecasts and evaluati ├── tests.yml └── region.txt - -The testing catalog is now defined in ``json`` format, which is the default catalog used by ``floatcsep``, as it allows the storage of metadata. +.. important:: + Although not necessary, the testing catalog is here defined in the ``.json`` format, which is the default catalog used by ``floatcsep``, as it allows the storage of metadata. .. note:: - An user-defined catalog can be saved as ``json`` with :meth:`CSEPCatalog.write_json() ` using ``pycsep`` + A catalog can be stored as ``.json`` with :meth:`CSEPCatalog.write_json() ` using ``pycsep`` Configuration @@ -48,13 +53,14 @@ Configuration In this example, the time, region and catalog specifications are written in the ``config.yml`` file. -.. literalinclude:: ../../examples/case_b/config.yml +.. literalinclude:: ../../tutorials/case_b/config.yml + :caption: tutorials/case_b/config.yml :language: yaml :lines: 3-15 -whereas the models' and tests' configurations are referred to external files for readability +whereas the models' and tests' configurations are referred to external files for better readability -.. literalinclude:: ../../examples/case_b/config.yml +.. literalinclude:: ../../tutorials/case_b/config.yml :language: yaml :lines: 17-18 @@ -63,31 +69,33 @@ Models ~~~~~~ The model configuration is now set in the ``models.yml`` file, where a list of model names specify their file paths. - .. literalinclude:: ../../examples/case_b/models.yml + .. literalinclude:: ../../tutorials/case_b/models.yml + :caption: tutorials/case_b/models.yml :language: yaml Evaluations ~~~~~~~~~~~ The evaluations are defined in the ``tests.yml`` file as a list of evaluation names, with their functions and plots (see :doc:`pycsep:concepts/evaluations`). In this example, we use the N-, M-, S- and CL-consistency tests, along with the comparison T-test. - .. literalinclude:: ../../examples/case_b/tests.yml + .. literalinclude:: ../../tutorials/case_b/tests.yml :language: yaml + :caption: tutorials/case_b/tests.yml .. note:: Plotting keyword arguments can be set in the ``plot_kwargs`` option - see :func:`~csep.utils.plots.plot_poisson_consistency_test` and :func:`~csep.utils.plots.plot_comparison_test` -. - .. note:: + .. important:: Comparison tests (such as the ``paired_t_test``) requires a reference model, whose name should be set in ``ref_model`` at the given test configuration. Running the experiment ---------------------- -The experiment can be run by simply navigating to the ``examples/case_b`` folder in the terminal an type. +The experiment can be run by simply navigating to the ``tutorials/case_b`` folder in the terminal an type. .. code-block:: console - floatcsep run config.yml + $ floatcsep run config.yml This will automatically set all the file paths of the calculation (testing catalogs, evaluation results, figures) and will display a summarized report in ``results/report.md``. diff --git a/docs/tutorials/case_c.rst b/docs/tutorials/case_c.rst new file mode 100644 index 0000000..a27f771 --- /dev/null +++ b/docs/tutorials/case_c.rst @@ -0,0 +1,95 @@ +.. _case_c: + +C - Multiple Time Windows +========================= + +The following example shows an experiment with **multiple time windows**. + +.. currentmodule:: floatcsep + +.. admonition:: **TL; DR** + + In a terminal, navigate to ``floatcsep/tutorials/case_c`` and type: + + .. code-block:: console + + $ floatcsep run config.yml + + After the calculation is complete, the results will be summarized in ``results/report.md``. + +.. contents:: Contents + :local: + + + +Experiment Components +--------------------- + +The source code can be found in the ``tutorials/case_c`` folder or in `GitHub `_. The input structure of the experiment is: + +:: + + case_c + └── models + ├── model_a.csv + ├── model_b.csv + ├── model_c.csv + └── model_d.csv + ├── config.yml + ├── catalog.json + ├── models.yml + ├── tests.yml + └── region.txt + +Configuration +------------- + +Time +~~~~ + + The time configuration now sets a sequence of time intervals between the start and end dates. + + .. literalinclude:: ../../tutorials/case_c/config.yml + :caption: tutorials/case_c/config.yml + :language: yaml + :lines: 3-7 + + .. note:: + + The time interval ``growth`` can be either ``cumulative`` (all windows start from ``start_date``) or ``incremental`` (each window starts from the previous window's end). + + The results of the experiment run will be associated with each time window (``2010-01-01_2011-01-01``, ``2010-01-01_2012-01-01``, ``2010-01-01_2013-01-01``, ...). + + + +Evaluations +~~~~~~~~~~~ + The experiment's evaluations are defined in ``tests.yml``, which can now include temporal evaluations (see :obj:`~floatcsep.utils.helpers.sequential_likelihood`, :obj:`~floatcsep.utils.helpers.sequential_information_gain`, :obj:`~floatcsep.utils.helpers.plot_sequential_likelihood`). + + .. literalinclude:: ../../tutorials/case_c/tests.yml + :language: yaml + :caption: tutorials/case_c/tests.yml + + .. note:: + + Plot arguments (title, labels, font sizes, axes limits, etc.) can be passed as a dictionary in ``plot_args`` (see the arguments details in :func:`~csep.utils.plots.plot_poisson_consistency_test`) + +Results +------- + +The :obj:`~floatcsep.commands.main.run` command + +.. code-block:: console + + $ floatcsep run config.yml + +now creates the result path tree for all time windows. + +* The testing catalog of the window is stored in ``results/{time_window}/catalog`` in ``json`` format. This is a subset of the global testing catalog. +* Human-readable results are found in ``results/{time_window}/evaluations`` +* Catalog and evaluation results figures in ``results/{time_window}/figures``. +* The complete results are summarized in ``results/report.md`` + +The report shows the temporal evaluations for all time-windows, whereas the discrete evaluations are shown only for the last time window. + + diff --git a/docs/examples/case_d.rst b/docs/tutorials/case_d.rst similarity index 64% rename from docs/examples/case_d.rst rename to docs/tutorials/case_d.rst index 01f270a..3fdc51e 100644 --- a/docs/examples/case_d.rst +++ b/docs/tutorials/case_d.rst @@ -1,14 +1,17 @@ -D - Catalog Queries and Model Repositories -========================================== +.. _case_d: -.. currentmodule:: floatcsep +D - Catalog and Model Queries +============================= -.. contents:: - :local: +The following example shows an experiment whose forecasts are **retrieved from a repository** (Zenodo - https://zenodo.org) and the testing **catalog** from an authoritative source **web service** (namely the gCMT catalog from the International Seismological Centre - http://www.isc.ac.uk). + + + +.. currentmodule:: floatcsep .. admonition:: **TL; DR** - In a terminal, navigate to ``floatcsep/examples/case_d`` and type: + In a terminal, navigate to ``floatcsep/tutorials/case_d`` and type: .. code-block:: console @@ -16,11 +19,14 @@ D - Catalog Queries and Model Repositories After the calculation is complete, the results will be summarized in ``results/report.md``. +.. contents:: Contents + :local: + -Artifacts ---------- +Experiment Components +--------------------- -The following example shows an experiment whose forecasts are retrieved from a repository (Zenodo - https://zenodo.org) and the testing catalog from an authoritative source web service (namely the gCMT catalog from the ISC - http://www.isc.ac.uk). The initial structure is: +The source code can be found in the ``tutorials/case_d`` folder or in `GitHub `_. The **initial** input structure of the experiment is: :: @@ -57,31 +63,33 @@ Configuration Catalog ~~~~~~~ - The ``catalog`` inset from ``config.yml`` now makes reference to a catalog query function, in this case :func:`~floatcsep.accessors.query_isc_gcmt`. + The ``catalog`` inset from ``config.yml`` now makes reference to a catalog query function, in this case :func:`~pycsep.query_gcmt`. - .. literalinclude:: ../../examples/case_d/config.yml - :language: yaml - :lines: 14-14 + .. literalinclude:: ../../tutorials/case_d/config.yml + :caption: tutorials/case_d/config.yml + :language: yaml + :lines: 14-14 ``floatcsep`` will automatically filter the catalog to the experiment time, spatial and magnitude windows of the experiment. .. note:: - Query functions exist in both ``pycsep`` and ``floatcsep`` (e.g. :func:`csep.query_comcat`, :func:`csep.query_bsi`, :func:`~csep.query_gcmt`, :func:`~csep.query_gns`). Only the name of the function is needed to retrieve the catalog. Refer to :obj:`csep` and :class:`floatcsep.accessors`. + Query functions are located in ``pycsep`` (e.g. :func:`csep.query_comcat`, :func:`csep.query_bsi`, :func:`csep.query_gcmt`, :func:`csep.query_gns`). Only the name of the function is needed to retrieve the catalog. Refer to :obj:`csep` API reference. Models ~~~~~~ The model configuration is set in ``models.yml``. - .. literalinclude:: ../../examples/case_d/models.yml - :language: yaml + .. literalinclude:: ../../tutorials/case_d/models.yml + :caption: tutorials/case_d/models.yml + :language: yaml * The option ``zenodo_id`` makes reference to the zenodo **record id**. The model ``team`` is found in https://zenodo.org/record/6289795, whereas the model ``wheel`` in https://zenodo.org/record/6255575. - * The option ``flavours`` allows multiple model sub-classes to be quickly instantiated. - * The ``zenodo`` (or ``git``) repositories could contain multiple files, each of which can be assigned to a flavour. + * The option ``flavours`` allows multiple model sub-classes to be quickly instantiated. + * When multiple flavours are passed, ``path`` refers to the folder where the models would be downloaded. * If a single file of the repository is needed (without specifying model flavours), ``path`` can reference to the file itself. For example, you can try replacing the whole WHEEL inset in ``models.yml`` to: @@ -96,7 +104,7 @@ Models Running the experiment ---------------------- - The experiment can be run by simply navigating to the ``examples/case_d`` folder in the terminal and typing. + The experiment can be run by simply navigating to the ``tutorials/case_d`` folder in the terminal and typing. .. code-block:: console diff --git a/docs/tutorials/case_e.rst b/docs/tutorials/case_e.rst new file mode 100644 index 0000000..061e6e0 --- /dev/null +++ b/docs/tutorials/case_e.rst @@ -0,0 +1,139 @@ +.. _case_e: + +E - A Time-Independent Experiment +================================= + +This example shows how to run a realistic testing experiment (based on https://doi.org/10.4401/ag-4844) while summarizing the concepts from the previous tutorials. + +.. currentmodule:: floatcsep + +.. admonition:: **TL; DR** + + In a terminal, navigate to ``floatcsep/tutorials/case_e`` and type: + + .. code-block:: console + + $ floatcsep run config.yml + + After the calculation is complete, the results will be summarized in ``results/report.md``. + +.. contents:: Contents + :local: + + + +Experiment Components +--------------------- + +The source code can be found in the ``tutorials/case_e`` folder or in `GitHub `_. The input structure of the experiment is: + +:: + + case_e + └── models + ├── gulia-wiemer.ALM.italy.10yr.2010-01-01.xml + ├── meletti.MPS04.italy.10yr.2010-01-01.xml + └── zechar.TripleS-CPTI.italy.10yr.2010-01-01.xml + ├── config.yml + ├── models.yml + └── tests.yml + +.. note:: + This experiment has only a subset of the original models and evaluations. + + +Configuration +------------- + + +Time +~~~~ + + The time configuration is manifested in the ``time-config`` inset. + + .. literalinclude:: ../../tutorials/case_e/config.yml + :caption: tutorials/case_e/config.yml + :language: yaml + :lines: 3-7 + +Region +~~~~~~ + + The testing region is the official Italy CSEP Region obtained from :obj:`csep.core.regions.italy_csep_region`. + + .. literalinclude:: ../../tutorials/case_e/config.yml + :caption: tutorials/case_e/config.yml + :language: yaml + :lines: 9-15 + + +Catalog +~~~~~~~ + + The catalog is obtained from an authoritative source, namely the Bollettino Sismico Italiano (http://terremoti.ingv.it/en/bsi ), using the function :func:`csep.query_bsi` + + .. literalinclude:: ../../tutorials/case_e/config.yml + :caption: tutorials/case_e/config.yml + :language: yaml + :lines: 17-17 + +Models +~~~~~~ + The models are set in ``models.yml``. For simplicity, there are only three of the nineteen models originally submitted to the Italy Experiment. + + .. literalinclude:: ../../tutorials/case_e/models.yml + :caption: tutorials/case_e/models.yml + :language: yaml + + The ``.xml`` format is automatically detected and parsed by ``floatcsep`` readers. + + .. important:: + + The forecasts are defined in ``[Earthquakes / 10-years]``, which is specified with the ``forecast_unit`` option (The default is `forecast_unit: 1`). + + .. note:: + + The ``use_db`` flag allows ``floatcsep`` to transform the forecasts into a database (HDF5), which speeds up the calculations. + +Post-Process +~~~~~~~~~~~~ + + Additional options for post-processing can set using the ``postprocess`` option. Here, we customize the forecasts plotting with: + + .. literalinclude:: ../../tutorials/case_e/config.yml + :language: yaml + :lines: 21-34 + + The forecasts are plotted and stored in ``tutorials/case_e/results/{timewindow}/forecasts/``. See :func:`~csep.utils.plots.plot_spatial_dataset` for forecast plotting options and :func:`~csep.utils.plots.plot_catalog` for the catalog placed on top of those plots. + + +Running the experiment +---------------------- + + The experiment can be run by navigating to the ``tutorials/case_e`` folder in the terminal and typing. + + .. code-block:: console + + $ floatcsep run config.yml + + This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. + + +Plot command +~~~~~~~~~~~~ + + If only the result plots are desired when the calculation was already completed before, you can type: + + .. code-block:: console + + $ floatcsep plot config.yml + + This can be used, for example, when an additional plot is desired without re-running the entire experiment. Try adding the following line to the ``postprocess`` inset of the ``config.yml`` file. + + .. code-block:: yaml + + postprocess: + plot_forecasts: + colormap: magma + + and re-run with the ``plot`` command. A forecast figure will re-appear in ``results/{window}/forecasts`` with a different colormap. Additional forecast and catalog plotting options can be found in the :func:`csep.utils.plots.plot_spatial_dataset` and :func:`csep.utils.plots.plot_catalog` ``pycsep`` functions. diff --git a/docs/tutorials/case_f.rst b/docs/tutorials/case_f.rst new file mode 100644 index 0000000..26438a3 --- /dev/null +++ b/docs/tutorials/case_f.rst @@ -0,0 +1,121 @@ +.. _case_f: + +F - Testing Catalog-Based Forecasts +=================================== + +This example shows how set up an experiment with a **time-dependent** model, whose forecast files already exist. + +.. currentmodule:: floatcsep + +.. admonition:: **TL; DR** + + In a terminal, navigate to ``floatcsep/tutorials/case_f`` and type: + + .. code-block:: console + + $ floatcsep run config.yml + + After the calculation is complete, the results will be summarized in ``results/report.md``. + +.. contents:: Contents + :local: + + +Experiment Components +--------------------- + + +The source files can be found in the ``tutorials/case_e`` folder or in `GitHub `_. The experiment structure is as follows: + +:: + + case_f + └── etas + ├── forecasts + ├── etas_2016-11-14_2016-11-15.csv (forecast files) + ... + └── etas_2016-11-20_2016-11-21.csv + ├── catalog.csv + ├── config.yml + ├── models.yml + └── tests.yml + +* The model to be evaluated (``etas``) is a collection of daily forecasts from ``2016-11-14`` until ``2016-11-21``. + +.. important:: + The forecasts must be located in a folder ``forecasts`` inside the model folder. This is meant for consistency with models based on source codes (see subsequent tutorials). + + +Model +----- + +The time-dependency of a model is manifested here by the provision of different forecasts, i.e., statistical descriptions of seismicity, for different time-windows. In this example, the forecasts were created from an external model https://github.com/lmizrahi/etas (`doi:10.1785/0220200231 `_), with which the experiment has no interface. This means that we use **only the forecast files** and no source code. We leave the handling of a model source code for subsequent tutorials. + + + +Configuration +------------- + + +Time +~~~~ + + The configuration is analogous to time-independent models with multiple time-windows (e.g., case C) with the exception that a ``horizon`` could be defined instead of ``intervals``, which is the forecast time-window length. The experiment's class should now be explicited as ``exp_class: td``. + + .. literalinclude:: ../../tutorials/case_f/config.yml + :caption: tutorials/case_f/config.yml + :language: yaml + :lines: 3-7 + +.. note:: + **floatCSEP** is flexible with the definition of time windows/deltas. Alternative string inputs for ``horizon`` can be ``1-day``, ``1 day``, ``1d``, etc. + +Catalog +~~~~~~~ + + The catalog ``catalog.json`` was obtained *previously* by using ``query_geonet`` and it was filtered to the testing period. However, it can be re-queried by changing its definition to: + + .. code-block:: yaml + + catalog: query_geonet + +Models +~~~~~~ + + Some additional arguments should be passed to a **time-dependent** model, such as its class ('td' for time-dependent) and the number of simulations. + + .. literalinclude:: ../../tutorials/case_f/models.yml + :caption: tutorials/case_f/config.yml + :language: yaml + :lines: 1-4 + +.. note:: + For consistency with time-dependent models that will create forecasts from a source code, the ``path`` should point to the folder of the model, which itself should contain a sub-folder named ``{path}/forecasts`` where the files are located. + +.. important:: + Note that for catalog-based forecasts, the model should explicit the number of simulations. This is meant for forecast files that contain synthetic catalogs with zero-event simulations, and therefore do not contain the total number of synthetic catalogs used. + +Tests +~~~~~ + + With time-dependent models, now catalog evaluations found in :obj:`csep.core.catalog_evaluations` can be used. + + + .. literalinclude:: ../../tutorials/case_f/tests.yml + :language: yaml + + .. note:: + It is possible to assign two plotting functions to a test, whose ``plot_args`` and ``plot_kwargs`` can be placed indented beneath + + +Running the experiment +---------------------- + + The experiment can be run by simply navigating to the ``tutorials/case_h`` folder in the terminal and typing. + + .. code-block:: console + + $ floatcsep run config.yml + + This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. + diff --git a/docs/tutorials/case_g.rst b/docs/tutorials/case_g.rst new file mode 100644 index 0000000..1edfaa7 --- /dev/null +++ b/docs/tutorials/case_g.rst @@ -0,0 +1,204 @@ +.. _case_g: + +G - Testing a Time-Dependent Model +================================== + +Here, we set up a time-dependent model from its **source code** for an experiment. + +.. admonition:: **TL; DR** + + In a terminal, navigate to ``floatcsep/tutorials/case_g`` and type: + + .. code-block:: console + + $ floatcsep run config.yml + + After the calculation is complete, the results will be summarized in ``results/report.md``. + + +.. currentmodule:: floatcsep + +.. contents:: Contents + :local: + + + +Experiment Components +--------------------- + +The example folder contains also, along with the already known components (configurations, catalog), a sub-folder for the **source code** of the model ``pymock``. The components of the experiment (and model) are: + +:: + + case_g + └── pymock (Model's source code) + ├── input (input interface to floatcsep) + ├── args.txt (model arguments) + └── catalog.csv (dynamically allocated catalog) + ├── pymock + ├── libs.py (helper functions) + └── main.py (main routines) + └── forecasts (output interface to floatcsep) + ... (forecasts should be stored here when the model is run) + ├── run.py (One of the possibilities to run the model) + ├── pyproject.toml (Build instructions) + ├── setup.cfg (Build instructions) + ├── setup.py (Build instructions) + ├── requirements.txt(Build instructions) + ├── Dockerfile (Build instructions) + └── README.md (Information) + + ├── catalog.csv + ├── config.yml + ├── models.yml + ├── custom_plot_script.py + └── tests.yml + +* The model to be evaluated (``pymock``) is a source code that generates forecasts for multiple time windows. + +* The testing catalog ``catalog.csv`` works also as the input catalog, by being filtered until the testing ``start_date`` and allocated in `pymock/input` dynamically (before each time the model is run) + +.. _Model: + +Model +----- + +The experiment's complexity increases from time-independent to dependent mostly because we now need a **Model** (source code) to generate forecasts that changes for every time-window. The model main components are: + + +* **Input**: The input consists in input **data** and **arguments**. + + 1. The **input data** is, at the very least, a catalog filtered until the forecast beginning. The catalog will be automatically allocated by ``floatcsep`` prior to each model's run (e.g., a single forecast run) in the `{model}/input` folder. It is stored in the ``csep.ascii`` format for simplicity's sake (see :doc:`pycsep:concepts/catalogs`). + + .. literalinclude:: ../../tutorials/case_g/catalog.csv + :caption: tutorials/case_g/catalog.csv + :lines: 1-2 + + 2. The **input arguments** controls how the model's source code works. The minimum arguments to run a model are the forecast ``start_date`` and ``end_date``, which will be modified dynamically during an experiment with multiple time-windows. The experiment system will access `{model}/input/args.txt` and change the values of ``start_date = {datetime}`` and ``end_date = {datetime}`` before the model is run. Additional arguments can be set by convenience, such as (not limited to) ``catalog`` (the input catalog name), ``n_sims`` (number of synthetic catalogs) and random ``seed`` for reproducibility. + +* **Output**: The model's output are the synthetic catalogs, which should be allocated in `{model}/forecasts/{filename}.csv` by the source code after each rone. The format is identically to ``csep_ascii``, but unlike in an input catalog, the ``catalog_id`` column should be modified for each synthetic catalog starting from 0. The file name follows the convention `{model_name}_{start}_{end}.csv`, where ``start`` and ``end`` folows the `%Y-%m-%dT%H:%M:%S.%f` - ISO861 FORMAT + +* **Model build**: Inside the model source code, there are multiple options to build it. A standard python ``setup.cfg`` is given, which can be built inside a python ``venv`` or ``conda`` managers. This is created and built automatically by ``floatCSEP``, as long as the the model build instructions are correctly set up. + +* **Model run**: The model should be run with a simple command, e.g. **entrypoint**, to which only ``arguments`` could be passed if desired. The ``pymock`` model contains multiple example of entrypoints, but the modeler should use only one for clarity. + + 1. A `python` call with arguments + + .. code-block:: console + + $ python run.py input/args.txt + + 2. Using a binary entrypoint with arguments (for instance, defined in the python build instructions: ``pymock/setup.cfg:entry_point``) + + .. code-block:: console + + $ pymock input/args.txt + + 3. A single binary entrypoint without arguments . + + .. code-block:: console + + $ pymock + + This means that the source code should internally read the input data and arguments, ``input/catalog.csv`` and ``input/args.txt`` files respectively. + +.. important:: + + The model should be conceptualized as a **black-box**, whose only interface/interaction with the ``floatcsep`` system is to receive an input (i.e., input catalog and arguments) and generates an output (the forecasts). + + +Configuration +------------- + + +Time +~~~~ + + The configuration is identical to time-independent models, with the exception that now a ``horizon`` can be defined instead of ``intervals``, which is the forecast time-window length. The experiment's class should now be explicited as ``exp_class: td`` + + .. literalinclude:: ../../tutorials/case_g/config.yml + :caption: tutorials/case_g/config.yml + :language: yaml + :lines: 3-7 + +Catalog +~~~~~~~ + + The catalog was obtained `previous to the experiment` using ``query_bsi``, but it was filtered from 2006 onwards, so it has enough data for the model calibration. + +Models +~~~~~~ + + Additional arguments should be passed to time-independent models. + + .. literalinclude:: ../../tutorials/case_g/models.yml + :caption: tutorials/case_g/models.yml + :language: yaml + :lines: 1-7 + + 1. Now ``path`` points to the folder where the source is installed. Therefore, the input and the forecasts should be allocated ``{path}/input`` and ``{path}/forecasts``, respectively. + 2. The ``func`` option is the shell command with which the model is run. As seen in the `Model`_ section, this could be either ``pymock``, ``pymock input/args.txt`` or ``python run.py input/args``. We use the simplest option ``pymock``, but you are welcome to try different entrypoints. + + .. note:: + The ``func`` command will be run from the model's directory and a model containerization (e.g., ``Dockerfile``, ``conda``). + + 3. The ``func_kwargs`` are extra arguments that will annotated to the ``input/args.txt`` file every time the model is run, or will be passed as extra arguments to the ``func`` call (Note that the two options are identical). This is useful to define sub-classes of models (or flavours) that uses the same source code, but a different instantiation. + 4. The ``build`` option defines the style of container within which the model will be placed. Currently in **floatCSEP**, only the python module ``venv``, the package manager ``conda`` and the containerization manager ``Docker`` are currently supported. + + .. important:: + For these tutorials, we use ``venv`` sub-environments, but we recommend ``Docker`` to set up real experiments. + + +Tests +~~~~~ + + With time-dependent models, now catalog evaluations found in :obj:`csep.core.catalog_evaluations` can be used. + + + .. literalinclude:: ../../tutorials/case_g/tests.yml + :caption: tutorials/case_g/tests.yml + :language: yaml + + .. note:: + It is possible to assign two plotting functions to a test, whose ``plot_args`` and ``plot_kwargs`` can be placed indented beneath + + +Custom Post-Process +~~~~~~~~~~~~~~~~~~~ + + Additional to the default :func:`~floatcsep.postprocess.plot_handler.plot_results`, :func:`~floatcsep.postprocess.plot_handler.plot_catalogs`, :func:`~floatcsep.postprocess.plot_handler.plot_forecasts` functions, a custom plotting function(s) can be set within the ``postprocess`` configuration + + .. literalinclude:: ../../tutorials/case_g/config.yml + :caption: tutorials/case_g/config.yml + :language: yaml + :lines: 22-23 + + This option provides `hook` for a python script and a function within as: + + .. code-block:: console + + {python_sript}:{function_name} + + The requirements are that the script to be located within the same directory as the configuration file, whereas the function must receive a :class:`floatcsep.experiment.Experiment` as argument + + .. literalinclude:: ../../tutorials/case_g/custom_plot_script.py + :caption: tutorials/case_g/custom_plot_script.py + :language: python + :lines: 6-13 + + + + In this way, the plot function can use all the :class:`~floatcsep.experiment.Experiment` attributes/methods to access catalogs, forecasts and test results. The script ``tutorials/case_g/custom_plot_script.py`` can also be viewed directly on `GitHub `_, where it is exemplified how to access the experiment data in runtime. + + +Running the experiment +---------------------- + + The experiment can be run by simply navigating to the ``tutorials/case_g`` folder in the terminal and typing. + + .. code-block:: console + + $ floatcsep run config.yml + + This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. + diff --git a/docs/tutorials/case_h.rst b/docs/tutorials/case_h.rst new file mode 100644 index 0000000..667c083 --- /dev/null +++ b/docs/tutorials/case_h.rst @@ -0,0 +1,215 @@ +.. _case_h: + +H - A Time-Dependent Experiment +=============================== + +Here, we run an experiment that access, containerize and execute multiple **time-dependent models**, and then proceeds to evaluate the forecasts once they are created. + +.. admonition:: **TL; DR** + + In a terminal, navigate to ``floatcsep/tutorials/case_h`` and type: + + .. code-block:: console + + $ floatcsep run config.yml + + After the calculation is complete, the results will be summarized in ``results/report.md``. + +.. currentmodule:: floatcsep + +.. contents:: Contents + :local: + + + +Experiment Components +--------------------- + +The experiment input files are: + +:: + + case_h + ├── catalog.csv + ├── config.yml + ├── tests.yml + └── models.yml + +* The ``models.yml`` contains the instructions to clone and build the source codes from software repositories (e.g., gitlab, Github), and how to interface them with **floatCSEP**. Once downloaded and built, the experiment structure should look like: + +:: + + case_h + ├── models + ├── etas + └── ... (ETAS source code) + ├── pymock_poisson + └── ... (Poisson-PyMock source code) + └── pymock_nb + └── ... (Negative-Binomial-PyMock source code) + ├── catalog.csv + ├── config.yml + ├── tests.yml + ├── custom_report.py + └── models.yml + +Configuration +------------- + +Models +~~~~~~ + +As in :ref:`Tutorial G`, each **Model** requires to build and execute a source code to generate forecasts. The instructions for each model are located within ``models.yml``, which we further explain here: + +.. note:: + The ``models.yml`` will define how to interface **floatCSEP** to each Model, implying that a Model should be developed, or adapted to ensure the interface requirements specified below. + +1. The repository URL of each model and their specific versions (e.g., commit hash, tag, release) are specified as: + + .. literalinclude:: ../../tutorials/case_h/models.yml + :caption: tutorials/case_h/models.yml + :language: yaml + :lines: 1-3, 11-13, 21-23 + +2. A ``path`` needs to be indicated for each model, to both download the repository contents therein and from where the source code will be executed. + + .. literalinclude:: ../../tutorials/case_h/models.yml + :caption: tutorials/case_h/models.yml + :language: yaml + :lines: 1-4 + :emphasize-lines: 4 + :lineno-match: + + .. important:: + The implies that the inputs (catalog and argument file) should be read by the model from a ``{path}/input`` folder, and the forecast outputs stored into a folder ``{path}/forecasts``. + +2. There is some flexibility to interface **floatCSEP** with a model. For instance, a different `filepath` can be set for the argument file: + + .. literalinclude:: ../../tutorials/case_h/models.yml + :caption: tutorials/case_h/models.yml + :language: yaml + :lines: 5 + :lineno-match: + + .. note:: + **floatCSEP** can read `.txt`, `.json` and `.yml` format-styled argument files. In all cases, the minimum required arguments, should be formatted as: + + .. code-block:: console + + #.txt + start_date = {DATESTRING} + end_date = {DATESTRING} + + .. code-block:: yaml + + #.yml + start_date: {DATESTRING} + end_date: {DATESTRING} + + .. code-block:: json + + //.json + "start_date": "{DATESTRING}", + "end_date": "{DATESTRING}" + + **floatcsep** will dynamically modify the ``start_date`` and ``start_date`` for each time window run, and any extra arguments will just be added for all time-windows. + +4. The ``func`` entry indicates how the models are invoked from a shell terminal. + + .. literalinclude:: ../../tutorials/case_h/models.yml + :caption: tutorials/case_h/models.yml + :language: yaml + :lines: 1,6,11,15,21,25 + + .. important:: + Please refer to :ref:`Tutorial G` for example of how to set up ``func`` for the model and interface it to **floatCSEP**. + +5. A prefix for the resulting forecast filepaths can be specified beforehand for each model. Without specifying this, the default is ``{model_name}`` (e.g., `etas`, `Poisson Mock`, `Negbinom Mock`). + + .. literalinclude:: ../../tutorials/case_h/models.yml + :caption: tutorials/case_h/models.yml + :language: yaml + :lines: 21, 26 + + The experiment will read the forecasts as: + + .. code-block:: + + {model_path}/{forecasts}/{prefix}_{start}_{end}.csv + + where ``start`` and ``end`` follow either the ``%Y-%m-%dT%H:%M:%S.%f`` - ISO861 FORMAT, or the short date version ``%Y-%m-%d`` if the windows are set by midnight. + +6. Additional function arguments can be passed to the model with the entry ``func_kwargs``. We perhaps noted that both Poisson Mock and Negbinom Mock use the same source code. With ``func_kwargs`` a different subclass can be defined for the same source code (in this case, a Negative-Binomial number distribution instead of Poisson). + + .. literalinclude:: ../../tutorials/case_h/models.yml + :caption: tutorials/case_h/models.yml + :language: yaml + :lines: 11,17-20,21,27-31 + + +Time +~~~~ + + The configuration is identical to time-independent models, with the exception that now a ``horizon`` can be defined instead of ``intervals``, which is the forecast time-window length. The experiment's class should now be explicited as ``exp_class: td`` + + .. literalinclude:: ../../tutorials/case_h/config.yml + :caption: tutorials/case_h/config.yml + :language: yaml + :lines: 3-7 + +Catalog +~~~~~~~ + + The catalog was obtained `previous to the experiment` using ``query_bsi``, but it was filtered from 2006 onwards, so it has enough data for the model calibration. + + +Tests +~~~~~ + + With time-dependent models, now catalog evaluations found in :obj:`csep.core.catalog_evaluations` can be used. + + + .. literalinclude:: ../../tutorials/case_h/tests.yml + :caption: tutorials/case_h/tests.yml + :language: yaml + + .. note:: + It is possible to assign two plotting functions to a test, whose ``plot_args`` and ``plot_kwargs`` can be placed indented beneath + + +Custom Post-Process +~~~~~~~~~~~~~~~~~~~ + + A custom reporting function can be set within the ``postprocess`` configuration to replace the :func:`~floatcsep.postprocess.reporting.generate_report`: + + .. literalinclude:: ../../tutorials/case_h/config.yml + :caption: tutorials/case_h/config.yml + :language: yaml + :lines: 22-23 + + This option provides `hook` for a python script and a function within as: + + .. code-block:: console + + {python_sript}:{function_name} + + The requirements are that the script to be located within the same directory as the configuration file, whereas the function must receive a :class:`floatcsep.experiment.Experiment` as argument + + .. literalinclude:: ../../tutorials/case_h/custom_report.py + :language: yaml + :lines: 5-11 + + In this way, the report function use all the :class:`~floatcsep.experiment.Experiment` attributes/methods to access catalogs, forecasts and test results. The script ``tutorials/case_h/custom_report.py`` can also be viewed directly on `GitHub `_, where it is exemplified how to access the experiment artifacts. + + +Running the experiment +---------------------- + + The experiment can be run by simply navigating to the ``tutorials/case_h`` folder in the terminal and typing. + + .. code-block:: console + + $ floatcsep run config.yml + + This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``. + diff --git a/examples/case_f/tests.yml b/examples/case_f/tests.yml deleted file mode 100644 index 7966964..0000000 --- a/examples/case_f/tests.yml +++ /dev/null @@ -1,19 +0,0 @@ -- Catalog_N-test: - func: catalog_evaluations.number_test - plot_func: - - plot_number_test: - plot_args: - title: 1 - name: 1 - - plot_consistency_test: - plot_kwargs: - one_sided_lower: True -- Catalog_S-test: - func: catalog_evaluations.spatial_test - plot_func: - - plot_consistency_test: - plot_kwargs: - one_sided_lower: True - - - diff --git a/examples/case_g/tests.yml b/examples/case_g/tests.yml deleted file mode 100644 index 67315da..0000000 --- a/examples/case_g/tests.yml +++ /dev/null @@ -1,14 +0,0 @@ -- Catalog_N-test: - func: catalog_evaluations.number_test - plot_func: - - plot_number_test: - plot_args: - title: 1 - name: 1 - - plot_consistency_test: - plot_kwargs: - one_sided_lower: True - - - - diff --git a/floatcsep/evaluation.py b/floatcsep/evaluation.py index e63954e..eb74b76 100644 --- a/floatcsep/evaluation.py +++ b/floatcsep/evaluation.py @@ -250,7 +250,7 @@ def compute( self.results_repo.write_result(evaluation_result, self, model, timewindow) def read_results( - self, window: Union[str, Sequence[datetime.datetime]], models: List[Model] + self, window: Union[str, Sequence[datetime.datetime]], models: Union[Model, List[Model]] ) -> List: """ Reads an Evaluation result for a given time window and returns a list of the results for diff --git a/floatcsep/experiment.py b/floatcsep/experiment.py index 2cdadaf..e5b50c5 100644 --- a/floatcsep/experiment.py +++ b/floatcsep/experiment.py @@ -188,8 +188,11 @@ def __init__( def __getattr__(self, item: str) -> object: """ - Override built-in method to return attributes found within. - :attr:`region_config` or :attr:`time_config` + Override built-in method to return the experiment attributes by also using the command + ``experiment.{attr}``. Adds also to the experiment scope the keys of + :attr:`region_config` or :attr:`time_config`. These are: ``start_date``, ``end_date``, + ``timewindows``, ``horizon``, ``offset``, ``region``, ``magnitudes``, ``mag_min``, + `mag_max``, ``mag_bin``, ``depth_min`` depth_max . """ try: @@ -206,7 +209,7 @@ def __getattr__(self, item: str) -> object: ) from None def __dir__(self): - """Adds time and region configs keys to instance scope.""" + """Adds the time and region configs keys the to instance scope.""" _dir = ( list(super().__dir__()) + list(self.time_config.keys()) + list(self.region_config) @@ -296,6 +299,12 @@ def get_model(self, name: str) -> Model: if model.name == name: return model + def get_test(self, name: str) -> Model: + """Returns an Evaluation by its name string.""" + for test in self.tests: + if test.name == name: + return test + def stage_models(self) -> None: """ Stages all the experiment's models. See :meth:`floatcsep.model.Model.stage` diff --git a/floatcsep/postprocess/plot_handler.py b/floatcsep/postprocess/plot_handler.py index c8d0671..f6531c4 100644 --- a/floatcsep/postprocess/plot_handler.py +++ b/floatcsep/postprocess/plot_handler.py @@ -96,6 +96,8 @@ def plot_forecasts(experiment: "Experiment") -> None: # If catalog option is passed, catalog is plotted on top of the forecast if plot_forecast_config.get("catalog"): cat_args = plot_forecast_config.get("catalog", {}) + if cat_args is True: + cat_args = {} experiment.catalog_repo.get_test_cat(window).plot( ax=ax, extent=ax.get_extent(), diff --git a/floatcsep/utils/helpers.py b/floatcsep/utils/helpers.py index 75b4821..030c2dc 100644 --- a/floatcsep/utils/helpers.py +++ b/floatcsep/utils/helpers.py @@ -159,12 +159,16 @@ def read_time_cfg(time_config, **kwargs): if "offset" in time_config.keys(): time_config["offset"] = parse_timedelta_string(time_config["offset"]) - if experiment_class == "ti": - time_config["timewindows"] = timewindows_ti(**time_config) - return time_config - elif experiment_class == "td": - time_config["timewindows"] = timewindows_td(**time_config) - return time_config + if not time_config.get("timewindows"): + if experiment_class == "ti": + time_config["timewindows"] = timewindows_ti(**time_config) + elif experiment_class == "td": + time_config["timewindows"] = timewindows_td(**time_config) + else: + time_config["start_date"] = time_config["timewindows"][0][0] + time_config["end_date"] = time_config["timewindows"][-1][-1] + + return time_config def read_region_cfg(region_config, **kwargs): @@ -340,7 +344,7 @@ def timewindows_td( Creates the testing intervals for a time-dependent experiment. Note: - The following arg combinations are possible: + The following are combinations are possible: - (start_date, end_date, timeintervals) - (start_date, end_date, timehorizon) - (start_date, timeintervals, timehorizon) @@ -391,10 +395,11 @@ def timewindows_td( except ValueError as e_: raise ValueError( "The following experiment parameters combinations are possible:\n" - " (start_date, end_date)\n" - " (start_date, end_date, intervals)\n" - " (start_date, end_date, timewindow)\n" - " (start_date, intervals, timewindow)\n" + " (start_date, end_date, timeintervals)\n" + " (start_date, end_date, timehorizon)\n" + " (start_date, timeintervals, timehorizon)\n" + " (start_date, end_date, timehorizon, timeoffset)\n" + " (start_date, timeinvervals, timehorizon, timeoffset)\n" ) # if growth == 'incremental': diff --git a/pyproject.toml b/pyproject.toml index ef57c31..1c9d0f9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,6 +4,7 @@ build-backend = "setuptools.build_meta" [tool.pytest.ini_options] addopts = "--cov=floatcsep" +norecursedirs = ["tutorials", "tests/artifacts"] testpaths = [ "tests", ] diff --git a/requirements_dev.txt b/requirements_dev.txt index 9c8799a..bb7a198 100644 --- a/requirements_dev.txt +++ b/requirements_dev.txt @@ -26,8 +26,11 @@ seaborn shapely sphinx sphinx-autoapi +sphinx_design sphinx-gallery sphinx-rtd-theme +sphinx-toolbox +sphinx_copybutton tables tox vcrpy==4.3.1 diff --git a/setup.cfg b/setup.cfg index 3528ec5..5fa97fe 100644 --- a/setup.cfg +++ b/setup.cfg @@ -74,8 +74,11 @@ dev = shapely sphinx sphinx-autoapi + sphinx_design + sphinx-toolbox sphinx-gallery sphinx-rtd-theme + sphinx_copybutton tables tox vcrpy==4.3.1 diff --git a/tests/artifacts/models/template/Dockerfile b/tests/artifacts/models/template/Dockerfile index dd5f778..b20e588 100644 --- a/tests/artifacts/models/template/Dockerfile +++ b/tests/artifacts/models/template/Dockerfile @@ -40,7 +40,7 @@ WORKDIR $MODEL_PATH COPY --chown=$USER_UID:$USER_GID {relpaths_host} $MODEL_PATH ## Ideally, the folder/files to be copied should be explicited -# COPY --chown=$USER_UID:$USER_GID source examples tests docs parameters setup $MODEL_PATH +# COPY --chown=$USER_UID:$USER_GID source tutorials tests docs parameters setup $MODEL_PATH # ______________________________ # 4- Build the model source code @@ -49,7 +49,7 @@ COPY --chown=$USER_UID:$USER_GID {relpaths_host} $MODEL_PATH ## Provide a set of instructions to build the model inside the docker image, and run the unit tests RUN setup && run_tests -## See examples in +## See tutorials in #### Python (https://git.gfz-potsdam.de/csep-group/rise_italy_experiment/models/mockup_py) #### R (https://git.gfz-potsdam.de/csep-group/rise_italy_experiment/models/mockup_R) diff --git a/tests/integration/test_model_interface.py b/tests/integration/test_model_interface.py index 318a708..ae2c0d8 100644 --- a/tests/integration/test_model_interface.py +++ b/tests/integration/test_model_interface.py @@ -139,7 +139,7 @@ def setUpClass(cls) -> None: cls._dir = os.path.join(path, "../", "artifacts", "models") cls._alm_fn = os.path.join( path, - "../../examples", + "../../tutorials", "case_e", "models", "gulia-wiemer.ALM.italy.10yr.2010-01-01.xml", @@ -201,7 +201,7 @@ def test_forecast_ti_from_hdf5(self): def tearDownClass(cls) -> None: alm_db = os.path.join( cls._path, - "../../examples", + "../../tutorials", "case_e", "models", "gulia-wiemer.ALM.italy.10yr.2010-01-01.hdf5", diff --git a/tests/qa/test_data.py b/tests/qa/test_data.py index 8441d87..c2db68f 100644 --- a/tests/qa/test_data.py +++ b/tests/qa/test_data.py @@ -10,14 +10,14 @@ class DataTest(unittest.TestCase): @staticmethod def get_runpath(case): return os.path.abspath( - os.path.join(__file__, "../../..", "examples", f"case_{case}", f"config.yml") + os.path.join(__file__, "../../..", "tutorials", f"case_{case}", f"config.yml") ) @staticmethod def get_rerunpath(case): return os.path.abspath( os.path.join( - __file__, "../../..", "examples", f"case_{case}", "results", f"repr_config.yml" + __file__, "../../..", "tutorials", f"case_{case}", "results", f"repr_config.yml" ) ) diff --git a/tests/unit/test_readers.py b/tests/unit/test_readers.py index a9e4bf8..f4935a1 100644 --- a/tests/unit/test_readers.py +++ b/tests/unit/test_readers.py @@ -71,7 +71,7 @@ def test_parse_csv_qtree(self): def test_parse_xml(self): fname = os.path.join( self._path, - "../../examples", + "../../tutorials", "case_e", "models", "gulia-wiemer.ALM.italy.10yr.2010-01-01.xml", diff --git a/examples/case_a/best_model.dat b/tutorials/case_a/best_model.dat similarity index 79% rename from examples/case_a/best_model.dat rename to tutorials/case_a/best_model.dat index a5b5003..55957d9 100644 --- a/examples/case_a/best_model.dat +++ b/tutorials/case_a/best_model.dat @@ -1,4 +1,3 @@ # lon_min lon_max lat_min lat_max depth_min depth_max mag_min mag_max rate mask 0 1 0 1 0 70 6 7 5e-1 1 -1 2 0 1 0 70 6 7 5e-1 0.1 - +1 2 0 1 0 70 6 7 5e-1 0.1 \ No newline at end of file diff --git a/examples/case_a/case_a.py b/tutorials/case_a/case_a.py similarity index 100% rename from examples/case_a/case_a.py rename to tutorials/case_a/case_a.py diff --git a/examples/case_a/catalog.csep b/tutorials/case_a/catalog.csep similarity index 100% rename from examples/case_a/catalog.csep rename to tutorials/case_a/catalog.csep diff --git a/examples/case_a/config.yml b/tutorials/case_a/config.yml similarity index 85% rename from examples/case_a/config.yml rename to tutorials/case_a/config.yml index 1835b0d..ceab9a3 100644 --- a/examples/case_a/config.yml +++ b/tutorials/case_a/config.yml @@ -23,3 +23,7 @@ tests: func: poisson_evaluations.number_test plot_func: plot_poisson_consistency_test +postprocess: + plot_forecasts: + cmap: magma + catalog: True diff --git a/examples/case_a/region.txt b/tutorials/case_a/region.txt similarity index 100% rename from examples/case_a/region.txt rename to tutorials/case_a/region.txt diff --git a/examples/case_b/case_b.py b/tutorials/case_b/case_b.py similarity index 100% rename from examples/case_b/case_b.py rename to tutorials/case_b/case_b.py diff --git a/examples/case_b/catalog.json b/tutorials/case_b/catalog.json similarity index 100% rename from examples/case_b/catalog.json rename to tutorials/case_b/catalog.json diff --git a/examples/case_b/config.yml b/tutorials/case_b/config.yml similarity index 100% rename from examples/case_b/config.yml rename to tutorials/case_b/config.yml diff --git a/examples/case_b/models.yml b/tutorials/case_b/models.yml similarity index 100% rename from examples/case_b/models.yml rename to tutorials/case_b/models.yml diff --git a/examples/case_b/models/model_a.csv b/tutorials/case_b/models/model_a.csv similarity index 100% rename from examples/case_b/models/model_a.csv rename to tutorials/case_b/models/model_a.csv diff --git a/examples/case_b/models/model_b.csv b/tutorials/case_b/models/model_b.csv similarity index 100% rename from examples/case_b/models/model_b.csv rename to tutorials/case_b/models/model_b.csv diff --git a/examples/case_b/models/model_c.csv b/tutorials/case_b/models/model_c.csv similarity index 100% rename from examples/case_b/models/model_c.csv rename to tutorials/case_b/models/model_c.csv diff --git a/examples/case_b/models/model_d.csv b/tutorials/case_b/models/model_d.csv similarity index 100% rename from examples/case_b/models/model_d.csv rename to tutorials/case_b/models/model_d.csv diff --git a/examples/case_b/region.txt b/tutorials/case_b/region.txt similarity index 100% rename from examples/case_b/region.txt rename to tutorials/case_b/region.txt diff --git a/examples/case_b/tests.yml b/tutorials/case_b/tests.yml similarity index 100% rename from examples/case_b/tests.yml rename to tutorials/case_b/tests.yml diff --git a/examples/case_c/case_c.py b/tutorials/case_c/case_c.py similarity index 100% rename from examples/case_c/case_c.py rename to tutorials/case_c/case_c.py diff --git a/examples/case_c/catalog.json b/tutorials/case_c/catalog.json similarity index 100% rename from examples/case_c/catalog.json rename to tutorials/case_c/catalog.json diff --git a/examples/case_c/config.yml b/tutorials/case_c/config.yml similarity index 100% rename from examples/case_c/config.yml rename to tutorials/case_c/config.yml diff --git a/examples/case_c/models.yml b/tutorials/case_c/models.yml similarity index 100% rename from examples/case_c/models.yml rename to tutorials/case_c/models.yml diff --git a/examples/case_c/models/model_a.csv b/tutorials/case_c/models/model_a.csv similarity index 100% rename from examples/case_c/models/model_a.csv rename to tutorials/case_c/models/model_a.csv diff --git a/examples/case_c/models/model_b.csv b/tutorials/case_c/models/model_b.csv similarity index 100% rename from examples/case_c/models/model_b.csv rename to tutorials/case_c/models/model_b.csv diff --git a/examples/case_c/models/model_c.csv b/tutorials/case_c/models/model_c.csv similarity index 100% rename from examples/case_c/models/model_c.csv rename to tutorials/case_c/models/model_c.csv diff --git a/examples/case_c/models/model_d.csv b/tutorials/case_c/models/model_d.csv similarity index 100% rename from examples/case_c/models/model_d.csv rename to tutorials/case_c/models/model_d.csv diff --git a/examples/case_c/region.txt b/tutorials/case_c/region.txt similarity index 100% rename from examples/case_c/region.txt rename to tutorials/case_c/region.txt diff --git a/examples/case_c/tests.yml b/tutorials/case_c/tests.yml similarity index 100% rename from examples/case_c/tests.yml rename to tutorials/case_c/tests.yml diff --git a/examples/case_d/config.yml b/tutorials/case_d/config.yml similarity index 87% rename from examples/case_d/config.yml rename to tutorials/case_d/config.yml index dd2726b..a4d4917 100644 --- a/examples/case_d/config.yml +++ b/tutorials/case_d/config.yml @@ -14,6 +14,3 @@ region_config: catalog: query_gcmt models: models.yml test_config: tests.yml - -postprocess: - plot_forecasts: True \ No newline at end of file diff --git a/examples/case_d/models.yml b/tutorials/case_d/models.yml similarity index 100% rename from examples/case_d/models.yml rename to tutorials/case_d/models.yml diff --git a/examples/case_d/tests.yml b/tutorials/case_d/tests.yml similarity index 100% rename from examples/case_d/tests.yml rename to tutorials/case_d/tests.yml diff --git a/examples/case_e/case_e.py b/tutorials/case_e/case_e.py similarity index 100% rename from examples/case_e/case_e.py rename to tutorials/case_e/case_e.py diff --git a/examples/case_e/catalog.json b/tutorials/case_e/catalog.json similarity index 100% rename from examples/case_e/catalog.json rename to tutorials/case_e/catalog.json diff --git a/examples/case_e/config.yml b/tutorials/case_e/config.yml similarity index 97% rename from examples/case_e/config.yml rename to tutorials/case_e/config.yml index 1f235e5..5fc86c9 100644 --- a/examples/case_e/config.yml +++ b/tutorials/case_e/config.yml @@ -20,6 +20,7 @@ test_config: tests.yml postprocess: plot_forecasts: + cmap: magma region_border: True basemap: stock_img clabel_fontsize: 14 diff --git a/examples/case_e/models.yml b/tutorials/case_e/models.yml similarity index 100% rename from examples/case_e/models.yml rename to tutorials/case_e/models.yml diff --git a/examples/case_e/models/gulia-wiemer.ALM.italy.10yr.2010-01-01.xml b/tutorials/case_e/models/gulia-wiemer.ALM.italy.10yr.2010-01-01.xml similarity index 100% rename from examples/case_e/models/gulia-wiemer.ALM.italy.10yr.2010-01-01.xml rename to tutorials/case_e/models/gulia-wiemer.ALM.italy.10yr.2010-01-01.xml diff --git a/examples/case_e/models/meletti.MPS04.italy.10yr.2010-01-01.xml b/tutorials/case_e/models/meletti.MPS04.italy.10yr.2010-01-01.xml similarity index 100% rename from examples/case_e/models/meletti.MPS04.italy.10yr.2010-01-01.xml rename to tutorials/case_e/models/meletti.MPS04.italy.10yr.2010-01-01.xml diff --git a/examples/case_e/models/zechar.TripleS-CPTI.italy.10yr.2010-01-01.xml b/tutorials/case_e/models/zechar.TripleS-CPTI.italy.10yr.2010-01-01.xml similarity index 100% rename from examples/case_e/models/zechar.TripleS-CPTI.italy.10yr.2010-01-01.xml rename to tutorials/case_e/models/zechar.TripleS-CPTI.italy.10yr.2010-01-01.xml diff --git a/examples/case_e/tests.yml b/tutorials/case_e/tests.yml similarity index 100% rename from examples/case_e/tests.yml rename to tutorials/case_e/tests.yml diff --git a/examples/case_f/catalog.json b/tutorials/case_f/catalog.json similarity index 100% rename from examples/case_f/catalog.json rename to tutorials/case_f/catalog.json diff --git a/examples/case_f/config.yml b/tutorials/case_f/config.yml similarity index 95% rename from examples/case_f/config.yml rename to tutorials/case_f/config.yml index 68adf11..d62af88 100644 --- a/examples/case_f/config.yml +++ b/tutorials/case_f/config.yml @@ -1,4 +1,4 @@ -name: case_g +name: case_f time_config: start_date: 2016-11-14T00:00:00 diff --git a/examples/case_f/etas/forecasts/etas_2016-11-14_2016-11-15.csv b/tutorials/case_f/etas/forecasts/etas_2016-11-14_2016-11-15.csv similarity index 100% rename from examples/case_f/etas/forecasts/etas_2016-11-14_2016-11-15.csv rename to tutorials/case_f/etas/forecasts/etas_2016-11-14_2016-11-15.csv diff --git a/examples/case_f/etas/forecasts/etas_2016-11-15_2016-11-16.csv b/tutorials/case_f/etas/forecasts/etas_2016-11-15_2016-11-16.csv similarity index 100% rename from examples/case_f/etas/forecasts/etas_2016-11-15_2016-11-16.csv rename to tutorials/case_f/etas/forecasts/etas_2016-11-15_2016-11-16.csv diff --git a/examples/case_f/etas/forecasts/etas_2016-11-16_2016-11-17.csv b/tutorials/case_f/etas/forecasts/etas_2016-11-16_2016-11-17.csv similarity index 100% rename from examples/case_f/etas/forecasts/etas_2016-11-16_2016-11-17.csv rename to tutorials/case_f/etas/forecasts/etas_2016-11-16_2016-11-17.csv diff --git a/examples/case_f/etas/forecasts/etas_2016-11-17_2016-11-18.csv b/tutorials/case_f/etas/forecasts/etas_2016-11-17_2016-11-18.csv similarity index 100% rename from examples/case_f/etas/forecasts/etas_2016-11-17_2016-11-18.csv rename to tutorials/case_f/etas/forecasts/etas_2016-11-17_2016-11-18.csv diff --git a/examples/case_f/etas/forecasts/etas_2016-11-18_2016-11-19.csv b/tutorials/case_f/etas/forecasts/etas_2016-11-18_2016-11-19.csv similarity index 100% rename from examples/case_f/etas/forecasts/etas_2016-11-18_2016-11-19.csv rename to tutorials/case_f/etas/forecasts/etas_2016-11-18_2016-11-19.csv diff --git a/examples/case_f/etas/forecasts/etas_2016-11-19_2016-11-20.csv b/tutorials/case_f/etas/forecasts/etas_2016-11-19_2016-11-20.csv similarity index 100% rename from examples/case_f/etas/forecasts/etas_2016-11-19_2016-11-20.csv rename to tutorials/case_f/etas/forecasts/etas_2016-11-19_2016-11-20.csv diff --git a/examples/case_f/etas/forecasts/etas_2016-11-20_2016-11-21.csv b/tutorials/case_f/etas/forecasts/etas_2016-11-20_2016-11-21.csv similarity index 100% rename from examples/case_f/etas/forecasts/etas_2016-11-20_2016-11-21.csv rename to tutorials/case_f/etas/forecasts/etas_2016-11-20_2016-11-21.csv diff --git a/examples/case_f/models.yml b/tutorials/case_f/models.yml similarity index 100% rename from examples/case_f/models.yml rename to tutorials/case_f/models.yml diff --git a/tutorials/case_f/tests.yml b/tutorials/case_f/tests.yml new file mode 100644 index 0000000..e2bf37a --- /dev/null +++ b/tutorials/case_f/tests.yml @@ -0,0 +1,18 @@ +- Catalog_N-test: + func: catalog_evaluations.number_test + plot_func: + - plot_number_test: + plot_args: + title: Test distribution + - plot_consistency_test: + plot_kwargs: + one_sided_lower: True +- Catalog_S-test: + func: catalog_evaluations.spatial_test + plot_func: + - plot_consistency_test: + plot_kwargs: + one_sided_lower: True + + + diff --git a/examples/case_g/catalog.csv b/tutorials/case_g/catalog.csv similarity index 100% rename from examples/case_g/catalog.csv rename to tutorials/case_g/catalog.csv diff --git a/examples/case_g/config.yml b/tutorials/case_g/config.yml similarity index 100% rename from examples/case_g/config.yml rename to tutorials/case_g/config.yml diff --git a/examples/case_g/custom_plot_script.py b/tutorials/case_g/custom_plot_script.py similarity index 91% rename from examples/case_g/custom_plot_script.py rename to tutorials/case_g/custom_plot_script.py index 6f9d62f..c19c656 100644 --- a/examples/case_g/custom_plot_script.py +++ b/tutorials/case_g/custom_plot_script.py @@ -5,7 +5,11 @@ def main(experiment): """ - Example custom plot function + Example custom plot function (Observed vs. forecast rates in time) + + Args: + experiment: a floatcsep.experiment.Experiment class + """ # Get all the timewindows diff --git a/examples/case_g/models.yml b/tutorials/case_g/models.yml similarity index 100% rename from examples/case_g/models.yml rename to tutorials/case_g/models.yml diff --git a/examples/case_g/pymock/Dockerfile b/tutorials/case_g/pymock/Dockerfile similarity index 100% rename from examples/case_g/pymock/Dockerfile rename to tutorials/case_g/pymock/Dockerfile diff --git a/examples/case_g/pymock/README.md b/tutorials/case_g/pymock/README.md similarity index 100% rename from examples/case_g/pymock/README.md rename to tutorials/case_g/pymock/README.md diff --git a/examples/case_g/pymock/input/args.txt b/tutorials/case_g/pymock/input/args.txt similarity index 100% rename from examples/case_g/pymock/input/args.txt rename to tutorials/case_g/pymock/input/args.txt diff --git a/examples/case_g/pymock/pymock/__init__.py b/tutorials/case_g/pymock/pymock/__init__.py similarity index 100% rename from examples/case_g/pymock/pymock/__init__.py rename to tutorials/case_g/pymock/pymock/__init__.py diff --git a/examples/case_g/pymock/pymock/libs.py b/tutorials/case_g/pymock/pymock/libs.py similarity index 100% rename from examples/case_g/pymock/pymock/libs.py rename to tutorials/case_g/pymock/pymock/libs.py diff --git a/examples/case_g/pymock/pymock/main.py b/tutorials/case_g/pymock/pymock/main.py similarity index 100% rename from examples/case_g/pymock/pymock/main.py rename to tutorials/case_g/pymock/pymock/main.py diff --git a/examples/case_g/pymock/pyproject.toml b/tutorials/case_g/pymock/pyproject.toml similarity index 100% rename from examples/case_g/pymock/pyproject.toml rename to tutorials/case_g/pymock/pyproject.toml diff --git a/examples/case_g/pymock/requirements.txt b/tutorials/case_g/pymock/requirements.txt similarity index 100% rename from examples/case_g/pymock/requirements.txt rename to tutorials/case_g/pymock/requirements.txt diff --git a/examples/case_g/pymock/run.py b/tutorials/case_g/pymock/run.py similarity index 100% rename from examples/case_g/pymock/run.py rename to tutorials/case_g/pymock/run.py diff --git a/examples/case_g/pymock/setup.cfg b/tutorials/case_g/pymock/setup.cfg similarity index 100% rename from examples/case_g/pymock/setup.cfg rename to tutorials/case_g/pymock/setup.cfg diff --git a/examples/case_g/pymock/setup.py b/tutorials/case_g/pymock/setup.py similarity index 100% rename from examples/case_g/pymock/setup.py rename to tutorials/case_g/pymock/setup.py diff --git a/tutorials/case_g/tests.yml b/tutorials/case_g/tests.yml new file mode 100644 index 0000000..445ed39 --- /dev/null +++ b/tutorials/case_g/tests.yml @@ -0,0 +1,13 @@ +- Catalog_N-test: + func: catalog_evaluations.number_test + plot_func: + - plot_number_test: + plot_args: + title: Test distribution + - plot_consistency_test: + plot_kwargs: + one_sided_lower: True + + + + diff --git a/examples/case_h/catalog.csv b/tutorials/case_h/catalog.csv similarity index 100% rename from examples/case_h/catalog.csv rename to tutorials/case_h/catalog.csv diff --git a/examples/case_h/config.yml b/tutorials/case_h/config.yml similarity index 88% rename from examples/case_h/config.yml rename to tutorials/case_h/config.yml index ca5c7f9..3d93d70 100644 --- a/examples/case_h/config.yml +++ b/tutorials/case_h/config.yml @@ -1,4 +1,4 @@ -name: case_f +name: case_h time_config: start_date: 2016-8-25T00:00:00 @@ -20,5 +20,5 @@ model_config: models.yml test_config: tests.yml postprocess: + report: custom_report.py:main plot_catalog: False - report: custom_report.py:main \ No newline at end of file diff --git a/examples/case_h/custom_report.py b/tutorials/case_h/custom_report.py similarity index 84% rename from examples/case_h/custom_report.py rename to tutorials/case_h/custom_report.py index 2d4d0cf..d705d5d 100644 --- a/examples/case_h/custom_report.py +++ b/tutorials/case_h/custom_report.py @@ -3,10 +3,19 @@ def main(experiment): + """ + Args: + experiment: a floatcsep.experiment.Experiment class + + """ + # Access the last time-window timewindow = experiment.timewindows[-1] + + # Convert the time-window to a string timestr = timewindow2str(timewindow) + # Instantiates a Report object and adds a title and objectives report = MarkdownReport() report.add_title(f"Experiment Report - {experiment.name}", "") report.add_heading("Objectives", level=2) @@ -17,6 +26,7 @@ def main(experiment): ] report.add_list(objs) + # Adds an input figure report.add_figure( f"Input catalog", [ @@ -44,4 +54,5 @@ def main(experiment): width=200, ) + # Stores the report report.save(experiment.registry.abs(experiment.registry.run_dir)) diff --git a/examples/case_h/models.yml b/tutorials/case_h/models.yml similarity index 100% rename from examples/case_h/models.yml rename to tutorials/case_h/models.yml index e2c99f7..dad36f1 100644 --- a/examples/case_h/models.yml +++ b/tutorials/case_h/models.yml @@ -1,8 +1,8 @@ - etas: giturl: https://git.gfz-potsdam.de/csep/it_experiment/models/vetas.git repo_hash: v3.2 - args_file: input/args.json path: models/etas + args_file: input/args.json func: etas-run func_kwargs: n_sims: 100 @@ -23,10 +23,10 @@ repo_hash: v0.1 path: models/pymock_nb func: pymock + prefix: pymock func_kwargs: n_sims: 100 mag_min: 3.5 - apply_mc_to_lambda: True - distribution: negbinom seed: 23 - prefix: pymock + distribution: negbinom + apply_mc_to_lambda: True diff --git a/examples/case_h/tests.yml b/tutorials/case_h/tests.yml similarity index 100% rename from examples/case_h/tests.yml rename to tutorials/case_h/tests.yml