You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/examples/case_a.rst
+20-27Lines changed: 20 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,9 @@
1
1
A - Simple Model and Catalog
2
2
============================
3
3
4
-
.. currentmodule:: floatcsep
4
+
The following example shows the definition of a testing experiment of a single **time-independent** forecast against a catalog.
5
5
6
-
.. contents::
7
-
:local:
6
+
.. currentmodule:: floatcsep
8
7
9
8
.. admonition:: **TL; DR**
10
9
@@ -16,11 +15,15 @@ A - Simple Model and Catalog
16
15
17
16
After the calculation is complete, the results will be summarized in ``results/report.md``.
18
17
18
+
.. contents::
19
+
:local:
20
+
:depth: 2
21
+
19
22
20
23
Experiment Components
21
24
---------------------
22
25
23
-
The following example shows the definition of a testing experiment of a single **time-independent** forecast against a catalog. The example files are found in ``floatcsep/examples/case_a``. The input structure of the experiment is:
26
+
The source code can be found in the ``examples/case_a`` folder or in `GitHub <https://github.com/cseptesting/floatcsep/blob/main/examples/case_a>`_. The directory structure of the experiment is:
24
27
25
28
::
26
29
@@ -34,14 +37,17 @@ The following example shows the definition of a testing experiment of a single *
34
37
* The testing region ``region.txt`` consists of a grid with two 1ºx1º bins, defined by its bottom-left nodes. The grid spacing is obtained automatically. The nodes are:
* The testing catalog ``catalog.csep`` contains only one event and is formatted in the :meth:`~pycsep.utils.readers.csep_ascii` style (see :doc:`pycsep:concepts/catalogs`). Catalog formats are detected automatically
* The forecast ``best_model.dat`` to be evaluated is written in the ``.dat`` format (see :doc:`pycsep:concepts/forecasts`). Forecast formats are detected automatically (see :mod:`floatcsep.utils.readers.ForecastParsers`)
The time configuration is manifested in the ``time_config`` inset. The simplest definition is to set only the start and end dates of the experiment. These are always UTC date-times in isoformat (``%Y-%m-%dT%H:%M:%S.%f`` - ISO861):
The region - a file path or a :mod:`pycsep` function, such as :obj:`~csep.core.regions.italy_csep_region` (check the available regions in :mod:`csep.core.regions`) -, the depth limits and magnitude discretization are defined in the ``region_config`` inset.
It is defined in the ``catalog`` inset. This should only make reference to a catalog file or a catalog query function (e.g. :func:`~csep.query_comcat`). ``floatcsep`` will automatically filter the catalog to the experiment time, spatial and magnitude frames:
95
+
It is defined in the ``catalog`` inset. This should only make reference to a catalog **file** or a catalog **query function** (e.g. :func:`~csep.query_comcat`). **floatCSEP** will automatically filter the catalog to the experiment time, spatial and magnitude frames:
The model configuration is set in the ``models`` inset with a list of model names, which specify their file paths (and other attributes). Here, we just set the path as ``best_model.dat``, whose format is automatically detected.
The experiment's evaluations are defined in the ``tests`` inset. It should be a list of test names, making reference to their function and plotting function. These can be either defined in ``pycsep`` (see :doc:`pycsep:concepts/evaluations`) or manually. In this example, we employ the consistency N-test: its function is :func:`csep.core.poisson_evaluations.number_test`, whereas its plotting function correspond to :func:`csep.utils.plots.plot_poisson_consistency_test`
The command ``floatcsep run <config>`` can be called from any working directory, as long as the specified file paths (e.g. region, models) are relative to the ``config.yml`` file.
131
-
132
-
133
-
Plot command
134
-
~~~~~~~~~~~~
135
-
136
-
If only the result plots are desired, when the calculation was already completed, you can type:
137
-
138
-
.. code-block:: console
139
-
140
-
$ floatcsep plot config.yml
141
-
142
-
This can be used, for example, when an additional plot is desired. Try adding to ``config.yml`` the following lines
143
-
144
-
.. code-block:: yaml
145
-
146
-
postproc_config:
147
-
plot_forecasts: True
141
+
The command ``floatcsep run {config_file}`` can be called from any working directory, as long as the specified file paths (e.g. region, models) are relative to the ``config.yml`` file.
148
142
149
-
and re-run with the ``plot`` command. A forecast figure will appear in ``results/{window}/forecasts``
150
143
151
144
Results
152
145
~~~~~~~
@@ -155,13 +148,13 @@ Results
155
148
156
149
* The testing catalog of the window is stored in ``results/{window}/catalog`` in ``json`` format. This is a subset of the global testing catalog.
157
150
* Human-readable results are found in ``results/{window}/evaluations``
158
-
* Catalog and evaluation results figures in ``results/{window}/figures``.
151
+
* Catalog, forecasts and evaluation results figures in ``results/{window}/figures``.
159
152
* The complete results are summarized in ``results/report.md``
160
153
161
154
162
155
Advanced
163
-
--------
156
+
~~~~~~~~
164
157
165
-
The experiment run logic can be seen in the file ``case_a.py``, which executes the same example but in python source code. The run logic of the terminal commands ``run``, ``plot`` and ``reproduce`` can be found in :mod:`floatcsep.commands.main`
158
+
The experiment run logic can be seen in the file ``case_a.py``, which executes the same example but in python source code. The run logic of the terminal commands ``run``, ``plot`` and ``reproduce`` can be found in :mod:`floatcsep.commands.main`, and can be customized by creating a script similar to ``case_a.py``.
Copy file name to clipboardExpand all lines: docs/examples/case_b.rst
+16-10Lines changed: 16 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,9 @@
1
1
B - Multiple Models and Tests
2
2
=============================
3
3
4
-
.. currentmodule:: floatcsep
4
+
The following example is an experiment including **multiple** time-independent forecasts and **multiple** evaluations.
5
5
6
-
.. contents::
7
-
:local:
6
+
.. currentmodule:: floatcsep
8
7
9
8
.. admonition:: **TL; DR**
10
9
@@ -16,11 +15,15 @@ B - Multiple Models and Tests
16
15
17
16
After the calculation is complete, the results will be summarized in ``results/report.md``.
18
17
18
+
.. contents::
19
+
:local:
20
+
:depth: 2
21
+
19
22
20
23
Experiment Components
21
24
---------------------
22
25
23
-
The following example is an experiment including **multiple** time-independent forecasts and evaluations. The input structure of the experiment is:
26
+
The source code can be found in the ``examples/case_b`` folder or in `GitHub <https://github.com/cseptesting/floatcsep/blob/main/examples/case_b>`_. The input structure of the experiment is:
24
27
25
28
::
26
29
@@ -36,11 +39,11 @@ The following example is an experiment including **multiple** time-independent f
36
39
├── tests.yml
37
40
└── region.txt
38
41
39
-
40
-
The testing catalog is now defined in ``json`` format, which is the default catalog used by ``floatcsep``, as it allows the storage of metadata.
42
+
.. important::
43
+
Although not necessary, the testing catalog is here defined in the ``.json`` format, which is the default catalog used by ``floatcsep``, as it allows the storage of metadata.
41
44
42
45
.. note::
43
-
An user-defined catalog can be saved as ``json`` with :meth:`CSEPCatalog.write_json() <csep.core.catalogs.CSEPCatalog.write_json>` using ``pycsep``
46
+
A catalog can be stored as ``.json`` with :meth:`CSEPCatalog.write_json() <csep.core.catalogs.CSEPCatalog.write_json>` using ``pycsep``
44
47
45
48
46
49
Configuration
@@ -49,10 +52,11 @@ Configuration
49
52
In this example, the time, region and catalog specifications are written in the ``config.yml`` file.
Plotting keyword arguments can be set in the ``plot_kwargs`` option - see :func:`~csep.utils.plots.plot_poisson_consistency_test` and :func:`~csep.utils.plots.plot_comparison_test` -.
78
84
79
-
.. note::
85
+
.. important::
80
86
Comparison tests (such as the ``paired_t_test``) requires a reference model, whose name should be set in ``ref_model`` at the given test configuration.
81
87
82
88
Running the experiment
@@ -87,7 +93,7 @@ The experiment can be run by simply navigating to the ``examples/case_b`` folder
87
93
88
94
.. code-block:: console
89
95
90
-
floatcsep run config.yml
96
+
$ floatcsep run config.yml
91
97
92
98
This will automatically set all the file paths of the calculation (testing catalogs, evaluation results, figures) and will display a summarized report in ``results/report.md``.
Copy file name to clipboardExpand all lines: docs/examples/case_c.rst
+26-15Lines changed: 26 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,9 @@
1
1
C - Multiple Time Windows
2
2
=========================
3
3
4
-
.. currentmodule:: floatcsep
4
+
The following example shows an experiment with **multiple time windows**.
5
5
6
-
.. contents::
7
-
:local:
6
+
.. currentmodule:: floatcsep
8
7
9
8
.. admonition:: **TL; DR**
10
9
@@ -16,11 +15,15 @@ C - Multiple Time Windows
16
15
17
16
After the calculation is complete, the results will be summarized in ``results/report.md``.
18
17
18
+
.. contents::
19
+
:local:
20
+
21
+
19
22
20
23
Experiment Components
21
24
---------------------
22
25
23
-
The following example shows an experiment with multiple time windows. The input structure of the experiment is:
26
+
The source code can be found in the ``examples/case_c`` folder or in `GitHub <https://github.com/cseptesting/floatcsep/blob/main/examples/case_c>`_. The input structure of the experiment is:
24
27
25
28
::
26
29
@@ -42,11 +45,12 @@ Configuration
42
45
Time
43
46
~~~~
44
47
45
-
The time configuration now set the time intervals between the start and end dates.
48
+
The time configuration now sets a sequence of time intervals between the start and end dates.
The experiment's evaluations are defined in ``tests.yml``, which can now include temporal evaluations (see :func:`~floatcsep.extras.sequential_likelihood`, :func:`~floatcsep.extras.sequential_information_gain`, :func:`~floatcsep.utils.plot_sequential_likelihood`).
65
+
The experiment's evaluations are defined in ``tests.yml``, which can now include temporal evaluations (see :func:`~floatcsep.utils.helpers.sequential_likelihood`, :func:`~floatcsep.utils.helpers.sequential_information_gain`, :func:`~floatcsep.utils.helpers.plot_sequential_likelihood`).
Plot arguments (title, labels, font sizes, axes limits, etc.) can be passed as a dictionary in ``plot_args`` (see details in :func:`~csep.utils.plots.plot_poisson_consistency_test`)
73
+
Plot arguments (title, labels, font sizes, axes limits, etc.) can be passed as a dictionary in ``plot_args`` (see the arguments details in :func:`~csep.utils.plots.plot_poisson_consistency_test`)
69
74
70
75
Results
71
76
-------
72
77
73
-
The :obj:`~floatcsep.cmd.main.run` command creates the result path tree for all time windows.
78
+
The :obj:`~floatcsep.cmd.main.run` command
79
+
80
+
.. code-block:: console
81
+
82
+
$ floatcsep run config.yml
83
+
84
+
now creates the result path tree for all time windows.
74
85
75
-
* The testing catalog of the window is stored in ``results/{window}/catalog`` in ``json`` format. This is a subset of the global testing catalog.
76
-
* Human-readable results are found in ``results/{window}/evaluations``
77
-
* Catalog and evaluation results figures in ``results/{window}/figures``.
86
+
* The testing catalog of the window is stored in ``results/{time_window}/catalog`` in ``json`` format. This is a subset of the global testing catalog.
87
+
* Human-readable results are found in ``results/{time_window}/evaluations``
88
+
* Catalog and evaluation results figures in ``results/{time_window}/figures``.
78
89
* The complete results are summarized in ``results/report.md``
79
90
80
-
The report now shows the temporal evaluations for all time-windows, whereas the discrete evaluations are shown only for the last time window.
91
+
The report shows the temporal evaluations for all time-windows, whereas the discrete evaluations are shown only for the last time window.
Copy file name to clipboardExpand all lines: docs/examples/case_d.rst
+15-9Lines changed: 15 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,11 @@
1
1
D - Catalog Queries and Model Repositories
2
2
==========================================
3
3
4
-
.. currentmodule:: floatcsep
4
+
The following example shows an experiment whose forecasts are **retrieved from a repository** (Zenodo - https://zenodo.org) and the testing **catalog** from an authoritative source **web service** (namely the gCMT catalog from the International Seismological Centre - http://www.isc.ac.uk).
5
+
5
6
6
-
.. contents::
7
-
:local:
7
+
8
+
.. currentmodule:: floatcsep
8
9
9
10
.. admonition:: **TL; DR**
10
11
@@ -16,11 +17,14 @@ D - Catalog Queries and Model Repositories
16
17
17
18
After the calculation is complete, the results will be summarized in ``results/report.md``.
18
19
20
+
.. contents::
21
+
:local:
22
+
19
23
20
24
Experiment Components
21
25
---------------------
22
26
23
-
The following example shows an experiment whose forecasts are retrieved from a repository (Zenodo - https://zenodo.org) and the testing catalog from an authoritative source web service (namely the gCMT catalog from the ISC - http://www.isc.ac.uk). The initialstructure is:
27
+
The source code can be found in the ``examples/case_d`` folder or in `GitHub <https://github.com/cseptesting/floatcsep/blob/main/examples/case_d>`_. The **initial** input structure of the experiment is:
24
28
25
29
::
26
30
@@ -60,8 +64,9 @@ Catalog
60
64
The ``catalog`` inset from ``config.yml`` now makes reference to a catalog query function, in this case :func:`~pycsep.query_gcmt`.
* The option ``zenodo_id`` makes reference to the zenodo **record id**. The model ``team`` is found in https://zenodo.org/record/6289795, whereas the model ``wheel`` in https://zenodo.org/record/6255575.
80
86
81
-
* The option ``flavours`` allows multiple model sub-classes to be quickly instantiated.
82
-
83
87
* The ``zenodo`` (or ``git``) repositories could contain multiple files, each of which can be assigned to a flavour.
84
88
89
+
* The option ``flavours`` allows multiple model sub-classes to be quickly instantiated.
90
+
85
91
* When multiple flavours are passed, ``path`` refers to the folder where the models would be downloaded.
86
92
87
93
* If a single file of the repository is needed (without specifying model flavours), ``path`` can reference to the file itself. For example, you can try replacing the whole WHEEL inset in ``models.yml`` to:
0 commit comments