@@ -10,9 +10,10 @@ For this, a basic 'graphics test' assertion operation is provided in the method
1010match against a stored reference.
1111A "graphics test" is any test which employs this.
1212
13- At present (Iris version 1.10), such tests include the testing for modules
14- `iris.tests.test_plot ` and `iris.tests.test_quickplot `, and also some other
15- 'legacy' style tests (as described in :ref: `developer_tests `).
13+ At present, such tests include the testing for modules `iris.tests.test_plot `
14+ and `iris.tests.test_quickplot `, all output plots from the gallery examples
15+ (contained in `docs/iris/example_tests `), and a few other 'legacy' style tests
16+ (as described in :ref: `developer_tests `).
1617It is conceivable that new 'graphics tests' of this sort can still be added.
1718However, as graphics tests are inherently "integration" style rather than true
1819unit tests, results can differ with the installed versions of dependent
@@ -38,80 +39,110 @@ Testing actual plot results introduces some significant difficulties :
3839Graphics Testing Strategy
3940=========================
4041
41- Prior to Iris 1.10, all graphics tests compared against a stored reference
42- image with a small tolerance on pixel values.
42+ In the Iris Travis matrix, and over time, graphics tests must run with
43+ multiple versions of Python, and of key dependencies such as matplotlib.
44+ To make this manageable, the "check_graphic" test routine tests against
45+ multiple alternative 'acceptable' results. It does this using an image "hash"
46+ comparison technique which avoids storing reference images in the Iris
47+ repository itself, to avoid space problems.
4348
44- From Iris v1.11 onward, we want to support testing Iris against multiple
45- versions of matplotlib (and some other dependencies).
46- To make this manageable, we have now rewritten "check_graphic" to allow
47- multiple alternative 'correct' results without including many more images in
48- the Iris repository.
4949This consists of :
5050
51- * using a perceptual 'image hash' of the outputs (see
52- https://github.com/JohannesBuchner/imagehash) as the basis for checking
51+ * The 'check_graphic' function uses a perceptual 'image hash' of the outputs
52+ (see https://github.com/JohannesBuchner/imagehash) as the basis for checking
5353 test results.
54- * storing the hashes of 'known accepted results' for each test in a
55- database in the repo (which is actually stored in
56- ``lib/iris/tests/results/imagerepo.json ``).
57- * storing associated reference images for each hash value in a separate public
58- repository, currently in https://github.com/SciTools/test-images-scitools ,
59- allowing human-eye judgement of 'valid equivalent' results.
60- * a new version of the 'iris/tests/idiff.py' assists in comparing proposed
61- new 'correct' result images with the existing accepted ones.
62-
63- BRIEF...
64- There should be sufficient work-flow detail here to allow an iris developer to:
65-
66- * understand the new check graphic test process
67- * understand the steps to take and tools to use to add a new graphic test
68- * understand the steps to take and tools to use to diagnose and fix an graphic test failure
69-
70-
71- Basic workflow
72- ==============
73-
74- If you notice that a graphics test in the Iris testing suite has failed
75- following changes in Iris or any of its dependencies, this is the process
76- you now need to follow:
77-
78- #. Create a directory in iris/lib/iris/tests called 'result_image_comparison'.
79- #. From your Iris root directory, run the tests by using the command:
80- ``python setup.py test ``.
81- #. Navigate to iris/lib/iris/tests and run the command: ``python idiff.py ``.
82- This will open a window for you to visually inspect the changes to the
83- graphic and then either accept or reject the new result.
84- #. Upon acceptance of a change or a new image, a copy of the output PNG file
85- is added to the reference image repository in
86- https://github.com/SciTools/test-images-scitools. The file is named
87- according to the image hash value, as ``<hash>.png ``.
88- #. The hash value of the new result is added into the relevant set of 'valid
89- result hashes' in the image result database file,
90- ``tests/results/imagerepo.json ``.
91- #. The tests must now be re-run, and the 'new' result should be accepted.
92- Occasionally there are several graphics checks in a single test, only the
93- first of which will be run should it fail. If this is the case, then you
94- may well encounter further graphical test failures in your next runs, and
95- you must repeat the process until all the graphical tests pass.
96- #. To add your changes to Iris, you need to make two pull requests. The first
97- should be made to the test-images-scitools repository, and this should
98- contain all the newly-generated png files copied into the folder named
99- 'image_files'.
100- #. The second pull request should be created in the Iris repository, and should
101- only include the change to the image results database
102- (``tests/results/imagerepo.json ``) :
103- This pull request must contain a reference to the matching one in
104- test-images-scitools.
54+ * The hashes of known 'acceptable' results for each test are stored in a
55+ lookup dictionary, saved to the repo file
56+ ``lib/iris/tests/results/imagerepo.json `` .
57+ * An actual reference image for each hash value is stored in a *separate *
58+ public repository : https://github.com/SciTools/test-iris-imagehash .
59+ * The reference images allow human-eye assessment of whether a new output is
60+ judged to be 'close enough' to the older ones, or not.
61+ * The utility script ``iris/tests/idiff.py `` automates checking, enabling the
62+ developer to easily compare proposed new 'acceptable' result images against the
63+ existing accepted reference images, for each failing test.
10564
106- Note: the Iris pull-request will not test out successfully in Travis until the
107- test-images-scitools pull request has been merged : This is because there is
108- an Iris test which ensures the existence of the reference images (uris) for all
109- the targets in the image results database.
11065
66+ How to Add New 'Acceptable' Result Images to Existing Tests
67+ ========================================
68+
69+ When you find that a graphics test in the Iris testing suite has failed,
70+ following changes in Iris or the run dependencies, this is the process
71+ you should follow:
72+
73+ #. Create a new, empty directory to store temporary image results, at the path
74+ ``lib/iris/tests/result_image_comparison `` in your Iris repository checkout.
75+
76+ #. **In your Iris repo root directory **, run the relevant (failing) tests
77+ directly as python scripts, or by using a command such as
78+ ``python -m unittest discover paths/to/test/files ``.
79+
80+ #. **In the ** ``iris/lib/iris/tests `` **folder **, run the command: ``python idiff.py ``.
81+ This will open a window for you to visually inspect side-by-side 'old', 'new'
82+ and 'difference' images for each failed graphics test.
83+ Hit a button to either "accept", "reject" or "skip" each new result ...
84+
85+ * If the change is *"accepted" * :
86+
87+ * the imagehash value of the new result image is added into the relevant
88+ set of 'valid result hashes' in the image result database file,
89+ ``tests/results/imagerepo.json `` ;
90+
91+ * the relevant output file in ``tests/result_image_comparison `` is
92+ renamed according to the image hash value, as ``<hash>.png ``.
93+ A copy of this new PNG file must then be added into the reference image
94+ repository at https://github.com/SciTools/test-iris-imagehash.
95+ (See below).
96+
97+ * If a change is *"skipped" * :
98+
99+ * no further changes are made in the repo.
100+
101+ * when you run idiff again, the skipped choice will be presented again.
111102
112- Fixing a failing graphics test
113- ==============================
103+ * If a change is *"rejected" * :
114104
105+ * the output image is deleted from ``result_image_comparison ``.
115106
116- Adding a new graphics test
117- ==========================
107+ * when you run idiff again, the skipped choice will not appear, unless
108+ and until the relevant failing test is re-run.
109+
110+ #. Now re-run the tests. The 'new' result should now be recognised and the
111+ relevant test should pass. However, some tests can perform *multiple * graphics
112+ checks within a single testcase function : In those cases, any failing
113+ check will prevent the following ones from being run, so a test re-run may
114+ encounter further (new) graphical test failures. If that happens, simply
115+ repeat the check-and-accept process until all tests pass.
116+
117+ #. To add your changes to Iris, you need to make two pull requests :
118+
119+ * (1) The first PR is made in the test-iris-imagehash repository, at
120+ https://github.com/SciTools/test-iris-imagehash.
121+
122+ * First, add all the newly-generated referenced PNG files into the
123+ ``images/v4 `` directory. In your Iris repo, these files are to be found
124+ in the temporary results folder ``iris/tests/result_image_comparison ``.
125+
126+ .. Note ::
127+
128+ The ``result_image_comparison `` folder is covered by a project
129+ ``.gitignore `` setting, so those files *will not show up * in a
130+ ``git status `` check.
131+
132+ * Then, run ``python recreate_v4_files_listing.py ``, to update the file
133+ which lists available images, ``v4_files_listing.txt ``.
134+
135+ * Create a PR proposing these changes, in the usual way.
136+
137+ * (2) The second PR is created in the Iris repository, and
138+ should only include the change to the image results database,
139+ ``tests/results/imagerepo.json `` :
140+ The description box of this pull request should contain a reference to
141+ the matching one in test-iris-imagehash.
142+
143+ Note: the Iris pull-request will not test out successfully in Travis until the
144+ test-iris-imagehash pull request has been merged : This is because there is
145+ an Iris test which ensures the existence of the reference images (uris) for all
146+ the targets in the image results database. N.B. likewise, it will *also * fail
147+ if you forgot to run ``recreate_v4_files_listing.py `` to update the image-listing
148+ file in test-iris-imagehash.
0 commit comments