ImageSource.save Optics Block #1337

j-c-c · 2025-11-03T19:49:25Z

Save ImageSource's with Relion >= 3.1 convention for starfiles, with separate data_optics and data_particle blocks.

Some other additions in this PR are:

Always store amplitudes in doubles
Rename _rlnAmplitude metadata field to _aspireAmplitude, since _rlnAmplitude is not a valid Relion field name
Save ImageSource mrcs files with pixel/voxel size in header. Some Relion command line tools extract pixel size (angpix) from here.
Save with optics block required fields _rlnImageSize and _rlnImageDimensionality
Testing for added features
Loading new file format in Relion was verified using several Relion CLI tools with Relion version 5.0.1

garrettwrong

Nice, thanks. Couple minor things noted.

Can you add a test that covers slicing the source? I don't think arbitrary slicing will mess up your logic, but it might in the future. Better to add test now while it works :D.

If you haven't let's make sure you can convert some real STAR files that we use. (Read them in and write them out in a re-usable way, ideally with the user doing nothing...). If we can't, that is going to be a problem and I'm not sure we could take the code yet. This may be fine already, so don't interpret it as a code criticism 😇 .

Can we document in a tutorial an example that shows generating a realistic Simulation, saving with this new format, and loading with Relion? (This can be a non-running, display only, example). I see your testing that we can load one back ourselves, but, we're probably not the target application...

Thanks!

garrettwrong · 2025-11-04T20:01:25Z

src/aspire/source/image.py

+        optics_block["_rlnOpticsGroup"] = []
+        optics_block["_rlnOpticsGroupName"] = []
+        for field in optics_value_fields:
+            optics_block[field] = []


This is good. Try using a defaultdict(list) for optics_block and see if that reduces some of this code.

Fact check me, but I believe all Python dicts are now naturally ordered for versions of Python we support.

Yeah, that's cleaner. Thanks.

Yep, you're correct about dicts being ordered.

garrettwrong · 2025-11-04T20:02:18Z

src/aspire/source/image.py

+        if "_rlnImageSize" not in metadata:
+            metadata["_rlnImageSize"] = np.full(self.n, self.L, dtype=int)
+
+        if "_rlnImageDimensionality" not in metadata:


what in the world

garrettwrong · 2025-11-04T20:08:41Z

src/aspire/source/relion.py

        metadata = RelionStarFile(self.filepath).get_merged_data_block()

+        # Promote legacy _rlnAmplitude column to the ASPIRE-specific name.
+        if "_rlnAmplitude" in metadata and "_aspireAmplitude" not in metadata:


I appreciate you are trying to make old files forward compatible. Nice idea, but perhaps too dangerous. Consider if Relion decides tomorrow they want to use _rlnAmplitude ...

How much stuff is actually going to break if we do not do this...?

It's not going to break anything. I just had it in for compatibility with old files. Removed.

garrettwrong · 2025-11-04T20:10:04Z

src/aspire/utils/relion_interop.py

 # of certain key fields used in the codebase,
 # which are originally read from Relion STAR files.
 relion_metadata_fields = {
+    "_aspireAmplitude": float,


Wasn't the whole point that _aspireAmplitude doesn't belong in relion_metadata_fields 😇 ?...

oops. Removed.

garrettwrong · 2025-11-04T20:10:41Z

tests/test_array_image_source.py

-    np.testing.assert_equal(src.amplitudes.dtype, dtype)

-    # offsets are always stored as doubles
+    # offsets and amplitudes are always stored as doubles


good idea, thanks

garrettwrong · 2025-11-04T20:14:06Z

tests/test_simulation.py

+    sim.save(starpath, overwrite=True)
+
+    star = RelionStarFile(str(starpath))
+    assert star.relion_version == "3.1"


just curious, are we exactly making 3.1 only, or some later version or ? What versions are possible?

It's really 3.1 up to the current version. The format has stayed the same, with the added optics block, since 3.1.

garrettwrong · 2025-11-04T20:16:21Z

tests/test_simulation.py

+    np.testing.assert_array_equal(optics["_rlnImageSize"], np.full(kv_ct, res))
+    np.testing.assert_array_equal(optics["_rlnImageDimensionality"], np.full(kv_ct, 2))
+
+    # Due to Simulation random indexing, voltages will be unordered


I don't think that is true. You should have the order of filters in the simulation. I think this ordering would follow from that one if I understand correctly.

Maybe "unordered" is the wrong word here. The optics["_rlnVoltage"])'s will be in the order they were encountered during save. That will not necessarily be the same ascending order the voltages were in when we created the ctf filters (which get applied randomly to the images).

garrettwrong · 2025-11-04T20:18:46Z

tests/test_simulation.py



+@pytest.mark.parametrize("batch_size", [1, 6])
+def test_simulation_save_sets_voxel_size(tmp_path, batch_size):


Nice, thanks

codecov · 2025-11-05T21:14:56Z

Codecov Report

❌ Patch coverage is 98.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90.68%. Comparing base (34f82da) to head (5cb9557).

Files with missing lines	Patch %	Lines
src/aspire/source/image.py	98.14%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1337      +/-   ##
===========================================
+ Coverage    90.65%   90.68%   +0.02%     
===========================================
  Files          134      134              
  Lines        14431    14487      +56     
===========================================
+ Hits         13082    13137      +55     
- Misses        1349     1350       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…-generated dummy values when loading. Unit test.

j-c-c self-assigned this Nov 3, 2025

j-c-c added cleanup extern Relating to external changes labels Nov 3, 2025

j-c-c requested a review from garrettwrong November 4, 2025 14:59

garrettwrong requested changes Nov 4, 2025

View reviewed changes

j-c-c added 22 commits November 14, 2025 08:10

initial ImageSource.save with optics block

f709929

add _rlnImageSize column to metadata when saving.

133a3d7

Add _rlnImageSize to coordinate source test.

bc01f4c

use rlnImageName to get n_rows

0b1204f

test for sim save with optics block

e4e24b0

add _rlnImageDimensionality

7c88e0a

test save/load roundtrip w/ phase_flip.

3e78f9a

update coord source test

816923c

cleanup

85b9fd4

remove unused var

10056b1

missing optics fields warning. warning test. code comments.

6a84644

removed unused import.

35b0532

deprecate _rlnAmplitude in favor of private __amplitude field.

4da7ee8

use _aspireAmplitude to retain ampitudes for saved files

1ef064c

ensure pixel size is added to mrc header

020e38c

test mrc pixel_size in header

65fbfcd

Use defaultdict

5c28890

Remove old file compatibility

8ab9ee7

Remove _aspireAmplitude from relion_metadata_fields dict.

60942fa

Test save on source slices

7091078

Always write an optics block

195f44c

Gallery example simulation -> relion

a3d6381

j-c-c force-pushed the save_optics_group branch from 934a2cd to a3d6381 Compare November 14, 2025 13:11

clean up gallery

7324533

j-c-c added 2 commits November 14, 2025 15:14

_aspireMetadata 'no_ctf' tag for missing optics fields. Detect ASPIRE…

665e038

…-generated dummy values when loading. Unit test.

test cleanup

5cb9557



		@pytest.mark.parametrize("batch_size", [1, 6])
		def test_simulation_save_sets_voxel_size(tmp_path, batch_size):

ImageSource.save Optics Block #1337

Are you sure you want to change the base?

ImageSource.save Optics Block #1337

Uh oh!

Conversation

j-c-c commented Nov 3, 2025

Uh oh!

garrettwrong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Nov 5, 2025 •

edited

Loading