`preprocess` command line option: expand and integrate with EmanSource

[chris-langfield]

We have a preprocess command that takes a number of arguments, but is tailored to the RelionSource class. The task here is to evaluate whether this command is as useful as it could be and to see whether it can be expanded to include options for preprocessing an EmanSource type dataset.

preprocess command

[janden]

We should be careful with this. There's a temptation to make the command-line tools work with a very wide variety of inputs, but this increase maintenance. A simpler approach might just be to have a converter of sorts that extracts the images and writes them to a star–mrcs pair.

On the other hand, the relion_preprocess command seems to take either star file or box files as input, so perhaps it is not so bad. The question would then be how to do it elegantly.

[chris-langfield]

That's true, technically if you .save() an EmanSource it saves as .mrcs stacks of particles with a STAR file, but with all null/default values for the metadata. I haven't tried loading that itself as a RelionSource yet.

I guess we would want to decide whether to try to populate some of that information (e.g. CTF) before saving?

[janden]

As discussed during the meeting, it would make sense to just output a star file without those metadata values and work towards RelionSource accepting such bare-bones star files.

[chris-langfield]

What if we just had a command line utility to quickly convert an EmanSource type data source into a RelionSource compatible one. It could even have a flag that enables the preprocessing steps not requiring Metadata. So the flow could be

• Download dataset with micrographs and coordinates
• Run cmd line utility, which crops the images and optionally runs downsample, whiten, etc and saves to a Star and a .mrcs stack
• Then you have basically a "clean" RelionSource with no metadata, which you could run further Relion tools on, or ours, and plug into RelionSource constructor for further work?

[chris-langfield]

It could wrap up the sophistication of the class I wrote, and for advanced users, the class/API for working with the coordinates source would still be there

[janden]

I think this makes sense. This is essentially what relion_preprocess does IIRC.

[chris-langfield]

This idea is implemented with the extract-particles cmd in #520, with the exception that it does not have preprocessing options. This could be done afterwards using the preprocess command and it's probably not worth packing these options into extract-particles as well.

[janden]

Normally, I would agree in keeping these functionalities separate. However, the issue is that preprocess has one very important step, downsampling, which can significantly reduce storage requirements. To keep essentially two copies of the data (one in the micrograph and one in full-resolution projection images) may be prohibitive in applications.

[chris-langfield]

Closed by #551