Components and configuration

Overview

A Murfey deployment is comprised of multiple components, illustrated in the diagram below. These consist of:

• A RabbitMQ message broker instance for passing messages between processing services
• A PostgreSQL database instance for keeping track of data collection and processing information
• The Murfey server which handles data collection registration and processing requests
• A Murfey client which watches for data and notifies the Murfey server

In addition there are optional components:

• ISPyB laboratory information management system for integration with user visit information and for processing results recording for display with the PATo web frontend
• A Graylog instance for log recording

Configuration and credentials files

Murfey will require a variety of configuration and credentials files to control the processes it will execute and secure the connections between its components. In this section, we show example configuration files for a Murfey setup where SPA TEM data is transferred from a Windows PC to a Linux server and storage system for subsequent processing.

Client configuration

An example of the .murfey file:

[Murfey]
instrument_name = krios
server = 192.168.0.0:8000
token = antidisestablishmentarianism

Machine configuration

An example of the murfey-machine-config.yaml file that has been set up for a TEM:

instrument-name:
    # General microscope information
    display_name: Krios
    instrument_name: krios
    machine_override: ""
    image_path: /home/User/Pictures/my_microscope.png

    # Acquisition hardware and software (TEM only)
    camera: K3
    superres: true
    flip_gain: ""
    calibrations:
    magnification:
        10000: 8.000
        20000: 4.000
        40000: 2.000
        80000: 1.000
        160000: 0.500
    acquisition_software:
    - epu
    software_versions:
      epu: 1.00
    data_required_substrings:
    epu:
        .mrc:
        - fractions
        .tiff:
        - fractions

    # Client-side directories
    data_directories:
    - "D:/DoseFractions"
    create_directories:
    - atlas
    analyse_created_directories:
    - atlas
    gain_reference_directory: "C:/ProgramData/Gatan/Reference Images"

    # Data transfer settings
    data_transfer_enabled: true
    rsync_basepath: /home/User/my-data
    rsync_module: data
    rsync_url: "http://192.168.0.2"
    allow_removal: true

    # Data processing settings
    processing_enabled: true
    processed_directory_name: processed
    processed_extra_directory: ""
    recipes:
    em-spa-bfactor: em-spa-bfactor
    em-spa-class2d: em-spa-class2d
    em-spa-class3d: em-spa-class3d
    em-spa-preprocess: em-spa-preprocess
    em-spa-refine: em-spa-refine
    em-tomo-align: em-tomo-align
    em-tomo-preprocess: em-tomo-preprocess
    default_model: my_particle_picking_ml_model.h5
    model_search_directory: processing
    initial_model_search_directory: processing/initial_model
    external_executables: {}
    external_environment: {}

    # Network and security settings
    security_configuration_path: /home/User/murfey-security-config.yaml
    murfey_url: "http://192.168.0.0:8000"
    frontend_url: "http://192.168.0.0:3000"
    instrument_server_url: "http://192.168.0.1:8000"
    node_creator_queue: "node_creator"

Security configuration

An example of the murfey-security-config.yaml file:

murfey_db_credentials: /home/User/murfey-db-credentials.yaml
crypto_key: supercalifragilisticexpialidocious
sqlalchemy_pooling: false

auth_key: hippopotomonstrosesquippedaliophobia
auth_algorithm: HS256
auth_type: "password"
cookie_key: "cookie_auth"
session_validation: ""
session_token_timeout: 30000

rabbitmq_credentials: /home/User/rabbitmq-credentials.yaml
feedback_queue: murfey_feedback

graylog_host: 192.168.0.5
graylog_port: 9000

ispyb_credentials: ""

PostgreSQL database credentials

An example of the murfey-db-credentials.yaml file:

username: murfey
password: llanfair­pwllgwyngyll­gogery­chwyrn­drobwll­llan­tysilio­gogo­goch
host: 192.168.0.4
port: 5432
database: murfey_db

RabbitMQ credentials

An example of the rabbitmq-credentials.yaml file:

[rabbit]
plugin: pika
host: 192.168.0.6
port: 5672
username: murfey
password: pneumonoultramicroscopicsilicovolcanoconiosis
vhost: zocalo