Skip to content

Study Config

Jump to bottom Edit New page
Dawn E. Smith edited this page Aug 13, 2018 · 11 revisions

This file holds metadata for a study, configures the expected scans for each scan site, helps to properly tag them, and holds settings for pipelines that the data is run through.

NOTE: All keys are case sensitive. If Datman is complaining about KeyErrors for any of these settings check that your spelling and case are correct.

Table of Contents

Study Metadata Block

This block holds descriptive information about the study, and some configuration needed to retrieve new scans from an sftp server.

Required Settings

  • MRUSER: The user name used to access this study's data on the sftp server configured. Only needs to be set if FTPSERVER is set in the site config or the study's config file.
  • MRFOLDER: The folder name or relative path to the folder on the sftp server in which the data will be found. If more than one folder exists a regex can be used to match them all. e.g. ASDD*MR to match all folders beginning with 'ASDD' and ending in 'MR'. Or a list can be used, e.g. '['FOLDER1', 'FOLDER2']'. You can also use include a regex in the list of folder names. This only needs to be set if FTPSERVER is set in the site config or the study's config file.
  • PROJECTDIR: The name of the folder that will hold all of the study's contents
  • STUDY_TAG: The tag used in the Datman ID for all of this study's data. This tag can be overridden by SITE_TAGS in the site config block if different sites will use slightly different study tags (not advisable because it makes the data harder to work with for humans). See here for more info

Optional Settings

These settings are useful documentation for humans but are not read by any Datman scripts and therefore are not strictly required to be defined.

  • FullName: The full name of the study.
  • Description: A freeform description of the study and the data it contains.
  • PrimaryContact: The name of the contact for the study.

Example Metadata

This is an excerpt from our 'STOPPD' study. Note that in this example MRFOLDER is making use of a list of regexes instead of a single folder name.

MRUSER : [UNAME1, UNAME2]                             
MRFOLDER : ["STOP*MR", "SRSD*MR"]                
PROJECTDIR: STOPPD 
STUDY_TAG: STOPPD
FullName: 
Description: Effects of Maintenance Treatment with Olanzapine vs. Placebo on Brain Structure
PrimaryContact: Aristotle Voineskos

REDCap Settings

For any study or site that expects a 'Scan Completed' redcap survey this should be added to the study config file:

USES_REDCAP: True

If every site in a study collects these surveys (even if some surveys come from remote servers, see below) add it outside of the 'Sites' block so it turns on the setting for all sites in one line. If only one or a few sites collect the survey you can add this line inside of that site's config block alongside the ExportInfo and XNAT_Archive settings. This setting is read by dashboard/add_study_info.py to update the dashboard's database and turn on warning messages when surveys are missing.

The dashboard pulls in MR Scan Comments from a 'Scan Completed' instrument in REDCap using REDCap's Data Entry Trigger. For external REDCap instances, the firewall prevents this from happening. As an alternative, datman has a dm_redcap_scan_completed.py script. To use this, the following REDCap fields need to be included in the study's config file:

REDCAP_URL: 'https://redcap.example.com/redcap/api/'  # REDCap API url
REDCAP_TOKEN: 'redcap-token'  # file within the study metadata folder that contains the API token
REDCAP_PROJECTID: '1'  # Study ID within REDCap
REDCAP_EVENTID: {'control_arm_1': '0001', 'case_arm_1': '0002'}  # dictionary of REDCap EventID mapping
REDCAP_INSTRUMENT: 'scan_completed'  # REDCap instrument name
REDCAP_SUBJ: 'mr_id'  # valid DATMAN MR session label
REDCAP_DATE: 'mr_date'  # Relevant date field within instrument
REDCAP_STATUS: 'mr_status'  # Indicator of whether MR Session was completed or not
REDCAP_COMMENTS: 'mr_comments'  # Relevant comment field within instrument

Sites Block

Each scan site for a study needs its own 'site block' of configuration. This section takes the form

Sites:
   SITE1:
        <site1 config>
   SITE2:
        <site2 config here>
   ...
   SITEN:
        <siteN config here>

where SITEX should match the 'site' tag from the Datman style ID.

Expected Settings

These settings should be defined for each defined site.

  • XNAT_Archive: This should contain the name of the archive on XNAT where this scan site's data will be stored. That is, it should match XNAT's 'Project ID' field for the project this site will upload to / download from.
  • ExportInfo: This should hold one entry for each tag this site will use. Entries take the form TAG: {setting1: X, setting2: X, ... settingN: X}. The tag from the Datman style file name acts as the 'TAG' field and at least the settings below must be defined for each entry. In addition, defaults from the ExportSettings for that tag can be overridden here by being included in the tag's settings.
    • Pattern: A string or a python style regex that will match the series description field in the DICOM headers for all scans intended to receive this tag. For example, a 'DTI60-1000' tag meant to be assigned to scans with a series description of 'somethingDTI-60something' might have a pattern set to 'DTI.60' or even just 'DTI' if that is unique enough not to match other series descriptions. If there are multiple different values the series descriptions might take for a single tag a list of strings / regexes can be used in place of a single one.
    • Count: The number of matching series expected for each session at this site. e.g. if a session is expected to collect one T1 for set Count: 1 for the T1 tag, if three DTI60-1000s are taken set Count: 3 for that tag, etc.

Optional Site Settings

Unique Site SFTP Server

Including these tags in a site block will let you configure a different sftp server just for a site:

  • FTPSERVER: This overrides the default server that is usually set in the site wide config.
  • FTPPORT: An optional setting that allows a site to use a a non-standard port for the sftp server
  • MRFTPPASS: This should be set to the name of the file in the metadata folder that will hold this site's sftp account password. A full path can be provided if the file is stored outside of the metadata folder.
  • MRUSER: Overrides the default MRUSER for the study. See the section on study metadata for more
  • MRFOLDER: Overrides the default MRFOLDER for the study. See the section on study metadata for more

Fetching Site Data from an External XNAT Server

Including these tags will allow you to use xnat_fetch_remote.py to pull in data from an external XNAT server for a site:

  • XNAT_source: The URL for the remote XNAT server to pull from
  • XNAT_source_archive: The Project ID on the XNAT server that holds this site's data
  • XNAT_source_credentials: The name of the text file in the metadata folder that will hold the username and password on separate lines (in that order).

Overriding the Study Tag

Due to historical reasons, to override the study tag for a site you have to use a unique tag instead of just re-declaring the same key with a different value. This can be done by including the SITE_TAGS setting in a site block and must be set to a list even if there is only one alternate tag that a site uses (again because of ancient design decisions, sorry). Note that this doesn't completely override the study tag. Data from a site that defines SITE_TAGS may have study tags matching those tags OR the study's tag defined in STUDY_TAG.

Configuring Additional Redcap Servers

If a site uses a different redcap server to host its 'Scan Completed' surveys (or any other surveys that you may need to pull in) you can add the 'REDCAPAPI' setting to a site block to declare a new server just for that site.

Example Sites Block

Below is an example of two scan sites named 'CMH' and 'UT1'. Specific things to notice in this example:

  • CMH is overriding the study tag and indicating data from this site may have a study tag of 'STUDY02' instead
  • CMH has an example of a list of regexes for its PDT2 and is overriding qc_type from ExportSettings to ignore this series for QC purposes.
  • CMH has enclosed its RST pattern in a list. In this instance a list isn't needed, but doing so is OK even for a single pattern.
  • CMH is configuring an external XNAT server to pull data from
  • UT1 has configured a non-default sftp server
  • UT1 is overriding the default formats from ExportSettings for its 'OBS' series
  • UT1 is setting its own redcap server
Sites:
  CMH:
    SITE_TAGS: ['STUDY02']
    XNAT_Archive: STUDY1_CMH
    XNAT_source: https://xnat.remoteserver.ca
    XNAT_source_archive: camh_scans
    XNAT_source_credentials: camh_login_creds.txt
    ExportInfo:
      T1:         { Pattern: 'T1w_MPR_vNav$',         Count: 2}
      RST:        { Pattern: ['(?i)Rest'],            Count: 1}
      PDT2:       { Pattern: ['T2.DE','T2DE'],        Count: 1,      qc_type: ignore}
  UT1:
     XNAT_Archive: STUDY1_UT1
     REDCAPAPI: https://ut1redcapserver.ca/redcap/api/
     MRUSER: exampleuser
     MRFOLDER: 'DICOMS/STUDY1'
     FTPSERVER: exampleserver.ca
     FTPPORT: 18777
     MRFTPPASS: my_passfile_name.txt
     ExportInfo:
       T1:         { Pattern: 'T1w_MPR_vNav$',         Count: 1}
       OBS:        { Pattern: 'Observ',                Count: 3,     formats: ['nii']}

Pipeline Config

This section documents any configuration that must be set in the study config for various nightly pipelines to run.

FreeSurfer

Settings for freesurfer must be inside a block that starts with 'freesurfer:'.

Required settings:

  • tags: This should be the tag name assigned to T1 scans, or a list of the tag names if there is more than one type of T1 tag used in this study. This setting is used to locate anatomical scans to be fed into recon-all with the '-i' argument.
    • Example: tags: 'MYT1TAG' or tags: ['MYT1TAG1', 'MYT1TAG2', ..., 'MYT1TAGN']

Optional settings:

  • T2: This provides the tag (or tags) assigned to T2 scans for this study. If this option is set recon-all will use any T2 anatomical scans it finds for a subject in addition to the T1s.
    • Example: T2: 'MYT2TAG' or T2: ['MYT2TAG1', 'MYT2TAG2', ..., 'MYT2TAGN']
  • FLAIR: This provides the tag (or tags) assigned to FLAIR scans for this study. If this option is set recon-all will use any FLAIR scans it finds for a subject in addition to the T1s.
    • Example: FLAIR: 'MYFLAIRTAG' or FLAIR: ['MYFLAIRTAG1', 'MYFLAIRTAG2', ..., 'MYFLAIRTAGN']
  • nu_iter: This should be a dictionary that contains the value to set FreeSurfer's 'nu_iter' to for each site.
    • Example: Assuming sites 'CMH' and 'MRP' are defined nu_iter: {CMH: 4, MRP: 8}

Example

Assuming a study where T1 scans are tagged with T1-Bravo', T2 scans tagged with T2-TSEand FLAIR scans just taggedFLAIR`, and assuming a single site named 'CMH' that wants nu_iter set to 6 the following freesurfer configuration could be used.

freesurfer:
    tags: 'T1-Bravo'
    FLAIR: 'FLAIR'
    T2: 'T2-TSE'
    nu_iter: {CMH: 6}

fMRI pipeline

These settings are used for the epitome / script-it fMRI pipeline. All settings should be inside of a block named 'fmri:' and within this block specific fMRI 'experiment' pipelines are configured. For example, if a study had imitate observe task data and resting state data that study might use something like:

fmri:
    imob:
        <imitate and observe settings here>
    rest:
        <resting state settings here>

The name of each block inside 'fmri:' will then become the name of an output folder nested within the destination path set for 'fmri'.

Required Settings

  • export: This should be set to a list of strings to look for in output file names to verify that the pipeline has run correctly.
    • Example: If the Datman assets script task.sh is run this might be set to ['filtered', 'scaled', 'T1', 'MNI-nonlin', 'volsmooth'], in which case dm_proc_fmri.py will look for files in the output directory containing each of these strings and consider outputs incomplete if it does not find a match for all items in the list.
  • pipeline: This specifies one of the script-it scripts stored in Datman's assets folder to run. Currently accepted options are: 'task.sh', 'rest.sh', 'rest-sprl.sh'
    • Example: If task data (e.g. imitate observe) is being configured pipeline: 'task.sh'
  • tags: This should contain the tag (or a list of tags) to search for when locating input files.
    • Example: If configuring an imitate-observe pipeline with scans tagged 'IMI' and 'OBS' respectively, settings would be tags: ['IMI', 'OBS']. Or for a resting state pipeline with only the 'RST' tag, tags: 'RST'
  • del: Sets the number of TRs to remove from the beginning of each run.
    • Example: del: 4 will delete the first 4 TRs
  • tr: Length of the TRs in seconds (Decimals allowed)
    • Example: tr: 2
  • dims: Isotropic voxel dimensions of MNI space data
    • Example: dims: 3
  • conn: This specifies a list of tags used to identify connectivity files. This setting is read by dm_proc_rest.py. Usually just set it to MNI-nonlin.
    • Example: conn: ['MNI-nonlin']
  • glm: This specifies a list of tags used to identify input files for dm_proc_ea.py and dm_proc_imob.py. Usually just set it to volsmooth
    • Example: glm: ['volsmooth']

Example

Assuming that the task pipeline needed to be configured for empathic accuracy files with a tag of 'EMP' and the rest pipeline for resting state data with a tag of 'RST' the following settings might be used:

fmri:
    ea:
        export: ['filtered', 'scaled', 'T1', 'MNI-nonlin', 'volsmooth']
        pipeline: 'task.sh'
        tags: 'EMP'
        del: 4
        tr: 2
        dims: 3
        conn: ['MNI-nonlin']
        glm: ['volsmooth']
    rest:
        export: ['lowpass', 'T1', 'MNI-nonlin', 'volsmooth']
        pipeline: 'rest.sh'
        tags: 'RST'
        del: 4
        tr: 2
        dims: 3
        conn: ['MNI-nonlin']

Unring

Settings for the Unring pipeline must be inside a block that starts with unring:

Required Settings

  • tags: This should contain a list of scan tags to search for in the nrrd folder. Any scan that matches one of these tags will be run through Unring.

Example

Assuming only scans tagged 'DTI60-1000' needed to run through the pipeline, the following could be used:

unring:
    tags: ['DTI60-1000']

Overriding Paths for a Study

If a 'Paths' block is declared inside a study config file it will completely replace the site wide defaults, allowing you to restructure individual studies. Unfortunately, even if only one or two paths need to be changed the other paths will still need to be declared here too since the 'Paths' set here will completely hide the 'Paths' for the site wide config.

For example, if you only wanted to change the location of the nifti folder for a study adding this to your study config does not work:

Paths:
  nii: some/other/place

Because Datman will no longer be able to see the settings for the other expected paths. To fix this, just copy the other path settings into the study's 'Path' block so it can find them again.

Example Study Template

Below is a rough template that could be copied and modified when making new study config files. Not all keys / features have been used here. See the respective sections on this page for more info.

## Metadata here
MRUSER : <sftp username here>                          
MRFOLDER : <sftp folder name / relative path here>               
PROJECTDIR: <project folder name here> 
STUDY_TAG: <datman style study tag here>
FullName: <full study name here>
Description: <study description here>
PrimaryContact: <PI / primary contact name here>

## Site(s) configured here
Sites:
    <site name 1>:
        XNAT_Archive: <XNAT Project ID here>
        ExportInfo:
            <tag>:       { Pattern: [<series descr pattern here>], Count: <expected num here>]

## Pipelines configured here
freesurfer:
    tags: <T1 tag here>

fmri: 
    rest:
        export: ['lowpass', 'T1', 'MNI-nonlin', 'volsmooth']
        pipeline: 'rest.sh'
        tags: 'RST'
        del: 4
        tr: 2.5
        dims: 3
        conn: ['MNI-nonlin']
unring:
    tags: [<file tags here>]
Add a custom footer

Contents

Clone this wiki locally