pipeline2p module

Module for feeding ThorLabs imaging datasets into a 2p imaging analysis pipeline

(c) 2015 C. Schmidt-Hieber GPLv3

class pipeline2p.ThorExperiment(fn2p, ch2p='A', area2p=None, fnsync=None, fnvr=None, roi_subset='', mc_method='hmmc', detrend=False, nrois_init=200, roi_translate=None, root_path='', ftype='thor', dx=None, dt=None, seg_method='cnmf', maxtime=None, ignore_sync_errors=False)[source]

Helper class to feed ThorLabs imaging datasets into a 2p imaging analysis pipeline

fn2p

str – File path (relative to root_path) leading to directory that contains tiff series

ch2p

str, optional – Channel. Default: “A”

area2p

str, optional – Brain area code (e.g. “CA1”). Default: None

fnsync

str, optional – Thorsync directory name. Default: None

fnvr

str, optional – VR file name trunk. Default: None

roi_subset

str, optional – String appended to roi label to distinguish between roi subsets. Default: “”

mc_method

str, optional – Motion correction method. One of “hmmc”, “dft”, “hmmcres”, “hmmcframe”, “hmmcpx”, “calblitz”. Default: “hmmc”

detrend

bool, optional – Whether to detrend fluorescence traces. Default: False

nrois_init

int, optional – Estimate of the number of ROIs. Default: 200

roi_translate

2-tuple of ints, optional – Apply ROI translation in x and y. Default: None

root_path

str, optional – Root directory leading to fn2p. Default: “”

seg_method

str, optional – One of “thunder” (ROIs are identified by thunder’s ICA), “sima” (ROIs are identified by SIMA’s stICA), “ij” (an ImageJ RoiSet is used), “cnmf” (constrained non-negative matrix factorization). Default: “cnmf”

maxtime

float, optional – Limit data to maxtime. Default: None

ignore_sync_errors

bool, optional – Whether to ignore mismatch between imaging and VR recording file lengths. Default: False

to_haussio(mc=False)[source]

Convert experiment to haussio.HaussIO

Parameters:mc (bool, optional) – Use motion corrected images. Default: False
Returns:dataset – A haussio.HaussIO instance
Return type:haussio.HaussIO
to_sima(mc=False, haussio_data=None)[source]

Convert experiment to sima.ImagingDataset

Parameters:
  • mc (bool, optional) – Use motion corrected images. Default: False
  • haussio_data (haussio.HaussIO, optional) – A pre-existing HaussIO object to save memory. Default: None
Returns:

dataset – A sima.ImagingDataset instance
Return type:

sima.ImagingDataset
pipeline2p.activity_level(data, infer_threshold=0.15, roi_subset='')[source]

Determine the ratio of active over inactive neurons

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • infer_threshold (float, optional) – Activity threshold of spike inference. Default: 0.15
  • roi_subset (str) – Roi subset to be processed
Returns:

level – Number of active and inactive neurons
Return type:

int, int
pipeline2p.contiguous_stationary(speed, speed_time, speed_thr, time_thr)[source]

Find contiguous stationary periods

Parameters:
  • speed (numpy.ndarray) – Running speed
  • speed_time (numpy.ndarray) – Time of running speed
  • speed_thr (float) – Speed threshold
  • time_thr (float) – Maximal resting duration If the resting period is shorter than time_thr, it will be counted as a non-stationary period
Returns:

running_mask – Boolean mask denoting stationary periods
Return type:

numpy.ndarray
pipeline2p.eta(measured, vrdict, evcodelist)[source]

Compute event-triggered average of fluorescence data

Parameters:
  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • vrdict (dict) – Dictionary with processed VR data
  • evcodelist (list of str) – List of event types to trigger on
pipeline2p.extract_rois(signal_label, dataset, rois, data, haussio_data)[source]

Extract fluorescence data from ROIs

Parameters:
  • signal_label (str) – Label of signal to be extracted
  • dataset (sima.Imaging.Dataset) – sima dataset
  • rois (sima.ROI.ROIList) – sima ROIList
  • data (ThorExperiment) – The ThorExperiment to be processed
  • haussio_data (haussio.HaussIO) – haussio.HaussIO instance
Returns:

  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • zproj (numpy.ndarray) – z-projected fluorescence image
pipeline2p.extract_signals(signal_label, rois, data, haussio_data, infer=True)[source]

Extract fluorescence data from ROIs

Parameters:
  • signal_label (str) – Label of signal to be extracted
  • rois (sima.ROI.ROIList) – sima ROIList to be plotted
  • data (ThorExperiment) – The ThorExperiment to be processed
  • haussio_data (haussio.HaussIO) – haussio.HaussIO instance
  • infer (bool, optional) – Perform spike inference. Default: True
Returns:

  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • zproj (numpy.ndarray) – z-projected fluorescence image
  • spikes (numpy.ndarray) – Spike inference for each ROI
pipeline2p.get_rois_cnmf(data, haussio_data, vrdict, speed_thr, time_thr, nrois_init, roi_iceberg=0.9)[source]

Identify ROIs, extract fluorescence and infer spikes using constrained non-negative matrix factorization (CNMF)

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • haussio_data (haussio.HaussIO) – haussio.HaussIO instance
  • vrdict (dict) – Dictionary with processed VR data
  • speed_thr (float) – Speed threshold
  • time_thr (float) – Maximal resting duration If the resting period is shorter than time_thr, it will be counted as a non-stationary period
  • nrois_init (int) – Estimate of the number of ROIs.
  • roi_iceberg (float, optional) – Relative level at which ROI contours will be plotted. Default: 0.9
Returns:

  • rois (sima.ROI.ROIList) – sima ROIList to be plotted
  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • zproj (numpy.ndarray) – z-projected fluorescence image
  • spikes (numpy.ndarray) – Spike inference values
  • vrdict (dict) – Dictionary with processed VR data
pipeline2p.get_rois_ij(data, haussio_data, infer=True)[source]

Extract fluorescence data from ImageJ ROIs

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • haussio_data (haussio.HaussIO) – haussio.HaussIO instance
  • infer (bool, optional) – Perform spike inference. Default: True
Returns:

  • rois (sima.ROI.ROIList) – sima ROIList to be plotted
  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • zproj (numpy.ndarray) – z-projected fluorescence image
  • spikes (numpy.ndarray) – Spike inference values
pipeline2p.get_rois_sima(data, haussio_data, infer=True)[source]

Extract fluorescence data from ROIs that are identified by sima’s stICA

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • haussio_data (haussio.HaussIO) – haussio.HaussIO instance
  • infer (bool, optional) – Perform spike inference. Default: True
Returns:

  • rois (sima.ROI.ROIList) – sima ROIList to be plotted
  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • zproj (numpy.ndarray) – z-projected fluorescence image
  • spikes (numpy.ndarray) – Spike inference values
pipeline2p.get_rois_thunder(data, haussio_data, sc, infer=True, speed=None, nrois_init=100)[source]

Extract fluorescence data from ROIs that are identified by thunder’s ICA. If running speed is available, ROIs will be determined during running periods. Otherwise, MAXFRAMES_ICA at the beginning of the recording will be used.

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • haussio_data (haussio.HaussIO) – haussio.HaussIO instance
  • sc (SparkContext) – Thunder wrapper for a Spark context
  • infer (bool, optional) – Perform spike inference. Default: True
  • speed (numpy.ndarray, optional) – Running speed. Default: None
  • nrois_init (int, optional) – Initial estimate of the number of ROIs
Returns:

  • rois (sima.ROI.ROIList) – sima ROIList to be plotted
  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • zproj (numpy.ndarray) – z-projected fluorescence image
  • spikes (numpy.ndarray) – Spike inference values
pipeline2p.get_vr_maps(data, measured, spikes, vrdict, method)[source]

Read and assemble VR data

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • spikes (numpy.ndarray) – Spike inference values
  • vrdict (dict) – Dictionary with processed VR data
  • method (str) – Method that was used to extract ROIs
Returns:

mapdict – Dictionary with processed VR data. Contains:

“t_2p”: Time points of 2p frames, shape (nt2p)

“DFoF_2p”: Fluorescence for each roi, shape (nrois, nt2p)

“spikes_2p”: Spike inference values for each roi, shape (nrois, nt2p)

“t_vr”: Time points of VR values, shape (ntvr)

“posy_vr”: Position in VR along y, shape (ntvr)

“speed_vr”: Speed in VR, shape (ntvr)

“events”: List of events, each containing the time of the event (event.time) and a 2-character event code (event.code)

“t_ev_matlab”: List of event times for processing with MATLAB

“events_matlab”: Numerical event codes for processing with MATLAB

“fluomap”: Mean Fluorescence values against space, shape (nrois, 2, nbins)

fluomap[:, 0, :] is position along y

fluomap[:, 1, :] is fluorescence along y

“infermap”: Spike inference values against space, shape (nrois, 2, nbins)

infermap[:, 0, :] is position along y

infermap[:, 1, :] is fluorescence along y
Return type:

dict
pipeline2p.infer_spikes(dataset, signal_label, measured)[source]

Perform spike inference

Parameters:
  • dataset (sima.ImagingDataset) – Dataset to be processed
  • signal_label (str) – Label of signal to be processed
Returns:

  • inference (ndarray of float) – The inferred normalized spike count at each time-bin. Values are normalized to the maximium value over all time-bins.
  • fit (ndarray of float) – The inferred denoised fluorescence signal at each time-bin.
  • parameters (dict) – Dictionary with values for ‘sigma’, ‘gamma’, and ‘baseline’.
pipeline2p.norm(sig)[source]

Normalize data to have range [0,1]

Parameters:sig (numpy.ndarray) – Data to be normalized
Returns:norm – Normalized data
Return type:numpy.ndarray
pipeline2p.plot_rois(rois, measured, haussio_data, zproj, data_path, pdf_suffix='', spikes=None, infer_threshold=0.15, region='', mapdict=None, lopass=1.0, plot_events=False, minimaps=None, dpi=1200, selected_rois=None, decoded=None)[source]

Plot ROIs on top of z-projected image, extracted fluorescence, spike inference, fluorescence and spike inference against position (if available)

Parameters:
  • rois (sima.ROI.ROIList) – sima ROIList to be plotted
  • measured (numpy.ndarray) – Processed fluorescence data for each ROI
  • haussio_data (haussio.HaussIO) – haussio.HaussIO instance
  • zproj (numpy.ndarray) – z-projected fluorescence image
  • data_path (str) – Path to data directory
  • pdf_suffix (str, optional) – Suffix appended to pdf. Default: “”
  • spikes (numpy.ndarray, optional) – Spike inference values. Default: None
  • infer_threshold (float, optional) – Spike inference threshold. Default: 0.15
  • region (str, optional) – Brain region. Default: “”
  • mapdict (dict, optional) – Dictionary containing processed VR data. Default: None
  • lopass (float, optional) – Lowpass filter frequency for plotted traces. Default: 1.0
  • plot_events (bool, optional) – Plot events. Default: False
  • selected_rois (list of ints, optional) – Indices of ROIs to be plotted. Default: None (plots all ROIs)
pipeline2p.process_data(data, detrend=False, base_fraction=0.2, zscore=True)[source]

Compute Delta F / F_0 and detrend if required

Parameters:
  • data (numpy.ndarray) – Fluorescence trace, shape: (nrois, nframes)
  • detrend (bool, optional) – Detrend fluorescence traces. Default: False
  • base_fraction (float, optional) – Bottom fraction to be used for F_0 computation. If None, F_0 is set to data.mean(). Default: 0.05
  • zscore (bool, optional) – Use z score instead of mean
Returns:

ret_data – Processed data
Return type:

numpy.ndarray
pipeline2p.thor_extract_roi(data, sc=None, infer=True, infer_threshold=0.15, selected_rois=None, roi_iceberg=0.9, decoded_only=False)[source]

Extract and process fluorescence data from ROIs

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • sc (SparkContext) – A SparkContext
  • infer (bool, optional) – Perform spike inference. Default: True
  • infer_threshold (float, optional) – Activity threshold of spike inference. Default: 0.15
  • nrois_init (int, optional) – Initial estimate of number of ROIs. Default: 200
  • selected_rois (list of ints, optional) – Indices of ROIs to be plotted. Default: None (plots all ROIs)
  • roi_iceberg (float, optional) – Relative level at which CNMF ROI contours will be plotted. Default: 0.9
  • decoded_only (bool, optional) – Only plot spatial decoding. Default: False
pipeline2p.thor_preprocess(data, ffmpeg='ffmpeg', compress=False)[source]

Read in ThorImage dataset, apply motion correction, export motion-corrected tiffs, produce movie of corrected and uncorrected data

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • ffmpeg (str, optional) – Path to ffmpeg binary. Default: movies.FFMPEG global variable
  • compress (boolean, optional) – Compress resulting raw file with xz. Default: False
pipeline2p.xcorr(data, chan, roi_subset1='DG', roi_subset2='CA3', infer_threshold=0.15)[source]

Compute cross correlation between fluorescence extracted from two roi subsets

Parameters:
  • data (ThorExperiment) – The ThorExperiment to be processed
  • chan (str) – Channel character
  • roi_subset1 (str, optional) – Roi subset 1 suffix. Default: “DG”
  • roi_subset2 (str, optional) – Roi subset 2 suffix. Default: “CA3”
  • infer_threshold (float, optional) – Spike inference threshold. Default: 0.15
Returns:

high_xcs – List of roi indices with xcorr values > 0.5
Return type:

list of 2-tuple of ints
class pipeline2p.ThorExperiment(fn2p, ch2p='A', area2p=None, fnsync=None, fnvr=None, roi_subset='', mc_method='hmmc', detrend=False, nrois_init=200, roi_translate=None, root_path='', ftype='thor', dx=None, dt=None, seg_method='cnmf', maxtime=None, ignore_sync_errors=False)[source]

Helper class to feed ThorLabs imaging datasets into a 2p imaging analysis pipeline

fn2p

str – File path (relative to root_path) leading to directory that contains tiff series

ch2p

str, optional – Channel. Default: “A”

area2p

str, optional – Brain area code (e.g. “CA1”). Default: None

fnsync

str, optional – Thorsync directory name. Default: None

fnvr

str, optional – VR file name trunk. Default: None

roi_subset

str, optional – String appended to roi label to distinguish between roi subsets. Default: “”

mc_method

str, optional – Motion correction method. One of “hmmc”, “dft”, “hmmcres”, “hmmcframe”, “hmmcpx”, “calblitz”. Default: “hmmc”

detrend

bool, optional – Whether to detrend fluorescence traces. Default: False

nrois_init

int, optional – Estimate of the number of ROIs. Default: 200

roi_translate

2-tuple of ints, optional – Apply ROI translation in x and y. Default: None

root_path

str, optional – Root directory leading to fn2p. Default: “”

seg_method

str, optional – One of “thunder” (ROIs are identified by thunder’s ICA), “sima” (ROIs are identified by SIMA’s stICA), “ij” (an ImageJ RoiSet is used), “cnmf” (constrained non-negative matrix factorization). Default: “cnmf”

maxtime

float, optional – Limit data to maxtime. Default: None

ignore_sync_errors

bool, optional – Whether to ignore mismatch between imaging and VR recording file lengths. Default: False

to_haussio(mc=False)[source]

Convert experiment to haussio.HaussIO

Parameters:mc (bool, optional) – Use motion corrected images. Default: False
Returns:dataset – A haussio.HaussIO instance
Return type:haussio.HaussIO
to_sima(mc=False, haussio_data=None)[source]

Convert experiment to sima.ImagingDataset

Parameters:
  • mc (bool, optional) – Use motion corrected images. Default: False
  • haussio_data (haussio.HaussIO, optional) – A pre-existing HaussIO object to save memory. Default: None
Returns:

dataset – A sima.ImagingDataset instance
Return type:

sima.ImagingDataset