Segmentation of neurons from fluorescence calcium recordings beyond real time

Fluorescent genetically encoded calcium indicators and two-photon microscopy help understand brain function by generating large-scale in vivo recordings in multiple animal models. Automatic,

fast and accurate active neuron segmentation is critical when processing these videos. Here we developed and characterized a novel method, Shallow U-Net Neuron Segmentation (SUNS), to

quickly and accurately segment active neurons from two-photon fluorescence imaging videos. We used temporal filtering and whitening schemes to extract temporal features associated with

active neurons, and used a compact shallow U-Net to extract spatial features of neurons. Our method was both more accurate and an order of magnitude faster than state-of-the-art techniques

when processing multiple datasets acquired by independent experimental groups; the difference in accuracy was enlarged when processing datasets containing few manually marked ground truths.

We also developed an online version, potentially enabling real-time feedback neuroscience experiments.

The trained network weights, and the optimal hyperparameters can be accessed at https://github.com/YijunBao/SUNS_paper_reproduction/tree/main/paper_reproduction/training%20results. The

output masks of all neuron segmentation algorithms can be accessed at https://github.com/YijunBao/SUNS_paper_reproduction/tree/main/paper_reproduction/output%20masks%20all%20methods. We used

three public datasets to evaluate the performance of SUNS and other neuron segmentation algorithms. We used the videos of ABO dataset from

https://github.com/AllenInstitute/AllenSDK/wiki/Use-the-Allen-Brain-Observatory-%E2%80%93-Visual-Coding-on-AWS, and we used the corresponding manual labels created from our previous work,

https://github.com/soltanianzadeh/STNeuroNet/tree/master/Markings/ABO. We used the Neurofinder dataset from https://github.com/codeneuro/neurofinder, and we used the corresponding manual

labels created from our previous work, https://github.com/soltanianzadeh/STNeuroNet/tree/master/Markings/Neurofinder. We used the videos and manual labels of CaImAn dataset from

https://zenodo.org/record/1659149. A more detailed description of how we used these dataset can be found in the readme of

https://github.com/YijunBao/SUNS_paper_reproduction/tree/main/paper_reproduction.

Code for SUNS can be accessed at https://github.com/YijunBao/Shallow-UNet-Neuron-Segmentation_SUNS51. The version to reproduce the results in this paper can be accessed at

https://github.com/YijunBao/SUNS_paper_reproduction52.

We acknowledge support from the BRAIN Initiative (NIH 1UF1-NS107678, NSF 3332147), the NIH New Innovator Program (1DP2-NS111505), the Beckman Young Investigator Program, the Sloan Fellowship

and the Vallee Young Investigator Program received by Y.G. We acknowledge Z. Zhu for early characterization of the SUNS.

Department of Biomedical Engineering, Duke University, Durham, NC, USA

Yijun Bao, Somayyeh Soltanian-Zadeh, Sina Farsiu & Yiyang Gong

Department of Ophthalmology, Duke University Medical Center, Durham, NC, USA

Department of Neurobiology, Duke University, Durham, NC, USA

Y.G. conceived and designed the project. Y.B. and Y.G. implemented the code for SUNS. Y.B. and S.S.-Z. implemented the code for other algorithms for comparison. Y.B. ran the experiment.

Y.B., S.S.-Z., S.F. and Y.G. analysed the data. Y.B., S.S.-Z., S.F. and Y.G. wrote the paper.

Peer review information Nature Machine Intelligence thanks Xue Han and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We determined the temporal matched filter kernel by averaging calcium transients within a moderate SNR range; these transients likely represent the temporal response to single action

potentials2. a, Example data show all background-subtracted fluorescence calcium transients of all GT neurons in all videos in the ABO 275 μm dataset that showed peak SNR (pSNR) in the

regime 6 < pSNR < 8 (gray). We minimized crosstalk from neighboring neurons by excluding transients during time periods when neighboring neurons also had transients. We normalized all

transients such that their peak values were unity, and then averaged these normalized transients into an averaged spike trace (red). We used the portion of the average spike trace above e–1

(blue dashed line) as the final template kernel. b, When analyzing performance on the ABO 275 μm dataset through ten-fold leave-one-out cross-validation, using the temporal kernel determined

in (a) within our temporal filter scheme achieved significantly higher F1 score than not using a temporal filter or using an unmatched filter (*P