Augmented Reality for Brain Tumour Surgery

Augmented reality (AR) visualization in image-guided neurosurgery (IGNS) allows a surgeon to see rendered preoperative medical datasets (e.g. MRI/CT) from a navigation system merged with the surgical field of view. Combining the real surgical scene with the virtual anatomical models into a comprehensive visualization has the potential of reducing the cognitive burden of the surgeon by removing the need to map preoperative images and surgical plans from the navigation system to the patient. Furthermore, it allows the surgeon to see beyond the visible surface of the patient, directly at the anatomy of interest, which may not be readily visible.

 

ar

Figure: Augmented reality visualizations from our neuronavigation system. The surgeon used AR for craniotomy planning on the skin (A), the bone (B), the dura (C), and also after the craniotomy on the cortex (D). In A, the orange arrow indicated the posterior boundary of the tumour and the blue arrow indicates the planned posterior boundary of the craniotomy that will allow access to the tumour. The yellow arrow shows the medial extent of the tumour, which is also the planned craniotomy margin. In B, the surgeon uses the augmented reality view to trace around the tumour in order to determine the size of the bone flap to be removed. In C, AR is used prior to the opening of the dura and in D the tumour is visualized on the cortex prior to its resection.

Video

Publications

  1. M. Kersten-Oertel, I. J. Gerard , S. Drouin , J. A. Hall , D. L. Collins. “Intraoperative Craniotomy Planning for Brain Tumour Surgery using Augmented Reality”, to be presented at CARS 2016. 
  2. I. J. Gerard, M. Kersten-Oertel, S. Drouin, J. A. Hall, K. Petrecca, D. De Nigris, T. Arbel and D. L. Collins. (2016) “Improving Patient Specific Neurosurgical Models with Intraoperative Ultrasound and Augmented Reality Visualizations in a Neuronavigation Environment,” in 4th Workshop on Clinical Image-based Procedures: Translational Research in Medical Imaging, LNCS 9401, pp. 1–8.*** Best Paper
  3. Kersten-Oertel, M., Gerard, I. J., Drouin, S., Mok, K., Petrecca, K., & Collins, D. L. (2015) Augmented Reality for Brain Tumour Resections. Int J CARS, 10(1):S260.

Image processing pipelines

Large-scale MRI studies of brain development or disease progression require processing of large amount of datasets. Often, acquired in longitudinal fashion, where each subject is followed over period of several years and each dataset contain multiple image modalities.

LP

Our group have developed number of methods to automatically process MRI scans from such projects, taking into account the fact that data from the same subject might be spread across multiple time points. These methods are optimized to increase sensitivity to the longitudinal changes in the subject’s brain and to minimize amount of manual interventions required to completely process datasets. Where the input data consists of the raw mri scans and output is the anatomical measurements that are useful for bio-statisticians.

References

  1. Aubert-Broche B, Fonov VS, García-Lorenzo D, Mouiha A, Guizard N, Coupé P, Eskildsen SF, Collins DL. A new method for structural volume analysis of longitudinal brain MRI data and its application in studying the growth trajectories of anatomical brain structures in childhood. Neuroimage. 2013 Nov 15;82:393-402. doi: 10.1016/j.neuroimage.2013.05.065

Software

Software for longitudinal processing of MRI data is currently only available for collaborators.

Cerebellum and its lobules segmentation

1403365_10151970272361428_660646102_o

The human cerebellum has the highest growth rate of all brain structures during the late fetal and early postnatal life, it plays an integrating role in various neuronal networks, and it shows pathological volume changes in various neuro-psychiatric disorders.

We have developed accurate and fully automatic method to segment cerebellum and it’s lobules using combination of non-linear registration and non-local patch-based label fusion.

References:

  • Katrin Weier, Vladimir Fonov, Karyne Lavoie, Julien Doyon, D Louis Collins. Rapid automatic segmentation of the human cerebellum and its lobules (RASCAL)—Implementation and application of the patch‐based label‐fusion technique with a template library to segment the human cerebellum. Human brain mapping 2014. DOI: 10.1002/hbm.22529
  • Katrin Weier, Vladimir Fonov, Bérengère Aubert-Broche, Douglas L Arnold, Brenda Banwell, D Louis Collins. Impaired growth of the cerebellum in pediatric-onset acquired CNS demyelinating disease. Multiple Sclerosis Journal, 2015 DOI: 10.1177/1352458515615224

Data:

Software:

Brain extraction

beast_picture_crop

Accurate delineation of brain tissue on MRI scan is important step in many image processing pipelines. We have created a method “BEaST: Brain extraction based on nonlocal segmentation technique”, that utilizes library of manually delineated scans for accurate and automatic brain extraction.

References

  1. Eskildsen SF, Coupe P, Fonov V, Manjon JV, Leung KK, Guizard N, Wassef SN, Ostergaard LR, Collins DL; Alzheimer’s Disease Neuroimaging Initiative. BEaST: brain extraction based on nonlocal segmentation technique. Neuroimage. 2012 Feb 1;59(3):2362-73.  DOI: 10.1016/j.neuroimage.2011.09.012

Software

BEaST method is included in the minc-toolkit, available in Software section.

Infant Atlases 0-4.5 years

nihpd_obj2_asym_axial

We present an unbiased magnetic resonance imaging template brain volume for pediatric data from birth to 4.5y age range. These volumes were created using 317 scans from 108 children enrolled in the NIH-funded MRI study of normal brain development (Almli et al., 2007, Evans and Group 2006).

Tools for using these atlases can be found in the Software section.

Publications

The following publications should be referenced when using this atlas:

VS Fonov, AC Evans, RC McKinstry, CR Almli and DL Collins Unbiased nonlinear average age-appropriate brain templates from birth to adulthood NeuroImage, Volume 47, Supplement 1, July 2009, Page S102 Organization for Human Brain Mapping 2009 Annual Meeting, DOI: 10.1016/S1053-8119(09)70884-5

License

Copyright (C) 1993–2004 Vladimir S. Fonov, Louis Collins, McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies. The authors and McGill University make no representations about the suitability of this software for any purpose. It is provided “as is” without express or implied warranty. The authors are not responsible for any data loss, equipment damage, property loss, or injury to subjects or patients resulting from the use or misuse of this software package.

Viewing

To view the atlases online, click on the appropriate JIV2 link in the Download section below.

Online viewing requires Java browser support. The Java Internet Viewer (JIV2) used here is available for download and personal use under the GNU general public license (GPL).

When viewing, the stereotaxic coordinates (X,Y,Z) are displayed in the first row below the volumes. One can use the left most mouse button to click on any image, and the other cross-sectional images will be updated with the appropriate position. You can also hold the middle/rocker mouse button down while moving up or down, to pan through the image plane. Holding ‘Shift’ with the left or middle button will enable dragging and zooming. When looking at the images, remember that left is left and right is right!

Download

You can download templates constructed for different age ranges. For each age range you will get an average T1w, T2w, PDw maps normalized between 0 and 100. Also each age range includes a binary brain mask. NOTE that these templates are smaller then standard MNI-152 template, so if you use them to perform registration in stereotaxic space it will be different coordinate system. It is possible to transform results to MNI-152 space by applying following scaling: 1.21988 in x direction, 1.23510 in y direction and 1.28654 in z direction. Also, you can apply transformation defined in nihpd_asym_44–60_tal.xfm file.

Automated Registration

registration_pic

The interpretation of magnetic resonance (MR) images of the human brain is facilitated when different data sets can be compared by visual inspection of equivalent anatomical planes. Quantitative analysis with pre-defined atlas templates often requires the initial alignment of atlas and image planes.

A completely automatic method has been developed, to register a given volumetric data set to a template. Once the data set is resampled by the transformation recovered by the algorithm, anatomical atlases can be matched to the corresponding voxels of the MRI scan. The use of a standardized stereotaxic space also allows the direct comparison, voxel-to-voxel, of two or more data sets.

References:

  1. DL Collins, P Neelin, TM Peters and AC Evans, Automatic 3D Inter-Subject Registration of MR Volumetric Data in Standardized Talairach Space,Journal of Computer Assisted Tomography, 18(2) p192-205, 1994
  2. DL Collins, CJ Holmes, TM Peters and AC Evans, Automatic 3D model-based neuro-anatomical segmentation Human Brain Mapping, 3(3) p 190-208, 1995

Software:

Automated registration tools are included in minc-toolkit software package, see Software section for details

Pediatric atlases (4.5–18.5y)

nihpd_asym_all_sm

We present an unbiased standard magnetic resonance imaging template brain volume for pediatric data from the 4.5 to 18.5y age range. These volumes were created using data from 324 children enrolled in the NIH-funded MRI study of normal brain development (Almli et al., 2007, Evans and Group 2006). Tools for using these atlases can be found in the Software section.

Viewing

To view the atlases online, click on the appropriate JIV2 link in the Download section below.

Online viewing requires Java browser support. The Java Internet Viewer (JIV2) used here is available for download and personal use under the GNU general public license (GPL).

When viewing, the MNI stereotaxic coordinates (X,Y,Z) are displayed in the first row below the volumes. One can use the left most mouse button to click on any image, and the other cross-sectional images will be updated with the appropriate position. You can also hold the middle/rocker mouse button down while moving up or down, to pan through the image plane. Holding ‘Shift’ with the left or middle button will enable dragging and zooming. When looking at the images, remember that left is left and right is right!

Methods

The pediatric average atlases are comprised of:

Image pre-processing included non-uniform intensity correction (Sled, 1998) and intensity normalization to a range of 0–100. All T1w MRI data was then transformed into the Talairach-like MNI stereotaxic space using minctracc (Collins, Neelin et al. 1994). Brain masking was performed using BET (Smith, 2002). Age-based subgroups of subjects were created, and all scans within each group were then automatically re-registered to the stereotaxic space using the appropriate template. For each group, an iterative nonlinear co-registration algorithm (Grabner, Janke et al. 2006, Fonov, 2010), was applied to obtain the group averages. The T1-based transformation was then applied to the T2, PD and tissue classified volumes to generate average atlases for these data. Methodological details can be found in (Fonov, 2010).

Publications

The following publications should be referenced when using this atlas:

VS Fonov, AC Evans, K Botteron, CR Almli, RC McKinstry, DL Collins and BDCG, Unbiased average age-appropriate atlases for pediatric studies, NeuroImage, In Press, ISSN 1053–8119, DOI:10.1016/j.neuroimage.2010.07.033

VS Fonov, AC Evans, RC McKinstry, CR Almli and DL Collins Unbiased nonlinear average age-appropriate brain templates from birth to adulthood NeuroImage, Volume 47, Supplement 1, July 2009, Page S102 Organization for Human Brain Mapping 2009 Annual Meeting, DOI: 10.1016/S1053-8119(09)70884-5

License

Copyright (C) 1993–2004 Vladimir S. Fonov, Louis Collins, McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies. The authors and McGill University make no representations about the suitability of this software for any purpose. It is provided “as is” without express or implied warranty. The authors are not responsible for any data loss, equipment damage, property loss, or injury to subjects or patients resulting from the use or misuse of this software package.

Comparing different ages

To compare between different paediatric atlases, click here and choose the desired ages. Note that the adult template is also included for reference (18.5–43 y.o): ICBM 152 Nonlinear atlases version 2009.

Download

You can download templates constructed for different age ranges. For each age range you will get an average T1w, T2w, PDw maps normalized between 0 and 100 and tissue probability maps, with values between 0 and 1. Also each age range includes a binary brain mask.

Left-Right Symmetric templates

Asymmetric (natural) templates

Contact

For questions related to the MNI NIHPD atlases (rather than the website), contact Vladimir Fonov

MNI Average Brain (305 MRI)

MNI Average Brain (305 MRI) Stereotaxic Registration Model

mni305_linThis is a version of the MNI Average Brain (an average of 305 T1-weighted MRI scans, linearly transformed to Talairach space) specially adapted for use with the MNI Linear Registration Package (mni_reg).

Methods
In order to overcome the idiosyncrasies of using a single subject brain as a template, in the early 1990s Evans and colleagues introduced the concept of a statistical MRI atlas for brain mapping (Evans et al., 1992a,b, 1993). The MNI305 atlas was constructed in two steps.

First, anatomical landmarks were manually identified in T1-weighted MRI scans from young healthy subjects (239 males, 66 females, age 23.4 +/- 4.1 years). These landmarks were chosen from the Talairach and Tournoux atlas and thus the final aver- age and space approximated Talairach space. Landmarks from each subject were fitted together via least-squares linear regression that matched the resulting AC-PC line to the original Talairach and Tournoux atlas. This yielded a first-pass average T1-weighted MRI volume.

Second, each native MRI volume was automatically mapped to the manually-derived average MRI to reduce the impact of order effects, manual errors and to create a sharper average. The mapping was not performed according to Talairach’s piecewise linear model but used a whole-brain linear (9-parameter) image similarity residual (Collins et al., 1994). The resultant template is thus an approx- imation of the original Talairach space and the Z-coordinate is approximately +3.5 mm relative to the Talairach coordinate. This process resulted in the original MNI305 atlas that has sub- sequently defined the MNI space. Note that, under constraints of linear alignment, residual non-linear anatomical variability across subjects gives rise to a “virtual convolution” (Evans et al., 1993) that somewhat enlarges the template compared with most individual brains.

Publications

  1. Collins, D.L., Neelin, P., Peters, T.M., Evans, A.C., 1994. Automatic 3-D intersubject reg- istration of MR volumetric data in standardized Talairach space. J. Comput. Assist. Tomogr. 18 (2), 192–205.
  2. Evans, A.C., Collins, D.L., Milner, B., 1992a. An MRI-based stereotaxic atlas from 250 young normal subjects. Proc 22nd Annual Symposium, Society for Neuroscience, 18, p. 408.
  3. Evans, A.C., Marrett, S., Neelin, P., Collins, D.L., Worsley, K., Dai, W., Milot, S., Meyer, E., Bub, D., 1992b. Anatomical mapping of functional activation in stereotactic coordi- nate space. NeuroImage 1 (1), 43–63.
  4. Evans, A.C., Collins, D.L., Mills, S.R., Brown, E.D., Kelly, R.L., Peters, T.M., 1993. 3D statis- tical neuroanatomical models from 305 MRI volumes. Proc IEEE-Nuclear Science Symposium and Medical Imaging Conference, pp. 1813–1817.

License
Copyright (C) 1993–2009 Louis Collins, McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies. The authors and McGill University make no representations about the suitability of this software for any purpose. It is provided “as is” without express or implied warranty. The authors are not responsible for any data loss, equipment damage, property loss, or injury to subjects or patients resulting from the use or misuse of this software package.

Download
Download archives containing average t1w model, brain mask and head mask: MINC1 3.6MB MINC2 3.7MB NIFTI 4.8MB

Colin 27 Average Brain 2008

Colin 27 Average Brain, Stereotaxic Registration Model, high-resolution version 2008

mni_colin27_2008

The anatomical phantom is derived from T1, T2, PD-weighted images formed from the average of 27, 11 and 12 scans respectively, of the same normal subject. These volumes are defined at a 0.5mm isotropic voxel grid in Talairach space, with dimensions 362*434*362 (XxYxZ) and start coordinates −90,−126,−72 (x,y,z). A discrete phantom was created by storing the label of the most important fraction class at each voxel location

Publications

  1. Holmes CJ, Hoge R, Collins DL, Woods R, Toga AW, Evans AC. “Enhancement of MR images using registration for signal averaging.” J Comput Assist Tomogr. 1998 Mar-Apr;22(2):324–33. https://dx.doi.org/10.1097/00004728-199803000-00032
  2. B Aubert-Broche, AC Evans, and DL Collins, “A new improved version of the realistic digital brain phantom,” NeuroImage, vol. 32, no. 1, pp. 138–45, 2006. https://www.ncbi.nlm.nih.gov/pubmed/16750398

License
Copyright (C) 1993–2009 Louis Collins, McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies. The authors and McGill University make no representations about the suitability of this software for any purpose. It is provided “as is” without express or implied warranty. The authors are not responsible for any data loss, equipment damage, property loss, or injury to subjects or patients resulting from the use or misuse of this software package.

Download
Download archives containing average t1w, t2w and pdw scan, and discrete tissue classification: 1: Cerebro-spinal fluid, 2: Gray Matter, 3: White Matter, 4: Fat, 5: Muscles, 6: Skin and Muscles, 7: Skull, 9: Fat 2, 10: Dura, 11: Marrow, 12: Vessels
MINC1 222MB MINC2 223MB NIFTI 291MB

Fuzzy segmentation

mni_colin27_2008_fuzzy
Download
Download archives containing 12 volumetric fuzzy volumes that define the spatial distribution for different tissues where voxel intensity is proportional to the fraction of tissue within the voxel (the integral of all tissue components is equal to 1).
MINC1 54MB MINC2 57MB NIFTI 90MB

Colin 27 Average Brain 1998

Stereotaxic Registration Model, original 1998 version

mni_colin27_1998This is a stereotaxic average of 27 T1-weighted MRI scans of the same individual. In 1998, a new atlas with much higher definition than MNI305s was created at the MNI. One individual (CJH) was scanned 27 times and the images linearly registered to create an average with high SNR and structure definition (Holmes et al., 1998). This average was linearly registered to the average 305. Ironically, this dataset was not originally intended for use as a stereotaxic template but as the sub- strate for an ROI parcellation scheme to be used with ANIMAL non-linear spatial normalization (Collins et al., 1995), i.e. it was intended for the purpose of segmentation, NOT stereotaxy. As a single brain atlas, it did not capture anatomical variability and was, to some degree, a reversion to the Talairach approach.

However, the high definition proved too attractive to the community and, after non-linear mapping to fit the MNI305 space, it has been adopted by many groups as a stereotaxic template (e.g., AFNI, Cox,; Brainstorm, Tadel et al., 2011; SPM, Litvak et al., 2011; Fieldtrip, Oostenveld et al., 2011).

Methods

This average dataset was created in a two step process. First, each of the 27 T1-weighted scans were registered to stereotaxic space using the mritotal procedure and resampled onto a 1mm grid in stereotaxic space. All 27 scans were averaged together to create an initial average. This average volume was used as a target for the second phase of registration where each original T1-weighted MRI was re-registered in stereotaxic space. This procedure has the advantage of removing the small variance in intra-subject mapping in stereotaxic space associated with the use of a multi-subject average.

 Publications

Holmes CJ, Hoge R, Collins L, Woods R, Toga AW, Evans AC. “Enhancement of MR images using registration for signal averaging.” J Comput Assist Tomogr. 1998 Mar-Apr;22(2):324–33. https://dx.doi.org/10.1097/00004728-199803000-00032

 License
Copyright (C) 1993–2009 Louis Collins, McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies. The authors and McGill University make no representations about the suitability of this software for any purpose. It is provided “as is” without express or implied warranty. The authors are not responsible for any data loss, equipment damage, property loss, or injury to subjects or patients resulting from the use or misuse of this software package.

Download

Download archives containing average t1w scan, brain mask and head mask: MINC1 13MB MINC2 13MB NIFTI 24MB

YouTube-logo-full_color