ChimeraX Medical Imaging and Virtual Reality Progress Report

Tom Goddard and Elaine Meng
Feb 25, 2019

Here are improvements made to ChimeraX for viewing DICOM medical imaging data on desktop displays and with virtual reality headsets. These new features were added December 15, 2018 to February 28, 2019 under a contract with Medical Science and Computing and the National Institute of Allergy and Infectious Diseases.

(1) Tutorial for viewing DICOM image data in VR. Elaine Meng developed a tutorial covering many of the improvements described below.

(2) ChimeraX DICOM Reference. Contains detailed descriptions of many of the features described below.

(3) Read DICOM files. ChimeraX can open DICOM image series.

• New File / Open DICOM Folder... menu entry. Many DICOM datasets consist of multiple .dcm files under multiple directories often with one 3D image per directory. The Open Folder menu entry searches all subdirectories and opens all images found.
• Menu File / Open... can be used to open individual .dcm files or to select multiple files.
• Types of DICOM data read. ChimeraX can read several types of DICOM files (example data sets) with modalities:
  • CT - computed tomography. Also other 3d image modalities such as MR (magnetic resonance).
  • SEG - segmentations.
  • Time series.
  • RTDOSE - radiotherapy dose distribution.
  • RTSTRUCT - contour lines in planes segmenting structures of interest.
  
• Initial appearance. A single central xy plane is shown for 3D image data. The initial brightness profile has 0, 0.9 an 1.0 brightness levels at -1000, 300, 3000 Hounsfield units with linear ramps between these levels.

(4) Appearance presets. A new volume command option "appearance" allows setting standard brightness and color curves taken from Horos, for example, volume #1 appearance "Airways II".

(5) Windowing and leveling VR mode. Added a new VR hand controller mode and mouse mode to do windowing (changing width of range of data values shown) and leveling (shifting range of data values shown).

(6) Data planes VR mode. Added a new VR hand controller mode and mouse mode to move slice plane through data set. Also allows moving 3-orthogonal planes.

(7) New toolbar buttons for image display. Added seven new image toolbar icons to control the appearance of image data: show single plane, show orthogonal planes, show full data, use axis aligned planes when rendering, use view aligned planes when rendering, airways brightness profile, default brightness profile.

(8) Inspecting image values. Hovering the mouse over a 2D image slice reports the grid index and data value in the 3D image.

(9) VR user interface improvements.

• Active buttons. Made buttons rise when VR pointer hovers over them for improved feedback on when pointer is close enough to push button.
• Hand-controller mode indicators. Added buttons to VR hand controller cones so user can see what modes are currently assigned to each button.
• Multi-person VR user interface. Allow participants in VR session to see the user interface of other participants -- for remotely training people to use the VR user interface.

Colormap on GPU	Colormap on CPU

(10) Better quality image rendering. Render 3d images as a stack of planes perpendicular to the view direction ("view aligned planes"). The older method used a stack of planes aligned with the data axes and switched planes when the data is rotated causing visible jumps in brightness particularly for data where z plane spacing is much larger than xy spacing.

(11) Realtime color, brightness and transparency changes in VR. Allowed mapping image values to colors on the graphics processor (GPU) instead of the CPU for smooth interactive windowing and leveling in VR. Old slower CPU method caused severe flicker in VR. This also improved appearance reducing noise by interpolating the image data values on the GPU. See images comparing GPU coloring to CPU coloring.

(12) Realtime image cropping in VR. Allow keeping all image data on graphics processor so that cropping to show subregions or moving a slab, or moving orthogonal planes through the data is fast and flicker-free in VR. Formerly only the shown part of the image was kept on the GPU.