Tom Goddard
December 16, 2016
To show models in HTC Vive or Oculus Rift virtual reality headsets using ChimeraX first open data, for example, command "open 3103 from emdb" opens a human nuclear pore density map from the Electron Microscopy Databank, start SteamVR (for Oculus headsets you will have to start the Oculus runtime first), then use ChimeraX command
vr on
The open models should then appear in the VR headset. You can move around the models to view from them different perspectives. The HTC Vive hand controllers can be used to move the models by pulling the trigger on either hand controller and moving or rotating the controller. Oculus Touch hand controllers may also work although I have not tested this. To scale the models to larger or smaller size, hold down the trigger button on both controllers and move the controllers closer together (scale to smaller size), or further apart. To turn off rendering in the VR headset and reenable rendering on the desktop display use the command
vr off
If the model is stuttering in the headset caused by not rendering fast enough, then you should not continue viewing it as this can easily and suddenly make you feel nauseous, and this motion sickness may persist for 24 hours after use. Even without stuttering, if you feel at all nauseous stop using the headset. Progression from feeling slightly queasy to needing to throw-up can happen fast.
SteamVR shows the physical room bounds as a blue mesh when either the hand controllers or headset gets close to a physical wall (within about 2-3 feet, or 1 meter). Models can be scaled to sizes larger than the physical room.
Small models only. Large molecular structures (more than a few thousand atoms) render too slowly and will cause stuttering (flickering) in the headset. Also large density map surfaces cause stuttering, a 200**3 map at full resolution should work but larger sizes may require using subsampling (step 2). It is also helpful to switch ChimeraX to simple lighting to increase rendering speed (command "light simple"). Current headsets render at 90 frames per second in two eyes, so the effective rendering speed to avoid flickering is about 200 frames per second. On a conventional desktop display a rendering speed of 10 frames per second is adequate. So the graphics demands are much higher for the virtual reality headset.
Desktop display stops rendering. The ChimeraX graphics on your normal computer display will stop updating when you turn on VR. This is because rendering to the conventional display at slower 60 frames per second slows down the headset rendering, so updating the graphics in the conventional display is disabled. You can show what the VR headset user sees using the SteamVR menu entry "Mirror Display".
Don't use side view. The ChimeraX side view tool will slow down rendering in the headset causing stuttering. The side view should be closed while use the headset.
Windows 10 only. We have tested this with Windows 10 with an Nvidia Geforce GTX 1080 graphics card and also with an AMD Radeon RX 480 graphics card. At this time (Dec 2016) SteamVR is only available on Windows, not Mac OS or Linux.
Difficult to change model display. To change the display style of molecules or maps you can use the ChimeraX user interface on the conventional display with mouse and keyboard. That requires taking off the headset and is very inconvenient. You can also display the desktop in the headset using an HTC Vive hand controller button (the "system" button?). The Vive controller blue laser pointer will appear and can be used to press icons on the ChimeraX user interface to change molecule style (ribbon, spheres, stick, ...). A Vive hand controller side button brings up a virtual keyboard that can be used with the laser pointer to type commands. As of November 2016 pressing the return key or backspace key on the virtual keyboard does nothing. So you can't actually enter a ChimeraX command this way. This is a SteamVR bug discussed online by many people.
Can't select models. There is no ability to select molecules, atoms, residues, or maps.
Clipping when models far away. If you scale the models to very large size, parts more than 500 meters away (in physical room dimensions) will be clipped. This may seem like a large distance, but as you scale to large size it may not look like the model is getting bigger because beyond 10 meters your stereoscopic depth perception is not effective. So as parts of the model get both larger and farther away at the same rate, you don't perceive the size change. To gauge model size it can be useful to move a hand controller near a physical wall so the blue mesh indicating physical room walls will indicate a reference distance.
Dynamic scenes may render too slow. Dynamic (time varying) scenes are more interesting to view than static scenes. We have successfully displayed 3d optical microscopy time series of crawling cells using the vseries command but it requires using the vseries play command caching options so that all surface depictions are precomputed. Otherwise the VR headset rendering is too slow. Also current headsets update at 90 frames per second, so you may want to change the scene at a slower pace, for example using the vseries command maxFrameRate option (updates per second). We have not yet tried molecular dynamics trajectories using the coordset command. There is no option to cache the graphics for trajectories.
ChimeraX can record 360 degree immersive movies that can be played back on virtual reality headsets. Movies have the advantage that complex scenes that cannot render fast enough interactively can be recorded in a movie and played back on even basic cell-phone virtual reality headsets such as the Samsung GearVR. A drawback is that the viewer cannot move around in the scene -- the movie is recorded from one vantage point that may move but is not under the viewer's control. The viewer can look in any direction.
The procedure to record a 360 degree virtual reality movie is to set the ChimeraX camera mode to a 360 degree format, for example using command "camera 360sbs". For stereoscopic movies two eye views are recorded and the image frames for the two eyes are placed side-by-side (360sbs) or left eye on top of right eye (360tb). It is also possible to record monoscopic 360 (camera 360), although it is not possible to directly see depth when the movie is played back in a headset. The Chimera "movie record" and "movie encode" commands are used to record the movie frames and make a single movie file from them (H.264 format). Other ChimeraX commands, such as move and turn are used to change the viewpoint or display style and colors during the movie recording.
Here is a simple example ChimeraX command script flypore.cxc that loads a human nuclear pore electron microscopy map, sets the camera mode, and records a movie flying through the pore.
# Open human nuclear pore density map.
open 3103 from emdb
# Set contour level for density map surface
vol #1 level 2.5
# Move 1500 Angstroms closer before starting movie
move z 1500
# Set stereoscopic side-by-side 360 degree camera mode with eye separation to 25 Angstroms
camera 360sbs eyeSeparation 25
# Start capturing movie frames 4800 by 1200 pixels
movie record size 4800,1200
# Move into pore with 2000 steps each 1 Angstrom
move z 1 2000
wait 2000
# Write H.264 encoded movie
movie encode ~/Desktop/flypore.mp4 framerate 90
Movie file flypore.mp4.
Movie duration. This movie is 2000 frames running at 90 frames per second for a running time of 22 seconds. It is encoded in H.264 format and is 27 Mbytes. This size is about 100 Mbytes per minute or 6 Gbytes per hour but there is very slow motion in this movie so compression is better than typical. The disk space used by the uncompressed (ppm format) image frames before the movie was encoded was about 33 Gbytes, about 1000 times the final movie size -- recording a longer movie requires a lot of disk space. It took about 15 minutes to render the movie (about 30x slower than realtime). It moves 200 nanometers towards the pore in steps of .1 nanometer. (The data coordinates are in Angstroms, motion of 2000 Angstroms, with steps of 1 Angstrom).
Eye separation / physical scale. In recording a stereoscopic (two eye) movie it is important to choose an appropriate eye separation specified in units of the molecular scene (typically Angstroms). This controls how large and close the models will appear in the headset. For the nuclear pore that is 1000 Angstroms in diameter with a central channel of about 500 Angstroms, the above script sets the eye separation to 25 Angstroms. If your eyes physically are two inches apart, this will make the pore channel appear 20 times larger (= 500 / 25) or 40 inches in diameter, a size that your head can comfortably fit through.
Lighting and shadows. Lighting is problematic when recording 360 degree videos. Using simple lighting (command "light simple") provides one directional light and one fill light positioned for looking in the -z axis direction in scene coordinates. Looking at models in the opposite direction will not show any diffuse or specular lighting highlights, instead just flat coloring. The same is true of full lighting. With soft lighting (command "light soft") ambient lighting from all directions with shadowing is used but there is no directional light.
Only default model locations. Stereoscopic 360 camera mode does not work correctly (scene depth is wrong) for models that have been moved within the scene. This is a bug in OpenGL shaders for this camera mode. It does works correctly for camera motions, as in the above script.
Movie resolution and format. It is desirable to at least match the resolution of the headset. Current headsets have 1200 pixels in the vertical dimension. Higher resolution recording can be advantageous because the VR headset distorts the image to correct for lens optics making some image pixels larger. The width of the side by side video should be 4 times the height. The left and right eye images each are an equirectangular projection where longitude angle goes from 0 to 360 degrees along the horizontal axis and lattitude angle goes from -90 to 90 degrees along the vertical axis. To make image pixels have equal angular height and width the movie should have aspect ratio 4:1. The screen size of the ChimeraX graphics does not need to match this aspect ratio. The desired size is specified as a "movie record" command option.
Stereoscopic eye positions. The movie recording include views from two side-by-side camera representing your two eyes. Depending on the direction you look the eyes are positioned differently. Since the movie is recorded without knowing which direction the viewer is looking it renders each vertical strip (meridian) as if you are looking directly at that strip (eye to eye line is perpendicular to view direction). This allows viewing 360 degrees along the equator with good stereoscopic perception (although it is strictly only correct at the central vertical strip in the field of view). The north and south pole directions (where the meridians meet) will not provide good stereoscopic depth perception.
Movie frame rate. It is desirable to record the movie for playback at a frame rate matching the headset, 90 frames per second in current VR headsets. This minimizes the flickering appearance during fast object motions.