Having had so much interest from our work capturing video footage for virtual reality in the operating theatre, we thought it would be a good idea to share some of our research with the community and explore some of the different options which we have considered.
Types of Camera
There are increasingly more and more ways to capture video for virtual reality, all of which have their pros and cons. Below are a few of the camera setups we have considered, along with each cameras use case.
360 Mirror Rig
One of the earliest ways to capture 360 degree video was with a mirror rig. In this setup, the camera points directly at a convex mirror, which reflects the scene around it. Because there are no additional cameras or angles to stitch together, you can ‘unwrap’ the resulting image into a stitch error-free 360 video. The problem with this approach however is that in order to ‘unwrap’ the mirror image, you have to distort the video footage, effectively stretching and tearing the video pixels apart, which results in a massive loss of quality. Even when using a camera with a large sensor and resolution size such as a RED Epic, the loss of quality is too severe when designing footage for the Oculus Rift, particularly when dealing with such a sensitive use case such as surgery, where every detail matters.
The alternative to this is to use more cameras.
We captured our first demonstration using a 3D printed rig, housing several commercially available GoPro action cameras. After capturing the operation, the footage from each of the cameras is stitched together into a 360 degree, spherical video. Whilst it may seem like GoPro’s are a somewhat strange choice for creating Virtual Reality content, they are actually rather suited. Because of their very small size, each camera can be relatively close to the next camera, which means that the difference in perspective or ‘parallax’ will be less between cameras, creating a much cleaner end result without any glaring ‘stitch errors’. Not only are the cameras small, they also support high frame rates & ‘better than HD’ resolution, vital for viewing comfortably in virtual reality.
However, this setup does have its limitations. The first of which being that the footage captured is only in 2D, lacking the ability to perceive depth. In order to record 360 degree video in 3D, a slightly different approach is required
GoPro 3D Stereo Array
Whilst there are some GoPro rigs which capture 3D 360, unfortunately, the majority of these rigs just aren’t practical. Many of these 3D 360 camera rigs use what we call ‘stereo pairs’, where they use several pairs of cameras, one camera for each eye. The problem is that when using two pairs of cameras for each eye, you introduce a world of problems when trying to create a seamless image. Stitch errors are bad enough in 2D 360 videos. In 3D, stitch errors verge on painful. As we want to create content that medical students can watch for extended periods, we have decided against using these types of 3D 360 GoPro rigs moving forwards.
3D Depth Cameras
One goal for us would be to create a 3D video that you can walk round in, much like you would in a video game. In order to capture video content like this, we have looked into using RGB + depth cameras such as Microsoft’s Kinect, that use additional cameras to capture depth information in addition to regular video. By using multiple depth cameras positioned around the object, we can capture an object from every angle and merge the footage together to create a genuine 3D video object.
We conduced our first trial of Kinect cameras in the operating theatre last week and will be writing up about our findings in the coming weeks. Below is a clip that was captured from the monitor whilst the video was captured. Whilst the scene captured is showing only depth information, you can see that by moving the mouse, you can effectively move around the scene and see from multiple perspectives.
The potential of using this type of technology to create more life-like experiences is mind blowing. And whilst capturing an entire surgical procedure using this technology might not yet practical, we are committed to continue experimenting with 3D video in preparation for when the technology catches up with our vision.
Google Jump & Jaunt Neo
Earlier this year, Google announced a new camera setup named ‘Project Jump’, which uses 16 GoPros in a circular array. Rather than trying to minimize the difference in parallax by keeping cameras as close to one another as possible like the GoPro array, Google have taken a different approach by deliberately introducing more parallax to the setup by adding more cameras. Although there is more perspective or ‘parallax’ shift across the overall setup, the difference in parallax between each individual camera is actually smaller and is therefore less noticeable from one camera to the next. By deliberately adding global parallax into the camera array, they can use this parallax shift to determine the depth of the scene. After generating a depth map, they can then create 3D or ‘stereo’ views.
Another company creating content with this type of computational photography is Jaunt, who have recently announced a new camera called the Neo (as in, ‘The One’). Last month, Dr Christian Assad, an interventional endovascular cardiologist and entrepreneur working in Santa Clara, was given the opportunity to use Jaunt’s camera to capture 2 clinical cases.
We are tremendously excited to announce that Dr Assad has just joined us at Medical Realities as an advisor and look forward to working alongside him pursuing our vision for global access to surgical health training via virtual reality. We’ll write another post to introduce Christian and the work he has been involved with, however in the meantime, you can read up more about Dr Assad’s work with Jaunt here.
3D camera systems
In another attempt to capture 3D depth, we have trialed several 3D camera setups, including the GoPro DUAL HERO system with wide angle lenses to provide footage which covers approximately 180 degrees field of view.
Due to the lightness of the camera system, it is also possible to mount this camera on the surgeons head using a head strap. However, whilst this gives the ultimate ‘surgeon’s eye perspective’, I challenge anybody to watch footage like this for more than a few minutes without feeling nauseous. Hollywood has known for years about the dangers of putting people off 3D by using fast panning & the negative effects of 3D are only heightened in VR if the content is not created with the viewer in mind. Any significant movements that the camera makes which are not under the control of the user can cause sickness.
We have however found it to be very effective to use 3D cameras to capture a closeup of the surgical field of view to accompany a 360 degree video, rather than replace it.
It’s clear that with the recent resurgence of virtual reality, swarms of camera manufacturers both new and old will join in the race to build the ultimate capture device for virtual reality. Companies such as Giroptic & Richo are battling to produce the best consumer camera whilst new manufacturers like Sphericam are aiming for the prosumer space. These types of cameras will be vital in bringing 360 degree video training to the masses, enabling anybody to capture high quality content without the need to send the footage off to be stitched externally or to spend hours behind a computer. On the opposite end of the spectrum, we have companies like Matterport, StereoLabs & Duo3D who are devleoping their RGB+Depth capture tech and companies like Lytro, who have received $50million to build the ultimate ‘lightfield’ camera for virtual reality.
We’ll write another article on lightfield cameras in the future but for the time being, it’s safe to say the future is very, very exciting.
Interested in using 360 degree video in one of your projects?
The Medical Realities team has extensive experience of capturing video for medical applications in virtual reality. If you have any questions about virtual reality or virtual reality apps for healthcare, call us or email us here. We would love to hear from you.