3D isn't just for cinemas and flatscreen TVs - it's also coming to smartphones and tablets, and you won't need special glasses. Paul Costigan, chief operating officer and APAC president of 3D chipset company Movidius, talks to Wireless Asia editor John C Tanner about what it takes to enable 3D video in cell phones, and why this time around, 3D will outlast the novelty factor that usually drives it before it fizzles out.
Wireless Asia: Let's start with what Movidius does in the 3D space.
Paul Costigan: To explain that, it's easier to step back and explain how 3D works. One of the objections people have with 3D is fatigue and discomfort, and a lot of that is down to technology. What you're doing in 3D is sending a different image to each eye, and your eyes are used to processing the same image and using depth cues to see it and converging it all. If you present two images, if there's any difference in focus or color or angular rotation, your brain gets seriously confused. So there's techniques like rectification and convergence and other things we have to do.
What do rectification and convergence do?
Rectification basically fixes errors during the capture process. We can't just capture an image like with 2D, there's a hell of a lot of stuff that has to be done first.
Convergence is the process of deciding what you're going to concentrate on when you're looking at an image. One solution for this is called "fixed convergence", where they converge at about 1 to 4 meters. Our solution is different: we decide what the main object is the frame is and focus on that. That means we can offer a 3D viewing experience from 75 cm to infinity. So if you point our camera to trees in the distance, for example, we can still give a depth effect for those distant objects, whereas simpler 3D systems can't do that.
Also, when you mount a camera on a phone, you can have up to seven mounting errors on a lot of different axes: up and down, in and out, and rotational errors. So we have to compensate for those, which eases the job of the manufacturer. We also have to compensate for things like lens angle variance and color balancing.
How does your 3D chipset fit into the device architecture?
We've come up with this multicore processor, and we act as a slave to a host processor. We sit on the memory bus, and we take the cameras and use the memory bus for data exchange between us and the host processor.
Our architecture is software-programmable, so obviously we can allow simple upgrades to track emerging 3D standards or even new safety recommendations and so on. Once we put our chip in, it sits as a co-processor beside the main application processor in a tablet or a smartphone - very simple to integrate. But we can offer new features on that very same chip through different software builds.
Also, our multicore processor addresses a problem you're beginning to see. If you look at the kind of application processors going into cell phones and tablets and so on, they're all based on ARM processors. The ARM is a fantastic general-purpose RISC that's getting faster and bigger, but as you get up to 1 GHz and 2 GHz, it's killing the battery power. It also needs to be supplemented by dedicated hardware in the chip for the graphics, audio and all that.
So our approach is, we still have that central RISC core, but we use our multimedia processors to be the hardware accelerators, and depending on how they're configured, they can take care of video or graphics or audio, whatever. Basically we have a smaller chip and we only need to clock it at 180 MHz, whereas a modern-day application processor is up at the GHz level, so we get a big, big power benefit from that. At 65-nm, 180 MHz is nothing.
What does this mean in terms of BOM for mobile devices?
You have to add two things to enable 3D - one is the 3D barrier on the LCD screen that divides what each eye is sent. The barriers add less than a 2x multiple on the LCDs right now and it's coming down. It'll approach close to the price of a 2D LCD once the volumes are there. The other add is a chip like ours, the 3D processor, which adds maybe an extra 20% to the total BOM, but it's coming down too.
One outstanding issue is brightness - the barrier knocks about 7% off the brightness of the LCD on a touch screen. That's being solved too.
Mobile operators see video as a major app, but also a bandwidth-hungry one. Will 3D add much to their traffic load?
The clever thing there is that 3D technology has been encapsulated within the existing H.264 standard - it's just a normal H.264 transmission that just happens to have the left and right images packed side by side. So the impact on bandwidth isn't as much as you might think. Plus, if you're converting 2D to 3D on the device, then it's even less of an issue. But to be honest, when LTE comes, that won't even be an issue.
Your first product is a 2D-to-3D converter, but 2D-to-3D films have not gotten the rave reactions of native-3D films like Avatar. Will that be less of an issue on smaller screens that don't require 3D glasses?
I think it will, but for a different reason. When you watch video on a handset, it's a true one-to-one experience, and you can constantly adjust the distance and the angle of the screen to give yourself the best viewing experience. It sounds simple when you say it - people don't know they're doing it, but they are. 3D viewing is very angle-dependent, especially for glasses-free 3D.
How do you determine when 3D is achieving that balance?
We can't. It's a totally individual experience. Some people in fact can't see 3D - there's a certain percentage of people who just can't see it. It also depends on the distance between your eyes and all sorts of stuff. That's why we have to make it adjustable, with settings the user can manipulate to get the best viewing experience.
So what metrics or benchmarks do you use to determine that it's ready for the public?
What we aim for in our normal setting is that about 70% of the effect is behind the screen. What you basically have are the parallaxes where your eye is focused. When I look at you now, there's stuff behind you and in front of you, so if the screen is "here", we can adjust the depth of the image so 70% of the items are virtually behind that screen and 30% to the front.
3D has famously had some false starts over the decades. Once the novelty factor wears off, what's going to keep driving demand for it?
Well, it's mostly a video thing, as opposed to still images and things. The other thing is that you're going to see a body of consumers that genuinely expect stuff to be 3D. My kids now expect to see the 3D version of the new movie.
3D has had some false starts, mainly because of the quality, but now is the time for a few reasons. One is that the content is beginning to appear consistently. Another is that the technology genuinely is there to enable comfortable 3D viewing on a mobile platform. And these are active screens, so if people don't want to watch in 3D, they can turn it off and it disappears.
Also, we understand now that there is a trade-off between the "wow" factor and the comfort factor. We can "wow" you, but you'll get tired. So we can make it adjustable. The other thing is that people are also now realizing that 3D doesn't have to mean dinosaurs jumping at you from the screen, and that if you create a depth effect, it's more pleasing and a lot less fatiguing on the eyes. If you've seen Avatar, there were only a couple of dramatic scenes where the stuff came out at you, and most of the 3D effect was depth-based. That was a key milestone in the development of 3D.