3D Video & The MVC Standard

http://www.nokiaresearch.com/files/3D_Video.pdf

Three-dimensional video services are predicted to be the next big thing in entertainment and communication industries. Concepts using 3D video in science fiction have fueled the excitement around this topic. With the aid of recent advances in capture, display and compression technologies, 3D video is taking the next step towards becoming a reality in the consumer domain with realistic opportunities.

Applications of 3D Video

One of the most exciting applications that could be enabled by 3D video is telepresence (or virtual reality) - delivering to people the feeling of being immersed in a place other than their actual location. This could enable new ways of collaboration, and increase the efficiency of online meetings.

A similar application is the extension of traditional TV entertainment using displays capable of 3D rendering. The simplest form of 3DTV is a stereoscopic TV, where the display is capable of rendering two views, one for each eye, so the scene is perceived as 3D.

More sophisticated displays are able to render and display multiple views in such a way that the perceived 3D video depends on the user’s location with respect to the TV For example, the user can move his/her head to see what is behind a certain object in a scene while watching TV. This feature brings a new form of interactivity and an immersive feeling to TV viewers that have never been experienced before.

The Joint Video Team of the Video Coding Experts Group of the ITU-T and the Moving Picture Experts Group (MPEG) of ISO/IEC has recently devoted part of its effort to extend the widely deployed H.264/AVC standard to enable these new services and developed the Multiview Video Coding (MVC) extension. The MVC standard has the following features that are useful in terms of enabling many of the 3D video services:

1. Compression of 3D Video

Multiview video sequences are captured by different cameras in different positions and through different angles, they are just different representations of the same scene. Therefore, there exists an additional redundancy in the view direction for 3D video, called view redundancy. The MVC standard utilizes prediction between different views to exploit this redundancy and improve the compression ratio.

2. Scalability and Adaptation

To meet the requirements of the applications exemplified above, scalability and adaptability also need to be supported. For example, in 3DTV scenario advanced displays capable of displaying multiple views would be decoding more views than stereoscopic displays that display only two views. The MVC standard defines efficient ways so that any subset of the views is easily separable from the entire bitstream. As the regular two dimensional TV applications are still widely used, backwards compatibility has been an important target for MVC. The MVC standard achieved backwards compatibility by defining the bitstream so that a compliant H.264/AVC decoder can decode a single 2D view and discard the rest of the data, whereas a compliant MVC decoder can decode all the views and generate the 3D video. Backwards compatibility is also supported by the related communication protocols for transport over the MPEG-2 Transport Stream and the Internet Protocol (IP) – a device capable of receiving an H.264/AVC stream over the MPEG-2 Transport Stream or the Real-Time Transport Protocol over IP is also capable of receiving an MVC stream over these protocols.

3. Computational Complexity

The amount of information that needs to be processed in 3D video is significantly higher than that of regular 2D video. Moreover, the additional dependency between views could potentially become problematic in terms of implementation complexity. In order to enable practical implementations and reduce the complexity of 3D video codecs, the MVC standard utilizes efficient ways to buffer the pictures used for prediction, and enable parallel processing of separate views. In addition, the MVC standard does not change the underlying coding tools used in the H.264/AVC standard, so that the existing hardware accelerators and optimized software implementations that are widely deployed could be reused.

4. Nokia 3D Video Research

4.1 Mobile 3D Video

With recent advances in capture and display technologies, 3D video communication and entertainment services are expected to become reality and redefine the communication and entertainment services with high immersion. To enable these exciting new services, Joint Video Team (JVT) of ITU-T and MPEG has devoted its recent efforts to extend the widely deployed H.264/AVC standard for multi-view video coding (MVC). MVC provides superior network robustness and compression performance for delivering 3D video by e.g. taking advantage of the inter-view dependencies of the different visual channels. In addition, its backwards compatibility with H.264/AVC codecs makes it widely interoperable in environments having both 2D and 3D capable devices.

4.2 Nokia's MVC Software

Nokia Research Center has developed a highly optimized MVC decoder running on Maemo platform, and demonstrated a natural real-time 3D movie playback on a modified Nokia N800 internet tablet with an integrated auto-stereoscopic display. The prototype was showcased at IEEE 3DTV-CON and Nokia World 2008. The source code implementing the 3D Video playback demos mentioned above is available for download now. The Nokia’s MVC software includes two modules:

1. MVC Encoder, which runs in PC and configured to generate MVC compliant bitstreams
2. MVC Decoder is a player, fully optimized for the Maemo platform, and decodes stereoscopic MVC bitstreams.

Download

Nokia Research Center welcomes any feedback on the software, in terms of bug-fixes or additional functionalities. Please send all your questions/comments toh264sw@nokia.com or to project team.

4.3 More Information on MVC

MVC Whitepaper

For more detailed information, please refer to the below standardization documents and academic papers that discuss the MVC standard and 3D Video services in more detail.

Standard Documentation and the Software

- Joint Draft 8.0 on Multiview Video Coding [http://wftp3.itu.int/av-arch/jvt-site/2008_07_Hannover/JVT-AB204.zip]
- WD 4 Reference Software for Multiview Video Coding (MVC) [http://wftp3.itu.int/av-arch/jvt-site/2009_01_Geneva/JVT-AD207.zip]
- RTP Payload Format for MVC Video (draft) [http://tools.ietf.org/html/draft-wang-avt-rtp-mvc-03]
- MVC File Format (draft) [http://wg11.sc29.org/meetings/87_Lausanne/w10444/w10444.zip] (available to MPEG members only)

Detailed Information on the MVC Standard

- Merkle, P.; Muller, K.; Smolic, A.; Wiegand, T., "Efficient Compression of Multi-View Video Exploiting Inter-View Dependencies Based on H.264/MPEG4-AVC,"; Multimedia and Expo, 2006 IEEE International Conference on , vol., no., pp.1717-1720, 9-12 July 2006
- Ying Chen;Ye-Kui Wang; Ugur, K.; Hannuksela, M.M., Lainema J.; Gabbouj M., "3D video services with the emerging MVC standard," EURASIP Journal on Advances in Signal Processing, Volume 2009, Article ID 786015, 13 pages, doi:10.1155/2009/786015.
- Ugur, K.; Hui Liu; Lainema, J.; Gabbouj, M.; Houqiang Li, "Parallel Encoding - Decoding Operation for Multiview Video Coding with High Coding Efficiency," 3DTV Conference, 2007 , vol., no., pp.1-4, 7-9 May 2007
- Ying Chen; Ye-Kui Wang; and Gabbouj, M.; "Buffer requirement analyses for multiview video coding," 26th Picture Coding Symposium (PCS 2007), Nov. 2007, Lisbon, Portugal.
- Smolic, A.; Kauff, P., "Interactive 3-D Video Representation and Coding Technologies," Proceedings of the IEEE , vol.93, no.1, pp.98-110, Jan. 2005
- Vetro, A.; Matusik,W.; Pfister, H; Jun Xin; "Coding Approaches for End-to-End 3D TV Systems," Picture Coding Symposium 2004 (PCS-2004), San Francisco, California, USA, December 2004.

Nokia’s Mobile MVC Prototype

- Willner, K.; Ugur, K.; Salmimaa, M.; Hallapuro, A.; Lainema, J., "Mobile 3D Video Using MVC and N800 Internet Tablet," 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, 2008 , vol., no., pp.69-72, 28-30 May 2008

Improvements on top of MVC

Below are some improvements Nokia Research Center has been working on to improve the performance of MVC even further.

- Ying Chen; Ye-Kui Wang; Hannuksela, M.M.; Gabbouj, M., "Picture-level adaptive filter for asymmetric stereoscopic video," 2008 IEEE International Conference on Image Processing (ICIP 2008), Oct. 2008, San Diego, CA, USA.
- Ying Chen; Shujie Liu; Ye-Kui Wang; Hannuksela, M.M.; Houqiang Li; Gabbouj, M., "Low-complexity asymmetric multiview video coding," Multimedia and Expo, 2008 IEEE International Conference on , vol., no., pp.773-776, June 23 2008-April 26 2008
- Shujie Liu; Ying Chen; Ye-Kui Wang; Gabbouj, M.; Hannuksela, M.M.; Houqiang Li, "Frame loss error concealment for multiview video coding," Circuits and Systems, 2008. ISCAS 2008. IEEE International Symposium on , vol., no., pp.3470-3473, 18-21 May 2008
- Ying Chen; Ye-Kui Wang; Hannuksela, M.M.; Gabbouj, M., "Single-loop decoding for multiview video coding," Multimedia and Expo, 2008 IEEE International Conference on , vol., no., pp.605-608, June 23 2008-April 26 2008

4.3 Project Team:

Antti Hallapuro, Miska Hannuksela, Jani Lainema, Marja Salmimaa, Kemal Ugur, Kai Willner, Ye-Kui Wang, Ying Chen