I’ve been saying for a while that the best way out of the Web video codec mess is formal standardization of a royalty-free video codec and that formal standards groups like MPEG and others should step up to the task.

Of course, I mean a real, bona-fide standardization process, not a dubious rubber-stamping “ratification” nor a half-hearted kick-the-can-down-the-road affair.

Below is a draft personal response to a public call for comments by the ISO MPEG committee on issues related to standardizing a royalty-free video codec.

—————————————————————————————————————————————————————————————–

(DRAFT) Response to MPEG Request for Comments
on Option-1 (Royalty-Free) Video Coding

At its October 2010 meeting, MPEG requested public comment on industry needs and performance targets for an Option-1 (royalty-free) video codec standard [1].

MPEG should enlarge its portfolio of standards by offering some that are expected to be royalty free [2], taking into account the following points:

Google has proposed ratification of its video codec VP8 as mandatory by IETF, and has stated it will remove h.264 support from Chrome:

In November, 2010, Google filed a proposed Internet-Draft, standards track document for real-time communication over the Web that states:

“In video, the VP8 codec [vp8] MUST be supported..” [3]

As to removing support of h.264 from Chrome, Google stated in January, 2011:

“Specifically, we are supporting the WebM (VP8) and Theora video codecs, and will consider adding support for other high-quality open codecs in the future. Though H.264 plays an important role in video, as our goal is to enable open innovation, support for the codec will be removed and our resources directed towards completely open codec technologies.” [4]

In discussing ratification of VP8 as a mandatory-to-implement specification by IETF even without a formal standardization process, the author of the Google proposed Internet-Draft has opined that making VP8 mandatory to implement is reasonable, and if IETF does not address the issue, proponents “may have to start up a third effort in some forum that’s willing to define the profile needed for interoperability” [5].

The Internet and World Wide Web are fundamentally based on royalty-free standards [6] and therefore need and expect a royalty-free video codec standard [7].

Tim Berners-Lee in an article in the Scientific American in November, 2010 restated the often-stated point that the Web’s richness, diverse user base and different (free and pay) business models require that royalty-free standards are and must be the foundation of the Web:

“The basic Web technologies that individuals and companies need to develop powerful services must be available for free, with no royalties.

… Open, royalty-free standards that are easy to use create the diverse richness of Web sites, from the big names such as Amazon, Craigslist and Wikipedia to obscure blogs written by adult hobbyists and to homegrown videos posted by teenagers.

… Openness also means you can build your own Web site or company without anyone’s approval. When the Web began, I did not have to obtain permission or pay royalties to use the Internet’s own open standards, such as the well-known transmission control protocol (TCP) and Internet protocol (IP). Similarly, the Web Consortium’s royalty-free patent policy says that the companies, universities and individuals who contribute to the development of a standard must agree they will not charge royalties to anyone who may use the standard.

… Open, royalty-free standards do not mean that a company or individual cannot devise a blog or photo-sharing program and charge you to use it. They can. And you might want to pay for it if you think it is “better” than others. The point is that open standards allow for many options, free and not.” [8]

The World Wide Web Consortium recently stated in the context of the much-watched deliberation about a royalty-free codec for the upcoming HTML5 specification:

“The W3C HTML Working Group has not identified a Royalty-Free video codec or container format that would satisfy all parties. … W3C is still highly interested in finding a solution in this space.”[9]

Looking forward, trends like cloud computing, the “Internet of devices”, and the many business models and usage scenarios thriving on the Internet all point to the continuing necessity of placing royalty-free, rather than royalty-bearing, standards at the foundation of the open Internet.

MPEG (WG11 of ISO SC29) has the competence and responsibility to standardize an Option-1 (royalty-free) video codec.

With a 20+ year history, MPEG, now has a portfolio of standards and technologies at or approaching patent expiration from which to assemble a royalty free standard. [10]

MPEG has deliberated at multiple meetings and a quorum of SC29 national bodies have expressed support [11].

The ISO/IEC/ITU patent policy provides a process framework [12], and related bodies like IETF are already moving forward with royalty-free codec standardization [13].

Recent changes in h.264 licensing have been rejected by key Internet industry leadership as inadequate to meet the need for a fully royalty-free standard [14].

As three of many examples, Web browser vendors Mozilla [15], Opera [16], and Google [4] have expressed that these new license terms from MPEG LA are unacceptable to them.

MPEG’s well-established methodology of defining a test model and evaluating improvements to it is the appropriate approach for Option-1 standardization.

MPEG has a long-established and successful work method of defining and then improving a test plan based on pre-defined, rigorous methodology [17].  This test model approach is ideally suited for Option-1, royalty-free standardization, which requires a two-step process of first defining a generic video test model of a known royalty-free foundation, then only allowing additions and modifications of known IPR source and licensing.

It would be unwise, counterproductive, and against long-established MPEG practice for MPEG to publicly set only a “hard” performance target like “a 2x coding gain, in comparison to MPEG-1” for an Option-1 (royalty-free) codec.

A narrow measure of video codec performance is a metric such as image quality (like PSNR) against bandwidth, but the practical reality is that all viable video codec standards activities must and do incorporate trade-offs of multiple features, application profiles and requirements, and platform constraints.

For example, MPEG-2 wisely did not set a hard performance target over MPEG-1; rather, MPEG-2’s core rationale [18] was specifically aimed at adding features like interlace to make the overall standard more generically useful and supportive of specific application profiles:

Rationale

…The general aim of MPEG-2 is to support a broad number of features and operating points which were not the focus of MPEG-1, and in so doing to establish an essentially generic, i.e. application independent standard, which will support a small number of key application profiles.”

Performance-related requirements were only specified as “optimized” image quality, and balanced against multiple parameters like multi-resolution scalability, transmission and storage channel-coding and error-recovery schemes, and  low coding-decoding delay:

“Requirements

MPEG-2 Video extends the capabilities of MPEG-1 with efficient methods to encode interlaced video formats. MPEG-2 Video key requirements include: optimized image quality in ranges from about 3 to 15 Mbit/s; support for various interlaced (as well as progressive) video formats; provision for multi-resolution bit-stream and decoder scalability; random accessibility to support efficient channel-hopping and editability; compatibility with both MPEG-1 and the CCITT H.261 recommendation for video telecommunications; adaptability to various transmission and storage channel-coding and error-recovery schemes; provision for low coding-decoding delay.

And the workplan required only an evaluation process to identify independently confirmable, appreciable improvements in picture quality to make improvements in a short time:

“ Workplan

… A video test model has been established, and several key modules in its algorithmic block diagram are subject to improvement up until March 1993.  For a proposed alternative to be selected to replace the current method, two independent experts must confirm experimentally that the proposed method yields appreciably improved picture quality.  This methodology permits MPEG’s dozens of experts from the world’s top video coding laboratories to make tremendous improvements to the state-of-the-art over a remarkably short time”

The 2001 ISO/ITU Terms of Reference for AVC and h.264 [19] did set a 50% performance target as one of 9 requirements in Annex 1, but it separately required a royalty-free baseline for which the priority requirement was to be its demonstrably royalty free IPR.  And the overall aim was limited to “offer the best possible technical performance under the practical constraints of being implementable on various platforms and for various applications enabled by the relevant ITU-T Recommendations and ISO/IEC International Standards”

Similarly, the High-Performance Video Coding activity has wisely acknowledged that optimizing performance over some ranges may produce worse performance over others [20]:

“3 Requirements

3.1 Compression Performance

A substantially greater bitrate reduction over MPEG-4 AVC High Profile is required for the target application(s); at no point of the entire bitrate range shall HVC be worse than existing standard(s).”

In sum, while performance efficiency is undoubtedly an important requirement of any viable video codec standard, requirements must include and weigh trade-offs of multiple factors.  As described above, MPEG’s long-established methodology of first establishing a test model (in this case an Option-1/royalty-free test model) and then evaluating (Option-1) improvements to it, within the constraints of multiple application and platform requirements, is the appropriate requirements methodology.

References

[1]     “Resolutions of the 94th Meeting”, ISO/IEC JTC 1/SC29 N11553, Guangzhou, CN, October 2010, http://www.itscj.ipsj.or.jp/sc29/open/29view/29n11604c.htm:

“MPEG requests that companies comment on the following topics relating to Option-1 licensable video coding:

1. The relevance of pursuing such a standards activity within MPEG, particularly with respect to current market conditions and industry needs.

2. What are the specific video codec performance targets that may be required in order to secure the desired level of market adoption? As an example, current discussions related to an Option-1 codec have considered a 2x coding gain, in comparison to MPEG-1, as a minimum performance target.”

[2]     Leonardo Chiariglione, The missed award speech, May 2008, http://www.chiariglione.org/leonardo/publications/epo2008/index.asp:

“I believe MPEG should enlarge its portfolio of standards by offering some that are expected to be royalty free and typically less performing and with less functionality next to those that are state of the art, more performing and with more functionality.

… “[F]air – reasonable – non discriminatory” used to be meaningful words when standards were designed for the needs of one industry whose members generally shared the business model according to which the standard would be used.

… I fear that the virtuous circle … whereby the reward from innovation is used to create more innovation may be coming to an end.

… [T]he problem is not “how many cents, tens of cent or euros of licensing fee is fair and reasonable” but “how can licensing be fair and reasonable without specifying a business model”.”

[3]     IETF Internet Draft, Standards Track, “Overview: Real Time Protocols for Brower-based Applications”, H. Alvestrand (Google), November 11, 2010, http://tools.ietf.org/html/draft-alvestrand-dispatch-rtcweb-protocols-00.

Section 5, Data formats, states:

“This document specifies a minimum baseline that will be supported by all implementations of this specification, and leaves further codecs to be included at the will of the implementor.

… In video, the VP8 codec [vp8] MUST be supported.” …

[4]     “HTML Video Codec Support in Chrome”, Mike Jazayeri, Google Product Manager, January 11, 2011, http://blog.chromium.org/2011/01/html-video-codec-support-in-chrome.html

[5]     January 13, 2011, Re: [dispatch] Charter proposal: The activity hitherto known as “RTC-WEB at IETF”, http://www.ietf.org/mail-archive/web/dispatch/current/msg03119.html:

“My personal thinking is that a mandatory codec is reasonable for the profile document that I tried to create in draft-alvestrand-dispatch-rtcweb-protocols, while it is not reasonable for the protocol document of draft-alvestrand-dispatch-rtcweb-datagram. But if the IETF decides that it doesn’t want to address this issue … we (the people who want interoperability of RTC-Web applications) may have to start up a third effort in some forum that’s willing to define the profile needed for interoperability”

[6]     See, for example, W3C Patent Policy, http://www.w3.org/Consortium/Patent-Policy-20040205/:

“In order to promote the widest adoption of Web standards, W3C seeks to issue Recommendations that can be implemented on a Royalty-Free (RF) basis. Subject to the conditions of this policy, W3C will not approve a Recommendation if it is aware that Essential Claims exist which are not available on Royalty-Free terms.”

[7]     For example, the Open Video Alliance, founded in 2009 by Mozilla, Kaltura, Participatory Culture Foundation, and Yale ISP, has issued five “Principles for an Open Video Ecosystem”, which clearly indicate that royalty-free open standards are the only acceptable approach for open web video standards:

“Open Standards for Video — Video standards (formats, codecs, metadata, etc.) should be open, interoperable, and royalty free”

http://openvideoalliance.org/wiki/index.php?title=Some_principles_for_open_video

[8]     Tim Berners-Lee, “Long Live the Web: A Call for Continued Open Standards and Neutrality”, Scientific American, November 22, 2010,  http://www.scientificamerican.com/article.cfm?id=long-live-the-web

[9]     W3C HTML5 FAQ, http://www.w3.org/html/wiki/FAQs, last modified 23 November 2010:

Does HTML5 provide for Royalty-Free video and audio codecs?

Question: Since no video codec and container format have been specified yet, CPs and service providers have to prepare multiple versions of same video contents for browsers supporting different codecs and container formats with HTML5. Therefore, it would be nice to specify (mandatorily) supported codec(s) and container format(s). When do you estimate this can be done? Or is it possible that this can be done at all?

The W3C HTML Working Group has not identified a Royalty-Free video codec or container format that would satisfy all parties. There are various requirements to consider, including the W3C Royalty-Free licensing commitments and various open source projects (Mozilla, Webkit). W3C is still highly interested in finding a solution in this space. At the moment, two video codecs seem to cover all major Web browsers.

See also Jeremy Kirk, “Browser vendor squabbles cause W3C to scrap codec requirement”, Infoworld, July 2, 2009 http://infoworld.com/d/developer-world/browser-vendor-squabbles-cause-w3c-scrap-codec-requirement-974

[10]   See MPEG 20th Year Anniversary Commemoration, Tokyo, November 8, 2008, http://www.itscj.ipsj.or.jp/forum/forum2008MPEG20.html;       Hiroshi Yasuda, “MPEG Birth to Practical Use”, November 8, 2008, http://www.itscj.ipsj.or.jp/forum/forum2008MPEG20/01MPEG20_Yasuda.pdf; Cliff Reader, “Video Coding IPR Issues”, http://www.avs.org.cn/avsdoc/2003-7-30/Cliff.pdf; Cliff Reader, “History of MPEG Video Compression–Ver. 4.0,” 99 pp., document marked Dec. 16, 2003.

[11]    Resolutions, the 92nd SC 29/WG 11 Meeting, 2010-04-19/23, Dresden, Germany, SC 29/WG 11 N 11241, http://www.itscj.ipsj.or.jp/sc29/open/29view/29n11185c.htm:

“Given that there is a desire for using royalty free video coding technologies for some applications such as video distribution over the Internet, MPEG wishes to enquire of National Bodies about their willingness to commit to active participation (as defined by Section 6.2.1.4 of the JTC1 directives) in developing a Type-1 video coding standard.”

Resolutions, the 93rd SC 29/WG 11 Meeting, 2010-07-26/30, Geneva, Switzerland  [SC 29/WG 11 N 11356], http://www.itscj.ipsj.or.jp/sc29/open/29view/29n11382c.htm:

“The Requirements group requests National Bodies and experts to provide contributions on the draft Option 1 Licensing Video Coding documents”

[12]   See Part 1, section 3 of “Guidelines for Implementation of the Common Patent Policy for ITU-T/ITU-R/ISO/IEC” (1 March 2007). The guidelines for the common patent policy encourage disclosure as early as possible by participants and third parties of any known patents or applications:

“[A]ny party participating in the work of the Organizations should, from the outset, draw their attention to any known patent or to any known pending patent application, either their own or of other organizations.

In this context, the words “from the outset” imply that such information should be disclosed as early as possible during the development of the Recommendation | Deliverable.

…  In addition to the above, any party not participating in Technical Bodies may draw  the attention of the Organizations to any known Patent, either their own and/or of any third-party.

[13]   http://tools.ietf.org/wg/codec/

[14]   See for example, “Think H.264 is Now Royalty-Free?  Think Again – and the ‘Open Source’ Defense is No Defense to MPEG-LA”, Peter Csathy, CEO Sorenson Media, Sept. 20, 2010,  http://blog.sorensonmedia.com/2010/09/think-h-264-is-now-royalty-free-think-again-and-the-open-source-defense-is-no-defense-to-mpeg-la/

It appears that many may  have been initially misinformed that MPEG-4 AVC was to become entirely royalty free, as highlighted by Peter Csathy, CEO of encoder vendor Sorenson Media:

“But, you say, MPEG LA recently announced that it will no longer charge royalties for the use of H.264. Yes, it’s true – MPEG LA recently bowed to mounting pressure from, and press surrounding, WebM and announced something that kind of sounds that way. But, I caution you to read the not-too-fine print. H.264 is royalty-free only in one limited case – for Internet video that is delivered free to end users. Read again: for (1) Internet delivery that is (2) delivered free to end users. In the words of MPEG LA’s own press release, “Products and services other than [those] continue to be royalty-bearing.”

This inspires speculation that there may be underlying and less charitable motives in play:

“MPEG LA, in fact, continues to make “noises” that even Google’s royalty free WebM gift to the world (which resulted from its acquisition of On2 and its VP8 video codec) infringes the rights of its patent holders.”

[15]   “Mozilla shrugs off ‘forever free’ H.264 codec license: Uh, will H.264 even be relevant in 4 years?”, Cade Metz, August 26, 2010, http://www.zdnet.com/blog/hardware/mozilla-unmoved-by-royalty-free-h264/9499

[16]   “Opera still won’t support H.264 video”, Jan Vermeulen, October 1, 2010, http://mybroadband.co.za/news/internet/15547-Opera-still-wont-support-H264-video.html

[17]   See for example, MPEG2 Workplan, http://mpeg.chiariglione.org/meetings/london/london_press.htm:

“For the Video work, a disciplined methodology has been established to enable experimentation on various modules of the video encoding and decoding system to proceed in parallel.  A video test model has been established, and several key modules in its algorithmic block diagram are subject to improvement up until March 1993.  For a proposed alternative to be selected to replace the current method, two independent experts must confirm experimentally that the proposed method yields appreciably improved picture quality.  This methodology permits MPEG’s dozens of experts from the world’s top video coding laboratories to make tremendous improvements to the state-of-the-art over a remarkably short time.”

[18]   MPEG-1 rationale, requirements and workplan are publicly described in MPEG Press Release, 20th  Meeting, Nov. 6, 1992, http://mpeg.chiariglione.org/meetings/london/london_press.htm

[19]   “Terms of Reference for a Joint Project between ITU-T Q.6/SG16 and ISO/IEC JTC 1/SC 29/WG11 for the Development of new Video Coding Recommendation and International Standard”, ISO/IEC JTC 1/SC 29/WG 11 N4400, December 2001,  http://www.itscj.ipsj.or.jp/sc29/29w12911jvt.pdf

[20]   “Vision, Applications and Requirements for High-Performance Video Coding (HVC)”, ISO/IEC JTC1/SC29/WG11/N11096, January 2010, Kyoto, JP, http://mpeg.chiariglione.org/working_documents/explorations/hvc/HVC_VR.zip