The road to the promised land.
For more than 6 years, we have been working on and looking forward to a simpler way to build RTC (Real Time Communications) applications on the web. In order for this technology to truly show its value, the major browser vendors needed to show up.
Mobile, mobile, mobile.
Now that Apple has joined the party in earnest, does the technology have the coverage required in order for developers to make good use of WebRTC on mobile devices? Let’s find out.
Until now, in order for WebRTC to work on iOS, we were relegated to wrapping WebRTC code in Objective-C and Swift, in our native iOS apps. Basically, we had to take the Chrome code and build an app that was sent to the app store for approval and wait in line, like all the other chumps (yours truly included). Conversely, on Android we could run much of that same code from our desktop Chrome apps, on the Android device as well, within reason of course.
Now that Safari and Chrome are shipping compatible WebRTC on mobile, we get to reuse the same code, right!? Well, mostly, they are different code bases, after all.
A word about hardware acceleration.
If ubiquitous mobile video is to take off, the battery life of the device has to last more than the length of the 10 minute video call (ok, I am exaggerating a bit, but I think you get the point) and the performance needs to be at least adequate enough to distinguish facial features. My bar is set a little higher, baby steps for now.
Without h/w acceleration the CPU is likely working too hard to encode the local video and decode the inbound video + service the other processes required at the same time. That really means there needs to be hardware onboard the device dedicated to video coding. That in turn means H.264, since there are very few vendors that offer VP8 or VP9 h/w acceleration.
Question: Does this mean that mobile apps written with VP8 will not be able to deliver decent mobile video conferencing?
Answer: No, not at all, but they will likely not be as performant as those taking advantage of hardware acceleration.
Suffice to say that SVC (Scalable Video Coding) usage would be another reason why we need h/w acceleration, but that’s for another day.
Who’s using what?
The majority of desktop and mobile WebRTC apps written today, are using VP8 for video.
Since Apple and Microsoft both use H.264 and Google uses VP8 and H.264 (recently shipped Open H.264 – on the desktop and mobile). Also, many of the Enterprise RTC developers are already on that H.264 bandwagon.
Question: If Apple and Microsoft devices ship with H.264, what is the case with Google Chrome on desktops and android, are they preferencing VP8?
Answer: Chrome for desktop and android now have H.264 native. Many of the Android devices that ship today all have H.264 hardware acceleration onboard. In order to understand which units have H.264 and hardware acceleration, you can run use the Android APIs to pull a list of available codecs, but in the case of WebRTC, you will only get H.264 in Android WebRTC if there is a h/w encoder on the device.
Is H.264 the answer for WebRTC video?
Here is a recent test:
Host 1 – (before joining):
macOS Sierra, Macbook, Safari (Technology Preview 32)
Host 2 (after joining):
Android 7, Samsung 7, Chrome 55
Host 1 (after joining):
According to the Chrome Status page, Chrome for Android should have H.264. So why is the session barfing when trying to set up video? The logs do not lie…
Safari – offer:
Chrome on android – answer:
Err, huh? No H.264 in reply?
So, I updated to latest Chrome on android (58) and tried again…
… et voilà!!
Next topic, paying the man!
Shipping your product with H.264 enabled, means you may potentially need to deal with the MPEG-LA royalty police for H.264 royalties, but there are some grey areas.
In the case of Apple and Microsoft, where H.264 royalties are already being paid for by the parent vendor, the WebRTC developer is riding on the coattails of papa bear, at least in theory.
Cisco’s generous OpenH.264 offer means that those using this binary module, can do so at potentially no cost:
We will not pass on our MPEG-LA licensing costs for this module, and based on the current licensing environment, this will effectively make H.264 free for use on supported platforms.
Q: If I use the source code in my product, and then distribute that product on my own, will Cisco cover the MPEG LA licensing fees which I’d otherwise have to pay?
A: No. Cisco is only covering the licensing fees for its own binary module, and products or projects that utilize it must download it at the time the product or project is installed on the user’s computer or device. Cisco will not be liable for any licensing fees incurred by other parties.
That seems to mean (I am no lawyer) every developer shipping WebRTC apps supporting Open H.264 binary module, get a free ride. Those using some other binary, or shipping the above source code for that module, could be on the hook for those royalties. That said, since there are royalties being paid by parent vendors where devices are shipping H.264 anyways, developers may not get hassled regardless.
So what did we learn here?
- Apple has joined the party, now we have a full complement of browser vendors!
- If you want to leverage WebRTC video to deliver a ubiquitous mobile and desktop experience for your users, you should likely consider including both H.264 and VP8.
- VP8 is (still) free and powers most of the WebRTC video out there today.
- You can make use of the Open H.264 project and get a free H.264 ride, albeit baseline AVC.
- WebRTC on Android does not support software encoding of H.264, so unless there is local hardware acceleration, H.264 will not be in the offer.
- H.264 is not fully enabled (or buggy) in Chrome 55 (I was using it on Samsung S7 Edge (Android 7), but it does work with Chrome 58.
- WebRTC is not DOA!
- SDP still sucks and ORTC can’t come soon enough!!
As a side note, it would be interesting to see something like this open sourced; VP8 / H.264 conversion without transcoding, if only to service the existing desktop apps currently running VP8 <-> mobile H.264. It would likely overwhelm the mobile device, but it would be cool if it worked!
Disclaimer: The views expressed by me are mine alone and do not necessarily represent the views or opinions of my employer.
Update: There is now some healthy conversation in the IETF WG around what “compliant” and “compatible” actually mean. More on this as it unfolds.
We are now in the final throes of a consensus call in the IETF around which video codec should be made mandatory for those building WebRTC apps, services et al, who wish to be considered “WebRTC Compliant”. The codec contenders are VP8 and H.264, in many forms and combinations.
This latest consensus call is for both codecs to be mandated for all WebRTC endpoints, or “dual MTI codec”. I am sure I will catch hell from someone on language but that is the essence of it. As one might expect, there are some that are in favor and some that are against a dual MTI video codec. Those in favor seem motivated to accept this based on the promise of interoperability that might follow and other reasons. As one might expect, we are all quite eager to put this debate to bed so we could get on with other work.
This is not a decision that should be made lightly. Let’s consider the implications. Imposing a dual MTI suggests that every developer that wishes to produce a WebRTC compliant app must implement both codecs.
I find it difficult to agree to mandate a dual MTI codec knowing that there are a great many developers who will not want or will be in a position to implement both codecs. Yes, many WebRTC SDK vendors will support both. Even if both codecs and their transports are provided as part or could be easily added to the application at compile time it doesn’t mean that every developer will want to implement or ship both codecs.
Bottom line is, according this consensus, if developers do not implement/ship both codecs they are not considered WebRTC compliant. To me, this seems like a rather unreasonable expectation. Developers should be able to choose which codec they ship, and not be forced to do 2x the work to become compliant.
I would love to hear from other developers on this. Do you plan on implementing both VP8 and H.264 in your apps?
A few things have happened that lead me to believe that H.264 has already won the WebRTC codec debate, regardless of what was not decided in the IETF RTCWEB or W3C WGs…
– Cisco delivers Open Source H.264 Codec via OpenH264.org
– Google and Cisco demonstrate H.264 interop in Chrome
– WebRTC Source Code for H.264 implementation surfaces in the WebRTC Developer mail list
So it looks like Chrome will end up supporting H.264 = they will support both VP8 & 264
Mozilla / Firefox already said they will support both.
Apple seems to be getting more involved, my guess is that they will announce some level of WebRTC support for Safari at WWDC. If they do, they will also favor H.264.
Since ORTC will be compatible with WebRTC to some degree, IE modern APIs group has indicated that ORTC is “under consideration”. Microsoft also seems to favor H.264.
That covers all the major browsers.
It feels like the MTI Video codec issue is a non-issue.
The IETF RTCWEB WG Chairs have not given up on the MTI video codec issue and they seem determined to see this through. Weather you feel that is a good use of our time or not there will be a vote, which is something generally not done in the IETF.
Jabber remote attendance will not be counted this time around, from WG Chair – Cullen Jennings on the subject…
“If there are any people in that category that wish to vote, could they email the chairs with full name and affiliation.”
Magnus Westerlund <firstname.lastname@example.org>
Cullen Jennings <email@example.com>
Ted Hardie <firstname.lastname@example.org>
Here is the full note from the chairs today…
The WebRTC ecosystem needs to avoid interoperability failure to grow
optimally. The RTCWEB working group took on the task of establish Audio
and Video MTI codecs as part of meeting that need. We have not succeeded
in finishing that task for video using normal IETF process, but it is
We (WG chairs) are proposing that the working group consent to a method
that will establish an MTI, even if the MTI chosen does not have rough
consensus. We would far prefer the normal IETF process, but it is not
proving workable for this selection.
We initially proposed a method from RFC 3929 (external review team), but
now believe that the working group would not consent to that method.
Instead we are proposing a method that leaves the decision in the hands
of the WG.
The method we propose is based on Instant-runoff voting,
http://en.wikipedia.org/wiki/Instant-runoff_voting, with the
understanding that the choice will be the winner according to the
Instant-runoff voting process.
The steps in the proposed process are these (1-5):
1) Establish a final list of alternatives, based on the WG’s input to
Gonzalo’s email on the 13th of November that requires any additions to
provided by end of the 27th of November.
2) Establish those eligible to vote. Any participant in the
working group process either electronically or in-person as of yesterday
(20th of November). Who has participated in the Working group process
will be anyone that can be identified from:
– The Blue Sheets for any RTCWEB WG session during an IETF meeting or
an interim meeting since the WG’s creation.
– posting of at least one email to the RTCWEB mailing list
The voter must at time of voting prove their eligibility, by pointing to
the mail archive or a particular blue sheet/meeting. Please verify your
3) Start the the voting period. Those eligible and willing to vote send
their ballot to a vote collector (Matt Lepinski, former Nomcom chair)
within two weeks using email. The vote collector will check when
receiving a ballot the that the voter is eligible and send a
confirmation email on receiving the ballot. During the balloting period
the vote collector will keep all ballots secret.
– The voter MUST rank ALL alternatives in their ballot from the most
preferred, marked with rank 1, the second most with 2, all the way
to the least preferred marked with rank N.
4) When the voting period is over the ballot collector will publish the
results as well as all ballots, including the voters name to the RTCWEB
WG mailing list. This enables all voting individuals to verify that
their ballot is unmodified. And allows anyone to verify the result of
5) The selection is recorded in the drafts.
— End of Process Proposal —
This message initiates the first step in the working group consensus
call process. Namely a one week comment and discussion period for the
After that week the WG chairs will update, if necessary, the proposal.
Then using the normal IETF process in which anyone is eligible to
participate, the chairs will ask for (rough) consensus to adopt this
extraordinary process to achieve the working group’s stated goals. The
end date for this consensus call is 2-weeks after the announcement of
the consensus call.
If the working group does not consent to using this extraordinary
process, we will hold a consensus call if the WG can accept
“WebRTC entities MUST support at least one of H.264 or VP8.”.
If there is failure to establish consensus even for this statement, the
chairs conclude that the WG can’t establish what to say about a MTI
The WG Chairs
Here are the choices..
The proposed alternatives
The following alternatives has been proposed:
- All entities MUST support H.264
- All entities MUST support VP8
- All entities MUST support both H.264 and VP8
- Browsers MUST support both H.264 and VP8, other entities MUST support at least one of H.264 and VP8
- All entities MUST support at least one of H.264 and VP8
- All entities MUST support H.261
- There is no MTI video codec
- 5+6, i.e. All entities MUST support H.261 and all entities MUST support at least one of H.264 and VP8
- All entities MUST support Theora.
- All entities SHOULD support both H.264 and VP8. All entities MUST at least implement one of those.
Entities that do not support both H.264 and VP8 MUST implement H.261.
The deadline to propose additional alternatives are: 27th of November 2013
This may change so keep an eye on – http://trac.tools.ietf.org/wg/rtcweb/trac/wiki/WikiStart
Chair: Taking the discussion back to the list.
Harold: We are witnessing a massive abuse of IETF policies. We have spent months on this.
Long line at the mic now
NEITHER CODEC WAS SELECTED
VP8 – about 30% of the room
Jabber: 30 for VP8 and 10 for 264
H.264 – about half the room
Chair / Ted Hardie:
Regarding the consensus call, raise your hand for either or both but not neither. This is not a call but simply calrification
Chair / Cullen: Agreed, hugs coming your way.
Martin: Keep in mind there are some that can’t get up and talk about that.
Chair / Cullen: We want to ask if you can’t live with either one of these codecs please get to the mic.
Tim Terryberry: Re: JR we need a royalty bearing codec if webrtc is to take off. I don’t think I need to tell the people in this room how excited everyone is about video in a browser without a plugin, ll of those are VP8.
Justin ? : Risk and Trust. Free codecs sounds good but today that’s not a reality. One of the advantages of going through an SVDO process is that the members must declare IPR, free codecs do not go through that. VP8 is not a free and the wind codec, it’s owned by Google. Google controls all of what VP8 is, can they be trusted. Who has implemented VP8 from the spec, not source.
?? – Technology is for people. The decision is for what we have today, not yesterday. Google has done a great job.
Harold Alvestrand: If people think that this announcement from Cisco of “later to be provided code” then I have real issues with the IETF process. Sooner or later we should find a way forward to rely on un-encumbered patent- less codecs, and if not now, then when?
Eric Rescorla: Mozilla intends to support 264 and VP8. We intend to support 264 via Cisco’s code/plugin on desktop. We will try hard to use the hw codecs since that works better on mobile. The user experience, during the install the codec will be installed (part of the install process). VP8 will be shipped with both mobile and desktop.
Bernard Aboba: It’s ok JR we do love you, hugs later.
Blackberry ?: Like the idea we would have a track towards better quality. The license is a big issue. VP8 is not a specification in the MPEG LA, but since it’s not there we have no mitigation there so we can’t make VP8 the MTI.
Randell Jesup: Those who are not making a browser but instead a server like asterisk or the like, their users would also have to download this plugin codec as well. It’s unclear if this would fall into the personal non-commercial use category. IT is not going to like this either, dealing with plugin codecs will make their lives potentially rather difficult.
JR: Arguing that we must choose a MTI codec or they can’t ship
Robin Raymond: I think it’s great that this new 264 codec is out there but if we make it mandatory the little guys lose. Mobile can’t download a binary, it must be part of the app before deployed. If you need H.264 you can add it if you need it but don’t force that royalty bearing codecs on all of us.
JR: Users don’t need to get plugins, apps do that.
Dean Willis: I don’t build browsers. For the last 7 years I have made my career on IPR consulting. If we were to pick 264 as MTI and since Cisco is not part the MPEG LA there seems to be risk there. Seems to me no choice works as well.
Roman Mahy: The most compelling items were the embedded platforms. My experience it was at one point very hard to find hw accelerated encoding, in the past.
Peter Thatcher: We turned on transcoding for a well known chat service and no one noticed. We are better off allowing the market to decide as Martin indicated months ago.
Justin Uberti: Regarding Adam’s comments regarding someone recording calls does not the understand the current laws. If you want to ship h.264 you can, no one is stopping you but not as the MTI codec.
John Peterson: This Wg has made plenty of decisions re: interop and we have saddled this WG with SIP /OA, but I think we should go with H.264
Roberto ?: The bias seems to be present, that larger companies have an easier time sending people here. If you choose to make 264 the MTI you are cutting out the small guys. I think the best option is to choose neither.
Matt ?: There are plenty of lawsuits against 264 so to say there is no risk with 264 that is simply not true. External plugins are not cool.
Ted Hardie (independent) : I offered JR a hug as he seemed to feel the free 264 effort was not appreciated. One of things we were trying to avoid is a plugin scenario and this (other things were added) is the reason we can’t invoke 264 as the MTI video codec. Royalties will push away numerous developers, big and small. Since 264 is now free (not free from royalty), we can add that codec as a non-mti codec.
JR: Transcoding degrades quality.
Martin Thompson: Let’s stop messing about with semantics, 264 is clearly the correct choice here so let’s get on with it.
Chair / Cullen Response: That was put to the list months ago and no one came forth with a proposal so that is not on the table today.
Jeremy ?: We could also consider both codecs.
Jean ?: Calling Nokia to the mic to clarify why there is a patent claim on VP8 and not on 264.
Peter Thatcher: Seems there are 2 broad categories, H.264 is better and the patent argument. The first is simply perspective and the second the patent being mentioned with Nokia is not resolved and if we are making a decision to implement 264 maybe those making that decision should wait until that is resolved.
Justin Uberti: Interop is misunderstood. Transcoding is cheap so that does not hold in the commercial world. VP8 is shipping in plenty of endpoints. Hardware Codecs are quite a ways off, there is LOTS of works that needs to be done here.
Monty Montgomery: If we go down this 264 path we are staking our future on a binary law environment.
Harold: To Cisco, please publish your findings
JR / Cisco Conclusion
JR Arguing Risk Analysis
JR Arguing Risk Assessment
JR Arguing Distribution
JR arguing the results in Germany (re: Nokia VP8 patent being put down in Germany) may or may not be a consideration
JR arguing that VP8 is a target for patent trolls
JR arguing patent risk
JR arguing both codecs perform nearly the same so let’s get past that
Justin asking, which of these represent (slide below) actual Real-time encoode/decode
Cisco response: much smaller percentage
H.264 vs VP8 Deployment & Standard
H.264 Shipping Date
Jonathan Rosenberg on H.264 for Cisco is at the mic
—- that was fast, thanks Harold! —-
Harold is at the mic..
NOTE: This post may not be entirely accurate. I am not the fastest typist in the world and I ad-lib when I can to get the general gist in the post.
Recently had an interview with Dave Michels on the MTI video codec debate and Object RTC. Greatly appreciate it Dave, thanks!
When I have questions about codecs and IETF standards, I turn to Erik Lagerway. I caught him in the midst of packing for his trip to the upcoming IETF 88 meeting. Erik was a co-founder of Xten Networks (now a part of CounterPath). His current endeavor, Hookflash, is right in the center of WebRTC. Erik co-wrote WEBRTC Object API, an IETF Informational Draft, and the Draft Report W3C ORTC.
More at TalkingPointz.com
Robin did a bang-up job at the Object RTC Walkthrough yesterday, check out the slides and video recordings here.
Thanks again to Microsoft Open Technologies for sponsoring the room and refreshments!
As we can see, MTI Video codec is not present in the agenda. It seems it was pushed back until the IPR issues surrounding VP8 have come to some sort of conclusion. Maybe we should include both VP8 and H.264 for MTI, that or the industry will do it for us if we wait long enough.
RTCWEB IETF 87
Chairs: Magnus Westerlund, Cullen Jennings, Ted Hardie
August 1, 2013
9:00 to 11:30
Should SDES be part of WebRTC security practice and, if so, how?
Presentations: 30 minutes
Discussion: 45 minutes
Post-Plan A/Plan B MMUSIC discussion of impact to RTCWEB documents
Presentation: 30 minutes
Discussion: 30 minutes
Security document updates
Presentation: 5 minutes
Discussion: 10 minutes
August 2, 2013
11:20 to 13:30
Chair review of dependencies in other groups: 10 minutes
Use Case Requirements updates:
Presentations:: 15 minutes
Discussion: 15 minutes
Issues list presentation: 45 minutes
Discussion: 45 minutes
The plot thickens…
VP8 Patent Cross-license Agreement
Google is in the process of preparing an agreement that will assist companies and developers with the adoption of VP8 technology by making available a royalty-free license to certain patents that are necessary for the implementation of VP8 and which are owned by Google and a number of other major technology companies.
Google is considering licensing these patents pursuant to the terms of the attached draft patent license agreement. This draft is under review by Google and is therefore currently non-binding and does not constitute an offer to license any patents. A formal, binding agreement will be posted to this site in the coming weeks. Thus, the below draft is for informational purposes only.
The Primary Licensors are each of the following entities and, subject to any noted exclusions, all of their respective Affiliates (as defined in the VP8 Patent Cross-License Agreement).
CIF Licensing LLC
Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
Koninklijke Philips Electronics N.V.
LG Electronics Inc.
Mitsubishi Electric Corporation
MPEG LA, LLC
Samsung Electronics Co., Ltd.
I have been supporting VP8 as the MTI codec in the WebRTC / RTCWEB WG, for various reasons including…
- Royalty Free (drives innovation)
- Open Source (Source code is optimized for RTC and readily available)
- Supported by a rather large company (Google)
It seems some of us may be changing our minds (myself included), for various reasons, eg…
- H.264 is prolific in RTC today. If we are to have interoperability with other endpoints out there we need 264.
- Open Source. H.264 optimized source for ARM and X86 its coming, Cullen said so 🙂
- Supported by many rather large companies (MPEG-LA for starters)
- VP8 IPR is coming under heavy fire. Nokia being one firm that has boldly stated that they hold patents on VP8 and will enforce them, and apparently there is at least one more VP8 patent holder out there that is keeping rather quiet, which is rather disconcerting.
- Since H.264 utilizes hardware acceleration, battery consumption on mobile devices should be lower as well.
- Quality is arguably better in some cases, this can be somewhat subjective.
So where does this leave the innovators that need free software to create free software? Great question, and here are some potential answers…
- If there is Open Source H.264 software out there that has been optimized for Real-time Communication (encoding and decoding in real-time) there are no “software” fees (excluding MPEG-LA), that is one less, rather large, obstacle.
- MPEG-LA (last I looked) has stated that there are no patent fees associated with deployments under 100k endpoints. This is great for smaller deployments but larger deployments could still have a problem here?
Some are saying that the bulk of all mobile devices that could support video have a t least one license for H.264 already, so why then would there be another royalty for H.264 on that very same device? This is an interesting argument and one that may in fact provide the innovative developers with a leg up when it comes time to debate the issue with the MPEG-LA. That being said, most innovative developers I know want nothing to do with legal debates.
There is also another factor to consider, many of the MPEG-LA patent holders are behind WebRTC, so why then would they shoot themselves in the foot and hamper adoption of WebRTC by litigating those who adopt it? There has also been talk of creating a new H.264 license in the MPEG-LA related directly to WebRTC / RTCWEB, which would be free of royalties. Talk is cheap.
Those vendors who only run one codec today will have likely already chosen H.264 for their existing deployments and will likely vote 264 as the MTI codec. At the end of the day, the prudent vendors looking for the most coverage in WebRTC interoperability will support both H.264 and VP8, regardless of which codec is selected as MTI.
After weighing all the options it seems (to me at least) that H.264 is the better choice today, now we just need an open source (with a BSD or MTI license or the like) H.264 implementation that has been optimized for WebRTC.
I am interested in hearing what the developers out there think about this. What do you think? Should H.264 be the MTI (Mandatory to Implement) video codec for WebRTC?