The road to the promised land.
For more than 6 years, we have been working on and looking forward to a simpler way to build RTC (Real Time Communications) applications on the web. In order for this technology to truly show its value, the major browser vendors needed to show up.
Mobile, mobile, mobile.
Now that Apple has joined the party in earnest, does the technology have the coverage required in order for developers to make good use of WebRTC on mobile devices? Let’s find out.
Until now, in order for WebRTC to work on iOS, we were relegated to wrapping WebRTC code in Objective-C and Swift, in our native iOS apps. Basically, we had to take the Chrome code and build an app that was sent to the app store for approval and wait in line, like all the other chumps (yours truly included). Conversely, on Android we could run much of that same code from our desktop Chrome apps, on the Android device as well, within reason of course.
Now that Safari and Chrome are shipping compatible WebRTC on mobile, we get to reuse the same code, right!? Well, mostly, they are different code bases, after all.
A word about hardware acceleration.
If ubiquitous mobile video is to take off, the battery life of the device has to last more than the length of the 10 minute video call (ok, I am exaggerating a bit, but I think you get the point) and the performance needs to be at least adequate enough to distinguish facial features. My bar is set a little higher, baby steps for now.
Without h/w acceleration the CPU is likely working too hard to encode the local video and decode the inbound video + service the other processes required at the same time. That really means there needs to be hardware onboard the device dedicated to video coding. That in turn means H.264, since there are very few vendors that offer VP8 or VP9 h/w acceleration.
Question: Does this mean that mobile apps written with VP8 will not be able to deliver decent mobile video conferencing?
Answer: No, not at all, but they will likely not be as performant as those taking advantage of hardware acceleration.
Suffice to say that SVC (Scalable Video Coding) usage would be another reason why we need h/w acceleration, but that’s for another day.
Who’s using what?
The majority of desktop and mobile WebRTC apps written today, are using VP8 for video.
Since Apple and Microsoft both use H.264 and Google uses VP8 and H.264 (recently shipped Open H.264 – on the desktop and mobile). Also, many of the Enterprise RTC developers are already on that H.264 bandwagon.
Question: If Apple and Microsoft devices ship with H.264, what is the case with Google Chrome on desktops and android, are they preferencing VP8?
Answer: Chrome for desktop and android now have H.264 native. Many of the Android devices that ship today all have H.264 hardware acceleration onboard. In order to understand which units have H.264 and hardware acceleration, you can run use the Android APIs to pull a list of available codecs, but in the case of WebRTC, you will only get H.264 in Android WebRTC if there is a h/w encoder on the device.
Is H.264 the answer for WebRTC video?
Here is a recent test:
Host 1 – (before joining):
macOS Sierra, Macbook, Safari (Technology Preview 32)
Host 2 (after joining):
Android 7, Samsung 7, Chrome 55
Host 1 (after joining):
According to the Chrome Status page, Chrome for Android should have H.264. So why is the session barfing when trying to set up video? The logs do not lie…
Safari – offer:
Chrome on android – answer:
Err, huh? No H.264 in reply?
So, I updated to latest Chrome on android (58) and tried again…
… et voilà!!
Next topic, paying the man!
Shipping your product with H.264 enabled, means you may potentially need to deal with the MPEG-LA royalty police for H.264 royalties, but there are some grey areas.
In the case of Apple and Microsoft, where H.264 royalties are already being paid for by the parent vendor, the WebRTC developer is riding on the coattails of papa bear, at least in theory.
Cisco’s generous OpenH.264 offer means that those using this binary module, can do so at potentially no cost:
We will not pass on our MPEG-LA licensing costs for this module, and based on the current licensing environment, this will effectively make H.264 free for use on supported platforms.
Q: If I use the source code in my product, and then distribute that product on my own, will Cisco cover the MPEG LA licensing fees which I’d otherwise have to pay?
A: No. Cisco is only covering the licensing fees for its own binary module, and products or projects that utilize it must download it at the time the product or project is installed on the user’s computer or device. Cisco will not be liable for any licensing fees incurred by other parties.
That seems to mean (I am no lawyer) every developer shipping WebRTC apps supporting Open H.264 binary module, get a free ride. Those using some other binary, or shipping the above source code for that module, could be on the hook for those royalties. That said, since there are royalties being paid by parent vendors where devices are shipping H.264 anyways, developers may not get hassled regardless.
So what did we learn here?
- Apple has joined the party, now we have a full complement of browser vendors!
- If you want to leverage WebRTC video to deliver a ubiquitous mobile and desktop experience for your users, you should likely consider including both H.264 and VP8.
- VP8 is (still) free and powers most of the WebRTC video out there today.
- You can make use of the Open H.264 project and get a free H.264 ride, albeit baseline AVC.
- WebRTC on Android does not support software encoding of H.264, so unless there is local hardware acceleration, H.264 will not be in the offer.
- H.264 is not fully enabled (or buggy) in Chrome 55 (I was using it on Samsung S7 Edge (Android 7), but it does work with Chrome 58.
- WebRTC is not DOA!
- SDP still sucks and ORTC can’t come soon enough!!
As a side note, it would be interesting to see something like this open sourced; VP8 / H.264 conversion without transcoding, if only to service the existing desktop apps currently running VP8 <-> mobile H.264. It would likely overwhelm the mobile device, but it would be cool if it worked!
Disclaimer: The views expressed by me are mine alone and do not necessarily represent the views or opinions of my employer.
ORTC, WebRTC, H.264, VP8, RID, RtpEncoding, Simulcast and much more. Google, Microsoft and Hookflash leading the discussion, join us!
Fresh out of Google IO, Justin Uberti provides a WebRTC update via WebRTC Meetup in SFO at the Twilio HQ. Slides and demos are not visible, I am attempting to get a copy of the slides. UPDATE: Most of the slides were captured via photos.
Justin talking points:
– Renewed focus on mobile
– HD bitrates and bandwidth estimation
– Goal H.264 coming to Chrome 45 via Cisco’s OpenH264 (whoa!)
– VP9 & hardware support
– Demo on Nexus 6 using VP9 and hardware encoder
What’s coming next..
– Mobile performance
– Complete call setup should be 500ms
– Encryption (we don’t hold the keys)
– ECDSA coming soon!
– HW encode on android capable of 1080p
– New Echo Cancellation via DAEC (Delay Agnostic Echo Canceller)
– Mobile Networks
– Network Handoff
– Scaling Quality
– Better performance on lossy networks
New domain for “WebRTC and Web Audio resources”
Q What’s the story on spec deviation?
A We want to make sure we add promises to the spec.
Q Get Stats?
A Working on it
Q Unified plan support
A Organizationally challenged and taking back seat to encoding performance and other “on fire” must fix immediately
Q What is going to evolve in screen sharing in spec and Chrome?
A Things work “ok” for screen sharing but not great for some things like scrolling, people are also interested in using in tabs versus window. Screen refresh is not as fast as we would like but we think we have fixed that.
Q Changing framerate and resolution mid-call?
A RTPSender gives you some of these knobs (Note: Object from ORTC Spec!), which is on its way.
Q Battery life for hw encoded apps?
A 3 categories, voice only, video on sw, video on hw. Video demo was on hw at 1080p at 30% of CPU. HW video will compete with a baseband voice call on wifi.
Feross Aboukhadijeh & John Hiesey (creators of PeerCDN
– Using WebRTC DataChannel to stream content
– Demo: can’t see the screen
– Hosting websites in Browsers via WebTorrent
– NAT traversal via regular STUN / TURN
Q Justin asks, what will it take to have this work with existing bittorrent clients
A They need to add WebRTC, then it will work
Hookflash is now providing WebRTC and ORTC Consulting services.
Professional Services Pack
For organizations looking for an expert ORTC / WebRTC developer to provide assistance or become a part of their teams for specific projects. Depending on your needs, the Hookflash subject matter expert can help with any aspect of your project, including consulting, design, development, and QA. The PSP (Professional Services Pack) is available in increments of 40 hours. We have streamlined the engagement process for the PSP to enable a quick start.
Complete Project Development
For companies needing specific development expertise or staff augmentation, we offer complete life cycle development services, including use-case definition, requirements specification, estimates, project planning, design, development, QA, and app store deployment.
Where we shine: C++ (low level, client and server), Assembler (eg. CODEC enhancement), Mobile development: Objective-C (iOS) , Java (android) and Server: Node.js
For more information please visit Hookflash or call 1.855.HOOKFLASH (466.5352) ext.2
The Object RTC initiative is a project supported by Hookflash, Microsoft, Google and others. The ORTC C++ Library is a project maintained by Hookflash. To sponsor ORTC Lib projects send an email to firstname.lastname@example.org
webrtcH4cKS: ~ ORTC is not the “Other” RTC: Q&A with ORTC CG Chair Robin Raymond
Char Hart of webrtchacks.com interviewed ORTC Chair – Robin Raymond on a range of topics. Excerpt:
Biggie vs. Tupac. Gates vs. Jobs. Apple vs. Samsung. Nothing catches people’s attention for no legitimate reason like a feud. Unfortunately this isn’t just a celebrity phenomenon. Feuds have been endemic even to real communications as well. From the very beginning, Elisha Gray’s dispute with Alexander Graham Bell over the original telephone patent showed the industry has a propensity for squabbles…
It touches on some interesting points, namely “a WebRTC app is not defined by its wrapper.” Here is an excerpt from the post..
Google Chrome was the first browser to support WebRTC, and most of the new applications rely on Chrome in order to be plugin-free. Other browsers that support WebRTC include Mozilla Firefox and Opera. These browsers use much of the same code, yet compatibility issues exist because Chrome has considerably more capability than what’s specified in the WebRTC specification….
- Sept 30 – Oct 2 / Chicago – IIT Real-time Communications Conference
(For me, this is “the” objective gathering of the brightest technical minds in the RTC space.) Robin Raymond will be speaking on ORTC / WebRTC 1.1 and also Cloud + P2P Communications.
- 30-31 Oct / Santa Clara – W3C TPAC / WebRTC WG Meeting
(W3C Technical Plenary / Advisory Committee Meetings Week which includes WebRTC Working Group meetings. This should be a rather interesting set of meetings for the WebRTC WG, for a variety of reasons.)
(tba) Oct ? / Web – W3C ORTC Community Group Meeting
- Nov 4-6 / Santa Clara – Cloud Expo / WebRTC Summit
(One of the bigger events, plenty more happening here than just WebRTC.) Erik Lagerway will be speaking on Real-time Communications, PaaS & Cloud Communications.)
- Nov 9-14 / Honolulu – IETF Meeting 91 / RTCWEB WG Meetings
- Nov 18-20 / San Jose – WebRTCworld Conference West
- Dec 16-18 / Paris – WebRTC Conference Expo Paris
The first ORTC Public Draft Specification has been published, authored by Hookflash, Microsoft, and Google. (http://ortc.org/wp-content/uploads/2014/08/ortc.html ) This specification extends WebRTC 1.0 with new functionality to create a WebRTC 1.1 API with exceptional flexibility and no loss of compatibility.
Like WebRTC, ORTC (Object Real-time Communication) enables plugin-free real-time communications for mobile, web and cloud, but is specifically tailored to provide the direct control needed to enable advanced multimedia and conferencing features.
“We heard developers say that they wanted more direct control over the technologies available in WebRTC. At the same time, we didn’t want existing developers to have to start over with a new API. ORTC is our proposal for how we can accomplish both of these things – a new set of APIs for direct control, that builds off the existing WebRTC 1.0 API set. As an evolution of the existing API, we consider this WebRTC 1.1” comments Justin Uberti, Google Tech Lead, WebRTC. “We’re grateful to Hookflash for their work to get ORTC off the ground. They have been instrumental in making this cross-industry collaboration happen, and we look forward to continuing our work with them.”
This newly published public draft has come a long way since the W3C ORTC Community Group was formed in mid-2013. As it has progressed from an initial set of ideas to a fleshed-out draft complete enough for implementations, several companies have gotten closely involved, with Microsoft and Google now joining Hookflash as authors of the emerging specification.
The W3C ORTC Community Group now numbers more than 60 participants.
“We believe the contributions to WebRTC 1.1 / ORTC will allow web communications technology to become ubiquitous and transcend nearly all communications technologies that came before it” says Hookflash Co-founder, Erik Lagerway, “We are honored to be working with some of the brightest minds at Google, Microsoft, and the other contributing members in the ORTC CG to mature WebRTC into a universal go-to toolkit enabling communications across the globe.”
Hookflash enables real-time social, mobile, and web communications for integration of voice, video, messaging with federated identity into world leading software, enterprise, applications, networks, mobile and computing devices. Hookflash and Open Peer are trademarks of Hookflash Inc.
Developers can register at (http://fly.hookflash.me) to start using the Hookflash RTC service and toolkits today. For more information on Hookflash RTC toolkits and White Labeling please visit Hookflash http://hookflash.com.
Come and work at one of the coolest companies in the space! We’re now hiring for these development positions: iOS, Android, Node.js & C++ send us your resume: email@example.com.
Hookflash – Trent Johnsen
855-466-5352 Ext: 1