SDP inside WebRTC is bad for SIP
For those who don’t know, SDP is an old school standards-based text format (pre-1998) for describing media, codecs, state and networking information offered by devices for use in real-time communications and more recently as the proposed format for with WebRTC. I’ve written in the past about my disdain for SDP. To me, using SDP inside the browser for WebRTC seems akin to requiring all new computers use the graphics processing unit from the Commodore 64 for all future graphics engines. As cool as it might have been in its day, it is not exactly up to the task anymore and should be left to the realm of nostalgia.
My original thinking was that the SIP guys would really love SDP in the browser since SDP is the primary media description format they use. But I must recast my opinion to say it’s really bad for the SIP folks as well. Here’s why…
- An increasing need for Session Border Controllers: As it stands, the SDP that comes from SIP devices will need to be re-written and perhaps even put through some kind of Session Border Controller (SBC)/Proxy to maintain compatibility. SIP devices could face update cycles tied to browser updates. There are some companies in the industry who sell proxies that would greatly benefit from compatibility issues (as their role is to fix them) but I would hate to think that the IETF/W3C has been usurped by those vendors to push a solution that is not to benefit the entire internet industry and end users.
- SIP feature-creep: One thing I do know about SIP vendors is they love to add their own extensions to SIP to add their favourite competitive features they offer with their devices/networks. This allows them to claim “support” for something their competition does not support. To that end, I’ve noticed a continuous stream of feature requests to the browser vendors from the SIP world (and I’m certain the alignment to a SIP vendor’s own preferred feature is pure coincidence).
The irony is that if the SIP vendors had insisted that the browsers only offer a good core media/RTC engine, they could have implemented many of the features they now demand from the browsers vendors themselves without waiting for Google, Mozilla, Opera, Microsoft and Apple and the rest of the industry to agree. Talk about a SIP vendor’s dream in being able to offer some unique feature for their network! But now they have to wait and wait and hope and whatever ends up being released in the browsers will work for them and not introduce even more problems and incompatibilities to their networks.
Browser vendors will become the choke-point.
Microsoft argued so strongly against SDP and offer/answer, they have not agreed to support WebRTC and instead produced a competing specification called CU-RTCWeb. Their proposal starts from the premise of having a good media engine/RTC controllable at a lower layer would be far better for the industry and they have (so far) not released their Internet Explorer browser with WebRTC support. Whatever your feelings about Microsoft or their particulars of their proposal, they are right about SDP offer/answer and without their market share being onboard, it will hurt WebRTC’s adoption rate, especially in the Enterprise. Apple is sitting on the sidelines giving no indication their position while the industry sorts this out. I’d love nothing more for the industry than to have all vendors on the same page and agree to implement “something” usable, but it seems the SDP offer/answer model is not helping and in fact hindering that effort.
Where do we go from here?
Cross-posted to IETF RTCWEB Mail list
Written by Robin Raymond
Edited by: Erik Lagerway
SDP the WebRTC Boat Anchor
I originally created the last blog post on why I have a really strongly dislike for SDP in WebRTC / RTCWEB. I was asked by Justin Uberti to repost my sentiments to the RTCWEB IETF mailing list. This is that summary, which I will be posting to the mailing list, apologizes in advance for the length.
I’ll try to compile my various points as to why I think we need a lower level RTCWEB API that does not include SDP.
My issues with SDP can be summarized as:
- unneeded – much too high level an API
- arcane format – legacy and problematic
- lack of API contact
- doesn’t truly solve goal of compatibility to legacy systems
Some will argue we need a higher level exchange “blob” like structured data (like SDP) to get two browsers talking to each other and that it makes it simple and it’s advantageous. I think they are horribly mistaken on this front.
SDP isn’t just an opaque blob; it’s offer/answer but I’ll get to offer/answer problems later.
To address a first point: do we need such an exchange SDP “blob” format in the first place? All media can be done without SDP given an intelligent stream API. No opaque blob is necessary at all. An API already exists to create these streams (albeit somewhat lacking if we remove the SDP ‘blob’). This API helps “simplify” creating this blob for later exchange. But the blob is truly not needed. Each side could in fact create the desired streams, pass in the appropriate media information such as codecs and ICE candidates and chose the socket pair to multiplex upon. Yes, it’s a bit more low level but it certainly can be done (and cleanly).
The larger issue, should it be lower level? Yes, in my opinion it should be. Some might say it will be too complicated for “web developers”. Nonsense! Web developers are a smart bunch. Libraries like jQuery, Promises, Node, WebGL, etc are a testament to their capabilities. The argument that it would be too difficult doesn’t wash with me. Obviously, wrappers can and will be created to simplify access for those who want higher level access. Already a half-dozen companies are vying for position as “simplified” access to RTCWEB above what the current “simplified SDP” is doing already.
The API needs to be at lower level. A lower level API will increase compatibility by setting the barrier to entry lower to what is needed to be RTCWEB compliant. A lower level API will allow for more innovative thinking when building new technologies and combinations of ideas. This will not be the case if we limit ourselves by mandating SDP in WebRTC.
Imagine being able to create streams as desired and mix and match the “pin connections” of the streams however needed via mixers, and then allowing for final rendering. So long two peers agree to communicate (via ICE) and access to sensitive assets like cameras/microphones are granted, why shouldn’t the manipulation of the streams be up to the programmer?
That’s the API we need. I not only prefer it but I think its vital to the success for WebRTC. There’s no SDP offer/answer needed. There’s no shortage of really smart people out there who would know how to produce a great API proposal. There’s no security threat introduced by managing streams with a solid API.
The SDP format itself is arcane and rooted in old world legacy reasoning. One of the primary goals was to be within a really tiny packet MTU. It’s difficult to extend and the parse rules are all over the map.
RTCWEB intentionally did not dictate a stack signaling protocol – and thank you! This ensures RTCWEB isn’t tied to SIP, XMPP/Jingle or perhaps Skype on the IE browsers. That allows for untold possible signaling scenarios, from silo websites using simple web sockets to supporting other future protocols (like Open Peer).
But SDP is a signaling protocol, just at the media level. Boo! You are forcing all protocols in the future into an offer/answer model. Despite what some might say, it’s not the only model. For example, Open Peer is stateless between peers. Each side provides its expectations of what it can send and what it expects to receive and a connection is formed based on that information. Further, we do not renegotiate streams, at least not in the SDP sense.
I would agree with Microsoft’s argument that SDP creates a brittle protocol. Changes in the future will force renegotiations as a reaction to what’s going on locally into a need to maintain a paired state to the remote side (including the complexities should both sides simultaneously renegotiate). If you get into multiple party handshakes, it can be hell. We made a conscious decision to be stateless/independent in Open Peer, based on our expertise and history when working with SIP. With SDP offer/answer, I’ll be forced to use this unnatural state machine just to support offer/answer and forever hamper our ability to dynamically change offers until fully accepted in a round trip from the remote party (not to mention the complexities in group conversations). Offer / answer will add untold headaches, but I don’t want to complain just because it’s “harder” for us. It’s not just programming difficulties; it breaks our concept of a good P2P on the wire signaling protocol.
The browser vendors have to create an API anyway – from the “initiator” side to be able to get an SDP blob. But the receiver side gets all sorts of implied logic and behavior when it receives this blob. Whatever untold elements exist in the SDP, it must understand (or reject even though it might be compatible). Some might say “wonderful” as it makes it all free/easy. But like the saying goes, free has a cost. The cost is a loss of control over what behaviours are wanted by the receiving party. The remote browser has to support all features that exist in the SDP, no matter how crazy and who is to say the SDP is going to the browser anyway? You’ll get versioning issues across SDP, all bundled into the offer / answer state machine.
Developers will sneak in additional features our protocol doesn’t want or isn’t compatible to support across devices by turning on features locally in SDP for their website, where these new bits get thrown into the SDP bundle and then sent to the remote party unaware any new feature which it won’t support (at least nor properly). That local website might be very happy with their new feature working as it will work “for them” but they’ll break federation to other sites. Some clever piggybacking (inside SDP) features will be hidden from the protocol layer that transport it, which could bring down the services across domains.
What will happen when browser vendors attempt to add features? They’ll add it, but another problem that will emerge. All new features will likely end up be expressed in the SDP somehow. Those things will break services and testing with a limited set of RTCWEB compliant browsers and devices will be insufficient. This will explode compatibility problems well beyond anything the browser vendors could imagine. Basically, it will become very risky to innovate at the browser level.
With the stream primitives only, I can build whatever state machines that are needed for the particular features I need (from none when I’m just swapping codecs, to more complex when I want to do dynamic re-pinning i/o to untold mixers). Now, throw SDP in there and we have to baseline all that into a common understanding of what it means, disallowing me from doing it because its unsupported in the SDP or force me to have a hybrid where I transport this SDP with all my additional information and have to coordinate my state machine with the browsers offer/answer state machine.
In Open Peer, we setup new streams (on same sockets) and quick-swap behind the scenes if the media must change. This greatly simplifies our model for us. Would I impose our model on anyone else? No. I’m not advocating a stateless SDP model either. Again, I’m saying no media signaling is required at all.
An API only would lower the bar of browsers being able to interoperate at the media level, since they only have to be compatible at the lower stream level. This removes the concerns about SDP compatibility issues (including the untold extensions that will happen to handle more powerful features and all that it implies and complex behaviours associated with SDP offer/answer, including rollback and ‘m=’ stability). Further, steams are easy to make compatible even if individually their API sets aren’t up to par to their counterparts across browsers and devices.
This also solves an issue regarding the data channel. There is no need for the data channel to be tied to an offer/answer exchange in the media at all. They are separate things entirely (as well they should be). For example, in Open Peer’s case the data channel gets formed well in advanced of media to maintain our document subscription/notification model between peers and media channels are open, closed and swapped as required.
The moment we use SDP we have to address one fundamental issue. Is the SDP blob meant to be transported in full format only to the destination “as is” or is it okay to mess with the SDP by intermediates? This is extremely important. SDP is a not just a format. It is an extendable specification. This mean it will be extended in unknown ways and as such those extensions can affect behavior and add/change or remove functionality.
The protocol we have written won’t use SDP (unless forced by RTCWEB). We’d prefer to transport the information exchange ourselves via a more palatable and future thinking method than what SDP allows (in fact, we do use a JSON format on the wire). This means we’d likely parse and tear apart the information contained inside the SDP and then reassemble a new SDP on the remote side. But that would be a very BAD idea for us to do in practice and likely force us to use the SDP format forever and deliver this SDP blob inside our JSON format “as is”.
The reason why it’s bad to disassemble/reassemble the SDP is because it can be arbitrarily extended at will without knowledge of what is going on internally. New features could be added to the SDP without it being understood. This might sound like a beneficial feature but it’s actually dangerous.
With an API, it’s a contract. You don’t change the contact arbitrarily because it has implications. An API can be extended, but with the current API you know exactly what you get thus you can predict behavior. SDP is not such. It can be changed arbitrarily and there’s no guarantee the two will match. Intermediates can and will mess with the SDP (as demonstrated in the SIP world). If you allow 3rd party people like me to modify it, we’ll lose the additional “features” in the name of transforming SDP into something palatable/compatible. But we really shouldn’t, as it’s way too dangerous; that means SDP will be imposed at the signaling protocol layer on us even though we don’t want it.
If you allow modifications, SDP is a compatibility nightmare. I was the original author of the popular softphone client (X-Lite) and I understand the compatibility issues that happen with SDP/SIP. It was modified many different ways and extended over and over, and in crazy ways. People couldn’t even get basic things like ICE right, let alone all those crazy things they did to SDP across venders. It was a mess. Everyone extending it in every which way imaginable and it created a nightmare of issues; the formats just weren’t compatible.
As a side note, SBC (Session Border Controllers) vendors probably love SDP. That means job security. They would constantly rewrite SDP between end points to ensure ‘compatibility’. This is what RTCWEB has to look forward to if we adopt SDP permanently. Yes, be very afraid.
If browsers use SDP, they inherit this mess and explode it. And compatibility issues won’t be limited to browser SDPs. There is a huge swath of legacy systems that will deliver a mess of SDP to the browsers and SBCs that will manipulate the SDP in untold ways. This will reintroduce the problems with SIP into the world of RTCWEB. The browsers will be locked to those legacy systems and they will drag the browser back like a ball and chain and it will limit the browser’s ability to innovate.
I can’t stress this enough: SDP is not required for compatibility. In fact, it’s a hindrance to it. I think the SIP vendors think if the browsers use SDP they will gain lots more compatibility. They won’t. They are better off writing a JS library that talks the SDP they understand if they want SDP, rather than trying to mix/match browser SDP into the mix. SDP that is likely offered by RTCWEB will not be compatible anyway with many (perhaps most) legacy systems out there. Many of these systems still don’t support the latest ICE specifications and it’s been a long time standard. Imagine RTCWEB being tied down by these systems later and breaking millions of end points during a casual update of the browser.
SDP binds things that don’t need to be bound. We can negotiate all the streams independently. Maybe we want 6 video streams then suddenly want to change that to zero. Why do we need to preserve six dead video media SDP lines? With SDP, we force them all bound to this SDP bundle that has to be negotiated together. It’s wholly unneeded and doesn’t allow flexibility of the streams.
I’m going to take a draft example of what I mean:
Nothing wrong with the draft in an SDP/SIP mindset but I’m going to take it from a totally different non-SDP angle. I have to say, the ideas presented are very good. I appreciate FEC, and synchronizing streams is cool. But SDP isn’t needed to do it. Let me as the programmer worry about how to manage streams and the features on the streams and associations between the streams via an API only.
Point 4, 5 and 6 in the specification all have to do with the complexities of having to describe the intentions of mixing in SDP. So no comment beyond “don’t use SDP”.
As for 7.1 – “this is because the sender choses the SSRC” – only true because we are forced to use SDP and the assumptions is that it’s SIP. We could have the receiver dictate what the sender should use in advance of any media. In our case, we establish in advance what we want from all parties before even “ringing” the other party. We do not have SSRC collisions as we reversed the scenario allowing the receiver to pick the expected SSRC. Coordinating the streams is a problem with SIP because of how they do forking/conferencing but not for Open Peer. We do not fork like they do. We negotiate each location independently and statelessly. This specification forces this issue on Open Peer. If SIP has problems with streams arriving early to their stateful offer/answer then let them worry about “how” they intend to match the streams at a higher SDP layer and get this draft out of the RTCWEB track on the SIP track. To be clear, the proposal seems entirely reasonable and intelligent for SIP/SDP. But it’s way to SIP centric for general purpose.
On that note, I do need in the API is an ability to dictate the SSRC when I create an RTP stream for sending (should I care to do that).
7.2 Multiple render
Again this is an issue of SIP/SDP. We can control the SSRCs to split them out to allow multiplexing easily on the same RTP ports with multiple parties/sources. If given the primitives to control the streams just, this specification could be used to dictate how to negotiate issues in their space.
7.2.1 I’m feeling the pain. How about just giving me an API where I can indicate what streams are FEC associated.
7.3 Give me API to give crypto keys to RTP layer. Let me handle the fingerprint and security myself beyond that.
8. Let’s just say politely that I would not want to be the developer assigned to programming around all this stuff.
Again, a perfect illustration why I don’t want SDP.
Media is complicated for good reason as there are many untold use cases. The entire IETF/W3C discussion around video constraints illustrates some of the complexities and competing desires for just one single media type. If we tie ourselves to SDP we are limiting ourselves big time, and some of the cool future stuff will be horribly hampered by it.
To conclude, can I work around SDP? Sure, just like browsers can patch around IE 6.0. But believe that having a more media stream centric API without the SDP offer/answer will simplify your release and increase compatibility, allow for newer and stronger protocols and more importantly allow many future capabilities that others can’t imagine if you don’t attach the boat anchor that is SDP.
Microsoft feels so strongly against the current RTCWEB path they decided to go the way of counter proposal with CU-RTC Web. I can’t comment on their specification (yet) but I can understand their strong sentiment.
For those who don’t know me, I’m the Chief Architect at Hookflash and author of the new P2P protocol, Open Peer. I used to be the CTO/ Chief Scientist at Xten, now CounterPath, and I’m the original author of the X-Lite/X-PRO/eyeBeam SIP softphone clients. I wish I had given my feedback earlier on this subject. To be honest, following and participating in the standards tracks requires huge time devotion to which I’ve not had enough of and perhaps I thought (naively so) the smart people in the IETF would naturally make the wisest choices, in the best interests of all! As you can tell, I am not at all happy with where we sit today. RTCWEB / WebRTC & SDP… brutal.
NOTE: Will be cross posted to RTCWEB IETF mailing list.
SDP in WebRTC? Who cares…
I would like to believe that I’m not hopelessly confused and outdated with regards to what is going on with RTCWEB. Last I checked my head is not stuck in the sand nor have I been buried under a rock for the last several years. I recently watched the February 7th, 2013 netcast talking about the data channel and questions about how it relates the SDP and the SDP ‘application m-line’.
For the love of all that is human, why is SDP part of RTCWEB efforts at all?
To be clear, I’m talking about a few specific aspects of SDP: the format, the exchange of SDP between browsers and media negotiations via the offer/answer model (and all that it implies regarding the negotiation of media streams). Come to think of it, all that makes SDP, well… SDP. I know what some will say: We need to exchange some kind of blob-like information between browsers so they can talk, that’s why SDP is used. And I would respond “of course”! Beyond arguing how arcane SDP is as a format, RTCWEB was specifically designed not to do signaling stuff at all. That part was purposefully (and wisely IMHO) left out of the specification so that the future was wide open for whatever it might hold in creative solutions.
What we really need in order to do future stuff in the browsers (yet remain compatible with the past) is a good API for a lower level media engine to create, destroy, control and manage media streams. That’s it. Write an engine that doesn’t take SDP, but manages much lower level streams and allow the programmer to dictate how they are plumbed together, which are active and inactive, and give events for the streams as they progress.
That’s the API I want. There’s no SDP offer/answer needed. There’s no shortage of really smart people out there who would know how to produce a great API proposal.
Such an API would lower the bar of browsers being able to interoperate at the media level. This removes the concerns about SDP compatibility issues (including the untold extensions that will happen to handle more powerful features and all that it implies and complex behaviours associated with SDP offer/answer, including rollback and ‘m=’ stability). If the browsers support RTP, ICE and codecs, and can stream then they are pretty much compatible even if individually their API sets aren’t up to par to their counterparts.
This also solves an issue regarding the data channel. There is no need for the data channel to be tied to an offer/answer exchange in the media at all. They are separate things entirely (as well they should be). For example, in Open Peer’s case the data channel gets formed in advanced to maintain our document subscription model between peers and media channels are open and closed as required.
Those who still want to do full on SDP can do SDP. Those who want stateless SDP-like exchanges can do exactly that. Those who want to negotiate media once and leave the streams alone can do so.
As an example, let’s examine Open Peer’s use case. Open Peer does not have, nor does it need or want a stateful offer/answer model. It also doesn’t support or require media renegotiation. Open Peer offers the ports and codecs (including offering to multiple parties the same port sets) and establishes the desired media. Call and media state is completely separated out. From then if alternative media is needed, a new media dialog is created to replace the existing one and then a ‘quick swap’ happens and the media streams are rewired appropriately to the correct inputs and outputs without renegotiation, at least this is not a renegotiation in the offer/answer sense of the meaning. Further, Open Peer allows either side to change its media without waiting for the other party’s answer.
Media is complicated for good reason as there are many use cases. The entire IETF/W3C discussion around video constraints illustrates some of the complexities and competing desires for just one single media type. If we tie ourselves to SDP we are limiting ourselves big time, and some of the cool future stuff will be horribly hampered by it.
Let’s face it, browsers are moving toward becoming sandboxed operating systems. So why do we not give an appropriate API low level as it deserves that allows for flexible futuristic application writing? Complicated and powerful HTML 5 APIs are being well received, so why can’t the same be true for lower level RTCWEB APIs?
I know Microsoft has argued the API is too high level and they’ve even gone to the trouble of submitting their own specification with CU-RTC Web and splintering and fragmenting efforts. I don’t presume to represent this stance regarding SDP, nor will I go into the merits of their offering, but I think they are right in principle. And for saying so, I’ve got my rotten tomato and egg shield in position.
In the Trenches with RTCWEB and Real-time Video
The concept of video streaming seems extraordinarily simple. One side has a camera and the other side has a screen. All one has to do is move the video images from the camera to the screen and that’s it. But alas, it’s nowhere near that simple.
Cameras are input sources, but they have a variety of modes for which they can operate. Each camera has it’s own dimensions (width/height), aspect ratio (the ratio of width/height) and frame rate. Cameras are often capable of recording at selectable input formats, for example, SD, or HD formats, which dictate their pixel dimensions and aspect ratio sizes (e.g. 4:3 or 16:9). If a camera opens in one format and switches to another there can be a time penalty before the video starts streaming again from the camera (thus switching modes needs to be minimized or avoided entirely). On portable devices, the camera can be orientated in a variety of ways and dynamically change its pixel dimensions and aspect ratio on the fly as the device is physically rotated.
Some devices have multiple camera inputs (e.g. front camera or rear camera). Each source input need not be identical in dimensions nor capability and the user can choose the input on the fly. Further, there are even cameras that record multiple angles (e.g. 3D) simultaneously, but I’m not sure if that should be covered right now even though 3D TVs are all the rage (at least from Hollywood’s perspective).
If I could equate cameras to a famous movie quote: they are like a box of chocolates, you never know what you are going to get.
Cameras aren’t the only sources though. Pre-recorded video can be used as a source just as much as a camera. Pre-recorded video has a fixed width, height and aspect ratio, but it must be considered as a potential video source.
The side receiving video typically renders the video onto a display. These output displays are also known as a type of video sink. There are other types of video sinks though, such as a video recording sink or even videoconferencing sink. Each has it’s own unique attributes that vary and the output width and height of these video sinks vary greatly.
Some video recording sinks work best when they receive the maximum resolution possible. While others might desire a fixed width/height (as it’s intended for later viewing on a particular fixed size output device). When video is displayed in a webpage, it might be rendered to a fixed width/height or there might be flexibility in the size of the output. For example, the page can have a fixed width, but the video height could be adjustable (up to a maximum viewable area), or vice versa with the width being the adjustable axis. In some cases both dimensions can adjust automatically larger or smaller.
Some output areas are adjustable in size when manually manipulated by a user. In such cases the user can dynamically resize the output areas larger or smaller as desired (up to a maximum width and height). Other output screens are fixed in size entirely and can never be adjusted. Still other devices adjust their output dimensions based upon the physical rotation of the device.
The problem is how do you fit the source size into the video sink’s area? A camera can be completely different in dimensions and aspect ratio than the area for the video sink. The old adage “how do you fit a square peg in a round hole” seems to apply.
In the simplest case, the video source (camera) would be exactly the same size as the output area or the output area would be adjustable in the range to match the camera source. But what happens when they don’t match?
The good news is that video can be scaled up or down to fit. The bad news is that scaling has several problems. Making a small image into a big image makes the image appear pixelated and ugly. Making a bigger image smaller is better (except there are consequences for processing and bandwidth).
Aspect ratio is also a big problem. Anyone who’s watched a widescreen movie on a standard screen TV (or vice versa) will understand this problem. There are basically three solutions for this problem. One solution is shrinking the wide screen to fix into the narrow and put “black bars” above and below the image, known as letterboxing (or pillarboxing on the other axis). Another solution is to expand the image large enough while maintaining aspect ratio so there are no black bars (but with the side effect that some of the image is cropped because it’s too big to fit in the viewing area). Another method is to stretch the image making images look taller or fatter. Fortunately that technique is largely discredited, although still selectively used at times.
Some people might argue that displaying video using a letterboxing/pillarboxing technique is too undesirable to ever be used. They would prefer video was stretched to fit the display area and any superfluous video image edges are automatically cropped off. Videophiles might gasp at such a suggestion for the very idea that discarding part of an image is nearing sacrilege. In practical terms, it’s both user preference as well as context that determine which technique is best.
As an example of why context is important, consider video rendered to the entire view screen (i.e. full screen mode). In this context, letterboxing/pillarboxing might be perfectly acceptable, as those black bars become part of the background of the video terminal. In a different context, black bars in the middle of a beautifully formatted web page might be horrifically ugly and unacceptable under any circumstance.
The complexities for video are far from over. When users place video calls, the source and the video sink are often not physically located together. That means that the video has to go from the source to the video sink located on different machines/devices and across a network.
When video is transmitted across a network pipe, a few important considerations must be factored. A network pipe has a maximum bandwidth that fluctuates with usage and saturation. Attempt to send too large a video and the video will become choppy and glitch badly. Even where the network pipe is sufficiently large, bandwidth has a cost associated, thus it’s wasteful to send a super high quality image to a device that is incapable of rendering it to the original quality. To waste less bandwidth, a codec is used to compress images and thus preserve network bandwidth as much as possible (the cost being the bigger an image, the more CPU required to compress the image using the codec).
As a general rule…
- a source should never send video images to the remote video sink that ends up being discarded or at a higher quality than the receiver is capable rendering, as it’s a waste of bandwidth as well as CPU processing power. For example, do not send HD video to a device only capable of displaying SD quality.
Too bad this general rule above has to have an exception. There are cases where the video cannot be scaled down before sending, although rare. Nonetheless, this exception cannot be ignored. Some servers offer pre-recorded video and do not scale the video at the source because doing so would require expensive hardware processing power to transcode the recorded video. Likewise, a simple device might be too underpowered or hard wired to its output format to be capable of scaling the video appropriately for the remote video sink.
The question becomes which end (source or sink) manipulates the video? And then there are the questions of how and what does each side need to know to do the right thing to get the video in the correct format for the other side?
I can offer a few suggestions that will help. Again, as to the general rules..
- A source should always attempt to send what a video sink expects and nothing more
- A source should never attempt to stretch the source image larger than the original source image’s dimensions.
- If the source is incapable of adjusting the dimensions to the video sink completely, it does so as much as it is capable and then the video sink must finish the job of adjusting the image before final rendering.
- The source must understand the video sink can change dimensions and aspect ratio anytime with a moment’s notice. As such, there must be a set of active “current” video properties the source must be aware of at all times with regard to the video sink.
- The “current” properties include the active width and height of the video sink (or maximum width or height should the area be automatically adjustable). The area needs to be flagged as safe for letterboxing/pillarboxing or not. If the area is unable to accept letterbox or pillarbox then the image must ultimately be adjusted to fill the rendered output area. Under such a situation the source could and should pre-crop the image before sending knowing the final dimensions used.
- The source needs to know the maximum possible resolution the output video sink is capable of producing to not waste its own CPU opening a camera at a higher resolution than will ever be possible to render (e.g. an iPad sending to an iPhone device). Unfortunately, this needs to be a list of maximum rendered output dimensions as a device might have multiple combinations (such as an iPhone device suddenly turned on its side).
I’m skeptical if a reciprocal minimum resolution is ever needed (or even possible). For example, an area may be deemed letterbox/pillarbox unsafe and the image is just too small to fit a minimum dimension (and thus would have to be stretched upon rendering). In the TV world, an image is simply stretched to fit upon output (typically while maintaining aspect ratio). Yes, a stretched image can become pixilated and that sucks, but there are smoothing algorithms that do a reasonable job within reasonable limitations. People playing DVDs on Blu-ray players with HD TVs are familiar with such processes, which magically outputs the DVD video image to the HD TV output size. Perhaps a “one pixel by one pixel” source connected to an HD (1920×1080) output would be the extreme case of unacceptable, but what would anyone expect in such a circumstance? That’s like hooking up an Atari 2600 to an HD TV. There’s only so much that can be done to smooth out the image, as the source image quality just isn’t available. But that doesn’t mean the image shouldn’t be displayed at all!
Another special case happens when a source cannot be scaled down for whatever reason before transmission and the receiving video sink is incapable of scaling it down further to display (due to bandwidth or CPU limitations on the device). The CPU limitation might be pre-known, but the bandwidth might not. In theory the sink could report failures to the source and cause a scale back in frame rate (i.e. cause the sender to send fewer images rather than smaller images). If CPU and bandwidth conditions are pre-known, then a maximum acceptable dimension and bandwidth could be elected by the video sink thus such a non dimension adjusting source must be incapable of connecting.
Aside from the difficulties in building good RTC video technology, those involved in RTCWEB / WebRTC have yet to agree on which codecs are Mandatory to Implement (MTI), which isn’t helping things at all. Since MTI Video is on the agenda for IETF 86 in Orlando maybe we will see it happen soon. If there is a decision (that’s a big IF), what is likely to happen is that there will be two or more MTI video codecs, which means we will need to support codec swapping and all the heavy lifting related thereto.
From the Sidelines, My Introduction into RTCWEB
I’ve been following the RTCWEB standardization for a while now from an architecture and technology standpoint. For the most part, I’ve been quiet and I’ve assumed a rather neutral stance in regards to the RTCWEB process when it comes to Open Peer, but my opinion has changed and I can no longer maintain a neutral standpoint.
There are many companies taking stances who all need to have a say in what happens because they want to make sure their technology does not get left out in the cold when RTCWEB comes into reality, as most people think this technology will be huge with consumers and businesses. The big guys with SIP, XMPP and Skype have various established offerings and they are married to existing technology that is difficult to change. They need to make sure that RTCWEB closely follows, or at the very least, does not hinder their own technology from functioning otherwise they will get left behind. The process of adapting existing systems to a new standard is understandably costly.
Thus, I have to ask, who am I with Open Peer to come along and push back against these tides as a new protocol when I have much more flexibility in our implementation than existing deployed systems? Further, the Open Peer protocol particularities didn’t even exist until recently and it has been under revision as Hookflash tested the implementation. We’ve just recently published our specification and source code and we’ve just undergone a significant update based on internal and external feedback from our initial implementations.
To be honest, architecting, designing and implementing a brand new protocol with such an ambitious scale for a small company has kept me extremely engaged and busy. I could listen to what’s happening from a 1,000-foot high perspective, but unfortunately that has also been a factor in my personal ability to participate. I don’t think it’s a great secret for those already involved that it takes immense devotion of time resources to follow the details, let alone participate in these long drawn procedures in ratifying a specification complex as RTCWEB that spans two organizations, namely the IETF and W3C groups. This is unfortunate that such time commitments are so huge as I think having those on the front lines much more actively involved would be healthy, but I digress.
In reality though, Hookflash is in a unique position with Open Peer. I am working on this protocol with a clean slate and a future thinking sense. I do not have the old technology shackles and I didn’t have to design with legacy deployed services in mind which would no doubt confound my decision making process. Likewise, I’ve had the experience of these legacy systems to help avoid their pitfalls (specifically as the original author of the X-Lite/X-Pro SIP softphone client for CounterPath years back with SIP).
The Open Peer implementation goes beyond basic call flow signaling and even beyond peer-to-peer signaling and incorporates identity and federation concepts with strong privacy and security considerations in mind.
Having just completed the next iteration of the protocol that is going through internal testing, I plan to spend much more time actively examining the details of RTCWEB standards. Even though I’m later to the table representing a newer company with newer technology, I hope the input will be welcome to the discussion. I do understand that decisions may be too immovable to change peoples’ minds and there is an active amount of established legacy systems, but hopefully coming from a unique perspective will help bring fresh blood and deeper insight. Forgive me if I argue points already lost, but I will always explain my reasoning for wanting to push certain aspects even if I am ignored in the end.
Ultimately, Open Peer will leverage RTCWEB and the implementation will adapt accordingly to the mutually agreed standards. I’m still going to give my opinion for whatever it is worth and I hope to prove it worthy, unique and valuable.
There are many bright people involved in the process and many companies with unique corporate political angles and agendas. My perspective and motivation will be straight up front. I want RTCWEB to succeed as soon as possible, but with an equal emphasis on ensuring the technology is sound from the future perspective as well, obviously in relation to plans with Open Peer utilizing RTCWEB.