The LocalRealtimeMediaStream Object

The LocalRealtimeMediaStream represents a locally-sourced stream of real-time media.

The constructor for LocalRealtimeMediaStream ignores any information in the RealtimeMediaDescription that it is passed that is not applicable to the provided track.

void play()

Causes the encoding of the stream and the corresponding creation of packets to be resumed.

void pause()

Causes the encoding of the stream and the corresponding creation of packets to be temporarily suspended.

readonly attribute boolean playing

This value is true if the stream is playing. A playing stream generates packets on the transport if the transport is open.

This value can be true if the underlying RealtimeTransport is currently closed.

The RemoteRealtimeMediaStream Object

A RemoteRealtimeMediaStream represents a stream of real-time media that originates on the network.

The constructor to RemoteRealtimeMediaStream constructs a MediaStreamTrack based on the provided RealtimeMediaDescription. If the browser is unable to construct this object, then an error is thrown. This error might be thrown if the media description includes information for both audio and video tracks.

attribute UnknownPacketTypeEventHandler? onunknownpackettype

Called when an unknown packet type arrives on the stream.

The RealtimeMediaDescriptionDictionary Object

Since a media description is likely to be exchanged over a signaling channel, this object represents the value of a RealtimeMediaDescription without the corresponding methods.

sequence<RtpStreamDescription> streams

A description of the RTP streams that comprise the real-time stream.

DOMString cname

The RTCP Canonical Name (CNAME) used to identify this browser as a source of media in RTCP.

sequence<RtpExtension>? rtpExceptions = null

The set of RTP extensions that are used by the described stream(s). Omission of the attribute indicates that extensions are not supported, an empty set indicates that extensions are supported but ignored.

sequence<RtcpFeature> rtcpFeatures

The set of RTCP features that are used by the described stream(s).

sequence<CodecDescription> codecs

A description of the set of codecs that are used by the described stream(s).

byte? priority

The relative priority of the stream.

The RtpStreamDescription Object

A RtpStreamDescription describes a single codec and how it is applied to both encoding or decoding. This includes the packet type for the codec and any codec-specific parameters.

unsigned int ssrc

Each RTP stream has a unique, 32-bit synchronization source (SSRC). This value is used to identify a single stream in an RTP session. The browser allocates a new, random SSRC from the set of available SSRC values for each stream, though this value can be overridden by the application. The browser MAY alter this value if it detects an SSRC collision within the RTP session.

unsigned long bandwidth?

The maximum bandwidth, in bits per second, allowed for this stream. This instructs the browser to constrain the stream to fit within this limit. The browser is able to choose the method it uses to meet this constraint.

The RtpExtension Object

RTP Extensions [[!RFC5285]] enables the addition of extension information to the RTP header. This enumeration identifies specific extensions that might be enabled. A set of these values may be used to describe each real-time stream. Omitting the set indicates that header extensions cannot be used on the stream.

enum RtpExtension {
    "rapid-sync", "c2m-alevel", "m2c-alevel"
};
        

rapid-sync

The RTP extension that enables rapid synchronization [[!RFC6051]].

c2m-alevel

The client-to-mixer audio level extension [[!RFC6464]] with mandatory header encryption.

m2c-alevel

The mixer-to-client audio level extension [[!RFC6465]] with mandatory header encryption.

The RtcpFeature Object

The real-time feedback channel provided by RTCP [[!RFC3550]] supports a number of optional features. This enumeration identifies specific features. A set of these values are used to describe each real-time stream.

enum RtcpFeature {
    "avpf", "nack", "pli", "sli", "rpsi", "reduced", "rtx", "fir", "tmmbr", "tstr"
};
          

avpf

The extended profile for RTCP-based feedback, AVPF [[!RFC4585]].

nack

The RTCP generic NACK [[!RFC4585]] feedback feature.

pli

RTCP picture loss indication (PLI) message [[!RFC4585]].

sli

RTCP slice loss indication (SLI) message [[!RFC4585]].

rpsi

RTCP reference picture selection indication (RPSI) message [[!RFC4585]].

reduced

Reduced sized RTCP [[!RFC5506]].

rtx

RTP retransmission [[!RFC4588]].

fir

RTCP full intra-frame request (FIR) message [[!RFC5104]].

tmmbr

Temporary maximum media stream bit rate request (TMMBR) [[!RFC5104]].

tstr

Temporal-spatial trade-off (TSTR) [[!RFC5104]].

The CodecDescription Object

A CodecDescription describes a single codec and how it is applied to both encoding or decoding. This includes the packet type for the codec and any codec-specific parameters.

When generating a RealtimeMediaDescription, the browser MUST specify values for all fields. Values can be omitted from RealtimeMediaDescriptionDictionary values, which allows the browser to select values for any unspecified values.

DOMString type

The MIME media type for the codec. For example, video/H264 or audio/PCMA.

byte packetType

The packet type that is used for this particular codec.

unsigned long clockRate

The clock rate for this codec description, specified in Hertz.

boolean silenceSuppression = true

For audio codecs that support silence suppression, this value determines if the codec is permitted to perform silence suppression.

DOMString fmtp

Format-specific parameters for the codec, taken from the SDP definition for the codec.

The RealtimeMediaDescriptionConstraints Object

The constraints structure is defined in [[!GETUSERMEDIA]]. RealtimeMediaDescriptionConstraints uses this scheme with a different set of possible constraints.

The following constraints are defined:

rtcp

A set of RTCP features (in an array) that contains all the features that are permitted and desired. This constraint can be met if the RTCP features on all streams are exactly the provided set.

silenceSuppression

A boolean value that determines the value for the silenceSuppression parameter of the CodecDescription.

bandwidth

An unsigned long value for the overall bandwidth allocated to the stream (or set of streams).

Design Goals

It is possible to achieve the basic goals in a number of ways. This design attempts to provide the maximum flexibility in the API, with minimal complexity.

To that end, the API ensures that the browser contains as little of the implementation of the transport establishment code as possible. This provides the using application more control over application logic.

This API also reduces the amount of application state that the browser maintains. This reduces the coupling of browser and application implementations, simplifies API interactions, gives applications greater control over browser behavior and enables the creation of more robust applications. For example, a browser with less accumulated state can recover necessary state faster in the event of a reload than otherwise.

Non-Goals

It is not a goal of this API to provide an application with the ability to create minimal realtme applications. Browser APIs rarely change once deployed, which means that a constraint imposed by an API is difficult to remove. In complete contrast, applications on the web are able to evolve rapidly. Moving the bulk of the necessary logic to application code ensures greater scope for innovation. A simple set of basic primitives are better enablers.

The successful creation of RTP streams requires signaling. Subjecting that signaling to standardization would constrain how it manifests, limiting the flexibility of applications that use these APIs.

Real-Time Transport Usage

Establishing real-time media transport requires the interaction of two objects:

• The RealtimePort is the end point for a real-time transport. This represents a UDP port, opened by the browser. The RealtimePort encapsulates the functions necessary to perform an ICE negotiation [[!RFC5245]]. In ICE terms, a RealtimePort is a local candidate.
• The RealtimeTransport is a peer-to-peer bearer for secured media. It binds a local RealtimePort to a remote RealtimePort and enables the bidirectional exchange of media packets.

In order to establish a transport between a local peer and remote peer, the following process is applied:

1. The local peer opens one or more real-time ports.
2. The local peer then has to learn of the ports that its remote peer has opened. This uses a signaling channel specific to the application. For instance, a web application could use previously HTTP requests or Websockets connections for this purpose.
3. A process of discovery is used to find a local and remote port pair (a candidate pair) that can exchange UDP packets. One or more connectivity checks are made from different local ports toward different remote ports. A successful connectivity check indicates that packets can reach the peer and that the peer consents to receive packets.
4. Finally, a real-time transport is established on the pair of ports. A security context is established so that secured media packets are able to flow in both directions between peers. Real-Time media streams can then be added to the transport.

In the first step of this scenario one or more RealtimePort objects are created in different ways. The browser allocates a local UDP port on each of the available network interfaces and returns these to the application.

    RealtimePort.openLocalPorts(function(ports) {
        signalingChannel.send('candidates', ports);
        localCandidates.push(ports);
    });
      

Each real-time port consists of an IP address, a port number, a username fragment and password. This information is exchanged with the remote peer using whatever signaling mechanism is chosen by the application.

Other forms of real-time port might be necessary for successful traversal of middleboxes like firewalls and NATs. For instance, a relay port (i.e., relay candidate) uses a relay server to forward packets. Because the relay server is publically reachable, many of the concerns with middlebox traversal do not apply to it, enabling communication in more challenging network conditions.

    RealtimePort.allocateRelay(relay, function(error, relayCandidates) {
        if (!error) {
            signalingChannel.send('candidate', relayCandidate);
        }
    });
      

Local and remote real-time ports are paired and connectivity checking is used for each pair to determine which ports can be used for exchanging packets.

    function receivePeerCandidates(remotePorts) {
        var candidatePairs = localCandidates.forEach(function(local) {
            local.addEventListener('checksuccess', checkSuccessCallback);
            return remotePorts.forEach(function(remote) {
                local.check(remote);
            });
        });
    }
      

Of the successful connectivity checks, one or more pairs are selected. A RealtimeTransport is then created from the pair of ports.

    function checkSuccessCallback(event) {
        var local = event.target;
        var remote = event.remote;
        RealtimeTransport.createTransport(local, remote, options, function(err, transport) {
            realtimeTransport = transport;
        });
    }
      

The resulting transport object can be used to securely exchange media with the peer.

The RemoteRealtimePort Object

The RemoteRealtimePort object is used to describe a remote real-time port. It is a simple dictionary of values, sufficient to describe how to connect to a remote peer.

DOMString ip

The IP address (v4 or v6) for the real-time port.

[EnforceRange] unsigned short port

The UDP port number for the real-time port.

DOMString ufrag

The ICE username fragment for the port.

DOMString username

The username is used to specify the STUN USERNAME attribute explicitly. If present, this parameter is used in preference to the username fragment. See server reflexive addresses for an explanation of this parameter.

DOMString pwd

The ICE password for the candidate. This should be set unless this is being used to describe a STUN or TURN server that doesn't require authentication.

The RealtimePort Object

A local RealtimePort provides limited access to the browsers STUN [[!RFC5389]] implementation which provides a means to trigger connectivity checking, the gathering of server reflexive addresses and allocation of TURN relays.

The application cannot select or alter the values chosen for local ports. The browser MAY select any UDP port number, and MUST select values for ufrag and pwd that have sufficient random entropy that they cannot be easily guessed. This guarantees that only entities that receive this information are able to successfully make a connectivity check. It also allows peers to identify unique browsers.

readonly attribute DOMString ip

The IP address (v4 or v6) for the real-time port.

readonly attribute unsigned short port

The UDP port number for the real-time port.

readonly attribute DOMString ufrag

The ICE username fragment for the port.

readonly attribute DOMString pwd

The ICE password for the port.

static void openLocalPorts (RealtimePortCallback callback)

Open ports on the local host. One port is opened for each network interface on the host. Calls the provided callback when complete.

static void allocateRelay (TurnServer turnServer, RealtimePortCallback callback)

Allocate a relay port on a TURN server using a newly created local base.

void allocateRelay (TurnServer turnServer, RelayCallback callback)

Allocate a relay port on a TURN server over UDP using this local real-time port as the base.

[TreatNonCallableAsNull] attribute EventHandler? onnetworkchange

An event handler for being informed of changes to network connectivity. Note: This is a candidate for moving to navigator or even the network information API.

[TreatNonCallableAsNull] attribute ConnectivityCheckEventHandler? onchecksent

Called when a connectivity check is sent to a remote peer. The event passed MUST NOT include the STUN Binding request.

[TreatNonCallableAsNull] attribute ConnectivityCheckEventHandler? onchecksuccess

Called when a valid connectivity check response is received from a remote peer.

[TreatNonCallableAsNull] attribute ConnectivityCheckEventHandler? onremotecheck

Called when a remote peer successfully performs a connectivity check against this port.

[TreatNonCallableAsNull] attribute EventHandler? onclose

Invoked when the real-time port is closed.

readonly attribute boolean open

Whether the port is open and can be used. This attribute is true when the object is passed to the application and false after the close method is invoked.

long check (RemoteRealtimePort remote, optional StunAttribute... attributes)

Perform a connectivity check on the remote port. Returns a handle on the check that can be used to cancel it.

void cancelCheck (long checkHandle)

Cancel an outstanding or enqueued connectivity check. This guarantees that a onchecksuccess event is never triggered for the specified check.

boolean status (RemoteRealtimePort remoteRealtimePort)

Query the status of a particular remote port with respect to this port. Returns true iff the remote port was successfully checked recently. The specific time is based on the browser's configured consent expiration timer.

void close ()

Closes the port and any associated open transports that use the port.

callback RealtimePortCallback = void (DOMString? error, sequence<RealtimePort> ports);
          

callback ConnectivityCheckEventHandler = void (ConnectivityCheckEvent event);
          

RealtimePort Lifecycle

A local RealtimePort is created by the browser in response to a call to openLocalPorts(callback) or allocateRelay(callback). Each RealtimePort is given to the application with the open attribute set to true.

    function createRelayCandidate() {
        var relay = { ip: '192.0.2.75', port: 7754, username: 'user', password: 'password1', turn: 'udp' };
        RealtimePort.allocateRelay(relay, function(error, relayCandidates) {
            if (error) {
                reportError(error);
            } else {
                signalingChannel.send('candidate', relayCandidate);
                scheduleConnectivityChecks(relayCandidate);
            }
        });
    }
          

enum TurnRelayTransport {
    "udp", "tcp", "tls"
};
          

Connectivity Checking

To establish a bidirectional transport flow between peers, both are required to successfully complete one or more connectivity check requests. Connectivity checks are used to open ports on NAT and firewall middleboxes, to validate end-to-end connectivity and to establish that the peer is willing to receive packets.

Network Interface Changes

Network interfaces are imperfect and impermanent. They can change, appear and disappear with no warning, especially on mobile devices. Reacting to changes in connectivity is a crucial part of providing good real-time transport.

The networkchange event is registered against a local RealtimePort.

Ideally, this would be globally registered, but since RealtimePort doesn't have a constructor, this might not be necessary. Again, ideally this is something for the network information API to solve.

The browser fires a networkchange event when it detects a potential change in the status of the network. Events that trigger this callback include changes to open network interfaces and the addition or removal of network interfaces.

Privacy Considerations

Applications that establish peer-to-peer transports require that the IP addresses of a peer are signaled to the remote peer. This can pose a privacy exposure even though an IP address can only be loosely correlated with a person. For instance, it is possible to use IP addresses to determine the physical location of a person.

In some applications, establishing a peer-to-peer transport occurs prior to establishing user consent for the session. This can be necessary to remove the delays associated with transport setup that might otherwise occur after session acceptance. Exposing IP address information prior to acceptance provides the initiator of the session a way to collect the IP address of even an unwilling peer.

Applications are encouraged to only signal relay ports prior to gaining explicit consent from users.

The RealtimeTransport Object

void createTransport (RealtimePort local, RemoteRealtimePort remote, TransportOptions options, TransportCallback callback)

Factory method for creating a transport.

readonly attribute RealtimePort localPort

The local real-time port that is used for this transport.

readonly attribute SimpleRealtimePort remotePort

The remote real-time port that is used for this transport.

readonly attribute boolean dtls

Whether key negotiation for the transport is being controlled using DTLS.

readonly attribute RemoteCertificateInformation? remoteCertificate

Information about the certificate offered by the remote peer. This attribute is only set if the transport is using DTLS for key negotiation.

readonly attribute SrtpSecurityDescription? outboundSdes

The SRTP security description for outbound media.

readonly attribute SrtpSecurityDescription? inboundSdes

The SRTP security description for inbound media.

readonly attribute boolean open

Whether the transport is currently open and able to receive data. Newly created transports always start with this value set to true. Note that absence or expiry of consent from the remote peer can cause a transport to be unable to send on an open transport.

readonly attribute unsigned long bandwidth

The maximum available bandwidth on this transport, as detected by the browser. This value can change over time. Users of the transport operate within this envelope; packets that exceed this limit within any interval (browser-selected) are discarded.

void close()

This closes the transport.

attribute EventHandler? onclose

This event callback is fired when the transport is closed.

readonly attribute boolean consent

Whether the transport is currently open and able to send data. This depends on there being a successful consent check toward the remote peer within a browser-configured period. This valid is false when consent toward the remote peer has expired.

attribute EventHandler? onconsent

This event callback is fired when consent checks toward the remote peer succeed after having failed.

attribute EventHandler? onconsentexpired

This event callback is fired when consent checks toward the remote peer time out and no packets are able to be sent on this transport.

attribute EventHandler? onbandwidthchange

This event callback is fired when the browser learns of a change in available bandwidth on the transport. The browser MAY limit the rate of notifications, or suppress notification for small (10% or less) changes in available bandwidth.

Transport Creation Options

Specifying RealtimeTransportOptions can be used to alter the characteristics of a transport.

RealtimeTransportMode mode = "dtls-srtp"

The mode used for the transport.

SrtpSecurityDescription outboundSdes

A description of the security parameters for outbound streams of RTP media secured with SRTP [[!RFC3711]]. Used only when SRTP keys are negotiated through signaling.

SrtpSecurityDescription inboundSdes

A description of the security parameters for inbound streams of RTP media secured with SRTP [[!RFC3711]]. Used only when SRTP keys are negotiated through signaling.

DtlsRole? role

The role that the local browser takes in the DTLS [[!RFC5764]] handshake, either 'client' or 'server'.

CertificateValidationCallback validate

This callback is triggered when the certificate for the remote peer is received. This allows the application to accept or reject a peer based on the proffered certificate. This is only valid for a DTLS transport. When omitted, the certificates is automatically accepted.

sequence<DOMString> cipherSuites

A set of DTLS cipher suites that can be negotiated for DTLS, using the Description from the TLS cipher suite registry. The browser cannot negotiate any suite that is not included in this set.

enum RealtimeTransportMode {
    "srtp", "dtls-srtp"
};
          

enum DtlsRole {
    "client", "server"
};
          

callback CertificateValidationCallback = boolean (RemoteCertificateInformation remoteCertificate);
          

function validateCertificate(certInfo) {
    return certInfo.isValidForDomain('example.com');
}