Access to raw audio data

[lminiero]

I've been tinkering with WASM codecs a bit, and one thing I figured out (well, others did before me, I only realized it looking what they did!) was that I had to "highjack" the hidden support for L16 in browsers to get access to uncompressed audio (via SDP munging, since it's officially unsupported) and then use Insertable Streams to take care of the encoding/decoding at the JS level. @fippo suggested I open an issue here, since standardizing access to raw audio data may indeed facilitate a lot of the audio-related use cases that have started appearing.

[alvestrand]

Wouldn't this also be addressed by adding audio to mediacapture-transform, as discussed in w3c/mediacapture-transform#29 ?

[aboba]

May I suggest that the need goes beyond just obtaining access to raw audio, but also to integrating the WASM codecs without having to trick WebRTC into thinking that it is sending and receiving another codec, such as Opus. IMHO, the use of custom audio codecs is becoming increasingly common, particularly in cloud gaming (Section 3.2.1).

[alvestrand]

Custom audio codecs requires SDP munging, but should otherwise be doable with proposed extensions to encoded-transform.
However, getting L16 should be doable with codec control. Do we have the APIs to enable L16 when needed?

[aboba]

Discussed at TPAC 2023. It was noted that custom audio codecs have uses beyond streaming and so may deserve a distinct use case.

[dontcallmedom-bot]

This issue had an associated resolution in WebRTC TPAC 2023 meeting – 12 September 2023 (Low Latency Streaming: Game Streaming use case):

RESOLUTION: detach #80 from low-latency use case; create a new use case related to pluggable audio codecs

[dontcallmedom-bot]

This issue was mentioned in WEBRTCWG-2023-12-05 (Page 19)