Archival suitability

[cyanreg]

@JeromeMartinez I've been working on this for a while now. @robUx4 helped out a bit as well, and broadcast folks also gave opinions. The plan is to eventually become part of the FFmpeg project, and get it submitted for an IETF standard once it's proven.
Could you or someone else review it, and give some feedback on how usable it would be for archival purposes? The bytestream is still very much not frozen.

The current specifications can be found here: https://cyanreg.github.io/avtransport/

It's got some features which may be of interest:

• 64 bit presentation and duration timestamps
• Actual timebase
• Only container to include mandatory error correction data on the headers themselves
• Large FEC range, using RaptorQ
• Extra resilient headers (each FEC segment carries a piece of the main header)
• Flexible keyframe indexing
• Lossless cutting of compressed video (any frame dependencies can be made invisible)
• Stream packet data compression
• A reasonable uncompressed video and audio encapsulation
• For H264 and HEVC, a real 64-bit dts timestamp.
• ICC profile support
• Color lookup table support (currently, only Adobe .cube files, which are pretty much the standard)
• DNG support
• Timestamp clock, allowing for jitter compensation
• Unified, cycle-accurate clock for exact synchronization between multiple streams.

Whenever I wondered what I should implement, I just took a look at Mastroska spec issues you've opened :)

It's definitely not as lightweight as Matroska is, each stream packet header is 36 bytes. Also, unlike Matroska, there is no clear file structure - AVTransport is a stream of packets, nothing more. Reordering, deduplication, caching, and so on are handled by the demuxer.

Also, there is no per-packet CRC, or a CRC at all - every header field is protected by error correction codes (which can be used to check integrity as well). The codes are standard irregular LDPCs with a coding ratio of 0.78, able to correct a lot more than a few bit flips.
As for the stream data, the optional FEC is RaptorQ, which is absurdly efficient at both finding faults and fixing them.
FEC may be applied to both individual packets, and groups of packets (which may include multiple streams, this is the traditional way FEC is done).

It was designed to be robust enough to be saved on tape, or sent over pure serial/RF/satellite/optical links, although it's mainly going to be used for point-to-point/point-to-many internet transmissions, and files.

[cyanreg]

@JeromeMartinez ping

[JeromeMartinez]

@JeromeMartinez ping

No specific thoughts on my side, on the container/transport level we rely on Matroska for the moment and we don't see a need to focus on something else, for the moment, maybe except FEC (there is a draft of addition on top of Matroska for that).

[cyanreg]

Thanks. I wasn't really asking if you would consider switching, if Matroska works for you, that's fine. I was more or less wondering broadly if it would be usable for archival, feature-wise and reliability-wise.

[cyanreg]

@dericed Could you take a look at it as well, if you have time?
It doesn't have to be archival-specific, just a general review would be fine too.

The plan is for it to be an official FFmpeg/Videolan project (like RIST), and possibly send it to the IETF once it's mature and used enough.

[cyanreg]

@JeromeMartinez sorry to bother you, could you ping @dericed?