avformat/matroskaenc: Don't assert when writing huge files

EBML numbers are variable length numbers: Only seven bits of every byte
are available to encode the number, the other bits encode the length of
the number itself. So an eight byte EBML number can only encode numbers
in the range 0..(2^56 - 1). And when using EBML numbers to encode the
length of an EBML element, the EBML number corresponding to 2^56 - 1 is
actually reserved to mean that the length of the corresponding element
is unknown.

And therefore put_ebml_length() asserted that the length it should
represent is < 2^56 - 1. Yet there was nothing that actually guaranteed
this to be true for the Segment (the main/root EBML element of a
Matroska file that encompasses nearly the whole file). This commit
changes this by checking in advance how big the length is and only
updating the number if it is representable at all; if not, the unknown
length element is not touched.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

libavformat/matroskaenc.c

@@ -2542,9 +2542,13 @@ static int mkv_write_trailer(AVFormatContext *s)
         }
 
     after_cues:
+    /* Lengths greater than (1ULL << 56) - 1 can't be represented
+     * via an EBML number, so leave the unknown length field. */
+    if (endpos - mkv->segment_offset < (1ULL << 56) - 1) {
         if ((ret64 = avio_seek(pb, mkv->segment_offset - 8, SEEK_SET)) < 0)
             return ret64;
         put_ebml_length(pb, endpos - mkv->segment_offset, 8);
+    }
 
         ret = mkv_write_seekhead(pb, mkv, 1, mkv->info.pos);
         if (ret < 0)