asfdec_o: reject size > INT64_MAX in asf_read_unknown

Both avio_skip and detect_unknown_subobject use int64_t for the size
parameter.

This fixes a segmentation fault due to infinite recursion.
Reviewed-by: Alexandra Hájková <alexandra.khirnova@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

libavformat/asfdec_o.c

@@ -178,6 +178,9 @@ static int asf_read_unknown(AVFormatContext *s, const GUIDParseTable *g)
     uint64_t size   = avio_rl64(pb);
     int ret;
 
+    if (size > INT64_MAX)
+        return AVERROR_INVALIDDATA;
+
     if (asf->is_header)
         asf->unknown_size = size;
     asf->is_header = 0;