h264_mp4toannexb: Don't forget numOfPictureParameterSets

The format of an AVCDecoderConfigurationRecord, the out-of-band
extradata of H.264 in mp4, is as follows: First four bytes containing
version, profile and level, one byte for the length size and one byte
each for the number of SPS, followed by the SPS (each with its own size
field), followed by a byte containing the number of PPS followed by the
PPS with their size fields. While the number of SPS/PPS may be zero, the
bytes containing these numbers are mandatory. Yet the byte containing
the number of PPS has been ignored in two places:
1. In the initial check for whether the extradata can contain an
AVCDecoderConfigurationRecord. The minimum size is 7, not 6.
2. No check is made for whether the extradata ended right after the last
byte of the last SPS of the SPS array. Instead the first byte of the
padding is read as if it were part of the extradata and contained the
number of PPS (namely zero, given that the padding is zeroed). No error
or warning was ever raised.
This has been changed. Such truncated extradata is now considered
invalid; the check for 2. has been incorporated into the general size
check.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

libavcodec/h264_mp4toannexb_bsf.c

@@ -102,8 +102,8 @@ static int h264_extradata_to_annexb(AVBSFContext *ctx, const int padding)
         unit_size   = bytestream2_get_be16u(gb);
         total_size += unit_size + 4;
         av_assert1(total_size <= INT_MAX - padding);
-        if (bytestream2_get_bytes_left(gb) < unit_size) {
-            av_log(ctx, AV_LOG_ERROR, "Packet header is not contained in global extradata, "
+        if (bytestream2_get_bytes_left(gb) < unit_size + !sps_done) {
+            av_log(ctx, AV_LOG_ERROR, "Global extradata truncated, "
                    "corrupted stream or invalid MP4/AVCC bitstream\n");
             av_free(out);
             return AVERROR_INVALIDDATA;
@@ -154,7 +154,7 @@ static int h264_mp4toannexb_init(AVBSFContext *ctx)
         (extra_size >= 4 && AV_RB32(ctx->par_in->extradata) == 1)) {
         av_log(ctx, AV_LOG_VERBOSE,
                "The input looks like it is Annex B already\n");
-    } else if (extra_size >= 6) {
+    } else if (extra_size >= 7) {
         ret = h264_extradata_to_annexb(ctx, AV_INPUT_BUFFER_PADDING_SIZE);
         if (ret < 0)
             return ret;