Commit bb4c9d0a authored by Philip Langdale's avatar Philip Langdale

avcodec: Don't assume separate u and v planes in ff_alloc_picture

alloc_frame_buffer in ff_alloc_picture asserts that the linesize
of planes 1 and 2 are the same. If the pixfmt has a single uv
plane, like NV12, this won't be true.

So, let's only do this check if there are more than 2 planes.

We never hit this with previous hw formats because they don't set
linesize to meaningful values, but the cuda hw format sets the
values based on the underlying data layout.
parent 54f8ac19
......@@ -22,6 +22,7 @@
#include "libavutil/avassert.h"
#include "libavutil/common.h"
#include "libavutil/pixdesc.h"
#include "avcodec.h"
#include "motion_est.h"
......@@ -155,7 +156,8 @@ static int alloc_frame_buffer(AVCodecContext *avctx, Picture *pic,
return -1;
}
if (pic->f->linesize[1] != pic->f->linesize[2]) {
if (av_pix_fmt_count_planes(pic->f->format) > 2 &&
pic->f->linesize[1] != pic->f->linesize[2]) {
av_log(avctx, AV_LOG_ERROR,
"get_buffer() failed (uv stride mismatch)\n");
ff_mpeg_unref_picture(avctx, pic);
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment