Commit def04022 authored by Andreas Rheinhardt's avatar Andreas Rheinhardt Committed by Michael Niedermayer

avcodec/zmbvenc: Correct offset in buffer

zmbvenc allocates a buffer for a picture with padding on all four sides:
The stride is choosen so large that it already contains padding on the
right; the height also includes padding rows. The padding on the right
of each row is also reused as padding for the left of the next row. So
one still needs to add padding on the left for the first row. This is done
by offsetting the actual pointer used to access the picture from the
pointer returned by av_mallocz and the formula for this offset was
wrong, because it ignored that a pixel can take more than one byte when
calculating the offset resulting from the left padding of the first row.

This fixes accesses outside of the allocated buffer that were reported
in tickets #7980 and #7994. No writes were ever attempted outside of
the buffer.
Signed-off-by: 's avatarAndreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: 's avatarTomas Härdin <tjoppen@acc.umu.se>
Signed-off-by: 's avatarMichael Niedermayer <michael@niedermayer.cc>
parent 223a2c2a
...@@ -409,7 +409,7 @@ static av_cold int encode_init(AVCodecContext *avctx) ...@@ -409,7 +409,7 @@ static av_cold int encode_init(AVCodecContext *avctx)
*/ */
c->pstride = FFALIGN((avctx->width + c->lrange) * c->bypp, 16); c->pstride = FFALIGN((avctx->width + c->lrange) * c->bypp, 16);
prev_size = FFALIGN(c->lrange * c->bypp, 16) + c->pstride * (c->lrange + avctx->height + c->urange); prev_size = FFALIGN(c->lrange * c->bypp, 16) + c->pstride * (c->lrange + avctx->height + c->urange);
prev_offset = FFALIGN(c->lrange, 16) + c->pstride * c->lrange; prev_offset = FFALIGN(c->lrange * c->bypp, 16) + c->pstride * c->lrange;
if (!(c->prev_buf = av_mallocz(prev_size))) { if (!(c->prev_buf = av_mallocz(prev_size))) {
av_log(avctx, AV_LOG_ERROR, "Can't allocate picture.\n"); av_log(avctx, AV_LOG_ERROR, "Can't allocate picture.\n");
return AVERROR(ENOMEM); return AVERROR(ENOMEM);
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment