Commit aacf6b3a authored by John Brooks's avatar John Brooks Committed by Kostya Shishkov

rv34: fix invalid memory access for small video dimensions

For small video dimensions calculations of the upper bound for pixel
access may result in negative value. Using an unsigned comparison
works only if the greater operand is non-negative. This is fixed by
doing edge emulation explicitly for such conditions.
Signed-off-by: 's avatarKostya Shishkov <kostya.shishkov@gmail.com>
parent 98f24ecd
......@@ -680,8 +680,9 @@ static inline void rv34_mc(RV34DecContext *r, const int block_type,
srcY += src_y * s->linesize + src_x;
srcU += uvsrc_y * s->uvlinesize + uvsrc_x;
srcV += uvsrc_y * s->uvlinesize + uvsrc_x;
if( (unsigned)(src_x - !!lx*2) > s->h_edge_pos - !!lx*2 - (width <<3) - 4
|| (unsigned)(src_y - !!ly*2) > s->v_edge_pos - !!ly*2 - (height<<3) - 4){
if(s->h_edge_pos - (width << 3) < 6 || s->v_edge_pos - (height << 3) < 6 ||
(unsigned)(src_x - !!lx*2) > s->h_edge_pos - !!lx*2 - (width <<3) - 4 ||
(unsigned)(src_y - !!ly*2) > s->v_edge_pos - !!ly*2 - (height<<3) - 4) {
uint8_t *uvbuf = s->edge_emu_buffer + 22 * s->linesize;
srcY -= 2 + 2*s->linesize;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment