Commit 2f014567 authored by Xi Wang's avatar Xi Wang Committed by Michael Niedermayer

rtpenc: fix overflow checking in avc_mp4_find_startcode()

The check `start + res < start' is broken since pointer overflow is
undefined behavior in C.  Many compilers such as gcc/clang optimize
away this check.

Use `res > end - start' instead.  Also change `res' to unsigned int
to avoid signed left-shift overflow.
Signed-off-by: 's avatarXi Wang <xi.wang@gmail.com>
Signed-off-by: 's avatarMichael Niedermayer <michaelni@gmx.at>
parent 713dea58
......@@ -31,14 +31,14 @@
static const uint8_t *avc_mp4_find_startcode(const uint8_t *start, const uint8_t *end, int nal_length_size)
{
int res = 0;
unsigned int res = 0;
if (end - start < nal_length_size)
return NULL;
while (nal_length_size--)
res = (res << 8) | *start++;
if (start + res > end || res < 0 || start + res < start)
if (res > end - start)
return NULL;
return start + res;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment