This makes libavcodec/libx264.c behave more similar to the x264 command line
util

Fixes Ticket3307

Implementation based on x264
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>