Also don't include it on files that don't need it.

This reduces differences with libav
Tested-by: Timothy Gu <timothygu99@gmail.com>
Reveiwed-by: Carl Eugen Hoyos <ceffmpeg@gmail.com>
Signed-off-by: James Almer <jamrial@gmail.com>