avcodec/nvenc: split large functions into smaller ones

Functions names and scopes are from libav. This commit basically moves
code around without changing it; it shouldn't change any functionality
except some small leak fixes on error paths.
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>