nvenc doesn't support P016, but we have two problems today:

1) We declare support for YUV444P16 which nvenc also doesn't support.
   We do this because it's the only pix_fmt we have that can
   approximate nvenc's internal format that is YUV444P10 with data in
   MSBs instead of LSBs. Because the declared format is a 16bit one,
   it will be preferrentially chosen when encoding >10bit content,
   but that content will normally be YUV420P12 or P016 which should
   get mapped to P010 and not YUV444P10.

2) Transcoding P016 content with nvenc should be possible in a pure
   hardware pipeline, and that can't be done if nvenc doesn't say it
   accepts P016. By mapping it to P010, we can use it, albeit with
   truncation. I have established that swscale doesn't know how to
   dither to 10bits so we'd get truncation anyway, even if we tried
   to do this 'properly'.

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Currently the resource is only ever unregistered when the
registered_frames array is fully in use and an unmapped entry is re-used
and cleaned up.
I'm pretty sure the frame will have been cleaned up before that happens,
so I'm kinda surprised this never blew up.
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

If some logic like vsync in ffmpeg.c duplicates frames, it might pass
the same frame twice, which will result in a crash due it being
effectively mapped and unmapped twice.
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

In function process_output_surface(), the return value is 0 on the path
that av_mallocz() returns a NULL pointer. 0 indicates success, which
deviates from the fact. Return "AVERROR(ENOMEM)" instead of "0".
Signed-off-by: Pan Bian <bianpan2016@163.com>
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Reviewed-by: Timo Rothenpieler <timo@rothenpieler.org>

This reduces code repetition, and will allow adding further push/pop
refinement for D3D11 devices in future commits.

It is detected as a part of the libdl check.

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Interlaced encoding profits from it, or might even need it in some
players.
No harm in enabling it unconditionally.
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

The previous default sets the allocated surfaces to 32 unless it is
user-overridden or the lookahead parameter is set.

Change the surfaces calculation for default, B-frames and lookahead scenario.

Previously, if a session allocates x surfaces, only x-1 surfaces are
used (due to combination of output delay and lock toggle logic).

hw accelerated transcode (h264_cuvid -> h264_nvenc with -hwaccel cuvid) was
broken after the filtergraph initialization was changed to intialize decoder
first followed by encoder (commit af1761f7).
During initialzing encoder with bframes, local buffers are allocated
internally in encoder which fails since no cuda context is available. Now
pushing the correct cuda context before encoder initialization fixes the issue.
Also adding push/pop cuda ctx during create/destroy/map/unmap resources and
destroy encoder session.
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Fixes #6260

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

This patch aims to reduce the number of input/output surfaces
NVENC allocates per session. Previous default sets allocated surfaces to 32
(unless there is user specified param or lookahead involved). Having large
number of surfaces consumes extra video memory (esp for higher resolution
encoding). The patch changes the surfaces calculation for default, B-frames,
lookahead scenario respectively.

The other change involves surface selection. Previously, if a session
allocates x surfaces, only x-1 surfaces are used (due to combination
of output delay and lock toggle logic). To prevent unused surfaces,
changing surface rotation to using predefined fifo.
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Diego Biurrun <diego@biurrun.de>

qmin and qmax are not necessary for nvenc vbr.

Also fix for using 2 pass vbr mode for slow preset through ctx->flag NVENC_TWO_PASSES.
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

qmin and qmax are not necessary for nvenc vbr.
Enforcing this constraint, doesn't allow user to use vbr 2 pass mode without explicity setting the qmin and qmax options
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>

The map is a sparse array and does not need a empty element to terminate
it.

The empty element is stored after the last one inserted in the list,
overwriting whichever element was next with zeros.

Bug-Id: 1029
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

This error is treated specially by the API.

CC: libav-stable@libav.org