Previously, the function would lead to an infinite wait (by
returning AVERROR(EAGAIN)) on sockets indicating an error
via either of these poll flags.
Signed-off-by: Martin Storsjö <martin@martin.st>