Input on/off state can change in request_samples(), which can result in
a state where only the first input is active. get_available_samples()
will then return 0, and request_frame() will fail with EAGAIN even
though there is data on the single active input.

Take this into account and check the number of active inputs again after
calling request_samples().