After f8990c5f we properly set non-blocking
mode which makes the connect() call return always 0 even if no connection can
be established.

Fix this by always doing a poll after calling connect(). Also there was some
leftover copy paste code which checks for various errors which are simply not
possible with SRT.
Signed-off-by: Marton Balint <cus@passwd.hu>