If another peer is sending unicast packets to the same port that
we are listening on, those packets can end up being received despite
using source specific multicast. For those cases, manually check the
source address of received packets against the intended source address.

This only handles the case when the source list is one single IP
address for now, which probably is the most common case.

Based on a patch by Ed Torbett.
Signed-off-by: Martin Storsjö <martin@martin.st>