If we've received packets on the same socket before, the return
packets are sent to that address. If we've only received packets
on the other socket, try to guess the source port for the other
one assuming the basic +1/-1 logic.
Signed-off-by: Martin Storsjö <martin@martin.st>