poll() is only used by networking code, so the fallback should
only be built if networking is enabled.  Also remove CONFIG_FFSERVER
condition from the declarations.

This should fix building on systems without poll(), broken
by a8475bbd.
Signed-off-by: Mans Rullgard <mans@mansr.com>