fate.texi

\input texinfo @c -*- texinfo -*-

@settitle FATE Automated Testing Environment
@titlepage
@center @titlefont{FATE Automated Testing Environment}
@end titlepage

@top

@contents

@chapter Introduction

FATE provides a regression testsuite embedded within the Libav build system.
It can be run locally and optionally configured to send reports to a web
aggregator and viewer @url{http://fate.libav.org}.

It is advised to run FATE before submitting patches to the current codebase
and provide new tests when submitting patches to add additional features.

@chapter Running FATE

@section Samples and References
In order to run, FATE needs a large amount of data (samples and references)
that is provided separately from the actual source distribution.

To inform the build system about the testsuite location, pass
@option{--samples=<path to the samples>} to @command{configure} or set the
@var{SAMPLES} Make variable or the @var{LIBAV_SAMPLES} environment variable
to a suitable value.

To use a custom wrapper to run the test, pass @option{--target-exec} to
@command{configure} or set the @var{TARGET_EXEC} Make variable.

The dataset is available through @command{rsync}, is possible to fetch
the current sample using the straight rsync command or through a specific
@ref{Makefile target}.

@example
# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite
@end example

@example
# make fate-rsync SAMPLES=fate-suite
@end example


@chapter Manual Run
FATE regression test can be run through @command{make}.
Specific Makefile targets and Makefile variables are available:

@anchor{Makefile target}
@section FATE Makefile targets

@table @option
@item fate-list
List all fate/regression test targets.

@item fate-rsync
Shortcut to download the fate test samples to the specified testsuite location.

@item fate
Run the FATE test suite (requires the fate-suite dataset).
@end table

@section FATE Makefile variables
@table @option
@item V
Verbosity level, can be set to 0, 1 or 2.

@table @option
@item 0
show just the test arguments

@item 1
show just the command used in the test

@item 2
show everything
@end table

@item SAMPLES
Specify or override the path to the FATE samples at make time, it has a
meaning only while running the regression tests.

@item THREADS
Specify how many threads to use while running regression tests, it is
quite useful to detect thread-related regressions.

@item THREAD_TYPE
Specify which threading strategy test, either @var{slice} or @var{frame},
by default @var{slice+frame}

@item CPUFLAGS
Specify a mask to be applied to autodetected CPU flags.

@item TARGET_EXEC
Specify or override the wrapper used to run the tests.

@item GEN
Set to @var{1} to generate the missing or mismatched references.
@end table

@example
    make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
@end example

@chapter Automated Tests
In order to automatically testing specific configurations, e.g. multiple
compilers, @command{tests/fate.sh} is provided.

This shell script builds Libav, runs the regression tests and prepares
a report that can be sent to @url{http://fate.libav.org/} or directly
examined locally.

@section Testing Profiles
The configuration file passed to @command{fate.sh} is shell scripts as well.

It must provide at least a @var{slot} identifier, the @var{repo} from
which fetch the sources, the @var{samples} directory, a @var{workdir} with
enough space to build and run all the tests.
Optional submit command @var{fate_recv} and a @var{comment} to describe
the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process
can be passed.

@example
slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
#branch=release/10                      # the branch to test
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@@fate.libav.org"  # command to submit report
comment=                                # optional description
build_only=     # set to "yes" for a compile-only instance that skips tests

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
as=
cc=
ld=
target_os=
sysroot=
target_exec=
target_path=
target_samples=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'
@end example

@section Special Instances
The @var{TARGET_EXEC} option provides a way to run FATE wrapped in
@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets
through @command{ssh}.

@section Submitting Reports
In order to send reports you need to create an @command{ssh} key and send it
to @email{root@@libav.org}.