fate.texi 4.73 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
\input texinfo @c -*- texinfo -*-

@settitle FATE Automated Testing Environment
@titlepage
@center @titlefont{FATE Automated Testing Environment}
@end titlepage

@top

@contents

@chapter Introduction

FATE provides a regression testsuite embedded within the Libav build system.
It can be run locally and optionally configured to send reports to a web
aggregator and viewer @url{http://fate.libav.org}.

It is advised to run FATE before submitting patches to the current codebase
and provide new tests when submitting patches to add additional features.

@chapter Running FATE

@section Samples and References
In order to run, FATE needs a large amount of data (samples and references)
that is provided separately from the actual source distribution.

To inform the build system about the testsuite location, pass
@option{--samples=<path to the samples>} to @command{configure} or set the
29
@var{SAMPLES} Make variable or the @var{LIBAV_SAMPLES} environment variable
30 31
to a suitable value.

32 33 34
To use a custom wrapper to run the test, pass @option{--target-exec} to
@command{configure} or set the @var{TARGET_EXEC} Make variable.

35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62
The dataset is available through @command{rsync}, is possible to fetch
the current sample using the straight rsync command or through a specific
@ref{Makefile target}.

@example
# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite
@end example

@example
# make fate-rsync SAMPLES=fate-suite
@end example


@chapter Manual Run
FATE regression test can be run through @command{make}.
Specific Makefile targets and Makefile variables are available:

@anchor{Makefile target}
@section FATE Makefile targets
@table @option
@item fate-list
List all fate/regression test targets.
@item fate-rsync
Shortcut to download the fate test samples to the specified testsuite location.
@item fate
Run the FATE test suite (requires the fate-suite dataset).
@end table

63
@section FATE Makefile variables
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
@table @option
@item V
Verbosity level, can be set to 0, 1 or 2.
@table @option
    @item 0
    show just the test arguments
    @item 1
    show just the command used in the test
    @item 2
    show everything
@end table
@item SAMPLES
Specify or override the path to the FATE samples at make time, it has a
meaning only while running the regression tests.
@item THREADS
Specify how many threads to use while running regression tests, it is
quite useful to detect thread-related regressions.
81 82 83
@item THREAD_TYPE
Specify which threading strategy test, either @var{slice} or @var{frame},
by default @var{slice+frame}
84 85
@item CPUFLAGS
Specify a mask to be applied to autodetected CPU flags.
86 87
@item TARGET_EXEC
Specify or override the wrapper used to run the tests.
88 89 90
@end table

@example
91
    make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
92 93 94 95 96 97
@end example

@chapter Automated Tests
In order to automatically testing specific configurations, e.g. multiple
compilers, @command{tests/fate.sh} is provided.

98 99 100
This shell script builds Libav, runs the regression tests and prepares
a report that can be sent to @url{http://fate.libav.org/} or directly
examined locally.
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141

@section Testing Profiles
The configuration file passed to @command{fate.sh} is shell scripts as well.

It must provide at least a @var{slot} identifier, the @var{repo} from
which fetch the sources, the @var{samples} directory, a @var{workdir} with
enough space to build and run all the tests.
Optional submit command @var{fate_recv} and a @var{comment} to describe
the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process
can be passed.

@example
slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@@fate.libav.org"  # command to submit report
comment=                                # optional description

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
cc=
target_os=
sysroot=
target_exec=
target_path=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'
@end example

142 143 144 145 146
@section Special Instances
The @var{TARGET_EXEC} option provides a way to run FATE wrapped in
@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets
through @command{ssh}.

147 148 149 150
@section Submitting Reports
In order to send reports you need to create an @command{ssh} key and send it
to @email{root@@libav.org}.
The current server fingerprint is @var{a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6}