fate.texi 6.59 KB
Newer Older
1 2
\input texinfo @c -*- texinfo -*-

3
@settitle FFmpeg Automated Testing Environment
4
@titlepage
5
@center @titlefont{FFmpeg Automated Testing Environment}
6 7
@end titlepage

8
@node Top
9 10 11 12 13 14
@top

@contents

@chapter Introduction

15 16
  FATE is an extended regression suite on the client-side and a means
for results aggregation and presentation on the server-side.
17

18 19 20 21
  The first part of this document explains how you can use FATE from
your FFmpeg source directory to test your ffmpeg binary. The second
part describes how you can run FATE to submit the results to FFmpeg's
FATE server.
22

23 24
  In any way you can have a look at the publicly viewable FATE results
by visiting this website:
25

26
  @url{http://fate.ffmpeg.org/}
27

28 29
  This is especially recommended for all people contributing source
code to FFmpeg, as it can be seen if some test on some platform broke
30
with their recent contribution. This usually happens on the platforms
31
the developers could not test on.
32

33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
  The second part of this document describes how you can run FATE to
submit your results to FFmpeg's FATE server. If you want to submit your
results be sure to check that your combination of CPU, OS and compiler
is not already listed on the above mentioned website.

  In the third part you can find a comprehensive listing of FATE makefile
targets and variables.


@chapter Using FATE from your FFmpeg source directory

  If you want to run FATE on your machine you need to have the samples
in place. You can get the samples via the build target fate-rsync.
Use this command from the top-level source directory:

@example
make fate-rsync SAMPLES=fate-suite/
make fate       SAMPLES=fate-suite/
@end example

  The above commands set the samples location by passing a makefile
variable via command line. It is also possible to set the samples
location at source configuration time by invoking configure with
`--samples=<path to the samples directory>'. Afterwards you can
invoke the makefile targets without setting the SAMPLES makefile
variable. This is illustrated by the following commands:
59 60

@example
61 62 63
./configure --samples=fate-suite/
make fate-rsync
make fate
64 65
@end example

66 67 68 69 70 71
  Yet another way to tell FATE about the location of the sample
directory is by making sure the environment variable FATE_SAMPLES
contains the path to your samples directory. This can be achieved
by e.g. putting that variable in your shell profile or by setting
it in your interactive session.

72
@example
73
FATE_SAMPLES=fate-suite/ make fate
74 75
@end example

76 77 78 79 80
@float NOTE
Do not put a '~' character in the samples path to indicate a home
directory. Because of shell nuances, this will cause FATE to fail.
@end float

81 82 83
To use a custom wrapper to run the test, pass @option{--target-exec} to
@command{configure} or set the @var{TARGET_EXEC} Make variable.

84 85 86 87

@chapter Submitting the results to the FFmpeg result aggregation server

  To submit your results to the server you should run fate through the
88
shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs
89 90 91 92 93 94 95
to be invoked with a configuration file as its first argument.

@example
tests/fate.sh /path/to/fate_config
@end example

  A configuration file template with comments describing the individual
96
configuration variables can be found at @file{doc/fate_config.sh.template}.
97 98 99

@ifhtml
  The mentioned configuration template is also available here:
100
@verbatiminclude fate_config.sh.template
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123
@end ifhtml

  Create a configuration that suits your needs, based on the configuration
template. The `slot' configuration variable can be any string that is not
yet used, but it is suggested that you name it adhering to the following
pattern <arch>-<os>-<compiler>-<compiler version>. The configuration file
itself will be sourced in a shell script, therefore all shell features may
be used. This enables you to setup the environment as you need it for your
build.

  For your first test runs the `fate_recv' variable should be empty or
commented out. This will run everything as normal except that it will omit
the submission of the results to the server. The following files should be
present in $workdir as specified in the configuration file:

@itemize
    @item configure.log
    @item compile.log
    @item test.log
    @item report
    @item version
@end itemize

124 125
  When you have everything working properly you can create an SSH key pair
and send the public key to the FATE server administrator who can be contacted
126
at the email address @email{fate-admin@@ffmpeg.org}.
127 128 129 130 131 132 133

  Configure your SSH client to use public key authentication with that key
when connecting to the FATE server. Also do not forget to check the identity
of the server and to accept its host key. This can usually be achieved by
running your SSH client manually and killing it after you accepted the key.
The FATE server's fingerprint is:

134 135 136 137 138 139
@table @option
@item RSA
   d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51
@item ECDSA
   76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86
@end table
140

141 142 143 144 145
  If you have problems connecting to the FATE server, it may help to try out
the @command{ssh} command with one or more @option{-v} options. You should
get detailed output concerning your SSH configuration and the authentication
process.

146 147 148 149 150 151 152
  The only thing left is to automate the execution of the fate.sh script and
the synchronisation of the samples directory.


@chapter FATE makefile targets and variables

@section Makefile targets
153 154 155

@table @option
@item fate-rsync
156
Download/synchronize sample files to the configured samples directory.
157 158

@item fate-list
159
Will list all fate/regression test targets.
160

161 162 163 164
@item fate
Run the FATE test suite (requires the fate-suite dataset).
@end table

165 166
@section Makefile variables

167 168 169
@table @option
@item V
Verbosity level, can be set to 0, 1 or 2.
170 171 172 173 174 175
    @itemize
        @item 0: show just the test arguments
        @item 1: show just the command used in the test
        @item 2: show everything
    @end itemize

176 177 178
@item SAMPLES
Specify or override the path to the FATE samples at make time, it has a
meaning only while running the regression tests.
179

180 181 182
@item THREADS
Specify how many threads to use while running regression tests, it is
quite useful to detect thread-related regressions.
183

184 185 186
@item THREAD_TYPE
Specify which threading strategy test, either @var{slice} or @var{frame},
by default @var{slice+frame}
187

188
@item CPUFLAGS
189
Specify CPU flags.
190

191 192
@item TARGET_EXEC
Specify or override the wrapper used to run the tests.
193 194 195
The @var{TARGET_EXEC} option provides a way to run FATE wrapped in
@command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets
through @command{ssh}.
196

197 198
@item GEN
Set to @var{1} to generate the missing or mismatched references.
199 200
@end table

201 202
@section Examples

203
@example
204
make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
205
@end example