visualtest/README.txt
author Philipp Wiesemann <philipp.wiesemann@arcor.de>
Wed, 17 Aug 2016 21:04:50 +0200
changeset 10275 ba7dc3ebb81e
parent 7924 fcb86d323770
permissions -rwxr-xr-x
Emscripten: Fixed opening previously closed joystick.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
7924
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     1
/*!
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     2
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     3
\mainpage Visual and Interactive Test Automation for SDL 2.0
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     4
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     5
\section license_sec License
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     6
Check the file \c COPYING.txt for licensing information.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     7
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     8
\section intro_sec Introduction
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     9
The goal of this GSoC project is to automate the testing of testsprite2.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    10
testsprite2 takes 26 parameters which have thousands of valid combinations and is
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    11
used to validate SDL's window, mouse and rendering behaviour. By having a test
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    12
harness that runs testsprite2 with various command line argument strings and
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    13
validates the output for each run, we can make testing an easier task for
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    14
maintainers, contributors and testers. The test harness can be used by a continuous
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    15
integration system (like buildbot or jenkins) to validate SDL after checkins.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    16
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    17
SDL Homepage: http://libsdl.org/
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    18
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    19
\section build_sec Building
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    20
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    21
\subsection build_linux Building on Linux/Cygwin
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    22
<tt>./autogen.sh; ./configure; make;</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    23
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    24
\subsection build_windows Building on Windows
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    25
Use the Visual Studio solution under \c SDL/VisualC/visualtest.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    26
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    27
\section docs_sec Documentation
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    28
Documentation is available via Doxygen. To build the documentation, cd to the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    29
SDL/visualtest/docs directory and run \c doxygen. A good starting point for
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    30
exploring the documentation is \c SDL/visualtest/docs/html/index.html
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    31
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    32
\section usage_sec Usage
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    33
To see all the options supported by the test harness, just run \c testharness
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    34
with no arguments.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    35
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    36
At the moment the following options are supported:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    37
\li \c sutapp - Path to the system under test (SUT) application
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    38
\li \c sutargs - Launch the SUT with the specified arguments string
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    39
\li \c timeout - The maximum time after which the SUT process will be killed;
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    40
	passed as hh:mm:ss; default 00:01:00
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    41
\li \c variator - Which variator to use; see \ref variators_sec
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    42
\li \c num-variations - The number of variations to run for; taken to be 
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    43
	1 for the random variator and ALL for the exhaustive variator by default
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    44
\li \c no-launch - Just print the arguments string for each variation without
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    45
	launching the SUT or performing any actions
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    46
\li \c parameter-config - A config file that describes the command line parameters
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    47
	supported by the SUT; see \ref paramconfig_sec or the sample *.parameters files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    48
	for more details
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    49
\li \c action-config - A config file with a list of actions to be performed while
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    50
	the SUT is running; see \ref actionconfig_sec or the sample *.actions files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    51
\li \c output-dir - Path to the directory where screenshots should be saved; is
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    52
	created if it doesn't exist; taken to be "./output" by default
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    53
\li \c verify-dir - Path to the directory with the verification images; taken to 
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    54
	be "./verify" by default
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    55
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    56
Paths can be relative or absolute.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    57
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    58
Alternatively, the options can be passed as a config file for convenience:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    59
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    60
<tt>testharness \-\-config testsprite2_sample.config</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    61
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    62
For a sample, take a look at the *.config files in this repository.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    63
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    64
We can also pass a config file and override certain options as necessary:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    65
<tt>testharness \-\-config testsprite2_sample.config \-\-num-variations 10</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    66
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    67
Note: You may find it convenient to copy the SUT executable along with any
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    68
resources to the test harness directory. Also note that testsprite2 and its
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    69
resources (icon.bmp) are automatically copied when using the Visual Studio
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    70
solution.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    71
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    72
\subsection usageexamples_subsec Usage examples:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    73
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    74
Passing a custom arguments string:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    75
<tt>testharness \-\-sutapp testsprite2 \-\-sutargs "\-\-cyclecolor \-\-blend mod
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    76
\-\-iterations 2" \-\-action-config xyz.actions</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    77
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    78
Using the random variator:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    79
<tt>testharness \-\-sutapp testsprite2 \-\-variator random \-\-num-variations 5
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    80
\-\-parameter-config xyz.parameters \-\-action-config xyz.actions</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    81
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    82
\subsection config_subsec Config Files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    83
Config files are an alternate way to pass parameters to the test harness. We
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    84
describe the paramters in a config file and pass that to the test harness using
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    85
the \-\-config option. The config file consists of lines of the form "x=y" where
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    86
x is an option and y is it's value. For boolean options, we simply give the name
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    87
of the option to indicate that it is to be passed to the testharness.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    88
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    89
The hash '#' character can be used to start a comment from that point to the end
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    90
of the line.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    91
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    92
\section paramconfig_sec The SUT Parameters File
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    93
To generate variations we need to describe the parameters the will be passed to
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    94
the SUT. This description is given in a parameters file. Each line of the parameters
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    95
file (except the blank lines) represents one command line option with five
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    96
comma separated fields:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    97
<tt>name, type, values, required, categories</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    98
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    99
\li \c name is the name of the option, e.g., \c \-\-cyclecolor.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   100
\li \c type can have one of three values - integer, boolean and enum.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   101
\li \c values - for integer options this is the valid range of values the option
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   102
	can take, i.e., [min max]. For enum options this is a list of strings that
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   103
	the option can take, e.g., [val1 val2 val3]. For boolean options this field
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   104
	is ignored.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   105
\li \c required - true if the option is required, false otherwise.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   106
\li \c categories - a list of categories that the option belongs to. For example,
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   107
	[video mouse audio]
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   108
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   109
Just like with config files, hash characters can be used to start comments.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   110
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   111
\subsection additionalnotes_subsec Additional Notes
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   112
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   113
\li If you want to have an option that always takes a certain value, use an enum
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   114
	with only one value.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   115
\li Currently there isn't any way to turn an option off, i.e., all options will
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   116
	be included in the command line options string that is generated using the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   117
	config. If you don't want an option to be passed to the SUT, remove it from
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   118
	the config file or comment it out.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   119
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   120
\section variators_sec Variators
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   121
Variators are the mechanism by which we generate strings of command line arguments
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   122
to test the SUT with. A variator is quite simply an iterator that iterates through
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   123
different variations of command line options. There are two variators supported at
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   124
the moment:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   125
\li \b Exhaustive - Generate all possible combinations of command line arguments
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   126
	that are valid.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   127
\li \b Random - Generate a random variation each time the variator is called.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   128
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   129
As an example, let's try a simple .parameters file:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   130
<tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   131
\-\-blend, enum, [add mod], false, [] \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   132
\-\-fullscreen, boolean, [], false, []
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   133
</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   134
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   135
The exhaustive variator would generate the following four variations:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   136
<tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   137
\-\-blend add \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   138
\-\-blend mod \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   139
\-\-blend add \-\-fullscreen \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   140
\-\-blend mod \-\-fullscreen \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   141
</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   142
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   143
The random variator would simply generate a random variation like the following:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   144
<tt>\-\-blend mod</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   145
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   146
\section actionconfig_sec The Actions File
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   147
Once the SUT process has been launched, automated testing happens using a mechanism
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   148
called actions. A list of actions is read from a file and each action is performed
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   149
on the SUT process sequentially. Each line in the actions file describes an action.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   150
The format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   151
There are five actions supported at the moment:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   152
\li \b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   153
	images are saved as \c [hash]_[i].bmp where \c [hash] is the 32 character long
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   154
	hexadecimal MD5 hash of the arguments string that was passed to the SUT while
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   155
	launching it and \c i is the window number. i = 1 is an exceptional case
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   156
	where the \c _[i] is dropped and the filename is simply \c [hash].bmp\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   157
	Note: The screenshots are only of the window's client area.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   158
\li \b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   159
	comparing them against a verification image. Each \c [hash]_i.bmp image output
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   160
	by the SCREENSHOT action is compared against a \c [hash].bmp image in the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   161
	verify-dir.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   162
\li \b QUIT - Gracefully quits the SUT process. On Windows this means sending a
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   163
	WM_CLOSE message to each window owned by the SUT process. On Linux it means
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   164
	sending a SIGQUIT signal to the SUT process.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   165
\li \b KILL - Forcefully kills the SUT process. This is useful when the SUT process
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   166
	doesn't respond to the QUIT action.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   167
\li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \c [args]
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   168
	as the arguments string.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   169
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   170
Just like with config files, hash characters can be used to start comments.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   171
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   172
\section contint_sec Continuous Integration (CI)
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   173
One of the goals of the project was to create a test harness that integrates
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   174
with CI systems to provide automated visual and interactive testing to SDL.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   175
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   176
At the moment the test harness can be run in two modes that are useful for CI:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   177
\li Crash testing mode - launch the SUT with every variation and all parameters,
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   178
	report to the CI if there's a crash
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   179
\li Visual testing mode - launch and visually verify the SUT for a smaller subset
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   180
	of the parameters
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   181
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   182
Look at the launch_harness.sh/launch_harness.cmd for an example scripts that run the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   183
test harness for all variations with all parameters and report an error on a crash.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   184
The script uses the testsprite2_crashtest config, so remember to copy those files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   185
over to the test harness executable directory along with the script.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   186
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   187
\section todo_sec TODOs
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   188
\li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   189
	VERIFY x, y, w, h
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   190
\li Add support for spaces between the equals sign in test harness config files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   191
\li Implement the SCREENSHOT action on Linux
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   192
\li Add a pairwise variator
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   193
\li Add actions to inject keyboard/mouse events
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   194
\li Add actions to manipulate the SUT window, e.g., minimize, restore, resize
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   195
\li Add support to load and save screenshots as .pngs instead of .bmps
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   196
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   197
\section issues_sec Known Issues
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   198
\li The QUIT action does not work on a testsprite2 process with multiple windows.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   199
	This appears to be an issue with testsprite2.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   200
\li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   201
	--fullscreen option is supplied. It works with --fullscreen-desktop, however.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   202
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   203
\section moreinfo_sec More Information
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   204
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   205
Author Contact Info:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   206
Apoorv Upreti \c \<apoorvupreti@gmail.com\>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   207
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   208
Other useful links:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   209
- Project Repository: https://bitbucket.org/nerdap/sdlvisualtest
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   210
- Project Wiki: https://github.com/nerdap/autotestsprite2/wiki
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   211
- Project Blog: http://nerdap.github.io
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   212
- Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   213
- Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   214
*/