visualtest/README.txt
author Ryan C. Gordon <icculus@icculus.org>
Fri, 12 Aug 2016 19:59:00 -0400
changeset 10266 c09f06c4e8c8
parent 7924 fcb86d323770
permissions -rwxr-xr-x
emscripten: send fake mouse events for touches, like other targets do. (This really should be handled at the higher level and not in the individual targets, but this fixes the immediate bug.)
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
7924
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     1
/*!
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     2
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     3
\mainpage Visual and Interactive Test Automation for SDL 2.0
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     4
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     5
\section license_sec License
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     6
Check the file \c COPYING.txt for licensing information.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     7
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     8
\section intro_sec Introduction
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     9
The goal of this GSoC project is to automate the testing of testsprite2.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    10
testsprite2 takes 26 parameters which have thousands of valid combinations and is
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    11
used to validate SDL's window, mouse and rendering behaviour. By having a test
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    12
harness that runs testsprite2 with various command line argument strings and
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    13
validates the output for each run, we can make testing an easier task for
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    14
maintainers, contributors and testers. The test harness can be used by a continuous
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    15
integration system (like buildbot or jenkins) to validate SDL after checkins.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    16
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    17
SDL Homepage: http://libsdl.org/
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    18
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    19
\section build_sec Building
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    20
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    21
\subsection build_linux Building on Linux/Cygwin
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    22
<tt>./autogen.sh; ./configure; make;</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    23
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    24
\subsection build_windows Building on Windows
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    25
Use the Visual Studio solution under \c SDL/VisualC/visualtest.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    26
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    27
\section docs_sec Documentation
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    28
Documentation is available via Doxygen. To build the documentation, cd to the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    29
SDL/visualtest/docs directory and run \c doxygen. A good starting point for
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    30
exploring the documentation is \c SDL/visualtest/docs/html/index.html
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    31
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    32
\section usage_sec Usage
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    33
To see all the options supported by the test harness, just run \c testharness
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    34
with no arguments.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    35
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    36
At the moment the following options are supported:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    37
\li \c sutapp - Path to the system under test (SUT) application
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    38
\li \c sutargs - Launch the SUT with the specified arguments string
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    39
\li \c timeout - The maximum time after which the SUT process will be killed;
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    40
	passed as hh:mm:ss; default 00:01:00
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    41
\li \c variator - Which variator to use; see \ref variators_sec
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    42
\li \c num-variations - The number of variations to run for; taken to be 
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    43
	1 for the random variator and ALL for the exhaustive variator by default
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    44
\li \c no-launch - Just print the arguments string for each variation without
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    45
	launching the SUT or performing any actions
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    46
\li \c parameter-config - A config file that describes the command line parameters
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    47
	supported by the SUT; see \ref paramconfig_sec or the sample *.parameters files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    48
	for more details
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    49
\li \c action-config - A config file with a list of actions to be performed while
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    50
	the SUT is running; see \ref actionconfig_sec or the sample *.actions files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    51
\li \c output-dir - Path to the directory where screenshots should be saved; is
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    52
	created if it doesn't exist; taken to be "./output" by default
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    53
\li \c verify-dir - Path to the directory with the verification images; taken to 
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    54
	be "./verify" by default
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    55
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    56
Paths can be relative or absolute.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    57
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    58
Alternatively, the options can be passed as a config file for convenience:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    59
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    60
<tt>testharness \-\-config testsprite2_sample.config</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    61
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    62
For a sample, take a look at the *.config files in this repository.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    63
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    64
We can also pass a config file and override certain options as necessary:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    65
<tt>testharness \-\-config testsprite2_sample.config \-\-num-variations 10</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    66
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    67
Note: You may find it convenient to copy the SUT executable along with any
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    68
resources to the test harness directory. Also note that testsprite2 and its
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    69
resources (icon.bmp) are automatically copied when using the Visual Studio
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    70
solution.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    71
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    72
\subsection usageexamples_subsec Usage examples:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    73
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    74
Passing a custom arguments string:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    75
<tt>testharness \-\-sutapp testsprite2 \-\-sutargs "\-\-cyclecolor \-\-blend mod
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    76
\-\-iterations 2" \-\-action-config xyz.actions</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    77
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    78
Using the random variator:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    79
<tt>testharness \-\-sutapp testsprite2 \-\-variator random \-\-num-variations 5
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    80
\-\-parameter-config xyz.parameters \-\-action-config xyz.actions</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    81
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    82
\subsection config_subsec Config Files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    83
Config files are an alternate way to pass parameters to the test harness. We
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    84
describe the paramters in a config file and pass that to the test harness using
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    85
the \-\-config option. The config file consists of lines of the form "x=y" where
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    86
x is an option and y is it's value. For boolean options, we simply give the name
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    87
of the option to indicate that it is to be passed to the testharness.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    88
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    89
The hash '#' character can be used to start a comment from that point to the end
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    90
of the line.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    91
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    92
\section paramconfig_sec The SUT Parameters File
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    93
To generate variations we need to describe the parameters the will be passed to
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    94
the SUT. This description is given in a parameters file. Each line of the parameters
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    95
file (except the blank lines) represents one command line option with five
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    96
comma separated fields:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    97
<tt>name, type, values, required, categories</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    98
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    99
\li \c name is the name of the option, e.g., \c \-\-cyclecolor.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   100
\li \c type can have one of three values - integer, boolean and enum.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   101
\li \c values - for integer options this is the valid range of values the option
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   102
	can take, i.e., [min max]. For enum options this is a list of strings that
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   103
	the option can take, e.g., [val1 val2 val3]. For boolean options this field
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   104
	is ignored.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   105
\li \c required - true if the option is required, false otherwise.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   106
\li \c categories - a list of categories that the option belongs to. For example,
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   107
	[video mouse audio]
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   108
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   109
Just like with config files, hash characters can be used to start comments.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   110
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   111
\subsection additionalnotes_subsec Additional Notes
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   112
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   113
\li If you want to have an option that always takes a certain value, use an enum
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   114
	with only one value.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   115
\li Currently there isn't any way to turn an option off, i.e., all options will
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   116
	be included in the command line options string that is generated using the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   117
	config. If you don't want an option to be passed to the SUT, remove it from
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   118
	the config file or comment it out.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   119
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   120
\section variators_sec Variators
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   121
Variators are the mechanism by which we generate strings of command line arguments
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   122
to test the SUT with. A variator is quite simply an iterator that iterates through
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   123
different variations of command line options. There are two variators supported at
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   124
the moment:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   125
\li \b Exhaustive - Generate all possible combinations of command line arguments
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   126
	that are valid.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   127
\li \b Random - Generate a random variation each time the variator is called.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   128
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   129
As an example, let's try a simple .parameters file:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   130
<tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   131
\-\-blend, enum, [add mod], false, [] \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   132
\-\-fullscreen, boolean, [], false, []
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   133
</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   134
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   135
The exhaustive variator would generate the following four variations:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   136
<tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   137
\-\-blend add \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   138
\-\-blend mod \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   139
\-\-blend add \-\-fullscreen \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   140
\-\-blend mod \-\-fullscreen \n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   141
</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   142
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   143
The random variator would simply generate a random variation like the following:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   144
<tt>\-\-blend mod</tt>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   145
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   146
\section actionconfig_sec The Actions File
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   147
Once the SUT process has been launched, automated testing happens using a mechanism
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   148
called actions. A list of actions is read from a file and each action is performed
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   149
on the SUT process sequentially. Each line in the actions file describes an action.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   150
The format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   151
There are five actions supported at the moment:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   152
\li \b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   153
	images are saved as \c [hash]_[i].bmp where \c [hash] is the 32 character long
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   154
	hexadecimal MD5 hash of the arguments string that was passed to the SUT while
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   155
	launching it and \c i is the window number. i = 1 is an exceptional case
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   156
	where the \c _[i] is dropped and the filename is simply \c [hash].bmp\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   157
	Note: The screenshots are only of the window's client area.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   158
\li \b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   159
	comparing them against a verification image. Each \c [hash]_i.bmp image output
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   160
	by the SCREENSHOT action is compared against a \c [hash].bmp image in the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   161
	verify-dir.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   162
\li \b QUIT - Gracefully quits the SUT process. On Windows this means sending a
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   163
	WM_CLOSE message to each window owned by the SUT process. On Linux it means
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   164
	sending a SIGQUIT signal to the SUT process.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   165
\li \b KILL - Forcefully kills the SUT process. This is useful when the SUT process
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   166
	doesn't respond to the QUIT action.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   167
\li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \c [args]
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   168
	as the arguments string.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   169
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   170
Just like with config files, hash characters can be used to start comments.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   171
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   172
\section contint_sec Continuous Integration (CI)
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   173
One of the goals of the project was to create a test harness that integrates
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   174
with CI systems to provide automated visual and interactive testing to SDL.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   175
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   176
At the moment the test harness can be run in two modes that are useful for CI:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   177
\li Crash testing mode - launch the SUT with every variation and all parameters,
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   178
	report to the CI if there's a crash
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   179
\li Visual testing mode - launch and visually verify the SUT for a smaller subset
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   180
	of the parameters
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   181
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   182
Look at the launch_harness.sh/launch_harness.cmd for an example scripts that run the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   183
test harness for all variations with all parameters and report an error on a crash.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   184
The script uses the testsprite2_crashtest config, so remember to copy those files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   185
over to the test harness executable directory along with the script.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   186
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   187
\section todo_sec TODOs
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   188
\li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   189
	VERIFY x, y, w, h
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   190
\li Add support for spaces between the equals sign in test harness config files
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   191
\li Implement the SCREENSHOT action on Linux
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   192
\li Add a pairwise variator
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   193
\li Add actions to inject keyboard/mouse events
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   194
\li Add actions to manipulate the SUT window, e.g., minimize, restore, resize
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   195
\li Add support to load and save screenshots as .pngs instead of .bmps
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   196
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   197
\section issues_sec Known Issues
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   198
\li The QUIT action does not work on a testsprite2 process with multiple windows.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   199
	This appears to be an issue with testsprite2.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   200
\li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   201
	--fullscreen option is supplied. It works with --fullscreen-desktop, however.
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   202
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   203
\section moreinfo_sec More Information
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   204
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   205
Author Contact Info:\n
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   206
Apoorv Upreti \c \<apoorvupreti@gmail.com\>
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   207
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   208
Other useful links:
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   209
- Project Repository: https://bitbucket.org/nerdap/sdlvisualtest
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   210
- Project Wiki: https://github.com/nerdap/autotestsprite2/wiki
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   211
- Project Blog: http://nerdap.github.io
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   212
- Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   213
- Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip
fcb86d323770 Added Apoorv Upreti's GSoC2013 work: visual test suite.
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   214
*/