src/test/SDL_test_harness.c
author Brandon Schaefer <brandon.schaefer@canonical.com>
Thu, 05 Jun 2014 15:37:33 -0700
changeset 8832 408b0d563b27
parent 8831 9326ec96c132
child 8879 f6e4f24df1ac
permissions -rw-r--r--
Turns out visualstudio does not like PRIu64, soo lets just cast it to llu.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
     1
/*
8139
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
     2
  Simple DirectMedia Layer
8149
681eb46b8ac4 Fixed bug 2374 - Update copyright for 2014...
Sam Lantinga <slouken@libsdl.org>
parents: 8139
diff changeset
     3
  Copyright (C) 1997-2014 Sam Lantinga <slouken@libsdl.org>
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
     4
8139
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
     5
  This software is provided 'as-is', without any express or implied
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
     6
  warranty.  In no event will the authors be held liable for any damages
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
     7
  arising from the use of this software.
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
     8
8139
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
     9
  Permission is granted to anyone to use this software for any purpose,
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    10
  including commercial applications, and to alter it and redistribute it
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    11
  freely, subject to the following restrictions:
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    12
8139
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    13
  1. The origin of this software must not be misrepresented; you must not
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    14
     claim that you wrote the original software. If you use this software
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    15
     in a product, an acknowledgment in the product documentation would be
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    16
     appreciated but is not required.
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    17
  2. Altered source versions must be plainly marked as such, and must not be
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    18
     misrepresented as being the original software.
ad675d62631b Fixed spaces in source file license comment.
Philipp Wiesemann <philipp.wiesemann@arcor.de>
parents: 7721
diff changeset
    19
  3. This notice may not be removed or altered from any source distribution.
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    20
*/
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    21
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    22
#include "SDL_config.h"
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    23
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    24
#include "SDL_test.h"
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    25
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
    26
#include <stdio.h>
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
    27
#include <stdlib.h>
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
    28
#include <string.h>
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
    29
#include <time.h>
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
    30
6756
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
    31
/* Invalid test name/description message format */
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
    32
const char *SDLTest_InvalidNameFormat = "(Invalid)";
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
    33
6756
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
    34
/* Log summary message format */
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
    35
const char *SDLTest_LogSummaryFormat = "%s Summary: Total=%d Passed=%d Failed=%d Skipped=%d";
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
    36
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
    37
/* Final result message format */
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
    38
const char *SDLTest_FinalResultFormat = ">>> %s '%s': %s\n";
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
    39
7678
286c42d7c5ed OCD fixes: Adds a space after /* (glory to regular expressions!)
Gabriel Jacobo <gabomdq@gmail.com>
parents: 7191
diff changeset
    40
/* ! \brief Timeout for single test case execution */
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
    41
static Uint32 SDLTest_TestCaseTimeout = 3600;
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    42
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    43
/**
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    44
* Generates a random run seed string for the harness. The generated seed
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    45
* will contain alphanumeric characters (0-9A-Z).
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    46
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    47
* Note: The returned string needs to be deallocated by the caller.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    48
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    49
* \param length The length of the seed string to generate
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    50
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    51
* \returns The generated seed string
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    52
*/
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    53
char *
6772
5229a117ef97 Added audio test suite; minor code cleanups in test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6771
diff changeset
    54
SDLTest_GenerateRunSeed(const int length)
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    55
{
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    56
    char *seed = NULL;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    57
    SDLTest_RandomContext randomContext;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    58
    int counter;
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    59
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    60
    /* Sanity check input */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    61
    if (length <= 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    62
        SDLTest_LogError("The length of the harness seed must be >0.");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    63
        return NULL;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    64
    }
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    65
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    66
    /* Allocate output buffer */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    67
    seed = (char *)SDL_malloc((length + 1) * sizeof(char));
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    68
    if (seed == NULL) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    69
        SDLTest_LogError("SDL_malloc for run seed output buffer failed.");
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
    70
        SDL_Error(SDL_ENOMEM);
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    71
        return NULL;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    72
    }
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    73
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    74
    /* Generate a random string of alphanumeric characters */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    75
    SDLTest_RandomInitTime(&randomContext);
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
    76
    for (counter = 0; counter < length; counter++) {
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    77
        unsigned int number = SDLTest_Random(&randomContext);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    78
        char ch = (char) (number % (91 - 48)) + 48;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    79
        if (ch >= 58 && ch <= 64) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    80
            ch = 65;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    81
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    82
        seed[counter] = ch;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    83
    }
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
    84
    seed[length] = '\0';
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    85
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    86
    return seed;
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    87
}
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    88
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
    89
/**
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    90
* Generates an execution key for the fuzzer.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    91
*
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    92
* \param runSeed        The run seed to use
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    93
* \param suiteName      The name of the test suite
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    94
* \param testName       The name of the test
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
    95
* \param iteration      The iteration count
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    96
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    97
* \returns The generated execution key to initialize the fuzzer with.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    98
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
    99
*/
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   100
Uint64
6772
5229a117ef97 Added audio test suite; minor code cleanups in test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6771
diff changeset
   101
SDLTest_GenerateExecKey(char *runSeed, char *suiteName, char *testName, int iteration)
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   102
{
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   103
    SDLTest_Md5Context md5Context;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   104
    Uint64 *keys;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   105
    char iterationString[16];
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   106
    Uint32 runSeedLength;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   107
    Uint32 suiteNameLength;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   108
    Uint32 testNameLength;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   109
    Uint32 iterationStringLength;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   110
    Uint32 entireStringLength;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   111
    char *buffer;
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   112
7721
bdf8c2c84e14 Christoph Mallon: Replace strlen(x) == 0 (O(n)) by x[0] == '\0' (O(1)).
Sam Lantinga <slouken@libsdl.org>
parents: 7678
diff changeset
   113
    if (runSeed == NULL || runSeed[0] == '\0') {
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   114
        SDLTest_LogError("Invalid runSeed string.");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   115
        return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   116
    }
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   117
7721
bdf8c2c84e14 Christoph Mallon: Replace strlen(x) == 0 (O(n)) by x[0] == '\0' (O(1)).
Sam Lantinga <slouken@libsdl.org>
parents: 7678
diff changeset
   118
    if (suiteName == NULL || suiteName[0] == '\0') {
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   119
        SDLTest_LogError("Invalid suiteName string.");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   120
        return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   121
    }
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   122
7721
bdf8c2c84e14 Christoph Mallon: Replace strlen(x) == 0 (O(n)) by x[0] == '\0' (O(1)).
Sam Lantinga <slouken@libsdl.org>
parents: 7678
diff changeset
   123
    if (testName == NULL || testName[0] == '\0') {
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   124
        SDLTest_LogError("Invalid testName string.");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   125
        return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   126
    }
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   127
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   128
    if (iteration <= 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   129
        SDLTest_LogError("Invalid iteration count.");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   130
        return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   131
    }
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   132
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   133
    /* Convert iteration number into a string */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   134
    SDL_memset(iterationString, 0, sizeof(iterationString));
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   135
    SDL_snprintf(iterationString, sizeof(iterationString) - 1, "%d", iteration);
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   136
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   137
    /* Combine the parameters into single string */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   138
    runSeedLength = SDL_strlen(runSeed);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   139
    suiteNameLength = SDL_strlen(suiteName);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   140
    testNameLength = SDL_strlen(testName);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   141
    iterationStringLength = SDL_strlen(iterationString);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   142
    entireStringLength  = runSeedLength + suiteNameLength + testNameLength + iterationStringLength + 1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   143
    buffer = (char *)SDL_malloc(entireStringLength);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   144
    if (buffer == NULL) {
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   145
        SDLTest_LogError("Failed to allocate buffer for execKey generation.");
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   146
        SDL_Error(SDL_ENOMEM);
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   147
        return 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   148
    }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   149
    SDL_snprintf(buffer, entireStringLength, "%s%s%s%d", runSeed, suiteName, testName, iteration);
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   150
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   151
    /* Hash string and use half of the digest as 64bit exec key */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   152
    SDLTest_Md5Init(&md5Context);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   153
    SDLTest_Md5Update(&md5Context, (unsigned char *)buffer, entireStringLength);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   154
    SDLTest_Md5Final(&md5Context);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   155
    SDL_free(buffer);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   156
    keys = (Uint64 *)md5Context.digest;
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   157
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   158
    return keys[0];
6717
2acd95060548 Add log, assert and harness (partial) to test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents:
diff changeset
   159
}
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   160
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   161
/**
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   162
* \brief Set timeout handler for test.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   163
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   164
* Note: SDL_Init(SDL_INIT_TIMER) will be called if it wasn't done so before.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   165
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   166
* \param timeout Timeout interval in seconds.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   167
* \param callback Function that will be called after timeout has elapsed.
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   168
*
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   169
* \return Timer id or -1 on failure.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   170
*/
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   171
SDL_TimerID
6772
5229a117ef97 Added audio test suite; minor code cleanups in test lib
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6771
diff changeset
   172
SDLTest_SetTestTimeout(int timeout, void (*callback)())
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   173
{
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   174
    Uint32 timeoutInMilliseconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   175
    SDL_TimerID timerID;
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   176
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   177
    if (callback == NULL) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   178
        SDLTest_LogError("Timeout callback can't be NULL");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   179
        return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   180
    }
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   181
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   182
    if (timeout < 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   183
        SDLTest_LogError("Timeout value must be bigger than zero.");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   184
        return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   185
    }
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   186
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   187
    /* Init SDL timer if not initialized before */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   188
    if (SDL_WasInit(SDL_INIT_TIMER) == 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   189
        if (SDL_InitSubSystem(SDL_INIT_TIMER)) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   190
            SDLTest_LogError("Failed to init timer subsystem: %s", SDL_GetError());
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   191
            return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   192
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   193
    }
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   194
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   195
    /* Set timer */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   196
    timeoutInMilliseconds = timeout * 1000;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   197
    timerID = SDL_AddTimer(timeoutInMilliseconds, (SDL_TimerCallback)callback, 0x0);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   198
    if (timerID == 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   199
        SDLTest_LogError("Creation of SDL timer failed: %s", SDL_GetError());
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   200
        return -1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   201
    }
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   202
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   203
    return timerID;
6718
918ba414168b Update assert API in test lib; add to and harness; add test lib to VS2010 and VS2012 solution; fix VS2012 solution; fix compiler warning
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6717
diff changeset
   204
}
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   205
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   206
/**
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   207
* \brief Timeout handler. Aborts test run and exits harness process.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   208
*/
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   209
void
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   210
    SDLTest_BailOut()
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   211
{
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   212
    SDLTest_LogError("TestCaseTimeout timer expired. Aborting test run.");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   213
    exit(TEST_ABORTED); /* bail out from the test */
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   214
}
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   215
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   216
/**
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   217
* \brief Execute a test using the given execution key.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   218
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   219
* \param testSuite Suite containing the test case.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   220
* \param testCase Case to execute.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   221
* \param execKey Execution key for the fuzzer.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   222
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   223
* \returns Test case result.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   224
*/
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   225
int
6830
0cd3e2e9b2c5 Add tests to audio suite; update harness filter logic
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6772
diff changeset
   226
SDLTest_RunTest(SDLTest_TestSuiteReference *testSuite, SDLTest_TestCaseReference *testCase, Uint64 execKey)
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   227
{
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   228
    SDL_TimerID timer = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   229
    int testCaseResult = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   230
    int testResult = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   231
    int fuzzerCount;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   232
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   233
    if (testSuite==NULL || testCase==NULL || testSuite->name==NULL || testCase->name==NULL)
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   234
    {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   235
        SDLTest_LogError("Setup failure: testSuite or testCase references NULL");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   236
        return TEST_RESULT_SETUP_FAILURE;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   237
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   238
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   239
    if (!testCase->enabled)
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   240
    {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   241
        SDLTest_Log((char *)SDLTest_FinalResultFormat, "Test", testCase->name, "Skipped (Disabled)");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   242
        return TEST_RESULT_SKIPPED;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   243
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   244
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   245
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   246
    /* Initialize fuzzer */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   247
    SDLTest_FuzzerInit(execKey);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   248
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   249
    /* Reset assert tracker */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   250
    SDLTest_ResetAssertSummary();
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   251
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   252
    /* Set timeout timer */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   253
    timer = SDLTest_SetTestTimeout(SDLTest_TestCaseTimeout, SDLTest_BailOut);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   254
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   255
    /* Maybe run suite initalizer function */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   256
    if (testSuite->testSetUp) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   257
        testSuite->testSetUp(0x0);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   258
        if (SDLTest_AssertSummaryToTestResult() == TEST_RESULT_FAILED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   259
            SDLTest_LogError((char *)SDLTest_FinalResultFormat, "Suite Setup", testSuite->name, "Failed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   260
            return TEST_RESULT_SETUP_FAILURE;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   261
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   262
    }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   263
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   264
    /* Run test case function */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   265
    testCaseResult = testCase->testCase(0x0);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   266
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   267
    /* Convert test execution result into harness result */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   268
    if (testCaseResult == TEST_SKIPPED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   269
        /* Test was programatically skipped */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   270
        testResult = TEST_RESULT_SKIPPED;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   271
    } else if (testCaseResult == TEST_STARTED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   272
        /* Test did not return a TEST_COMPLETED value; assume it failed */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   273
        testResult = TEST_RESULT_FAILED;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   274
    } else if (testCaseResult == TEST_ABORTED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   275
        /* Test was aborted early; assume it failed */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   276
        testResult = TEST_RESULT_FAILED;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   277
    } else {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   278
        /* Perform failure analysis based on asserts */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   279
        testResult = SDLTest_AssertSummaryToTestResult();
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   280
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   281
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   282
    /* Maybe run suite cleanup function (ignore failed asserts) */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   283
    if (testSuite->testTearDown) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   284
        testSuite->testTearDown(0x0);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   285
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   286
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   287
    /* Cancel timeout timer */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   288
    if (timer) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   289
        SDL_RemoveTimer(timer);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   290
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   291
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   292
    /* Report on asserts and fuzzer usage */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   293
    fuzzerCount = SDLTest_GetFuzzerInvocationCount();
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   294
    if (fuzzerCount > 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   295
        SDLTest_Log("Fuzzer invocations: %d", fuzzerCount);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   296
    }
7189
414be1d64060 Update test harness to handle test return codes; fix comment format in harness; update Main test suite to handle globally disabled features
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6885
diff changeset
   297
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   298
    /* Final log based on test execution result */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   299
    if (testCaseResult == TEST_SKIPPED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   300
        /* Test was programatically skipped */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   301
        SDLTest_Log((char *)SDLTest_FinalResultFormat, "Test", testCase->name, "Skipped (Programmatically)");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   302
    } else if (testCaseResult == TEST_STARTED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   303
        /* Test did not return a TEST_COMPLETED value; assume it failed */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   304
        SDLTest_LogError((char *)SDLTest_FinalResultFormat, "Test", testCase->name, "Failed (test started, but did not return TEST_COMPLETED)");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   305
    } else if (testCaseResult == TEST_ABORTED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   306
        /* Test was aborted early; assume it failed */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   307
        SDLTest_LogError((char *)SDLTest_FinalResultFormat, "Test", testCase->name, "Failed (Aborted)");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   308
    } else {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   309
        SDLTest_LogAssertSummary();
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   310
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   311
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   312
    return testResult;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   313
}
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   314
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   315
/* Prints summary of all suites/tests contained in the given reference */
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   316
void SDLTest_LogTestSuiteSummary(SDLTest_TestSuiteReference *testSuites)
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   317
{
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   318
    int suiteCounter;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   319
    int testCounter;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   320
    SDLTest_TestSuiteReference *testSuite;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   321
    SDLTest_TestCaseReference *testCase;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   322
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   323
    /* Loop over all suites */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   324
    suiteCounter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   325
    while(&testSuites[suiteCounter]) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   326
        testSuite=&testSuites[suiteCounter];
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   327
        suiteCounter++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   328
        SDLTest_Log("Test Suite %i - %s\n", suiteCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   329
            (testSuite->name) ? testSuite->name : SDLTest_InvalidNameFormat);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   330
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   331
        /* Loop over all test cases */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   332
        testCounter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   333
        while(testSuite->testCases[testCounter])
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   334
        {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   335
            testCase=(SDLTest_TestCaseReference *)testSuite->testCases[testCounter];
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   336
            testCounter++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   337
            SDLTest_Log("  Test Case %i - %s: %s", testCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   338
                (testCase->name) ? testCase->name : SDLTest_InvalidNameFormat,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   339
                (testCase->description) ? testCase->description : SDLTest_InvalidNameFormat);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   340
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   341
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   342
}
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   343
6756
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
   344
/* Gets a timer value in seconds */
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
   345
float GetClock()
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
   346
{
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   347
    float currentClock = (float)clock();
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   348
    return currentClock / (float)CLOCKS_PER_SEC;
6756
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
   349
}
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   350
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   351
/**
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   352
* \brief Execute a test suite using the given run seed and execution key.
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   353
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   354
* The filter string is matched to the suite name (full comparison) to select a single suite,
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   355
* or if no suite matches, it is matched to the test names (full comparison) to select a single test.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   356
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   357
* \param testSuites Suites containing the test case.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   358
* \param userRunSeed Custom run seed provided by user, or NULL to autogenerate one.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   359
* \param userExecKey Custom execution key provided by user, or 0 to autogenerate one.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   360
* \param filter Filter specification. NULL disables. Case sensitive.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   361
* \param testIterations Number of iterations to run each test case.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   362
*
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   363
* \returns Test run result; 0 when all tests passed, 1 if any tests failed.
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   364
*/
6768
22c37bf0afbc Fixed const correctness in the test harness
Sam Lantinga <slouken@libsdl.org>
parents: 6763
diff changeset
   365
int SDLTest_RunSuites(SDLTest_TestSuiteReference *testSuites[], const char *userRunSeed, Uint64 userExecKey, const char *filter, int testIterations)
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   366
{
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   367
    int totalNumberOfTests = 0;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   368
    int failedNumberOfTests = 0;
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   369
    int suiteCounter;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   370
    int testCounter;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   371
    int iterationCounter;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   372
    SDLTest_TestSuiteReference *testSuite;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   373
    SDLTest_TestCaseReference *testCase;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   374
    const char *runSeed = NULL;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   375
    char *currentSuiteName;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   376
    char *currentTestName;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   377
    Uint64 execKey;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   378
    float runStartSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   379
    float suiteStartSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   380
    float testStartSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   381
    float runEndSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   382
    float suiteEndSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   383
    float testEndSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   384
    float runtime;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   385
    int suiteFilter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   386
    char *suiteFilterName = NULL;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   387
    int testFilter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   388
    char *testFilterName = NULL;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   389
    int testResult = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   390
    int runResult = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   391
    Uint32 totalTestFailedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   392
    Uint32 totalTestPassedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   393
    Uint32 totalTestSkippedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   394
    Uint32 testFailedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   395
    Uint32 testPassedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   396
    Uint32 testSkippedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   397
    Uint32 countSum = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   398
    char *logFormat = (char *)SDLTest_LogSummaryFormat;
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   399
    SDLTest_TestCaseReference **failedTests;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   400
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   401
    /* Sanitize test iterations */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   402
    if (testIterations < 1) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   403
        testIterations = 1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   404
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   405
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   406
    /* Generate run see if we don't have one already */
7721
bdf8c2c84e14 Christoph Mallon: Replace strlen(x) == 0 (O(n)) by x[0] == '\0' (O(1)).
Sam Lantinga <slouken@libsdl.org>
parents: 7678
diff changeset
   407
    if (userRunSeed == NULL || userRunSeed[0] == '\0') {
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   408
        runSeed = SDLTest_GenerateRunSeed(16);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   409
        if (runSeed == NULL) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   410
            SDLTest_LogError("Generating a random seed failed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   411
            return 2;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   412
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   413
    } else {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   414
        runSeed = userRunSeed;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   415
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   416
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   417
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   418
    /* Reset per-run counters */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   419
    totalTestFailedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   420
    totalTestPassedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   421
    totalTestSkippedCount = 0;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   422
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   423
    /* Take time - run start */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   424
    runStartSeconds = GetClock();
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   425
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   426
    /* Log run with fuzzer parameters */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   427
    SDLTest_Log("::::: Test Run /w seed '%s' started\n", runSeed);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   428
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   429
	/* Count the total number of tests */
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   430
    suiteCounter = 0;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   431
    while (testSuites[suiteCounter]) {
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   432
        testSuite=(SDLTest_TestSuiteReference *)testSuites[suiteCounter];
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   433
        suiteCounter++;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   434
        testCounter = 0;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   435
        while (testSuite->testCases[testCounter])
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   436
        {
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   437
            testCounter++;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   438
			totalNumberOfTests++;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   439
		}
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   440
	}
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   441
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   442
	/* Pre-allocate an array for tracking failed tests (potentially all test cases) */
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   443
	failedTests = (SDLTest_TestCaseReference **)SDL_malloc(totalNumberOfTests * sizeof(SDLTest_TestCaseReference *));
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   444
	if (failedTests == NULL) {	
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   445
	   SDLTest_LogError("Unable to allocate cache for failed tests");
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   446
           SDL_Error(SDL_ENOMEM);	   
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   447
           return -1;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   448
	}
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   449
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   450
    /* Initialize filtering */
7721
bdf8c2c84e14 Christoph Mallon: Replace strlen(x) == 0 (O(n)) by x[0] == '\0' (O(1)).
Sam Lantinga <slouken@libsdl.org>
parents: 7678
diff changeset
   451
    if (filter != NULL && filter[0] != '\0') {
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   452
        /* Loop over all suites to check if we have a filter match */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   453
        suiteCounter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   454
        while (testSuites[suiteCounter] && suiteFilter == 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   455
            testSuite=(SDLTest_TestSuiteReference *)testSuites[suiteCounter];
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   456
            suiteCounter++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   457
            if (testSuite->name != NULL && SDL_strcmp(filter, testSuite->name) == 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   458
                /* Matched a suite name */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   459
                suiteFilter = 1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   460
                suiteFilterName = testSuite->name;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   461
                SDLTest_Log("Filtering: running only suite '%s'", suiteFilterName);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   462
                break;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   463
            }
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   464
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   465
            /* Within each suite, loop over all test cases to check if we have a filter match */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   466
            testCounter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   467
            while (testSuite->testCases[testCounter] && testFilter == 0)
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   468
            {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   469
                testCase=(SDLTest_TestCaseReference *)testSuite->testCases[testCounter];
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   470
                testCounter++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   471
                if (testCase->name != NULL && SDL_strcmp(filter, testCase->name) == 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   472
                    /* Matched a test name */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   473
                    suiteFilter = 1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   474
                    suiteFilterName = testSuite->name;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   475
                    testFilter = 1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   476
                    testFilterName = testCase->name;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   477
                    SDLTest_Log("Filtering: running only test '%s' in suite '%s'", testFilterName, suiteFilterName);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   478
                    break;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   479
                }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   480
            }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   481
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   482
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   483
        if (suiteFilter == 0 && testFilter == 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   484
            SDLTest_LogError("Filter '%s' did not match any test suite/case.", filter);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   485
            SDLTest_Log("Exit code: 2");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   486
            return 2;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   487
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   488
    }
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   489
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   490
    /* Loop over all suites */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   491
    suiteCounter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   492
    while(testSuites[suiteCounter]) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   493
        testSuite=(SDLTest_TestSuiteReference *)testSuites[suiteCounter];
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   494
        currentSuiteName = (char *)((testSuite->name) ? testSuite->name : SDLTest_InvalidNameFormat);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   495
        suiteCounter++;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   496
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   497
        /* Filter suite if flag set and we have a name */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   498
        if (suiteFilter == 1 && suiteFilterName != NULL && testSuite->name != NULL &&
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   499
            SDL_strcmp(suiteFilterName, testSuite->name) != 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   500
                /* Skip suite */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   501
                SDLTest_Log("===== Test Suite %i: '%s' skipped\n",
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   502
                    suiteCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   503
                    currentSuiteName);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   504
        } else {
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   505
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   506
            /* Reset per-suite counters */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   507
            testFailedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   508
            testPassedCount = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   509
            testSkippedCount = 0;
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   510
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   511
            /* Take time - suite start */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   512
            suiteStartSeconds = GetClock();
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   513
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   514
            /* Log suite started */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   515
            SDLTest_Log("===== Test Suite %i: '%s' started\n",
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   516
                suiteCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   517
                currentSuiteName);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   518
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   519
            /* Loop over all test cases */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   520
            testCounter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   521
            while(testSuite->testCases[testCounter])
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   522
            {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   523
                testCase=(SDLTest_TestCaseReference *)testSuite->testCases[testCounter];
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   524
                currentTestName = (char *)((testCase->name) ? testCase->name : SDLTest_InvalidNameFormat);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   525
                testCounter++;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   526
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   527
                /* Filter tests if flag set and we have a name */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   528
                if (testFilter == 1 && testFilterName != NULL && testCase->name != NULL &&
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   529
                    SDL_strcmp(testFilterName, testCase->name) != 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   530
                        /* Skip test */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   531
                        SDLTest_Log("===== Test Case %i.%i: '%s' skipped\n",
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   532
                            suiteCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   533
                            testCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   534
                            currentTestName);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   535
                } else {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   536
                    /* Override 'disabled' flag if we specified a test filter (i.e. force run for debugging) */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   537
                    if (testFilter == 1 && !testCase->enabled) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   538
                        SDLTest_Log("Force run of disabled test since test filter was set");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   539
                        testCase->enabled = 1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   540
                    }
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   541
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   542
                    /* Take time - test start */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   543
                    testStartSeconds = GetClock();
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   544
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   545
                    /* Log test started */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   546
                    SDLTest_Log("----- Test Case %i.%i: '%s' started",
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   547
                        suiteCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   548
                        testCounter,
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   549
                        currentTestName);
7721
bdf8c2c84e14 Christoph Mallon: Replace strlen(x) == 0 (O(n)) by x[0] == '\0' (O(1)).
Sam Lantinga <slouken@libsdl.org>
parents: 7678
diff changeset
   550
                    if (testCase->description != NULL && testCase->description[0] != '\0') {
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   551
                        SDLTest_Log("Test Description: '%s'",
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   552
                            (testCase->description) ? testCase->description : SDLTest_InvalidNameFormat);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   553
                    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   554
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   555
                    /* Loop over all iterations */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   556
                    iterationCounter = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   557
                    while(iterationCounter < testIterations)
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   558
                    {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   559
                        iterationCounter++;
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   560
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   561
                        if (userExecKey != 0) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   562
                            execKey = userExecKey;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   563
                        } else {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   564
                            execKey = SDLTest_GenerateExecKey((char *)runSeed, testSuite->name, testCase->name, iterationCounter);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   565
                        }
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   566
8832
408b0d563b27 Turns out visualstudio does not like PRIu64, soo lets just cast it to llu.
Brandon Schaefer <brandon.schaefer@canonical.com>
parents: 8831
diff changeset
   567
                        SDLTest_Log("Test Iteration %i: execKey %llu", iterationCounter, (long long unsigned)execKey);
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   568
                        testResult = SDLTest_RunTest(testSuite, testCase, execKey);
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   569
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   570
                        if (testResult == TEST_RESULT_PASSED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   571
                            testPassedCount++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   572
                            totalTestPassedCount++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   573
                        } else if (testResult == TEST_RESULT_SKIPPED) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   574
                            testSkippedCount++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   575
                            totalTestSkippedCount++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   576
                        } else {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   577
                            testFailedCount++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   578
                            totalTestFailedCount++;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   579
                        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   580
                    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   581
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   582
                    /* Take time - test end */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   583
                    testEndSeconds = GetClock();
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   584
                    runtime = testEndSeconds - testStartSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   585
                    if (runtime < 0.0f) runtime = 0.0f;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   586
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   587
                    if (testIterations > 1) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   588
                        /* Log test runtime */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   589
                        SDLTest_Log("Runtime of %i iterations: %.1f sec", testIterations, runtime);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   590
                        SDLTest_Log("Average Test runtime: %.5f sec", runtime / (float)testIterations);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   591
                    } else {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   592
                        /* Log test runtime */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   593
                        SDLTest_Log("Total Test runtime: %.1f sec", runtime);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   594
                    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   595
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   596
                    /* Log final test result */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   597
                    switch (testResult) {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   598
                    case TEST_RESULT_PASSED:
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   599
                        SDLTest_Log((char *)SDLTest_FinalResultFormat, "Test", currentTestName, "Passed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   600
                        break;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   601
                    case TEST_RESULT_FAILED:
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   602
                        SDLTest_LogError((char *)SDLTest_FinalResultFormat, "Test", currentTestName, "Failed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   603
                        break;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   604
                    case TEST_RESULT_NO_ASSERT:
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   605
                        SDLTest_LogError((char *)SDLTest_FinalResultFormat,"Test", currentTestName, "No Asserts");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   606
                        break;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   607
                    }
6763
9cbd31a3450b Test lib updates: updated harness to support filtering, added surface comparer, updated interface to test images; added Render test suite from GSOC project
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6760
diff changeset
   608
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   609
                    /* Collect failed test case references for repro-step display */
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   610
                    if (testResult == TEST_RESULT_FAILED) {
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   611
                        failedTests[failedNumberOfTests] = testCase;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   612
                        failedNumberOfTests++;
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   613
                    }
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   614
                }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   615
            }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   616
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   617
            /* Take time - suite end */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   618
            suiteEndSeconds = GetClock();
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   619
            runtime = suiteEndSeconds - suiteStartSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   620
            if (runtime < 0.0f) runtime = 0.0f;
6756
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
   621
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   622
            /* Log suite runtime */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   623
            SDLTest_Log("Total Suite runtime: %.1f sec", runtime);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   624
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   625
            /* Log summary and final Suite result */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   626
            countSum = testPassedCount + testFailedCount + testSkippedCount;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   627
            if (testFailedCount == 0)
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   628
            {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   629
                SDLTest_Log(logFormat, "Suite", countSum, testPassedCount, testFailedCount, testSkippedCount);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   630
                SDLTest_Log((char *)SDLTest_FinalResultFormat, "Suite", currentSuiteName, "Passed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   631
            }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   632
            else
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   633
            {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   634
                SDLTest_LogError(logFormat, "Suite", countSum, testPassedCount, testFailedCount, testSkippedCount);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   635
                SDLTest_LogError((char *)SDLTest_FinalResultFormat, "Suite", currentSuiteName, "Failed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   636
            }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   637
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   638
        }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   639
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   640
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   641
    /* Take time - run end */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   642
    runEndSeconds = GetClock();
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   643
    runtime = runEndSeconds - runStartSeconds;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   644
    if (runtime < 0.0f) runtime = 0.0f;
6756
398073b195bb Refactor/fix test lib harness, assert and log component; add harness driver; port platform suite from GSOC code
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6727
diff changeset
   645
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   646
    /* Log total runtime */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   647
    SDLTest_Log("Total Run runtime: %.1f sec", runtime);
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   648
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   649
    /* Log summary and final run result */
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   650
    countSum = totalTestPassedCount + totalTestFailedCount + totalTestSkippedCount;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   651
    if (totalTestFailedCount == 0)
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   652
    {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   653
        runResult = 0;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   654
        SDLTest_Log(logFormat, "Run", countSum, totalTestPassedCount, totalTestFailedCount, totalTestSkippedCount);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   655
        SDLTest_Log((char *)SDLTest_FinalResultFormat, "Run /w seed", runSeed, "Passed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   656
    }
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   657
    else
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   658
    {
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   659
        runResult = 1;
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   660
        SDLTest_LogError(logFormat, "Run", countSum, totalTestPassedCount, totalTestFailedCount, totalTestSkippedCount);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   661
        SDLTest_LogError((char *)SDLTest_FinalResultFormat, "Run /w seed", runSeed, "Failed");
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   662
    }
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   663
8605
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   664
    /* Print repro steps for failed tests */
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   665
    if (failedNumberOfTests > 0) {
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   666
        SDLTest_Log("Harness input to repro failures:");
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   667
        for (testCounter = 0; testCounter < failedNumberOfTests; testCounter++) {
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   668
          SDLTest_Log(" --seed %s --filter %s", runSeed, failedTests[testCounter]->name);
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   669
        }
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   670
    }
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   671
    SDL_free(failedTests);
57faccca4fab Fix bug/add test coverage for SDLTest_GenerateRunSeed helper; improve test harness adding output of repro steps for failures; improve negative test for SDL_GetError/SDL_SetError
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 8149
diff changeset
   672
7191
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   673
    SDLTest_Log("Exit code: %d", runResult);
75360622e65f File style cleanup for the SDL 2.0 release
Sam Lantinga <slouken@libsdl.org>
parents: 7189
diff changeset
   674
    return runResult;
6721
53b71f45a53a Added to harness in test lib (work in progress)
Andreas Schiffler <aschiffler@ferzkopp.net>
parents: 6718
diff changeset
   675
}