mojoshader_lexer.re
author Ryan C. Gordon <icculus@icculus.org>
Wed, 18 Apr 2012 00:19:32 -0400
changeset 1095 bc3d2c6e06cf
parent 959 8a5d46db0cc6
child 1121 4142681f9fda
permissions -rw-r--r--
glsl: Implemented most of the missing srcmods. Reread the GLSL spec, and it turns out that "vec3(x,y,z) - 3.0" is legal syntactic sugar: the compiler knows to subtract 3.0 from each of the three components in that vec3. This made this simpler than having to tapdance to generate correct constant vectors ourselves, and it's easier to read.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     1
/**
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     2
 * MojoShader; generate shader programs from bytecode of compiled
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     3
 *  Direct3D shaders.
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     4
 *
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     5
 * Please see the file LICENSE.txt in the source's root directory.
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     6
 *
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     7
 *  This file written by Ryan C. Gordon.
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     8
 */
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
     9
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    10
// This was originally based on examples/pp-c.re from re2c: http://re2c.org/
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    11
//   re2c is public domain code.
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    12
//
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    13
// You build mojoshader_lexer_preprocessor.c from the .re file with re2c...
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    14
// re2c -is -o mojoshader_lexer_preprocessor.c mojoshader_lexer_preprocessor.re
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    15
//
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    16
// Changes to the lexer are done to the .re file, not the C code!
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    17
//
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    18
// Please note that this isn't a perfect C lexer, since it is used for both
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    19
//  HLSL and shader assembly language, and follows the quirks of Microsoft's
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    20
//  tools.
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    21
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    22
#define __MOJOSHADER_INTERNAL__ 1
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    23
#include "mojoshader_internal.h"
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    24
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    25
typedef unsigned char uchar;
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    26
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    27
/*!max:re2c */
798
5dd67cc04cf9 Bunch of small tweaks to make this compile as C++ code without errors/warnings.
Ryan C. Gordon <icculus@icculus.org>
parents: 697
diff changeset
    28
#define RET(t) return update_state(s, eoi, cursor, token, (Token) t)
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    29
#define YYCTYPE uchar
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    30
#define YYCURSOR cursor
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    31
#define YYLIMIT limit
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    32
#define YYMARKER s->lexer_marker
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    33
#define YYFILL(n) { if ((n) == 1) { cursor = sentinel; limit = cursor + YYMAXFILL; eoi = 1; } }
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    34
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    35
static uchar sentinel[YYMAXFILL];
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    36
651
05ad72d120a5 Lexer now stores token value to IncludeState.
Ryan C. Gordon <icculus@icculus.org>
parents: 649
diff changeset
    37
static Token update_state(IncludeState *s, int eoi, const uchar *cur,
05ad72d120a5 Lexer now stores token value to IncludeState.
Ryan C. Gordon <icculus@icculus.org>
parents: 649
diff changeset
    38
                          const uchar *tok, const Token val)
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    39
{
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    40
    if (eoi)
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    41
    {
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    42
        s->bytes_left = 0;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    43
        s->source = (const char *) s->source_base + s->orig_length;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    44
        if ( (tok >= sentinel) && (tok < (sentinel+YYMAXFILL)) )
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    45
            s->token = s->source;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    46
        else
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    47
            s->token = (const char *) tok;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    48
    } // if
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    49
    else
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    50
    {
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    51
        s->bytes_left -= (unsigned int) (cur - ((const uchar *) s->source));
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    52
        s->source = (const char *) cur;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    53
        s->token = (const char *) tok;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    54
    } // else
648
243e35876142 Have lexer calculate token size now.
Ryan C. Gordon <icculus@icculus.org>
parents: 642
diff changeset
    55
    s->tokenlen = (unsigned int) (s->source - s->token);
651
05ad72d120a5 Lexer now stores token value to IncludeState.
Ryan C. Gordon <icculus@icculus.org>
parents: 649
diff changeset
    56
    s->tokenval = val;
05ad72d120a5 Lexer now stores token value to IncludeState.
Ryan C. Gordon <icculus@icculus.org>
parents: 649
diff changeset
    57
    return val;
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    58
} // update_state
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    59
649
527cfe6f2d45 Renamed preprocessor_internal_lexer to preprocessor_lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 648
diff changeset
    60
Token preprocessor_lexer(IncludeState *s)
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    61
{
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    62
    const uchar *cursor = (const uchar *) s->source;
679
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    63
    const uchar *token = cursor;
595
a72705c067dc Handle lexing of comment ends better.
Ryan C. Gordon <icculus@icculus.org>
parents: 594
diff changeset
    64
    const uchar *matchptr;
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    65
    const uchar *limit = cursor + s->bytes_left;
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
    66
    int eoi = 0;
564
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
    67
    int saw_newline = 0;
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    68
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    69
/*!re2c
596
8efa254e8840 Renamed "any" to "ANY" to match other rules.
Ryan C. Gordon <icculus@icculus.org>
parents: 595
diff changeset
    70
    ANY = [\000-\377];
597
832dbfa63509 Bogus chars in the lexer now return a token that signifies this.
Ryan C. Gordon <icculus@icculus.org>
parents: 596
diff changeset
    71
    ANYLEGAL = [a-zA-Z0-9_/'*=+%^&|!#<>()[{}.,~^:;? \t\v\f\r\n\-\]\\];
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    72
    O = [0-7];
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    73
    D = [0-9];
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    74
    L = [a-zA-Z_];
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    75
    H = [a-fA-F0-9];
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    76
    E = [Ee] [+-]? D+;
959
8a5d46db0cc6 HLSL uses h or H as a float literal suffix, meaning "half".
Ryan C. Gordon <icculus@icculus.org>
parents: 910
diff changeset
    77
    FS = [fFhH];
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    78
    IS = [uUlL]*;
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    79
    ESC = [\\] ([abfnrtv?'"\\] | "x" H+ | O+);
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    80
    PP = "#" [ \t]*;
594
de6f8df3b4fe A couple endline fixes in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 584
diff changeset
    81
    NEWLINE = ("\r\n" | "\r" | "\n");
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    82
    WHITESPACE = [ \t\v\f]+;
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    83
*/
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    84
679
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    85
    // preprocessor directives are only valid at start of line.
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    86
    if (s->tokenval == ((Token) '\n'))
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    87
        goto ppdirective;  // may jump back to scanner_loop.
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    88
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    89
scanner_loop:
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    90
    if (YYLIMIT == YYCURSOR) YYFILL(1);
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    91
    token = cursor;
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
    92
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    93
/*!re2c
631
5045dcc9fdaf Lexer now handles '\\' continuation at the end of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 616
diff changeset
    94
    "\\" [ \t\v\f]* NEWLINE  { s->line++; goto scanner_loop; }
5045dcc9fdaf Lexer now handles '\\' continuation at the end of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 616
diff changeset
    95
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    96
    "/*"            { goto multilinecomment; }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    97
    "//"            { goto singlelinecomment; }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    98
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
    99
    L (L|D)*        { RET(TOKEN_IDENTIFIER); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   100
    
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   101
    ("0" [xX] H+ IS?) | ("0" D+ IS?) | (D+ IS?) |
596
8efa254e8840 Renamed "any" to "ANY" to match other rules.
Ryan C. Gordon <icculus@icculus.org>
parents: 595
diff changeset
   102
    (['] (ESC|ANY\[\r\n\\'])* ['])
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   103
                    { RET(TOKEN_INT_LITERAL); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   104
    
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   105
    (D+ E FS?) | (D* "." D+ E? FS?) | (D+ "." D* E? FS?)
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   106
                    { RET(TOKEN_FLOAT_LITERAL); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   107
    
596
8efa254e8840 Renamed "any" to "ANY" to match other rules.
Ryan C. Gordon <icculus@icculus.org>
parents: 595
diff changeset
   108
    (["] (ESC|ANY\[\r\n\\"])* ["])
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   109
                    { RET(TOKEN_STRING_LITERAL); }
631
5045dcc9fdaf Lexer now handles '\\' continuation at the end of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 616
diff changeset
   110
584
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   111
    ">>="           { RET(TOKEN_RSHIFTASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   112
    "<<="           { RET(TOKEN_LSHIFTASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   113
    "+="            { RET(TOKEN_ADDASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   114
    "-="            { RET(TOKEN_SUBASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   115
    "*="            { RET(TOKEN_MULTASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   116
    "/="            { RET(TOKEN_DIVASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   117
    "%="            { RET(TOKEN_MODASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   118
    "^="            { RET(TOKEN_XORASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   119
    "&="            { RET(TOKEN_ANDASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   120
    "|="            { RET(TOKEN_ORASSIGN); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   121
    "++"            { RET(TOKEN_INCREMENT); }
9aea8877564d Updated lexer for HLSL.
Ryan C. Gordon <icculus@icculus.org>
parents: 564
diff changeset
   122
    "--"            { RET(TOKEN_DECREMENT); }
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   123
    ">>"            { RET(TOKEN_RSHIFT); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   124
    "<<"            { RET(TOKEN_LSHIFT); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   125
    "&&"            { RET(TOKEN_ANDAND); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   126
    "||"            { RET(TOKEN_OROR); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   127
    "<="            { RET(TOKEN_LEQ); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   128
    ">="            { RET(TOKEN_GEQ); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   129
    "=="            { RET(TOKEN_EQL); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   130
    "!="            { RET(TOKEN_NEQ); }
865
4eb06464212d Added stringify operator ("#") to preprocessor.
Ryan C. Gordon <icculus@icculus.org>
parents: 798
diff changeset
   131
    "#"             { RET(TOKEN_HASH); }
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   132
    "##"            { RET(TOKEN_HASHHASH); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   133
    "("             { RET('('); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   134
    ")"             { RET(')'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   135
    "["             { RET('['); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   136
    "]"             { RET(']'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   137
    "."             { RET('.'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   138
    ","             { RET(','); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   139
    "&"             { RET('&'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   140
    "!"             { RET('!'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   141
    "~"             { RET('~'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   142
    "-"             { RET('-'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   143
    "+"             { RET('+'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   144
    "*"             { RET('*'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   145
    "/"             { RET('/'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   146
    "%"             { RET('%'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   147
    "<"             { RET('<'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   148
    ">"             { RET('>'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   149
    "^"             { RET('^'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   150
    "|"             { RET('|'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   151
    ":"             { RET(':'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   152
    "{"             { RET('{'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   153
    "}"             { RET('}'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   154
    "="             { RET('='); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   155
    "?"             { RET('?'); }
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   156
690
4710df464f13 Moved asm comment processing into the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 679
diff changeset
   157
    ";"             { if (s->asm_comments) goto singlelinecomment; RET(';'); }
4710df464f13 Moved asm comment processing into the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 679
diff changeset
   158
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   159
    "\000"          { if (eoi) { RET(TOKEN_EOI); } goto bad_chars; }
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   160
635
be3b428802a1 Try to make #error lexing match gcc's.
Ryan C. Gordon <icculus@icculus.org>
parents: 631
diff changeset
   161
    WHITESPACE      { if (s->report_whitespace) RET(' '); goto scanner_loop; }
564
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   162
    NEWLINE         { s->line++; RET('\n'); }
597
832dbfa63509 Bogus chars in the lexer now return a token that signifies this.
Ryan C. Gordon <icculus@icculus.org>
parents: 596
diff changeset
   163
    ANY             { goto bad_chars; }
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   164
*/
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   165
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   166
multilinecomment:
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   167
    if (YYLIMIT == YYCURSOR) YYFILL(1);
595
a72705c067dc Handle lexing of comment ends better.
Ryan C. Gordon <icculus@icculus.org>
parents: 594
diff changeset
   168
    matchptr = cursor;
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   169
// The "*\/" is just to avoid screwing up text editor syntax highlighting.
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   170
/*!re2c
564
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   171
    "*\/"           {
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   172
                        if (saw_newline)
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   173
                            RET('\n');
635
be3b428802a1 Try to make #error lexing match gcc's.
Ryan C. Gordon <icculus@icculus.org>
parents: 631
diff changeset
   174
                        else if (s->report_whitespace)
be3b428802a1 Try to make #error lexing match gcc's.
Ryan C. Gordon <icculus@icculus.org>
parents: 631
diff changeset
   175
                            RET(' ');
564
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   176
                        goto scanner_loop;
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   177
                    }
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   178
    NEWLINE         {
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   179
                        s->line++;
595
a72705c067dc Handle lexing of comment ends better.
Ryan C. Gordon <icculus@icculus.org>
parents: 594
diff changeset
   180
                        token = matchptr;
564
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   181
                        saw_newline = 1;
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   182
                        goto multilinecomment;
c669568326fb Reworked preprocessor newline management.
Ryan C. Gordon <icculus@icculus.org>
parents: 555
diff changeset
   183
                    }
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   184
    "\000"          {
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   185
                        if (eoi)
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   186
                            RET(TOKEN_INCOMPLETE_COMMENT);
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   187
                        goto multilinecomment;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   188
                    }
596
8efa254e8840 Renamed "any" to "ANY" to match other rules.
Ryan C. Gordon <icculus@icculus.org>
parents: 595
diff changeset
   189
    ANY             { goto multilinecomment; }
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   190
*/
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   191
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   192
singlelinecomment:
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   193
    if (YYLIMIT == YYCURSOR) YYFILL(1);
595
a72705c067dc Handle lexing of comment ends better.
Ryan C. Gordon <icculus@icculus.org>
parents: 594
diff changeset
   194
    matchptr = cursor;
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   195
/*!re2c
595
a72705c067dc Handle lexing of comment ends better.
Ryan C. Gordon <icculus@icculus.org>
parents: 594
diff changeset
   196
    NEWLINE         { s->line++; token = matchptr; RET('\n'); }
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   197
    "\000"          { if (eoi) { RET(TOKEN_EOI); } goto singlelinecomment; }
596
8efa254e8840 Renamed "any" to "ANY" to match other rules.
Ryan C. Gordon <icculus@icculus.org>
parents: 595
diff changeset
   198
    ANY             { goto singlelinecomment; }
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   199
*/
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   200
679
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   201
ppdirective:
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   202
    if (YYLIMIT == YYCURSOR) YYFILL(1);
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   203
/*!re2c
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   204
        PP "include"    { RET(TOKEN_PP_INCLUDE); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   205
        PP "line"       { RET(TOKEN_PP_LINE); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   206
        PP "define"     { RET(TOKEN_PP_DEFINE); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   207
        PP "undef"      { RET(TOKEN_PP_UNDEF); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   208
        PP "if"         { RET(TOKEN_PP_IF); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   209
        PP "ifdef"      { RET(TOKEN_PP_IFDEF); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   210
        PP "ifndef"     { RET(TOKEN_PP_IFNDEF); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   211
        PP "else"       { RET(TOKEN_PP_ELSE); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   212
        PP "elif"       { RET(TOKEN_PP_ELIF); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   213
        PP "endif"      { RET(TOKEN_PP_ENDIF); }
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   214
        PP "error"      { RET(TOKEN_PP_ERROR); }
910
3ee487d85ecb Handle #pragma (or at least, don't fail on them).
Ryan C. Gordon <icculus@icculus.org>
parents: 865
diff changeset
   215
        PP "pragma"     { RET(TOKEN_PP_PRAGMA); }
679
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   216
        WHITESPACE      { goto ppdirective; }
697
765a85a989a2 Fixed crash in lexer on empty input stream.
Ryan C. Gordon <icculus@icculus.org>
parents: 690
diff changeset
   217
765a85a989a2 Fixed crash in lexer on empty input stream.
Ryan C. Gordon <icculus@icculus.org>
parents: 690
diff changeset
   218
        ANY             {
765a85a989a2 Fixed crash in lexer on empty input stream.
Ryan C. Gordon <icculus@icculus.org>
parents: 690
diff changeset
   219
                            token = cursor = (const uchar *) s->source;
765a85a989a2 Fixed crash in lexer on empty input stream.
Ryan C. Gordon <icculus@icculus.org>
parents: 690
diff changeset
   220
                            limit = cursor + s->bytes_left;
765a85a989a2 Fixed crash in lexer on empty input stream.
Ryan C. Gordon <icculus@icculus.org>
parents: 690
diff changeset
   221
                            goto scanner_loop;
765a85a989a2 Fixed crash in lexer on empty input stream.
Ryan C. Gordon <icculus@icculus.org>
parents: 690
diff changeset
   222
                        }
679
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   223
*/
3892ebde7d99 Fixed lexer to only accept preprocessor directives at start of a line.
Ryan C. Gordon <icculus@icculus.org>
parents: 651
diff changeset
   224
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   225
bad_chars:
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   226
    if (YYLIMIT == YYCURSOR) YYFILL(1);
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   227
/*!re2c
601
bb1484be4e1b Reworked and documented preprocessor tokens.
Ryan C. Gordon <icculus@icculus.org>
parents: 597
diff changeset
   228
    ANYLEGAL        { cursor--; RET(TOKEN_BAD_CHARS); }
642
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   229
    "\000"          {
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   230
                        if (eoi)
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   231
                        {
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   232
                            assert( !((token >= sentinel) &&
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   233
                                     (token < sentinel+YYMAXFILL)) );
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   234
                            eoi = 0;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   235
                            cursor = (uchar *) s->source_base + s->orig_length;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   236
                            RET(TOKEN_BAD_CHARS);  // next call will be EOI.
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   237
                        }
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   238
                        goto bad_chars;
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   239
                    }
624ab8696f9b Enormous amount of tapdancing to handle EOI better in the lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 635
diff changeset
   240
597
832dbfa63509 Bogus chars in the lexer now return a token that signifies this.
Ryan C. Gordon <icculus@icculus.org>
parents: 596
diff changeset
   241
    ANY             { goto bad_chars; }
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   242
*/
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   243
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   244
    assert(0 && "Shouldn't hit this code");
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   245
    RET(TOKEN_UNKNOWN);
649
527cfe6f2d45 Renamed preprocessor_internal_lexer to preprocessor_lexer.
Ryan C. Gordon <icculus@icculus.org>
parents: 648
diff changeset
   246
} // preprocessor_lexer
555
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   247
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   248
// end of mojoshader_lexer_preprocessor.re (or .c) ...
940821555fda Initial work on preprocessor. Not yet complete!
Ryan C. Gordon <icculus@icculus.org>
parents:
diff changeset
   249