Building Scanners with Lex

Building Scanners with Lex

• There is a tool for building scanners that is very similar to Yacc named Lex.

• While Yacc views the input file as a unit onto which it must impose some structure, Lex views the input as a stream of many small entities it must recognize.

  A Lex definition is therefore a series of descriptions of the different types of "small entities" that might appear paired with "actions" (more C code) describing what to do if they appear.

• In Lex, a notation for writing regular expressions is used to describe the types of tokens the scanner should look for.

• Like a Yacc input file, a Lex input file has three parts separated from one another by lines beginning with a pair of percent signs.

  1. The first section contains declaration of two types:

     • C declarations to be included as part of the final C source for the parser function "yylex()".

     • declarations of identifers that can be used as predefined regular expression in the next "rule" section.

  2. The second section is the actual rules describing the scanner. Each rule consists of a regular expression and some C code to execute when the regular expression is "matched".

     • The two must appear on a line together separated by some blank space.

     • It is typical to place the C code within braces (this is required if there is more than a single statement).

  3. The third section can be used to add definitions of functions called by scanner actions.

• The fun part is the syntax for regular expressions.

  • Any non-special character (all alphabetics, numerics and some punctuation) can be used as regular expressions that match themselves.

  • Two regular expressions (including simple ones like single characters) concatenated together match any string formed by concatenating anything that matched the first regular expression with something that matched the second.

  • A regular expression formed by placing a " | " between two other expression matches anything that matches either of the sub-expressions.

  • parentheses can be used for grouping.

  • A regular expression followed by a "*" matches zero or more copies of the string matched by the original expression.

  • A regular expression followed by a "+" matches one or more copies of the original. A regular expression followed by a ? matches 0 or 1 copies of the original.

  • The regular expression . matches any single character.

  • The regular expression formed by placing a group of characters between square brackets ( i.e. [ and ]) matches any single character in the group.

  • Within square brackets, a sequence of consequtive characters can abbreviated using a dash as in "[a-z]".

  • etc.

• There are many other notations supported (read the handout). The most significant is that Lex needs to provide some way to tell it that the user wants to match a string containing a character like "(" or "*" that would normally be interpreted as a metacharacter.

  • Any quoted string (double quotes) matches itself.

  • A character preceded by a backslash, "\", matches the character itself.

• To improve the clarity of a Lex specification, you can associate names with commonly used or complex subexpressions.
  • The definitions of such names appear in the first section of the Lex input file (i.e. before the first %%).
  • Each definition is just a line containing a name separated from a regular expression by blanks and or tabs.
  • To use such a name in a later regular expression, you just surround the name with curly braces.

    alpha [a-zA-Z]
    ...
    %%
    {alpha}+     printf("I found a word"); 
    

• It is possible that several of the regular expressions used in rules in a Lex rule file might match prefixes of the same input. In such cases, Lex chooses the longest possible match and, among matches of equal length, the match with the earliest rule in the file.

• As in Yacc, the actions one pairs with the regular expressions in a Lex file are just bits of C code.

  • To return a token, simply include a "return" statement that returns the token's token number (which can be referred to symbolically using one of the #defined names from the y.tab.h file Yacc produces).

  • If you also need to associate a token value with the token, some extra code must be included in the action to assign the appropriate value to the variable "yylval".

  • "yylval" is actually defined external to Lex (by Yacc) and is of type YYSTYPE (the semantic value type used in the associated Yacc file).

• Within the actions, there are some special variables that can be used to access information about the strings matched:

  • yytext is the string matched.

  • yyleng is the length of the match.

• If you don't include a return in an action, Lex will happilly go on matching substrings to regular expressions.

• If no regular expression matches some portion of the input, Lex will copy the unmatched character to standard output.

• With these facts about Lex in mind, the following is an example of a Lex specfication appropriate for our dates example.

  %{
  
  #include "tn.h"
  
  typedef union {
    int num;
    char *str;
  } YYSTYPE;
  
  extern YYSTYPE yylval;
  
  char * makecopy( char * str);
  
  %}
  
  alfa [a-zA-Z]
  digit [0-9]
  
  %%
  ","       { return ','; }
  "/"       { return '/'; }
  {alfa}+   { yylval.str = makecopy(yytext); return MONTH;  }
  {digit}+  { yylval.num = atoi(yytext); return NUMBER; }
  .         { }
  
  %%
  
  char * makecopy( char * str) {
   char * result;
  
   result = malloc( strlen( str) + 1);
   strcpy(result,str);
   return result;
  }
  

Building Scanners with Lex