DOCUMENTATION.en.rdoc

REX: Ruby Lex for Racc¶ ↑

About¶ ↑

Lexical Scanner Generator for Ruby, with Racc.

Usage¶ ↑

rex [options] grammarfile

-o  --output-file  filename   designated output filename.
-s  --stub                    append stub main for debug.
-i  --ignorecase              ignore char case
-C  --check-only              syntax check only.
    --independent             independent mode.
-d  --debug                   print debug information
-h  --help                    print usage.
    --version                 print version.
    --copyright               print copyright.

Default Output Filename¶ ↑

It destinate from foo.rex  to foo.rex.rb.
This name is for a follow description.

  require 'foo.rex'

Grammar File Format¶ ↑

A definition is given in order of a header part, a rule part, 
and the footer part.  One or more sections are included in a rule part.
As for each section, the head of the sentence starts by the keyword.

Summary:

  [Header Part]
  "class" Foo
  ["option"
    [options] ]
  ["inner"
    [methods] ]
  ["macro"
    [macro-name  regular-expression] ]
  "rule"
    [start-state]  pattern  [actions]
  "end"
  [Footer Part]

Grammar File Example¶ ↑

class Foo
macro
  BLANK         \s+
  DIGIT         \d+
rule
  {BLANK}
  {DIGIT}       { [:NUMBER, text.to_i] }
  .             { [text, text] }
end

Header Part ( Optional )¶ ↑

All the contents described before the definition of a rule part are 
posted to head of the output file.

Footer Part ( Optional )¶ ↑

All the contents described after the definition of a rule part are 
posted to tail of the output file.

Rule Part¶ ↑

Rule part is from the line which begins from the "class" keyword 
to the line which begins from the "end" keyword.
The class name outputted after a keyword "class" is specified.
If embellished with a module name, it will become a class in a module.
The class which inherited Racc::Parser is generated.

Rule Header Example¶ ↑

class Foo
class Bar::Foo

Option Section ( Optional )¶ ↑

"option" is start keyword.

  "ignorecase"  when pattern match, ignore char case.
  "stub"        append stub main for debug.
  "independent" independent mode, for it is not inherited Racc.

Inner Section ( Optional )¶ ↑

"inner" is start keyword.
The contents defined here are defined by the inside of the class 
of the generated scanner.

Macro Section ( Optional )¶ ↑

"macro" is start keyword.
One regular expression is named.
A blank character (0x20) can be included by escaping by \ .

Macro Section Example¶ ↑

DIGIT         \d+
IDENT         [a-zA-Z_][a-zA-Z0-9_]*
BLANK         [\ \t]+
REMIN         \/\*
REMOUT        \*\/

Rule Section¶ ↑

"rule" is start keyword.

[state]  pattern  [actions]

state: Start State ( Optional )¶ ↑

A start state is expressed with the identifier which prefaces ":".
When the continuing alphabetic character is a capital letter, 
it will be in an exclusive start state.
When it is a small letter, it will be in an inclusive start state.
Initial value and default value of a start state is nil.

pattern: String Pattern¶ ↑

The regular expression for specifying a character string.
The macro definition bundled with { } can be used for description 
of a regular expression.  A macro definition is used in order to use 
a regular expression including a blank.

actions: Processing Actions ( Optional )¶ ↑

Action is performed when a pattern is suited.
The processing which creates a suitable token is defined.
The arrangement whose token has the second clause of classification 
and a value, or nil.
The following elements can be used in order to create a token.

  lineno    Line number     ( Read Only )
  text      Matched string  ( Read Only )
  state     Start state     ( Read/Write )

Action is bundled with { }.  It is the block of Ruby.
Don't use the function to change the flow of control exceeding a block.
( return, exit, next, break, ... )
If action is omitted, the character string which matched will be canceled 
and will progress to the next scan.

Rule Part Example¶ ↑

      {REMIN}                 { state = :REM ; [:REM_IN, text] }
:REM  {REMOUT}                { state = nil ; [:REM_OUT, text] }
:REM  (.+)(?={REMOUT})        { [:COMMENT, text] }
      {BLANK}
      -?{DIGIT}               { [:NUMBER, text.to_i] }
      {WORD}                  { [:word, text] }
      .                       { [text, text] }

Comment ( Optional )¶ ↑

From "#" to the end of the line becomes a comment in each line.

Usage for Generated Class¶ ↑

scan_setup()¶ ↑

The event for initializing at the time of the execution start of a scanner.
It is redefined and used.

scan_str( str )¶ ↑

Parse the string described by the defined grammar.
Token is stored in an inside.

scan_file( filename )¶ ↑

Parse the file described by the defined grammar.
Token is stored in an inside.

next_token¶ ↑

One token stored in the inside is taken out.
The last returns nil.

Notice¶ ↑

This specification is provisional and may be changed without a preliminary 
announcement.