Sometimes you can't tokenize free of the parsing context. Is "MODULEPROCEDUREFOO" a Fortran statement that begins a new module called 'procedurefoo', or a reference to a module procedure 'foo'? Fortran compilers that use table-driven parsing typically feed state back to a "statement classifier", and the class of the statement determines how to tokenize it. Fortran compilers that use recursive descent typically just operate on characters; it seems to me to be more robust and error-recoverable to do it that way.