LexerInterface Class Reference

#include <lexer.h>

Detailed Description

Lexer class. This class provides functionality to scan text files. the stream is tokenized for easy reading.

Public Member Functions

MAXON_METHOD Result< void > Init (UrlOrInputStream &&fileName, LEXERINITFLAGS flags, const String &stringEscapeCharacters, const StringDecodingRef &stringDecoding, LEXERSYMBOLFLAGS firstParseFlag=LEXERSYMBOLFLAGS::NONE)
 
MAXON_METHOD Result< void > AddOperator (const String &chars)
 
MAXON_METHOD Result< void > Close ()
 
MAXON_METHOD Result< Utf32CharReadNextChar ()
 
MAXON_METHOD Result< BoolSkipSpaces ()
 
MAXON_METHOD Result< BoolSkipLineToEnd (Bool collectSkipped=false)
 
MAXON_METHOD Result< LEXERSYMBOLReadNextSymbol (LEXERSYMBOLFLAGS flags=LEXERSYMBOLFLAGS::NONE)
 
MAXON_METHOD Result< void > ReadNextSymbolExpected (LEXERSYMBOL expectedSymbols, LEXERSYMBOLFLAGS flags=LEXERSYMBOLFLAGS::NONE)
 
MAXON_METHOD void RewindSymbol ()
 
MAXON_METHOD LEXERSYMBOL GetSymbol () const
 
MAXON_METHOD Int GetLeadingSpace () const
 
MAXON_METHOD String GetIdent () const
 
MAXON_METHOD Result< void > GetFloat (Float32 &val) const
 
MAXON_METHOD Result< void > GetFloat (Float64 &val) const
 
MAXON_METHOD Result< void > GetInt (Int32 &val) const
 
MAXON_METHOD Result< void > GetInt (Int64 &val) const
 
MAXON_METHOD Result< void > GetUInt (UInt32 &val) const
 
MAXON_METHOD Result< void > GetUInt (UInt64 &val) const
 
MAXON_METHOD Result< Int64GetCurrentReadPosition () const
 
MAXON_METHOD Result< Int64GetSymbolPosition () const
 
MAXON_METHOD Bool CheckIdent (const Char *cmp) const
 
MAXON_METHOD Bool CheckIdent (const String &cmp) const
 
MAXON_METHOD Int GetLine () const
 
MAXON_METHOD void SetStringQuoteCharacters (const String &quoteChar)
 

Static Public Member Functions

static MAXON_METHOD LexerInterfaceAlloc (MAXON_SOURCE_LOCATION_DECLARATION)
 

Private Member Functions

 MAXON_INTERFACE_NONVIRTUAL (LexerInterface, MAXON_REFERENCE_NORMAL, "net.maxon.interface.lexer", MAXON_IMPLEMENTATION_MODULE("net.maxon.kernel"))
 

Member Function Documentation

◆ MAXON_INTERFACE_NONVIRTUAL()

MAXON_INTERFACE_NONVIRTUAL ( LexerInterface  ,
MAXON_REFERENCE_NORMAL  ,
"net.maxon.interface.lexer"  ,
MAXON_IMPLEMENTATION_MODULE("net.maxon.kernel")   
)
private

◆ Alloc()

Parameters
[in]allocLocationSource location.

◆ Init()

MAXON_METHOD Result<void> Init ( UrlOrInputStream &&  fileName,
LEXERINITFLAGS  flags,
const String stringEscapeCharacters,
const StringDecodingRef &  stringDecoding,
LEXERSYMBOLFLAGS  firstParseFlag = LEXERSYMBOLFLAGS::NONE 
)

Initializes the lexer class. This functions opens the stream.

Parameters
[in]fileNameUrl or Input Stream.
[in]flagsControl flags for the lexer. See LEXERINITFLAGS for description.
[in]stringEscapeCharactersDefines characters that allow to escape " ' and the escape character itself within strings. By Default this is '\'.
[in]stringDecodingString encoding for this file. If a nullptr is passed (recommended) then StreamConversions::UtfTextDecoder will be used to auto-detect the file encoding.
[in]firstParseFlagThe lexer will parse the first symbol from the file with ReadNextSymbol. This flag will be passed to the call.
Returns
OK on success.

◆ AddOperator()

MAXON_METHOD Result<void> AddOperator ( const String chars)

Adds an operator symbol to the lexer. This is a symbol which consists of several non-letter characters such as && or +=. The lexer will then return such a character sequence as a single symbol of type LEXERSYMBOL::OPERATOR.

Parameters
[in]charsThe symbol to add.

◆ Close()

MAXON_METHOD Result<void> Close ( )

Closes the stream.

◆ ReadNextChar()

MAXON_METHOD Result<Utf32Char> ReadNextChar ( )

Reads the next character out of the stream.

Returns
Returns the character that has been read. This function returns 0 if the end of the file has been reached.

◆ SkipSpaces()

MAXON_METHOD Result<Bool> SkipSpaces ( )

Skips the spaces from the current position.

Returns
True if the function was successful. False if the end of the file has been reached.

◆ SkipLineToEnd()

MAXON_METHOD Result<Bool> SkipLineToEnd ( Bool  collectSkipped = false)

Skips line to the end. If the lexer is in NEED_EOL mode the method stops at line end. Otherwise it will advance to the next line and stops when non-whitespace is found.

Parameters
[in]collectSkippedIf true then the skipped bytes are returned with the next GetIdent() call.
Returns
True if the function was successful. False if the end of the file has been reached. Otherwise an error.

◆ ReadNextSymbol()

Reads the next symbol in the stream. GetIdent() returns the name of the symbol/ident/number/operator.

Parameters
[in]flagsSee LEXERSYMBOLFLAGS for details.
Returns
Next detected symbol. LEXERSYMBOL::ENDOFFILE if the end of file has been reached.

◆ ReadNextSymbolExpected()

MAXON_METHOD Result<void> ReadNextSymbolExpected ( LEXERSYMBOL  expectedSymbols,
LEXERSYMBOLFLAGS  flags = LEXERSYMBOLFLAGS::NONE 
)

Reads the next symbol in the stream and checks if it's identical to requiredSymbols. GetIdent() returns the name of the symbol/ident/number/operator.

Parameters
[in]expectedSymbolsSymbol that is required.
[in]flagsSee LEXERSYMBOLFLAGS for details.
Returns
Next detected symbol. LEXERSYMBOL::ENDOFFILE if the end of file has been reached.

◆ RewindSymbol()

MAXON_METHOD void RewindSymbol ( )

Rewinds the current symbol so that the next ReadNextSymbol returns exactly the same as before.

◆ GetSymbol()

MAXON_METHOD LEXERSYMBOL GetSymbol ( ) const

Returns the current symbol.

◆ GetLeadingSpace()

MAXON_METHOD Int GetLeadingSpace ( ) const

Returns the number of leading spaces of the symbol. Tabs count as up to 8 spaces.

◆ GetIdent()

MAXON_METHOD String GetIdent ( ) const

Returns the current String ident.

◆ GetFloat() [1/2]

MAXON_METHOD Result<void> GetFloat ( Float32 val) const

In case of LEXERSYMBOL::NUMBER this function returns the floating point value of this number.

Parameters
[out]valReference to the value.

◆ GetFloat() [2/2]

MAXON_METHOD Result<void> GetFloat ( Float64 val) const

In case of LEXERSYMBOL::NUMBER this function returns the floating point value of this number.

Parameters
[in]valReference to the value.

◆ GetInt() [1/2]

MAXON_METHOD Result<void> GetInt ( Int32 val) const

In case of LEXERSYMBOL::NUMBER this function returns the integer value of this number.

Parameters
[in]valReference to the value.

◆ GetInt() [2/2]

MAXON_METHOD Result<void> GetInt ( Int64 val) const

In case of LEXERSYMBOL::NUMBER this function returns the integer value of this number.

Parameters
[in]valReference to the value.

◆ GetUInt() [1/2]

MAXON_METHOD Result<void> GetUInt ( UInt32 val) const

In case of LEXERSYMBOL::NUMBER this function returns the integer value of this number.

Parameters
[in]valReference to the value.

◆ GetUInt() [2/2]

MAXON_METHOD Result<void> GetUInt ( UInt64 val) const

In case of LEXERSYMBOL::NUMBER this function returns the integer value of this number.

Parameters
[in]valReference to the value.

◆ GetCurrentReadPosition()

MAXON_METHOD Result<Int64> GetCurrentReadPosition ( ) const

Returns the current reading position.

◆ GetSymbolPosition()

MAXON_METHOD Result<Int64> GetSymbolPosition ( ) const

Returns the position of the current symbol.

Returns
The position of the current symbol.

◆ CheckIdent() [1/2]

MAXON_METHOD Bool CheckIdent ( const Char cmp) const

Checks against the current identifier.

Parameters
[in]cmpC-style string to compare with.
Returns
Returns true if identical.

◆ CheckIdent() [2/2]

MAXON_METHOD Bool CheckIdent ( const String cmp) const

Checks against the current identifier.

Parameters
[in]cmpString to compare with.
Returns
Returns true if identical.

◆ GetLine()

MAXON_METHOD Int GetLine ( ) const

Returns the current line number within the file.

◆ SetStringQuoteCharacters()

MAXON_METHOD void SetStringQuoteCharacters ( const String quoteChar)

Sets the quote characters that are used to detect Strings when LEXERSYMBOLFLAGS::ALLOWSTRINGS is used.

Parameters
[in]quoteCharDefines characters that is used to detect quoted strings. By Default this is "\"'".