Writing Regular Expressions for Lexer Tokenization in Rust

Mục tiêu: A guide on creating regular expressions for tokenizing elements of a programming language syntax in Rust.

Writing Regular Expressions for Lexer Tokenization in Rust

When building a lexer for your custom programming language interpreter, regular expressions (regex) are essential for accurately matching and tokenizing different elements of your language syntax. In this article, we’ll focus on writing regex patterns for various token types, ensuring they are correctly identified during the tokenization process.

Understanding Token Types

Before writing regex patterns, let’s identify the basic token types your language will support:

Identifiers: Names for variables, functions, etc. (e.g., myVariable)
Keywords: Reserved words (e.g., fn, if, else)
Literals: Numerical values (e.g., 123, 45.67)
Operators: Symbols for operations (e.g., +, -, *)
Symbols: Parentheses, braces, etc. (e.g., (, {)

Writing Regex Patterns

Here are the regex patterns for each token type:

Identifiers:
Pattern: [a-zA-Z_][a-zA-Z0-9_]*
Matches: Letters, digits, and underscores, starting with a letter or underscore.
Keywords:
Pattern: (fn|if|else|for|while|return|let|const)
Matches: Specific reserved words in your language.
Integer Literals:
Pattern: \d+
Matches: One or more digits.
Boolean Literals:
Pattern: (true|false)
Matches: Boolean values true or false.
Operators:
Pattern: (\+|\-|\*|\/|\%|\=\=|\!\=|\<|\>|\=\=|\!\=)
Matches: Arithmetic and comparison operators.
Symbols:
Pattern: (\(|\)|\{|\}|\[|\]|,|;|:)
Matches: Various symbols used in the language structure.

Implementing the Lexer

Here’s how you can implement these regex patterns in Rust:

use regex::Regex;

// Define token types
#[derive(Debug)]
enum Token {
    Identifier(String),
    Keyword(String),
    Integer(i32),
    Boolean(bool),
    Operator(String),
    Symbol(String),
    Error(String),
}

// Lexer struct
struct Lexer {
    input: String,
    position: usize,
    line: usize,
    column: usize,
}

impl Lexer {
    fn new(input: String) -> Self {
        Lexer {
            input,
            position: 0,
            line: 1,
            column: 1,
        }
    }

    // Method to get the next token
    fn get_token(&mut self) -> Token {
        // Define all regex patterns
        let identifier_regex = Regex::new(r"^[a-zA-Z_][a-zA-Z0-9_]*").unwrap();
        let keyword_regex = Regex::new(r"^(fn|if|else|for|while|return|let|const)").unwrap();
        let integer_regex = Regex::new(r"^\d+").unwrap();
        let boolean_regex = Regex::new(r"^(true|false)").unwrap();
        let operator_regex = Regex::new(r"^[+\-*/%]=|!=|<=|>=|==|!=|<|>|=|").unwrap();
        let symbol_regex = Regex::new(r"^()[\]{};,:").unwrap();

        let substr = &self.input[self.position..];

        // Try to match each pattern in order
        if let Some(match_) = identifier_regex.find(substring) {
            let token = match_.as_str().to_string();
            self.position += match_.end();
            self.column += match_.end();
            return Token::Identifier(token);
        }
        // Similar checks for other token types...

        // If no pattern matches, return error
        Token::Error(format!("Unexpected character: {}", substr.chars().nth(0).unwrap()))
    }
}

Key Considerations

Order of Checking: The order in which you check for tokens matters. For example, keywords should be checked before identifiers to avoid misclassifying reserved words.
Whitespace Handling: Include patterns for whitespace and comments to skip over them during tokenization.
Position Tracking: Keep track of line and column numbers for better error reporting.

Next Steps

After defining your regex patterns and implementing the basic lexer, you’ll want to:

Implement Parser: Use the tokens generated by the lexer to build an Abstract Syntax Tree (AST).
Add Error Handling: Expand your error handling to provide meaningful messages for invalid syntax.
Test Thoroughly: Write test cases to ensure your lexer correctly identifies all token types.

Implementing Lexer Logic in Rust

Mục tiêu: The task involves implementing a lexer in Rust that tokenizes input source code into keywords, identifiers, literals, operators, and symbols. It includes defining a Lexer struct, handling whitespace and comments, and generating tokens with appropriate error handling.

Implementing the Lexer Logic in Rust

The lexer is a critical component of any interpreter or compiler. Its primary responsibility is to take the input source code as a string and convert it into a sequence of tokens. These tokens represent the basic elements of the language such as keywords, identifiers, literals, operators, and symbols.

In this task, we will implement the lexer logic that will tokenize the input source code based on the regular expressions we defined in the previous task. We will create a Lexer struct that will handle the tokenization process and produce a stream of tokens.

Step-by-Step Implementation

Lexer Struct Definition

We start by defining a Lexer struct that will hold the input string, the current position in the input, and the current line and column numbers. This struct will be used to maintain the state of the lexer as it processes the input.

/// A struct that represents the lexer state
pub struct Lexer {
input: String,
position: usize,
line: usize,
column: usize,
}

impl Lexer {
/// Creates a new Lexer instance with the given input string
pub fn new(input: String) -> Self {
Self {
input,
position: 0,
line: 1,
column: 1,
}
}

/// The main method that tokenizes the input and returns tokens pub fn lex(&mut self) -> Token { // Implementation of the lex method }

Token Definition

Before we implement the lexer logic, we need to define the Token enum that will represent the different types of tokens. Each token will have a TokenKind and a String value.

/// Represents the different types of tokens
#[derive(Debug, PartialEq)]
pub enum TokenKind {
Identifier,
Keyword,
Integer,
Float,
Operator,
Symbol,
Comment,
Whitespace,
Error,
}

/// Represents a token produced by the lexer
#[derive(Debug)]
pub struct Token {
pub kind: TokenKind,
pub value: String,
pub line: usize,
pub column: usize,
}

Lexer Logic Implementation

The lex method is where the core tokenization logic resides. This method will process each character in the input string and determine the type of token it represents.

pub fn lex(&mut self) -> Token {
// Skip whitespace and comments
self.skip\_whitespace\_and\_comments();

   // Check if we've reached the end of the input
   if self.position >= self.input.len() {
       return Token {
           kind: TokenKind::Error,
           value: "Unexpected end of input".to_string(),
           line: self.line,
           column: self.column,
       };
   }

   // Get the current character
   let current_char = self.input[self.position..].chars().next().unwrap();

   // Check for identifier or keyword
   if current_char.is_alphabetic() {
       let mut identifier = String::new();
       while self.position < self.input.len() &&
            self.input[self.position..].chars().next().unwrap().is_alphanumeric() {
           identifier.push(current_char);
           self.position += 1;
           current_char = self.input[self.position..].chars().next().unwrap();
       }

       // Determine if the identifier is a keyword
       let kind = match identifier.as_str() {
           "fn" => TokenKind::Keyword,
           "let" => TokenKind::Keyword,
           _ => TokenKind::Identifier,
       };

       self.column += identifier.len();
       return Token {
           kind,
           value: identifier,
           line: self.line,
           column: self.column - identifier.len(),
       };
   }

   // Check for number (integer or float)
   if current_char.is_digit(10) || current_char == '.' {
       let mut number = String::new();
       let mut has_decimal = false;
       let mut has_exponent = false;

       while self.position < self.input.len() &&
            (current_char.is_digit(10) ||
             (current_char == '.' && !has_decimal) ||
             (current_char == 'e' && !has_exponent) ||
             (current_char == 'E' && !has_exponent)) {
           number.push(current_char);
           self.position += 1;
           current_char = self.input[self.position..].chars().next().unwrap();

           if current_char == '.' {
               has_decimal = true;
           } else if current_char == 'e' || current_char == 'E' {
               has_exponent = true;
           }
       }

       return Token {
           kind: TokenKind::Float,
           value: number,
           line: self.line,
           column: self.column,
       };
   }

   // Check for operator or symbol
   for op in &["+", "-", "*", "/", "%", "=", "==", "!=", ">", "<", ">=", "<="] {
       if self.input[self.position..].starts_with(op) {
           let value = op.to_string();
           let length = op.len();
           self.position += length;
           self.column += length;
           return Token {
               kind: TokenKind::Operator,
               value,
               line: self.line,
               column: self.column - length,
           };
       }
   }

   // Handle remaining cases
   match current_char {
       '(' => {
           self.position += 1;
           self.column += 1;
           return Token {
               kind: TokenKind::Symbol,
               value: "(".to_string(),
               line: self.line,
               column: self.column,
           };
       }
       ')' => {
           self.position += 1;
           self.column += 1;
           return Token {
               kind: TokenKind::Symbol,
               value: ")".to_string(),
               line: self.line,
               column: self.column,
           };
       }
       ',' => {
           self.position += 1;
           self.column += 1;
           return Token {
               kind: TokenKind::Symbol,
               value: ",".to_string(),
               line: self.line,
               column: self.column,
           };
       }
       ';' => {
           self.position += 1;
           self.column += 1;
           return Token {
               kind: TokenKind::Symbol,
               value: ";".to_string(),
               line: self.line,
               column: self.column,
           };
       }
       _ => {
           // If none of the above, it's an error
           self.position += 1;
           self.column += 1;
           return Token {
               kind: TokenKind::Error,
               value: format!("Unexpected character: {}", current_char),
               line: self.line,
               column: self.column,
           };
       }
   }    }

/// Skips whitespace and comments in the input fn skip_whitespace_and_comments(&mut self) { while self.position < self.input.len() { let current_char = self.input[self.position..].chars().next().unwrap();

       // Skip whitespace
       if current_char.is_whitespace() {
           self.position += 1;
           if current_char == '\n' {
               self.line += 1;
               self.column = 1;
           } else {
               self.column += 1;
           }
           continue;
       }

       // Skip line comments starting with //
       if current_char == '/' && self.position + 1 < self.input.len() &&
          self.input[self.position + 1..].chars().next().unwrap() == '/' {
           // Move to the end of the line
           let mut i = self.position;
           while i < self.input.len() && self.input[i..].chars().next().unwrap() != '\n' {
               i += 1;
           }
           self.position = i;
           self.line += 1;
           self.column = 1;
           continue;
       }

       // Skip block comments enclosed in /* */
       if current_char == '/' && self.position + 1 < self.input.len() &&
          self.input[self.position + 1..].chars().next().unwrap() == '*' {
           let mut i = self.position + 2;
           while i < self.input.len() &&
                 !(self.input[i..].chars().next().unwrap() == '*' &&
                   i + 1 < self.input.len() &&
                   self.input[i + 1..].chars().next().unwrap() == '/') {
               i += 1;
           }
           if i < self.input.len() && self.input[i..].chars().next().unwrap() == '*' &&
              i + 1 < self.input.len() && self.input[i + 1..].chars().next().unwrap() == '/' {
               i += 2;
           }
           self.position = i;
           // Update line and column counts
           let mut lines = self.input[self.position..].chars().collect::<Vec<char>>();
           let line_count = lines.iter().filter(|c| **c == '\n').count();
           self.line += line_count;
           self.column = self.position - self.position.lastIndexOf('\n').unwrap_or(self.position);
           continue;
       }

       // If none of the above, break the loop
       break;
   }    } ```

}

1. **Testing the Lexer**

To ensure that the lexer works correctly, we can write some test cases:

```rust
#[cfg(test)]
mod tests {
use super::\*;

#[test] fn test_lexer() { let input = “ fn main() { let x = 10; let y = 20.5; println!("Sum: {}", x + y); } “;

   let mut lexer = Lexer::new(input.to_string());
   let tokens: Vec<Token> = loop {
       let token = lexer.lex();
       if token.kind == TokenKind::Error {
           break vec![token];
       }
       // Add more conditions to collect tokens
   };

   // Assert tokens
   // ...    } ```

}

1. **Handling Line and Column Tracking**

The lexer keeps track of the current line and column as it processes the input. This is essential for providing accurate error messages. The `skip_whitespace_and_comments` method updates the line and column counts as it processes whitespace and comments.

1. **Error Handling**

The lexer includes basic error handling for unexpected characters. If the lexer encounters a character that doesn't match any token pattern, it returns an error token with the unexpected character and its position.

#### Next Steps

Now that we have implemented the lexer, the next step is to implement the parser. The parser will take the tokens produced by the lexer and construct an abstract syntax tree (AST) based on the grammar rules we defined earlier.

#### Further Reading

* [Lexer Implementation in Rust](https://doc.rust-lang.org/book/ch08-00-parsing-command-line-arguments.html)
* [Tokenization in Programming Languages](https://en.wikipedia.org/wiki/Lexical_analysis)
* [Rust String Manipulation](https://doc.rust-lang.org/std/string/)

This implementation provides a solid foundation for the lexer component of our interpreter. It handles various token types, skips whitespace and comments, and includes basic error handling.

## Handling Different Token Types in Rust Lexer

*Mục tiêu: A guide on implementing token handling in a Rust-based interpreter, covering token types, lexer implementation, and line tracking.*

---

# Handling Different Token Types in Rust Lexer

Handling different token types is a crucial part of implementing a lexer for your Rust-based interpreter. Tokens are the basic building blocks of your programming language, and correctly identifying them ensures that your parser can construct the correct Abstract Syntax Tree (AST). In this article, we'll explore how to handle various token types, including identifiers, keywords, operators, and literals.

## Understanding Token Types

Before diving into the implementation, let's review the different types of tokens your language needs to support. Based on your language's syntax, you should have tokens for:

* **Identifiers**: Names given to variables, functions, etc.
* **Keywords**: Reserved words like `if`, `else`, `while`, `fn`, etc.
* **Operators**: Symbols like `+`, `-`, `*`, `/`, `==`, etc.
* **Literals**: Numerical values (`123`, `45.67`), boolean values (`true`, `false`), etc.
* **Parentheses**: `(` and `)`
* **Control Symbols**: `{`, `}`, `;`, etc.

## Implementing Token Handling

### 1. Defining Token Types

First, let's define an enum to represent all possible token types. This enum will help in organizing and handling different tokens systematically:

#[derive(Debug, PartialEq)] pub enum Token { Identifier(String), Keyword(Keyword), Operator(Operator), Literal(Literal), LeftParen, RightParen, LeftBrace, RightBrace, Semicolon, Comma, Dot, Slash, Bang, Tilde, Plus, Minus, Star, Percent, Caret, Eq, Neq, Lt, Gt, Leq, Geq, Assign, Arrow, Colon, DoubleColon, // Add more tokens as needed }

// Define sub-enums for keywords, operators, and literals #[derive(Debug, PartialEq)] pub enum Keyword { If, Else, While, For, Fn, Let, Return, // Add more keywords as needed }

#[derive(Debug, PartialEq)] pub enum Operator { Add, Subtract, Multiply, Divide, Modulus, Equal, NotEqual, LessThan, GreaterThan, LessOrEqual, GreaterOrEqual, Assign, // Add more operators as needed }

#[derive(Debug, PartialEq)] pub enum Literal { Number(f64), Boolean(bool), String(String), // Add more literal types as needed }

### 2. Matching Tokens

The lexer's main job is to read the input character by character and produce tokens. For each character, the lexer needs to determine what type of token it belongs to. Here's how you can implement this:

pub struct Lexer { input: String, position: usize, line: usize, column: usize, }

impl Lexer { pub fn new(input: String) -> Self { Lexer { input, position: 0, line: 1, column: 1, } }

pub fn next_token(&mut self) -> Token {
    // Skip whitespace and comments
    while self.position < self.input.len() && self.is_whitespace_or_comment() {
        if self.input[self.position..].starts_with('\n') {
            self.line += 1;
            self.column = 1;
        } else {
            self.column += 1;
        }
        self.position += 1;
    }

    // Check for end of input
    if self.position >= self.input.len() {
        return Token::Eof;
    }

    let current_char = self.input[self.position];

    // Match identifiers and keywords
    if current_char.is_alphabetic() {
        let identifier = self.read_identifier();
        return match Keyword::from_str(&identifier) {
            Some(keyword) => Token::Keyword(keyword),
            None => Token::Identifier(identifier),
        };
    }

    // Match numbers
    if current_char.is_digit(10) || current_char == '.' {
        let number = self.read_number();
        return Token::Literal(Literal::Number(number));
    }

    // Match operators
    match current_char {
        '+' => {
            self.position += 1;
            return Token::Operator(Operator::Add);
        }
        '-' => {
            self.position += 1;
            return Token::Operator(Operator::Subtract);
        }
        '*' => {
            self.position += 1;
            return Token::Operator(Operator::Multiply);
        }
        '/' => {
            self.position += 1;
            return Token::Operator(Operator::Divide);
        }
        '%' => {
            self.position += 1;
            return Token::Operator(Operator::Modulus);
        }
        '=' => {
            if self.position + 1 < self.input.len() && self.input[self.position + 1] == '=' {
                self.position += 2;
                return Token::Operator(Operator::Equal);
            } else {
                self.position += 1;
                return Token::Assign;
            }
        }
        '!' => {
            if self.position + 1 < self.input.len() && self.input[self.position + 1] == '=' {
                self.position += 2;
                return Token::Operator(Operator::NotEqual);
            } else {
                self.position += 1;
                return Token::Bang;
            }
        }
        // Add more operator matching logic as needed
        _ => {}
    }

    // Handle parentheses and other symbols
    match current_char {
        '(' => {
            self.position += 1;
            return Token::LeftParen;
        }
        ')' => {
            self.position += 1;
            return Token::RightParen;
        }
        '{' => {
            self.position += 1;
            return Token::LeftBrace;
        }
        '}' => {
            self.position += 1;
            return Token::RightBrace;
        }
        ';' => {
            self.position += 1;
            return Token::Semicolon;
        }
        ',' => {
            self.position += 1;
            return Token::Comma;
        }
        '.' => {
            self.position += 1;
            return Token::Dot;
        }
        // Add more symbol matching as needed
        _ => {}
    }

    // If none of the above matches, it's an unknown token
    self.position += 1;
    return Token::Unknown(current_char.to_string());
}

// Helper methods for reading identifiers and numbers
fn read_identifier(&mut self) -> String {
    let start = self.position;
    while self.position < self.input.len() && self.input[self.position].is_alphanumeric() {
        self.position += 1;
    }
    String::from(&self.input[start..self.position])
}

fn read_number(&mut self) -> f64 {
    let start = self.position;
    while self.position < self.input.len() &&
          (self.input[self.position].is_digit(10) || self.input[self.position] == '.') {
        self.position += 1;
    }
    String::from(&self.input[start..self.position]).parse::<f64>().unwrap()
}

fn is_whitespace_or_comment(&mut self) -> bool {
    let c = self.input[self.position];
    c == ' ' || c == '\t' || c == '\n' || c == '/' && self.position + 1 < self.input.len() && self.input[self.position + 1] == '/'
} } ```

3. Handling Line and Column Tracking

Proper line and column tracking is essential for providing accurate error messages. The lexer should keep track of its current position in the input, including the line and column numbers. Update the line and column counters whenever you encounter a newline or when moving to the next character.

4. Error Handling

The lexer should handle unexpected characters gracefully. When encountering an unknown token, the lexer should emit an error token with information about the unexpected character and its position.

Example Usage

Here’s an example of how you can use the lexer to tokenize a simple input:

fn main() {
    let input = "
        let x = 5;
        if x > 10 {
            println!(\"x is greater than 10\");
        } else {
            println!(\"x is less than or equal to 10\");
        }
    ";

    let mut lexer = Lexer::new(input.to_string());
    loop {
        let token = lexer.next_token();
        match token {
            Token::Eof => break,
            _ => println!("{:?}", token),
        }
    }
}

Next Steps

Now that you’ve implemented the lexer’s ability to handle different token types, the next step is to implement the parser. The parser will take the tokens produced by the lexer and construct an Abstract Syntax Tree (AST) according to the language’s grammar rules.

Implementing Line and Column Tracking in Rust Lexer

Mục tiêu: Enhance error reporting in a Rust-based interpreter by implementing line and column tracking in the lexer.

Implementing Line and Column Tracking in Rust Lexer

To enhance error reporting in our Rust-based interpreter, we need to implement line and column tracking in the lexer. This will allow us to provide more precise error messages by pinpointing the exact location of issues in the source code.

Why Line and Column Tracking?

Better Error Reporting: When the lexer encounters an invalid character or syntax error, it can report the exact line and column number where the issue occurred.
Improved Developer Experience: Developers using our interpreter will appreciate the detailed feedback, making debugging easier.
Foundation for Parsing: Accurate line and column information will be crucial for the parser and subsequent error handling.

Approach

Position Tracking Struct: Create a Position struct to keep track of the current line and column numbers.
Lexer State: Integrate the Position struct into the lexer’s state to update the position as we process each character.
Character Processing: As the lexer processes each character, update the column number. When encountering a newline character, increment the line number and reset the column number.

Implementation Code

// Define a struct to hold the current position in the source code
#[derive(Debug, Clone, Copy)]
pub struct Position {
    pub line: usize,
    pub column: usize,
}

impl Position {
    // Create a new Position instance starting at line 1, column 1
    fn new() -> Self {
        Position {
            line: 1,
            column: 1,
        }
    }

    // Update the position when a newline character is encountered
    pub fn newline(&mut self) {
        self.line += 1;
        self.column = 1;
    }

    // Update the position for any other character
    pub fn next(&mut self) {
        self.column += 1;
    }
}

// Example usage within the lexer
pub struct Lexer {
    input: String,
    position: Position,
    // ... other lexer state ...
}

impl Lexer {
    pub fn new(input: String) -> Self {
        Lexer {
            input,
            position: Position::new(),
            // ... initialize other state ...
        }
    }

    // Process each character and update position
    pub fn process_input(&mut self) {
        for c in self.input.chars() {
            match c {
                '\n' => {
                    self.position.newline();
                    // ... handle newline in tokenization ...
                }
                _ => {
                    self.position.next();
                    // ... process the character for tokenization ...
                }
            }
        }
    }
}

Explanation

Position Struct: The Position struct maintains the current line and column numbers. It provides methods to handle newlines and character advancement.
Lexer Integration: The lexer now includes the Position struct as part of its state. As it processes each character, it updates the position accordingly.
Error Reporting: When an error occurs, the lexer can reference the Position to provide detailed information about where the issue occurred.

Next Steps

Now that we’ve implemented line and column tracking, the next steps would be:

Token Handling: Implement the logic to handle different types of tokens (identifiers, keywords, operators, etc.).
Error Handling: Use the position information to provide detailed error messages for invalid characters and unexpected input.

Implementing Error Handling in the Lexer for Rust Programming Language Interpreter

Mục tiêu: Adding error handling for invalid characters and unexpected input in a Rust-based lexer to ensure meaningful feedback for malformed input.

Implementing Error Handling in the Lexer for Rust Programming Language Interpreter

Error handling is a crucial part of building a robust lexer. In this task, we’ll focus on adding error handling for invalid characters and unexpected input in our Rust-based lexer. This will ensure that our interpreter can gracefully handle malformed input and provide meaningful feedback to the user.

Understanding the Need for Error Handling

When processing source code, the lexer must handle various edge cases, such as:

Invalid Characters: Characters that are not part of the language syntax (e.g., ‘@’, invalid Unicode characters).
Unexpected Input: Input that doesn’t match any defined token patterns.
Error Recovery: Continuing lexing after an error occurs to find subsequent errors in a single pass.

Approach to Error Handling

Custom Error Type: Define a custom error type to represent different kinds of lexing errors.
Error Tracking: Modify the lexer to track and collect errors while processing the input.
Position Tracking: Use line and column information to provide context for errors.
Error Reporting: Return errors along with tokens for the parser to handle appropriately.

Implementation Details

Let’s implement these concepts step by step.

// Define a custom error type for lexing errors
use thiserror::Error;

#[derive(Error, Debug)]
pub enum LexError {
    #[error("Invalid character '{0}' at line {1}, column {2}")]
    InvalidCharacter(char, usize, usize),
    #[error("Unexpected input '{0}' at line {1}, column {2}")]
    UnexpectedInput(String, usize, usize),
}

// Define a result type for lexing operations
pub type LexResult<T> = Result<T, LexError>;

Modifying the Lexer

We’ll modify the lexer to return a LexResult and collect errors:

// Updated lexer function signature
pub fn lex(source: &str) -> LexResult<Vec<Token>> {
    let mut tokens = Vec::new();
    let mut errors = Vec::new();

    let mut iter = source.chars().enumerate();
    let mut line = 1;
    let mut column = 1;

    while let Some((idx, c)) = iter.next() {
        // Update line and column positions
        if c == '\n' {
            line += 1;
            column = 1;
        } else {
            column += 1;
        }

        // Attempt to match different token types
        match c {
            // ... existing token matching logic ...

            // Handle invalid characters
            _ if c.is_ascii_graphic() => {
                // If character is not matched by any token pattern, treat as invalid
                let error = LexError::InvalidCharacter(c, line, column);
                errors.push(error);
            },
            _ => {
                // Handle non-printable characters
                let error = LexError::UnexpectedInput(c.to_string(), line, column);
                errors.push(error);
            }
        }
    }

    // If there are errors, return them
    if !errors.is_empty() {
        return Err(LexError::InvalidCharacter('?', line, column));
    }

    Ok(tokens)
}

Handling Errors in the Lexer

The lexer now collects errors and returns them using our custom error type. This allows the parser and higher-level components to handle errors gracefully.

Best Practices and Considerations

Error Recovery: After encountering an error, the lexer should attempt to recover and continue processing the input to find subsequent errors.
Position Tracking: Accurate line and column tracking is essential for meaningful error messages.
Error Messages: Error messages should be clear and provide enough context for the user to identify and fix issues.
Testing: Thoroughly test the lexer with various invalid inputs to ensure error handling works as expected.

Next Steps

Now that we’ve implemented error handling for the lexer, the next steps would be:

Integrate Error Handling with the Parser: Modify the parser to handle errors returned by the lexer.
Implement Error Reporting: Create a system to report errors to the user in a user-friendly format.
Add More Error Cases: Continue adding error handling for other potential issues, such as invalid token sequences.