Define the Lexical Token Enum in Rust

Mục tiêu: Create a Token enum in Rust to represent all possible lexical units for the SimpliLang compiler. This includes defining variants for keywords, literals, identifiers, operators, punctuation, and special tokens like EOF and Error.

With your comprehensive EBNF grammar and suite of .spl example files, the design phase of SimpliLang is officially complete. This is a fantastic achievement that provides a solid blueprint for implementation. We now transition from being language designers to compiler engineers.

Our first task is to build the initial component of the compiler’s frontend: the lexer (also known as a tokenizer or scanner). Before our compiler can understand the complex grammatical structure of a program (like a function definition), it must first break the raw source code text into a sequence of individual, meaningful “words” and symbols. These fundamental units are called tokens.

Think of it like reading an English sentence. Before you can understand the meaning of “The quick brown fox jumps.”, your brain first recognizes the individual words (“The”, “quick”, “brown”, “fox”, “jumps”) and the punctuation mark (“.”). The lexer performs this exact job for our SimpliLang source code. It scans the text character by character and groups them into discrete tokens, discarding irrelevant information like whitespace along the way. This process transforms a messy String of characters into a clean, structured stream of tokens, which is a much easier format for the next phase, the parser, to work with.

Our immediate goal is to define a data structure that can represent every possible token in our language. In Rust, the enum is the perfect tool for this, as it allows us to define a type that can be one of several distinct variants.

Creating the Token “Alphabet”

Let’s create the file that will house our Token enum. Inside your src/lexer/ directory, create a new file named token.rs.

touch src/lexer/token.rs

Now, open this new file. We will define a Token enum where each variant corresponds to a terminal symbol from our EBNF grammar or a category of symbols. We’ll group them by category for clarity.

// src/lexer/token.rs

/// Represents a single, discrete token in the SimpliLang source code.
///
/// The `#[derive(...)]` attribute automatically implements traits for our enum.
/// - `Debug`: Allows us to print the token for debugging using `println!("{:?}", token);`.
/// - `Clone`: Allows us to create copies of tokens.
/// - `PartialEq`: Allows us to compare two tokens for equality (useful for testing).
///   Note: Direct comparison of `f64` values can be tricky due to precision, but
///   for our compiler's purposes, this is generally sufficient.
#[derive(Debug, Clone, PartialEq)]
pub enum Token {
    // --- Keywords ---
    // Reserved words that have special meaning in the language.
    Fn,     // "fn"
    Let,    // "let"
    If,     // "if"
    Else,   // "else"
    Return, // "return"

    // --- Literals ---
    // These variants store the actual value of the literal found in the code.
    Int(i64),         // e.g., 123, 42
    Float(f64),       // e.g., 3.14, 9.81
    Bool(bool),       // "true" or "false"

    // --- Identifiers ---
    // Names for variables and functions, storing the name as a String.
    Identifier(String), // e.g., "my_variable", "add"

    // --- Operators ---
    // Symbols that perform operations.
    Assign,     // =
    Plus,       // +
    Minus,      // -
    Star,       // *
    Slash,      // /
    Equal,      // ==
    BangEqual,  // !=
    Less,       // <
    Greater,    // >

    // --- Punctuation / Delimiters ---
    // Characters that structure the code.
    LParen,     // (
    RParen,     // )
    LBrace,     // {
    RBrace,     // }
    Comma,      // ,
    Colon,      // :
    Semicolon,  // ;
    Arrow,      // ->

    // --- Special Tokens ---
    // These are not generated from source text directly but are useful for the parsing process.

    /// Represents a character or sequence of characters that is not valid in SimpliLang.
    /// The string inside will contain the problematic character(s) for error reporting.
    Error(String),

    /// Represents the End of the File. This signals to the parser that there's no more input.
    Eof,
}

Dissecting the Token Enum

  • Value-Carrying Variants: Notice that Int, Float, Bool, and Identifier are not just simple tags. They carry data inside them. Token::Int(100) is a distinct value from Token::Int(42). This is a powerful feature of Rust enums. It means the token itself contains the literal value, so the parser doesn’t need to look back at the original source code to find it. We chose i64 and f64 to align with the design decisions we made for our int and float primitive types.
  • Keywords vs. Identifiers: It’s crucial for the lexer to distinguish between a keyword like let and an identifier that happens to be spelled “let”. We have separate variants for all keywords. The lexer’s logic will be: if a word matches a keyword, create the keyword token; otherwise, create an Identifier token.
  • Special Tokens (Error, Eof):
    • Error: No lexer is perfect. It needs a way to signal that it encountered something it didn’t recognize, like a @ or $ symbol. The Error token allows the lexer to report this without crashing.
    • Eof: This is a vital signal for the parser. When it receives the Eof token, it knows that the input stream has been fully consumed and it can proceed to validate the final structure.

Integrating with the Module System

Now that we’ve defined our Token enum, we need to make it accessible to the rest of our compiler. We’ll do this by updating the mod.rs file in our lexer directory, following the same re-exporting pattern we used in the ast module.

Open src/lexer/mod.rs and add the following code:

// src/lexer/mod.rs

// Declare a public module named `token`, which corresponds to the file `token.rs`.
pub mod token;

// Re-export the contents of the `token` module (specifically, our `Token` enum)
// so that other parts of the compiler can access it via `lexer::Token` instead
// of the more verbose `lexer::token::Token`.
pub use token::*;

By defining this Token enum, you have created the vocabulary of your language. You’ve given your compiler the building blocks it needs to start understanding the .spl files you wrote.

Next Steps

Our Token enum is a great start, but it’s missing a crucial piece of information. When the parser or semantic analyzer finds an error, it needs to be able to tell the user where the error occurred. “Syntax error” is not helpful, but “Syntax error on line 10, column 25” is.

In the very next task, we will enhance our Token structure to include this vital location information (line and column numbers).

Further Reading

Enhancing Compiler Tokens with Source Code Spans

Mục tiêu: Refactor the lexer’s token structure by separating the token’s type (TokenKind) from its location (Span) to enable precise error reporting and compiler diagnostics.

Excellent work defining the complete “alphabet” for SimpliLang with the Token enum. You’ve created a comprehensive vocabulary that our compiler can use to understand the source code. However, a token’s type is only half the story. To build a compiler that provides helpful feedback to its users, we must also know where each token came from.

The Power of Location: From “What” to “Where”

Imagine your compiler finds a type error. A message like “Type Mismatch” is technically correct but incredibly unhelpful. A developer’s immediate questions would be, “Where? Which part of my code has the mismatch?” This is where location information becomes indispensable. By tracking the line and column number of every single token, we can later produce human-friendly error messages, often called diagnostics, that pinpoint the exact location of a problem.

A great diagnostic looks like this:

Error: Mismatched types
  --> examples/my_program.spl:5:21
   |
 5 | let x: int = 10 + false;
   |                     ^^^^^ expected int, found bool

To generate such messages, every token needs to carry its location with it throughout the entire compilation process. We will achieve this by enhancing the token structure we just created.

Introducing Spans: Capturing Source Code Position

A common and robust way to represent a token’s location is with a span. A span represents a contiguous slice of the source file, marked by a start and end position. For maximum flexibility, we will store these positions as byte offsets from the beginning of the file. This is more versatile than line/column numbers because it’s easier to work with, and line/column information can always be calculated from the byte offsets later when needed.

Let’s refactor our token definition to incorporate this concept. This involves a common pattern in compiler development:

  1. Rename our Token enum to TokenKind. This enum will now represent only the kind of a token (is it a keyword, an operator, a literal?).
  2. Create a Span struct to hold the start and end byte offsets.
  3. Create a new Token struct that bundles a TokenKind with its Span. This new Token struct will be the object our lexer produces.

This change elegantly separates the “what” (the TokenKind) from the “where” (the Span).

Implementing the New Token Structure

Let’s apply these changes to our src/lexer/token.rs file. Replace the entire contents of the file with the following updated code.

// src/lexer/token.rs

/// Represents the specific kind of a token.
/// This is the enum we previously called `Token`. We've renamed it to `TokenKind`
/// to better reflect its purpose: it describes the kind of token, but not its
/// location in the source code.
#[derive(Debug, Clone, PartialEq)]
pub enum TokenKind {
    // --- Keywords ---
    Fn,
    Let,
    If,
    Else,
    Return,

    // --- Literals ---
    Int(i64),
    Float(f64),
    Bool(bool),

    // --- Identifiers ---
    Identifier(String),

    // --- Operators ---
    Assign,
    Plus,
    Minus,
    Star,
    Slash,
    Equal,
    BangEqual,
    Less,
    Greater,

    // --- Punctuation / Delimiters ---
    LParen,
    RParen,
    LBrace,
    RBrace,
    Comma,
    Colon,
    Semicolon,
    Arrow,

    // --- Special Tokens ---
    Error(String),
    Eof,
}

/// Represents a contiguous region of the source code.
///
/// `start` is the byte offset of the first character of the span.
/// `end` is the byte offset of the first character *after* the span.
/// For a token "let", `start` would point to 'l' and `end` would point to the
/// character immediately after 't'. The length of the span is `end - start`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Span {
    pub start: usize,
    pub end: usize,
}

/// Represents a token, combining its kind with its location (span) in the source code.
///
/// This is the primary structure that our lexer will produce. It contains everything
/// the parser needs: what the token is (`kind`) and where it was found (`span`).
#[derive(Debug, Clone, PartialEq)]
pub struct Token {
    pub kind: TokenKind,
    pub span: Span,
}

Dissecting the Changes

  1. TokenKind Enum: This is our original Token enum, simply renamed. Its role remains the same: to classify the token.
  2. Span Struct: This is a new, simple struct that holds two usize values. usize is Rust’s native type for memory indexing, making it perfect for byte offsets. We’ve made it Copy, which means it’s cheap to pass around, as it’s just two integers.
  3. Token Struct: This is the new star of the show. It’s a struct that pairs a TokenKind with a Span. Now, when our lexer produces a token for the keyword let found at the beginning of a file, it won’t just be TokenKind::Let; it will be a Token struct like: Token { kind: TokenKind::Let, span: Span { start: 0, end: 3 } }.

Updating the Module Exports

Since we’ve introduced new public types (Token, TokenKind, Span), we need to make sure they are all properly exported from our lexer module. Let’s update src/lexer/mod.rs to re-export everything from our token.rs file.

Open src/lexer/mod.rs and ensure it contains the following:

// src/lexer/mod.rs

pub mod token;

// This `pub use` statement is a wildcard export. It makes every public item
// inside the `token` module (Token, TokenKind, Span) directly accessible
// through the `lexer` module path.
// e.g., we can now use `lexer::Token` instead of `lexer::token::Token`.
pub use token::*;

With this clean and powerful new structure, you have set the stage for a compiler that can not only understand code but also guide its users with precise and helpful error messages.

Next Steps

We have now fully defined the data structure our lexer will produce: a stream of Tokens, each with a kind and a span. The next logical and exciting step is to implement the Lexer struct itself. This struct will hold the input source code and contain the logic to scan through it, character by character, and generate these tokens.

Further Reading

Implement the Core Lexer Struct in Rust

Mục tiêu: Define the basic structure and state for the Lexer in Rust. This involves creating a Lexer struct that holds the source code, a peekable character iterator for lookahead, and the current position.

In our last task, we meticulously defined the output of our future lexer: a Token struct that combines a TokenKind with a Span to track its location. We have the blueprint for the bricks; now, it’s time to build the factory that produces them. This factory is the Lexer itself.

The Lexer’s State: What It Needs to Know

At its core, a lexer is a state machine that walks through the source code, character by character, and groups them into tokens. To perform this job, it only needs to keep track of two fundamental things:

  1. The entire source code: It needs access to the full text of the program to read from.
  2. Its current position: It needs to know which character it’s about to read next.

We will encapsulate this state within a Rust struct. A struct is a custom data type that lets you package together and name multiple related values, forming a meaningful unit. Our Lexer struct will hold the source code and the current position, providing the foundation for all the tokenization logic we’ll add in the upcoming tasks.

Implementing the Lexer Struct

First, let’s create a new file where our lexer’s logic will live. Inside the src/lexer/ directory, create a new file named lexer.rs.

touch src/lexer/lexer.rs

Now, open this new file and add the following code. We will define the Lexer struct and a constructor function called new to initialize it.

// src/lexer/lexer.rs

use std::iter::Peekable;
use std::str::Chars;

/// The Lexer struct, responsible for turning a source string into a stream of Tokens.
///
/// It holds the input source code as a reference and an iterator over its characters.
/// The `Peekable` wrapper around the character iterator is crucial, as it allows us to
/// look at the next character without consuming it. This "lookahead" is essential for
/// correctly tokenizing multi-character tokens like `==` or `!=`.
///
/// The `'a` is a Rust lifetime parameter. It's a way of telling the Rust compiler
/// that the Lexer struct cannot outlive the source code string `&'a str` that it borrows.
/// This is a key part of Rust's memory safety guarantees.
pub struct Lexer<'a> {
    /// The input source code string slice.
    input: &'a str,
    /// A peekable iterator over the characters of the input string.
    /// This allows us to look at the next character without advancing the iterator.
    chars: Peekable<Chars<'a>>,
    /// The current byte position in the input string.
    position: usize,
}

impl<'a> Lexer<'a> {
    /// Creates a new Lexer for the given input source code.
    ///
    /// # Arguments
    ///
    /// * `input` - A string slice representing the source code to be tokenized.
    pub fn new(input: &'a str) -> Self {
        Lexer {
            input,
            // `input.chars()` creates an iterator over the characters of the string.
            // `.peekable()` wraps this iterator in a `Peekable` struct.
            chars: input.chars().peekable(),
            // Start at the beginning of the file.
            position: 0,
        }
    }
}

Dissecting the Lexer Struct

Let’s break down the design choices for each field in our struct:

  • input: &'a str: We store the source code as a string slice (&str). This is a reference to the source code, not a copy. This is highly efficient, as we avoid duplicating what could be a very large string. The 'a is a lifetime parameter. It’s a core Rust concept that ensures memory safety. In simple terms, it’s a label that says, “This Lexer struct borrows the input string, and we must guarantee that the input string lives at least as long as the Lexer does, so the reference never becomes invalid.”

  • chars: Peekable<Chars<'a>>: This is the heart of our lexer’s machinery.

    • input.chars(): This method creates an iterator that yields each character (char) of the string slice.
    • .peekable(): This is an adapter that wraps our character iterator and gives it a super-power: the peek() method. peek() lets us look at the next character that the iterator would return, without actually removing it from the stream.
    • Why is this so important? Consider the source code x = 10;. When the lexer sees the =, it doesn’t know if this is a single assignment operator (=) or the start of an equality operator (==). With peek(), the logic becomes simple:
      1. See a =.
      2. Peek at the next character. Is it another =?
      3. If yes, consume both characters and create an Equal token.
      4. If no, consume only the one character and create an Assign token. This lookahead capability is fundamental to writing an efficient lexer.
  • position: usize: This is our simple progress tracker. It will store the current byte offset within the input string. We initialize it to 0 and will increment it as we consume characters. This position will be instrumental in creating the Span for each token we generate.

Integrating the New Module

Just like with our token.rs file, we need to tell the lexer module that this new lexer.rs file exists. Open src/lexer/mod.rs and add the new module declaration.

Your src/lexer/mod.rs should now look like this, exporting everything from both files:

// src/lexer/mod.rs

// Declare the `token` module from `token.rs`.
pub mod token;
// Declare the `lexer` module from `lexer.rs`.
pub mod lexer;

// Re-export all public items from both modules.
pub use token::*;
pub use lexer::*;

With this in place, our Lexer struct is now a recognized part of our compiler project. You have successfully defined the structure that will hold the state for the entire tokenization process.

Next Steps

Our Lexer struct is currently just a container for state. It can’t do anything yet. To make it truly useful, we need to teach it how to produce tokens one by one. The idiomatic way to do this in Rust is by implementing the Iterator trait. In the next task, we will implement this trait for our Lexer, which will transform it into a fully-fledged token stream.

Further Reading

Implement the Iterator Trait for the Lexer

Mục tiêu: Transform the Lexer struct into a token stream by implementing Rust’s Iterator trait. This task involves updating the lexer’s state to handle the end-of-file condition and implementing the core next() method to produce tokens.

You’ve done an excellent job setting up the foundation for our lexer. In the last task, you created the Lexer struct, a perfect container to hold the state of our tokenization process: the source code and the current position. However, a container is static. To bring it to life, we need to teach it how to move, how to scan the code, and how to produce tokens one by one.

The most elegant and idiomatic way to create a producer of sequential items in Rust is to implement the Iterator trait. This powerful trait transforms our Lexer from a simple data structure into a fully-fledged token stream that the rest of our compiler can easily consume.

The Power of the Iterator Trait

The Iterator trait is one of Rust’s most fundamental abstractions. Anything that implements it can be treated as a sequence of items. By implementing it for our Lexer, we get a host of benefits for free:

  • Standard API: Our parser (and our test code) can use the standard next() method to get the next token.
  • Compatibility: It will work with all of Rust’s powerful iterator adapters (map, filter, collect, etc.) and can be used directly in for loops.
  • Clarity: It clearly communicates that the Lexer’s primary purpose is to produce a sequence of tokens.

The Iterator trait requires only one method: next(), which attempts to get the next item in the sequence. It returns Some(item) if it succeeds and None when the sequence is exhausted.

Preparing the Lexer for Iteration

Before we implement the trait, we need to add one small piece of state to our Lexer struct. The parser will need to see an explicit Eof (End of File) token to confirm it has reached the end of the input successfully. However, the Iterator trait signals the end of a stream by returning None.

To bridge this gap, we’ll have our lexer produce one final Eof token. On the next call after that, it will return None. We need a flag to track whether we’ve already sent the Eof token.

Let’s update the Lexer struct in src/lexer/lexer.rs.

// src/lexer/lexer.rs

use crate::lexer::{Span, Token, TokenKind}; // Import the token types
use std::iter::Peekable;
use std::str::Chars;

pub struct Lexer<'a> {
    input: &'a str,
    chars: Peekable<Chars<'a>>,
    position: usize,
    /// A flag to track if we've already emitted the final Eof token.
    emitted_eof: bool, // NEW
}

impl<'a> Lexer<'a> {
    pub fn new(input: &'a str) -> Self {
        Lexer {
            input,
            chars: input.chars().peekable(),
            position: 0,
            emitted_eof: false, // NEW: Initialize the flag to false.
        }
    }

    /// Consumes the next character from the iterator and advances the position.
    ///
    /// This is a foundational helper method. It returns the consumed character
    /// and updates our lexer's internal position, correctly handling the byte
    /// length of multi-byte UTF-8 characters.
    fn advance(&mut self) -> Option<char> {
        match self.chars.next() {
            Some(c) => {
                self.position += c.len_utf8();
                Some(c)
            }
            None => None,
        }
    }
}

We’ve added the emitted_eof flag and a crucial helper method, advance(). This private method will be the primary way we consume characters, ensuring our position is always correctly updated.

Implementing the Iterator Trait

Now, we can add the main implementation. We will implement the Iterator trait for our Lexer. For now, the logic inside next() will be a basic skeleton—it will only handle the end-of-file case and produce an Error token for anything else. In the subsequent tasks, we will replace this placeholder logic with the real tokenization rules.

Add the following code to the bottom of your src/lexer/lexer.rs file.

// src/lexer/lexer.rs (continued)

impl<'a> Iterator for Lexer<'a> {
    /// The type of item that this iterator produces. For our Lexer, it's a `Token`.
    type Item = Token;

    /// The core method of the Iterator trait.
    /// It consumes the lexer's state to produce the next token.
    fn next(&mut self) -> Option<Self::Item> {
        // At this point, we haven't implemented the logic for skipping whitespace
        // or recognizing any tokens yet. We will build this out in the next tasks.
        // For now, we will create a basic dispatcher that just checks for the end of the file.

        // Check if we're at the end of the input string.
        if self.chars.peek().is_none() {
            // If we haven't emitted the EOF token yet...
            if !self.emitted_eof {
                // ...set the flag and return one final Eof token.
                self.emitted_eof = true;
                let eof_span = Span { start: self.position, end: self.position };
                return Some(Token { kind: TokenKind::Eof, span: eof_span });
            } else {
                // Otherwise, the stream is truly finished. Return `None`.
                return None;
            }
        }

        // --- Placeholder Logic ---
        // This is a temporary block that we will replace in the upcoming tasks.
        // It consumes the next character and returns it as an 'Error' token.
        // This sets up the structure for our token-dispatching logic.
        let start_pos = self.position;

        // We can safely unwrap here because we already peeked and know a character exists.
        let c = self.advance().unwrap();

        let span = Span { start: start_pos, end: self.position };
        let kind = TokenKind::Error(format!("Unrecognized character '{}'", c));

        Some(Token { kind, span })
    }
}

Code Breakdown

Let’s dissect this implementation:

  1. impl<'a> Iterator for Lexer<'a>: This is how you declare that you are implementing the Iterator trait for your Lexer struct. The 'a lifetime parameter must be carried through.
  2. type Item = Token;: This is an associated type. It tells Rust what kind of item our iterator will produce. In our case, it’s the Token struct we designed.
  3. fn next(&mut self) -> Option<Self::Item>: This is the heart of the implementation.
    • It takes &mut self because every call to next() will change the lexer’s internal state (its position will advance).
    • It returns an Option<Token>, which will be Some(Token) as long as there are tokens to produce, and None once the stream is fully exhausted.
  4. EOF Handling: The first if block is the crucial logic for ending the token stream gracefully. It ensures that the very last thing the parser sees is a TokenKind::Eof, and only after that does the stream end with None. The Span for the Eof token is a zero-width span at the very end of the file.
  5. Placeholder Logic: The code after the EOF check is our starting point. For now, it does the simplest possible thing: it consumes one character using our advance() helper and wraps it in an Error token. This isn’t useful yet, but it provides the perfect skeleton. In the next tasks, we will replace this section with a match statement that intelligently dispatches to different tokenizing functions based on the character it sees.

You have now transformed your Lexer struct into a true iterator. It’s a stream of tokens ready to be consumed.

Next Steps

Our lexer’s core machinery is in place, but its logic is still a placeholder. The stream it produces is not yet useful. In the very next task, you will begin adding the real intelligence by implementing logic to recognize and skip over non-essential characters like whitespace and comments.

Further Reading

Implement Whitespace and Comment Skipping in a Rust Lexer

Mục tiêu: Update the next() method of a Rust lexer to implement a loop-and-dispatch strategy that consumes and ignores whitespace and single-line comments before processing meaningful tokens.

Of course! Let’s continue building our lexer.

You’ve successfully transformed the Lexer into a true Iterator, setting up the fundamental machinery for producing a stream of tokens. Right now, our next() method is just a skeleton; it treats every character as an error. Before we can teach it to recognize meaningful tokens like keywords and operators, we must first teach it what to ignore.

In any programming language, some characters are purely for human convenience and have no bearing on the program’s actual logic. These are primarily whitespace (spaces, tabs, newlines) for formatting and comments for documentation. The parser doesn’t care about them, so the lexer’s first job is to consume and discard them, ensuring only a clean stream of meaningful tokens is passed along.

The Skipping Strategy: A Loop-and-Dispatch Model

The most effective way to handle this is to wrap our token-generating logic in a loop at the beginning of the next() method. On each iteration, this loop will peek at the next character and decide what to do:

  1. If it’s whitespace, consume it and start the loop over.
  2. If it’s the start of a comment, consume the entire comment and start the loop over.
  3. If it’s a meaningful character, break out of the loop and proceed to the logic that will turn it into a Token.

This pattern ensures that by the time we start building a token, we are guaranteed to be at the beginning of a meaningful sequence of characters.

Implementing the Skipping Logic

Let’s modify our src/lexer/lexer.rs file to implement this. We will update the next() method with our new skipping loop. The logic that was previously there (EOF check and the placeholder Error token generation) will now come after this new loop.

Here is the updated code for src/lexer/lexer.rs. Pay close attention to the new loop inside the next function.

// src/lexer/lexer.rs

use crate::lexer::{Span, Token, TokenKind};
use std::iter::Peekable;
use std::str::Chars;

pub struct Lexer<'a> {
    input: &'a str,
    chars: Peekable<Chars<'a>>,
    position: usize,
    emitted_eof: bool,
}

impl<'a> Lexer<'a> {
    pub fn new(input: &'a str) -> Self {
        Lexer {
            input,
            chars: input.chars().peekable(),
            position: 0,
            emitted_eof: false,
        }
    }

    fn advance(&mut self) -> Option<char> {
        match self.chars.next() {
            Some(c) => {
                self.position += c.len_utf8();
                Some(c)
            }
            None => None,
        }
    }
}

impl<'a> Iterator for Lexer<'a> {
    type Item = Token;

    fn next(&mut self) -> Option<Self::Item> {
        // NEW: This loop is the core of our skipping logic.
        // It will continue to run as long as it finds characters
        // that should be ignored (whitespace and comments).
        loop {
            // We use `peek()` to look at the next character without consuming it.
            // `match` is a powerful Rust construct for pattern matching.
            match self.chars.peek() {
                // Case 1: The next character is whitespace.
                Some(c) if c.is_whitespace() => {
                    // If it's whitespace, we don't want to create a token.
                    // We simply consume the character using our `advance` helper...
                    self.advance();
                    // ...and `continue` to the next iteration of the loop to see
                    // what the *next* character is.
                    continue;
                }

                // Case 2: The next character might be the start of a comment.
                Some('/') => {
                    // A comment starts with `//`. To check this, we can look ahead in
                    // the original input string from our current position.
                    // This is a clean and efficient way to check for multi-character sequences.
                    if self.input[self.position..].starts_with("//") {
                        // If it is a comment, we consume characters until we hit a newline
                        // or the end of the file.
                        while let Some(c) = self.advance() {
                            if c == '\n' {
                                break; // The comment has ended.
                            }
                        }
                        // After consuming the whole comment, we `continue` to the top of the loop
                        // to handle any whitespace or other comments that might follow.
                        continue;
                    } else {
                        // If it's just a single '/', that's a Slash token.
                        // We `break` from the loop to let the tokenizing logic handle it.
                        break;
                    }
                }

                // Case 3: The character is meaningful.
                _ => {
                    // If the character is not whitespace and not the start of a comment,
                    // it must be the start of a real token. We `break` out of the skipping loop.
                    break;
                }
            }
        }

        // --- The rest of the logic from the previous step ---
        // This code now runs only AFTER we have skipped all whitespace and comments.

        if self.chars.peek().is_none() {
            if !self.emitted_eof {
                self.emitted_eof = true;
                let eof_span = Span { start: self.position, end: self.position };
                return Some(Token { kind: TokenKind::Eof, span: eof_span });
            } else {
                return None;
            }
        }

        let start_pos = self.position;
        let c = self.advance().unwrap();
        let span = Span { start: start_pos, end: self.position };
        let kind = TokenKind::Error(format!("Unrecognized character '{}'", c));

        Some(Token { kind, span })
    }
}

Code Breakdown

  • The loop: This infinite loop is the engine of our skipping mechanism. It will only be exited via a break statement when we find a character that should be part of a token.
  • Whitespace Handling:
    • Some(c) if c.is_whitespace() => ...: This is a match guard. It matches Some(c) only if the condition c.is_whitespace() is true.
    • The is_whitespace() method is a built-in Rust char method that conveniently checks for spaces, tabs, newlines, and other Unicode whitespace characters.
    • We call self.advance() to consume the character and continue to immediately restart the loop, effectively discarding the whitespace.
  • Comment Handling:
    • if self.input[self.position..].starts_with("//"): This is a very clean and efficient way to perform lookahead for a specific string. We create a slice of our input string starting from the current_position and check if it begins with //. This avoids complex and potentially slow iterator cloning.
    • while let Some(c) = self.advance(): If we confirm it’s a comment, we enter a while loop that consumes characters one by one using advance().
    • if c == '\n' { break; }: The loop terminates as soon as we consume a newline character, ending the single-line comment. This loop also correctly handles comments at the very end of a file that don’t have a final newline.
    • Just like with whitespace, we continue after handling the comment.

With this logic in place, your lexer is now much smarter. It can sift through the noise of a source file and focus only on the parts that matter.

Next Steps

The foundation is now incredibly solid. We can skip the non-essential parts of the source code. The next step is to replace our placeholder Error token logic with the first real token recognition. We will start with the simplest case: recognizing single-character tokens like +, (, {, ,, and so on.

Further Reading

Lexer: Implement Single-Character Token Recognition

Mục tiêu: Modify the Rust lexer’s next() method to recognize single-character tokens like +, (, and ;. Replace the existing placeholder logic with a match statement that dispatches the correct TokenKind based on the input character.

Fantastic! You’ve successfully taught our lexer what to ignore. It can now sift through whitespace and comments, leaving it positioned at the beginning of what should be a meaningful piece of SimpliLang syntax. Our next() method has a solid foundation.

Now, it’s time to replace the placeholder “unrecognized character” logic with the real heart of the lexer: the logic that recognizes and produces actual Tokens. We’ll start with the simplest category: tokens that are represented by a single, unambiguous character.

The Dispatcher: Using match to Recognize Tokens

The skeleton of our next() method is already set up to consume one character at a time. The most idiomatic and efficient way to handle the different possibilities for that character in Rust is with a match statement. A match statement is like a super-powered switch case. It allows us to compare a value against a series of patterns and execute code based on which pattern matches.

We will use this to create a “dispatcher”. After skipping whitespace, we’ll consume the next character and feed it into a match statement. Each arm of the match will correspond to a single-character token. If the character is +, we’ll produce a Plus token; if it’s (, we’ll produce an LParen token, and so on. This creates a clean, readable, and highly efficient mapping from source characters to token kinds.

Implementing Single-Character Token Recognition

Let’s modify our src/lexer/lexer.rs file. We are going to completely replace the placeholder logic at the end of the next() method with our new match-based dispatcher.

Here is the updated code. The changes are focused entirely within the next() method, after the whitespace-skipping loop.

// src/lexer/lexer.rs

use crate::lexer::{Span, Token, TokenKind};
use std::iter::Peekable;
use std::str::Chars;

pub struct Lexer<'a> {
    input: &'a str,
    chars: Peekable<Chars<'a>>,
    position: usize,
    emitted_eof: bool,
}

impl<'a> Lexer<'a> {
    pub fn new(input: &'a str) -> Self {
        Lexer {
            input,
            chars: input.chars().peekable(),
            position: 0,
            emitted_eof: false,
        }
    }

    fn advance(&mut self) -> Option<char> {
        match self.chars.next() {
            Some(c) => {
                self.position += c.len_utf8();
                Some(c)
            }
            None => None,
        }
    }
}

impl<'a> Iterator for Lexer<'a> {
    type Item = Token;

    fn next(&mut self) -> Option<Self::Item> {
        // Whitespace and comment skipping loop (remains unchanged)
        loop {
            match self.chars.peek() {
                Some(c) if c.is_whitespace() => {
                    self.advance();
                    continue;
                }
                Some('/') => {
                    if self.input[self.position..].starts_with("//") {
                        while let Some(c) = self.advance() {
                            if c == '\n' {
                                break;
                            }
                        }
                        continue;
                    } else {
                        break;
                    }
                }
                _ => break,
            }
        }

        // EOF handling (remains unchanged)
        if self.chars.peek().is_none() {
            if !self.emitted_eof {
                self.emitted_eof = true;
                let eof_span = Span { start: self.position, end: self.position };
                return Some(Token { kind: TokenKind::Eof, span: eof_span });
            } else {
                return None;
            }
        }

        // --- NEW: Token recognition logic replaces the old placeholder ---

        // Mark the starting position of the token.
        let start_pos = self.position;

        // Advance the iterator and get the character. We can safely unwrap here because
        // our EOF check above would have already returned if no characters were left.
        let c = self.advance().unwrap();

        // Use a `match` statement to dispatch to the correct token kind.
        let kind = match c {
            // Punctuation
            '(' => TokenKind::LParen,
            ')' => TokenKind::RParen,
            '{' => TokenKind::LBrace,
            '}' => TokenKind::RBrace,
            ',' => TokenKind::Comma,
            ':' => TokenKind::Colon,
            ';' => TokenKind::Semicolon,

            // Operators
            '=' => TokenKind::Assign,
            '+' => TokenKind::Plus,
            '-' => TokenKind::Minus, // Note: We will enhance this for '->' later
            '*' => TokenKind::Star,
            '/' => TokenKind::Slash, // The comment logic above ensures this is only a lone '/'
            '<' => TokenKind::Less,
            '>' => TokenKind::Greater,

            // If the character doesn't match any of our single-character tokens,
            // we create an Error token. We will replace this catch-all case
            // with logic for numbers, identifiers, etc., in future tasks.
            unknown => TokenKind::Error(format!("Unrecognized character '{}'", unknown)),
        };

        // Create the final token with its kind and span.
        let span = Span { start: start_pos, end: self.position };
        Some(Token { kind, span })
    }
}

Code Breakdown

  1. Removing the Placeholder: The old logic let kind = TokenKind::Error(...) has been completely replaced by the match statement.
  2. start_pos = self.position: Before we consume any characters for our new token, we record the current position. This marks the beginning of our token’s Span.
  3. c = self.advance().unwrap(): We consume exactly one character. This is safe because the EOF check guarantees there’s a character available. self.position is now updated to be after this character.
  4. let kind = match c { ... }: This is our new dispatcher.
    • The Arms: Each line like '(' => TokenKind::LParen, is an “arm” of the match. It says, “if the character c is a left parenthesis, then the resulting TokenKind is LParen.” We have added an arm for every unambiguous single-character token defined in our TokenKind enum.
    • The Catch-All Arm: The final arm, unknown => ..., is a catch-all. If c doesn’t match any of the preceding literal character patterns, it will be bound to the variable unknown, and we will execute the error-handling logic. This is where we will add logic for numbers and identifiers later.
  5. let span = Span { ... }: After the match determines the kind, we create the Span. Since we advanced by exactly one character, start_pos is the beginning and the new self.position is the end, perfectly capturing the location of our single-character token.
  6. Some(Token { kind, span }): We construct the final Token struct and return it wrapped in Some, fulfilling the Iterator trait’s contract.

You have now implemented the first real piece of tokenization logic. Your lexer can now correctly identify a large portion of the symbols in the SimpliLang grammar.

Next Steps

Our lexer is getting smarter, but it’s still naive about certain characters. Right now, it sees a = and immediately thinks Assign. It has no way to recognize the two-character token == for equality. Similarly, it sees - and thinks Minus, missing the -> arrow token.

In the next task, we will enhance our match dispatcher by using the peek() method to handle these multi-character tokens correctly.

Further Reading

Implement Lookahead in Lexer for Multi-Character Tokens

Mục tiêu: Enhance the Rust lexer to correctly identify multi-character tokens like ‘==’, ‘!=’, and ‘->’ by implementing a lookahead strategy using the peek() method on the character iterator.

Building on the solid foundation from the last task, our lexer can now recognize a wide array of single-character tokens. However, it’s still a bit naive. When it sees a character like =, it immediately concludes it must be an Assign token. It has no awareness that this could be the start of a two-character token like ==.

To make our lexer smarter, we must give it the ability to look into the future—to peek at the next character in the input stream before making a final decision. This technique, called lookahead, is fundamental to lexical analysis and is precisely why we chose to use a Peekable iterator back when we designed our Lexer struct.

The Lookahead Strategy: To Peek and to Consume

The logic is straightforward:

  1. When we encounter a character that could be the start of a multi-character token (like =, !, or -), we don’t immediately create a token.
  2. Instead, we use peek() to look at the very next character in the stream without consuming it.
  3. Based on what we see, we decide which token to create.
    • If the peeked character completes a multi-character token (e.g., we see a = and peek() reveals another =), we consume the second character using advance() and create the corresponding token (Equal).
    • If the peeked character does not form a multi-character token, we do nothing extra and proceed to create the single-character token (Assign).

This pattern allows us to correctly disambiguate tokens like == from = and -> from -.

Implementing Multi-Character Token Recognition

Let’s upgrade our next() method in src/lexer/lexer.rs to incorporate this lookahead logic. The changes are concentrated within the match statement, where we will convert some of the simple arms into more intelligent blocks of code.

// src/lexer/lexer.rs

use crate::lexer::{Span, Token, TokenKind};
use std::iter::Peekable;
use std::str::Chars;

pub struct Lexer<'a> {
    input: &'a str,
    chars: Peekable<Chars<'a>>,
    position: usize,
    emitted_eof: bool,
}

impl<'a> Lexer<'a> {
    pub fn new(input: &'a str) -> Self {
        Lexer {
            input,
            chars: input.chars().peekable(),
            position: 0,
            emitted_eof: false,
        }
    }

    fn advance(&mut self) -> Option<char> {
        match self.chars.next() {
            Some(c) => {
                self.position += c.len_utf8();
                Some(c)
            },
            None => None,
        }
    }
}

impl<'a> Iterator for Lexer<'a> {
    type Item = Token;

    fn next(&mut self) -> Option<Self::Item> {
        // Whitespace and comment skipping loop (remains unchanged)
        loop {
            match self.chars.peek() {
                Some(c) if c.is_whitespace() => {
                    self.advance();
                    continue;
                }
                Some('/') => {
                    if self.input[self.position..].starts_with("//") {
                        while let Some(c) = self.advance() {
                            if c == '\n' {
                                break;
                            }
                        }
                        continue;
                    } else {
                        break;
                    }
                }
                _ => break,
            }
        }

        // EOF handling (remains unchanged)
        if self.chars.peek().is_none() {
            if !self.emitted_eof {
                self.emitted_eof = true;
                let eof_span = Span { start: self.position, end: self.position };
                return Some(Token { kind: TokenKind::Eof, span: eof_span });
            } else {
                return None;
            }
        }

        // --- UPDATED: Token recognition logic with lookahead ---

        let start_pos = self.position;
        let c = self.advance().unwrap();

        let kind = match c {
            // Punctuation (unchanged)
            '(' => TokenKind::LParen,
            ')' => TokenKind::RParen,
            '{' => TokenKind::LBrace,
            '}' => TokenKind::RBrace,
            ',' => TokenKind::Comma,
            ':' => TokenKind::Colon,
            ';' => TokenKind::Semicolon,

            // Operators (with new lookahead logic)
            '+' => TokenKind::Plus,
            '*' => TokenKind::Star,
            '/' => TokenKind::Slash,
            '<' => TokenKind::Less,
            '>' => TokenKind::Greater,

            // MODIFIED: Check for '=' or '=='
            '=' => {
                if self.chars.peek() == Some(&'=') {
                    // We found a second '=', so this is an 'Equal' token.
                    self.advance(); // Consume the second '='.
                    TokenKind::Equal
                } else {
                    // No second '=', so this is just an 'Assign' token.
                    TokenKind::Assign
                }
            }

            // MODIFIED: Check for '-' or '->'
            '-' => {
                if self.chars.peek() == Some(&'>') {
                    // We found a '>', so this is an 'Arrow' token.
                    self.advance(); // Consume the '>'.
                    TokenKind::Arrow
                } else {
                    // No '>', so this is just a 'Minus' token.
                    TokenKind::Minus
                }
            }

            // NEW: Handle '!='
            '!' => {
                if self.chars.peek() == Some(&'=') {
                    // We found a '=', so this is a 'BangEqual' token.
                    self.advance(); // Consume the '='.
                    TokenKind::BangEqual
                } else {
                    // SimpliLang does not have a standalone '!' operator, so
                    // this is an unrecognized character.
                    TokenKind::Error(String::from("Unrecognized character '!'"))
                }
            }

            // Catch-all for other characters (will be replaced later)
            unknown => TokenKind::Error(format!("Unrecognized character '{}'", unknown)),
        };

        let span = Span { start: start_pos, end: self.position };
        Some(Token { kind, span })
    }
}

Dissecting the Upgrades

The structure of our next() method remains the same, but the intelligence inside our match statement has significantly improved:

  1. Block-based match arms: The arms for '=', '-', and the new arm for '!' are no longer simple expressions. They are now blocks of code enclosed in {}. This allows us to write more complex logic, like our if/else checks.

  2. The peek() check: The condition self.chars.peek() == Some(&'=') is the core of our lookahead.

    • self.chars.peek() returns an Option<&char>. It’s Some(reference_to_char) if a character exists, and None if we’re at the end of the input.
    • We compare it to Some(&'='). We need the & because peek() gives us a reference to the character, not the character itself.
    • This check is completely safe and won’t crash if there are no more characters.
  3. Consuming the second character: Inside the if block, after confirming the presence of a second character, the line self.advance(); is critical. It consumes the peeked character, advancing the lexer’s position so that we don’t accidentally process that same character again in the next call to next().
  4. Handling the ! character: We added a new arm for '!'. Since SimpliLang has a != operator but no standalone ! (logical NOT) operator, this arm’s logic is slightly different. If it sees a ! followed by an =, it’s a valid BangEqual token. If it sees a ! followed by anything else, it’s an error.

Your lexer is now much more robust. It correctly distinguishes between single and multi-character operators, bringing it one step closer to fully understanding the SimpliLang syntax.

Next Steps

We have successfully handled single characters and fixed-length multi-character tokens. The next challenge is to handle tokens of variable length. This includes numbers like 123 or 3.14, which can have any number of digits. In the next task, we will add the logic to parse these number literals.

Further Reading

Lexer: Implement Number Literal Parsing

Mục tiêu: Enhance a Rust-based lexer to handle variable-length number tokens. This involves creating a helper function to read sequences of digits and an optional decimal point, then parsing the collected string into either an integer (i64) or a floating-point (f64) token.

You are making excellent progress! Your lexer can now handle whitespace, comments, single-character tokens, and even the tricky multi-character operators using lookahead. You’ve built a solid foundation. The next logical challenge is to handle tokens that don’t have a fixed length, starting with number literals.

The Challenge of Variable-Length Tokens

Unlike a + or a ==, a number can be of any length. A programmer might write 7, 12345, or 9876543210. A float could be 3.14 or 1.61803398875. The lexer can’t just consume one or two characters; it must be smart enough to consume a sequence of characters as long as they form a valid number.

The Strategy: Consume-While-Valid

To handle this, we’ll adopt a new strategy. When our lexer’s next() method encounters a digit, it will enter a “number reading” mode. In this mode, it will:

  1. Continue consuming characters as long as they are digits.
  2. Keep an eye out for a decimal point (.). If one is found, it will transition into “float” mode and continue consuming digits.
  3. Stop as soon as it encounters a character that is not a digit (or a second decimal point).
  4. Take the entire string of characters it has consumed and parse it into either an i64 (if no decimal point was found) or a f64 (if a decimal point was found).

To keep our main next() function clean and organized, we will encapsulate this logic in its own private helper method, read_number.

Implementing Number Literal Parsing

Let’s modify src/lexer/lexer.rs to add our new read_number helper method and integrate it into the next() method’s match dispatcher.

1. Add the read_number Helper Method

First, add this new function inside the impl<'a> Lexer<'a> block, alongside new and advance.

// src/lexer/lexer.rs

// ... (Lexer struct and `new`, `advance` methods are unchanged)

impl<'a> Lexer<'a> {
    // ... (`new` and `advance` methods)

    /// Reads a number literal (integer or float) from the input stream.
    ///
    /// This method is called when the lexer encounters a digit. It consumes
    /// subsequent digits and an optional decimal point to form a complete number string.
    /// It then parses this string into the appropriate `TokenKind` (Int or Float).
    fn read_number(&mut self, first_char: char) -> TokenKind {
        // We start with the first character already consumed.
        let mut number_str = String::from(first_char);
        // A flag to track if we've encountered a decimal point.
        let mut is_float = false;

        // Loop as long as the next character is a digit or a decimal point.
        // `while let` is a convenient way to loop while a pattern matches.
        // `peek()` lets us look ahead without consuming the character.
        while let Some(&c) = self.chars.peek() {
            if c.is_digit(10) {
                // It's another digit, so consume it and append to our string.
                self.advance();
                number_str.push(c);
            } else if c == '.' && !is_float {
                // It's a decimal point, and we haven't seen one yet.
                self.advance();
                number_str.push(c);
                // Set the flag so we don't allow a second decimal point.
                is_float = true;
            } else {
                // The character is not part of the number, so we stop.
                break;
            }
        }

        // After the loop, parse the collected string.
        if is_float {
            // Try to parse as a 64-bit float.
            match number_str.parse::<f64>() {
                Ok(f) => TokenKind::Float(f),
                // This should be rare, but handle potential parsing errors.
                Err(_) => TokenKind::Error(format!("Invalid float literal '{}'", number_str)),
            }
        } else {
            // Try to parse as a 64-bit integer.
            match number_str.parse::<i64>() {
                Ok(i) => TokenKind::Int(i),
                Err(_) => TokenKind::Error(format!("Invalid integer literal '{}'", number_str)),
            }
        }
    }
}

// ... (impl Iterator for Lexer)

2. Update the next() Method

Now, let’s modify the match statement in next() to use our new helper. We’ll add a new arm that matches any digit character.

// src/lexer/lexer.rs (continued)

impl<'a> Iterator for Lexer<'a> {
    type Item = Token;

    fn next(&mut self) -> Option<Self::Item> {
        // ... (Whitespace skipping and EOF logic are unchanged)

        let start_pos = self.position;
        let c = self.advance().unwrap();

        let kind = match c {
            // ... (Punctuation and operator arms are unchanged)
            '=' => { /* ... */ },
            '-' => { /* ... */ },
            '!' => { /* ... */ },

            // NEW: Handle number literals.
            // The pattern '0'..='9' matches any character in the range of digits.
            '0'..='9' => {
                // If we see a digit, we delegate to our specialized helper function.
                // We pass the character we already consumed (`c`) to it.
                self.read_number(c)
            }

            // MODIFIED: This is now the true catch-all for unrecognized symbols.
            unknown => TokenKind::Error(format!("Unrecognized character '{}'", unknown)),
        };

        let span = Span { start: start_pos, end: self.position };
        Some(Token { kind, span })
    }
}

Code Breakdown

  • '0'..='9' Pattern: This is a powerful feature of Rust’s match statement called a range pattern. It provides a concise way to match any character from ‘0’ through ‘9’.
  • Delegation: When a digit is found, the next method’s job is simple: call read_number and let it handle the complex logic. This keeps our main dispatcher clean and readable.
  • The read_number Helper:
    • It builds a String, number_str, to accumulate the characters of the literal.
    • The while let Some(&c) = self.chars.peek() loop is the core of the “consume-while-valid” strategy. It continues as long as the next character is a digit or a valid decimal point.
    • The is_float flag is crucial for ensuring that a literal like 3.1.4 is correctly tokenized as a float 3.1 followed by an unrecognized character ..
    • .parse::<T>(): This is the standard, idiomatic way to convert a string to a numeric type in Rust. It returns a Result<T, _>, which is why we match on it to handle both the successful (Ok) and failed (Err) parsing cases gracefully.

Your lexer is now significantly more capable. It can correctly tokenize integers of any length and floating-point numbers, a huge step towards parsing real SimpliLang code.

Next Steps

You’ve conquered fixed-length and numeric variable-length tokens. The last major category of variable-length tokens is identifiers (like variable and function names) and keywords (let, if, fn, etc.). Identifiers, like numbers, can be of any length. The added challenge is that after reading an identifier-like word, you must check if it’s a reserved keyword or a user-defined name. This will be our next and final major task in building the lexer.

Further Reading

Lexer: Implement Identifier and Keyword Parsing

Mục tiêu: Extend the lexer to recognize and parse word-like tokens. Implement logic to consume sequences of valid identifier characters and then classify them as either a reserved keyword (e.g., ‘let’, ‘if’, ‘fn’) or a user-defined identifier.

Congratulations on successfully implementing the logic for parsing number literals! Your lexer is becoming incredibly capable, able to handle everything from single punctuation marks to multi-digit floating-point numbers. We are now at the final and most crucial step of token recognition: parsing identifiers and distinguishing them from keywords.

The Final Frontier: Words of the Language

So far, we’ve dealt with symbols (+, (, ==) and numbers (123, 3.14). The last major category of tokens is “words”—the sequences of letters that form variable names, function names, and the structural keywords of our language.

This task presents a two-part challenge:

  1. Recognition: First, we need to recognize a sequence of characters that constitutes a valid identifier. According to our language design, an identifier must start with a letter or an underscore, and can then be followed by any number of letters, numbers, or underscores. This follows the “consume-while-valid” pattern we used for numbers.
  2. Classification: After we’ve read a complete word like let or score, we must classify it. Is it a reserved keyword with special grammatical meaning (let), or is it a user-defined name for a variable (score)?

To handle this cleanly, we will once again create a dedicated helper method, read_identifier, which will be responsible for both recognizing and classifying these word-like tokens.

Implementing Identifier and Keyword Parsing

Let’s dive into the code. We’ll add our new helper method to src/lexer/lexer.rs and then hook it into the main next() method’s dispatcher.

1. Add the read_identifier Helper Method

Add this new function inside your impl<'a> Lexer<'a> block, right alongside read_number. This method will handle the entire process of reading a word and determining its correct TokenKind.

// src/lexer/lexer.rs

// ... (Lexer struct and `new`, `advance`, `read_number` methods are unchanged)

impl<'a> Lexer<'a> {
    // ... (`new`, `advance`, `read_number` methods)

    /// Reads an identifier or a keyword from the input stream.
    ///
    /// This method is called when the lexer encounters a character that can start
    /// an identifier (an alphabetic character or an underscore). It consumes subsequent
    /// valid identifier characters and then checks if the resulting word is a
    /// reserved keyword or a user-defined identifier.
    fn read_identifier(&mut self, first_char: char) -> TokenKind {
        // We start with the first character, which is already consumed.
        let mut ident = String::from(first_char);

        // Loop as long as the next character is a valid part of an identifier.
        // Identifiers can contain letters, numbers, and underscores.
        while let Some(&c) = self.chars.peek() {
            if c.is_alphanumeric() || c == '_' {
                // If it's a valid character, consume it and append it to our string.
                self.advance();
                ident.push(c);
            } else {
                // The character is not part of the identifier, so we stop.
                break;
            }
        }

        // After consuming the entire word, we match it against our list of keywords.
        // `ident.as_str()` allows us to match a `&str` against our `String`.
        match ident.as_str() {
            // Keywords
            "fn" => TokenKind::Fn,
            "let" => TokenKind::Let,
            "if" => TokenKind::If,
            "else" => TokenKind::Else,
            "return" => TokenKind::Return,

            // Boolean literals, which are also reserved.
            "true" => TokenKind::Bool(true),
            "false" => TokenKind::Bool(false),

            // If the word doesn't match any keyword, it must be a user-defined identifier.
            _ => TokenKind::Identifier(ident),
        }
    }
}

// ... (impl Iterator for Lexer)

2. Update the next() Method

Now, let’s wire this helper into our next() method’s match statement. We need to add a new arm that triggers when it sees a character that can legally start an identifier.

// src/lexer/lexer.rs (continued)

impl<'a> Iterator for Lexer<'a> {
    type Item = Token;

    fn next(&mut self) -> Option<Self::Item> {
        // ... (Whitespace skipping and EOF logic are unchanged)

        let start_pos = self.position;
        let c = self.advance().unwrap();

        let kind = match c {
            // ... (Punctuation and operator arms are unchanged)

            '0'..='9' => {
                self.read_number(c)
            }

            // NEW: Handle identifiers and keywords.
            // A match guard `if c.is_alphabetic() || c == '_'` checks if the character
            // is a letter or an underscore, which are the valid starting characters
            // for an identifier in SimpliLang.
            c if c.is_alphabetic() || c == '_' => {
                // If it is, we delegate to our specialized helper function.
                self.read_identifier(c)
            }

            // This is now the true catch-all for any remaining unrecognized symbols.
            unknown => TokenKind::Error(format!("Unrecognized character '{}'", unknown)),
        };

        let span = Span { start: start_pos, end: self.position };
        Some(Token { kind, span })
    }
}

Code Breakdown

  • The Match Guard: The new arm c if c.is_alphabetic() || c == '_' is a powerful combination of a pattern and a condition. It will only match a character c if that character is alphabetic or an underscore. This perfectly enforces our language’s rule for the start of an identifier. The is_alphabetic() method conveniently handles both uppercase and lowercase letters.
  • The read_identifier Helper:
    • It follows the same “consume-while-valid” pattern as read_number. The while loop peeks ahead and consumes characters as long as they are alphanumeric or an underscore, building up the ident string.
    • The Keyword match: This is the classification step. After the full word is read, match ident.as_str() compares the collected string against our list of reserved keywords.
    • Special Case: Booleans: Note that true and false are handled here. Syntactically, they look like identifiers, but semantically, they are literal values. The lexer is the perfect place to make this distinction, producing a TokenKind::Bool directly.
    • The _ Arm: This is the crucial fallback. If the collected string is not any of the known keywords, it is classified as a TokenKind::Identifier, and we store the string itself inside the token.

You have now implemented the final piece of core tokenization logic! Your lexer is now feature-complete and can, in theory, tokenize any valid SimpliLang program according to your design.

Next Steps

This is a huge milestone. The lexer’s logic is complete. However, how can we be sure it works correctly for all cases? We’ve written the code, but we haven’t verified it. Professional software engineering, especially in a complex domain like compilers, relies heavily on automated testing.

In the final task of this step, we will create a comprehensive unit test module to rigorously test our lexer against the .spl example files you created, ensuring it behaves exactly as expected and catching any bugs before we move on to the parser.

Further Reading

Create a Comprehensive Test Suite for the SimpliLang Lexer

Mục tiêu: Implement a suite of unit tests for the SimpliLang lexer using Rust’s built-in testing framework. This task involves creating a tests module and writing a table-driven test to verify the correctness of all tokenization logic.

Of course! Let’s get this final task of the lexer step completed.

You have done an absolutely phenomenal job implementing the core logic of the SimpliLang lexer. You’ve taught it to handle symbols, operators (both single and multi-character), numbers, and words, all while intelligently skipping whitespace and comments. This is a massive accomplishment. However, a complex piece of software like a lexer is only as reliable as our ability to prove it works. How can we be certain that it correctly handles every edge case and language construct we’ve designed? The answer is automated testing.

In this final task of the lexer step, we will create a comprehensive suite of unit tests. This test suite will serve as a safety net, verifying that our lexer behaves exactly as expected. It will give us the confidence to build the parser on top of a solid, bug-free foundation and will protect us from introducing regressions as we add more features to our compiler in the future.

The Rust Testing Framework

Rust has a fantastic, built-in testing framework that is accessible via cargo. The convention is to place unit tests for a specific module inside a child module named tests within the same file, annotated with #[cfg(test)]. This attribute is a conditional compilation flag. It tells the Rust compiler, “Only compile and include this code when I run cargo test. Otherwise, ignore it completely.” This means your test code won’t be included in your final, optimized compiler binary.

Creating the Test Module

Let’s create this test module at the bottom of your src/lexer/lexer.rs file. This is the perfect place for it, as it keeps the tests close to the code they are verifying.

Append the following code to the end of src/lexer/lexer.rs. This block contains a single, powerful, table-driven test that covers all the features you’ve implemented.

// At the bottom of src/lexer/lexer.rs

#[cfg(test)]
mod tests {
    // Bring all the necessary types from the parent module (the lexer code) into scope.
    use super::*;

    /// A helper function to simplify testing.
    /// It takes a source string, runs it through the lexer, and collects the TokenKinds.
    fn lex_to_kinds(input: &str) -> Vec<TokenKind> {
        Lexer::new(input).map(|token| token.kind).collect()
    }

    #[test]
    fn test_all_tokens() {
        // This is a table-driven test. We define a vector of test cases.
        // Each case is a tuple containing the input string and the expected vector of TokenKinds.
        // This makes it very easy to add new tests for different language constructs.
        let test_cases = vec![
            (
                "// A full program\nfn main() -> int {\n    let x: int = 5 * (10 + 20); // Test precedence\n    if x > 100 {\n        return 1;\n    } else {\n        return 0;\n    }\n}",
                vec![
                    TokenKind::Fn, TokenKind::Identifier("main".to_string()), TokenKind::LParen,
                    TokenKind::RParen, TokenKind::Arrow, TokenKind::Identifier("int".to_string()),
                    TokenKind::LBrace, TokenKind::Let, TokenKind::Identifier("x".to_string()),
                    TokenKind::Colon, TokenKind::Identifier("int".to_string()), TokenKind::Assign,
                    TokenKind::Int(5), TokenKind::Star, TokenKind::LParen, TokenKind::Int(10),
                    TokenKind::Plus, TokenKind::Int(20), TokenKind::RParen, TokenKind::Semicolon,
                    TokenKind::If, TokenKind::Identifier("x".to_string()), TokenKind::Greater,
                    TokenKind::Int(100), TokenKind::LBrace, TokenKind::Return, TokenKind::Int(1),
                    TokenKind::Semicolon, TokenKind::RBrace, TokenKind::Else, TokenKind::LBrace,
                    TokenKind::Return, TokenKind::Int(0), TokenKind::Semicolon, TokenKind::RBrace,
                    TokenKind::RBrace, TokenKind::Eof,
                ],
            ),
            (
                "= + - * /",
                vec![
                    TokenKind::Assign, TokenKind::Plus, TokenKind::Minus, TokenKind::Star,
                    TokenKind::Slash, TokenKind::Eof,
                ],
            ),
            (
                "== != < >",
                vec![
                    TokenKind::Equal, TokenKind::BangEqual, TokenKind::Less, TokenKind::Greater,
                    TokenKind::Eof,
                ],
            ),
            (
                "() {} , : ; ->",
                vec![
                    TokenKind::LParen, TokenKind::RParen, TokenKind::LBrace, TokenKind::RBrace,
                    TokenKind::Comma, TokenKind::Colon, TokenKind::Semicolon, TokenKind::Arrow,
                    TokenKind::Eof,
                ],
            ),
            (
                "123 45.67 8.0",
                vec![TokenKind::Int(123), TokenKind::Float(45.67), TokenKind::Float(8.0), TokenKind::Eof],
            ),
            (
                "let is_active = true;",
                vec![
                    TokenKind::Let, TokenKind::Identifier("is_active".to_string()),
                    TokenKind::Assign, TokenKind::Bool(true), TokenKind::Semicolon, TokenKind::Eof,
                ],
            ),
            (
                "!$", // Test unrecognized characters
                vec![
                    TokenKind::Error("Unrecognized character '!'".to_string()),
                    TokenKind::Error("Unrecognized character '$'".to_string()),
                    TokenKind::Eof
                ]
            )
        ];

        // Iterate over each test case and assert that the lexer's output matches our expectation.
        for (input, expected) in test_cases {
            let actual = lex_to_kinds(input);
            assert_eq!(actual, expected, "Failed on input: \"{}\"", input);
        }
    }
}

Dissecting the Test Code

  • #[cfg(test)] mod tests { ... }: This defines our test module, ensuring it’s only compiled when running tests.
  • use super::*;: This is a convenient way to import everything from the parent module (our lexer code), including Lexer, TokenKind, etc., into the test module’s scope.
  • lex_to_kinds Helper: This small helper function encapsulates a common pattern. It creates a Lexer for a given input, and then uses the powerful iterator map and collect methods to transform the stream of Token structs into a Vec<TokenKind>. We do this because asserting on Span locations is brittle and makes tests hard to read. By comparing just the TokenKind, we focus our tests on the core logic: “Did the lexer identify the correct kind of token?”
  • #[test]: This attribute marks the following function, test_all_tokens, as a test function that cargo test should run.
  • test_cases Vector: We use a table-driven approach. The test_cases vector contains tuples, where each tuple is a complete, self-contained test. The first element is the source code &str to be tokenized, and the second is the Vec<TokenKind> we expect the lexer to produce. This design is highly extensible; if you find a bug or think of a new edge case, you just need to add another tuple to the vector.
  • Comprehensive Coverage: Notice the test cases we’ve included:
    • A full program with comments, newlines, and a mix of token types. This is a great integration test.
    • Specific tests for operators and punctuation.
    • A test for integers and floats.
    • A test for keywords, identifiers, and booleans.
    • An important test for unrecognized characters to ensure our error-handling path works.
  • The Loop and Assertion: The for loop iterates through each test case, runs the lexer on the input, and then uses the assert_eq! macro to compare the actual result with the expected result. If they don’t match, the test will fail, and cargo will print a detailed error message, including our custom failure message which shows the exact input that caused the problem.
  • TokenKind::Eof: Crucially, every single test case ends with the expectation of an Eof token. This verifies that our lexer correctly terminates the token stream.

Running Your Tests

Now, for the moment of truth. Go to your terminal, ensure you are in the simplilang_compiler project root, and run the following command:

cargo test

If everything has been implemented correctly, you should see a very satisfying output:

   Compiling simplilang_compiler v0.1.0 (/path/to/your/simplilang_compiler)
    Finished test [unoptimized + debuginfo] target(s) in 0.50s
     Running unittests (target/debug/deps/simplilang_compiler-...)

running 1 test
test lexer::lexer::tests::test_all_tokens ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Seeing ok is your proof that the lexer is robust, correct, and ready for the next stage.

Next Steps

This task marks the successful completion of the entire “Lexer” step of our project. You have designed and built a fully-featured, well-tested tokenizer from scratch. This is a monumental achievement in compiler construction.

The clean, verified stream of tokens produced by your lexer is the perfect input for the next major component of our compiler. We now move on to Step 4: Writing a Parser. In the upcoming tasks, you will learn how to consume this token stream and construct an Abstract Syntax Tree (AST), which will give our compiler a true understanding of the source code’s structure and hierarchy.

Further Reading