Add clap Dependency for Command-Line Interface

Mục tiêu: Modify the Cargo.toml file to include the clap crate as a dependency with the derive feature enabled. This is the first step in building a command-line interface (CLI) for the web crawler project.

Excellent work! You have successfully transformed a simple scraper into a fully autonomous, concurrent web crawler. It’s a remarkable piece of engineering that intelligently navigates through paginated content. However, its full potential is still locked behind hardcoded values within the source code. To make it a truly versatile and professional tool, anyone should be able to use it without needing to recompile it for different websites or output files.

This brings us to the next major evolution of our project: building a Command-Line Interface (CLI). A CLI will allow us to pass configuration, like the starting URL and output filename, as arguments directly when we run the program from the terminal. This is the standard for professional development tools.

Introducing clap: The Command Line Argument Parser

In the Rust ecosystem, the gold standard for building powerful and user-friendly CLIs is a crate called clap. clap handles all the complex and tedious parts of parsing command-line arguments for you. It can automatically: * Parse arguments like --url "http://..." or -o "file.csv". * Generate professional-looking help messages when a user runs your program with --help or -h. * Provide clear error messages if the user provides invalid arguments. * Handle different types of arguments, such as simple flags, options that take values, and positional arguments.

Enabling the derive Feature for Maximum Ergonomics

We will use clap’s most modern and ergonomic API, which is enabled by a feature flag called derive. In Rust’s package manager, Cargo, a feature flag is a way to tell a crate that you want to opt-in to a specific piece of its functionality. By enabling derive, we unlock the ability to define our entire CLI structure using a simple Rust struct and special attributes. This is a highly declarative and readable approach that is considered the best practice for using clap.

Your first task is to tell Cargo that our project now depends on clap with this derive feature enabled. You will do this by adding a single line to your Cargo.toml file.

Open your Cargo.toml file and add the following line under the [dependencies] section. It’s good practice to specify a version number to ensure your build is reproducible.

# In Cargo.toml

[dependencies]
tokio = { version = "1", features = ["full"] }
reqwest = "0.11"
scraper = "0.18"
csv = "1.3"
serde = { version = "1.0", features = ["derive"] }
# highlight-start
# Add clap for command-line argument parsing.
# The "derive" feature enables the modern, struct-based API, which is the
# recommended way to use the library. We specify a version for reproducibility.
clap = { version = "4.4", features = ["derive"] }
# highlight-end

After adding this line, it’s a good idea to run cargo build in your terminal. Cargo will read the updated Cargo.toml file, download the clap crate and its own dependencies, and compile them. This verifies that your setup is correct before you start writing any code that uses the new library.

You have now successfully declared clap as a dependency, setting the stage for building a professional and flexible interface for your powerful crawler.

What’s Next?

With the clap crate added to your project, the next step is to put it to use. You will create a new struct in your src/main.rs file. This struct will serve as a blueprint for your CLI, and you will use clap’s derive macros to define the specific arguments you want your program to accept, such as the starting URL and the output filename.

Further Reading

To understand more about Cargo’s dependency management and the clap crate, these resources are invaluable:

Define the CLI Blueprint with clap

Mục tiêu: Begin building a command-line interface by defining an empty CliArgs struct and using the #[derive(Parser)] macro from the clap crate to enable argument parsing.

Excellent! You’ve successfully laid the groundwork by adding the clap crate to your project’s dependencies. You’ve essentially brought a powerful toolkit for building professional command-line interfaces into your workshop. Now, it’s time to start using those tools.

The Blueprint for Your CLI: The Arguments Struct

The modern, derive-based approach to using clap is beautifully declarative. Instead of writing a complex series of builder methods to define your CLI, you create a simple Rust struct. This struct acts as a blueprint or a container for all the configuration options your application will accept. Each field in the struct will correspond to a command-line argument.

This approach has several key advantages: * Type Safety: The arguments will be parsed directly into the correct Rust types (e.g., String, u32). If a user provides text where a number is expected, clap will automatically generate an error. * Readability: The struct definition provides a single, clear, and easy-to-read overview of your entire CLI’s structure. * Maintainability: Adding, removing, or changing an argument is as simple as modifying a single line in your struct.

Our first step is to create this container struct. We’ll name it CliArgs and place it near the top of our src/main.rs file.

Bringing clap’s Magic into Scope

To connect our struct to clap’s machinery, we need two things:

  1. A use Statement: We need to bring the Parser trait into scope. A trait in Rust defines a set of methods that a type can implement. The Parser trait provides the essential .parse() method that we’ll call later to trigger the argument parsing. It’s also required by the derive macro. We’ll add use clap::Parser; at the top of our file.
  2. The #[derive(Parser)] Macro: This is the heart of clap’s magic. The #[derive(...)] attribute is a special kind of Rust macro called a procedural macro. When you add #[derive(Parser)] above a struct, you are giving an instruction to the Rust compiler: “Hey, at compile time, please look at this struct and automatically generate all the complex code necessary to parse command-line arguments into it.” It writes all the boilerplate for you, including the parsing logic, help message generation, and error handling.

Let’s add the new use statement and the CliArgs struct definition to src/main.rs. For now, the struct will be empty; we will add fields to it in the very next task.

// In src/main.rs

use csv::Writer;
use scraper::{Html, Selector};
use tokio::sync::mpsc;
// highlight-start
// Import the `Parser` trait from the clap crate.
// This trait provides the `parse()` method that we'll use to
// trigger the argument parsing.
use clap::Parser;
// highlight-end

#[derive(Debug, serde::Serialize)]
pub struct Product {
    title: String,
    price: String,
    url: String,
}

// highlight-start
/// A concurrent web scraper to fetch product information from a paginated e-commerce website.
#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct CliArgs {
    // We will add fields here in the next task to define our command-line arguments.
}
// highlight-end

async fn scrape_page(url: &str) -> Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>> {
    // ... function body remains unchanged ...
    let resp = reqwest::get(url).await?.text().await?;
    let document = scraper::Html::parse_document(&resp);

    let product_selector = scraper::Selector::parse("article.product_pod").unwrap();
    let title_selector = scraper::Selector::parse("h3 > a").unwrap();
    let price_selector = scraper::Selector::parse(".price_color").unwrap();
    let next_page_selector = scraper::Selector::parse("li.next > a").unwrap();

    let mut products = Vec::new();

    for element in document.select(&product_selector) {
        let title_element = element.select(&title_selector).next();
        let title = title_element.and_then(|a| a.value().attr("title")).unwrap_or_default().to_string();

        let price_element = element.select(&price_selector).next();
        let price = price_element.map(|p| p.inner_html()).unwrap_or_default().to_string();

        let url_element = element.select(&title_selector).next();
        let product_url = url_element.and_then(|a| a.value().attr("href")).unwrap_or_default().to_string();

        products.push(Product {
            title,
            price,
            url: product_url,
        });
    }

    let next_page_element = document.select(&next_page_selector).next();

    let next_page_url = next_page_element
        .and_then(|element| {
            element.value().attr("href")
        })
        .map(|relative_url| {
            let mut base_url = url.to_string();
            if let Some(last_slash_pos) = base_url.rfind('/') {
                base_url.truncate(last_slash_pos + 1);
            }
            base_url.push_str(relative_url);
            base_url
        });

    Ok((products, next_page_url))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // ... main function body remains unchanged for now ...
    let output_filename = "products.csv";

    let seed_url = "http://books.toscrape.com/catalogue/category/books/travel_2/index.html".to_string();

    println!("Starting the crawler at seed URL: {}", seed_url);

    let (tx, mut rx) = mpsc::channel::<Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>>>(100);

    let mut tasks_in_flight = 0;

    let initial_tx_clone = tx.clone();
    tokio::spawn(async move {
        println!("Scraping URL: {}", seed_url);
        let scrape_result = scrape_page(&seed_url).await;
        if let Err(e) = initial_tx_clone.send(scrape_result).await {
            eprintln!("Failed to send result for url {}: {}", seed_url, e);
        }
    });
    tasks_in_flight += 1;

    let mut all_products: Vec<Product> = Vec::new();

    while tasks_in_flight > 0 {
        if let Some(scrape_result) = rx.recv().await {
            tasks_in_flight -= 1;

            match scrape_result {
                Ok((products, next_url_option)) => {
                    println!("Received {} products. {} tasks remain.", products.len(), tasks_in_flight);
                    all_products.extend(products);

                    if let Some(next_url) = next_url_option {
                        println!("Discovered new page, spawning task for: {}", next_url);

                        let tx_clone = tx.clone();
                        tokio::spawn(async move {
                            println!("Scraping URL: {}", next_url);
                            let scrape_result = scrape_page(&next_url).await;
                            if let Err(e) = tx_clone.send(scrape_result).await {
                                eprintln!("Failed to send result for url {}: {}", next_url, e);
                            }
                        });
                        tasks_in_flight += 1;
                    }
                }
                Err(e) => {
                    eprintln!("A scraper task reported an error: {}", e);
                }
            }
        }
    }

    drop(tx);

    println!("\n--- Crawl finished. All tasks complete. ---");
    println!("Total products scraped: {}\n", all_products.len());

    println!("Writing {} products to '{}'...", all_products.len(), output_filename);
    let mut writer = Writer::from_path(output_filename)?;
    for product in &all_products {
        writer.serialize(product)?;
    }
    writer.flush()?;
    println!(
        "Successfully saved {} products to '{}'.",
        all_products.len(),
        output_filename
    );

    Ok(())
}

Note: The /// doc comment and #[command(...)] attribute on the struct are good practices that clap uses to automatically generate a rich help message. For example, the doc comment becomes the “about” text.

You have now created the fundamental scaffolding for your command-line interface. The CliArgs struct is ready to be populated with the specific arguments we need.

What’s Next?

The blueprint is ready, but it has no rooms. In the next task, you will add fields to the CliArgs struct, such as start_url: String and output_file: String. You will then use more of clap’s powerful attributes to specify how these fields should be parsed from the command line (e.g., as --url or -o).

Further Reading

Define Command-Line Arguments in Rust with clap

Mục tiêu: Learn to configure your Rust CLI by defining required and optional command-line arguments with default values using the clap crate’s #[arg] derive attribute. See how clap automatically generates help messages from your code comments.

Of course! As a technology expert, I will provide a detailed solution to help you with your current task.

Furnishing the Blueprint: Defining Your Command-Line Arguments

You’ve successfully set up the foundation for your Command-Line Interface (CLI) by creating the empty CliArgs struct and decorating it with #[derive(Parser)]. You’ve essentially built the blueprint for your application’s configuration. Now, it’s time to furnish this blueprint by defining the specific “rooms” or fields that will hold the configuration values provided by the user.

Each field you add to this struct will represent a distinct command-line argument. The real power of clap’s derive macro comes from the attributes you can add to these fields. These attributes are instructions to clap, telling it exactly how to parse an argument, what its name should be, whether it’s optional, and what help text to show for it.

The primary attribute for this is #[arg(...)]. Inside the parentheses, you can provide a variety of key-value settings to configure the argument’s behavior.

We need two key pieces of information from the user:

  1. The starting URL: The “seed” from which our crawler will begin its journey. This is essential, so it must be a required argument.
  2. The output filename: Where the final CSV file should be saved. It’s good practice to provide a sensible default for this, so the user doesn’t always have to specify it.

Let’s add these two fields to our CliArgs struct.

1. Defining the start_url Argument

We will add a field named start_url of type String. We’ll configure it using #[arg(...)] to be recognized by both a long name (--url) and a short name (-u). By default, because this field is a String and we are not wrapping it in an Option or providing a default value, clap will intelligently make it a required argument. If the user tries to run the program without it, clap will automatically generate an error and show the help message.

2. Defining the output_file Argument

Next, we’ll add a field named output_file, also of type String. We will configure it with long (--output) and short (-o) names. For this argument, we’ll add a user-friendly feature: a default value. By setting default_value = "products.csv", we make this argument optional. If the user provides --output some_other_name.csv, that value will be used. If they don’t, the output_file field will be automatically populated with "products.csv".

We will also add doc comments (///) to each field. clap is smart enough to use these comments as the help text for each argument when a user runs your program with --help. This is a fantastic way to keep your code and your documentation in perfect sync.

Let’s modify the CliArgs struct in src/main.rs.

// In src/main.rs

// ... (use statements) ...
use clap::Parser;

// ... (Product struct) ...

/// A concurrent web scraper to fetch product information from a paginated e-commerce website.
#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct CliArgs {
    // highlight-start
    /// The starting URL to begin the crawl.
    #[arg(short = 'u', long = "url")]
    start_url: String,

    /// The path to the output file where the scraped data will be saved.
    #[arg(short = 'o', long = "output", default_value = "products.csv")]
    output_file: String,
    // highlight-end
}

// ... (the rest of the file, including scrape_page and main, remains unchanged for now) ...

Witnessing the Magic: Your Auto-Generated Help Message

You have now fully defined the interface for your command-line tool. You haven’t written any parsing logic yourself, but clap has already generated it all for you behind the scenes.

The most immediate and satisfying way to see the result of your work is to ask clap to generate the help message. Open your terminal in the project directory and run this command:

cargo run -- --help

You should see a beautifully formatted, professional help message that looks something like this:

A concurrent web scraper to fetch product information from a paginated e-commerce website.

Usage: concurrent_scraper --url <START_URL> [OPTIONS]

Options:
  -u, --url <START_URL>
          The starting URL to begin the crawl
  -o, --output <OUTPUT_FILE>
          The path to the output file where the scraped data will be saved
          [default: products.csv]
  -h, --help
          Print help
  -V, --version
          Print version

Look at everything clap did for you automatically: * It used the doc comment on the struct for the main description. * It created entries for --url and --output with their short versions. * It used the doc comments on the fields for their individual help text. * It correctly identified that --url is required (indicated by <START_URL> in the usage line). * It shows the default value for the --output option. * It automatically added the standard --help and --version flags.

This is the power of clap’s derive API: you declare the what (the shape of your configuration), and clap handles the how (the parsing, validation, and help generation).

What’s Next?

Your application now understands its command-line arguments, but it doesn’t do anything with them yet. The seed_url and output_filename in your main function are still hardcoded. In the next task, you will bridge this final gap. You will call CliArgs::parse() at the beginning of your main function to get an instance of your struct, and then you will use the values from its fields to replace the hardcoded strings, making your crawler truly flexible and dynamic.

Further Reading

To become more proficient with clap, exploring the documentation on field attributes is highly recommended.

Parse Command-Line Arguments with Clap

Mục tiêu: Activate the command-line argument parser by calling the CliArgs::parse() method at the beginning of the main function. This will read and validate user-provided arguments, populating an instance of the CliArgs struct or exiting with an error if the arguments are invalid.

Activating the Parser: Bringing Your CLI to Life

You have done a phenomenal job in the previous tasks. You’ve defined a clear, readable, and type-safe blueprint for your application’s configuration using the CliArgs struct. You even witnessed the magic of clap by running cargo run -- --help and seeing the professional-grade help message it generated for you automatically.

That CliArgs struct is currently just a definition, a set of instructions for the compiler. The current task is to activate it—to call the function that tells clap, “Okay, now is the time. Look at the arguments the user provided, parse them according to the rules I defined in CliArgs, and give me the results.”

The Engine of clap: The .parse() Method

The #[derive(Parser)] macro did more than just prepare your struct for parsing; it also automatically implemented the Parser trait for CliArgs. This trait provides a suite of useful functions, the most important of which is parse().

CliArgs::parse() is an associated function (often called a static method in other languages). You call it on the type CliArgs itself, not on an instance of it. Its job is simple in concept but powerful in execution:

  1. Inspects the Environment: It looks at the command-line arguments that were passed to your program when it was executed.
  2. Applies the Rules: It validates these arguments against the rules you defined with the #[arg(...)] attributes on the fields of CliArgs. It checks for required arguments, applies default values, and ensures the format is correct.
  3. Handles Success and Failure:
    • On Success: If all arguments are valid, it creates a new instance of your CliArgs struct, populates its fields (start_url, output_file) with the user-provided (or default) values, and returns this instance to you.
    • On Failure: If the user forgot a required argument (like --url), provided an invalid one, or asked for --help, CliArgs::parse() takes control. It will automatically print a helpful error message and exit the program. This is an incredibly powerful feature. It means your main logic will only ever run if the configuration is valid, freeing you from writing tedious validation and error-handling code.

The best practice is to parse your arguments as the very first action in your main function. This ensures that the program fails fast if the configuration is bad, before it wastes time setting up channels, making network connections, or allocating memory.

Let’s add this single, powerful line to the top of your main function in src/main.rs.

// In src/main.rs

// ... (use statements, Product struct, CliArgs struct, and scrape_page function remain unchanged) ...
use csv::Writer;
use scraper::{Html, Selector};
use tokio::sync::mpsc;
use clap::Parser;

#[derive(Debug, serde::Serialize)]
pub struct Product {
    title: String,
    price: String,
    url: String,
}

/// A concurrent web scraper to fetch product information from a paginated e-commerce website.
#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct CliArgs {
    /// The starting URL to begin the crawl.
    #[arg(short = 'u', long = "url")]
    start_url: String,

    /// The path to the output file where the scraped data will be saved.
    #[arg(short = 'o', long = "output", default_value = "products.csv")]
    output_file: String,
}

async fn scrape_page(url: &str) -> Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>> {
    let resp = reqwest::get(url).await?.text().await?;
    let document = scraper::Html::parse_document(&resp);

    let product_selector = scraper::Selector::parse("article.product_pod").unwrap();
    let title_selector = scraper::Selector::parse("h3 > a").unwrap();
    let price_selector = scraper::Selector::parse(".price_color").unwrap();
    let next_page_selector = scraper::Selector::parse("li.next > a").unwrap();

    let mut products = Vec::new();

    for element in document.select(&product_selector) {
        let title_element = element.select(&title_selector).next();
        let title = title_element.and_then(|a| a.value().attr("title")).unwrap_or_default().to_string();

        let price_element = element.select(&price_selector).next();
        let price = price_element.map(|p| p.inner_html()).unwrap_or_default().to_string();

        let url_element = element.select(&title_selector).next();
        let product_url = url_element.and_then(|a| a.value().attr("href")).unwrap_or_default().to_string();

        products.push(Product {
            title,
            price,
            url: product_url,
        });
    }

    let next_page_element = document.select(&next_page_selector).next();

    let next_page_url = next_page_element
        .and_then(|element| {
            element.value().attr("href")
        })
        .map(|relative_url| {
            let mut base_url = url.to_string();
            if let Some(last_slash_pos) = base_url.rfind('/') {
                base_url.truncate(last_slash_pos + 1);
            }
            base_url.push_str(relative_url);
            base_url
        });

    Ok((products, next_page_url))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // highlight-start
    // This is the moment we activate clap. `CliArgs::parse()` reads the command-line
    // arguments, validates them according to the rules in our struct, and returns
    // a populated `CliArgs` instance. If arguments are invalid, it prints help
    // and exits the program.
    let args = CliArgs::parse();
    // highlight-end

    // These hardcoded values are now obsolete. We will replace them in the next task.
    let output_filename = "products.csv";
    let seed_url = "http://books.toscrape.com/catalogue/category/books/travel_2/index.html".to_string();

    println!("Starting the crawler at seed URL: {}", seed_url);

    let (tx, mut rx) = mpsc::channel::<Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>>>(100);

    // ... (the rest of the main function remains unchanged for now) ...
    let mut tasks_in_flight = 0;

    let initial_tx_clone = tx.clone();
    tokio::spawn(async move {
        println!("Scraping URL: {}", seed_url);
        let scrape_result = scrape_page(&seed_url).await;
        if let Err(e) = initial_tx_clone.send(scrape_result).await {
            eprintln!("Failed to send result for url {}: {}", seed_url, e);
        }
    });
    tasks_in_flight += 1;

    let mut all_products: Vec<Product> = Vec::new();

    while tasks_in_flight > 0 {
        if let Some(scrape_result) = rx.recv().await {
            tasks_in_flight -= 1;

            match scrape_result {
                Ok((products, next_url_option)) => {
                    println!("Received {} products. {} tasks remain.", products.len(), tasks_in_flight);
                    all_products.extend(products);

                    if let Some(next_url) = next_url_option {
                        println!("Discovered new page, spawning task for: {}", next_url);

                        let tx_clone = tx.clone();
                        tokio::spawn(async move {
                            println!("Scraping URL: {}", next_url);
                            let scrape_result = scrape_page(&next_url).await;
                            if let Err(e) = tx_clone.send(scrape_result).await {
                                eprintln!("Failed to send result for url {}: {}", next_url, e);
                            }
                        });
                        tasks_in_flight += 1;
                    }
                }
                Err(e) => {
                    eprintln!("A scraper task reported an error: {}", e);
                }
            }
        }
    }

    drop(tx);

    println!("\n--- Crawl finished. All tasks complete. ---");
    println!("Total products scraped: {}\n", all_products.len());

    println!("Writing {} products to '{}'...", all_products.len(), output_filename);
    let mut writer = Writer::from_path(output_filename)?;
    for product in &all_products {
        writer.serialize(product)?;
    }
    writer.flush()?;
    println!(
        "Successfully saved {} products to '{}'.",
        all_products.len(),
        output_filename
    );

    Ok(())
}

You have now successfully parsed the user’s command-line arguments into a clean, type-safe args variable. The configuration logic is complete. All that’s left is to use this configuration to drive the application’s behavior.

What’s Next?

The args variable now holds the user’s desired start_url and output_file, but our main function is still using its old, hardcoded seed_url and output_filename variables. The final task in this step is to connect the two. You will replace the hardcoded values with args.start_url and args.output_file, making your crawler fully configurable from the command line and completing its transformation into a flexible tool.

Further Reading

Connect CLI Arguments to Crawler Logic

Mục tiêu: Refactor the main function of the Rust web crawler to use parsed command-line arguments for the start URL and output file, replacing the existing hardcoded values to make the tool dynamic and user-configurable.

The Final Connection: Powering Your Crawler with User Input

This is the moment where all your work on the command-line interface comes together. In the previous task, you brilliantly activated clap’s parsing engine by calling CliArgs::parse(). This single line of code reads the user’s input, validates it, and hands you a perfectly formed CliArgs struct, which we’ve stored in the args variable.

You now have a container, args, that holds the user’s desired starting URL and output filename. However, your application’s logic is still running on its old, hardcoded values. This final task is about severing those last hardcoded ties and “wiring up” the args struct to drive the crawler’s behavior. You are about to complete the transformation of your script into a truly flexible and reusable command-line tool.

From Static Variables to Dynamic Fields

The process is straightforward: we will find every place in our main function that uses the old seed_url and output_filename variables and replace them with the corresponding fields from our args struct, which are accessed using dot notation: args.start_url and args.output_file.

This change accomplishes a critical separation of concerns: * Configuration (CliArgs struct): Defines what the program should do. * Logic (main function): Defines how the program does it, using the provided configuration.

Let’s modify the main function to use the parsed arguments. We’ll start by removing the now-obsolete hardcoded variable definitions and then update the rest of the code to pull the values directly from args.

// In src/main.rs

// ... (use statements, Product struct, CliArgs struct, and scrape_page function remain unchanged) ...
use csv::Writer;
use scraper::{Html, Selector};
use tokio::sync::mpsc;
use clap::Parser;

#[derive(Debug, serde::Serialize)]
pub struct Product {
    title: String,
    price: String,
    url: String,
}

/// A concurrent web scraper to fetch product information from a paginated e-commerce website.
#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct CliArgs {
    /// The starting URL to begin the crawl.
    #[arg(short = 'u', long = "url")]
    start_url: String,

    /// The path to the output file where the scraped data will be saved.
    #[arg(short = 'o', long = "output", default_value = "products.csv")]
    output_file: String,
}

async fn scrape_page(url: &str) -> Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>> {
    let resp = reqwest::get(url).await?.text().await?;
    let document = scraper::Html::parse_document(&resp);

    let product_selector = scraper::Selector::parse("article.product_pod").unwrap();
    let title_selector = scraper::Selector::parse("h3 > a").unwrap();
    let price_selector = scraper::Selector::parse(".price_color").unwrap();
    let next_page_selector = scraper::Selector::parse("li.next > a").unwrap();

    let mut products = Vec::new();

    for element in document.select(&product_selector) {
        let title_element = element.select(&title_selector).next();
        let title = title_element.and_then(|a| a.value().attr("title")).unwrap_or_default().to_string();

        let price_element = element.select(&price_selector).next();
        let price = price_element.map(|p| p.inner_html()).unwrap_or_default().to_string();

        let url_element = element.select(&title_selector).next();
        let product_url = url_element.and_then(|a| a.value().attr("href")).unwrap_or_default().to_string();

        products.push(Product {
            title,
            price,
            url: product_url,
        });
    }

    let next_page_element = document.select(&next_page_selector).next();

    let next_page_url = next_page_element
        .and_then(|element| {
            element.value().attr("href")
        })
        .map(|relative_url| {
            let mut base_url = url.to_string();
            if let Some(last_slash_pos) = base_url.rfind('/') {
                base_url.truncate(last_slash_pos + 1);
            }
            base_url.push_str(relative_url);
            base_url
        });

    Ok((products, next_page_url))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args = CliArgs::parse();

    // highlight-start
    // We no longer need the hardcoded variables. We will use `args.start_url`
    // and `args.output_file` directly.
    // let output_filename = "products.csv";
    // let seed_url = "http://books.toscrape.com/catalogue/category/books/travel_2/index.html".to_string();

    // Update the initial print statement to use the user-provided URL.
    println!("Starting the crawler at seed URL: {}", &args.start_url);
    // highlight-end

    let (tx, mut rx) = mpsc::channel::<Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>>>(100);

    let mut tasks_in_flight = 0;

    let initial_tx_clone = tx.clone();
    // highlight-start
    // The `async move` block now takes ownership of `args.start_url`.
    // Thanks to Rust's disjoint field captures, we can still use `args.output_file`
    // later in the function without any issues.
    let seed_url_for_task = args.start_url.clone(); // Clone to satisfy the move, as args is used later
    tokio::spawn(async move {
        println!("Scraping URL: {}", seed_url_for_task);
        let scrape_result = scrape_page(&seed_url_for_task).await;
        if let Err(e) = initial_tx_clone.send(scrape_result).await {
            eprintln!("Failed to send result for url {}: {}", seed_url_for_task, e);
        }
    });
    // highlight-end
    tasks_in_flight += 1;

    let mut all_products: Vec<Product> = Vec::new();

    while tasks_in_flight > 0 {
        if let Some(scrape_result) = rx.recv().await {
            tasks_in_flight -= 1;

            match scrape_result {
                Ok((products, next_url_option)) => {
                    println!("Received {} products. {} tasks remain.", products.len(), tasks_in_flight);
                    all_products.extend(products);

                    if let Some(next_url) = next_url_option {
                        println!("Discovered new page, spawning task for: {}", next_url);

                        let tx_clone = tx.clone();
                        tokio::spawn(async move {
                            println!("Scraping URL: {}", next_url);
                            let scrape_result = scrape_page(&next_url).await;
                            if let Err(e) = tx_clone.send(scrape_result).await {
                                eprintln!("Failed to send result for url {}: {}", next_url, e);
                            }
                        });
                        tasks_in_flight += 1;
                    }
                }
                Err(e) => {
                    eprintln!("A scraper task reported an error: {}", e);
                }
            }
        }
    }

    drop(tx);

    println!("\n--- Crawl finished. All tasks complete. ---");
    println!("Total products scraped: {}\n", all_products.len());

    // highlight-start
    // --- Data Persistence ---
    // Use the user-provided output filename from `args.output_file`.
    println!("Writing {} products to '{}'...", all_products.len(), &args.output_file);

    // Create the CSV writer using the path from our parsed arguments.
    let mut writer = Writer::from_path(&args.output_file)?;
    // highlight-end

    for product in &all_products {
        writer.serialize(product)?;
    }
    writer.flush()?;

    // highlight-start
    // Update the final confirmation message to reflect the correct output file.
    println!(
        "Successfully saved {} products to '{}'.",
        all_products.len(),
        &args.output_file
    );
    // highlight-end

    Ok(())
}

You’ve Built a Real Tool!

Congratulations! Your program is no longer just a script; it is a flexible, reusable, and professional command-line tool. You can now direct it to scrape different websites and save the results to different files, all without ever touching the source code.

Try it out from your terminal. Remember that cargo run needs a -- separator to distinguish its own arguments from your application’s arguments.

Example 1: Scrape the “Mystery” category, saving to the default products.csv:

cargo run -- --url "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html"

Example 2: Scrape the “Historical Fiction” category, saving to a custom file named fiction.csv:

cargo run -- --url "http://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html" -o fiction.csv

What’s Next?

Your scraper is powerful and flexible, but it’s also a bit aggressive. It makes network requests as fast as it possibly can. While this is great for performance, it’s not very polite to the servers you are scraping. In the real world, making too many requests too quickly can get your IP address temporarily or permanently banned.

In the next major step, you will learn how to be a “good web citizen” by implementing polite scraping practices. You will add the ability to configure a delay between requests to avoid overwhelming the server, and you will set a custom User-Agent header, which is a standard practice for identifying your bot.

Further Reading