Add a Delay Argument for Polite Scraping

Mục tiêu: Extend a Rust CLI web scraper by adding an optional ‘–delay’ argument using the ‘clap’ crate to enable rate limiting and polite scraping practices.

Becoming a Good Web Citizen: Implementing Polite Scraping

You have successfully built a powerful and flexible command-line tool. Your crawler can be pointed at any starting URL and will diligently scrape all connected pages, saving the results to a user-specified file. This is a huge milestone! However, with great power comes great responsibility.

Currently, your scraper is highly aggressive. As soon as one task finishes, another one starts, firing off HTTP requests as fast as your computer and network will allow. While this maximizes speed, it’s considered poor practice when interacting with web servers. Making too many requests in a short period can overload a website’s server, degrade performance for other users, and can lead to your IP address being temporarily or even permanently blocked.

This step is all about transforming your scraper from an aggressive bot into a “good web citizen” by implementing polite scraping practices. The first and most important of these practices is rate limiting: intentionally slowing down your requests to a reasonable pace.

We will achieve this by adding a new, optional command-line argument that allows the user to specify a delay, in milliseconds, to wait between requests. This gives the user control over the scraper’s “politeness” level.

Extending Your CLI with a Delay Argument

Since we’ve already done the hard work of setting up clap, adding a new argument is incredibly simple. We just need to add a new field to our CliArgs struct.

We will add a field named delay_ms of type u64. A u64 is an unsigned 64-bit integer, which is a perfect choice for representing a non-negative duration in milliseconds.

We’ll configure this new field using clap’s #[arg(...)] attribute with the following properties: * long = "delay": This will allow the user to specify the delay with a clear, descriptive flag, like --delay 500. * short = 'd': This provides a convenient shorthand: -d 500. * default_value = "0": This is a crucial setting. By providing a default value, we make the argument optional. If the user doesn’t specify a delay, it will default to 0, meaning the scraper will run at full speed as it does now. clap will automatically parse the string "0" into the u64 type of our field. * Doc Comment (///): Just as before, we’ll add a descriptive doc comment above the field. clap will use this text to explain the argument in the --help message, making our tool self-documenting.

Let’s modify the CliArgs struct in src/main.rs.

// In src/main.rs

// ... (use statements) ...
use clap::Parser;

// ... (Product struct) ...

/// A concurrent web scraper to fetch product information from a paginated e-commerce website.
#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct CliArgs {
    /// The starting URL to begin the crawl.
    #[arg(short = 'u', long = "url")]
    start_url: String,

    /// The path to the output file where the scraped data will be saved.
    #[arg(short = 'o', long = "output", default_value = "products.csv")]
    output_file: String,

    // highlight-start
    /// The delay in milliseconds between each HTTP request.
    #[arg(short = 'd', long = "delay", default_value = "0")]
    delay_ms: u64,
    // highlight-end
}

// ... (The rest of the file, including scrape_page and main, remains unchanged for this task) ...

That’s it! With just those few lines, you have successfully extended your CLI to support a new configuration option.

To see the result immediately, run the help command in your terminal:

cargo run -- --help

You will now see your new option integrated perfectly into the professional help message:

A concurrent web scraper to fetch product information from a paginated e-commerce website.

Usage: concurrent_scraper --url <START_URL> [OPTIONS]

Options:
  -u, --url <START_URL>
          The starting URL to begin the crawl
  -o, --output <OUTPUT_FILE>
          The path to the output file where the scraped data will be saved
          [default: products.csv]
  -d, --delay <DELAY_MS>
          The delay in milliseconds between each HTTP request
          [default: 0]
  -h, --help
          Print help
  -V, --version
          Print version

What’s Next?

Your application now knows about the desired delay, but it doesn’t yet act on it. The delay_ms value is parsed and stored in the args struct, but it’s not being used anywhere in the scraping logic.

In the next task, you will put this value to use. You will modify your scraping logic to incorporate an asynchronous “sleep” before each HTTP request is made, using the value provided by the user via this new CLI flag.

Implement Asynchronous Delay in a Rust Web Scraper

Mục tiêu: Modify an asynchronous Rust web scraper to implement a rate-limiting delay between HTTP requests. This involves using tokio::time::sleep to pause execution without blocking the Tokio runtime, making the scraper more polite to the target server.

Excellent! You’ve successfully added a new --delay option to your command-line interface, giving you the power to configure the scraper’s speed. Your application now parses this value and stores it, but it doesn’t yet act on it. This task is about bringing that configuration to life by implementing the actual delay mechanism.

The Art of Asynchronous Pausing with `tokio::time::sleep`

In a traditional, synchronous program, you might use std::thread::sleep to pause execution. However, in an asynchronous environment like ours, powered by Tokio, this would be a major mistake. Calling std::thread::sleep would block the entire OS thread, freezing not just the current task but also preventing any other asynchronous tasks managed by that thread from making progress. It defeats the entire purpose of non-blocking I/O.

This is where Tokio’s own time utilities come into play. The function tokio::time::sleep is the asynchronous equivalent. When you await a call to tokio::time::sleep, you are not blocking the thread. Instead, you are yielding control back to the Tokio runtime. You’re essentially telling the runtime, “Wake this task up after the specified duration has passed. In the meantime, feel free to use this thread to run other tasks that are ready.” This is the cornerstone of cooperative multitasking and is essential for building efficient, high-performance concurrent applications.

Our plan is simple:

Modify the scrape_page function to accept the delay value.
Inside scrape_page, right before making the HTTP request, we’ll call tokio::time::sleep.
Update the main function to pass the delay value from our parsed arguments into every spawned scraping task.

Modifying the Scraper to Respect the Delay

First, let’s update the scrape_page function. It needs to know how long to wait, so we’ll add a delay_ms: u64 parameter to its signature. Then, at the very top of the function, we’ll add our sleep logic. We’ll also add a check to only sleep if the delay is greater than zero, and a println! to give the user visual feedback that the rate limiting is working.

Next, we need to plumb this new parameter through our main function. The delay_ms value is available in the args struct. We need to pass this value into each tokio::spawn block so the worker task has access to it.

Here are the required changes in src/main.rs:

// In src/main.rs

// ... (use statements, Product struct, and CliArgs struct are unchanged) ...

// highlight-start
// The function signature is updated to accept the delay in milliseconds.
async fn scrape_page(url: &str, delay_ms: u64) -> Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>> {
    // If a non-zero delay is specified, we perform an asynchronous sleep.
    // This pauses the current task without blocking the entire thread,
    // allowing other tasks to run.
    if delay_ms > 0 {
        println!("Waiting for {}ms before scraping {}...", delay_ms, url);
        // `tokio::time::Duration::from_millis` creates a duration object.
        // `tokio::time::sleep` returns a 'Future' that completes after the duration.
        // We `.await` it to pause the execution of this function.
        tokio::time::sleep(tokio::time::Duration::from_millis(delay_ms)).await;
    }
// highlight-end

    let resp = reqwest::get(url).await?.text().await?;
    let document = scraper::Html::parse_document(&resp);

    // ... (rest of scrape_page function remains unchanged) ...

    let product_selector = scraper::Selector::parse("article.product_pod").unwrap();
    let title_selector = scraper::Selector::parse("h3 > a").unwrap();
    let price_selector = scraper::Selector::parse(".price_color").unwrap();
    let next_page_selector = scraper::Selector::parse("li.next > a").unwrap();

    let mut products = Vec::new();

    for element in document.select(&product_selector) {
        let title_element = element.select(&title_selector).next();
        let title = title_element.and_then(|a| a.value().attr("title")).unwrap_or_default().to_string();

        let price_element = element.select(&price_selector).next();
        let price = price_element.map(|p| p.inner_html()).unwrap_or_default().to_string();

        let url_element = element.select(&title_selector).next();
        let product_url = url_element.and_then(|a| a.value().attr("href")).unwrap_or_default().to_string();

        products.push(Product {
            title,
            price,
            url: product_url,
        });
    }

    let next_page_element = document.select(&next_page_selector).next();

    let next_page_url = next_page_element
        .and_then(|element| {
            element.value().attr("href")
        })
        .map(|relative_url| {
            let mut base_url = url.to_string();
            if let Some(last_slash_pos) = base_url.rfind('/') {
                base_url.truncate(last_slash_pos + 1);
            }
            base_url.push_str(relative_url);
            base_url
        });

    Ok((products, next_page_url))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args = CliArgs::parse();

    println!("Starting the crawler at seed URL: {}", &args.start_url);

    let (tx, mut rx) = mpsc::channel::<Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>>>(100);

    let mut tasks_in_flight = 0;

    let initial_tx_clone = tx.clone();
    let seed_url_for_task = args.start_url.clone();

    // highlight-start
    // Capture the delay value from our parsed arguments.
    // We must pass this into the spawned task.
    let delay_for_task = args.delay_ms;
    tokio::spawn(async move {
        // The spawned task now calls `scrape_page` with the delay argument.
        let scrape_result = scrape_page(&seed_url_for_task, delay_for_task).await;
        if let Err(e) = initial_tx_clone.send(scrape_result).await {
            eprintln!("Failed to send result for url {}: {}", seed_url_for_task, e);
        }
    });
    // highlight-end
    tasks_in_flight += 1;

    let mut all_products: Vec<Product> = Vec::new();

    while tasks_in_flight > 0 {
        if let Some(scrape_result) = rx.recv().await {
            tasks_in_flight -= 1;

            match scrape_result {
                Ok((products, next_url_option)) => {
                    println!("Received {} products. {} tasks remain.", products.len(), tasks_in_flight);
                    all_products.extend(products);

                    if let Some(next_url) = next_url_option {
                        println!("Discovered new page, spawning task for: {}", next_url);

                        let tx_clone = tx.clone();
                        // highlight-start
                        // We must also pass the delay to any newly discovered pages.
                        let delay_for_task = args.delay_ms;
                        tokio::spawn(async move {
                            let scrape_result = scrape_page(&next_url, delay_for_task).await;
                            if let Err(e) = tx_clone.send(scrape_result).await {
                                eprintln!("Failed to send result for url {}: {}", next_url, e);
                            }
                        });
                        // highlight-end
                        tasks_in_flight += 1;
                    }
                }
                Err(e) => {
                    eprintln!("A scraper task reported an error: {}", e);
                }
            }
        }
    }

    // ... (rest of main function remains unchanged) ...

    drop(tx);

    println!("\n--- Crawl finished. All tasks complete. ---");
    println!("Total products scraped: {}\n", all_products.len());

    println!("Writing {} products to '{}'...", all_products.len(), &args.output_file);

    let mut writer = Writer::from_path(&args.output_file)?;

    for product in &all_products {
        writer.serialize(product)?;
    }
    writer.flush()?;

    println!(
        "Successfully saved {} products to '{}'.",
        all_products.len(),
        &args.output_file
    );

    Ok(())
}

Test Your Polite Scraper

You have now fully implemented rate limiting! Run your application from the command line and provide a delay. A value like 500 (half a second) or 1000 (one second) will make the effect very clear.

# Run with a 500ms delay between requests
cargo run -- --url "http://books.toscrape.com/catalogue/category/books/travel_2/index.html" --delay 500

You will now see your new log messages in the terminal, followed by a noticeable pause before the “Scraping URL…” message for the next page appears. You are now scraping responsibly.

What’s Next?

Adding a delay is the most important part of polite scraping, but there’s another common practice: setting a custom User-Agent. A User-Agent is an HTTP header that identifies the client making the request. By default, reqwest uses a generic one. Setting a custom one (e.g., “MyAwesomeRustScraper/1.0”) is a good practice that helps website administrators identify your bot’s traffic.

In the next task, you will create a reusable reqwest::Client using its builder pattern. This will allow you to configure default settings, like a custom User-Agent, that will be applied to all requests made by your application.

Create a Reusable HTTP Client for Efficient Scraping

Mục tiêu: Refactor the web scraper to use a single, reusable reqwest::Client instance. This improves performance by leveraging connection pooling, avoiding the overhead of creating new TCP/TLS connections for each request. The client will be created using the builder pattern at the start of the main function.

You’ve taken a major step towards responsible scraping by implementing a configurable delay. This is a crucial practice for any real-world automated web client. Now, let’s continue enhancing our HTTP request logic, focusing not just on politeness but also on performance and best practices.

So far, we have been using reqwest::get(url). This is a fantastic convenience function provided by the reqwest library for making simple, one-off GET requests. Behind the scenes, reqwest::get() is a shorthand that actually does two things: it builds a new reqwest::Client, uses it to make a single request, and then discards it.

While this is fine for a quick test, for a high-performance application like ours that makes many requests, this approach has two significant drawbacks:

Inefficiency: Each call creates a new client and potentially a new underlying network connection. Establishing a new TCP and TLS connection for every single request is computationally expensive and slow.
Lack of Configuration: We have no way to set default options that should apply to all requests, such as setting a custom User-Agent header or configuring timeouts.

The professional solution is to create a single, reusable reqwest::Client instance that our entire application can share.

Introducing `reqwest::Client` and the Builder Pattern

A reqwest::Client is a powerful object that acts as your application’s main entry point for making HTTP requests. It’s designed to be created once and reused for the lifetime of your application. Its key benefit is that it manages an internal connection pool. When you make a request to a host, it opens a connection and, after the request is done, keeps that connection alive and ready in its pool. The next time you request something from the same host, it can reuse the existing connection, skipping the expensive setup process. This dramatically improves performance.

To create a Client, we use a very common and powerful design pattern in Rust called the Builder Pattern. Instead of creating the object directly with a function that might have a dozen parameters, we follow a three-step process:

Call reqwest::Client::builder() to get a ClientBuilder instance.
Call various configuration methods on the ClientBuilder to set the options we want (we’ll do this in the next task).
Call .build() on the builder to consume it and produce our final, configured Client object.

This pattern makes code highly readable and allows for the easy construction of complex objects.

Let’s create our client at the beginning of the main function. It should be one of the first things we do, as it’s a core piece of our application’s infrastructure.

// In src/main.rs

// ... (use statements, Product struct, CliArgs struct, and scrape_page function remain unchanged) ...

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args = CliArgs::parse();

    // highlight-start
    // Create a single, reusable reqwest::Client.
    // Using a Client is significantly more efficient than one-shot functions like `reqwest::get()`
    // because it manages an internal connection pool, which can reuse connections for
    // multiple requests to the same host. This avoids the overhead of establishing a new
    // TCP and TLS connection for every request.
    println!("Initializing HTTP client...");
    // We use the builder pattern to construct the client. `Client::builder()` returns
    // a `ClientBuilder`. The `.build()` method consumes the builder and returns
    // a `Result<Client, ...>`, so we use `?` to handle any potential errors during creation.
    let client = reqwest::Client::builder().build()?;
    // highlight-end

    println!("Starting the crawler at seed URL: {}", &args.start_url);

    let (tx, mut rx) = mpsc::channel::<Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>>>(100);

    // ... (the rest of the main function remains unchanged for this task) ...
    let mut tasks_in_flight = 0;

    let initial_tx_clone = tx.clone();
    let seed_url_for_task = args.start_url.clone();
    let delay_for_task = args.delay_ms;
    tokio::spawn(async move {
        let scrape_result = scrape_page(&seed_url_for_task, delay_for_task).await;
        if let Err(e) = initial_tx_clone.send(scrape_result).await {
            eprintln!("Failed to send result for url {}: {}", seed_url_for_task, e);
        }
    });
    tasks_in_flight += 1;

    let mut all_products: Vec<Product> = Vec::new();

    while tasks_in_flight > 0 {
        if let Some(scrape_result) = rx.recv().await {
            tasks_in_flight -= 1;

            match scrape_result {
                Ok((products, next_url_option)) => {
                    println!("Received {} products. {} tasks remain.", products.len(), tasks_in_flight);
                    all_products.extend(products);

                    if let Some(next_url) = next_url_option {
                        println!("Discovered new page, spawning task for: {}", next_url);

                        let tx_clone = tx.clone();
                        let delay_for_task = args.delay_ms;
                        tokio::spawn(async move {
                            let scrape_result = scrape_page(&next_url, delay_for_task).await;
                            if let Err(e) = tx_clone.send(scrape_result).await {
                                eprintln!("Failed to send result for url {}: {}", next_url, e);
                            }
                        });
                        tasks_in_flight += 1;
                    }
                }
                Err(e) => {
                    eprintln!("A scraper task reported an error: {}", e);
                }
            }
        }
    }

    drop(tx);

    println!("\n--- Crawl finished. All tasks complete. ---");
    println!("Total products scraped: {}\n", all_products.len());

    println!("Writing {} products to '{}'...", all_products.len(), &args.output_file);

    let mut writer = Writer::from_path(&args.output_file)?;

    for product in &all_products {
        writer.serialize(product)?;
    }
    writer.flush()?;

    println!(
        "Successfully saved {} products to '{}'.",
        all_products.len(),
        &args.output_file
    );

    Ok(())
}

You have now instantiated a powerful, efficient, and reusable HTTP client. While the program’s behavior hasn’t changed yet, you have put in place a critical piece of infrastructure that enables more advanced configurations and better performance.

What’s Next?

The client is created, but it’s not being used yet. The real power of the builder pattern is in the configuration you can apply before calling .build(). In the very next task, you will use this builder to set a default User-Agent header for all requests made by the client. This is another key aspect of polite scraping, as it helps website administrators identify your bot’s traffic. After that, we’ll plumb the client through to our scrape_page function and replace reqwest::get() for good.

Configure a Custom User-Agent for the HTTP Client

Mục tiêu: Update the reqwest::Client creation logic to set a custom User-Agent header using the builder pattern. This identifies the scraper to web servers, a crucial practice for ethical and robust web scraping.

You’ve made a significant leap forward in the previous task by creating a reusable reqwest::Client. This is a professional-grade upgrade that improves performance by managing a connection pool. You’ve essentially built the engine for all our future HTTP requests. Now, it’s time to tune that engine and give it a proper identity.

Giving Your Scraper an Identity with the `User-Agent` Header

When any client—a web browser, a mobile app, or our scraper—makes a request to a server, it can include a set of HTTP headers. These headers provide metadata about the request. One of the most standard and important headers is the User-Agent.

The User-Agent is a string that identifies the client software making the request. For example, when you visit a website with Firefox, the User-Agent might be something like Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0.

Why is this important for our scraper?

Transparency and Politeness: By setting a custom User-Agent, we are being transparent with the website owner. We are clearly identifying our traffic as coming from an automated bot, not a human. This is a fundamental principle of ethical scraping.
Identification: A descriptive User-Agent allows website administrators to easily identify traffic from your scraper in their logs. If your scraper is accidentally causing issues, they can see where the traffic is coming from. Best practices even suggest including a URL or contact email in your User-Agent so they can reach you.
Avoiding Blocks: Many websites, especially those with anti-bot measures, will automatically block or throttle requests that have a generic, default User-Agent (like the one reqwest uses by default) or no User-Agent at all. Setting a custom, unique one can help your scraper appear more like a legitimate, well-behaved bot.

Configuring the `User-Agent` with the Client Builder

The reqwest::ClientBuilder you created in the last task makes setting a default User-Agent incredibly easy. The builder has a .user_agent() method that takes a string. When you use this method, the User-Agent you provide will be automatically added to every single request made by the resulting Client instance.

This is the power of the builder pattern: you configure it once, and the behavior applies universally. Let’s add this configuration to the client creation logic in your main function.

// In src/main.rs

// ... (use statements, Product struct, CliArgs struct, and scrape_page function remain unchanged) ...

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args = CliArgs::parse();

    println!("Initializing HTTP client...");
    // highlight-start
    // We chain the `.user_agent()` method onto our builder.
    // This sets a default User-Agent header that will be sent with every request
    // made by this client. This is a crucial step in polite scraping, as it
    // identifies our bot to the server administrators.
    let client = reqwest::Client::builder()
        .user_agent("ConcurrentRustScraper/0.1.0 (https://github.com/user/repo)")
        .build()?;
    // highlight-end

    println!("Starting the crawler at seed URL: {}", &args.start_url);

    // ... (the rest of the main function remains unchanged for this task) ...
}

Code Breakdown:

reqwest::Client::builder(): This is our starting point, creating the ClientBuilder.
.user_agent("ConcurrentRustScraper/0.1.0 (https://github.com/user/repo)"): This is the new line. We call the user_agent method on the builder.
- The string ConcurrentRustScraper/0.1.0 is a good, descriptive name. It follows the common product/version format.
- Including a link to the project’s repository (or a contact page) in parentheses is a best practice. It gives the site admin a way to learn more about your scraper or contact you if needed. You should replace this with your actual repository URL.
.build()?: This final call consumes the now-configured builder and creates the Client instance, which will now carry our custom User-Agent as its default identity.

You have now created and configured a performant and polite HTTP client. It’s ready to be put to work.

What’s Next?

The client is fully configured and ready to go, but our scrape_page function is still using the old, inefficient reqwest::get() function. The next logical step is to replace that call. You will use the newly configured client to make the request, which involves a slightly different but more flexible syntax: client.get(url).send().await?.

Integrate the Shared `reqwest::Client` into the Scraping Function

Mục tiêu: Refactor the scrape\_page function to accept a reqwest::Client instance. Replace the one-shot reqwest::get() call with the client’s request builder (client.get(url).send().await) to leverage connection pooling and default headers for improved performance.

Fantastic! You’ve successfully built and configured a professional-grade reqwest::Client. It’s sitting in your main function, ready for action, equipped with an efficient connection pool and a polite User-Agent header. The infrastructure is in place. Now, it’s time to route our application’s traffic through this new, high-performance engine.

Currently, your scrape_page function is still using the one-shot reqwest::get() function. This is like having a finely tuned race car engine but still using a bicycle to get around. This task is about swapping out that old bicycle for the new engine, ensuring all our HTTP requests gain the benefits of the client we just built.

From a Simple Function Call to a Powerful Request Builder

The reqwest::get(url) function is a convenient shorthand. To use our custom client, we’ll switch to a more explicit and powerful syntax: client.get(url).send().await. Let’s break down this chain:

client.get(url): This is the first step. We call the .get() method on our client instance, passing the URL. This does not send the request immediately. Instead, it creates and returns a RequestBuilder object. This intermediate object is incredibly powerful; you could use it to add custom headers or a timeout that applies only to this specific request before sending it.
.send(): This method, called on the RequestBuilder, is what actually executes the HTTP request. It’s an asynchronous operation, so it returns a Future.
.await: We await the Future from .send() to get the result, which will be a Result<reqwest::Response, reqwest::Error>.

This new syntax is the key to unlocking the full power of our configured client. Every request built this way will automatically use the client’s connection pool and include the default User-Agent header we set.

To implement this, we need to modify scrape_page to accept the client as a parameter. Then, we can replace the old reqwest::get() call with our new syntax.

Here is the updated code for scrape_page:

// In src/main.rs

// ... (use statements, Product struct, CliArgs struct are unchanged) ...

// highlight-start
// The function signature is updated to accept a reference to the reqwest::Client.
// This allows us to use the shared, pre-configured client for all our HTTP requests.
async fn scrape_page(
    client: &reqwest::Client,
    url: &str,
    delay_ms: u64,
) -> Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>> {
// highlight-end
    if delay_ms > 0 {
        println!("Waiting for {}ms before scraping {}...", delay_ms, url);
        tokio::time::sleep(tokio::time::Duration::from_millis(delay_ms)).await;
    }

    // highlight-start
    // We replace the one-shot `reqwest::get()` with our new, more efficient client method.
    // 1. `client.get(url)` creates a `RequestBuilder` with our client's configuration (e.g., User-Agent).
    // 2. `.send()` dispatches the request and returns a `Future`.
    // 3. We `.await` the future to get the response.
    // 4. We then get the response body as text and parse it.
    let resp_text = client.get(url).send().await?.text().await?;
    let document = scraper::Html::parse_document(&resp_text);
    // highlight-end

    // ... (rest of scrape_page function, including selector logic, is unchanged) ...
    let product_selector = scraper::Selector::parse("article.product_pod").unwrap();
    let title_selector = scraper::Selector::parse("h3 > a").unwrap();
    let price_selector = scraper::Selector::parse(".price_color").unwrap();
    let next_page_selector = scraper::Selector::parse("li.next > a").unwrap();

    let mut products = Vec::new();

    for element in document.select(&product_selector) {
        let title_element = element.select(&title_selector).next();
        let title = title_element.and_then(|a| a.value().attr("title")).unwrap_or_default().to_string();

        let price_element = element.select(&price_selector).next();
        let price = price_element.map(|p| p.inner_html()).unwrap_or_default().to_string();

        let url_element = element.select(&title_selector).next();
        let product_url = url_element.and_then(|a| a.value().attr("href")).unwrap_or_default().to_string();

        products.push(Product {
            title,
            price,
            url: product_url,
        });
    }

    let next_page_element = document.select(&next_page_selector).next();

    let next_page_url = next_page_element
        .and_then(|element| {
            element.value().attr("href")
        })
        .map(|relative_url| {
            let mut base_url = url.to_string();
            if let Some(last_slash_pos) = base_url.rfind('/') {
                base_url.truncate(last_slash_pos + 1);
            }
            base_url.push_str(relative_url);
            base_url
        });

    Ok((products, next_page_url))
}

// ... (The main function will be modified in the next task) ...

By making this change, you have successfully upgraded your scraper’s core logic to use the efficient, reusable client. Every request it makes will now be faster (due to connection reuse) and more polite (due to the custom User-Agent).

What’s Next?

Our worker function, scrape_page, is now ready to receive and use the client. However, our manager, the main function, doesn’t yet know how to provide it. You’ve updated the function’s signature, which will now cause a compiler error in main where it’s called. This is expected! In the next task, you will fix this by passing the configured client from main down into each of your spawned scraping tasks.

The Solution: `Arc<T>`, the Atomically Reference-Counted Smart Pointer

The idiomatic Rust solution for sharing read-only access to data across threads and async tasks is the Arc<T> smart pointer. Arc stands for Atomically Reference-Counted. Let’s break down what that means:

Smart Pointer: It’s a struct that wraps our data (reqwest::Client in this case) and provides additional capabilities. The data it wraps is allocated on the heap.
Reference-Counted: Arc keeps track of how many “owners” or active references to the data exist. When you first create an Arc with Arc::new(client), the reference count is 1.
Cloning an Arc: When you call .clone() on an Arc, you are not cloning the underlying data (the expensive reqwest::Client). Instead, you are creating a new pointer to the same data on the heap and atomically (thread-safely) incrementing the reference count. This operation is extremely cheap.
Dropping an Arc: When an Arc pointer goes out of scope (e.g., at the end of a task), it decrements the reference count. The underlying data is only truly deallocated and cleaned up when the count reaches zero.

This mechanism is a perfect fit for our needs. We can create one Arc<reqwest::Client> in main. Then, for each task we spawn, we create a cheap clone of the Arc and move it into the task. All tasks will be pointing to the exact same Client instance, sharing its connection pool and configuration safely.

Let’s modify our main function to implement this pattern.

// In src/main.rs

// highlight-start
// Import the `Arc` smart pointer for shared ownership.
use std::sync::Arc;
// highlight-end
use csv::Writer;
use scraper::{Html, Selector};
use tokio::sync::mpsc;
use clap::Parser;

// ... (Product struct, CliArgs struct, and the updated scrape_page function are unchanged) ...
#[derive(Debug, serde::Serialize)]
pub struct Product {
    title: String,
    price: String,
    url: String,
}

/// A concurrent web scraper to fetch product information from a paginated e-commerce website.
#[derive(Parser, Debug)]
#[command(version, about, long_about = None)]
struct CliArgs {
    /// The starting URL to begin the crawl.
    #[arg(short = 'u', long = "url")]
    start_url: String,

    /// The path to the output file where the scraped data will be saved.
    #[arg(short = 'o', long = "output", default_value = "products.csv")]
    output_file: String,

    /// The delay in milliseconds between each HTTP request.
    #[arg(short = 'd', long = "delay", default_value = "0")]
    delay_ms: u64,
}

async fn scrape_page(
    client: &reqwest::Client,
    url: &str,
    delay_ms: u64,
) -> Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>> {
    if delay_ms > 0 {
        println!("Waiting for {}ms before scraping {}...", delay_ms, url);
        tokio::time::sleep(tokio::time::Duration::from_millis(delay_ms)).await;
    }

    let resp_text = client.get(url).send().await?.text().await?;
    let document = scraper::Html::parse_document(&resp_text);

    let product_selector = scraper::Selector::parse("article.product_pod").unwrap();
    let title_selector = scraper::Selector::parse("h3 > a").unwrap();
    let price_selector = scraper::Selector::parse(".price_color").unwrap();
    let next_page_selector = scraper::Selector::parse("li.next > a").unwrap();

    let mut products = Vec::new();

    for element in document.select(&product_selector) {
        let title_element = element.select(&title_selector).next();
        let title = title_element.and_then(|a| a.value().attr("title")).unwrap_or_default().to_string();

        let price_element = element.select(&price_selector).next();
        let price = price_element.map(|p| p.inner_html()).unwrap_or_default().to_string();

        let url_element = element.select(&title_selector).next();
        let product_url = url_element.and_then(|a| a.value().attr("href")).unwrap_or_default().to_string();

        products.push(Product {
            title,
            price,
            url: product_url,
        });
    }

    let next_page_element = document.select(&next_page_selector).next();

    let next_page_url = next_page_element
        .and_then(|element| {
            element.value().attr("href")
        })
        .map(|relative_url| {
            let mut base_url = url.to_string();
            if let Some(last_slash_pos) = base_url.rfind('/') {
                base_url.truncate(last_slash_pos + 1);
            }
            base_url.push_str(relative_url);
            base_url
        });

    Ok((products, next_page_url))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let args = CliArgs::parse();

    println!("Initializing HTTP client...");
    // highlight-start
    // We wrap our configured client in an `Arc` to enable safe, shared ownership
    // across multiple asynchronous tasks.
    let client = Arc::new(reqwest::Client::builder()
        .user_agent("ConcurrentRustScraper/0.1.0 (https://github.com/user/repo)")
        .build()?);
    // highlight-end

    println!("Starting the crawler at seed URL: {}", &args.start_url);

    let (tx, mut rx) = mpsc::channel::<Result<(Vec<Product>, Option<String>), Box<dyn std::error::Error>>>(100);

    let mut tasks_in_flight = 0;

    let initial_tx_clone = tx.clone();
    let seed_url_for_task = args.start_url.clone();
    let delay_for_task = args.delay_ms;
    // highlight-start
    // Clone the Arc for the first task. This is a cheap operation.
    let client_for_task = Arc::clone(&client);
    tokio::spawn(async move {
        // Now we pass a reference to the client into our scrape_page function.
        // The `&` dereferences the `Arc<Client>` to a `&Client`.
        let scrape_result = scrape_page(&client_for_task, &seed_url_for_task, delay_for_task).await;
        if let Err(e) = initial_tx_clone.send(scrape_result).await {
            eprintln!("Failed to send result for url {}: {}", seed_url_for_task, e);
        }
    });
    // highlight-end
    tasks_in_flight += 1;

    let mut all_products: Vec<Product> = Vec::new();

    while tasks_in_flight > 0 {
        if let Some(scrape_result) = rx.recv().await {
            tasks_in_flight -= 1;

            match scrape_result {
                Ok((products, next_url_option)) => {
                    println!("Received {} products. {} tasks remain.", products.len(), tasks_in_flight);
                    all_products.extend(products);

                    if let Some(next_url) = next_url_option {
                        println!("Discovered new page, spawning task for: {}", next_url);

                        let tx_clone = tx.clone();
                        let delay_for_task = args.delay_ms;
                        // highlight-start
                        // We do the same for every subsequent task we spawn.
                        let client_for_task = Arc::clone(&client);
                        tokio::spawn(async move {
                            // The new task gets its own pointer to the shared client.
                            let scrape_result = scrape_page(&client_for_task, &next_url, delay_for_task).await;
                            if let Err(e) = tx_clone.send(scrape_result).await {
                                eprintln!("Failed to send result for url {}: {}", next_url, e);
                            }
                        });
                        // highlight-end
                        tasks_in_flight += 1;
                    }
                }
                Err(e) => {
                    eprintln!("A scraper task reported an error: {}", e);
                }
            }
        }
    }

    // ... (rest of main function is unchanged) ...
    drop(tx);

    println!("\n--- Crawl finished. All tasks complete. ---");
    println!("Total products scraped: {}\n", all_products.len());

    println!("Writing {} products to '{}'...", all_products.len(), &args.output_file);

    let mut writer = Writer::from_path(&args.output_file)?;

    for product in &all_products {
        writer.serialize(product)?;
    }
    writer.flush()?;

    println!(
        "Successfully saved {} products to '{}'.",
        all_products.len(),
        &args.output_file
    );

    Ok(())
}

Congratulations! You have now fully integrated the performant and polite HTTP client into your application’s concurrent architecture. Your scraper is now a well-behaved, efficient, and robust tool. It reuses connections, identifies itself with a User-Agent, and respects the server by rate-limiting its requests.

What’s Next?

Your application is now feature-complete and follows best practices for interaction and performance. The final step in this project is to focus on internal code quality and maintainability. Your src/main.rs file has grown quite large, containing logic for the CLI, the scraper, and the main application loop. In the next step, you will restructure your project by refactoring this code into separate, logical modules (cli.rs, scraper.rs, etc.). This will make your codebase cleaner, easier to navigate, and more maintainable in the long run.