From a Single URL to a Vector of URLs

Mục tiêu: Update the main function to define a vector of URLs to be scraped. This changes the input from a single string to a collection, preparing the application for concurrent scraping.

Congratulations! You have successfully built and perfected the scrape_page function. It is now a complete, robust, and self-contained unit of work that can reliably process a single URL and return structured data. This function is the engine of our scraper. Now, it’s time to build the chassis around it and transform our program from a single-cylinder engine into a multi-core powerhouse.

So far, our main function has been a simple test harness, calling scrape_page for just one URL. To unlock true concurrency, we must first change our input from a single item to a collection of items. Our goal is to give main a list of jobs to perform, setting the stage for processing them all in parallel.

From a Single Task to a Work Queue

The first step in this architectural shift is to define our “work queue.” Instead of a single String or &str holding one URL, we will now use a vector to hold a list of URLs. A Vec<T> (vector) in Rust is a growable, list-like data structure that stores a sequence of values of the same type T.

We will create a vector of string literals. In Rust, a string literal like "http://..." has the type &'static str. Let’s break that down:

&: This indicates it’s a reference, meaning we are borrowing the data, not owning it.
str: This is Rust’s primitive string slice type.
'static: This is a special lifetime annotation. It means the data is guaranteed to be valid for the entire duration of the program’s execution. String literals are perfect for this because they are baked directly into your compiled program’s binary.

Using Vec<&'static str> is highly efficient for a hardcoded list of URLs because we are just storing a list of pointers to data that already exists in our program’s memory.

Let’s update main in src/main.rs to replace the single URL with a vector of several URLs. We will also remove the old call to scrape_page, as we will be replacing it with a new concurrent looping mechanism in the upcoming tasks.

// src/main.rs

use scraper::{Html, Selector};

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;
    let document = Html::parse_document(&html_content);

    let product_container_selector = Selector::parse("article.product_pod").unwrap();
    let product_title_selector = Selector::parse("h3 a").unwrap();
    let product_price_selector = Selector::parse("p.price_color").unwrap();

    let mut products: Vec<Product> = Vec::new();

    for product_element in document.select(&product_container_selector) {
        let title_element = product_element.select(&product_title_selector).next();
        let price_element = product_element.select(&product_price_selector).next();

        if let (Some(title_elem), Some(price_elem)) = (title_element, price_element) {
            let price = price_elem.text().collect::<String>();
            let title = title_elem.value().attr("title").unwrap_or_default().to_string();
            let relative_url = title_elem.value().attr("href").unwrap_or_default();
            // This hardcoded base URL will be improved later when we handle pagination.
            let url = format!("http://books.toscrape.com/catalogue/{}", relative_url.trim_start_matches("../"));

            products.push(Product {
                title,
                price,
                url,
            });
        }
    }

    Ok(products)
}

#[tokio::main]
async fn main() {
    // highlight-start
    // Instead of a single URL, we now have a vector of URLs to process.
    // The `vec!` macro is a convenient way to create a new vector.
    // Each element is a `&'static str`, a reference to a string literal
    // that lives for the entire program's duration.
    let urls_to_scrape = vec![
        "http://books.toscrape.com/catalogue/category/books/travel_2/index.html",
        "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
        "http://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
    ];

    println!("Starting the concurrent scraper for {} pages...", urls_to_scrape.len());

    // The old logic for scraping a single page has been removed.
    // In the following tasks, we will build a loop here that spawns an
    // asynchronous task for each URL in our `urls_to_scrape` vector.
    // highlight-end
}

By making this change, you have fundamentally altered the program’s objective. It is no longer about a single task but about managing a collection of tasks. This vector is the foundation upon which we will build our concurrent execution logic.

What’s Next?

With our list of jobs (urls_to_scrape) defined, the next logical step is to prepare a place to keep track of all the concurrent tasks we’re about to launch. In the next task, you will create another empty vector that will be used to store the “handles” for each spawned asynchronous task, allowing us to wait for all of them to complete their work.

Prepare for Concurrent Scraping: Create a JoinHandle Vector

Mục tiêu: Initialize a mutable vector to store the JoinHandles from spawned Tokio tasks. This vector is essential for managing and awaiting the completion of all concurrent scraping operations.

Excellent work! In the previous task, you successfully set up the foundational “work queue” for our scraper by creating a Vec of URLs. This was a critical shift in perspective, moving from a single job to a list of jobs. Now, with our to-do list in hand, we need to prepare a way to manage and track the concurrent operations we are about to launch.

Managing Asynchronous Work: The `JoinHandle`

When we ask tokio to run a task concurrently, we use a function called tokio::spawn. This function takes an asynchronous block of code (like our scrape_page function call), sends it off to the tokio runtime to be executed, and immediately returns without waiting for the task to finish. This is the essence of “spawning” a task—it lets the main program continue its own work while the spawned task runs in the background.

But if it returns immediately, how do we know when the task is done? And how do we get the result (the Vec<Product>) back from it?

The answer is that tokio::spawn returns a special type called a tokio::task::JoinHandle. A JoinHandle is a smart object that acts as a controller for the spawned task. It’s a type of Future, meaning it’s a placeholder for a value that will be ready later. We can hold onto this handle and, at a later time, .await it. Awaiting a JoinHandle will pause the current function’s execution until the spawned task has completed, at which point it will resolve to the value that the task returned (in our case, the Result<Vec<Product>, ...>).

A Container for Our Handles

Since we plan to iterate through our urls_to_scrape vector and spawn a new task for each URL, we are going to get a JoinHandle for each one. We need a place to store all of these handles so we can wait for all of them to complete after we’ve spawned them all.

The perfect data structure for this is, once again, a Vec! We will create an empty, mutable vector that will hold all the JoinHandles from our spawned scraping tasks.

Let’s add this to our main function in src/main.rs.

// src/main.rs

// ... (scrape_page function and Product struct remain unchanged) ...

#[tokio::main]
async fn main() {
    let urls_to_scrape = vec![
        "http://books.toscrape.com/catalogue/category/books/travel_2/index.html",
        "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
        "http://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
    ];

    println!("Starting the concurrent scraper for {} pages...", urls_to_scrape.len());

    // highlight-start
    // Create a mutable vector to hold the handles for each spawned task.
    // The `let mut` keyword is crucial because we will be adding elements
    // to this vector inside our upcoming loop.
    // The `vec![]` macro creates a new, empty vector. Rust's powerful type
    // inference will figure out the exact type of the vector's elements
    // (the JoinHandles) once we start pushing them in the next task.
    let mut tasks = vec![];
    // highlight-end
}

By adding let mut tasks = vec![];, you have now set up the container that will be essential for managing the lifecycle of our concurrent operations.

let mut: The mut keyword signifies that the tasks vector is mutable. This is a requirement because we will need to modify it by calling .push() to add each new JoinHandle as we create it. In Rust, variables are immutable by default, which is a key safety feature, so we must be explicit when we need to change a value.
vec![]: This is a convenient macro for creating a new, empty vector.

You now have both the list of work to be done (urls_to_scrape) and a place to track the workers (tasks). The stage is perfectly set for the next step.

What’s Next?

With our “to-do” list and our “in-progress” tracker ready, the next task is to connect them. You will write a for loop that iterates over the urls_to_scrape vector. Inside that loop, you will call tokio::spawn for each URL, kicking off the scraping process, and push the resulting JoinHandle into your new tasks vector.

Iterate Over URLs with a `for` Loop in Rust

Mục tiêu: Add a for loop to iterate over a vector of URLs. This sets up the structure for spawning a concurrent, asynchronous scraping task for each URL in a subsequent step.

You’ve perfectly set the stage for concurrency. In the last two tasks, you defined the “work queue” by creating a vec! of URLs and prepared a container to track our asynchronous workers by initializing an empty tasks vector. You have the inputs and a place for the outputs; now, let’s build the bridge between them.

The next logical step is to process our list of URLs one by one, and for each one, perform the action of spawning a new scraping task. The most idiomatic and clearest way to process a collection of items in Rust is with a for loop.

Iterating to Delegate: The `for` Loop

A for loop in Rust is a powerful construct for consuming an iterator. When you write a for loop over a vector (or many other collection types), Rust automatically creates an iterator for that collection. The loop then calls .next() on that iterator, assigning the resulting item to a variable, and executes the loop body. This repeats until the iterator returns None, signaling that the sequence is finished.

This approach is not only clean and readable but also safe. It completely avoids the possibility of common “off-by-one” errors that can happen with manual index-based loops in other languages.

In our specific case, the line for url in urls_to_scrape will iterate over our vector. By default, iterating over a Vec<T> yields references to its items, so on each pass of the loop, the url variable will be of type &&'static str. This might look a bit strange, but it’s exactly what we want! Our scrape_page function expects a &str, and Rust’s automatic dereferencing (Deref coercion) will seamlessly convert our &&'static str into the &str the function needs. This is a perfect example of Rust’s efficiency—we are simply passing around references (pointers) to the string data that’s already in our program’s binary, not making any expensive copies.

Let’s add the loop structure to our main function. For this task, we will only add the loop itself; the body where we spawn the task will be filled in the very next step.

Here are the changes for src/main.rs:

#[tokio::main]
async fn main() {
    let urls_to_scrape = vec![
        "http://books.toscrape.com/catalogue/category/books/travel_2/index.html",
        "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
        "http://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
    ];

    println!("Starting the concurrent scraper for {} pages...", urls_to_scrape.len());

    let mut tasks = vec![];

    // highlight-start
    // Now, we'll iterate over our list of URLs.
    // The `for url in urls_to_scrape` syntax creates a loop that will execute
    // its body once for each item in the `urls_to_scrape` vector.
    // On each iteration, the `url` variable will hold a reference to one of the
    // string literals from the vector (e.g., "http://...").
    for url in urls_to_scrape {
        // Inside this loop is where the magic of concurrency will happen.
        // For each URL, we will spawn a new asynchronous task that will call our
        // `scrape_page` function. We'll add the `tokio::spawn` call in the
        // very next task.
    }
    // highlight-end
}

You have now built the scaffolding for concurrent execution. If you run the program now with cargo run, it will simply print the “Starting…” message and exit. This is correct! We’ve set up the loop, but since its body is empty, it does nothing on each iteration. This is a crucial structural step. We have clearly separated the logic of “what work needs to be done” (the list of URLs) from the mechanism that will “start the work” (the loop).

What’s Next?

With the loop in place, the next, most exciting task is to fill its body. You will use tokio::spawn to take the url from each iteration and launch a new, independent, asynchronous scraping task. This is where you will truly bring your concurrent scraper to life.

Launch Concurrent Scraping Tasks with `tokio::spawn`

Mục tiêu: Within a loop, use tokio::spawn to launch a new asynchronous task for each URL. Address Rust’s ownership requirements by creating an owned String from the URL and using an async move block to transfer its ownership into the spawned task.

You’ve brilliantly constructed the loop that will serve as the launchpad for all our concurrent operations. You have the urls_to_scrape vector (the mission manifest) and the tasks vector (the mission control tracker). Now, it’s time to light the fuse and launch our rockets. Inside this loop, for each URL, we will command the tokio runtime to start a new, independent scraping task.

Launching a Concurrent Task with `tokio::spawn`

The core function for achieving concurrency in Tokio is tokio::spawn. This function is the heart of our high-performance design. When you call tokio::spawn, you are telling the Tokio runtime: “Here is a piece of asynchronous work. Please start running it for me in the background, on your pool of threads, whenever you have capacity. Don’t make me wait for it; just give me a handle so I can check on it later.”

This is a fundamental shift from the sequential code we’ve written before. Instead of calling scrape_page and waiting for it to finish before moving to the next URL, our for loop will now zip through all the URLs almost instantly, launching a new background task for each one. All these scraping tasks will then effectively run at the same time, making network requests and parsing HTML in parallel.

The Critical Rule of Concurrency: Ownership

Before we can spawn a task, we must satisfy Rust’s most important safety rule: ownership. The tokio::spawn function has a critical requirement: the task (or Future) you give it must have a 'static lifetime.

'static Lifetime: This is a special lifetime that means “this thing is valid for the entire duration of the program.” It’s a safety guarantee. The Tokio runtime cannot know how long your for loop or even your main function will last. A spawned task might end up running long after the loop that created it has finished. Therefore, the task cannot borrow any data that might disappear. It must own all the data it needs to do its job.

Our url variable inside the loop is a &str—a borrowed reference to a string literal. We cannot give this borrowed data to a new task because the task might outlive the reference.

The solution involves two steps:

Create Owned Data: We first convert the borrowed &str into an owned String using url.to_string(). This creates a new piece of data on the heap that is completely independent of our for loop.
Move Ownership into the Task: We use an async move block. This special syntax creates a Future and the move keyword forces this future to take ownership of any variables it uses from its environment. This is how we transfer ownership of our new String into the spawned task, satisfying the 'static lifetime requirement.

Let’s implement this inside the for loop in src/main.rs.

#[tokio::main]
async fn main() {
    let urls_to_scrape = vec![
        "http://books.toscrape.com/catalogue/category/books/travel_2/index.html",
        "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
        "http://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
    ];

    println!("Starting the concurrent scraper for {} pages...", urls_to_scrape.len());

    let mut tasks = vec![];

    for url in urls_to_scrape {
        // highlight-start
        // 1. Create an owned copy of the URL.
        // The `url` from the loop is a `&'static str`. Spawning a task requires
        // that the task owns all of its data. By converting the slice to a `String`,
        // we create data that can be moved.
        let url_owned = url.to_string();

        // 2. Spawn the task.
        // `tokio::spawn` takes a future and executes it on the Tokio runtime.
        // It immediately returns a `JoinHandle` without waiting for the future to complete.
        let task = tokio::spawn(async move {
            // The `async move` block creates a future that takes ownership of its
            // environment (in this case, `url_owned`). This is crucial.
            println!("Spawning task for URL: {}", url_owned);
            // Now, we call our scraper function with a reference to the owned String.
            scrape_page(&url_owned).await
        });
        // We will push this `task` handle into our `tasks` vector in the next step.
        // highlight-end
    }
}

What You’ve Just Built

With these changes, your loop now does something incredible. For each URL:

It creates a new, independent String.
It creates a new asynchronous task that takes ownership of that string.
It hands this task off to the tokio runtime to be scheduled and executed.
It receives a JoinHandle (stored in the task variable) in return. This JoinHandle is our link to the running task, which we can use later to get its result.

If you run cargo run now, you might see the “Spawning task for URL…” messages appear in a different order each time, or all at once! This is a clear sign that the tasks are being scheduled concurrently by the runtime.

What’s Next?

You are now successfully launching concurrent tasks, but you’re not yet tracking them. The task variable, which holds the JoinHandle, is created and then immediately discarded at the end of each loop iteration. In the next task, you will take this handle and push it into the tasks vector you created earlier. This will allow you to keep track of all the running scrapers so you can wait for them to finish and collect their results.

Track Concurrent Tasks by Storing JoinHandles

Mục tiêu: Complete the concurrent task launching loop by storing the JoinHandle returned from tokio::spawn into a vector. This enables the program to track all running asynchronous tasks before waiting for their completion.

You have reached the final piece of the “launch” phase. In the previous task, you successfully wrote the code to spawn a new, independent scraping task for each URL in your list. The tokio::spawn function handed you back a JoinHandle, a crucial object representing that running task. However, this handle was being created and then immediately discarded as the loop moved to the next iteration.

Now, we must complete the loop by collecting these handles. This is the moment we connect our “mission control” tracker (tasks vector) with the rockets we’ve just launched.

Tracking Your Workers: Storing the `JoinHandle`

Think of the JoinHandle as a tracking number for a package. Just sending a package isn’t enough; you need the tracking number to know when it has been delivered and to confirm its contents. In the world of asynchronous programming, simply spawning a task doesn’t guarantee completion. If our main function were to finish before our spawned tasks, the program would exit, and our scraping work would be terminated prematurely.

By storing each JoinHandle, we are creating a manifest of all the work that is currently in progress. Later, we can go through this manifest and wait for each task to report back that it’s finished. This ensures that our program waits for all scraping to be completed before it moves on or exits.

To add the JoinHandle to our tasks vector, we will use the standard .push() method. This method appends a new element to the end of a vector. This is precisely why we declared let mut tasks—the mut keyword gives us permission to modify the vector by adding elements to it.

Let’s add the final line inside our for loop in src/main.rs. This small addition is the linchpin that holds our entire concurrent strategy together.

// In src/main.rs

// ... (Product struct and scrape_page function are unchanged) ...

#[tokio::main]
async fn main() {
    let urls_to_scrape = vec![
        "http://books.toscrape.com/catalogue/category/books/travel_2/index.html",
        "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
        "http://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
    ];

    println!("Starting the concurrent scraper for {} pages...", urls_to_scrape.len());

    let mut tasks = vec![];

    for url in urls_to_scrape {
        let url_owned = url.to_string();

        let task = tokio::spawn(async move {
            println!("Spawning task for URL: {}", url_owned);
            scrape_page(&url_owned).await
        });

        // highlight-start
        // This is the crucial step: we take the `JoinHandle` returned by `tokio::spawn`
        // and add it to our `tasks` vector.
        // The `push` method moves ownership of the `task` handle into the vector.
        // After this loop finishes, our `tasks` vector will contain one handle
        // for each URL we are scraping.
        tasks.push(task);
        // highlight-end
    }
}

What Has Happened?

Let’s pause and appreciate what you’ve just built. When you run this code, the for loop executes with incredible speed. It doesn’t wait for any scraping to happen. It simply:

Takes a URL.
Spawns a background scraping task for it.
Receives a JoinHandle.
Pushes that handle into the tasks vector.
Repeats for the next URL.

In a fraction of a second, the loop will be finished. At this point, all your scraping tasks are running concurrently in the background, managed by the Tokio runtime. Your main function is now paused just after the loop, and the tasks vector is fully populated, holding a JoinHandle for every running job. You have successfully launched and are now tracking a fleet of concurrent workers!

What’s Next?

You’ve completed the “launch” phase. The next and final task in this step is the “rendezvous” phase. Now that you have a vector full of JoinHandles, you will iterate through this vector and .await each handle. This will effectively pause the main function until every single scraping task has completed its work, allowing you to safely collect all the results.

Aggregate Results from Concurrent Tokio Tasks

Mục tiêu: Wait for all spawned Tokio tasks to complete by iterating through their JoinHandles and using .await. Implement robust error handling for the nested Result to distinguish between task panics and application errors, and collect all successful results into a single list.

You have masterfully orchestrated the launch of your concurrent scrapers. In the previous tasks, you created a “mission manifest” of URLs and then looped through it, spawning a new, independent tokio task for each one. Your tasks vector is now fully populated, acting as a mission control board, with each JoinHandle representing a scraper that is out on the web, working in parallel.

The for loop that spawned these tasks finished almost instantly. But the work is still happening in the background. If our main function were to end now, the Tokio runtime would shut down, terminating all our scrapers mid-flight. Our current task is to perform the crucial “rendezvous” step: we must explicitly wait for every single task to complete its mission and report back with its findings.

The Rendezvous: Awaiting the `JoinHandle`

A JoinHandle is a type of Future. This means we can .await it. When we .await a JoinHandle, we are telling our main function to pause its own execution at that point and wait until the specific background task associated with that handle has finished running. Once the task is complete, .await will resolve to the value that the task returned.

This is how we “join” our main program flow back with our concurrent workers, ensuring all work is completed before the program exits. We will iterate through our tasks vector and .await each handle one by one.

Handling Two Layers of Potential Failure: The Nested `Result`

This is where error handling in a concurrent context gets interesting. When you .await a JoinHandle, the value you get back is a Result itself. Let’s look at the types:

Our scrape_page function returns: Result<Vec<Product>, Box<dyn std::error::Error>>. This is our Application-Level Result. It tells us if the scraping logic succeeded or failed (e.g., a network error).
Awaiting the JoinHandle returns: Result<T, tokio::task::JoinError>, where T is the value returned by the task. This is our Task-Level Result. It tells us if the task itself completed normally. A JoinError occurs if the task panics (a crash) or is cancelled.

When we combine these, the final type we get from task.await is a nested Result: Result<Result<Vec<Product>, Box<dyn std::error::Error>>, tokio::task::JoinError>

This might look intimidating, but it gives us precise control. We can now distinguish between a scraper failing because the website was down (Ok(Err(...))) versus the scraper failing because of a bug in our code that caused a panic (Err(...)).

We will use a for loop and a match statement to handle all these cases gracefully, collecting all the successful results into a final, master list of products.

Let’s add the final logic to your main function in src/main.rs.

#[tokio::main]
async fn main() {
    let urls_to_scrape = vec![
        "http://books.tscrape.com/catalogue/category/books/travel_2/index.html",
        "http://books.tscrape.com/catalogue/category/books/mystery_3/index.html",
        "http://books.tscrape.com/catalogue/category/books/historical-fiction_4/index.html",
    ];

    println!("Starting the concurrent scraper for {} pages...", urls_to_scrape.len());

    let mut tasks = vec![];

    for url in urls_to_scrape {
        let url_owned = url.to_string();
        let task = tokio::spawn(async move {
            println!("Spawning task for URL: {}", url_owned);
            scrape_page(&url_owned).await
        });
        tasks.push(task);
    }

    // This is the new part of our main function.
    // We create a new vector to store all products from all scraped pages.
    let mut all_products: Vec<Product> = Vec::new();

    // Now, we iterate through our vector of JoinHandles.
    for task in tasks {
        // We `.await` each handle. This pauses the loop until the task is complete.
        // `task.await` returns a `Result` to handle potential panics in the task.
        match task.await {
            Ok(result) => {
                // The task completed without panicking. `result` is the
                // `Result<Vec<Product>, ...>` returned by our `scrape_page` function.
                match result {
                    Ok(products) => {
                        // The scraping was successful. We extend our master list.
                        println!("Scraped {} products from a page.", products.len());
                        all_products.extend(products);
                    }
                    Err(e) => {
                        // The scraping logic failed for this page.
                        eprintln!("A scraping task failed with error: {}", e);
                    }
                }
            }
            Err(e) => {
                // The task itself failed to execute (e.g., it panicked).
                eprintln!("A spawned task failed to complete: {}", e);
            }
        }
    }

    // After all tasks are awaited, we can print the final results.
    println!("\n--- All scraping tasks complete! ---");
    println!("Total products scraped: {}", all_products.len());
    println!("--- Sample of all scraped products: ---");

    // Print the first 5 products for a clean preview.
    for product in all_products.iter().take(5) {
        println!("{:#?}", product);
    }
}

When you run your program now with cargo run, you will see the “Spawning…” messages, followed by the “Scraped…” messages as each task completes. Finally, you’ll see a grand total of all products scraped from all three pages combined. You have successfully built a truly concurrent web scraper!

What’s Next?

This pattern of spawning tasks, collecting handles, and then awaiting them in a separate loop is a common and valid approach called “structured concurrency.” However, as you can see, the nested Result handling can be a bit verbose. Furthermore, this method requires us to wait for all tasks to be launched before we can start processing any results.

In the next step, we will explore a more flexible and powerful pattern for communication between concurrent tasks: channels. Specifically, a multi-producer, single-consumer (mpsc) channel will allow our scraping tasks to send their results back to the main function as soon as they are ready, creating a real-time stream of data.