Define an Asynchronous Web Scraping Function in Rust

Mục tiêu: Define the signature for an asynchronous web scraping function, scrape\_page, in Rust. This involves setting up the tokio async runtime in the main function and constructing a robust function signature that uses Result and Box for idiomatic error handling.

Having meticulously crafted the Product struct, you’ve created the perfect data container for our scraped information. It’s the “what” of our project. Now, we’ll build the “how”: the function that will perform the actual scraping and produce instances of this struct. This is where we bring together the powerful asynchronous libraries we added earlier and write the core logic of our application.

Our first task is to define the function’s signature. This signature acts as a contract, clearly stating what the function needs to do its job (its inputs) and what it promises to give back (its output).

Introducing Asynchronous Functions in Rust

We will define this as an async fn. The async keyword is the foundation of Rust’s modern concurrency story. It transforms a regular function into a coroutine, a special kind of function that can be paused in the middle of its execution without blocking the entire program.

When our scrape_page function sends a network request, it will await the response. Instead of freezing the program, the async runtime (our tokio crate) will pause scrape_page and switch to another task that’s ready to do work. Once the network response arrives, the runtime will wake our function up and let it continue from where it left off. This is the key to handling thousands of requests concurrently and achieving high performance.

Modifying `main` to Power Our Async World

Because we are now introducing async functions, our main entry point also needs to be aware of the asynchronous runtime. The tokio crate provides a convenient attribute macro, #[tokio::main], to handle this. It transforms our main function into an entry point that starts the Tokio runtime and executes the async code within it.

We need to make this change first. Update your src/main.rs to modify the main function.

// highlight-start
#[tokio::main]
async fn main() {
// highlight-end
    println!("Hello, world!");
}

Now, let’s add the skeleton of our new scrape_page function.

// In src/main.rs

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

// highlight-start
/// An asynchronous function to scrape a single web page.
///
/// # Arguments
///
/// * `url` - A string slice that holds the URL of the page to scrape.
///
/// # Returns
///
/// A `Result` which is:
/// - `Ok(Vec<Product>)`: A vector of `Product` structs if scraping is successful.
/// - `Err(Box<dyn std::error::Error>)`: A boxed, dynamic error if anything fails.
async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    // We will implement the scraping logic in the upcoming tasks.
    // For now, we'll return an empty vector to satisfy the compiler.
    Ok(vec![])
}
// highlight-end

#[tokio::main]
async fn main() {
    println!("Hello, world!");
}

Deconstructing the Function Signature

Let’s break down this new function signature piece by piece, as every part is significant:

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>>

async fn scrape_page: We’ve declared an asynchronous function named scrape_page.
(url: &str): This defines the function’s single parameter.
- url: The name of the parameter.
- &str: The type is a “string slice”. This is a borrowed view of a string. We use &str instead of String because the function only needs to read the URL. This is more efficient as it avoids taking ownership and allocating new memory for the URL string.
->: This arrow indicates that the function returns a value.
Result<..., ...>: This is Rust’s standard, built-in type for handling operations that can either succeed or fail. It is an enum with two variants:
- Ok(T): Contains the value of type T from a successful operation.
- Err(E): Contains an error of type E from a failed operation. Using Result is the idiomatic way to handle errors in Rust, forcing us to deal with potential failures explicitly and making our code robust.
Vec<Product>: This is our success type (T). Vec stands for “vector,” which is Rust’s growable list type. If our function succeeds, it will return a vector containing all the Product structs it managed to scrape from the page.
Box<dyn std::error::Error>: This is our error type (E) and is a very common pattern in Rust applications. Let’s break it down further:
- std::error::Error: This is a trait—a contract defining shared behavior—that all standard error types in Rust implement. It represents the concept of “being an error.”
- dyn: The dyn keyword indicates we are using a trait object. This means we aren’t returning a specific error type (like reqwest::Error or csv::Error), but rather any type that implements the Error trait. This gives us immense flexibility, as our function might fail due to a network issue, an HTML parsing issue, or something else. We can return any of these different error types without changing our function’s signature.
- Box<...>: A Box is a smart pointer that allocates memory on the heap. We need it because the compiler can’t know the exact size of a dyn Error at compile time (it could be any type of error). The Box provides a pointer with a known size that points to the actual error data on the heap.

You have now defined a robust and idiomatic function skeleton. This signature sets a clear contract for the work ahead and provides a solid foundation for error handling and asynchronous execution.

What’s Next?

With the function’s framework in place, our next task is to begin filling in its logic. We will start by using the reqwest crate to make an actual HTTP request to the given URL and fetch the page’s HTML content.

Make an Asynchronous Web Request with Reqwest

Mục tiêu: Implement an asynchronous HTTP GET request using the reqwest crate inside an async function. Utilize .await to handle the Future and the ? operator for robust error propagation.

With the robust signature for your scrape_page function now in place, you’ve created a solid contract for the work ahead. That signature is like the blueprint for a machine; now it’s time to build the first gear. The very first action any web scraper must take is to reach out across the internet and request the content of a web page. For this, we will use the powerful reqwest crate you added earlier.

The First Step: Making the Request

Our goal is to take the url string slice passed into our function and use it to perform an HTTP GET request. This is the standard method web browsers use to fetch the contents of a webpage. The reqwest crate makes this incredibly straightforward with its get() function.

However, since we are in an async context, this operation isn’t instantaneous. Requesting data over a network takes time—milliseconds or even seconds—and this is precisely where the power of async/await shines.

Understanding `async`, `.await`, and Futures

When you call an async function like reqwest::get(), it doesn’t block your program while it waits for the server to respond. Instead, it immediately returns a special type called a Future.

A Future is a placeholder, a promise that a value will be available in the future. In our case, the reqwest::get(url) call returns a Future that will eventually resolve to a Result containing the server’s response.

To get the actual value out of the Future, we use the .await keyword. When the tokio runtime sees .await, it does something remarkable: if the value isn’t ready yet (i.e., we’re still waiting for the network), it will pause the execution of our scrape_page function and switch to running other tasks. Once the server’s response arrives, the runtime will “wake up” our function and resume it exactly where it left off. This mechanism is what allows our application to handle thousands of concurrent operations efficiently without getting stuck.

Robust Error Handling with the `?` Operator

Network requests can fail for many reasons: the URL might be invalid, the server could be down, or there might be a DNS resolution error. reqwest::get(url).await doesn’t return a reqwest::Response directly; it returns a Result<reqwest::Response, reqwest::Error>.

We need to handle the Err case. The most idiomatic and cleanest way to do this in Rust is with the question mark (?) operator.

The ? operator is syntactic sugar for error propagation. When you place it at the end of an expression that returns a Result, it does the following:

If the Result is Ok(value), it unwraps it and gives you the value (in our case, the reqwest::Response).
If the Result is Err(error), it immediately stops the execution of the current function (scrape_page) and returns the error to the function’s caller.

This is precisely why we designed our function to return a Result<..., Box<dyn std::error::Error>>. The ? operator seamlessly propagates any potential network error, and Rust’s type system automatically converts the specific reqwest::Error into our general Box<dyn std::error::Error> type.

Implementing the Request

Let’s add this logic to our scrape_page function in src/main.rs. We’ll call reqwest::get, .await its result, and use ? to handle any potential errors.

// src/main.rs

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    // Make an asynchronous GET request to the given URL.
    // The `.await` keyword pauses the function execution until the request is complete.
    // The `?` operator handles any potential errors (like network issues),
    // propagating them up to the caller.
    let response = reqwest::get(url).await?;

    // The response variable now holds a `reqwest::Response` struct.
    // In the next step, we will process this response to get the HTML body.

    Ok(vec![])
}

#[tokio::main]
async fn main() {
    // Let's create a hardcoded URL to test our function.
    // This is a sandbox website designed for scraping practice.
    let url_to_scrape = "http://books.toscrape.com/";

    // Call our scraping function.
    // We use a `match` block to handle the Result returned by the function.
    match scrape_page(url_to_scrape).await {
        Ok(products) => {
            // If everything went well, for now, we'll just print a success message.
            // Later, we will process the `products` vector.
            println!("Successfully fetched the page!");
        }
        Err(e) => {
            // If an error occurred, print it to the console.
            eprintln!("Error scraping page: {}", e);
        }
    }
}

Code Breakdown:

In scrape_page:
- let response = reqwest::get(url).await?;: This new line is the core of our task. It performs the request and assigns the successful reqwest::Response to the response variable. If it fails, the function returns early with an error.
In main:
- We’ve added a call to scrape_page to test our new logic. You can now run your program with cargo run.
- If your computer is connected to the internet, you should see the message: Successfully fetched the page!.
- Try changing the URL to something invalid, like "http://thissitedoesnotexist.invalid/", and run it again. You’ll see a descriptive error message printed, demonstrating our robust error handling in action!

You have now successfully implemented the first, most fundamental step of any web scraper. Your program can reach out to a server, request a page, and handle success and failure gracefully.

What’s Next?

The response variable we now have is a reqwest::Response struct. This struct contains all the information about the server’s response, including the status code, headers, and, most importantly, the response body (the HTML content). In the next task, we will extract this HTML body as a text string so we can prepare it for parsing.

Extract HTML Content from a Reqwest HTTP Response in Rust

Mục tiêu: Learn to extract the HTML body from a reqwest::Response object by using the asynchronous .text() method. This task involves awaiting the future to get the response body as a Rust String and handling potential errors.

In the last task, you successfully wrote the code to send an HTTP request and receive a response from the server. This was a monumental step! You now have a reqwest::Response object stored in your response variable. This object is a complete representation of the server’s reply, containing not just the page content but also status codes (like 200 OK or 404 Not Found), headers, and other metadata.

Our goal, however, is to get to the juicy center: the raw HTML content of the page. The Response object holds this content in its “body,” and our current task is to extract this body and turn it into a Rust String that we can work with.

From Response Body to Usable Text

The reqwest crate provides a convenient, asynchronous method for this exact purpose: .text(). This method is called on the Response object and handles several important details for us:

Consuming the Body: It reads the entire response body from the network stream. This body might be large and arrive in multiple packets; .text() handles this streaming process for us.
Decoding: Web content is just a sequence of bytes. To become a human-readable String, these bytes must be decoded using a character encoding. The .text() method intelligently inspects the Content-Type header of the response (e.g., Content-Type: text/html; charset=utf-8) to determine the correct encoding. If it can’t find one, it defaults to UTF-8, the standard encoding for the web.
Asynchronous Operation: Just like fetching the initial response, reading the full response body can take time. Therefore, .text() is also an async operation. It immediately returns a Future that will resolve to a Result<String, reqwest::Error> once the entire body has been downloaded and decoded.

Using `.await` and `?` for a Clean Workflow

Since .text() returns a Future, we must use .await to pause our function and wait for the result. And because this operation can fail (e.g., a network connection drops mid-download, or the content is not valid UTF-8), we again use the ? operator to gracefully handle any potential errors. It will either unwrap the Ok(String) to give us our HTML content or propagate the Err up to the caller of scrape_page.

Let’s update our scrape_page function to extract the HTML. We’ll also add a temporary println! statement to display the first 500 characters of the fetched HTML. This is a great way to verify that our code is working correctly and see the data we’ll be parsing in the next step.

Here are the changes for src/main.rs:

// src/main.rs

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;

    // The .text() method consumes the response body and returns it as a String.
    // This is also an async operation, so we await it.
    // The `?` will propagate any errors that occur during reading or decoding the body.
    let html_content = response.text().await?;

    // For debugging, let's print the first 500 characters of the HTML.
    // This confirms we've successfully downloaded the page content.
    // We use a slice `&html_content[..500]` to avoid printing the whole page.
    println!("Page content received (first 500 chars):\n{}", &html_content[..500]);

    Ok(vec![])
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    match scrape_page(url_to_scrape).await {
        // We use `_products` because we are not yet using the returned vector.
        // The underscore prefix tells the Rust compiler that this is intentional,
        // preventing an "unused variable" warning.
        Ok(_products) => {
            println!("\nSuccessfully fetched and read page body!");
        }
        Err(e) => {
            eprintln!("Error scraping page: {}", e);
        }
    }
}

Now, run your application with cargo run. You should see the first 500 characters of the HTML source code of the “Books to Scrape” website printed in your terminal, followed by the success message. You have successfully bridged the gap from a network response to a usable String in your program!

What’s Next?

You now have the raw material—a String containing the entire HTML of the page. While this is a huge step, it’s still just a sea of text. The next task is to bring in the scraper crate to parse this string into a structured Document Object Model (DOM), which will allow us to navigate the HTML tree and precisely select the elements we need.

Parse HTML Content into a DOM with `scraper`

Mục tiêu: Use the scraper::Html::parse\_document function to convert a raw HTML string into a navigable DOM tree, preparing the web page’s content for data extraction.

You’ve made fantastic progress! In the previous task, you successfully reached out to the internet with reqwest, downloaded the content of a web page, and stored it as a raw String in the html_content variable. This is a crucial milestone. However, in its current form, this string is just a massive, unstructured wall of text. To a program, it’s a meaningless sequence of characters like <div class="product_pod">....

To extract meaningful information, we must first give this text some structure. We need to convert it from a flat string into a hierarchical, tree-like data structure that our program can understand and navigate. This process is called parsing, and the resulting structure is known as the Document Object Model (DOM).

From Unstructured Text to a Navigable Tree: The DOM

Imagine you have a book written as one continuous paragraph. Finding the third sentence of the second paragraph in Chapter 5 would be impossible. But when the book is properly formatted with chapters, paragraphs, and sentences, the task becomes trivial. The DOM does the same for an HTML document. It represents the HTML as a tree of nodes, where each HTML tag (like <div>, <h1>, or <p>) is a branch, and the text inside them are the leaves.

Our tool for building this DOM tree is the scraper crate. It’s built on top of Firefox’s robust html5ever engine, meaning it’s designed to handle the messy, often imperfect HTML found on real-world websites.

Using `scraper::Html` to Parse the Document

The central type in the scraper crate for this task is scraper::Html. It provides a static method called parse_document(), which takes a string slice (&str) of HTML content and returns a parsed Html document object. This object is the root of our DOM tree, and it will be our entry point for searching and selecting specific elements on the page.

Let’s update our scrape_page function to perform this parsing step. We will take the html_content string from the previous step and pass it to scraper::Html::parse_document().

Here are the changes for src/main.rs. Notice how we build directly on the previous step’s result.

// src/main.rs

// We need to bring the `Html` type from the `scraper` crate into scope.
use scraper::Html;

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;

    // The `scraper::Html::parse_document` function takes the HTML string
    // and converts it into a structured, traversable document object.
    // We pass a reference `&html_content` because the parser only needs to
    // read the content, not take ownership of it, which is more efficient.
    let document = Html::parse_document(&html_content);

    // The `document` variable now holds the entire parsed HTML page.
    // In the next step, we'll learn how to query it using CSS selectors.

    // Let's remove the old `println!` and add a new one to confirm parsing.
    println!("Successfully parsed the HTML document.");

    Ok(vec![])
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    match scrape_page(url_to_scrape).await {
        Ok(_products) => {
            println!("\nScraping task completed successfully!");
        }
        Err(e) => {
            eprintln!("Error during scraping: {}", e);
        }
    }
}

Code Breakdown:

use scraper::Html;: At the top of the file, we add a use statement. This brings the Html type from the scraper crate into our current namespace, allowing us to refer to it directly as Html instead of the more verbose scraper::Html. This is a standard practice for improving code readability.
let document = Html::parse_document(&html_content);: This is the core of our current task. We call the parse_document function on the Html type, passing it a reference to our html_content string. The result, a fully parsed and queryable document, is stored in a new variable.

With this single line of code, you have transformed the raw HTML from a simple string into a rich, structured format. You can now run your program again with cargo run. The output should show the new confirmation message, indicating that the parsing was successful.

What’s Next?

You now have a document object that represents the entire web page. The next logical and exciting step is to start querying this document to find the specific data we’re looking for. We will learn how to define CSS selectors—powerful patterns that allow us to precisely target the HTML elements containing the product information.

Define CSS Selectors for Web Scraping in Rust

Mục tiêu: Learn how to use browser developer tools to find the correct CSS selectors for target data (like product titles and prices) and then define them in a Rust application using the scraper crate to prepare for data extraction from an HTML document.

You have successfully transformed the raw HTML string into a structured, navigable document object. This is a massive step forward! The document variable now holds the entire web page as a tree, but how do we tell our program which branches and leaves contain the information we want? The answer lies in using a powerful query language: CSS Selectors.

The “Address System” of a Web Page: CSS Selectors

Imagine you have a map of a city (the DOM), and you need to find all the post offices. You wouldn’t look at every single building. Instead, you’d look for buildings with specific signs or features. CSS Selectors are the “signs” we use to pinpoint specific elements on our HTML map.

Originally created to apply styles (like color and font size) to web pages, CSS Selectors have become an indispensable tool for web scraping due to their expressive and precise nature. They are patterns that match against elements in the DOM tree.

Examples include: * h1: Selects all <h1> elements. * .product: Selects all elements with the class product. * #main-content: Selects the single element with the ID main-content. * div p: Selects all <p> elements that are inside a <div>. * a[href]: Selects all <a> elements that have an href attribute.

Our task now is to find the correct “addresses” for the product information on our target website.

How to Find the Right Selectors: Using Browser Developer Tools

This is a hands-on skill every scraper developer must learn. We will use your web browser’s built-in developer tools to inspect the live website and discover the HTML structure.

Open the Target Page: In your web browser (like Chrome or Firefox), navigate to our scraping target: http://books.toscrape.com/.
Inspect an Element: Right-click on the title of any book on the page and choose “Inspect” or “Inspect Element” from the context menu.
Analyze the HTML: A new panel will open, showing you the page’s HTML source code, with the element you clicked on highlighted. Now, let’s find our selectors:
- Product Container: Look at the highlighted <a> tag for the title. Move your mouse up in the HTML panel. You’ll see that the entire book’s card (image, title, price, etc.) is wrapped inside an <article> tag with the class product_pod. This is our container! Every product on the page shares this structure.
  - Selector: article.product_pod
- Product Title & URL: Inside the <article class="product_pod">, you’ll find an <h3> tag which contains an <a> tag. This <a> tag holds two crucial pieces of information: its text is the book’s title, and its href attribute is the URL to the product page.
  - Selector: h3 a
- Product Price: Look just below the <h3> tag. You’ll find a <div> with the class product_price, which contains a <p> tag with the class price_color. The text inside this <p> tag is the price.
  - Selector: p.price_color

We now have the three patterns we need to precisely locate all the data on the page.

Defining Selectors in Your Rust Code

Now, let’s translate these patterns into our Rust code. The scraper crate provides a Selector type for this. We create an instance of Selector by parsing our selector string.

The function Selector::parse() is failable—it returns a Result because you could provide an invalid selector string (e.g., "!!invalid??"). For now, to keep our focus on the main path, we will use .unwrap(). This will get the Selector out of the Ok variant but will cause the program to panic if parsing fails. This is fine for our current stage, where we know our selectors are valid, but in a production application, you would handle this Result more gracefully.

Here are the changes for src/main.rs:

// src/main.rs

// Bring the `Selector` type into scope from the `scraper` crate.
use scraper::{Html, Selector};

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;
    let document = Html::parse_document(&html_content);

    // Define the CSS selectors for the data we want to extract.
    // `Selector::parse` returns a `Result`, so we use `.unwrap()` to get the `Selector`
    // or panic if the selector is invalid. This is acceptable for our known, valid selectors.

    // Selects the container for each product.
    let product_container_selector = Selector::parse("article.product_pod").unwrap();

    // Selects the title and URL from the `a` tag within the `h3` tag.
    let product_title_selector = Selector::parse("h3 a").unwrap();

    // Selects the price from the `p` tag with the class `price_color`.
    let product_price_selector = Selector::parse("p.price_color").unwrap();

    // The selector variables now hold compiled, efficient representations of our CSS patterns.
    // In the next step, we'll use these to query the `document`.

    Ok(vec![])
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    match scrape_page(url_to_scrape).await {
        Ok(_products) => {
            println!("\nScraping task completed successfully!");
        }
        Err(e) => {
            eprintln!("\nError during scraping: {}", e);
        }
    }
}

You have now armed your scraper with the precise knowledge of where to find the data. These Selector objects are compiled and highly optimized for efficient searching within the DOM.

What’s Next?

With our document parsed and our Selectors defined, the next logical step is to put them together. You will now use the document.select() method with your product_container_selector to get an iterator over all the product elements on the page, preparing us to loop through them and extract the final data points.

Find All Product Containers Using a CSS Selector

Mục tiêu: Learn to use the document.select() method with a CSS selector to find all matching elements on a parsed HTML page. This task explains how select() returns a lazy iterator and demonstrates how to count the found elements to verify your selector.

You have successfully defined the “addresses” for the data you want to scrape by creating your Selector objects. You have the parsed document (the map) and the selectors (the addresses). Now it’s time to put them together and actually find the elements on the map.

Querying the Document with `select()`

The scraper crate makes this process incredibly intuitive. The Html document object has a method called select(). This method is the workhorse of our parsing logic. You provide it with a reference to a Selector, and it searches the entire DOM tree, returning all the elements that match the selector’s pattern.

The select() method is highly efficient, but it’s important to understand what it returns. It doesn’t give you a Vec (a list) of all the found elements right away. Instead, it returns an iterator.

The Power of Iterators in Rust

An iterator is a fundamental concept in Rust that represents a sequence of items. The key feature of an iterator is that it is lazy. This means it doesn’t compute or collect all the items in the sequence at once. It produces them one by one, only when you ask for the next one. This is extremely memory-efficient, especially when dealing with potentially thousands of matching elements on a very large web page.

In our case, document.select(&product_container_selector) will return an iterator that, when we loop over it, will yield each product’s <article class="product_pod"> element one at a time. The type of item yielded by this iterator is an ElementRef. An ElementRef is essentially a lightweight reference or a “view” into a specific element within the parsed document. It allows us to inspect an element’s attributes, text content, and even search for child elements within it.

Finding All Product Containers

Let’s update our scrape_page function to use the product_container_selector to find all the product blocks on the page. We will also add a println! statement to count how many containers we found. This is an excellent way to verify that our selector is correct and that our code is working as expected. For the books.toscrape.com homepage, we expect to find 20 products.

Here are the changes for src/main.rs:

// src/main.rs

use scraper::{Html, Selector};

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;
    let document = Html::parse_document(&html_content);

    let product_container_selector = Selector::parse("article.product_pod").unwrap();
    let product_title_selector = Selector::parse("h3 a").unwrap();
    let product_price_selector = Selector::p.price_color").unwrap();

    // Use the `select` method on the `document` with our product container selector.
    // We pass a reference `&product_container_selector` because the method only
    // needs to borrow the selector to perform its search.
    // This returns a lazy iterator over all elements that match the selector.
    let product_containers = document.select(&product_container_selector);

    // For verification, let's count how many product containers we found.
    // The `.count()` method consumes the iterator, so if we want to use the iterator
    // again later (which we do!), we must first `.clone()` it. The clone is cheap
    // as it only copies the iterator's state, not the entire document.
    let count = product_containers.clone().count();
    println!("Found {} products on the page.", count);

    Ok(vec![])
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    match scrape_page(url_to_scrape).await {
        Ok(_products) => {
            println!("\nScraping task completed successfully!");
        }
        Err(e) => {
            eprintln!("\nError during scraping: {}", e);
        }
    }
}

Now, run your program with cargo run. You should see the following output in your terminal:

Found 20 products on the page.

Scraping task completed successfully!

This confirmation is a fantastic milestone! It proves that you have successfully:

Fetched the page.
Parsed the HTML into a structured document.
Defined a correct CSS selector.
Used the selector to query the document and find the exact number of product containers we expected.

What’s Next?

You now have an iterator, product_containers, that holds the sequence of all 20 product elements from the page. The next logical step is to loop through this iterator. For each product element, we will then run our more specific selectors (product_title_selector and product_price_selector) to extract the final pieces of data.

Loop Through Scraped Product Elements in Rust

Mục tiêu: Implement a for loop in Rust to iterate over the collection of scraped product HTML elements. This task focuses on setting up the iterative structure to process each product container individually, preparing for data extraction in the next step.

You’ve reached a pivotal moment in building our scraper. In the previous task, you successfully queried the parsed HTML document and created product_containers, an iterator that holds a reference to every single product block on the page. You verified this by counting them and seeing the expected number, 20.

Now, it’s time to process this sequence of elements. Instead of just counting them, we will iterate through them one by one, preparing the ground for extracting the specific details from each product.

Consuming the Iterator with a `for` Loop

In Rust, the most common and idiomatic way to work with an iterator is to use a for loop. The for loop takes ownership of the iterator and will execute its body once for each item the iterator produces, until the sequence is exhausted. This is a clean, safe, and efficient way to process a collection of items without needing to manage indices or worry about going out of bounds.

Each item produced by our product_containers iterator is of type scraper::ElementRef. An ElementRef is a lightweight, borrowed reference to a specific element (an HTML tag and its contents) within the main document. This is highly efficient because we are not copying large chunks of HTML data for each product; we are simply looking at different parts of the single document we have in memory.

Our plan is as follows:

Initialize an empty, mutable vector that will hold the final Product structs.
Use a for loop to iterate over the product_containers.
Inside the loop, each product_element will be our context for finding the title, price, and URL. (We will do the actual extraction in the next task).
After the loop finishes, we will return the vector of populated Product structs.

Let’s modify our scrape_page function to implement this looping structure. We will also update our main function to print the number of products found in the returned vector.

Here are the changes for src/main.rs:

// src/main.rs

use scraper::{Html, Selector};

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;
    let document = Html::parse_document(&html_content);

    let product_container_selector = Selector::parse("article.product_pod").unwrap();
    let product_title_selector = Selector::parse("h3 a").unwrap();
    let product_price_selector = Selector::parse("p.price_color").unwrap();

    // highlight-start
    // Create a mutable vector to store the scraped products.
    // We will populate this vector inside our loop.
    let mut products: Vec<Product> = Vec::new();

    // Iterate over each element that matches the product container selector.
    // The `for` loop will consume the `product_containers` iterator.
    for product_element in document.select(&product_container_selector) {
        // Inside this loop, `product_element` is an `ElementRef`.
        // This `ElementRef` gives us a focused view of a single product's HTML,
        // from which we will extract the title, price, and URL in the next task.

        // For now, we will leave the loop body empty as we set up the structure.
    }

    // After the loop has processed all product elements, return the vector
    // of products, wrapped in `Ok`.
    Ok(products)
    // highlight-end
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    println!("Scraping URL: {}", url_to_scrape);

    match scrape_page(url_to_scrape).await {
        // highlight-start
        // If the function returns successfully, `products` will be the Vec<Product>.
        Ok(products) => {
            // We use {:#?} for "pretty-printing" the debug output, which is more readable.
            println!("Successfully scraped {} products!", products.len());
            println!("{:#?}", products);
        }
        // highlight-end
        Err(e) => {
            eprintln!("\nError during scraping: {}", e);
        }
    }
}

Code Breakdown

let mut products: Vec<Product> = Vec::new();: Before we start iterating, we create a new, empty, and mutable vector. This is where we’ll collect the Product instances as we create them.
for product_element in document.select(&product_container_selector): This is the core of our change. We directly loop over the result of document.select(). The previous logic for cloning and counting is no longer needed, as our goal now is to process the elements themselves.
Ok(products): At the end of the function, we return the products vector. If the loop found no products, it will correctly return an empty vector.
In main: We’ve updated the Ok arm of our match statement. It now receives the products vector and prints its length, followed by the contents of the vector itself. Because Product derives Debug, we can print it easily with {:#?}.

If you run the code now with cargo run, the output will be:

Scraping URL: http://books.toscrape.com/
Successfully scraped 0 products!
[]

This output is absolutely correct! We have successfully set up the loop to iterate through the 20 product elements, but since the loop’s body is empty, we haven’t actually created and added any Product structs to our products vector yet. The fact that the program runs, completes the loop, and returns an empty vector proves our structure is sound.

What’s Next?

You have built the complete scaffolding for data extraction. The for loop is ready and waiting. The next, and most exciting, task is to fill in the body of that loop. You will use the other selectors you defined (product_title_selector and product_price_selector) on each product_element to extract the title and price text.

Extract Product Details Within a Scraping Loop

Mục tiêu: Inside a loop iterating over product HTML elements, use the scraper crate in Rust to select and extract specific data points like title, price, and URL. This involves using selectors within the context of an ElementRef, handling Option types with if let, and collecting the data into a struct.

This is the moment where all our setup comes together. In the previous task, you brilliantly constructed the loop that iterates over each product’s HTML block. That loop is the engine of our scraper, and now, it’s time to add the machinery to pull out the valuable data.

Inside the loop, our product_element variable is an ElementRef, which represents a focused view of a single product’s HTML. A key feature of ElementRef is that it has its own .select() method. This allows us to run our more specific selectors not against the whole page, but only within the context of that single product block. This is how we’ll isolate the title and price for each item.

Extracting the Title and URL

Let’s start with the product’s title and its unique URL. We’ll use our product_title_selector (h3 a) on the product_element.

product_element.select(&product_title_selector) will give us an iterator over the <a> tags inside the <h3>.
We know there’s only one such tag per product, so we’ll call .next() on the iterator. This returns an Option<ElementRef>. Using Option is Rust’s safe way of handling cases where an element might not be found, preventing crashes.
If we find the element (Some(element)), we can then inspect it. Looking at books.toscrape.com, we find two key pieces of information on the <a> tag:
- The full title is in the title attribute. The visible text is often truncated with an ellipsis (…), so reading the attribute is more reliable. We can get this with .value().attr("title").
- The URL is in the href attribute. We get this with .value().attr("href").
The extracted URL will be a relative path (e.g., catalogue/a-light-in-the-attic_1000/index.html). To make it a complete, clickable link, we must join it with the website’s base URL.

Extracting the Price

Extracting the price follows a similar pattern, but instead of an attribute, we need the actual text content of the element.

We’ll use our product_price_selector (p.price_color) on the product_element.
Again, we call .select(...).next() to get an Option<ElementRef>.
To get the visible text content of an element, we use the .text() method. This method returns an iterator over all the text nodes inside the element (e.g., £, 51.77).
We then use .collect::<String>() to join all these text pieces into a single, clean String.

Let’s fill in the body of our for loop in src/main.rs.

// src/main.rs

use scraper::{Html, Selector};

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;
    let document = Html::parse_document(&html_content);

    let product_container_selector = Selector::parse("article.product_pod").unwrap();
    let product_title_selector = Selector::parse("h3 a").unwrap();
    let product_price_selector = Selector::parse("p.price_color").unwrap();

    let mut products: Vec<Product> = Vec::new();

    for product_element in document.select(&product_container_selector) {
        // --- Start of New Code ---

        // Extract the title and URL.
        // We search within the current `product_element` for an element matching the title selector.
        let title_element = product_element.select(&product_title_selector).next();
        // The price follows the same pattern.
        let price_element = product_element.select(&product_price_selector).next();

        // We use `if let` to safely unwrap the `Option`s. This ensures that if a product
        // is missing a title or a price for some reason, our scraper doesn't crash.
        if let (Some(title_elem), Some(price_elem)) = (title_element, price_element) {

            // Extract the text content of the price element.
            // `.text()` returns an iterator of text nodes, which we `.collect()` into a String.
            let price = price_elem.text().collect::<String>();

            // Extract the 'title' attribute for the full book title.
            // We use `.value().attr()` to get the value of an HTML attribute.
            let title = title_elem.value().attr("title").unwrap_or_default().to_string();

            // Extract the 'href' attribute for the relative URL and make it absolute.
            let relative_url = title_elem.value().attr("href").unwrap_or_default();
            let url = format!("http://books.toscrape.com/{}", relative_url);

            // Create a new `Product` instance with the scraped data.
            let product = Product {
                title,
                price,
                url,
            };

            // Add the newly created product to our vector.
            products.push(product);
        }
        // --- End of New Code ---
    }

    Ok(products)
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    println!("Scraping URL: {}", url_to_scrape);

    match scrape_page(url_to_scrape).await {
        Ok(products) => {
            println!("Successfully scraped {} products!", products.len());
            // We only print the first 3 products for a cleaner output.
            for product in products.iter().take(3) {
                 println!("{:#?}", product);
            }
        }
        Err(e) => {
            eprintln!("\nError during scraping: {}", e);
        }
    }
}

Code Explanation

if let (Some(title_elem), Some(price_elem)) = (title_element, price_element): This is a powerful Rust pattern. It checks if both title_element and price_element contain a Some value. Only if both elements are found will the code inside the curly braces execute. This makes our scraper robust against missing data on the page.
.unwrap_or_default().to_string(): When we extract the attributes, we use unwrap_or_default(). If the href or title attribute is missing for some reason, this will prevent a panic and provide an empty string slice ("") instead, which we then convert to an owned String.
format!("http://books.toscrape.com/{}", relative_url): This macro safely and efficiently constructs the full URL string by prepending the base URL to the relative path we scraped.
products.push(product): This is the final step inside the loop. We take the Product we just created and add it to our list of results.

Now, run your program with cargo run. The output should be spectacular! You’ll see a confirmation that 20 products were scraped, followed by the neatly structured data for the first few books, each with a title, price, and a full, valid URL.

What’s Next?

You have successfully written the core logic for scraping a single page! Your scraper can now fetch a page, parse it, and extract structured data into a Vec<Product>. This is a huge accomplishment. The next step will be to evolve this from a single-page script into a truly concurrent scraper by modifying our main function to process a list of URLs simultaneously.

Assemble Scraped Data into Product Structs

Mục tiêu: Instantiate a Product struct with the scraped title, price, and URL, and then push the newly created struct into a vector to collect all the scraped items.

You have arrived at the final, most rewarding task of this step. In the previous tasks, you masterfully fetched, parsed, and looped through the HTML, culminating in the extraction of raw text for the product’s title, price, and URL inside your for loop. You have the individual ingredients; now it’s time to follow the recipe we created earlier and assemble them into a finished dish: a Product struct.

From Raw Data to Structured Information

Right now, inside your loop, you have three separate string variables. While this data is correct, it’s “loose.” The real power of our design comes from bundling this related information into the Product struct we so carefully defined. This act of instantiation transforms scattered data points into a single, meaningful, and type-safe unit of information.

Instantiating a Struct

To create an instance of a struct in Rust, you use the StructName { ... } syntax, specifying a value for each field. For example:

let my_product = Product {
    title: String::from("A Great Book"),
    price: String::from("£10.00"),
    url: String::from("http://..."),
};

However, Rust has a wonderful piece of syntactic sugar called field init shorthand. If you have a variable with the exact same name as a struct field, you can simply write the variable name instead of field: variable. Since we’ve already created variables named title, price, and url, our instantiation becomes much cleaner and more idiomatic:

// This is equivalent to the verbose version above
let product = Product {
    title,
    price,
    url,
};

This shorthand not only saves typing but also makes the code’s intent clearer.

Collecting the Results

Once we have a Product instance, we need to store it. Recall that before the for loop, we created an empty, mutable vector: let mut products: Vec<Product> = Vec::new();. This vector is our collection basket. For each product we instantiate, we will add it to this vector using the .push() method. The .push() method appends an element to the end of the vector. This action also moves the ownership of the product instance into the vector, which is a core concept in Rust ensuring memory safety.

Let’s put this final piece of logic into our scrape_page function.

Here are the changes for src/main.rs. We are simply adding the final two lines inside the if let block.

// src/main.rs

use scraper::{Html, Selector};

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;
    let document = Html::parse_document(&html_content);

    let product_container_selector = Selector::parse("article.product_pod").unwrap();
    let product_title_selector = Selector::parse("h3 a").unwrap();
    let product_price_selector = Selector::parse("p.price_color").unwrap();

    let mut products: Vec<Product> = Vec::new();

    for product_element in document.select(&product_container_selector) {
        let title_element = product_element.select(&product_title_selector).next();
        let price_element = product_element.select(&product_price_selector).next();

        if let (Some(title_elem), Some(price_elem)) = (title_element, price_element) {

            let price = price_elem.text().collect::<String>();
            let title = title_elem.value().attr("title").unwrap_or_default().to_string();
            let relative_url = title_elem.value().attr("href").unwrap_or_default();
            let url = format!("http://books.toscrape.com/{}", relative_url);

            // highlight-start
            // Create a new `Product` instance using the data we just scraped.
            // We are using the "field init shorthand" here because our variable names
            // (title, price, url) match the struct's field names.
            let product = Product {
                title,
                price,
                url,
            };

            // Add the newly created product to our vector of products.
            // This moves the `product` instance into the `products` vector.
            products.push(product);
            // highlight-end
        }
    }

    Ok(products)
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    println!("Scraping URL: {}", url_to_scrape);

    match scrape_page(url_to_scrape).await {
        Ok(products) => {
            println!("\nSuccessfully scraped {} products!", products.len());
            // Let's print the first 3 products for a clean preview of our data.
            // Using `iter().take(3)` is an efficient way to process just the first few items.
            println!("--- Sample of Scraped Products ---");
            for product in products.iter().take(3) {
                 // The `{:#?}` format specifier triggers "pretty-printing" for structs
                 // that derive `Debug`, making the output much more readable.
                 println!("{:#?}", product);
            }
        }
        Err(e) => {
            eprintln!("\nError during scraping: {}", e);
        }
    }
}

Congratulations! Run your program now with cargo run. You should see a truly satisfying output: a confirmation that 20 products were scraped, followed by the neatly printed, structured data for the first three books.

Scraping URL: http://books.toscrape.com/

Successfully scraped 20 products!
--- Sample of Scraped Products ---
Product {
    title: "A Light in the Attic",
    price: "£51.77",
    url: "http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
}
Product {
    title: "Tipping the Velvet",
    price: "£53.74",
    url: "http://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
}
Product {
    title: "Soumission",
    price: "£50.10",
    url: "http://books.toscrape.com/catalogue/soumission_998/index.html",
}

You have officially completed the entire core logic for scraping a single web page. This scrape_page function is now a complete, robust, and reusable unit of work.

What’s Next?

You have mastered the art of scraping a single page. But the project is called a Concurrent Web Scraper. In the next major step, we will elevate your program from a simple script to a high-performance tool. You will modify the main function to take a list of URLs and spawn a separate asynchronous task for each one, allowing them all to be scraped simultaneously. This is where you’ll truly harness the power of tokio and Rust’s async/await capabilities.

Return the Scraped Products from the Function

Mục tiêu: Complete the scrape\_page function by returning the vector of collected products. Wrap the products vector in the Ok variant of the Result enum to signify a successful operation and fulfill the function’s return type signature.

You have successfully navigated the most intricate part of this step. Your for loop is now a well-oiled machine: it iterates through each product, extracts the raw data, and as of the last task, assembles that data into a Product struct which it then pushes into a collecting vector. The final piece of the puzzle is to complete the function’s contract by returning this vector of fully-formed products to the caller.

Fulfilling the Function’s Promise

Let’s revisit the function signature we so carefully crafted at the beginning of this step:

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>>

This signature is a promise. It says, “Give me a URL, and I promise to return either a Result containing a vector of Products on success (Ok(Vec<Product>)), or a Result containing an error on failure (Err(...)).”

Throughout the function, we’ve diligently handled the failure path. Every ? operator was a potential early exit, returning an Err if anything went wrong. Now, after the for loop has successfully completed, we have reached the end of the success path. Our products vector is full of the data we worked so hard to collect. The final action is to wrap this vector in the Ok variant of the Result enum and return it.

The line Ok(products) at the end of the function does exactly this. It signifies that the entire scraping process for this page was successful and that the payload of this success is the products vector.

Ownership and the Return Value

A critical concept in Rust is happening here: ownership transfer. When we created the products vector, the scrape_page function owned it. When the for loop pushed Product instances into it, the vector took ownership of each product. Now, when we return Ok(products), the ownership of the entire vector and all the data it contains is moved out of the scrape_page function and given to whatever code called it (in our case, the main function). The scrape_page function is now finished with its work, and its resources are cleaned up, but the data it produced lives on in the main function, ready for the next stage of processing.

Let’s look at the complete, final version of the scrape_page function. It now represents a fully self-contained and robust piece of logic for scraping a single web page.

// src/main.rs

use scraper::{Html, Selector};

#[derive(Debug, serde::Serialize)]
pub struct Product {
    pub title: String,
    pub price: String,
    pub url: String,
}

async fn scrape_page(url: &str) -> Result<Vec<Product>, Box<dyn std::error::Error>> {
    // 1. Fetch the HTML from the URL.
    let response = reqwest::get(url).await?;
    let html_content = response.text().await?;

    // 2. Parse the HTML into a structured document.
    let document = Html::parse_document(&html_content);

    // 3. Define the selectors for finding the data.
    let product_container_selector = Selector::parse("article.product_pod").unwrap();
    let product_title_selector = Selector::parse("h3 a").unwrap();
    let product_price_selector = Selector::parse("p.price_color").unwrap();

    // 4. Create a vector to hold the results.
    let mut products: Vec<Product> = Vec::new();

    // 5. Loop through each product container on the page.
    for product_element in document.select(&product_container_selector) {
        let title_element = product_element.select(&product_title_selector).next();
        let price_element = product_element.select(&product_price_selector).next();

        // 6. Extract and clean the data, handling potential missing elements.
        if let (Some(title_elem), Some(price_elem)) = (title_element, price_element) {

            let price = price_elem.text().collect::<String>();
            let title = title_elem.value().attr("title").unwrap_or_default().to_string();
            let relative_url = title_elem.value().attr("href").unwrap_or_default();
            let url = format!("http://books.toscrape.com/{}", relative_url);

            // 7. Instantiate a `Product` and add it to our vector.
            products.push(Product {
                title,
                price,
                url,
            });
        }
    }

    // 8. Fulfill the function's contract by returning the collected data
    //    wrapped in the `Ok` variant of the `Result` enum.
    //    This signifies a successful operation.
    Ok(products)
}

#[tokio::main]
async fn main() {
    let url_to_scrape = "http://books.toscrape.com/";

    println!("Scraping URL: {}", url_to_scrape);

    match scrape_page(url_to_scrape).await {
        Ok(products) => {
            println!("\nSuccessfully scraped {} products!", products.len());
            println!("--- Sample of Scraped Products ---");
            for product in products.iter().take(3) {
                 println!("{:#?}", product);
            }
        }
        Err(e) => {
            eprintln!("\nError during scraping: {}", e);
        }
    }
}

You have now completed a major milestone. This scrape_page function is the heart of your application, and it is now fully functional.

What’s Next?

You’ve mastered scraping a single page. But the project is named a Concurrent Web Scraper for a reason. In the next step, you will graduate from scraping one page to scraping many pages at the same time. You will modify the main function to manage a list of URLs, spawn a separate asynchronous task for each one, and truly unlock the high-performance capabilities of tokio and Rust.