Web scraping is nothing new. However, the technologies that are used to build websites are constantly developing. Hence, the techniques that have to be used to scrape a website have to adapt.
A lot of websites use front-end frameworks like React, Vue.js, Angular, etc., which load the content (or parts of the content) after the initial DOM is loaded. This especially applies to performance-optimized e-commerce websites, where price and production information are loaded asynchronously.
This is where Puppeteer comes in. It opens a headless Chrome instance to render a page.
Getting Started – Prerequisites
Let us get started by installing Node.js on our system by initializing a new npm (Node Package Manager) instance. npm allows us to install further packages easily. To begin, run the following command:
querySelector() to parse retrieved HTML.
That’s it. We are finished with all the prerequisites. Let’s start working on our actual web scraper.
Building the Scraper
Let us create a new file, called
index.js and start by importing the previously installed Puppeteer library.
Next, we launch a new headless Chrome window. The
await command tells Puppeteer to wait to proceed to the next line until the related statement is completed. Following this pattern, we open up our e-commerce site and tell the browser to wait until the element that contains all information that we want to scrape is visible.
In our case, this element is a div-container, labeled with the
price-box__price class. Now the website is in the state where all information that is relevant to us is visible.
As a next step, we are going to use a function called
evaluate(). It allows us to interfere with the rendered website, which is what we need to do if we want to scrape it.
In the code snippet above, we first select the desired product information and save them into the variables
currency. Next, we save them to an object and declare
null as a fallback value in case the property does not exist.
console.log() statement will return the gathered information, as shown in the screenshot below:
We now want to get rid of the linebreaks and all spacing. Additionally, we want to convert the property
amount to an integer value.
This is the complete and finalized code snippet:
This will output the data in an expected and well-structured way:
Of course, you would want to scrape a lot more data properties when applying and running this in a serious web project. But this tutorial is about the concept behind it.
This article was used as a basis for the creation of this article.