Web scraping with NodeJS: An Ultimate Guide -

In the early days, JavaScript was less famous for scraping data from web pages. Nowadays, the scenario is totally changed as JavaScript is becoming more popular and used for the front-end and back-end of the application. Also, we can use JavaScript to scrape data using the front-end and back-end.

For example, we can use the vanilla JavaScript code to scrape the data by running it into the browser. Furthermore, we can use JavaScript with NodeJS to write back-end code to scrape data from any web page.

This guide will explore the different NodeJS libraries to scrape the data from various web pages. We have chosen a total of 5 different NodeJS libraries, like JQuery, Axios and Cheerio, Playwright, Puppeteer, and Selenium, to scrape the data. We will learn how to use all the above libraries to scrape data via example. Also, we will look at the pros and cons of every library. At last, we will compare all libraries so that users can choose which library they should use for web scrapping according to their requirements and comfortability.

What is Web Scraping?

Web scraping allows us to fetch data from a different webpage. Sometimes, we don’t get data using API (Application programming interface). In such cases, we require to use web scrapping to get data directly from the webpage.

HTML Parsing and Data extraction

HTML parsing and data extraction is part of web scraping. In web scraping, we need to get the row HTML of the particular web page to get data from the web page by making the HTTP ‘GET’ request. It is also called HTML parsing.

Once we get the row HTML of the web page, we can find the required HTML element by identifiers such as class, id, tag, etc. After getting the required HTML element from the row HTML of the webpage, we can extract its inner HTML (Data), also called data extraction.

Applications of Web Scraping

We have seen the overview of HTML parsing and data extraction in web scraping. Now, we will learn the uses and applications of web scraping in this section.

Price monitoring – We can scrape the price data of products from the competitor’s website, and according to that, we can decrease and increase the price on our e-commerce store.
Collection data – We can collect the row data from various webpage to train the machine learning models.
Media monitoring – We can track some events and news using web scraping.
Real Estate – We can use web scrapping to get property data from the real estate agent’s websites.
Lead generation – We can fetch the user’s contact information, such as email or phone number, using web scrapping and use them for lead generation and marketing purposes.

We have seen applications of web scraping. Now, it’s time to do web scrapping with fun!

Before we start web scrapping with NodeJS, we need to create a NodeJS application. Users can follow the steps below to create a NodeJS application.

Step 1 – Users should have NodeJS installed on their local computer.

Step 2 – Create a directory, and open the terminal in the directory.

Step 3 – Enter the below command in the directory to create a new Node application.

npm init -y

Step 5 – NodeJS application is created. Users need to open the project in the code editor and create the app.js file in the project directory. Make sure that the app.js and package.JSON files are in the same directory.

Step 6 – Users can use the below command to run the code of the app.js file.

node app.js

Below, we will scrape the data from the various web pages of the https://dev.to/ website.

1. JQuery

JavaScript contains many libraries, and JQuery is one of them. We can use JQuery to utilize HTML in a better way than JavaScript. The best thing about JQuery is that we don’t need to set the node application to use JQuery with HTML, as it only works on the client side.

We can add JQuery to the HTML file using the path of locally installed JQuery files or via CDN. The best way to use JQuery is CDN, as we don’t need to download the latest version of JQuery when the new version is released.

We will scrape the tags related to web development and the total posts published for each tag from the https://dev.to/tags using JQuery. Also, we have highlighted the tag and total post count with the red mark in the below image.

Scrapping a webpage using JQuery

Step 1 – First, create an HTML file and write a boilerplate code. After that, add the JQuery CDN to the <head> tag of the HTML file, as shown below.

<head>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.3/jquery.min.js"
    integrity="sha512-STof4xm1wgkfm7heWqFJVn58Hm3EtS31XFaagaa8VMReCXAkQnJZ+jEy8PCC/iT18dFy95WcExNHFTqLyp72eQ=="
    crossorigin="anonymous" referrerpolicy="no-referrer"></script>
</head>

Step 2 – Next, use the get() method of the AJAX to make a get request to the webpage and get the row HTML of the webpage. Also, add a web page URL as the first get() method’s parameter.

$.get("https://dev.to/tags", function (data) {

}

Step 3 – Inside the callback function, access the web development tags and counts of posts for a particular tag using the class name.

        var $html = $(data);
        var tags = $html.find(".crayons-tag");
        var posts = $html.find(".mb-3");

Step 4 – Use the for loop, and extract the innerHTML of every tag to show them on the web page.

      for (var i = 0; i < tags.length; i++) {
          let htmlText = `<b>Tag Name:</b> ${tags[i].innerText} <b>Posts:</b> ${posts[i].innerText}`;
        }

Step 5 – Users can append the tag and post counts to the HTML.

Full Example code

Below, we have written the full example code to scrape data using JQuery.

<HTML>

<head>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.3/jquery.min.js"
    integrity="sha512-STof4xm1wgkfm7heWqFJVn58Hm3EtS31XFaagaa8VMReCXAkQnJZ+jEy8PCC/iT18dFy95WcExNHFTqLyp72eQ=="
    crossorigin="anonymous" referrerpolicy="no-referrer"></script>
</head>

<body>
  <h1>Scraping Dev.to Tags and total posts published</h1>
  <div class="demo-container"></div>

  <script>
    function scrapDevToTags() {
      // Scraping the URL using JQuery
      $.get("https://dev.to/tags", function (data) {
        // Parsing the HTML data using JQuery
        var $html = $(data);
        // Selecting the tags from the HTML data
        var tags = $html.find(".crayons-tag");

        // Selecting the posts from the HTML data
        var posts = $html.find(".mb-3");
        // Creating a container to display the tags
        var container = $("<div></div>");
        // Looping through the tags
        for (var i = 0; i < tags.length; i++) {
          // Creating a tag element
          var tag = $("<div></div>");

          let htmlText = `<b>Tag Name:</b> ${tags[i].innerText} <b>Posts:</b> ${posts[i].innerText}`;
          // Adding the tag name to the tag element
          tag.html(htmlText);
          // Adding the tag element to the container
          container.append(tag);
        }
        // Displaying the container on the page
        $(".demo-container").append(container);
      });
    }
    scrapDevToTags();
  </script>
</body>

</HTML>

When the user opens the above HTML file in the browser, it gives an output like the one shown below.

The best thing about using JQuery to scrape data is that developers don’t need to set up the Node application, as they can create HTML files and add scrapping code to the <script> tag. Furthermore, we can write the client-side code to scrape the data from web pages using JQuery.

Also, JQuery is faster than other libraries and provides cross-browser support, which is why JQuery is the first for web scraping in our list.

Many websites blocks accessing their resources using the AJAX get request due to security purpose. So, we can’t scrape the data of such websites using JQuery, which is the disadvantage of JQuery.

2. Axios And Cheerio

The Axios is also one of the most popular JavaScript libraries for HTTP requests. Here, we can use it to make the HTTP ‘get’ request to the web page and to get the row HTML of the web pages.

Once we get the row HTML of the web page using Axios, we can use Cheerio to filter out or extract the required data by parsing the HTML. Basically, cheerio works as a web crawler.

We will scrape the author names of every post from https://dev.to/t/webdev as shown in the below image.

Scrapping a webpage using Axios and Cheerio

Step 1 – First, we need to install the Axios and Cheerio NPM packages to our node application. Open the terminal in the project directory and run the below command.

npm i axios cheerio

Step 2 – Make a get request using the axios.get() method to get the row HTML of the web page.

axios.get(url)
    .then(response => {
      // handle HTML data here
    })
    .catch(console.error);
}

Step 3 – Now, parse the required data using the Cheerio in the then() block.

      const htmlData = response.data;
      const $ = cheerio.load(htmlData);
      const postAuthors = [];
      $('a.crayons-story__secondary').each(function() {
        const author = $(this).text().trim();
        postAuthors.push(author);
      });

In the above code, response.data gives us HTML. The cheerio.load() method loads the HTML from the data.

The $('a.crayons-story__secondary') is used to get all author names. After that, we used each() method to iterate through all authors and get the text of every HTML element. The trim() method is used to remove the white spaces from the author’s name.

Full Example Code

Below, we have written the full example code to scrape data using the Axios and Cheerio NPM packages.

// function to scrape the post authors from https://dev.to/t/webdevs using axios and cheerio
const axios = require('axios');
const cheerio = require('cheerio');

const url = 'https://dev.to/t/webdev';

function scrapUsingAxios(url) {

  axios.get(url)
    .then(response => {
      // get response data
      const htmlData = response.data;
      const $ = cheerio.load(htmlData);
      const postAuthors = [];
      $('a.crayons-story__secondary').each(function() {
        const author = $(this).text().trim();
        postAuthors.push(author);
      });
      console.log(postAuthors);
    })
    .catch(console.error);
}

scrapUsingAxios(url);

In the above output, users can observe that only authors’ names are extracted from all web page posts.

The Axios and Cheerio provide better performance for web scrapping without any doubt, but there are some limitations of Axios and Cheerio.

We can use Axios and Cheerio to scrape the static web pages. In real-time development, most websites’ content never remains static. The solution to scrape the data from the dynamic web page is to use headless browsers.

To learn to scrap, we can use Axios and Cheerio, but in real-time development, users should use the libraries like Playwright or Puppeteer, which we have covered in the below sections.

3. Playwright

The playwright NPM package allows us to scrape data using headless browsing. Let’s see the brief overview of headless browsing.

Headless browsing

Whenever we open any webpage in the browser, it shows the proper GUI (Graphic user interface) with HTML and CSS. As the ‘headless’ browser word suggests, it is without a head, meaning GUI. When we open any web page in a headless browser, it opens the webpage without GUI, which is why it is lighter than other browsers.

However, we can perform every JavaScript action with the headless browser, like clicking and event triggering. Developers can use headless browsing to scrape the data when they don’t require to render the webpage but need to perform the HTML parsing or data extraction.

Here, we will scrape the post titles from the https://dev.to/t/webdev web page as shown below.

Scrapping the web page with Playwright

The Playwright also uses headless browsing; it launches the new browser window and opens up the webpage in headless mode.

You can follow the below steps to scrape data using Playwright.

Step 1 – First, we need to install the Playwright NPM package by executing the below command in the terminal.

npm install playwright

Step 2 – Launch the browser window using Playwright.

  const window = await playwright.chromium.launch({
    headless: true,
  });

Step 3 – Open the web page in the headless browser window.

  const webPage = await window.newPage();
  await webPage.goto(url);

Step 4 – Access the HTML elements using the identifiers.

const posts = await webPage.$$("div.crayons-story");
  for (let post of posts) {
    const postTitle = await post.$eval(
      ".crayons-story__hidden-navigation-link",
      (el) => el.textContent
    );
    postTitles.push(postTitle);
  }

In the above code, we have fetched all postcards using the webPage.$$("div.crayons-story"). After that, we iterate through every postcard using the for-of loop and get the HTML child element with the class name “.crayons-story__hidden-navigation-link”. We get the content of the HTML element using the textContent property.

Full example code

Here is what the complete code looks like:

const playwright = require("playwright");

const url = "https://dev.to/t/webdev";

async function scrapUsingPlayWright(url) {

  const window = await playwright.chromium.launch({
    headless: true,
  });

 
  const webPage = await window.newPage();
  await webPage.goto(url);

  
  const postTitles = [];
  const posts = await webPage.$$("div.crayons-story");
  for (let post of posts) {
    const postTitle = await post.$eval(
      ".crayons-story__hidden-navigation-link",
      (el) => el.textContent
    );
    postTitles.push(postTitle);
  }

  // Print the result
  console.log(postTitles);
  await window.close();
}

scrapUsingPlayWright(url);

In the above output, users can see all post titles in the string format.

The PlayWright library is more efficient for real-time development as it supports headless browsing. However, extracting the data from webpages requires writing more complex code than Axios and JQuery.

Also, the main advantage of using Playwright is that it provides cross-browser support. So, we can use it with any browser, which means we can launch a headless browser window for any browser like Chrome, Firefox, safari, opera, etc.

4. Puppeteer

The Puppeteer is also a new library in the market, like Playwright, to scrape data from web pages and for automation. The Puppeteer supports almost all Chromium-based browsers and partially supports Firefox, and Microsoft Edge browser, which is the limitation of using the Puppeteer library.

The Puppeteer also supports headless browsing like the Playwright. We can automate the tasks using Puppeteer.

We will scrape the post titles from the https://dev.to/t/ubuntu webpage using Puppeteer.

Scrapping data with Puppeteer

Step 1 – Execute the below command in the project directory to install the Puppetter in the Node application.

npm install puppeteer

Step 2 – Launch the chromium window.

const window = await puppeteer.launch();

Step 3 – Open the web page in the headless browser window.

const webPage = await window.newPage();

await webPage.goto(url);

Step 4 – Use the evaluate() method to get data from the web page.

const data = await webPage.evaluate(() => {
    const results = [];
    const items = document.querySelectorAll(".crayons-story__title a");
    items.forEach((item) => {
      results.push(item.innerText);
    });
    return results;
  });

Here, we have used the querySelectorAll() method to select all elements by class name inside the anonymous function. Once the querySelectorAll() method returns items, we used the forEach() method to iterate through every story card and pushed the innerText to the results list.

Step 5 – At last, close the browser window when scrapping completes.

window.close();

Full example code

Here is the complete example code for web scraping with Puppeteer.

const puppeteer = require("puppeteer");

const url = "https://dev.to/t/ubuntu";

async function scrapUsingPuppeteer(url) {
  const window = await puppeteer.launch();
  const webPage = await window.newPage();

  await webPage.goto(url);

  const data = await webPage.evaluate(() => {
    const results = [];
    const items = document.querySelectorAll(".crayons-story__title a");
    items.forEach((item) => {
      results.push(item.innerText);
    });
    return results;
  });

  console.log(data);
  await window.close();
}

scrapUsingPuppeteer(url);

5. Selenium

Selenium is at last on our list of scrapping libraries. Selenium requires to use of a web driver to extract data from the web pages.

Selenium also launches the headless browser window, but it requires setting up the web drivers, which makes it more complex to use Selenium. Also, loading the webdriver takes longer, So Selenium is less efficient than Playwright and Puppeteer libraries.

Here, we will extract the post titles from https://dev.to/latest using Selenium.

Scrapping webpages with Selenium

Step 1 – Selenium requires setting up the Chrome web driver. Developers can either manually set up the Chrome web driver or use the chrome-web driver NPM package. Here, we will use the chrome-driver NPM package for ease. Use the below command to install Selenium and the chrome webdriver in your node application.

npm install chromedriver selenium-webdriver

Step 2 – Import the selenium and chromedriver in the app.js file.

require("chromedriver");

// Include selenium webdriver
let swd = require("selenium-webdriver");
let until = swd.until;
let browser = new swd.Builder();
let By = swd.By;

Step 3 – Wait until the web driver finds the element with a particular class.

await driver.get(url);
    await driver.wait(
      until.elementLocated(By.css("div.crayons-story__body")),
      10000
    );

Step 4 – Access HTML elements after finding them.

let articles = await driver.findElements(By.css("div.crayons-story__body"));

Step 5 – Iterate through all elements, get their child elements, and access its content.

for (let i = 0; i < articles.length; i++) {
      let article = await articles[i].findElement(
        By.css("h2.crayons-story__title")
      );
      let title = await article.getText();

      console.log(title);
    }

Full example code

Here we have written full example code.

// Include the chrome driver
require("chromedriver");

// Include selenium webdriver
let swd = require("selenium-webdriver");
let until = swd.until;
let browser = new swd.Builder();
let By = swd.By;
let driver = browser.forBrowser("chrome").build();

const url = "https://dev.to/latest";

async function scrapUsingSelenium(url) {
  try {
    await driver.get(url);
    await driver.wait(
      until.elementLocated(By.css("div.crayons-story__body")),
      10000
    );
    let articles = await driver.findElements(By.css("div.crayons-story__body"));
    for (let i = 0; i < articles.length; i++) {
      let article = await articles[i].findElement(
        By.css("h2.crayons-story__title")
      );
      let title = await article.getText();

      console.log(title);
    }
  } catch (error) {
    console.log(error);
  } finally {
    await driver.quit();
  }
}

scrapUsingSelenium(url);

In the above output, users can observe that we have scrapped the data using Selenium. When you run the above code, you can feel how slow Selenium is compared to Puppeteer.

However, the main benefit of using Selenium over Puppeteer is that it is compatible with most browsers. In contrast, Pupeeter is used only with Chromium-based browsers and is supported by JavaScript.

Here, we have compared all five libraries based on their features in the comparison table.

Library	Key Features	Key drawbacks	Applicable Scenarios
JQuery	Lightweight and easy to use	Limited functionality	Scraping simple web pages with basic HTML structure
Axios and Cheerio	Promise-based HTTP client	Limited support for dynamic websites	Scraping static websites with complex HTML structure
Playwright	Cross-browser automation	Steep learning curve	Scraping dynamic websites with complex UI interactions
Puppeteer	Chrome automation and control	Limited support for other browsers	Scraping dynamic websites with complex UI interactions
Selenium	Cross-browser automation	Slow and resource-intensive	Scraping dynamic websites with complex UI interactions

Users can see every library’s main benefits and drawbacks in the above comparison table. Also, users check the applicable scenario to know when they should use which library.

Conclusion

Selenium is the oldest library of the above five libraries, and Playwright and Puppeteer are new compared to other libraries. Users should ask themselves what purpose they want to use the scrapping library before they use any.

If you want to scrape static web pages of a single page, you can use JQuery and Axios. You might prefer Playwright to scrape complex and dynamic web pages using the headless browser and get support for the cross-browser platform. If you need to scrape data using NodeJS and need only Chromium-based browser support, Puppeteer may serve better. If you have very limited knowledge of Playwright or Puppetter, you can go for Selenium.

Web scraping with NodeJS: An Ultimate Guide

What is Web Scraping?

HTML Parsing and Data extraction

Applications of Web Scraping

1. JQuery

Scrapping a webpage using JQuery

Full Example code

2. Axios And Cheerio

Scrapping a webpage using Axios and Cheerio

Full Example Code

3. Playwright

Headless browsing

Scrapping the web page with Playwright

Full example code

4. Puppeteer

Scrapping data with Puppeteer

Full example code

5. Selenium

Scrapping webpages with Selenium

Full example code

Conclusion

Leave a Comment Cancel reply

Reach out to us for a consultation.