Puppeteer

Puppeteer is a Node library which provides a high-level API to control Chrome or chromium over the DevTools Protocol. Puppeteer runs headless by default, but it can be configured to run full Chrome. It was maintained by the Chrome DevTools team and an awesome open-source community. When you install the puppeteer, it downloads the recent version of chromium that is guaranteed to work with the API. The benefit of the puppeteer was, it allowed access to the measurement of loading and rendering times provided by the Chrome Performance Analysis tool. Most things that you can do manually in the browser can be done using Puppeteer such as generate screenshots and PDF of pages, automate form submission, UI testing, keyboard etc.,

A Short Demo for Web Scraping, e2e with Puppeteer

Chrome vs Chromium

Chrome

• Chrome is a proprietary web browser developed and maintained by Google.
• Chrome has automatic updates, browsing data, and native support for Flash.
• Chrome has sandbox support.

Chromium :

• Chromium is an open-source web browser developed and maintained by the Chromium Projects
• Chromium has no auto updates, browsing data, or Flash support.
• Chromium also has sand box support. But some linux distributions may disable the sandbox support.

Prerequisites

To use Puppeteer, you have to install Node.js on your machine.

Installation

npm i puppeteer

Web Scraping

Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page. Therefore, web crawling is a main component of web scraping, to fetch pages for later processing. Once fetched, then extraction can take place. The content of a page may be parsed, searched, reformatted, its data copied into a spreadsheet, and so on. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else.

Headless Browser

A headless browser is a web browser without a graphical user interface. It is often called a scraper or a crawler to read and interact with it. It provides automated control of a web page in an environment similar to popular web browsers, but are executed via a command-line interface or using network communication. They are particularly useful for testing web pages as they are able to render and understand HTML the same way a browser would, including styling elements such as page layout, colour, font selection and execution of JavaScript and AJAX which are usually not available when using other testing methods.

Headless Chrome

Headless Chrome is a way to run the Chrome browser in a headless environment without the full browser UI. One of the benefits of using Headless Chrome is that your JavaScript tests will be executed in the same environment as users of your site. Headless Chrome gives you a real browser context without the memory overhead of running a full version of Chrome. In that we need not to specify the headless option when the browser will launch.

Headless Chrome Example

const puppeteer = require("puppeteer");
(async () => {
    try {
        const browser = await puppeteer.launch({ headless: true, defaultViewport:null });
        const page = await browser.newPage();
        await page.goto("https://www.google.com/", {
        waitUntil: "networkidle2",
            timeout: 0
        });
        await page.screenshot({ path: "screen.png", fullPage: true });
        await page.pdf({ path: "screen.pdf", format: "A4" });
        console.log("Image and PDF generated");
        await browser.close();
    } catch (error) {
        console.log("We got an error!!!");
    }
})();

Headful Chrome

Headful chrome means displaying the browser graphical user interface and it is very useful for debugging. In that we need to set the headless option to false when the browser will launch.

const puppeteer = require("puppeteer");               
(async () => {
    try {
        const browser = await puppeteer.launch({ headless: false, defaultViewport:null });
        const page = await browser.newPage();
        await page.goto("https://www.google.com/", {
            waitUntil: "networkidle2",
            timeout: 0
        });
        await page.screenshot({ path: "screen.png", fullPage: true });
        await page.pdf({ path: "screen.pdf", format: "A4" });
        console.log("Image and PDF generated");
        await browser.close();
    } catch (error) {
        console.log("We got an error!!!");
    }
})();

Automated Scripts

An automation script consists of a launch point, variables with corresponding binding values, and the source code. You use wizards to create the components of an automation script. You create scripts and launch points or you create a launch point and associate the launch point with an existing script. It provides many benefits including faster execution of repetitive tasks, ability to parallelize workloads and improved test coverage for your website. In the below example, we have to create a simple login process when the user enters the email and password , then they will redirects to their respective page.

const puppeteer = require("puppeteer");
const CRED = {
    user: "developer@gmail.com",
    pass: "password"
};
const ID = {
    login: "#email",
    pass: "#pass"
};
(async () => {
    try {
        const browser = await puppeteer.launch({
            headless: false,
            defaultViewport: null
        });
        const page = await browser.newPage();
        await page.goto("https://facebook.com");
        await page.waitForSelector(ID.login);
        await page.type(ID.login, CRED.user);
        await page.type(ID.pass, CRED.pass);
        await page.click("#loginbutton");
        await page.waitForNavigation({ waitUntil: "load" });
        console.log("Logged in Successfully!!!!");
        await browser.close();
    } catch (error) {
        console.log("We got an error!!!");
    }
})();

Web Scraping in Puppeteer

The first step of web-scraping is to acquire the selectors. A selector is just a path to the data. You have to acquire the selectors when inspect the element of the page, the developer tools window will open. In the Elements tab of Developer Tools, right-click the highlighted element and select CopySelector. In the below example, instead of map we have to use forEach for looping the data.

const puppeteer = require("puppeteer");
(async function () {
    try {
        const browser = await puppeteer.launch({
            headless: true,
            defaultViewport: null
        });
        const page = await browser.newPage();
        await page.goto(
            "https://examples.wufoo.com/reports/example-contact-form-report/"
        );
        const data = await page.evaluate(() => {
        const name = document.querySelectorAll(
            #"#gridHolder > table > tbody > tr"
        );
        let rows = [];
        name.forEach(data => {
        const td = data.querySelectorAll("td");
        let val = {};
        #val = {
            #...val,
            #serialno: td[0].innerText,
            #message: td[1].innerText,
            #email_address: td[2].innerText,
            #datecreated: td[3].innerText
        };
        rows = [...rows, val];
        });
        return rows;
        });
    } catch (error) {
        console.log("We got an error!!!");
    }
})();