IOVIO - How to start with Puppeteer and HTTP tracing

Puppeteer is a Node library which provides a high-Level API to control Chrome over the Dev Tools protocol. Basically, everything you do manually in the browser can be done via Puppeteer. A couple of examples are:

Generate screenshots and PDFs of pages.
Automate form submissions, UI testing, keyboard input.
Create an up-to-date, automated testing environment.
Capture a timeline trace to help you find performance issues.
Test Chrome extensions.

In this article, I will focus on performance and will give you a simple step by step approach on how to start with Puppeteer and create a short HTTP tracing, which is logging all the web browser interactions.

A couple of weeks ago we started an experiment at the client-side to check if Puppeteer can help us with performance testing a web application. Where we first use Jmeter in combinarion with our Selenium Page objects we are planning to switch to Puppeteer. The main reason for this is that the Selenium framework is evolved in such a way it’s not that easy anymore to build a jar from the framework and make it work together with JMeter. Reading about Puppeteer made me curious and is apparently very efficient and easy to script.

Before you start

Before we start scripting a couple of preconditions need to be in place:

Code editor
Install NPM, Node JS, Puppeteer

To start scripting a Code editor needs to be available otherwise we are not able to script and run our code. If you don’t have a code editor installed on your device I will recommend you install Visual Studio Code. Visual Studio Code is free and available on Linux, MacOs, and Windows.

After installing our Code Editor we are going to install Node Js. This is needed because Puppeteer is a Node library. When installing Node JS we get immediately NPM installed on our device.

Before we are going to install Puppeteer we are first going to create a project directory:

C:\>mkdir iovio_test

‍

C:\>cd iovio_test

‍

In this project directory we will install Puppeteer with the following command:

C:\iovio_test>npm i puppeteer

When the installation is done we are ready to code!!

Let’s start coding

First of all, let’s create an empty JavaScript file. When this is done we are going to import the puppeteer library by adding the following line:

const puppeteer = require('puppeteer');

After this, we can make use of the functions puppeteer is offering. We will be able to start a chrome browser, navigate to a specific page and close the browser again:

const puppeteer = require('puppeteer');
(async () => {

// Opening a browser with the give arg.
const browser = await puppeteer.launch({
    headless: false,
    devtools: false,
    ignoreHTTPSErrors: true,
    args: [
        '--start-fullscreen',
        '--window-size=1920,1040',
        '--no-sandbox'
    ]
});

// Opening a new page
const page = await browser.newPage();

// Set windows height en width
await page.setViewport({ width: 1600, height: 900 });

// Navigate to specific url and wait till network traffic is idle
await page.goto('https://qualibrate.com', {waitUntil: "networkidle0"});

// Close the browser
await browser.close();
})();

‍

Let’s check if this works by starting the little script with the following command in your terminal. Don’t forget to use the file name of the file that you created. In my case this is test.js:

C:\iovio_test>node test.js

‍

If your script runs successfully let’s add some extra steps and do a bit of navigation to the blog section of this web page. Your script will look like the following then:

const puppeteer = require('puppeteer');
 
(async () => {
 
// Opening a browser with the give arg.
const browser = await puppeteer.launch({headless: false, devtools: false, ignoreHTTPSErrors: true, args: [
    '--start-fullscreen',
    '--window-size=1920,1040',
    '--no-sandbox'
    ]});
 
// Opening a new page
const page = await browser.newPage();
 
// Set windows height en width
await page.setViewport({ width: 1600, height: 900 });
 
// Navigate to specific url and wait till network traffic is idle
await page.goto('https://qualibrate.com', {waitUntil: "networkidle0"});
 
// Click on menu item Blog
const menuElement = await page.$x("//a[contains(text(), 'Blog')]");
await menuElement[0].click();
 
await page.waitForNavigation({waitUntil: "networkidle0"});
 
// Close the browser
await browser.close();
})();

To test your script you can use the following command in your terminal:

C:\iovio_test>node test.js

HTTP Tracing

If your script ran successfully it’s time for the next step: we are going to make a HTTP tracing during the execution time of the script. The first action we need to do is to add an extra package with the name puppeteer-har. We can get this package via NPM by the command:

C:\iovio_test>npm i puppeteer-har

When the package is finished with downloading, we can extend our script with the following code at the top of the script. Your first two lines of code will look like this:

const puppeteer = require('puppeteer');
const PuppeteerHar = require('puppeteer-har');

‍

After this is done we will place the code below between the opening of a new page and then set the height and width of the window. In this couple of lines of code, we are creating a timestamp for a unique filename and we are starting the tracing:

//opening a new page
const page = await browser.newPage();

// Create a timestamp 
var timestamp = (Date.now() / 1000 | 0);
const har = new PuppeteerHar(page);
 
// Start the HTTP Tracing
await har.start({ path: './'+timestamp+'results.har' });
console.log(`Har file started and profiling`);
 
// Set windows hight en width
await page.setViewport({ width: 1600, height: 900 });

‍

To stop the HTTP tracing we need to add an extra line of code at the end of the script:

// Stop the HTTP Tracing
await har.stop();
console.log(`Stop Har file and save`);

await browser.close();

‍

HAR Analyzer

After you run your script, a Har file is created. The format of the file is basically a JSON object with a particular field distribution. For this example, we are going to use an online HAR Analyzer but be careful. A Har file contains sensitive data like all the information you submitted. For this example everything is fine and we can use the HAR Analyzer from google.

Choose your file and open it. All the requests that happen during your recording will be visible in this analysis.

Summary and Next steps

After following the steps above you are able to

setup your environment;
install the prerequisites to start with coding:
create a simple script that is opening a page in a browser and performs a navigation step.
Starting a HTTP tracing which can be checked by a Har Analyzer.

For the next article that I am going to write, we will go deeper into how to set up the TICK stack in a Docker container and send our performance results to an InfluxDB, which is part of the TICK stack. In this way, we can create a dashboard of the available data.

I hope you had fun reading and trying the examples above. If you have questions don’t hesitate to contact me for some support or to write in the comments below.

‍