How to Set a Proxy in Playwright 2024

In this article, we will tell you how to set up a proxy in Playwright (Playwright version for Node.js).

  Playwright is an excellent solution for browser automation. It has modern browsers such as Chromium, Firefox and WebKit. Playwright uses the open source CDP protocol to send commands. Playwright is mainly used for user interface testing and web scraping. In both cases, setting up proxy servers is an important part of the scenario.

  There are two ways to configure the Playwright proxy server:

• in a global browser instance,
• in the browser context.

The First Way to Set a Proxy in Playwright

Even if you are an experienced developer, there is a simple way to reduce the time you spend working and extend your free time, which you can spend even more usefully!

const browser = await chromium.launch({
  proxy: {
    server: 'http://myproxy.com:3128',
    username: 'usr',
    password: 'pwd'
  }
});

Note! The credentials, which are stored in the standard http://username:pw@host:port syntax, need to be adapted to the Playwright format. Let's take a closer look.

Another Way of Setting up a Proxy for Playwright

You can, of course, use Javascript or Typescript, but you will have to commit sensitive credentials to the git repository. You can keep it simple and use environment variables to use the .env file and the standard one-line proxy syntax:

PROXY_URL=http://user:pw@proxy-host:port

The next step is to initialize dotenv in your project's main file. Pre-install dotenv using the npm i dotenv command. Here's what happens:

import dotenv from 'dotenv'
dotenv.config()

function convertProxyToPlaywrightFormat(proxyUrl) {
    const url = new URL(proxyUrl);
    return {
        server: `${url.protocol}//${url.host}`,
        username: url.username,
        password: url.password
    };
}

const proxyOptions = convertProxyToPlaywrightFormat(proxyUrl);

With this, you can avoid having three proxy-only env variables (username, password, host) and replace them with just one variable.

  Now let's look at the complete code for this proxy in Playwright:

import 'dotenv/config';
import { chromium } from 'playwright';

function convertProxyToPlaywrightFormat(proxyUrl) {
    const url = new URL(proxyUrl);
    return {
        server: `${url.protocol}//${url.host}`,
        username: url.username,
        password: url.password
    };
}

async function main() {
    const proxyUrl = process.env.PROXY_URL;

    if (!proxyUrl) {
        console.error('Proxy URL not found in .env file');
        process.exit(1);
    }

    const proxyOptions = convertProxyToPlaywrightFormat(proxyUrl);

    const browser = await chromium.launch({
        proxy: proxyOptions,
    });

    const page = await browser.newPage();
    await page.goto('http://example.com');

    await browser.close();
}

main();

Web Scraping Playwright: Proxy Rotation and Retries

If you use rotating proxies, then the previous code version will not work for you. Proxies, although effective, make the connection to the target site less reliable, and there is always a risk that the connection will simply be interrupted. The code above will fail more often when using a proxy than it would without a proxy. However, there is a solution. You need to repeat the Playwright request. Here's how to do it:

import 'dotenv/config';
import { chromium } from 'playwright';

function convertProxyToPlaywrightFormat(proxyUrl) {
    const url = new URL(proxyUrl);
    return {
        server: `${url.protocol}//${url.host}`,
        username: url.username,
        password: url.password
    };
}

async function tryNavigate(page, url, maxRetries = 3) {
    for (let attempt = 1; attempt <= maxRetries; attempt++) {
        try {
            await page.goto(url);
            return; // If successful, return without throwing an error
        } catch (error) {
            console.error(`Attempt ${attempt} failed: ${error.message}`);
            if (attempt === maxRetries) {
                throw error; // Rethrow the last error if all retries fail
            }
        }
    }
}

async function main() {
    const proxyUrl = process.env.PROXY_URL;

    if (!proxyUrl) {
        console.error('Proxy URL not found in .env file');
        process.exit(1);
    }

    const proxyOptions = convertProxyToPlaywrightFormat(proxyUrl);
    const browser = await chromium.launch({
        proxy: proxyOptions,
    });

    try {
        const page = await browser.newPage();
        await tryNavigate(page, 'http://example.com');
    } catch (error) {
        console.error(`Failed to navigate: ${error.message}`);
    } finally {
        await browser.close();
    }
}

main();

In addition to retrying the request, you should reduce the number of resources you download. For example, limit the number of loaded pages. Then Playwright's work with a proxy will be more stable.

How to Set Different Proxies for one Playwright instance

Playwright requires large system resources, such as a significant amount of RAM. Therefore, sometimes it is necessary to reduce the load on the system. This can be done by using different proxies for different requests. This will reduce the number of browser instances that load the system. In the case of Playwright, Playwright contexts are used: BrowserContexts. They allow you to manage multiple independent browser sessions. At the same time, note that for the website, the two contexts look like two different browser sessions, although they are launched using the same platform, Playwright. Let's imagine we have a .env file:

PROXY_URL=http://user:pw@proxy-host:port
PROXY2_URL=http://user2:pw@proxy-host2:port

And this is an example of how you can use two different proxies in one Playwright session:

import 'dotenv/config';
import { chromium } from 'playwright';

function convertProxyToPlaywrightFormat(proxyUrl) {
    const url = new URL(proxyUrl);
    return {
        server: `${url.protocol}//${url.host}`,
        username: url.username,
        password: url.password
    };
}

async function main() {
    const proxyUrl = process.env.PROXY_URL;
    const proxy2Url = process.env.PROXY2_URL;

    if (!proxyUrl || !proxy2Url) {
        console.error('One or both proxy URLs not found in .env file');
        process.exit(1);
    }

    const proxyOptions = convertProxyToPlaywrightFormat(proxyUrl);
    const proxy2Options = convertProxyToPlaywrightFormat(proxy2Url);

    const browser = await chromium.launch();

    // Create two different contexts with different proxies
    const context1 = await browser.newContext({ proxy: proxyOptions });
    const context2 = await browser.newContext({ proxy: proxy2Options });

    const page1 = await context1.newPage();
    const page2 = await context2.newPage();

    // Do something with both pages. 
    // Cookies and sessions are not shared between page1 and page2
    await page1.goto('http://example.com');
    await page2.goto('http://example.com');

    // Close the browser contexts
    await context1.close();
    await context2.close();

    // Close the browser
    await browser.close();
}

main();