Skip to main content

Running Puppeteer Scenarios for Web scraping

This feature is used for more advanced ways of web scraping. Interact with the webpage you want to scrape using the Puppeteer automation tool.

This method allows running full-featured JavaScript scenarios with Puppeteer wrapped into a single POST request which can be made in any language from your side.

What can be made with Puppeteer:

  • Automated form submission
  • Keyboard input
  • Authorization / Login
  • Mouse clicks
  • Custom JavaScript execution
  • Waiting for CSS elements to appear
  • Extracting data by CSS selector
  • Page scrolling
  • XHR/AJAX requests interception

The /scenario endpoint takes data from context variable and and runs code from code variable.

Getting page content


"url": ""

puppeteer scenario:

module.exports = async function ({ page, context }) {
const { url } = context; // Read the `url` from context

await page.goto( // Docs:
{waitUntil: 'domcontentloaded'}

const data = await page.content();
return {
type: 'application/html',

The code above was minified with the online babel repl or, so being unable to have multi-line strings in JSON, you still can use it in the following curl call:

cURL (with an API token)

curl -X POST \
'' \
-H 'Content-Type: application/json' \
-d '{
"code": "module.exports = async ({ page, context }) => { const { url } = context; await page.goto( url, {waitUntil: \"domcontentloaded\"} ); const data = await page.content(); return { data, type: \"application\/html\",};};",
"context": {
"url": ""

Example: Logging in and getting data as JSON

As an example, we took demo page to show how to log in to an account to parse data. The 'context' variable is an empty object (because the URL is inside the code)

puppeteer scenario:

module.exports = async ({ page, context }) => {
await page.goto(
{waitUntil: 'domcontentloaded'}
// Fill inputs & click Login
await page.type('.block.login input', 'pushDEMO');
await page.type('.block.login [type=password]', '12345');
await'[title="Login securely"]');

// Waiting page load
await page.waitForSelector('.btableblock');

// Fetching data using JavaScript exec on the page
let payments = await page.evaluate(() => {
let result = [];
let elements = document.querySelectorAll('.balance .offscreen'); // get elements by selector
for (i=0; i<elements.length; i++) { // iterate over elements
return result; // returning data to 'payments' variable

return {
data: payments,
type: 'application/json',

Scenario wrapped into request

curl --location --request POST '' \
--header 'Content-Type: application/json' \
--data-raw '{"code":"module.exports=async({page:a,context:b})=>{await a.goto(\"https:\/\/\/en\/home\/login-online-banking\/demo-online-banking-pushtan.html\",{waitUntil:\"domcontentloaded\"}),await a.type(\".block.login input\",\"pushDEMO\"),await a.type(\".block.login [type=password]\",\"12345\"),await\"[title=\\\"Login securely\\\"]\"),await a.waitForSelector(\".btableblock\");let c=await a.evaluate(()=>{let a=[],b=document.querySelectorAll(\".balance .offscreen\");for(i=0;i<b.length;i++)a.push(b[i].innerText);return a});return{data:c,type:\"application\/json\"}};","context":{}}'


["24.705,53 EUR","1.000,00 EUR","1.000,00 EUR","-125,50 EUR","-125,50 EUR","18.235,00 EUR","18.235,00 EUR","1.000,00 USD","1.000,00 USD","1.897,45 EUR","1.897,45 EUR","2.378,90 EUR","2.378,90 EUR","97.458,32 USD","97.458,32 USD","558,91 EUR","558,91 EUR","26,12 EUR","26,12 EUR","24.705,53 EUR","52.000,00 EUR","2.000,00 EUR","2.000,00 EUR","15.000,00 EUR","15.000,00 EUR","35.000,00 EUR","35.000,00 EUR","52.000,00 EUR","145.550,80 EUR","47.473,85 EUR","47.473,85 EUR","98.076,95 EUR","98.076,95 EUR","145.550,80 EUR","-36.613,93 EUR","-36.613,93 EUR","-36.613,93 EUR","-880,00 EUR","-880,00 EUR","-880,00 EUR","510.000,00 EUR","450.000,00 EUR","450.000,00 EUR","60.000,00 EUR","60.000,00 EUR","510.000,00 EUR","694.762,40 EUR"]

Session Timeout

Puppeteer / Playwright

By default session timeout is 30 seconds. You can set your value (&timeout=<VALUE> in milliseconds) if it is needed for your script

// 60-second limit:


Add parameter &--proxy-server=browsercloud-proxies to API endpoint string to use built-in proxies