---
title: Scrape any webpage with a real browser
description: Scrape DOM elements from any page with a remote browser in the cloud.
source_url:
  html: https://testingbot.com/support/functions/scrape
  md: https://testingbot.com/support/functions/scrape/index.md
---
# Scrape Function

With the Scrape function you can visit any webpage on the internet with a remote, real browser running on TestingBot.

You can fetch specific elements from the page and TestingBot will return it as a structured JSON response.

By default, TestingBot will navigate to the URL and wait for all content to be loaded. TestingBot will then wait up to 30 seconds for an element to appear, assuming it is not yet in the DOM.

## Structured data

The Scrape function can return structured data for several specific use cases. See the list below on what we currently support:

- [Amazon Products Scraper API](https://testingbot.com/support/functions/scrape/amazon-products)

## Example

    curl -X POST https://cloud.testingbot.com/scrape?key=YOUR_KEY&secret=YOUR_SECRET&browserName=chrome&version=latest&platform=WIN11 \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://testingbot.com", "elements":[{"selector":"body h2"}]}'

This simple example will run a Puppeteer script on our service and do the following:

- Start a Headless Browser (latest version) on our cloud
- Connect with Puppeteer to the Browser and navigate to the URL you specified
- Look for any elements in the DOM that match the selector `body h2`
- Return a JSON response with the results that are found, see the example response below.

    [{"selector":"body h2","results":[{"html":"Automated Testing","text":"","width":340,"height":84,"top":1852.40625,"left":780,"attributes":[]},{"html":"Live Testing","text":"","width":150,"height":84,"top":2727.890625,"left":-146.71875,"attributes":[]},{"html":"+4800 Browsers & Devices","text":"","width":184,"height":168,"top":3455.265625,"left":780,"attributes":[]},{"html":"Integrate TestingBot into your setup","text":"Integrate TestingBot into your setup","width":549,"height":35,"top":2097.9375,"left":117.03125,"attributes":[]}]}]

## Specifying browser and version

See the example below to specify on which platform configuration you'd like to scrape a webpage on. You can specify a `browserName`, `version` and `platform`.

  ![OS selected](https://testingbot.com/assets/environments/svg/windows11-0e1b28bc0fdd5034d3e4d3dc8d346c500a8c6522facf4b45d0da56537c1f1c6d.svg) Windows 11 › ![Browser selected](https://testingbot.com/assets/environments/svg/chrome-c4081ff447d2d898d4afcb8f074a907c960e6f007716c1a1d119eee6803c4042.svg) Chrome 139 

Loading environments...

Please wait while we load the available browsers and platforms.

    

## Scrape Options

You can specify additional options to use while using the scrape functionality.

### Authenticate options

You can specify the [page.authenticate](https://pptr.dev/api/puppeteer.page.authenticate) credentials.

    curl -X POST https://cloud.testingbot.com/scrape?key=YOUR_KEY&secret=YOUR_SECRET&browserName=chrome&version=latest&platform=WIN11 \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://testingbot.com", "authenticate":{ "username": "user", "password": "passwd" }, "elements":[{"selector":"body h2"}]}'

### Goto options

You can specify the [page.goto](https://pptr.dev/api/puppeteer.page.goto) options and add `timeout` and `waitUntil` settings.

    curl -X POST https://cloud.testingbot.com/scrape?key=YOUR_KEY&secret=YOUR_SECRET&browserName=chrome&version=latest&platform=WIN11 \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://testingbot.com", "gotoOptions":{ "waitUntil": "networkidle2" }, "elements":[{"selector":"body h2"}]}'

### Extra Headers

You can specify the [page.setExtraHTTPHeaders](https://pptr.dev/api/puppeteer.page.setextrahttpheaders) options to add extra HTTP headers to the request that the TestingBot browser makes.

    curl -X POST https://cloud.testingbot.com/scrape?key=YOUR_KEY&secret=YOUR_SECRET&browserName=chrome&version=latest&platform=WIN11 \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://testingbot.com", "extraHeaders":{ "foo": "bar" }, "elements":[{"selector":"body h2"}]}'

### Disable Javascript

You can disable Javascript with the [page.setJavaScriptEnabled](https://pptr.dev/api/puppeteer.page.setjavascriptenabled) option.

    curl -X POST https://cloud.testingbot.com/scrape?key=YOUR_KEY&secret=YOUR_SECRET&browserName=chrome&version=latest&platform=WIN11 \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://testingbot.com", "javascriptEnabled":false, "elements":[{"selector":"body h2"}]}'

### Emulate Media

Change the CSS media type of the page with [page.emulateMediaType](https://pptr.dev/api/puppeteer.page.emulatemediatype).

    curl -X POST https://cloud.testingbot.com/scrape?key=YOUR_KEY&secret=YOUR_SECRET&browserName=chrome&version=latest&platform=WIN11 \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://testingbot.com", "emulateMedia":"print", "elements":[{"selector":"body h2"}]}'

### Looking for more help?

Have questions or need more information? Reach out via email or Slack.

[Email us](https://testingbot.com/contact/new)[Slack Join our Slack](https://join.slack.com/t/testingb0t/shared_invite/zt-3bcw9xch-jk19~6XPs_xBrsAgAedkCw)