# Web Scrape

> Fetch Google Search results, the text of a web page or site, RSS or Atom feed items, or TikTok posts, and pass them on as JSON to AI and list nodes.

Source: https://nodaro.ai/docs/nodes/automate/web-scrape

The **Web Scrape** node fetches content from the web and returns it as a JSON list of items. It has four sources: Google Search results, the text of a web page or a whole site, the items of an RSS or Atom feed, and the recent posts of a TikTok profile. Wire the result into a [Prompt](https://nodaro.ai/docs/nodes/automate/prompt) node to summarize or rewrite it, or into [Extract Field](https://nodaro.ai/docs/nodes/automate/extract-field) to run one generation per item.

- Found in: Automate › Get Content
- Output: data
- API type: `web-scrape`

## When to use it
- Pull the search results for a keyword into a content workflow.
- Read a web page or a site as Markdown, and let a Prompt node summarize it or extract facts.
- Read the newest items of a news site, a blog or a YouTube channel, on a schedule.
- Collect the recent posts of a TikTok profile for analysis or new ideas.

## Quick start
### Add the node

Press Tab on the canvas and choose **Automate › Get Content › Web Scrape**.

### Choose a source

Open the settings panel, choose a **Source** and fill in its fields, for example a **Query** for Google Search. The **Run** button shows the price of that source.

### Run it

Click **Run**. The first items appear on the node, and the **Results** tab of the settings panel lists them all.

### Use the items

Connect the **JSON** output to a [Prompt](https://nodaro.ai/docs/nodes/automate/prompt) node, which reads the JSON as text, or to [Extract Field](https://nodaro.ai/docs/nodes/automate/extract-field) to take one field from every item.

Workflow: Every morning the Schedule Trigger runs the whole workflow: Web Scrape reads a news feed, a Prompt node writes a summary, and Telegram Post sends it to a channel.

- Web Scrape → Prompt (prompt)
- Prompt → Telegram Post

## Sources

| Source | What it returns | Fields |
| --- | --- | --- |
| **Google Search** | Up to 10 search results for a query | **Query**. **Max results**, from 1 to 10, default 5. **Country code**, a 2-letter code such as `us`, to localize the results. |
| **Website Content (Markdown)** | The text of one page, or of up to 20 pages of a site, as Markdown | **Start URL**, the address as you type it in a browser. `https://` is optional. **Crawl mode**: **Single page** (the default) or **Site crawl, up to 20 pages**, which follows the links of the site from the start address. |
| **RSS Feed** | The items of an RSS or Atom feed | **Feed URL**, with or without `https://`. **Results limit**, from 1 to 50, default 10. |
| **TikTok** | The recent posts of a profile, or one post | **Profile or post URL**, with or without `https://`. **Results limit**, from 1 to 20, default 10. |

Every text field accepts `{}`. When a node is connected to the **URL / Query** input, its text replaces `{}`. A [Text](https://nodaro.ai/docs/nodes/automate/text) or [List](https://nodaro.ai/docs/nodes/automate/list) node can then drive the query or the address.

## RSS and Atom feeds

The **RSS Feed** source reads both feed formats: RSS 2.0, with the older 0.9x and 1.0 versions, and Atom. That includes YouTube channel feeds (`https://www.youtube.com/feeds/videos.xml?channel_id=...`), GitHub `releases.atom` feeds and the feeds that most blog platforms publish.

Every item has the same five fields, whatever the format of the feed:

| Field | Contains |
| --- | --- |
| `title` | The title of the item |
| `url` | The link to the item. Empty when the item has no link. |
| `description` | The summary or the content of the item. It can contain HTML. |
| `pubDate` | The publication date, in ISO 8601 format when the date can be read |
| `guid` | The unique ID of the item, or its link when it has no ID |

When the feed's server has a temporary problem, the node tries again, up to three requests within 30 seconds. A run that runs out of time fails with the error "RSS fetch timed out". An address that returns something other than a feed, such as a web page or an error page, fails the run. The error says what the address returned.

## Results

After a run, the node shows the first few items. The **Results** tab of the settings panel lists every item, with a **List** view and a **Raw JSON** view, and the **Copy JSON** and **Download JSON** buttons. Every item links to its source, such as the search result's page, the feed item or the TikTok video. Check a result there before you build on it.

The output, **JSON**, is the list of items. Three common ways to use it:

- **Summarize or rewrite it.** Connect it to the **Prompt** input of a [Prompt](https://nodaro.ai/docs/nodes/automate/prompt) node. The JSON arrives as text.
- **Take one field.** Connect it to [Extract Field](https://nodaro.ai/docs/nodes/automate/extract-field), for example to get the `title` of every feed item as a list.
- **Run once per item.** Send the list of extracted values to a generation node, which runs once for every value. See [Lists and batching](https://nodaro.ai/docs/concepts/lists-and-batching).

## Credits
| Source | Credits per run |
| --- | --- |
| Google Search | 30 |
| Website Content, single page | 10 |
| Website Content, site crawl | 50 |
| RSS Feed | 10 |
| TikTok | 10 |

The **Run** button on the node, the price next to each crawl mode and the total of a workflow run all show the price you pay.

- **A failed run is refunded in full.**
- **A run that finds nothing is charged.** It completed, so it costs the price of its source, and the node keeps its last good result next to the empty one. For the RSS Feed source, a real feed with no items counts as finding nothing.

## Long crawls

A site crawl follows up to 20 pages and often takes several minutes. In the editor and in a workflow run, you do not need to do anything. The node waits, and the pages appear on the node when the crawl finishes. If you reopen the workflow after the crawl finished in the background, the result is there too.

On a self-hosted install, the Google Search, Website Content and TikTok sources need the web-scraping key described in [Provider keys](https://nodaro.ai/docs/self-hosting/provider-keys), or a [connection to Nodaro Cloud](https://nodaro.ai/docs/self-hosting/cloud-connect). The RSS Feed source needs neither.

## Tips
- **Start with one page.** Use **Single page** to check that a site returns useful text before you pay for a **Site crawl**.
- **Poll on a schedule.** Add a [Schedule Trigger](https://nodaro.ai/docs/nodes/automate/schedule-trigger) to the workflow for a daily digest or a news-driven workflow.
- **Keep only what you need.** Extract one field, such as `title`, before a node that runs once per item, so that every run gets a short, clean text.

## From the API
`POST /v1/web-scrape` runs the node from code. A site crawl can take longer than the limit of about 100 seconds on a single HTTP request, so send `"respondAsync": true` for crawls. The call then answers at once with a `jobId`, and you read the pages from the completed job with `GET /v1/jobs/:id`. Without the flag, the request stays open until the scrape is done. A crawl that outlasts the request limit is cut off on the way back, although the job completes and is charged. See [Run a single node](https://nodaro.ai/docs/developers/api/nodes) and [Jobs](https://nodaro.ai/docs/developers/api/jobs).

## Frequently asked questions

### What can the Web Scrape node fetch?

Google Search results, the text of one web page or of up to 20 pages of a site as Markdown, the items of an RSS or Atom feed, and the recent posts of a TikTok profile. Every source returns a JSON list of items.

### How many credits does Web Scrape cost?

It depends on the source. Google Search costs 30 credits per run, one web page 10, a site crawl of up to 20 pages 50, an RSS or Atom feed 10, and TikTok 10. A failed run is refunded in full.

### Can Web Scrape read a whole website?

Yes. Choose Website Content (Markdown) and set Crawl mode to Site crawl. The node follows the links of the site from the start address, up to 20 pages, and returns each page as Markdown.

### How do I check a feed or a search every day?

Add a Schedule Trigger to the workflow, set the schedule, for example every day at 9:00, and turn the schedule on. Each scheduled run fetches fresh results and runs the nodes after Web Scrape.

### What happens when the scrape finds nothing?

The run completes and is charged, and the node keeps its last good result next to the empty one. For a feed, an address that returns something other than a feed fails the run instead, and the credits are refunded.