What can the Web Scrape node fetch?

Google Search results, the text of one web page or of up to 20 pages of a site as Markdown, the items of an RSS or Atom feed, and the recent posts of a TikTok profile. Every source returns a JSON list of items.

How many credits does Web Scrape cost?

It depends on the source. Google Search costs 30 credits per run, one web page 10, a site crawl of up to 20 pages 50, an RSS or Atom feed 10, and TikTok 10. A failed run is refunded in full.

Can Web Scrape read a whole website?

Yes. Choose Website Content (Markdown) and set Crawl mode to Site crawl. The node follows the links of the site from the start address, up to 20 pages, and returns each page as Markdown.

How do I check a feed or a search every day?

Add a Schedule Trigger to the workflow, set the schedule, for example every day at 9:00, and turn the schedule on. Each scheduled run fetches fresh results and runs the nodes after Web Scrape.

What happens when the scrape finds nothing?

The run completes and is charged, and the node keeps its last good result next to the empty one. For a feed, an address that returns something other than a feed fails the run instead, and the credits are refunded.

Web Scrape node — Google Search, website text, RSS feeds and TikTok posts as JSON

Fetch Google Search results, the text of a web page or site, RSS or Atom feed items, or TikTok posts, and pass them on as JSON to AI and list nodes.

The Web Scrape node fetches content from the web and returns it as a JSON list of items. It has four sources: Google Search results, the text of a web page or a whole site, the items of an RSS or Atom feed, and the recent posts of a TikTok profile. Wire the result into a Prompt node to summarize or rewrite it, or into Extract Field to run one generation per item.

When to use it

Pull the search results for a keyword into a content workflow.
Read a web page or a site as Markdown, and let a Prompt node summarize it or extract facts.
Read the newest items of a news site, a blog or a YouTube channel, on a schedule.
Collect the recent posts of a TikTok profile for analysis or new ideas.

Quick start

Add the node

Press Tab on the canvas and choose Automate › Get Content › Web Scrape.

Choose a source

Open the settings panel, choose a Source and fill in its fields, for example a Query for Google Search. The Run button shows the price of that source.

Run it

Click Run. The first items appear on the node, and the Results tab of the settings panel lists them all.

Use the items

Connect the JSON output to a Prompt node, which reads the JSON as text, or to Extract Field to take one field from every item.

Every morning the Schedule Trigger runs the whole workflow: Web Scrape reads a news feed, a Prompt node writes a summary, and Telegram Post sends it to a channel.

Sources

Source	What it returns	Fields
Google Search	Up to 10 search results for a query	Query. Max results, from 1 to 10, default 5. Country code, a 2-letter code such as `us`, to localize the results.
Website Content (Markdown)	The text of one page, or of up to 20 pages of a site, as Markdown	Start URL, the address as you type it in a browser. `https://` is optional. Crawl mode: Single page (the default) or Site crawl, up to 20 pages, which follows the links of the site from the start address.
RSS Feed	The items of an RSS or Atom feed	Feed URL, with or without `https://`. Results limit, from 1 to 50, default 10.
TikTok	The recent posts of a profile, or one post	Profile or post URL, with or without `https://`. Results limit, from 1 to 20, default 10.

Every text field accepts {}. When a node is connected to the URL / Query input, its text replaces {}. A Text or List node can then drive the query or the address.

RSS and Atom feeds

The RSS Feed source reads both feed formats: RSS 2.0, with the older 0.9x and 1.0 versions, and Atom. That includes YouTube channel feeds (https://www.youtube.com/feeds/videos.xml?channel_id=...), GitHub releases.atom feeds and the feeds that most blog platforms publish.

Every item has the same five fields, whatever the format of the feed:

Field	Contains
`title`	The title of the item
`url`	The link to the item. Empty when the item has no link.
`description`	The summary or the content of the item. It can contain HTML.
`pubDate`	The publication date, in ISO 8601 format when the date can be read
`guid`	The unique ID of the item, or its link when it has no ID

When the feed's server has a temporary problem, the node tries again, up to three requests within 30 seconds. A run that runs out of time fails with the error "RSS fetch timed out". An address that returns something other than a feed, such as a web page or an error page, fails the run. The error says what the address returned.

Results

After a run, the node shows the first few items. The Results tab of the settings panel lists every item, with a List view and a Raw JSON view, and the Copy JSON and Download JSON buttons. Every item links to its source, such as the search result's page, the feed item or the TikTok video. Check a result there before you build on it.

The output, JSON, is the list of items. Three common ways to use it:

Summarize or rewrite it. Connect it to the Prompt input of a Prompt node. The JSON arrives as text.
Take one field. Connect it to Extract Field, for example to get the title of every feed item as a list.
Run once per item. Send the list of extracted values to a generation node, which runs once for every value. See Lists and batching.

Credits

Source	Credits per run
Google Search	30
Website Content, single page	10
Website Content, site crawl	50
RSS Feed	10
TikTok	10

The Run button on the node, the price next to each crawl mode and the total of a workflow run all show the price you pay.

A failed run is refunded in full.
A run that finds nothing is charged. It completed, so it costs the price of its source, and the node keeps its last good result next to the empty one. For the RSS Feed source, a real feed with no items counts as finding nothing.

Long crawls

A site crawl follows up to 20 pages and often takes several minutes. In the editor and in a workflow run, you do not need to do anything. The node waits, and the pages appear on the node when the crawl finishes. If you reopen the workflow after the crawl finished in the background, the result is there too.

On a self-hosted install, the Google Search, Website Content and TikTok sources need the web-scraping key described in Provider keys, or a connection to Nodaro Cloud. The RSS Feed source needs neither.

Tips

Start with one page. Use Single page to check that a site returns useful text before you pay for a Site crawl.
Poll on a schedule. Add a Schedule Trigger to the workflow for a daily digest or a news-driven workflow.
Keep only what you need. Extract one field, such as title, before a node that runs once per item, so that every run gets a short, clean text.

From the API

POST /v1/web-scrape runs the node from code. A site crawl can take longer than the limit of about 100 seconds on a single HTTP request, so send "respondAsync": true for crawls. The call then answers at once with a jobId, and you read the pages from the completed job with GET /v1/jobs/:id. Without the flag, the request stays open until the scrape is done. A crawl that outlasts the request limit is cut off on the way back, although the job completes and is charged. See Run a single node and Jobs.

Web Scrape