Nodaro DocsRemove Duplicates
Remove repeated items from a list and keep the first copy of each. Compare whole items or one field, such as id or url, in scraped or merged lists.
The Remove Duplicates node drops the repeated items of a list and keeps the first copy of each. It compares whole items by default, or one field inside each item, such as id or url, which suits scraped records. The remaining items keep their order. The node is free.
When to use it
- A scrape returns the same post more than once. Drop the copies by their
idorurl. - You merged two lists that overlap and want each item once.
- You want each prompt of a batch to run only once.
Quick start
Add the node
Press Tab on the canvas and choose Automate › Lists & Batching › Remove Duplicates.
Connect a list
Wire a list into the List input, for example from Web Scrape, Merge Lists, Filter List, List or Split Text.
Choose what to compare
Open the settings panel. Leave Deduplicate by field on (whole item) to compare full items, or choose the field that identifies an item.
Run it
The Unique output carries the list without repeats. After a run, the panel previews the unique items and shows how many there are.
Inputs and outputs
| Input or output | Accepts or sends | What it does |
|---|---|---|
| List | A list of text or JSON items | The list to clean. |
| Unique | A list | The first copy of every item, in the original order. |
Settings
| Setting | What it does |
|---|---|
| Deduplicate by field | What makes two items the same. (whole item), the default, compares the full items as text. Choose a field detected from the upstream data, or Custom path… to type a dot path such as id or metadata.url. Each item is read as JSON when the path matches it. |
Credits
Remove Duplicates is free.
Tips
- Compare by an identifier. Two scrapes of the same post can differ in small details, such as a like count. Compare by
idorurlso that they still count as one post. - Merge and clean in one step. When you only need to merge lists and drop exact repeats, enable Remove duplicates after merge in Merge Lists.
Frequently asked questions
Related
Merge Lists
Filter List
Sort List
Web Scrape
Lists and batching
Last updated on
Sort List
Put a list in order by the whole item or by a field such as score or created_at, comparing as text, numbers or dates, ascending or descending.
Merge Lists
Combine several lists into one. Concatenate appends them in connection order, and Zip merges the items that share a position. Optionally drop duplicates.