# Remove Duplicates

> Remove repeated items from a list and keep the first copy of each. Compare whole items or one field, such as id or url, in scraped or merged lists.

Source: https://nodaro.ai/docs/nodes/automate/remove-duplicates

The **Remove Duplicates** node drops the repeated items of a list and keeps the first copy of each. It compares whole items by default, or one field inside each item, such as `id` or `url`, which suits scraped records. The remaining items keep their order. The node is free.

- Found in: Automate › Lists & Batching
- Output: data
- API type: `deduplicate`

## When to use it
- A scrape returns the same post more than once. Drop the copies by their `id` or `url`.
- You merged two lists that overlap and want each item once.
- You want each prompt of a batch to run only once.

## Quick start
### Add the node

Press Tab on the canvas and choose **Automate › Lists & Batching › Remove Duplicates**.

### Connect a list

Wire a list into the **List** input, for example from [Web Scrape](https://nodaro.ai/docs/nodes/automate/web-scrape), [Merge Lists](https://nodaro.ai/docs/nodes/automate/merge-lists), [Filter List](https://nodaro.ai/docs/nodes/automate/filter-list), [List](https://nodaro.ai/docs/nodes/automate/list) or [Split Text](https://nodaro.ai/docs/nodes/automate/split-text).

### Choose what to compare

Open the settings panel. Leave **Deduplicate by field** on **(whole item)** to compare full items, or choose the field that identifies an item.

### Run it

The **Unique** output carries the list without repeats. After a run, the panel previews the unique items and shows how many there are.

Workflow: Two searches are merged into one list, and Remove Duplicates keeps each result once by comparing its url.

- Web Scrape → Merge Lists (lists)
- Web Scrape → Merge Lists (lists)
- Merge Lists → Remove Duplicates (list)

## Inputs and outputs
| Input or output | Accepts or sends | What it does |
| --- | --- | --- |
| **List** | A list of text or JSON items | The list to clean. |
| **Unique** | A list | The first copy of every item, in the original order. |

## Settings
| Setting | What it does |
| --- | --- |
| **Deduplicate by field** | What makes two items the same. **(whole item)**, the default, compares the full items as text. Choose a field detected from the upstream data, or **Custom path…** to type a dot path such as `id` or `metadata.url`. Each item is read as JSON when the path matches it. |

## Credits
Remove Duplicates is free.

## Tips
- **Compare by an identifier.** Two scrapes of the same post can differ in small details, such as a like count. Compare by `id` or `url` so that they still count as one post.
- **Merge and clean in one step.** When you only need to merge lists and drop exact repeats, enable **Remove duplicates after merge** in [Merge Lists](https://nodaro.ai/docs/nodes/automate/merge-lists).

## Frequently asked questions

### Which copy of a repeated item does Remove Duplicates keep?

The first one. Remove Duplicates walks the list from the top, keeps the first item with each value, and drops every later copy. The remaining items keep their order.

### How do I remove duplicates by one field?

Set Deduplicate by field to the field, such as id or url, or choose Custom path… and type a dot path such as metadata.url. Two items with the same value in that field count as duplicates, even when their other fields differ.

### Does Remove Duplicates cost credits?

No. Remove Duplicates is free.

### Can Merge Lists remove duplicates too?

Yes. Merge Lists has a Remove duplicates after merge switch that drops repeated items by value. Use Remove Duplicates when you need to compare one field instead of the whole item.