> ## Documentation Index
> Fetch the complete documentation index at: https://docs.markifact.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Remove Duplicates

> Clean up lists by removing duplicate items to ensure unique values only.

**Remove Duplicates** takes a list and returns only unique items, eliminating any repeated values. Essential for cleaning data and ensuring accurate processing.

***

## When to Use It

* Clean campaign lists that may have duplicate entries
* Remove repeated URLs from sitemap extraction
* Deduplicate client lists from multiple sources
* Ensure unique keywords before processing

***

## Inputs

| Field    | Type | Required | Description                        |
| -------- | ---- | -------- | ---------------------------------- |
| **List** | List | Yes      | The list to remove duplicates from |

***

## Outputs

| Output          | Description                       |
| --------------- | --------------------------------- |
| **Unique List** | List with duplicate items removed |

***

## Credit Cost

Free to use - no credits required.

***

## Real-World Examples

**Clean Campaign Data:**

```
Google Ads Get Report → Remove Duplicates → Loop Over List
Before: ["Campaign A", "Campaign B", "Campaign A", "Campaign C", "Campaign B"]
After: ["Campaign A", "Campaign B", "Campaign C"]
```

**Deduplicate URL Lists:**

```
Extract URLs from Sitemap → Remove Duplicates → Count List Items
"Clean extracted URLs before processing to avoid duplicate work"
```

**Merge Client Lists:**

```
Multiple Sheets Read Data → Combine Lists → Remove Duplicates → Write to Sheets
"Merge client lists from different sources without duplicates"
```

**Keyword Cleaning:**

```
Generate List (from text) → Remove Duplicates → Loop Over List
"Process unique keywords only for campaign creation"
```

***

## How It Works

The node compares items and keeps only the first occurrence of each unique value:

**Example Process:**

```
Input List: ["apple", "banana", "apple", "cherry", "banana", "apple"]
Processing: Keeps first "apple", first "banana", first "cherry"
Output List: ["apple", "banana", "cherry"]
```

**Data Type Handling:**

* Text comparison is case-sensitive: "Apple" ≠ "apple"
* Numbers are compared by value: 123 = 123.0
* Empty values are treated as duplicates if multiple exist

***

## Tips

**Data Quality:**

* Always use this before loops to avoid processing the same item multiple times
* Helps reduce API calls and processing time

**List Merging:**

* Essential when combining data from multiple sources
* Prevents duplicate entries in final outputs

**Performance:**

* Reduces workflow execution time by eliminating redundant processing
* Especially important for large lists with many duplicates

**Case Sensitivity:**

* Remember that "Campaign A" and "campaign a" are different items
* Consider standardizing text case before deduplication if needed

***

## FAQ

<Accordion title="Does this preserve the original order of items?">
  Yes, the node keeps items in their original order. It removes duplicates but maintains the sequence of the first occurrence of each unique item.
</Accordion>

<Accordion title="How does it handle case sensitivity?">
  The comparison is case-sensitive, so "Apple" and "apple" are treated as different items. If you need case-insensitive deduplication, standardize the text case first.
</Accordion>

<Accordion title="What happens if my entire list contains duplicates?">
  You'll get a much shorter list with only unique items. For example, \["A", "A", "A", "B", "B"] becomes \["A", "B"].
</Accordion>

<Accordion title="Can this remove duplicates from complex data like campaign objects?">
  Yes, but it compares the entire data structure. Two campaign objects are only considered duplicates if every field matches exactly. For partial matching, use other filtering methods.
</Accordion>

<Accordion title="How do I know how many duplicates were removed?">
  Use Count List Items before and after Remove Duplicates, then subtract to see how many duplicates were found. This helps assess data quality.
</Accordion>

<Accordion title="Does this work with numbers and mixed data types?">
  Yes, it works with any data type. Numbers are compared by value (123 equals 123), text by exact match, and mixed lists handle each type appropriately.
</Accordion>