How To Remove Duplicates From the List In Python – Definitive Guide

Published on: | Last updated on: January 12, 2023

Python lists allow you to store multiple items in a single object.

You can remove duplicates from the List in Python using the list(set(your_list_name)) statement.

Basic Example

num_list = [10, 20,30,30,40,50,50]

unique_numbers = list(set(num_list))

unique_numbers

The new list contains only the unique elements, and the duplicates are removed.

Output

[40, 10, 50, 20, 30]

This tutorial teaches you the different methods to remove duplicates from the list in Python and when it is appropriate to use them.

Using Set() method

You can remove the duplicates from the list by using the set() constructor.

  • Create a set out of the list, the set will add only the unique items
  • The duplicates will be removed
  • With the resultant set, create a list. The new will contain only the unique items

This is the fastest way to remove the duplicates from the list when the list is not too extensive.

The order of items will NOT be preserved while using this method. Hence, if you want the order to be preserved, you need to use the alternate methods in this tutorial.

Code

num_list = [10, 20,30,30,40,50,50]

unique_numbers = list(set(num_list))

unique_numbers

Output

The new list will contain only the unique items.

    [40, 10, 50, 20, 30]

Using Dict.Keys Method

The fromkeys() method in the dict class creates a new dictionary with the keys from the supplied list.

  • The dictionary keys are always unique.
  • The duplicate values in the list are deleted while creating a dictionary from the list of values.
  • Convert the dictionary keys into a list using the list constructor.

The order of items will be preserved while using this method. Use this method when you want the order of the items to be preserved.

Code

lst = [20, 10, 20, 30,40]

unique_lst = list(dict.fromkeys(lst))

unique_lst

Output

The new list will contain only the unique items, and the item order is also preserved.

    [20, 10, 30, 40]

Using Numpy Unique

The unique() method returns the unique items from the array.

  1. Convert the list to a NumPy array using the np.array() method
  2. Use the NumPy array with the np.unique() method to get the unique values

This method will also NOT preserve the order of the items.

Code

import numpy as np

num_list = [10, 20,30,30,40,50,50]

x = np.array(num_list)

np.unique(x)

Output

The new list will contain only the unique items, and the item order is also preserved.

    array([10, 20, 30, 40, 50])

Remove Duplicates and Preserve Order

In this section, you’ll learn how to remove duplicates from the list in python and preserve the order of the items.

Use the list comprehension and set() to remove the duplicates.

  1. Create a list with duplicate items
  2. Create an empty set to store the unique values
  3. Use the list comprehension to iterate over the list. Check if the current iterated item is on the list
  4. If it exists, do nothing
  5. If it is NOT available, add it to the Set and return True to the list comprehension. The item will be added to the output list created by the list comprehension.

Code

num_list = [5, 10, 5, 10, 40, 20, 30]

unique_set = set()

unique = [x for x in num_list if not (x in unique_set or unique_set.add(x))]

unique

Output

    [5, 10, 40, 20, 30]

Remove Duplicates From List Using For Loop

This section explains how to remove duplicates from the List using the for loop.

This method also uses the list comprehension method to create a list of unique items.

Code

num_list = [5, 10, 5, 10, 40, 20, 30]

unique_set = set()

unique = [x for x in num_list if not (x in unique_set or unique_set.add(x))]

unique

Output

    [5, 10, 40, 20, 30]

Additional Resources

Leave a Comment