How To Remove Duplicates From the List In Python – Definitive Guide

Published by on

Python lists allow you to store multiple items in a single object.

You can remove duplicates from the List in Python using the list(set(your_list_name)) statement.

In this tutorial, you’ll learn the different methods to remove duplicates from the list in Python and when it is appropriate to use them.

If You’re in Hurry…

You can remove duplicates from the list in python by converting it into the set and converting it back to the list.

num_list = [10, 20,30,30,40,50,50]

unique_numbers = list(set(num_list))

unique_numbers

The new list contains only the unique elements, and the duplicates are removed.

Output

[40, 10, 50, 20, 30]

If You Want to Understand Details, Read on…

Different methods are available such as set() , dict.keys() and some libraries like NumPy to remove duplicates.

All the methods don’t preserve the order while removing duplicates. Hence, you’ll also see how to preserve the list item’s order while deleting duplicates.

Using Set() method

You can remove the duplicates from the list by using the set() constructor. While creating a set out of the list, the set will add only the unique items. The duplicates will be removed.

With the resultant set, you can again create a list. That list will contain only the unique items.

This is the fastest way to remove the duplicates from the list when the list is not too extensive.

The order of items will NOT be preserved while using this method. Hence, if you want the order to be preserved, you need to use the alternate methods in this tutorial.

Code

The code below demonstrates how to remove the duplicates from the list using the set() method.

num_list = [10, 20,30,30,40,50,50]

unique_numbers = list(set(num_list))

unique_numbers

Output

The new list will contain only the unique items.

    [40, 10, 50, 20, 30]

Using Dict.Keys Method

The second method to remove duplicates from the list is to use the dict.fromkeys() method.

The fromkeys() method in the dict class creates a new dictionary with the keys from the supplied list.

The dictionary keys are always unique. Hence, the duplicate values are deleted while creating a dictionary from the list of values.

Once you have the dictionary, you can convert it into a list using the list constructor.

The order of items will be preserved while using this method. Use this method when you want the order of the items to be preserved.

Code

The code below demonstrates how to use the dict.fromkeys() method to remove duplicates from the list.

lst = [20, 10, 20, 30,40]

unique_lst = list(dict.fromkeys(lst))

unique_lst

Output

The new list will contain only the unique items, and the item order is also preserved.

    [20, 10, 30, 40]

Using Numpy Unique

You can use the numpy.unique() method to remove duplicate values from a list.

The unique() method returns the unique items from the array.

  1. You need to convert the list to a NumPy array using np.array() method.
  2. Then, use the NumPy array with the np.unique() method to get the unique values.

This method will also not preserve the order of the items.

Code

The code below demonstrates how to convert a list to a NumPy array and get unique values from it.

import numpy as np

num_list = [10, 20,30,30,40,50,50]

x = np.array(num_list)

np.unique(x)

Output

The new list will contain only the unique items, and the item order is also preserved.

    array([10, 20, 30, 40, 50])

Remove Duplicates and Preserve Order

In this section, you’ll learn how to remove duplicates from the list in python and **preserve the order of the items. **

You’ll use the list comprehension and set() to remove the duplicates.

  1. Create a list with duplicate items
  2. Create an empty set to store the unique values
  3. Use the list comprehension to iterate over the list. Check if the current iterated item is in the list. If it exists, do nothing. If it is NOT available, add it to the Set and return True to the list comprehension. Hence the item will be added to the output list created by the list comprehension.

Code

The code below demonstrates how to use the list comprehension method to delete duplicates from the list in Python.

num_list = [5, 10, 5, 10, 40, 20, 30]

unique_set = set()

unique = [x for x in num_list if not (x in unique_set or unique_set.add(x))]

unique

Output

    [5, 10, 40, 20, 30]

Remove Duplicates From List Using For Loop

This section explains how to remove duplicates from the List using the for loop. This method also uses the list comprehension method to create a list of unique items.

Code

num_list = [5, 10, 5, 10, 40, 20, 30]

unique_set = set()

unique = [x for x in num_list if not (x in unique_set or unique_set.add(x))]

unique

Output

    [5, 10, 40, 20, 30]

Conclusion

You’ve learned the different methods to remove duplicates from the Python list. Additionally, you’ve learned which methods preserve the item order and which don’t.

You May Also Like

Leave a Comment