Python lists allow you to store multiple items in a single object.
You can remove duplicates from the List in Python using the list(set(your_list_name)) statement.
In this tutorial, you’ll learn the different methods to remove duplicates from the list in Python and when it is appropriate to use them.
If You’re in Hurry…
You can remove duplicates from the list
in python by converting it into the set
and converting it back to the list
.
num_list = [10, 20,30,30,40,50,50]
unique_numbers = list(set(num_list))
unique_numbers
The new list contains only the unique elements, and the duplicates are removed.
Output
[40, 10, 50, 20, 30]
If You Want to Understand Details, Read on…
Different methods are available such as set()
, dict.keys()
and some libraries like NumPy
to remove duplicates.
All the methods don’t preserve the order while removing duplicates. Hence, you’ll also see how to preserve the list item’s order while deleting duplicates.
Table of Contents
Using Set() method
You can remove the duplicates from the list by using the set() constructor. While creating a set
out of the list
, the set
will add only the unique items. The duplicates will be removed.
With the resultant set
, you can again create a list. That list will contain only the unique items.
This is the fastest way to remove the duplicates from the list
when the list is not too extensive.
The order of items will NOT be preserved while using this method. Hence, if you want the order to be preserved, you need to use the alternate methods in this tutorial.
Code
The code below demonstrates how to remove the duplicates from the list using the set()
method.
num_list = [10, 20,30,30,40,50,50]
unique_numbers = list(set(num_list))
unique_numbers
Output
The new list will contain only the unique items.
[40, 10, 50, 20, 30]
Using Dict.Keys Method
The second method to remove duplicates from the list is to use the dict.fromkeys() method.
The fromkeys()
method in the dict
class creates a new dictionary with the keys from the supplied list.
The dictionary keys are always unique. Hence, the duplicate values are deleted while creating a dictionary from the list of values.
Once you have the dictionary, you can convert it into a list
using the list
constructor.
The order of items will be preserved while using this method. Use this method when you want the order of the items to be preserved.
Code
The code below demonstrates how to use the dict.fromkeys()
method to remove duplicates from the list
.
lst = [20, 10, 20, 30,40]
unique_lst = list(dict.fromkeys(lst))
unique_lst
Output
The new list will contain only the unique items, and the item order is also preserved.
[20, 10, 30, 40]
Using Numpy Unique
You can use the numpy.unique() method to remove duplicate values from a list
.
The unique()
method returns the unique items from the array.
- You need to convert the
list
to a NumPy array usingnp.array()
method. - Then, use the
NumPy
array with thenp.unique()
method to get the unique values.
This method will also not preserve the order of the items.
Code
The code below demonstrates how to convert a list
to a NumPy array
and get unique values from it.
import numpy as np
num_list = [10, 20,30,30,40,50,50]
x = np.array(num_list)
np.unique(x)
Output
The new list will contain only the unique items, and the item order is also preserved.
array([10, 20, 30, 40, 50])
Remove Duplicates and Preserve Order
In this section, you’ll learn how to remove duplicates from the list in python and **preserve the order of the items. **
You’ll use the list comprehension and set()
to remove the duplicates.
- Create a list with duplicate items
- Create an empty set to store the unique values
- Use the list comprehension to iterate over the
list
. Check if the current iterated item is in thelist
. If it exists, do nothing. If it is NOT available, add it to theSet
and returnTrue
to the list comprehension. Hence the item will be added to the output list created by the list comprehension.
Code
The code below demonstrates how to use the list comprehension method to delete duplicates from the list in Python.
num_list = [5, 10, 5, 10, 40, 20, 30]
unique_set = set()
unique = [x for x in num_list if not (x in unique_set or unique_set.add(x))]
unique
Output
[5, 10, 40, 20, 30]
Remove Duplicates From List Using For Loop
This section explains how to remove duplicates from the List
using the for
loop. This method also uses the list comprehension method to create a list of unique items.
Code
num_list = [5, 10, 5, 10, 40, 20, 30]
unique_set = set()
unique = [x for x in num_list if not (x in unique_set or unique_set.add(x))]
unique
Output
[5, 10, 40, 20, 30]
Conclusion
You’ve learned the different methods to remove duplicates from the Python list. Additionally, you’ve learned which methods preserve the item order and which don’t.
You May Also Like
- How to Remove an element from a List Using Index in Python
- How To Get The Last Element Of A List In Python
- How To Create An Empty List in Python with a Certain Size
- How To Count Occurrences Of A List Item In Python
- How To Sum a list of numbers in Python
- Difference Between Del, Remove and Pop methods on Lists in Python – Detailed Guide