How to Check if a key exists in an S3 bucket using Boto3 Python?

S3 is a simple storage service provided by Amazon. A key uniquely identifies an object in an S3 bucket.

You can check if a key exists in an S3 bucket using the list_objects() method.

Any sub-object (subfolders) created under an S3 bucket is also identified using the key.

In this tutorial, you’ll learn the different methods available to check if a key exists in an S3 bucket using Boto3 Python.

The different methods are

• S3 Client List_objects() method
• S3 Object Load() method
• The S3fs exists() method

Prerequisites

• Boto3 – Additional package to be installed(Explained below)
• AWS Credentials – You can Generate the security credentials by clicking Your Profile Name -> My Security Credentials -> Access keys (access key ID and secret access key) option. This is necessary to create a session with your AWS account.
• Bucket_Name – Target S3 bucket name where you want to check if a key exists or not.
• Key – Name of the object that needs to be searched for.

Installing Boto3

If you’ve not installed boto3 yet, you can install it using the snippet below.

You can use the % symbol before pip to install packages directly from the Jupyter notebook instead of launching the Anaconda Prompt.

Snippet

%pip install boto3

Boto3 will be installed successfully.

Now, you can use it to access AWS resources.

Boto3 provides client, resource objects to interact with AWS. Read the difference between client and resources to understand when to use these appropriately.

Using List_objects_v2() Method in Boto3 Client

In this section, you’ll learn how to use the boto3 client to check if the key exists in the S3 bucket.

• list_objects_v2() method allows you to list all the objects in a bucket.
• Pass the key you want to check for existence using the prefix parameter.
• Objects with this prefix will be filtered in the results.

Returns

• List_objects_v2() returns a dictionary with multiple keys in it.
• If the key you’ve searched for doesn’t exist in the S3 bucket, the response dictionary object will not have a key called Contents in it.

Code

Using the response dictionary, you can check if the Contents key is available to check if the key exists in the S3 bucket, as shown in the following code.

import boto3

bucket='stackvidhya'

file_key = 'csv_files/IRIS.csv'

s3_client = boto3.client('s3', aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>')

result = s3_client.list_objects_v2(Bucket=bucket, Prefix=file_key)

if 'Contents' in result:
    print("Key exists in the bucket.")
else:
    print("Key doesn't exist in the bucket.")

Output

Key exists in the bucket.

This is how you can use the list_object_v2() method to check if a key exists in an S3 bucket using the Boto3 client.

Using S3 Object.Load() method in Boto3 Resource

In this section, you’ll learn how to check if a key exists in the S3 bucket using the Boto3 resource.

Boto3 resource doesn’t provide any method directly to check if the key exists in the S3 bucket.

Hence, you can load the S3 object using the load() method.

• If there is no exception thrown, then the key exists.
• If there is a client error thrown and the error code is 404, then the key doesn’t exist in the bucket.

While using this method, the program’s control flow is handled through exceptions, which is not recommended.

Code

Use the following code to check if a key exists in an S3 bucket using the Boto3 resource.

import boto3

bucket='stackvidhya'

file_key = 'csv_files/IRIS.csv'

session = boto3.Session(
    aws_access_key_id='<your_access_key_id>',
    aws_secret_access_key='<your_secret_access_key>'
)

s3 = session.resource('s3')

try:
    s3.Object(bucket, file_key).load()

except botocore.exceptions.ClientError as e:

    if e.response['Error']['Code'] == "404":

        print("Object Doesn't exists")

    else:

        print("Error occurred while fetching a file from S3. Try Again.")


else:
    print("Object Exists")

Output

Object Exists

This is how you can check if a key exists in an S3 bucket using Boto3.

Using S3FS

If you want to check if a key exists in the S3 bucket in Python without using Boto3, you can use the S3FS interface.

S3Fs is a Pythonic file interface to S3. It builds on top of botocore.

It provides a method exists() to check if a key exists in the S3 bucket.

It returns,

• True – If the key exists in the S3 bucket
• False – If the key doesn’t exist in the S3 bucket

Installing S3Fs

pip install s3fs

S3Fs is installed.

Use the code below to check if a key exists in the S3 bucket using the S3Fs interface.

Code

import s3fs

bucket='stackvidhya'

file_key = 'csv_files/IRIS.csv'

s3 = s3fs.S3FileSystem(anon=False, key='<your_access_key_id>', secret='<your_secret_access_key>')

path = bucket + '/' + file_key

if s3.exists(path):
    print('File Exists')
else:
    print('File doesn't exists')

Output

File Exists

You can also use this method to Check if S3 URI exists.

S3 URI will look like this s3://bucket_name//object_name.extension . You can generate this URL using the copy URI option available in the AWS S3 console.

This is the alternative method to check if a key exists in the S3 bucket using Python.

Conclusion

To summarise, you’ve learned how to check if a key exists in the S3 bucket using the Boto3 library.

The same steps can also be used to

• check if a prefix exists in the S3 bucket.
• check if a folder exists inside an S3 bucket.

If you have any questions, comment below.

Additional Resources

• How To Load Data From AWS S3 Into Sagemaker (Using Boto3 Or AWSWrangler)
• How To Write A File Or Data To An S3 Object Using Boto3
• How to List Contents of s3 Bucket Using Boto3 Python
• How To Write Pandas Dataframe As CSV To S3 Using Boto3 Python