S3 is a simple storage service provided by Amazon. A key uniquely identifies an object in an S3 bucket.
You can check if a key exists in an S3 bucket using the list_objects() method.
Any sub-object (subfolders) created under an S3 bucket is also identified using the key.
In this tutorial, you’ll learn the different methods available to check if a key exists in an S3 bucket using Boto3 Python.
The different methods are
- S3 Client List_objects() method
- S3 Object Load() method
- The S3fs exists() method
Prerequisites
Boto3
– Additional package to be installed(Explained below)- AWS Credentials – You can Generate the security credentials by clicking Your Profile Name -> My Security Credentials -> Access keys (access key ID and secret access key) option. This is necessary to create a session with your AWS account.
- Bucket_Name – Target S3 bucket name where you want to check if a key exists or not.
- Key – Name of the object that needs to be searched for.
Installing Boto3
If you’ve not installed boto3 yet, you can install it using the snippet below.
You can use the % symbol before pip to install packages directly from the Jupyter notebook instead of launching the Anaconda Prompt.
Snippet
%pip install boto3
Boto3 will be installed successfully.
Now, you can use it to access AWS resources.
Boto3 provides client, resource objects to interact with AWS. Read the difference between client and resources to understand when to use these appropriately.
Using List_objects_v2() Method in Boto3 Client
In this section, you’ll learn how to use the boto3 client to check if the key exists in the S3 bucket.
- list_objects_v2() method allows you to list all the objects in a bucket.
- Pass the
key
you want to check for existence using theprefix
parameter. - Objects with this prefix will be filtered in the results.
Returns
List_objects_v2()
returns a dictionary with multiple keys in it.- If the key you’ve searched for doesn’t exist in the S3 bucket, the response dictionary object will not have a key called
Contents
in it.
Code
Using the response
dictionary, you can check if the Contents
key is available to check if the key exists in the S3 bucket, as shown in the following code.
import boto3
bucket='stackvidhya'
file_key = 'csv_files/IRIS.csv'
s3_client = boto3.client('s3', aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>')
result = s3_client.list_objects_v2(Bucket=bucket, Prefix=file_key)
if 'Contents' in result:
print("Key exists in the bucket.")
else:
print("Key doesn't exist in the bucket.")
Output
Key exists in the bucket.
This is how you can use the list_object_v2()
method to check if a key exists in an S3 bucket using the Boto3 client.
Using S3 Object.Load() method in Boto3 Resource
In this section, you’ll learn how to check if a key exists in the S3 bucket using the Boto3 resource.
Boto3 resource doesn’t provide any method directly to check if the key exists in the S3 bucket.
Hence, you can load the S3 object using the load()
method.
- If there is no exception thrown, then the key exists.
- If there is a client error thrown and the error code is
404
, then the key doesn’t exist in the bucket.
While using this method, the program’s control flow is handled through exceptions, which is not recommended.
Code
Use the following code to check if a key exists in an S3 bucket using the Boto3 resource.
import boto3
bucket='stackvidhya'
file_key = 'csv_files/IRIS.csv'
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)
s3 = session.resource('s3')
try:
s3.Object(bucket, file_key).load()
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "404":
print("Object Doesn't exists")
else:
print("Error occurred while fetching a file from S3. Try Again.")
else:
print("Object Exists")
Output
Object Exists
This is how you can check if a key exists in an S3 bucket using Boto3.
Using S3FS
If you want to check if a key exists in the S3 bucket in Python without using Boto3, you can use the S3FS interface.
S3Fs is a Pythonic file interface to S3. It builds on top of botocore.
It provides a method exists() to check if a key exists in the S3 bucket.
It returns,
True
– If the key exists in the S3 bucketFalse
– If the key doesn’t exist in the S3 bucket
Installing S3Fs
pip install s3fs
S3Fs
is installed.
Use the code below to check if a key exists in the S3 bucket using the S3Fs
interface.
Code
import s3fs
bucket='stackvidhya'
file_key = 'csv_files/IRIS.csv'
s3 = s3fs.S3FileSystem(anon=False, key='<your_access_key_id>', secret='<your_secret_access_key>')
path = bucket + '/' + file_key
if s3.exists(path):
print('File Exists')
else:
print('File doesn't exists')
Output
File Exists
You can also use this method to Check if S3 URI exists.
S3 URI will look like this s3://bucket_name//object_name.extension
. You can generate this URL using the copy URI option available in the AWS S3 console.
This is the alternative method to check if a key exists in the S3 bucket using Python.
Conclusion
To summarise, you’ve learned how to check if a key exists in the S3 bucket using the Boto3 library.
The same steps can also be used to
- check if a prefix exists in the S3 bucket.
- check if a folder exists inside an S3 bucket.
If you have any questions, comment below.
Hi, if the “same key exist” as a question this guide is okay. But same key but newer version of it i have my local. How can i check the key with the same version exist?
Hi,
If the key exists, you can check if the contents of the file in S3 and the contents of the file in your local directory are the same.
For comparing text files, you can open the S3 file as a string using https://140.82.22.9/open-s3-object-as-string-in-boto3/ and open the local file using https://140.82.22.9/python-read-file-line-by-line/ and do the python string comparison.
If the file is a binary file, read how to open and read binary file in python https://140.82.22.9/python-read-binary-file/
Hope this helps.
Further questions are welcome.
Regards,
Vikram