S3 is a simple storage service provided by Amazon. A key uniquely identifies an object in an S3 bucket.
You can check if a key exists in an S3 bucket using the list_objects() method.
Any sub-object (subfolders) created under an S3 bucket is also identified using the key.
In this tutorial, you’ll learn the different methods available to check if a key exists in an S3 bucket using Boto3 Python.
The different methods are
- S3 Client List_objects() method
- S3 Object Load() method
- The S3fs exists() method
Boto3– Additional package to be installed(Explained below)
- AWS Credentials – You can Generate the security credentials by clicking Your Profile Name -> My Security Credentials -> Access keys (access key ID and secret access key) option. This is necessary to create a session with your AWS account.
- Bucket_Name – Target S3 bucket name where you want to check if a key exists or not.
- Key – Name of the object that needs to be searched for.
If you’ve not installed boto3 yet, you can install it using the snippet below.
You can use the % symbol before pip to install packages directly from the Jupyter notebook instead of launching the Anaconda Prompt.
%pip install boto3
Boto3 will be installed successfully.
Now, you can use it to access AWS resources.
Boto3 provides client, resource objects to interact with AWS. Read the difference between client and resources to understand when to use these appropriately.
Using List_objects_v2() Method in Boto3 Client
In this section, you’ll learn how to use the boto3 client to check if the key exists in the S3 bucket.
- list_objects_v2() method allows you to list all the objects in a bucket.
- Pass the
keyyou want to check for existence using the
- Objects with this prefix will be filtered in the results.
List_objects_v2()returns a dictionary with multiple keys in it.
- If the key you’ve searched for doesn’t exist in the S3 bucket, the response dictionary object will not have a key called
response dictionary, you can check if the
Contents key is available to check if the key exists in the S3 bucket, as shown in the following code.
import boto3 bucket='stackvidhya' file_key = 'csv_files/IRIS.csv' s3_client = boto3.client('s3', aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>') result = s3_client.list_objects_v2(Bucket=bucket, Prefix=file_key) if 'Contents' in result: print("Key exists in the bucket.") else: print("Key doesn't exist in the bucket.")
Key exists in the bucket.
This is how you can use the
list_object_v2() method to check if a key exists in an S3 bucket using the Boto3 client.
Using S3 Object.Load() method in Boto3 Resource
In this section, you’ll learn how to check if a key exists in the S3 bucket using the Boto3 resource.
Boto3 resource doesn’t provide any method directly to check if the key exists in the S3 bucket.
Hence, you can load the S3 object using the
- If there is no exception thrown, then the key exists.
- If there is a client error thrown and the error code is
404, then the key doesn’t exist in the bucket.
While using this method, the program’s control flow is handled through exceptions, which is not recommended.
Use the following code to check if a key exists in an S3 bucket using the Boto3 resource.
import boto3 bucket='stackvidhya' file_key = 'csv_files/IRIS.csv' session = boto3.Session( aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>' ) s3 = session.resource('s3') try: s3.Object(bucket, file_key).load() except botocore.exceptions.ClientError as e: if e.response['Error']['Code'] == "404": print("Object Doesn't exists") else: print("Error occurred while fetching a file from S3. Try Again.") else: print("Object Exists")
This is how you can check if a key exists in an S3 bucket using Boto3.
If you want to check if a key exists in the S3 bucket in Python without using Boto3, you can use the S3FS interface.
S3Fs is a Pythonic file interface to S3. It builds on top of botocore.
It provides a method exists() to check if a key exists in the S3 bucket.
True– If the key exists in the S3 bucket
False– If the key doesn’t exist in the S3 bucket
pip install s3fs
S3Fs is installed.
Use the code below to check if a key exists in the S3 bucket using the
import s3fs bucket='stackvidhya' file_key = 'csv_files/IRIS.csv' s3 = s3fs.S3FileSystem(anon=False, key='<your_access_key_id>', secret='<your_secret_access_key>') path = bucket + '/' + file_key if s3.exists(path): print('File Exists') else: print('File doesn't exists')
You can also use this method to Check if S3 URI exists.
S3 URI will look like this
s3://bucket_name//object_name.extension . You can generate this URL using the copy URI option available in the AWS S3 console.
This is the alternative method to check if a key exists in the S3 bucket using Python.
To summarise, you’ve learned how to check if a key exists in the S3 bucket using the Boto3 library.
The same steps can also be used to
- check if a prefix exists in the S3 bucket.
- check if a folder exists inside an S3 bucket.
If you have any questions, comment below.
2 thoughts on “How to Check if a key exists in an S3 bucket using Boto3 Python?”
Hi, if the “same key exist” as a question this guide is okay. But same key but newer version of it i have my local. How can i check the key with the same version exist?
If the key exists, you can check if the contents of the file in S3 and the contents of the file in your local directory are the same.
For comparing text files, you can open the S3 file as a string using https://126.96.36.199/open-s3-object-as-string-in-boto3/ and open the local file using https://188.8.131.52/python-read-file-line-by-line/ and do the python string comparison.
If the file is a binary file, read how to open and read binary file in python https://184.108.40.206/python-read-binary-file/
Hope this helps.
Further questions are welcome.