How To Specify Credentials When Connecting to AWS S3 Using Boto3?

Boto3 is an AWS SDK for python. You can interact with any AWS service using Boto3 when you’re programming with python if you have the access and the appropriate credentials.

You can specify credentials in boto3 using session = boto3.Session(aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>' ).

Other ways to pass credentials are,

• Passing credentials as parameters
• Using the AWS config file
• Using shared credentials file
• Using environment variables

If you’re in Hurry

You can use the below code snippet to specify credentials when creating a boto3.Session.

Snippet

import boto3
session = boto3.Session(
    aws_access_key_id='<your_access_key_id>',
    aws_secret_access_key='<your_secret_access_key>'
)

This is how you can specify credentials directly when creating a session to AWS S3.

If You Want to Understand Details, Read on…

In this tutorial, you’ll learn the different methods available to specify credentials when connecting to AWS services using boto3.

Prerequisites

• Generate the security credentials by clicking Your Profile Name -> My Security Credentials -> Access keys (access key ID and secret access key) option. This is necessary to create session with your AWS account.

Installing Boto3

If you’ve not installed boto3 yet, you can install it by using the below snippet.

You can use the % symbol before pip to install packages directly from the Jupyter notebook instead of launching the Anaconda Prompt.

Snippet

%pip install boto3

Boto3 will be installed successfully.

Now, you can use it to access AWS resources.

Passing credentials as parameters

In this section, you’ll learn how to pass the credentials directly during the creation of the boto3 Session or boto3 client. Read the difference between boto3 session, client, and resource to understand its differences and when to use it.

Creating Boto3 Session With Credentials

A session is an object to create a connection to AWS Service and manage the state of the connection. You can create a boto3 Session using the boto3.Session() method.

Along with other parameters, Session() accepts credentials as parameters namely,

• aws_access_key_id – Your access key ID
• aws_secret_access_key – Your secret access key

Once the session is created, you can access the resources by creating a resource. For example, you can access S3 by creating S3 resources using session.resource('s3').

Snippet

import boto3
session = boto3.Session(
    aws_access_key_id='<your_access_key_id>',
   aws_secret_access_key='<your_secret_access_key>'
)


#Then use the session to get the resource
s3 = session.resource('s3')

s3.Bucket('stackvidhya').upload_file('E:/temp/testfile.txt','file2_uploaded_by_boto3.txt')

Creating Boto3 Client With Credentials

The client is a low-level service class representing the AWS services. It provides methods similar to AWS API services.

You can create a boto3 client using the method boto3.client().

Along with other parameters, client() accepts credentials as parameters namely,

• aws_access_key_id – Your access key ID
• aws_secret_access_key – Your secret access key

Once the boto3 client is created, you can access the methods available on the boto3 client.

For example, boto3 the client provides the methods put_object() to upload files to the S3 bucket. With the client created, you can use put_object() method to upload files to the bucket as shown below.

Snippet

s3_client = boto3.client('s3', 
                      aws_access_key_id='<your_access_key_id>', 
                      aws_secret_access_key='<your_secret_access_key>', 
                      region_name='us-east-1'
                      )

s3_client.put_object(Body='Text Contents', Bucket='stackvidhya', Key='filename_by_client_put_object.txt')

This is how you can create boto3 client with credentials and use the methods provided by the client to access the AWS services.

Get Credentials Dynamically

In the previous section, you’ve learned how to create boto3 Session and client with the credentials.

After creating sessions and at the later point of your program, you may need to know the credentials again. For creating another session or a client object.

In that case, you can read credentials from boto3 Session using the get_credentials() method. You can get access_key id using the .access_key attribute and secret key using the .secret_key attribute.

Snippet

import botocore.session
session = botocore.session.get_session()

print(session.get_credentials().access_key)


print(session.get_credentials().secret_key)

Output

    <your_access_key_id>
    <your_secret_access_key>

This is how you can get the access key and the secret access from the already created session.

Using AWS Config File

In this section, you’ll learn how to configure AWS CLI with the credentials and use these credentials to create a boto3 session.

First, you need to install AWS CLI using the below command.

Snippet

%pip install awscli

Output

    Collecting awscliNote: you may need to restart the kernel to use updated packages.

    Successfully installed awscli-1.19.98 colorama-0.4.3 docutils-0.15.2 pyasn1-0.4.8 rsa-4.7.2

AWS CLI will be installed on your machine.

Now, you need to configure the security credentials and the default region to be used while using the AWS CLI commands.

You can do so by using the below command.

Snippet

aws configure

You’ll be asked for the access key id and secret access key and the default region to be used. Read how to install and configure AWS CLI to understand in detail.

Once the configuration is done, the details will be stored in the file ~/.aws/credentials and the content will look like below.

[default]
aws_access_key_id=<your_access_key_id>;
aws_secret_access_key=<your_secret_access_key>;

Fetch Credentials From Aws CLi Configuration File

You can fetch the credentials from the AWS CLI configuration file by using the below parameters.

• settings.AWS_SERVER_PUBLIC_KEY – To fetch the access key id
• settings.AWS_SERVER_SECRET_KEY – To fetch the secret key

You can use these in your python program to create a boto3 Session as shown below.

Snippet to Create Boto3 Session

Creating a boto3 Session using the settings from the config file:

import boto3
session = boto3.Session(
    aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
    aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
)


#Then use the session to get the resource
s3 = session.resource('s3')

Snippet to Create Boto3 Client

s3_client = boto3.client('s3', 
                      aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY, 
                      aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY, 
                      region_name=REGION_NAME
                      )

This is how you can install and configure the AWS CLI and specify the credentials using the CLI parameters to create boto3 session and client.

Using Shared Credentials File

You can also create a credentials file and store the credentials to connect to AWS services using the SDKs such as boto3.

Credential files are normally available in the location \.aws\credentials and it contains the access key id and the secret access keys.

Example

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

When this file is configured, you can directly use the parameters,

• settings.AWS_SERVER_PUBLIC_KEY – To fetch the access key id
• settings.AWS_SERVER_SECRET_KEY – To fetch the secret key

This is how you can use the shared credentials file to store and reuse the credentials in the SDKs such as boto3.

Using Environment Variables

Another option available to store the AWS credentials is to use the environment variables. When you set the environment variables, it is available as a global parameter. This will affect all the clients created using any SDKs unless it is overridden in the new config object.

The environment variables used to configure AWS credentials are,

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

You can configure these variables and used them elsewhere to access the credentials.

Solving Errors

You might face an error Boto3 unable to locate credentials when using the parameters settings.AWS_ACCESS_KEY_ID or settings.AWS_SECRET_ACCESS_KEY.

To solve this, check if the AWS CLI is rightly configured and has the credentials stored accordingly. If you’re trying to use the environment variables, double-check if you are able to access the environment variables from the system command line first.

If you still face problems, comment below with the full description. We will try to help you.

Conclusion

To summarize, you’ve learned how to specify credentials when creating boto3 Session or client. You’ve also learned how you can install and configure AWS CLI with the security credentials and how the credentials can be referred to in your program.

If you have any questions, comment below.

You May Also Like

• How To Load Data From AWS S3 Into Sagemaker (Using Boto3 Or AWSWrangler)
• How To Write A File Or Data To An S3 Object Using Boto3
• How to List Contents of s3 Bucket Using Boto3 Python