Add an S3 Bucket as File Storage in a Django Project

The following is a detailed explanation on how to set up an S3 bucket as file storage in a Django project.

What you need to have:

1. An AWS account
2. A Django project that needs a file storage system
3. pip install boto3
4. pip install django-storages

1. Set up an S3 bucket:

First we need to locate the S3 tab on the Amazon dashboard. Then click on “Create bucket”:

Create a new bucket

We will then need to enter a new “Bucket name”. After this, select the region where you’d like for this bucket to be. Other than that, we can keep the default settings:

After creating our new bucket we will need to edit the “Cross-origin resource sharing (CORS)”. Scroll down until you see the following and then press edit:

Edit S3 CORS permission

Then add in the following:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "POST",
            "GET",
            "PUT"
        ],
        "AllowedOrigins": [
            "*"
        ]
    }
]

For better security we should update AllowedOrigins to only trusted domains. For now, I will keep it as * which means that anyone who has permission can access the bucket and will update it in production.

2. Create a new user:

Now we need to make a new username with limited access so that our Django project can communicate with the S3 bucket though a more controlled and secure process.

First we need to go to the IAM dashboard and click Users and then click Add user on the top left of the page:

After, we need to make a new username. When choosing an Access type, select Programmatic access only:

Make a new user called “django_user‘

Now click Next:Permissions at the bottom right side of the page in order to give the required permission to our new username. We must then navigate to the Attach existing policies directly tab and and search for S3. Then, tick the box next to AmazonS3FullAccess:

Add permission to your Django user

Now click through the rest of the pages until you have fully created a new user. You must also not forget to save the user Access key ID and Secret access key of this new user for future authentications:

Saving credentials

3. Updating Django settings.py

It is not recommended to add any credentials to any code base. For this, I use django-environ which allows you to utilise 12factor inspired environment variables to configure the application. This allows you to set up the environment variables independent of your operating system.

We now need to add the saved credentials and our bucket name to the .env file and then call them in from the settings.py:

1. Add storages to installed app
2. AWS_ACCESS_KEY_ID = env('AWS_ACCESS_KEY_ID')
3. AWS_SECRET_ACCESS_KEY = env('AWS_SECRET_ACCESS_KEY')
4. AWS_STORAGE_BUCKET_NAME = env('AWS_STORAGE_BUCKET_NAME')
5. AWS_S3_FILE_OVERWRITE = False
6. AWS_DEFAULT_ACL = None 
7.DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

The naming/spelling of the above variables must be exactly as mentioned, so that django-storages can interact with them correctly.

These are all the changes that will be needed and your project should now be using the S3 backend for file storage. In case that you already have files in your models that are stored in a media folder, you can drag and drop these the whole folder in the bucket that you’ve just made, and the system will automatically link them to your model data field.

4. In conclusion:

In this tutorial we learned how to make a new S3 bucket, make a new username and provide it with the required permissions to programmatically connect to S3. We also learned how to update a Django project so use an S3 bucket for file storage.