AWS Storage Blog

Creating a simple public file repository on Amazon S3

Organizations often need to store and serve various types of static content, such as data sets, archives, and file repositories, on the web. This content is typically accessed by end-users for research and analysis purposes or general content distribution. However, presenting this content in a user-friendly and easily browsable manner can be a significant challenge, particularly when dealing with large volumes of data, high traffic, or complex file structures.

Previously, organizations had to set up FTP servers or configure web server features like Apache’s DirectoryIndex or Nginx’s Autoindex to create browsable directory listings for their static content. Many have now chosen Amazon Simple Storage Service (S3) as their preferred storage solution for static web content combined with Amazon CloudFront for a highly scalable, cost-optimized, and serverless solution to publish content on the web.

In this post, I walk through deploying a simple web-based file browser solution for publishing files in an Amazon S3 bucket. This approach utilizes S3 and CloudFront to provide a user-friendly interface for browsing your bucket’s contents, eliminating the need to deploy and maintain additional web server infrastructure. I’ll also review ways to quickly move data into the public S3 bucket, customize the solution with your own domain, and tailor the appearance to match your organization’s branding requirements.

Setting up a simple public file repository on Amazon S3: Solution overview and architecture

This solution uses a static website hosted on Amazon S3 fronted by CloudFront. Once deployed, users can upload files they wish to publish on the public internet to a specially configured “public files” S3 general purpose bucket. When users access the URL of the file browser application, the application lists the contents of the S3 bucket and renders the results in a traditional file browser hierarchical format.

Architecture diagram showing a web browser downloading public website code through CloudFront and an S3 bucket, then obtaining credentials, then using those credentials to list files in a seperate S3 bucket and rending that list of files in the browser.

  1. End user loads a web page from CloudFront and Amazon S3.
  2. After page load, the browser silently retrieves temporary credentials from Amazon Cognito.
  3. The browser uses the credentials from Cognito to list the files in the public “files” S3 bucket.
  4. Then, the file list is rendered in the browser as a traditional file browser/directory index.
  5. Optionally, a user can select one of the listed files to download it through CloudFront and Amazon S3.

Prerequisites

For this walkthrough, you must have the following prerequisites:

  • An AWS account
  • AWS SAM CLI installed and configured with credentials
  • Python version 3.11 installed and in your PATH variable
  • Choose an AWS Region where the required services are supported. Most AWS Regions are supported, but consult the availability page for details.
    • Amazon CloudFront (including Standard Log support)
    • Amazon S3
    • Amazon Cognito
    • AWS CloudFormation
    • AWS Lambda

Setting up a simple public file repository on Amazon S3: Walkthrough

The solution is packaged as an automated deployment through the AWS Serverless Application Model (AWS SAM) command line interface (CLI).

The AWS SAM CLI is an open-source command line tool used to locally build, test, debug, and deploy serverless applications defined with AWS SAM templates.

Download the source code and extract locally: AWS Samples – public-file-browser-for-amazon s3.

Build and deploy the stack

1. In a terminal, navigate to the ./sam/ directory within the downloaded code repository.

2. Run the following command to build and package the project for deployment:

sam build

3. Deploy the SAM template to your account. The wizard guides you through the process of deploying the SAM AWS CloudFormation. Details on this process are found in the SAM build documentation.

a. Run the following command:

sam deploy --guided --capabilities CAPABILITY_NAMED_IAM

b. Enter values for the deployment parameters:

i. Stack Name: Choose a unique CloudFormation stack name. End users do not see this.
ii. AWS Region: The supported AWS Region you chose in the Prerequisites section.
iii. SiteName: Publicly visible title for the site, shown on the top of the page and in the title bar.
iv. FilesOpenTabMode: How do you want the browser to react when a user selects a file in the following interface values. For a consistent user experience, In New Tab is recommended over In Same Tab.
v. VisibleStorageClasses: Comma delimited list of storage classes to show. Recommend keeping this default.
vi. CrossOriginRestriction: Browser security setting, set to * for first deployment, then see Step 6.

c. Select the default inputs for the remaining items with the last prompt before deployment being:

Deploy this changeset?

4. Wait for the deployment to complete. This process takes approximately five minutes with a final prompt stating:

Successfully created/updated stack - [STACK-NAME] in [REGION]

5. Once the deployment completes, note the following entries in the Outputs section.

a. FileBrowserURL: This URL is for the public web interface. Needed in Step 6.
b. PublicFilesBucket: The name of the S3 bucket for storing PUBLICLY ACCESSIBLE files that display in the user browser.
c. WebInterfaceAppBucket: The name of the S3 bucket that stores the code that runs the file browser web interface.

6. IMPORTANT: Complete Steps 3 and 4 again, keeping all values the same except for the CrossOriginRestriction parameter, and input the value from the FileBrowserURL output in Step 5. For example:

Parameter CrossOriginRestriction [*]: https://d111111abcdef8.cloudfront.net

The deployment of the public file browser for Amazon S3 web application is now completed. AWS SAM CLI uses CloudFormation to orchestrate the deployment of the front-end static website and public file storage S3 bucket.

A screenshot of a terminal window showing the successful output of the sam-deploy command. It shows the CloudFormation outputs showing the PublicFilesBucket, the FileBrowserURL, and the WebInterfaceAppBucket outputs.

Optionally, you can view the deployed resources by logging in to the AWS Management Console, navigating to the CloudFormation service, selecting the CloudFormation stack name selected in the preceding deployment, and selecting the Resources tab.

Setting up a simple public file repository on Amazon S3: Test the solution

In this testing phase, you upload contents to the configured “public files” S3 bucket deployed as part of this solution. After uploading, you browse the public URL and verify the upload files are listed.

  1. Log in to the S3 console, navigate to your list of buckets, and select the Amazon S3 bucket that matches the PublicFilesBucket output value from Step 5 in the setup instructions. The S3 bucket name should match public-file-browser-files-[...]. Select the Upload button.

A screenshot of the AWS Management Console's S3 Bucket contents view. The bucket is the bucket for public files and is empty. There is an arrow pointing to the Upload button.

2. Select the files you want to upload to test the solution. By default, S3 objects and resources are private. However, this solution has a specially configured bucket that makes all contents publicly accessible. You can review the remaining options on the page. The default options are acceptable for most use cases and include automatic encryption for stored files. Select Upload at the bottom of the page.

3. Wait for your upload to finish and show a successful status. The length of time to upload is based on your network connection speed and data size for the test files.

A screenshot of the AWS Management Console's S3 Bucket upload summary view. The screenshot shows seven files totallying over a hundred megabytes successfully uploaded with no failures.

4. Select the Close button to return to the S3 bucket contents and verify that the new files appear.

A screenshot of the AWS Management Console's S3 Bucket contents view. The bucket is the bucket for public files and has the contents which were previously uploaded.

5. Finally, view the public file browser for Amazon S3 web interface by going to the URL in the FileBrowserURL value from the output section of deployment Step 5. The uploaded files are listed in the web interface.

A screenshot of a web browser window showing the final deployed and configured interface for the public file browser solution. The web page shows the files uploaded to the public files bucket.

Next steps

With the public file browser for Amazon S3 solution fully deployed and tested, the next steps are to customize the solution by adding your own files. Optionally, you can move the solution to your DNS domain name, or modify the user interface. See the following resources if interested in further customizing the solution.

How do I move files into the public-file-browser-files-[...] quickly in bulk?

How do I use this solution with my own DNS domain name (such as publicfiles.example.com) instead of the CloudFront name?

How do I modify the public user interface, such as to add text, change the style, or add my logo?

  • The public website files are located in the public-file-browser-website-[...] bucket. These files can be downloaded, modified, and re-uploaded containing customizations. Note that CloudFront caches these files, so you must create an invalidation to clear the cache when a file is updated. See How do I remove a cached file from CloudFront?
  • If you choose to update the files in the ./website/ directory of the source code repository for future deployments, then you must update the ./sam/seed_s3_data/website.zip by following the instructions in the repository’s README.md file.

Additional considerations in security

As a best practice, the solution enables the following features:

Cleaning up

This is a serverless solution, thus the costs are directly related to usage – only a few cents monthly to store the website source code. The persistent cost to maintaining this application when not in use is in the data stored within the website, public files, and logging S3 buckets. Costs are also incurred for users accessing the website, such as CloudFront data transfer and Amazon S3 LIST/GET request costs. Notably there is no charge for Amazon S3 Data Transfer to CloudFront.

To avoid incurring future charges, follow these instructions to delete the resources created by the solution:

  1. In the Console, navigate to CloudFormation and select the stack name specified in the deployment section of this post.
  2. Select Delete near the top of the console and select delete again in the pop-up confirmation to begin deleting the stack. This process takes approximately five minutes.
  3. Deletion of the CloudFormation stack likely fails if there are contents remaining in the website and public files’ S3 buckets. You can choose to delete the stack anyway, thereby retaining these resources, or manually empty these S3 buckets before attempting the delete again.
  4. Empty and then delete the logging S3 bucket that stores access logs for the application. These are preserved by default for audit purposes. This S3 bucket is formatted as public-file-browser-logging-[...].

Conclusion

In this post I presented a simple web-based file browser solution for publishing files in an Amazon S3 bucket. I also showed ways to quickly move data into the public S3 bucket, customize the browser URL with your own domain, and change the appearance of the solution by customizing the code to match your organization’s branding requirements.

You can use the public file browser for Amazon S3 solution to reduce costs and provide a user-friendly interface for browsing your bucket’s contents. This approach allows for serving large volumes of data, high traffic, and complex file structures while also eliminating the need to deploy and maintain additional web server infrastructure. Additionally the solution’s cost directly scales with its usage so you’re not paying for idle infrastructure.

Each component of this application includes an AWS Free Tier. Use the AWS Free Tier to learn and experiment with new technologies.

More related resources from AWS

TAGS: , , , ,
Caesar Kabalan

Caesar Kabalan

Caesar Kabalan is a solutions architect at Amazon Web Services (AWS). He brings together his development and enterprise infrastructure experience to invent solutions to solve customer challenges. He also restores, tweaks, and reprograms 1970s-era pinball machines as a hobby.

Grant Joslyn

Grant Joslyn

Grant Joslyn is a solutions architect for the US state and local government public sector team at Amazon Web Services (AWS). He specializes in end user compute and cloud automation. He provides technical and architectural guidance to customers building secure solutions on AWS. He is a subject matter expert and thought leader for strategic initiatives that help customers embrace DevOps practices.