AWS Storage Blog
Enhance data access control with AWS Transfer Family and Amazon S3 Access Points
Sharing data is at the core of collaboration efforts sparking innovation. Many industries rely on secure data access solutions that scale with their business. From pharmaceutical companies exchanging research data with partners in Amazon S3, to financial customers migrating and modernizing transaction data with traditional file transfer protocols, there is a persistent ask for fine-grained access control (FGAC) in managed file transfer solutions. Organizations like these need mechanisms to simplify user data access patterns across complex datasets shared with internal departments and external partners.
In this post, I discuss how you can use AWS Transfer Family logical directories and Amazon S3 Access Points to enhance FGAC in your shared S3 datasets.
Overview of AWS Transfer Family access controls
AWS Transfer Family continues to evolve as a managed file transfer (MFT) solution with a variety of ways to configure access. Your options for implementing secure access include layering AWS native controls such as AWS Identity and Access Management (IAM) permissions, security groups, and S3 bucket policies. You can also take user authentication into your own hands by either integrating your existing authentication systems or choosing the service-managed approach. AWS Transfer Family also provides logical directories as a built-in feature that maps multiple S3 buckets and files to a virtual directory. Logical directories simplify complex folder structures for data distribution without replicating files across multiple users. With this feature, you define and lock a specific user’s visibility into the data file-structure, have chroot-capabilities, and hide S3 bucket names and paths from users.
Compatibility with Amazon S3 Access Point aliases
S3 Access Points, a feature of Amazon S3, simplifies data access for any AWS service or customer application that is storing data in S3. With each S3 Access Point, you can create a unique access control policy to easily define authorization patterns on your shared S3 datasets.
The recently launched S3 Access Points aliases allow AWS Transfer Family to absorb S3 Access Points benefits. Use these aliases interchangeably with S3 bucket names in AWS Transfer Family workflows to provide compatibility with S3 Access Points and subsequent functionality at no additional cost. Customers with shared datasets from data lakes, media archives, or user-generated content can now use these aliases to create individualized access points with tailored names and permissions across hundreds of applications.
Fine-grain access control (FGAC) with S3 Access Points and logical directories
Rather than manage permissions through a single bucket policy spanning hundreds of use cases, abstract and delegate these permissions to discrete S3 Access Points policies. This simplifies administering thousands of roles and policies with respective character limits, adds flexibility to changes in existing file-transfer folder structures, and streamlines common asymmetrical access patterns such as read/write vs. read-only. You can design and configure granular policies to meet the scale of your business across users, applications, teams, departments, organizations, and partners.
When you combine AWS Transfer Family logical directories and Amazon S3 Access Points, you simplify and scale FGAC patterns used to manage both specific users and user-group permissions. In this blog, I cover an example defining an asymmetrical access pattern using S3 Access Points and logical directories as a mechanism to minimize the management of IAM roles and S3 bucket policies.
Prerequisites
To get started, you’ll need:
- An AWS account
- Access and familiarity with AWS Command Line Interface (AWS CLI)
- An AWS Transfer Family server
- Access and familiarity with a Transfer Family SFTP/FTP(S) client
- Basic working knowledge of logical directories and S3 Access Point policies
Permission Definitions
I use a multi-folder dataset that is shared with end users stemming from internal teams and external organizations. The users have different expected access patterns on the same S3 dataset. Here, authorized actions are dependent on a user’s folder, role, and organization. My shared S3 dataset has many types of end-users, and I narrow it down to the following three:
- Admin: internal user with full permissions and viewing access into the bucket contents
- Alice: internal user with full read/write permissions, but only to specified user and shared folders
- Ben: external user with read/write permissions into a user folder and read-only permissions on shared folder
As I use logical directories and S3 Access Points for FGAC, I define the access patterns, S3 structure, and expected user exposure.
My access pattern driven by S3 policies is shown as followed:
My S3 structure configuration is shown as follows:
My expected user exposure driven by logical directories covers these three access patterns:
Note: The preceding asymmetrical access pattern can be achieved through a variety of the previously mentioned access controls. This example intends to show the basic functionality of using S3 Access Points with logical directories to minimize the number of roles and policies needed to manage. This becomes increasingly apparent with more complex user-patterns.
Walkthrough
To achieve FGAC via S3 Access Points and AWS Transfer Family logical directories I complete the following steps and walk you through each in more detail.
- Create S3 Access Points and retrieve the respective alias.
- Define your S3 policies.
- Define logical directory mappings with S3 Access Points alias.
- Test expected user access.
Create S3 Access Point and retrieve the alias
You can create S3 Access Points and retrieve the alias using the S3 console, REST API, or AWS CLI as shown. I create an access point named `read-only-ap` for bucket `bucket-name` in account 123456789012.
aws s3control create-access-point --bucket bucket-name --name read-only-ap --
account-id 123456789012
{
"AccessPointArn":"arn:aws:s3:region:123456789012:
accesspoint/read-only-ap","Alias": "read-only-ap-METADATA-s3alias"
}
Define your S3 policies
At this step, you can get strategic in defining authorization patterns that minimize policy overhead. Rather than creating a per-user or per-role policy, my goal is to narrow the scope of the `shared` folder to read-only access for external users like Ben. Therefore, my desired permissions state is simplified with a shorter deny-list than allow-list, and I remove the IAM management headache by simplifying permissions to two types of authorized bucket actions:
- Policy 1: A general permissive S3 bucket policy allowing read/write access to our bucket
- Policy 2: A more restricted S3 Access Points policy for read-only access to our bucket
I assume my users Admin, Alice, and Ben each have an IAM role including an IAM policy with full access to bucket-name, and a trusted relationship between AWS Transfer Family is defined. Regardless of how over permissive a user’s IAM role may be, if routed via this sample S3 Access Points Policy (Policy 2), the user is denied on putting and deleting objects:
{
"Version": "2012-10-17",
"Statement" : [
{
"Effect": "Deny",
"Principal" : {
"*"
},
"Action": ["s3:PutObject","s3:DeleteObject"],
"Resource":"arn:aws:s3:region:123456789012:accesspoint/read-only-ap/object/*"
}
]
}
Granted, if there isn’t a conflict with a mapped user’s IAM role, I can design these S3 bucket policies and S3 Access Points policies to scale permissions across different authorization patterns.
Define logical directory mappings using S3 Access Points alias
Now that I used bucket policies to define what my users can do, I use logical directories to specify what users are authorized to see upon file-transfer client entry. For contrast in Admin, I create a user via the CLI with the S3 bucket path exposed as a PATH-home directory type with no chroot-capability:
Admin:
aws transfer create-user --server-id server-id --user-name admin --ssh-public- key-body key--role role --home-directory-type PATH --home-directory /bucket-name
Using logical directories, I create users Alice and Ben to offer chroot-capabilities and granular control in folder and file visibility. For Alice, I use bucket-name as part of the logical directory target definition to make use of our more general permissive bucket policy (Policy 1).
Alice:
aws transfer create-user --server-id server-id --user-name alice --ssh-public-
key-body key --role role --home-directory-type LOGICAL --home-directory-mapping
‘[{"Entry": "/alice", "Target": "/bucket-name/internal/alice"},
{"Entry": "/shared", "Target": "/bucket-name/shared"}]’
For Ben, I replace bucket-name with the retrieved alias in HomeDirectoryMappings so that authorized actions on the shared folder are limited by the S3 Access Points policy (Policy 2).
Ben:
aws transfer create-user --server-id server-id --user-name ben --ssh-public-
key-body key --role role --home-directory-type LOGICAL --home-directory-mapping
‘[{"Entry": "/ben", "Target": "/bucket-name/external/ben"},
{"Entry": "/shared", "Target": "/read-only-ap-METADATA-s3alias/shared"}]’
Test expected user access
Using your file-transfer client of choice, you can test the following read/write commands to meet expectations. I tested each authenticated user’s read/write output using OpenSSH as a file-transfer client to test authorized actions on an SFTP-enabled server:
Admin
sftp> pwd Remote working directory: /fgac-tf-ap sftp> ls external internal shared // Admin has full view into folder structure sftp> put image.jpg Uploading image.jpg to /fgac-tf-ap/image.jpg image.jpg 100% 6828 160.6KB/s 00:00 sftp> ls external image.jpg internal shared // Admin can put file regardless of folder sftp> rm image.jpg Removing /fgac-tf-ap/image.jpg sftp> ls external internal shared // Admin can remove file sftp> cd shared sftp> ls image.png sftp> rm image.png Removing /fgac-tf-ap/shared/image.png //Admin can remove existing files in shared folder sftp> ls sftp> cd .. sftp> ls external internal shared sftp> cd external sftp> ls ben sftp> cd ben sftp> ls folder1 sftp> put image.jpg Uploading image.jpg to /fgac-tf-ap/external/ben/image.jpg image.jpg 100% 6828 175.9KB/s 00:00 sftp> ls folder1 image.jpg // Admin can put file in a user folder sftp> rm image.jpg Removing /fgac-tf-ap/external/ben/image.jpg sftp> cd .. sftp> ls ben // Admin can remove file in a user folder sftp> cd .. sftp> ls external internal shared sftp> cd .. sftp> ls … //
Note: Some clients may allow user to traverse up the folder even if user has scoped down policy. Learn more about this, chroot and logical directories in this blog
Alice
sftp> pwd Remote working directory: / sftp> ls alice shared // Alice can get user and shared folder sftp> cd alice sftp> ls folder1 folder2 sftp> put image.jpg Uploading image.jpg to /alice/image.jpg image.jpg 100% 6828 156.9KB/s 00:00 // Alice can put file in Alice folder sftp> ls folder1 folder2 image.jpg sftp> rm image.jpg Removing /alice/image.jpg sftp> ls folder1 folder2 // Alice can remove file in Alice folder sftp> cd .. sftp> ls alice shared sftp> cd shared sftp> ls sftp> put image.jpg Uploading image.jpg to /shared/image.jpg image.jpg 100% 6828 176.6KB/s 00:00 // Alice can put files in shared folder sftp> ls image.jpg
Note: Some clients may allow user to traverse up the folder even if user has scoped down policy. Learn more about this, chroot and logical directories in this blog
Ben
sftp> pwd Remote working directory: / sftp> ls ben shared // Ben can get user and shared folder sftp> cd ben sftp> ls folder1 sftp> cd folder1 sftp> ls sftp> sftp> put image.jpg Uploading image.jpg to /ben/folder/image.jpg Image.jpg 100% 15KB 1.9MB/s 00:00 sftp> ls image.jpg // Ben can put file in user folder sftp> rm image.jpg Removing /ben/folder1/image.jpg Sftp> ls Sftp> //Ben can remove file in user folder sftp> cd ../.. sftp> ls ben shared sftp> cd shared sftp> ls image.jpg sftp> put ben.jpg Uploading ben.jpg to /shared/ben.jpg 100% 6828 177.3KB/s 00:00 Couldn’t close file: Permission denied // Ben cannot put file in shared folder sftp> rm image.jpg Removing /shared/image.jpg Couldn’t delete file: Permission denied // Ben cannot delete file in shared folder
As expected, the Admin user has full read/write access and full visibility into the bucket. Alice has read/write permissions into both the shared and user folders. Ben is most limited with read-only views into the shared folder and read/write access into user folder.
If your output doesn’t provide the anticipated authorization, you can troubleshoot by walking through a user’s IAM role, the S3 bucket policies, S3 Access Points policies, logical directory mappings, and refresh the client connection.
Cleaning up
To avoid incurring future charges, delete the sample resources set up as part of this post including your S3 bucket, your IAM roles, and your AWS Transfer Family server or any respective users no longer needed.
For cost details, please refer to Amazon S3 pricing and AWS Transfer Family pricing pages.
Conclusion
In this post, I walked through integrating Amazon S3 Access Points with AWS Transfer Family logical directories to scale your permission management of shared datasets. Now, imagine scaling a similar pattern to thousands of users with niche access needs in your MFT. You no longer have to manage complex IAM roles or error-prone configurations in a single bucket policy. You can now build more complex variations with solutions that span across accounts and S3 buckets, delegate access control to individual access points, or have multiple complex entry-target pairings.
Leave a comment if you found this helpful or have any questions, and happy building!