Batch: Amazon S3
Introduction
In order to integrate with Banyan's AWS solution, you will be using the AWS CLI sync
command.
What we'll need from you
If your data is located in S3
This option is for situations where you have your files already in S3, and would like to move them from your bucket to ours.
- Company Name
- AWS Account ID
- The bucket name of the data that will be copied over
- The KMS Key ARN if your data is encrypted at rest
If the location is other then S3
This option is for situations where your data is on a drive or database and can be copied directly from a server.
- Company Name
- AWS Account ID
Creating a IAM user
Before you can assume the IAM Role created for your company, you have to create an IAM user which will be used by you to assume the set role. Since the role that we make references the IAM username, and the AWS Account ID that you provided us.
Navigate to the IAM console,
- create a new user named
banyan_input_s3
. - Select programmatic access,
- Hit next until you are shown your security credentials. Be sure to record them safely.
- Once the user is created, attach the following
inline policy
.
Important
Make sure to replace
COMPANY_NAME
with your company name.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::356687812700:role/banyan_input_s3_merchant_COMPANY_NAME"
}
]
}
Preparation before assuming the role
Note
- Install the latest version of the AWS CLI.
- The following work needs to be done under a
*UNIX system
.
First, we need to make sure you have the right environment setup.
If you've used the AWS CLI before
If you have used the AWS CLI in the past, and run the aws configure
command, you will have a folder called .aws
in the home directory of the user you are logged in as in the OS, with all the files needed in place. You can skip the next step.
If you've never used the AWS CLI before
If you have never used the AWS CLI, following the instructions below to create the folder and files.
mkdir ~/.aws
- to create the folder in your home directory.touch ~/.aws/config
- to create an empty file for the AWS CLI configuration.touch ~/.aws/credentials
- to create an empty file where the user credentials will go.
Setup the credentials
With the folder structure and files now in place, add the following content in the ~/.aws/config
file, making sure to replace COMPANY_NAME
with the one you provided:
[profile banyan]
role_arn = arn:aws:iam::356687812700:role/banyan_input_s3_merchant_COMPANY_NAME
source_profile = banyan_credentials
In the ~/.aws/credentials
file, add the following content inside, making sure to replace DATA
with the correct values you've saved when making the IAM user:
[banyan_credentials]
aws_access_key_id=DATA
aws_secret_access_key=DATA
Give us access to your resources
At this point all the policies are set for your role on our side, but since we are dealing with a cross account access, we can't just set in our policy that you can copy data from your bucket to ours, if this was the case anyone could access resources in different accounts. For this reason you have to give our account explicit access to your data for our Role that you are going to assume for it to work.
Bucket policy
- Go to the bucket that you provided the name to
- In the Permission tab scroll down until the
Bucket policy
section. - Click Edit, and add the following policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::356687812700:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:GetObjectTagging"
],
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME/*",
"arn:aws:s3:::YOUR-BUCKET-NAME"
]
}
]
}
KMS Key Policy
If you also provided us a KMS ARN Key, in this case you also have to update the Key Policy to allow our account to use the key to decrypt your data in the bucket by adding the following Policy Document in the already existing Key Policy
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::356687812700:root"
},
"Action": "kms:Decrypt",
"Resource": "THE-FULL-ARN-OF-THE-KEY"
}
Transferring data to our bucket
Important
Make sure to replace
COMPANY_NAME
with your company name.
Now that the data is in the right place, you can run the following command in the terminal which will copy the evaluation data in the folder where you'll run the command using the profile that you just made above. The CLI will take care of the IAM Role assumption.
Bucket to Bucket
In the terminal, run the following command to copy data from your bucket to ours.
aws s3 sync s3://YOUR-BUCKET-NAME s3://by-production-us-east-1-input-s3-COMPANY_NAME --delete --profile banyan
Drive to Bucket
In the terminal, run the following command to copy data from from the local drive to our bucket.
aws s3 sync . s3://by-production-us-east-1-input-s3-COMPANY_NAME --delete --profile banyan
Additionally you can copy, list and delete
Updated about 1 year ago