Batch: Amazon S3

Suggest Edits

Introduction

In order to integrate with Banyan's AWS solution, you will be using the AWS CLI sync command.

What we'll need from you

If your data is located in S3

This option is for situations where you have your files already in S3, and would like to move them from your bucket to ours.

  • Company Name
  • AWS Account ID
  • The bucket name of the data that will be copied over
  • The KMS Key ARN if your data is encrypted at rest

If the location is other then S3

This option is for situations where your data is on a drive or database and can be copied directly from a server.

  • Company Name
  • AWS Account ID

Creating a IAM user

Before you can assume the IAM Role created for your company, you have to create an IAM user which will be used by you to assume the set role. Since the role that we make references the IAM username, and the AWS Account ID that you provided us.

Navigate to the IAM console,

  1. create a new user named banyan_input_s3.
  2. Select programmatic access,
  3. Hit next until you are shown your security credentials. Be sure to record them safely.
  4. Once the user is created, attach the following inline policy.

📘

Important

Make sure to replace COMPANY_NAME with your company name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::356687812700:role/banyan_input_s3_merchant_COMPANY_NAME"
        }
    ]
}

Preparation before assuming the role

📘

Note

  1. Install the latest version of the AWS CLI.
  2. The following work needs to be done under a *UNIX system.

First, we need to make sure you have the right environment setup.

If you've used the AWS CLI before

If you have used the AWS CLI in the past, and run the aws configure command, you will have a folder called .aws in the home directory of the user you are logged in as in the OS, with all the files needed in place. You can skip the next step.

If you've never used the AWS CLI before

If you have never used the AWS CLI, following the instructions below to create the folder and files.

  1. mkdir ~/.aws - to create the folder in your home directory.
  2. touch ~/.aws/config - to create an empty file for the AWS CLI configuration.
  3. touch ~/.aws/credentials - to create an empty file where the user credentials will go.

Setup the credentials

With the folder structure and files now in place, add the following content in the ~/.aws/config file, making sure to replace COMPANY_NAME with the one you provided:

[profile banyan]
role_arn = arn:aws:iam::356687812700:role/banyan_input_s3_merchant_COMPANY_NAME
source_profile = banyan_credentials

In the ~/.aws/credentials file, add the following content inside, making sure to replace DATA with the correct values you've saved when making the IAM user:

[banyan_credentials]
aws_access_key_id=DATA
aws_secret_access_key=DATA

Give us access to your resources

At this point all the policies are set for your role on our side, but since we are dealing with a cross account access, we can't just set in our policy that you can copy data from your bucket to ours, if this was the case anyone could access resources in different accounts. For this reason you have to give our account explicit access to your data for our Role that you are going to assume for it to work.

Bucket policy

  1. Go to the bucket that you provided the name to
  2. In the Permission tab scroll down until the Bucket policy section.
  3. Click Edit, and add the following policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::356687812700:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR-BUCKET-NAME/*",
                "arn:aws:s3:::YOUR-BUCKET-NAME"
            ]
        }
    ]
}

KMS Key Policy

If you also provided us a KMS ARN Key, in this case you also have to update the Key Policy to allow our account to use the key to decrypt your data in the bucket by adding the following Policy Document in the already existing Key Policy

{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::356687812700:root"
    },
    "Action": "kms:Decrypt",
    "Resource": "THE-FULL-ARN-OF-THE-KEY"
}

Transferring data to our bucket

📘

Important

Make sure to replace COMPANY_NAME with your company name.

Now that the data is in the right place, you can run the following command in the terminal which will copy the evaluation data in the folder where you'll run the command using the profile that you just made above. The CLI will take care of the IAM Role assumption.

Bucket to Bucket

In the terminal, run the following command to copy data from your bucket to ours.

aws s3 sync s3://YOUR-BUCKET-NAME s3://by-production-us-east-1-input-s3-COMPANY_NAME --delete --profile banyan

Drive to Bucket

In the terminal, run the following command to copy data from from the local drive to our bucket.

aws s3 sync . s3://by-production-us-east-1-input-s3-COMPANY_NAME --delete --profile banyan

Additionally you can copy, list and delete

Updated almost 2 years ago