AWS S3 Data Transfer

Overview

Use AWS S3 CLI to transfer(upload/download) data between gpu machine and s3 bucket.

Install AWS CLI

Open a terminal and ssh into your instance, or use the terminal in the Jupyter Notebook UI.

Then run the following command to install the AWS CLI.

apt install awscli

Configure AWS CLI

Run the following command to configure the AWS CLI.

aws configure

Provide your Access Key ID, Secret Access Key, Region, and Output format.

#### Access Key ID and Secret Access Key
You can find your Access Key ID and Secret Access Key in the AWS Management Console.

Step 1: Sign in to AWS Console
Go to the AWS Management Console and log in with an account that has permissions to create and manage IAM (Identity and Access Management) users.

Step 2: Go to the IAM Service
In the AWS Console, search for and open the IAM service.
In the IAM Dashboard, navigate to Users in the sidebar.

Step 3: Create or Select a User
If you don’t already have a user, click Add users. Follow the prompts to create a new user and be sure to enable Programmatic access.
If you already have a user, click the username to open their details.

Step 4: Generate Access Keys
In the user’s details, go to the Security credentials tab.
Scroll down to Access keys and click Create access key.
Copy or download the Access Key ID and Secret Access Key shown on the screen.

Important: Make sure to save the Secret Access Key somewhere safe, as it won’t be shown again once you leave the page.

Download Data

Run the following command to download data from an S3 bucket to your instance.

aws s3 cp s3://your-bucket-name/path/to/your/file /local/path/to/download/
### download a folder
aws s3 cp --recursive s3://your-bucket-name/path/to/your/folder /local/path/to/download/

Upload Data

Run the following command to upload data from your instance to an S3 bucket.

aws s3 cp /local/path/to/upload s3://your-bucket-name/path/to/upload/
### upload a folder
aws s3 cp --recursive /local/path/to/upload s3://your-bucket-name/path/to/upload/