How to sync an entire Amazon S3 bucket locally

tl;dr

brew install awscli # MacOS
# apt update && apt install awscli -y # Ubuntu
# pip install --upgrade --user awscli # Unix
aws configure
aws s3 ls 
aws s3 sync s3://my-bucket . 

Ubuntu install

this is on a fresh Ubuntu system.

try first:

apt install awscli -y
echo `python --version` `pip --version`

install pip

curl -O https://bootstrap.pypa.io/get-pip.py && python get-pip.py
pip --version

install AWS CLI

pip install awscli --upgrade --user

install (Unix)

apt install python-pip -y
pip install --upgrade pip
pip install --upgrade --user awscli

to upgrade, run the same command above. To uninstall, pip uninstall awscli

configuration

aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: ENTER

For Amazon S3, region doesn’t matter. But you must enter a value, so pick whichever you like from here.

Default output format can be either json, text, or table. If you don’t specify an output format, json will be used. To skip having to specify anything hit ENTER.

You can change or update these later. The access_key and ID are stored in ~/.aws/credentials and the region and default format are saved in ~/.aws/config. You can also have multiple profiles with different configurations.

List and sync buckets

# List all buckets
aws s3 ls
# Sync the 'my-bucket' bucket on S3 to current directory
aws s3 sync s3://my-bucket .