How to sync an entire Amazon S3 bucket locally
tl;dr
brew install awscli # MacOS
# apt update && apt install awscli -y # Ubuntu
# pip install --upgrade --user awscli # Unix
aws configure
aws s3 ls
aws s3 sync s3://my-bucket .
Ubuntu install
this is on a fresh Ubuntu system.
try first:
apt install awscli -y
echo `python --version` `pip --version`
install pip
curl -O https://bootstrap.pypa.io/get-pip.py && python get-pip.py
pip --version
install AWS CLI
pip install awscli --upgrade --user
install (Unix)
apt install python-pip -y
pip install --upgrade pip
pip install --upgrade --user awscli
to upgrade, run the same command above. To uninstall, pip uninstall awscli
configuration
aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: ENTER
For Amazon S3, region doesn’t matter. But you must enter a value, so pick whichever you like from here.
Default output format can be either json
, text
, or table
. If you don’t specify an output format, json will be used. To skip having to specify anything hit ENTER.
You can change or update these later. The access_key and ID are stored in ~/.aws/credentials
and the region and default format are saved in ~/.aws/config
. You can also have multiple profiles with different configurations.
List and sync buckets
# List all buckets
aws s3 ls
# Sync the 'my-bucket' bucket on S3 to current directory
aws s3 sync s3://my-bucket .