This topic describes some of the commands you can use to manage Amazon S3 buckets and objects
using the
aws s3
commands in the AWS CLI. For commands not covered in this topic and additional command examples,
see the
aws s3
commands in the AWS CLI Reference.
The high-level aws s3 commands simplify managing Amazon S3 objects. These commands
enable you to manage the contents of Amazon S3 within itself and with local directories.
The profile that you use must have permissions that allow the AWS operations
performed by the examples.
Understand these Amazon S3 terms:
Bucket – A top-level Amazon S3 folder.
Prefix – An Amazon S3 folder in a bucket.
Object – Any item that's hosted in an Amazon S3
bucket.
Before you start
This section describes a few things to note before you use aws s3
commands.
Large object uploads
When you use aws s3 commands to upload large objects to an Amazon S3 bucket, the
AWS CLI automatically performs a multipart upload. You can't resume a failed upload when using
these aws s3 commands.
If the multipart upload fails due to a timeout, or if you manually canceled in the
AWS CLI, the AWS CLI stops the upload and cleans up any files that were created. This process
can take several minutes.
If the multipart upload or cleanup process is canceled by a kill command or system
failure, the created files remain in the Amazon S3 bucket. To clean up the multipart upload, use
the
s3api abort-multipart-upload command.
File properties and tags in multipart copies
When you use the AWS CLI version 1 version of commands in the aws s3 namespace to
copy a file from one Amazon S3 bucket location to another Amazon S3 bucket location, and that
operation uses multipart copy, no
file properties from the source object are copied to the destination object.
By
default, the AWS CLI version 2 commands in the s3 namespace that perform multipart
copies transfers all tags and the following set of properties from the source to the
destination copy: content-type, content-language,
content-encoding, content-disposition,
cache-control, expires, and metadata.
This can result in additional AWS API calls to the Amazon S3 endpoint that would not have
been made if you used AWS CLI version 1. These can include: HeadObject,
GetObjectTagging, and PutObjectTagging.
If you need to change this default behavior in AWS CLI version 2 commands, use the
--copy-props parameter to specify one of the following options:
default – The default value. Specifies
that the copy includes all tags attached to the source object and the properties
encompassed by the --metadata-directive parameter used for non-multipart
copies: content-type, content-language,
content-encoding, content-disposition,
cache-control, expires, and metadata.
metadata-directive – Specifies that the
copy includes only the properties that are encompassed by the
--metadata-directive parameter used for non-multipart copies. It
doesn't copy any tags.
none – Specifies that the copy includes
none of the properties from the source object.
Create a bucket
Use the
s3 mb command
to make a bucket. Bucket names must be globally unique (unique across all of Amazon S3) and should be DNS
compliant.
Bucket names can contain lowercase letters, numbers, hyphens, and periods. Bucket names
can start and end only with a letter or number, and cannot contain a period next to a hyphen
or another period.
Syntax
$ aws s3 mb [--options]
The following example creates the s3://amzn-s3-demo-bucket bucket.
$ aws s3 mb s3://amzn-s3-demo-bucket
List buckets and objects
To list your buckets, folders, or objects, use the
s3 ls command.
Using the command without a target or options lists all buckets.
Syntax
$ aws s3 ls [--options]
For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of available
options, see
s3 ls in the
AWS CLI Command Reference.
The following example lists all of your Amazon S3 buckets.
$ aws s3 ls
2018-12-11 17:08:50 amzn-s3-demo-bucket1
2018-12-14 14:55:44 amzn-s3-demo-bucket2
The following command lists all objects and prefixes in a bucket. In this example
output, the prefix example/ has one file named
MyFile1.txt.
$ aws s3 ls s3://amzn-s3-demo-bucket
PRE example/
2018-12-04 19:05:48 3 MyFile1.txt
You can filter the output to a specific prefix by including it in the command. The
following command lists the objects in bucket-name/example/
(that is, objects in bucket-name filtered by the prefix
example/).
$ aws s3 ls s3://amzn-s3-demo-bucket/example/
2018-12-06 18:59:32 3 MyFile1.txt
To display only the buckets and objects in a specific region, use the
--region options
$ aws s3 ls --region us-east-2
2018-12-06 18:59:32 3 MyFile1.txt
If you have a large list of buckets and objects, you can paginated the results using
the --max-items or --page-size options. The
--max-items option limits how many total buckets and objects are returned
in a call and the --page-size option limits how many of those are listed on a
page.
The following example removes the s3://amzn-s3-demo-bucket bucket.
$ aws s3 rb s3://amzn-s3-demo-bucket
By default, the bucket must be empty for the operation to succeed. To remove a bucket
that's not empty, you need to include the --force option. If you're using a
versioned bucket that contains previously deleted—but retained—objects, this
command does not allow you to remove the bucket. You must first
remove all of the content.
The following example deletes all objects and prefixes in the bucket, and then deletes
the bucket.
$ aws s3 rb s3://amzn-s3-demo-bucket --force
Delete objects
To delete objects in a bucket or your local directory, use the
s3 rm command.
Syntax
$ aws s3 rm [--options]
For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of options,
see
s3 rm in the
AWS CLI Command Reference.
The following example deletes filename.txt from
s3://amzn-s3-demo-bucket/example.
Use the
s3 mv command
to move objects from a bucket or a local directory. The s3 mv command copies the
source object or file to the specified destination and then deletes the source object or
file.
Syntax
$ aws s3 mv [--options]
For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of available
options, see
s3 mv in the
AWS CLI Command Reference.
Warning
If you are using any type of access point ARNs or access point aliases in your Amazon S3
source or destination URIs, you must take extra care that your source and destination Amazon S3
URIs resolve to different underlying buckets. If the source and destination buckets are the
same, the source file or object can be moved onto itself, which can result in accidental
deletion of your source file or object. To verify that the source and destination buckets
are not the same, use the --validate-same-s3-paths parameter, or set the
environment variable AWS_CLI_S3_MV_VALIDATE_SAME_S3_PATHS to
true.
The following example moves all objects from
s3://amzn-s3-demo-bucket/example to
s3://amzn-s3-demo-bucket/.
Use the
s3 cp command
to copy objects from a bucket or a local directory.
Syntax
$ aws s3 cp [--options]
You can use the dash parameter for file streaming to standard input (stdin)
or standard output (stdout).
Warning
If you're using PowerShell, the shell might alter the encoding of a CRLF or add a CRLF
to piped input or output, or redirected output.
The s3 cp command uses the following syntax to upload a file stream from
stdin to a specified bucket.
Syntax
$ aws s3 cp - [--options]
The s3 cp command uses the following syntax to download an Amazon S3 file stream
for stdout.
Syntax
$ aws s3 cp [--options] -
For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For the complete list of
options, see
s3 cp in the
AWS CLI Command Reference.
The following example copies all objects from
s3://amzn-s3-demo-bucket/example to
s3://amzn-s3-demo-bucket/.
The following example streams the
s3://amzn-s3-demo-bucket/filename.txt file to stdout
and prints the contents to the console.
$ aws s3 cp s3://amzn-s3-demo-bucket/filename.txt -hello world
The following example streams the contents of s3://bucket-name/pre to
stdout, uses the bzip2 command to compress the files, and
uploads the new compressed file named key.bz2 to
s3://bucket-name.
The
s3 sync
command synchronizes the contents of a bucket and a directory, or the contents of two buckets.
Typically, s3 sync copies missing or outdated files or objects between the source
and target. However, you can also supply the --delete option to remove files or
objects from the target that are not present in the source.
Syntax
$ aws s3 sync [--options]
For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of options,
see
s3 sync in
the AWS CLI Command Reference.
The following example synchronizes the contents of an Amazon S3 prefix named
path in the bucket named amzn-s3-demo-bucket with the
current working directory.
s3 sync updates any files that have a size or modified time that are
different from files with the same name at the destination. The output displays specific
operations performed during the sync. Notice that the operation recursively synchronizes
the subdirectory MySubdirectory and its contents with
s3://amzn-s3-demo-bucket/path/MySubdirectory.
$ aws s3 sync . s3://amzn-s3-demo-bucket/path
upload: MySubdirectory\MyFile3.txt to s3://amzn-s3-demo-bucket/path/MySubdirectory/MyFile3.txt
upload: MyFile2.txt to s3://amzn-s3-demo-bucket/path/MyFile2.txt
upload: MyFile1.txt to s3://amzn-s3-demo-bucket/path/MyFile1.txt
The following example, which extends the previous one, shows how to use the
--delete option.
// Delete local file
$ rm ./MyFile1.txt
// Attempt sync without --delete option - nothing happens
$ aws s3 sync . s3://amzn-s3-demo-bucket/path
// Sync with deletion - object is deleted from bucket
$ aws s3 sync . s3://amzn-s3-demo-bucket/path --delete
delete: s3://amzn-s3-demo-bucket/path/MyFile1.txt
// Delete object from bucket
$ aws s3 rm s3://amzn-s3-demo-bucket/path/MySubdirectory/MyFile3.txt
delete: s3://amzn-s3-demo-bucket/path/MySubdirectory/MyFile3.txt
// Sync with deletion - local file is deleted
$ aws s3 sync s3://amzn-s3-demo-bucket/path . --delete
delete: MySubdirectory\MyFile3.txt
// Sync with Infrequent Access storage class
$ aws s3 sync . s3://amzn-s3-demo-bucket/path --storage-class STANDARD_IA
When using the --delete option, the --exclude and
--include options can filter files or objects to delete during an s3
sync operation. In this case, the parameter string must specify files to exclude
from, or include for, deletion in the context of the target directory or bucket. The
following shows an example.
Assume local directory and s3://amzn-s3-demo-bucket/path currently in sync and each contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt
'''
// Sync with delete, excluding files that match a pattern. MyFile88.txt is deleted, while remote MyFile1.txt is not.
$ aws s3 sync . s3://amzn-s3-demo-bucket/path --delete --exclude "path/MyFile?.txt"
delete: s3://amzn-s3-demo-bucket/path/MyFile88.txt
'''
// Sync with delete, excluding MyFile2.rtf - local file is NOT deleted
$ aws s3 sync s3://amzn-s3-demo-bucket/path . --delete --exclude "./MyFile2.rtf"
download: s3://amzn-s3-demo-bucket/path/MyFile1.txt to MyFile1.txt
'''
// Sync with delete, local copy of MyFile2.rtf is deleted
$ aws s3 sync s3://amzn-s3-demo-bucket/path . --delete
delete: MyFile2.rtf
Frequently used options for s3 commands
The following options are frequently used for the commands described in this topic. For a
complete list of options you can use on a command, see the specific command in the
AWS CLI version 2 reference
guide.
acl
s3 sync and s3 cp can use the --acl option.
This enables you to set the access permissions for files copied to Amazon S3. The
--acl option accepts private, public-read, and
public-read-write values. For more information, see Canned
ACL in the Amazon S3 User Guide.
When you use the s3 cp, s3 mv, s3 sync, or
s3 rm command, you can filter the results by using the
--exclude or --include option. The --exclude
option sets rules to only exclude objects from the command, and the options apply in the
order specified. This is shown in the following example.
Local directory contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt
// Exclude all .txt files, resulting in only MyFile2.rtf being copied$ aws s3 cp . s3://amzn-s3-demo-bucket/path --exclude "*.txt"// Exclude all .txt files but include all files with the "MyFile*.txt" format, resulting in, MyFile1.txt, MyFile2.rtf, MyFile88.txt being copied$ aws s3 cp . s3://amzn-s3-demo-bucket/path --exclude "*.txt" --include "MyFile*.txt"// Exclude all .txt files, but include all files with the "MyFile*.txt" format, but exclude all files with the "MyFile?.txt" format resulting in, MyFile2.rtf and MyFile88.txt being copied$ aws s3 cp . s3://amzn-s3-demo-bucket/path --exclude "*.txt" --include "MyFile*.txt" --exclude "MyFile?.txt"
include
When you use the s3 cp, s3 mv, s3 sync, or
s3 rm command, you can filter the results using the
--exclude or --include option. The --include
option sets rules to only include objects specified for the command, and the options
apply in the order specified. This is shown in the following example.
Local directory contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt
// Include all .txt files, resulting in MyFile1.txt and MyFile88.txt being copied$ aws s3 cp . s3://amzn-s3-demo-bucket/path --include "*.txt"// Include all .txt files but exclude all files with the "MyFile*.txt" format, resulting in no files being copied$ aws s3 cp . s3://amzn-s3-demo-bucket/path --include "*.txt" --exclude "MyFile*.txt"// Include all .txt files, but exclude all files with the "MyFile*.txt" format, but include all files with the "MyFile?.txt" format resulting in MyFile1.txt being copied$ aws s3 cp . s3://amzn-s3-demo-bucket/path --include "*.txt" --exclude "MyFile*.txt" --include "MyFile?.txt"
grant
The s3 cp, s3 mv, and s3 sync commands
include a --grants option that you can use to grant permissions on the
object to specified users or groups. Set the --grants option to a list of
permissions using the following syntax. Replace Permission,
Grantee_Type, and Grantee_ID with your own values.
For more information about Amazon S3 access control, see Access control.
The following example copies an object into a bucket. It grants read
permissions on the object to everyone, and full permissions
(read, readacl, and writeacl) to the account
associated with user@example.com.
You can also specify a nondefault storage class (REDUCED_REDUNDANCY or
STANDARD_IA) for objects that you upload to Amazon S3. To do this, use the
--storage-class option.
The s3 cp, s3 mv, and s3 sync commands
include a --no-overwrite option that you can use to prevent overwriting
objects that already exist at the destination.
The following example copies an object from a bucket to the local directory only
if it does not already exist in the local directory.
When you use this option, the command is performed on all files or objects under the
specified directory or prefix. The following example deletes
s3://amzn-s3-demo-bucket/path and all of its contents.
