Amazon S3 ​

Sprocket supports transferring files to and from Amazon S3.

Supported Amazon S3 URLs ​

The following cloud storage URLs for Amazon S3 are supported:

• s3 schemed URLs in the form s3://<bucket>/<object> (see default region).
• https schemed URLs in the form https://<bucket>.s3.<region>.amazonaws.com/<object>.
• https schemed URLs in the form https://<region>.s3.amazonaws.com/<bucket>/<object>.

Authentication ​

Sprocket supports authentication to Amazon S3 using access keys.

Follow these instructions for creating a new access key for a user.

Environment variables (recommended) ​

Use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to configure Amazon S3 authentication in Sprocket.

This overrides any Amazon S3 authentication settings in sprocket.toml.

Command line options ​

Use the --aws-access-key-id and --aws-secret-access-key options to the sprocket run command to configure Amazon S3 authentication in Sprocket.

This overrides any Amazon S3 authentication settings in sprocket.toml.

Configuration file ​

Amazon S3 authentication can be configured with the run.storage.s3.auth section in sprocket.toml:

[run.storage.s3.auth]
access_key_id = "<access-key-id>"
secret_access_key = "<secret-access-key>"

Default region ​

For s3 schemed URLs, the default region is us-east-1.

The default region can be changed via the following configuration in sprocket.toml:

[run.storage.s3]
region = "<region>"

Additionally, the --aws-default-region CLI option to sprocket run or the AWS_DEFAULT_REGION environment variable can be used to override the default region from configuration.