Appearance

AWS S3 ​

Sprocket supports transferring files to and from AWS S3.

Supported AWS S3 URLs ​

The following cloud storage URLs for AWS S3 are supported:

  • s3 schemed URLs in the form s3://<bucket>/<object> (see default region).
  • https schemed URLs in the form https://<bucket>.s3.<region>.amazonaws.com/<object>.
  • https schemed URLs in the form https://<region>.s3.amazonaws.com/<bucket>/<object>.

Authentication ​

Sprocket supports authentication to AWS S3 using access keys.

Follow these instructions for creating a new access key for a user.

This overrides any AWS S3 authentication settings in sprocket.toml.

Environment variables ​

Use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to configure AWS S3 authentication in Sprocket.

This overrides any AWS S3 authentication settings in sprocket.toml.

Command line options ​

Use the --aws-access-key-id and --aws-secret-access-key options to the sprocket run command to configure AWS S3 authentication in Sprocket.

This overrides any AWS S3 authentication settings in sprocket.toml.

Configuration file ​

AWS S3 authentication can be configured with the run.storage.s3.auth section in sprocket.toml:

toml
[run.storage.s3.auth]
access_key_id = "<access-key-id>"
secret_access_key = "<secret-access-key>"

WARNING

On Unix operating systems, it is recommended that your sprocket.toml has an access permission of 0600 if it contains secrets like AWS S3 secret access keys.

Default region ​

For s3 schemed URLs, the default region is us-east-1.

The default region can be changed via the following configuration in sprocket.toml:

toml
[run.storage.s3]
region = "<region>"

Additionally, the --aws-default-region CLI option to sprocket run or the AWS_DEFAULT_REGION environment variable can be used to override the default region from configuration.