Using S3 Storage

Need help?

Schedule a consultation with an Azul performance expert.

When you use the Warp engine with -XX:CRaCEngine=warp, the checkpoint can be stored on S3-compatible buckets. Define the location starting with s3://, for example, -XX:CRaCCheckpointTo=s3://bucket/some-key.

You must configure the environment with credentials in ~/.aws or environment variables, as specified by the standard approach of the AWS SDK. You can finetune some settings with -XX:CRaCEngineOptions:

Setting Description Default

Setting	Description	Default
`s3.connections`	Set the number of connections to be used for upload or download.	`8`
`s3.endpoint`	The endpoint by default is configured to upload to AWS S3. Use this setting to change the endpoint URL and upload to another S3 compatible storage. The endpoint must start with `s3.` to let the AWS SDK recognize virtual-host-styled requests, for example, `https://s3.foobar.example.com`.	AWS S3
`s3.log.level`	Configure AWS logging level.	Off
`s3.log.prefix`	Configure AWS logging prefix.	`aws_sdk_`
`s3.upload.part_size`	Images stored in S3 can be split over several upload connections. AWS supports up to 10000 parts each between 5MB and 5GB. When your checkpoint is larger than 50GB, you might need to increase the value.	5MB
`s3.download.part_size`	Images stored in S3 can be split over several download connections and the size can be specified with this setting.	5MB

s3.connections

Set the number of connections to be used for upload or download.

8

s3.endpoint

The endpoint by default is configured to upload to AWS S3. Use this setting to change the endpoint URL and upload to another S3 compatible storage. The endpoint must start with s3. to let the AWS SDK recognize virtual-host-styled requests, for example, https://s3.foobar.example.com.

AWS S3

s3.log.level

Configure AWS logging level.

Off

s3.log.prefix

Configure AWS logging prefix.

aws_sdk_

s3.upload.part_size

Images stored in S3 can be split over several upload connections. AWS supports up to 10000 parts each between 5MB and 5GB. When your checkpoint is larger than 50GB, you might need to increase the value.

5MB

s3.download.part_size

Images stored in S3 can be split over several download connections and the size can be specified with this setting.

5MB

Note	All available settings for `-XX:CRaCEngineOptions=<value>` can be listed by starting your JVM with `-XX:CRaCEngine=warp -XX:CRaCEngineOptions=help`.