CRaC Command-Line Options and System Properties

Table of Contents

Command-Line Options
System Properties

Need help?

Schedule a consultation with an Azul performance expert.

Command-Line Options

Java command-line options that control the behavior of CRaC.

Command Line Option Description Default

Command Line Option	Description	Default
-XX:CRaCCheckpointTo=<value>	Directory path where the checkpoint must be created. The value is required. To store the checkpoint in an S3-compatible bucket (only available with the Warp CRaC engine), set the `<value>` to, for example, `s3://bucket/some-key`. See Using CRaC on Amazon Web Services (AWS) > S3 Storage
-XX:CRaCRestoreFrom=<value>	The directory path where the CRaC image is stored which must be used for the restore. It replaces the initializing virtual machine on successful restore. The value is required.
-XX:+/-CRaCKeepRunning	Flag defining to not terminate the process after the checkpoint.	false
-XX:+/-CRaCConcurrentMemoryLoading	Used for restore to start the restored process before the image is fully unpacked into memory. As applications usually don’t require all data to perform some useful work, this can improve the startup lag. Direct memory mapping (non-compressed) is extremely fast compared to regular loading. Compressed images are slower as they need to be uncompressed. `-XX:+CRaCConcurrentMemoryLoading` may improve the time to the first operation after restore for large images. For small images, the effect may be negative.	false
-XX:+/-CRaCDirectMapImage	Flag to define if the checkpoint image is mapped into memory directly (true), or read into anonymous memory map (false). Direct mapping usually results in faster restore, but operations accessing the memory for the first time might experience decreased performance as the content is paged-in lazily.	true
-XX:CRaCOptimizeImage=<value>	After checkpoint, the image can be automatically reordered, based on the loading order in the profiling runs. This option sets the number of profiling runs. The default value of 0 means no optimization. With Warp, you can use the option `-XX:CRaCOptimizeImage=N` to perform N training runs to create checkpoints. After the checkpoint image is created, Warp iteratively performs N restores and determine the optimal layout of the image.	0
-XX:CRaCMinPid=<value>	The minimal PID value for a process that is restored. See Debugging Coordinated Restore at Checkpoint Failures > Using the CRaCMinPid Option. The default value is 128 only if the Java process is started with PID 1 (e.g., in a container). For processes started with a different PID, the PID is not adjusted unless this option is set.	128
-XX:+/-CRaCResetStartTime	Reset JVM’s start time and uptime on restore.	true
-XX:CRaCEngine=<value>	Integrated engines in Zulu: `criu`: Engine using Linux CRIU (Checkpoint/Restore In Userspace). `warp`: Engine which doesn’t require any additional capabilities, neither for checkpoint nor for restore. `simengine` and `pauseengine`: Simulation engines used for development purposes only. For more information, see CRaC Engines.	criu
-XX:+/-CRaCIgnoreRestoreIfUnavailable	Ignore `-XX:CRaCRestoreFrom` and continue initialization if restore is unavailable. This is useful when the process accidentally inherits an open file descriptor from the parent.	false
-XX:CRaCIgnoredFileDescriptors=<value>	Comma-separated list of file descriptor numbers or paths. All the file descriptors greater than 2 (stdin, stdout and stderr are excluded automatically), that are not in this list, are closed when the VM starts.	null
-XX:CRaCAllowedOpenFilePrefixes=<value>	List of path prefixes for files that can be open during the checkpoint. CRaC won’t throw an error upon detecting these and leaves the handling up to the Checkpoint/Restore engine. This option applies only to files opened by native code. For files opened by Java code use `-Djdk.crac.resource-policies=…`.	`/var/lib/sss/mc/` (on Linux)
-XX:+/-CRaCImageCompression	Compress the CRaC image during the checkpoint. Upon restore, the compressed format is detected automatically.	false
-XX:+/-CRaCHeapDumpOnCheckpointException	Dump the heap when a `CheckpointException` is thrown because of a failing CRaC precondition.	false
-XX:+/-CRaCPrintResourcesOnCheckpoint	Print the resources used to decide on a CheckpointException. This is a diagnostic feature and must be preceded with `-XX:+UnlockDiagnosticVMOptions`.	false
-XX:+/-CRaCTraceStartupTime	Trace the startup time.	false
-XX:+/-CRaCDoThrowCheckpointException	Throw `CheckpointException` if resource handle is found that can’t be checkpointed.	true
-XX:CRaCMaxHeapSizeBeforeCheckpoint=<value>	The maximum size of the heap before the checkpoint. By default, this equals to `-Xmx`.	0
-XX:CPUFeatures=<value>	The CPU feature set. Use `-XX:CPUFeatures=0xnumber` with `-XX:CRaCCheckpointTo` when you get an error during `-XX:CRaCRestoreFrom` on a different machine (with another architecture). `native` is the default (and fastest) `ignore` disables the CPU features check `generic` is compatible but slower compared to `native`	native
-XX:+/-ShowCPUFeatures	Show the features of this CPU to be possibly used for the `-XX:CPUFeatures=0xnumber` option.	false
-XX:+/-IgnoreCPUFeatures	EXPERIMENTAL! Flag defining to continue to run after `-XX:CRaCRestoreFrom` finds out some CPU features are missing.	false

-XX:CRaCCheckpointTo=<value>

Directory path where the checkpoint must be created. The value is required.

To store the checkpoint in an S3-compatible bucket (only available with the Warp CRaC engine), set the <value> to, for example, s3://bucket/some-key. See Using CRaC on Amazon Web Services (AWS) > S3 Storage

-XX:CRaCRestoreFrom=<value>

The directory path where the CRaC image is stored which must be used for the restore. It replaces the initializing virtual machine on successful restore. The value is required.

-XX:+/-CRaCKeepRunning

Flag defining to not terminate the process after the checkpoint.

false

-XX:+/-CRaCConcurrentMemoryLoading

Used for restore to start the restored process before the image is fully unpacked into memory. As applications usually don’t require all data to perform some useful work, this can improve the startup lag.

Direct memory mapping (non-compressed) is extremely fast compared to regular loading. Compressed images are slower as they need to be uncompressed. -XX:+CRaCConcurrentMemoryLoading may improve the time to the first operation after restore for large images. For small images, the effect may be negative.

false

-XX:+/-CRaCDirectMapImage

Flag to define if the checkpoint image is mapped into memory directly (true), or read into anonymous memory map (false). Direct mapping usually results in faster restore, but operations accessing the memory for the first time might experience decreased performance as the content is paged-in lazily.

true

-XX:CRaCOptimizeImage=<value>

After checkpoint, the image can be automatically reordered, based on the loading order in the profiling runs. This option sets the number of profiling runs. The default value of 0 means no optimization.

With Warp, you can use the option -XX:CRaCOptimizeImage=N to perform N training runs to create checkpoints. After the checkpoint image is created, Warp iteratively performs N restores and determine the optimal layout of the image.

-XX:CRaCMinPid=<value>

The minimal PID value for a process that is restored. See Debugging Coordinated Restore at Checkpoint Failures > Using the CRaCMinPid Option.

The default value is 128 only if the Java process is started with PID 1 (e.g., in a container). For processes started with a different PID, the PID is not adjusted unless this option is set.

128

-XX:+/-CRaCResetStartTime

Reset JVM’s start time and uptime on restore.

true

-XX:CRaCEngine=<value>

Integrated engines in Zulu:

criu: Engine using Linux CRIU (Checkpoint/Restore In Userspace).
warp: Engine which doesn’t require any additional capabilities, neither for checkpoint nor for restore.
simengine and pauseengine: Simulation engines used for development purposes only.

For more information, see CRaC Engines.

criu

-XX:+/-CRaCIgnoreRestoreIfUnavailable

Ignore -XX:CRaCRestoreFrom and continue initialization if restore is unavailable. This is useful when the process accidentally inherits an open file descriptor from the parent.

false

-XX:CRaCIgnoredFileDescriptors=<value>

Comma-separated list of file descriptor numbers or paths. All the file descriptors greater than 2 (stdin, stdout and stderr are excluded automatically), that are not in this list, are closed when the VM starts.

null

-XX:CRaCAllowedOpenFilePrefixes=<value>

List of path prefixes for files that can be open during the checkpoint. CRaC won’t throw an error upon detecting these and leaves the handling up to the Checkpoint/Restore engine. This option applies only to files opened by native code. For files opened by Java code use -Djdk.crac.resource-policies=….

/var/lib/sss/mc/ (on Linux)

-XX:+/-CRaCImageCompression

Compress the CRaC image during the checkpoint. Upon restore, the compressed format is detected automatically.

false

-XX:+/-CRaCHeapDumpOnCheckpointException

Dump the heap when a CheckpointException is thrown because of a failing CRaC precondition.

false

-XX:+/-CRaCPrintResourcesOnCheckpoint

Print the resources used to decide on a CheckpointException. This is a diagnostic feature and must be preceded with -XX:+UnlockDiagnosticVMOptions.

false

-XX:+/-CRaCTraceStartupTime

Trace the startup time.

false

-XX:+/-CRaCDoThrowCheckpointException

Throw CheckpointException if resource handle is found that can’t be checkpointed.

true

-XX:CRaCMaxHeapSizeBeforeCheckpoint=<value>

The maximum size of the heap before the checkpoint. By default, this equals to -Xmx.

-XX:CPUFeatures=<value>

The CPU feature set. Use -XX:CPUFeatures=0xnumber with -XX:CRaCCheckpointTo when you get an error during -XX:CRaCRestoreFrom on a different machine (with another architecture).

native is the default (and fastest)
ignore disables the CPU features check
generic is compatible but slower compared to native

native

-XX:+/-ShowCPUFeatures

Show the features of this CPU to be possibly used for the -XX:CPUFeatures=0xnumber option.

false

-XX:+/-IgnoreCPUFeatures

EXPERIMENTAL! Flag defining to continue to run after -XX:CRaCRestoreFrom finds out some CPU features are missing.

false

System Properties

The following system properties for CRac can be set:

jdk.crac.resource-policies: see File Descriptor Policies
jdk.crac.collect-fd-stacktraces: see Debugging Coordinated Restore at Checkpoint Failures > File Descriptors in Java Code
jdk.crac.trace-startup-time=true: add this to print startup time values to track how long the Java-side of the restore takes. This prints to stdout:
- STARTUPTIME <SYSTEM NANOTIME> restore
- STARTUPTIME <SYSTEM NANOTIME> restore-finish
jdk.crac.globalContext.impl: this property has two options:
- OrderedContext (default): Lets you register the resource when the checkpoint/restore is already executing, but this Resource’s beforeCheckpoint() and afterRestore() methods won’t be called during this checkpoint/restore.
- BlockingOrderedContext: Lets you assert that there is no new Resource registered during execution of the checkpoint/restore. The thread trying to register a new resource is blocked (this can lead to a deadlock).

Set these properties when you execute Java, for example:

 java -XX:CRaCCheckpointTo=... -Djdk.crac.resource-policies=/path/to/file.yml ...