Visit Azul.com Support

Core Bundling

Table of Contents
Need help?
Schedule a consultation with an Azul performance expert.
Contact Us

If you contact your Azul technical support representative regarding a failure, you might be asked to provide core files or a core bundle that includes the core files and crash associated files, (such as, log files generated by the Azul Zing Builds of OpenJDK [Zing] instance, system log files, and versions of shared libraries used by the Zing instance). Providing this information, in addition to the core files, helps Azul to properly diagnose and quickly identify failures due to Azul Platform Prime.

If a failure occurs due to a Zing instance or Zing instance-related Java application failure a core file is created, provided ulimit -c unlimited is set. The uli`m`it setting is required to generate a core file.

Use the following two scripts for bundling the core files with the shared libraries that were in use by the Zing instance running the Java application when it failed:

  • core-bundler (introduced in Zing 19.06.0.0 for improved work with Docker containers)

  • zing-core-bundler (a previous version of the script)

About the core-bundler Script

The core-bundler script is a self-contained bash script introduced in the Azul Platform Prime 19.06.0.0. It works well in a container environment.

When the Azul Zing Builds of OpenJDK (Zing) crashes, a core file is dumped on a disk (if core dump is configured and enabled). See Core Bundling with Azul Platform Prime for more information.

The core-bundler script is used to create an encrypted bundle containing the core file along with object files used by the crashed application and log files produced by Zing.

It collects and bundles the core file along with the dynamic libraries to an encrypted gzipped tarball (.tgz.gpg) archive.

core-bundler requires gpg, gtar/tar, pigz/gzip, gawk/awk.

The core-bundler script does not require installation and is located in the <zvm installation directory>/etc/core-bundler directory once Zing is installed on your system.

Using core-bundler

The core-bundler syntax and options are listed below.

Usage:

 
core-bundler -h|--help
core-bundler <command> <args>
Command Description

create

Creates an encrypted archive with files needed to successfully debug the core dump.

extract

Extracts an encrypted core bundle to the current directory.

help

Provides information about a specific command usage.

Creating an Encrypted Bundle

To create a bundle, use the core-bundler create command:

 
core-bundler create [--hsfile <hsfile>] [--docker-container <cid> | --docker-image <iid> [-v <vlist> ...]] [<force-options> ...] <core>
Option Description

<core>

Specifies the path to the core file to create a bundle for.

<hsfile>

Specifies the path to the hs_err log file left by the same VM process.

Default: hs_err fileis in the same directory as the corefile.

<cid>

Docker container ID of the container that dumped the core or of a similar container (if the original one is not available).

<iid>

Docker image ID of the image that can be used to create a container similar to the one that dumped the core.

<vlist>

Docker volumes mount list (in the format of -v <src>:<dst>). Multiple -v options allowed.

<force-options>

core-bundler is interactive by default but you can force some/all actions by providing the following options:

  • --no-host-check forces bundling on the current host even if it differs from the host where coredump was produced

  • --no-preview does not open an EDITOR to preview the list of files to bundle

  • --no-space-check does not perform a free disk space check

  • --overwrite overwrites the bundle if a file with the same name already exists

For example:

 
$ /opt/zing/zing-jdk8/etc/core-bundler create core.57075
core-bundler version 0.3
Extracting hostname from the core file. Could take a while ... Done
The bundle file will contain core file, log files, executable, dynamic libraries and some system information.
For security reasons the bundle will be encrypted.
Do you want to review (and/or modify) list of files to be included before proceeding? ([Y]es / [N]o / [C]ancel) [default 'n']: n
Creating an encrypted bundle: /tmp/corelibs.57075_testhost.tgz.gpg ...
Encrypted bundle file has been created and is ready to be shared with Azul!
File: /tmp/corelibs.57075_testhost.tgz.gpg
** PLEASE DO NOT UPLOAD THE PASSWORD FILE TO AZUL **
Azul already has a private key which can decrypt the core bundle.
The password is only for your use, in case YOU need to extract files from the bundle.
To extract the file use core-bundler extract corelibs.57075_testhost.tgz.gpg.

Extracting an Encrypted Bundle

The password file that is created next to the bundle file can be used to decode/extract files from the bundle.

To extract an archive, use the core-bundler extract command:

 
core-bundler extract [--list-only] <encrypted-bundle>
Option Description

<encrypted-bundle>

Specifies the path to the encrypted bundle (corelibs.xxxx.tgz.gpg).

--list-only

Does not extract files. Lists the archive content only.

For example:

 
$ /opt/zing/zing-jdk8/etc/core-bundler extract /tmp/corelibs.57075_testhost.tgz.gpg
core-bundler version 0.3
Extracting data from /tmp/corelibs.57075_testhost.tgz.gpg ... Done

The above command extracts the content of the bundle to the current directory.

Dealing with Core Files from Docker Container Environment

Core files dumped by Zing while running in a container need to be co-bundled with the same dynamic libraries that were used by Zing during the crashed run. So the core-bundler script needs a way to access files from the container.

If the Docker container is either still alive or was started without the --rm option, one can use the --docker-container <cid> option to pass the ID of the container to the script.

In this case, the script will auto-detect all the mappings that are used by the container and prepare a bundle:

For example:

 
$ /opt/zing/zing-jdk8/etc/core-bundler create core.15 --docker-container 123abc456def
core-bundler version 0.3
The bundle file will contain core file, log files, executable, dynamic libraries and some system information.
For security reasons the bundle will be encrypted.
Do you want to review (and/or modify) list of files to be included before proceeding? ([Y]es / [N]o / [C]ancel) [default 'n']: n
Creating an encrypted bundle: /tmp/corelibs.15_123abc456def.tgz.gpg ...
Encrypted bundle file has been created and is ready to be shared with Azul!
File: /tmp/corelibs.15_123abc456def.tgz.gpg
PLEASE DO NOT UPLOAD THE PASSWORD FILE TO AZUL
Azul already has a private key which can decrypt the core bundle.
The password is only for your use, in case YOU need to extract files from the bundle.
To extract the file use core-bundler extract corelibs.15_123abc456def.tgz.gpg.

If the container is not available, you can use the --docker-image option to specify an image ID that could be used to re-create the environment used during the crashed run. It is important to provide an image that has the same set of dynamic libraries installed.

Ideally, it should be the same image that the container was created from.

Note, that in this case the script has no information about Docker Volumes mappings. So you have to provide them with -v <vlist> option(s):

For example:

 
$ core-bundler create core.15 --docker-image fed654cba321 -v /tmp:/cores -v /home/user/data:/data --no-preview
core-bundler version 0.3
Creating an encrypted bundle: /tmp/corelibs.15_45e64c65dd4b.tgz.gpg ...
Encrypted bundle file has been created and is ready to be shared with Azul!
File: /tmp/corelibs.15_45e64c65dd4b.tgz.gpg
PLEASE DO NOT UPLOAD THE PASSWORD FILE TO AZUL *
Azul already has a private key which can decrypt the core bundle.
The password is only for your use, in case YOU need to extract files from the bundle.
To extract the file use core-bundler extract corelibs.15_45e64c65dd4b.tgz.gpg.

About the zing-core-bundler Script

This section describes the zing-core-bundler script.

Azul Zing Builds of OpenJDK (Zing) Core Bundling

If automatic core bundling is not enabled, you can manually bundle the core files with the shared libraries that were in use by the Zing instance running the Java application when it failed.

If you are running a system based on Red Hat Enterprise Linux 5.2 or CentOS Linux 5.2, the automatic core bundler is not on, so core files need to be bundled manually. For this purpose the script zing-core-bundler is provided. This script, as part of the ZST installation, is located in the /usr/sbin subdirectory of the ZST installation.

The zing-core-bundler bundles the required files needed to debug a Zing core. It assembles, collects, and bundles the core file along with the shared objects, data, and executable files in-use to an encrypted gzipped tarball (.tgz.gpg) archive.

To manually bundle the core file with shared libraries:

  1. On RHEL, SLES, and Ubuntu systems, ensure the development tools package group is installed.

     
    yum groupinstall 'Development Tools'
    Note
    		 Selected development tools are required to bundle RHEL 5 core files. If they are not installed, the shared libraries, which were used at the time of the crash and are essential to processing the core file, are not included.

  2. Locate the core file.

    Core files are bundled and stored in the current working directory of the process that crashed.

    [NOTE]Typically, you create a core bundle on the system where the core file was created, for example host A. If you run the zing-core-bundler command with a copy of the core file on a different system, for example host B, then include the -bundle-on-current-host option.

  3. On Ubuntu systems, ensure that the strings command is available. If it is not, install the package, binutils.

     
    # apt-get install binutils

  4. If you use the jsvc tool to start your application, to generate Azul Platform Prime core files:

     
    # kill -SIGABRT <PID>

  5. Ensure /usr/sbin is in your Zing executable path. For example, add the following to appropriate profile file:

     
    export PATH=/usr/sbin:$PATH

  6. Run the zing-core-bundler command. Go to the Zing host command line where ZST is installed. To create a bundle with only related shared libraries that does not include the .jar files, run:

     
    $ /usr/sbin/zing-core-bundler -core core<pid>

    For example:

     
    $ /usr/sbin/zing-core-bundler -core core.1234 -getall -hsfile hs_err_pid1234.log

    A compressed and encrypted tar file ending in .tgz.gpg is returned.

    In this example, the package contains the core file, the .jar files, and because the -getall option is specified, it also contains the related shared libraries which were used in the Zing instance running the Java application.

    • The -getall option ensures the bundle contains all application related data including the .jar files.

    • The -hsfile option creates an additional file that provides additional information useful for support to identify issues.

  7. Locate the bundled libs.<corepid>.tgz.gpg file. The libs.<corepid>.tgz.gpg file is created in the current directory. For example, if you execute the command from /home/test, the bundled libs.<corepid>.tgz.gpg file is created in /home/test. Ensure you have write permissions in the executing directory (i.e. /home/test).

  8. Send the generated bundle to Azul for failure analysis.

Note

  • Encryption error

    If there is a file with the same name as the core bundle, the core bundle will not be encrypted and the following message is displayed in the core bundling response.

    INFO: Encrypting core bundle libs.1234.tgz

    …​

    gpg: libs.1234.tgz: encryption failed: File exists

    ERROR: Encryption failed.

    Warning
    		 Core bundle is not encrypted

    INFO: Core bundle at libs.1234.tgz

  • Bundler host error

    Typically, you create the bundle on the same system where the core file was created. If you are not creating the bundle on the originating core file system, and you did not specify the required option, the following message is displayed in the core bundling response.

    ERROR: The bundler must run on host <hostname>

    ERROR: Use the argument '-bundle-on-current-host' to bundle on this host

To create the bundle on a different system:

  1. Copy the core file from the originating core file system (host A) to host B.

  2. On host B, include the -bundle-on-current-host option when you run the zing-core-bundler command.

The zing-core-bundler syntax and options are listed below.

Option Description

-core

Specify the core file to be bundled.

-bundle-on-current-host

Create bundle on current host. This allows bundling a core file that was generated on another host.

-exepath

Specify the path of the binary that was used to invoke Zing.

-decrypt

Specify file to decrypt

-getall

Include all possible files associated with the crash. This includes shared objects, data, and executable files.

-getexe

Include non-java executable with the bundle. This is bundled if it is available on the host at the time of bundling.

-hsfile

Explicitly specify the path of the hs_err file.

-passwordfile

Specify decryption password file.

-help

Print options help.

-output-dir | -o

Bundle will be produced at the location specified.

Decrypting Core Files

Core bundles created through zing-core-pattern, whether they are automatically or manually created and bundled, are encrypted. Files contained in the core bundle are not encrypted individually. The core bundle filename format is: libs.<corepid>.tgz.gpg, where <corepid> is a unique ID specific to the process that encountered an error and crashed.

Note
 Core files are not encrypted if zing-core-pattern is disabled.

To decrypt the Zing core bundle:

  1. Locate the core bundle file.

  2. Run the command:

     
    $ /usr/sbin/zing-core-bundler –decrypt libs.<corepid>.tgz.gpg

    For example:

     
    $ /usr/sbin/zing-core-bundler -decrypt libs.12830.tgz.gpg
INFO: Decrypting file libs.12830.tgz.gpg
gpg: CAST5 encrypted session key
gpg: encrypted with 1 passphrase
gpg: encrypted with RSA key, ID 2934892D
SUCCESS: Decrypted file libs.12830.tgz

Core Bundling with Docker Containers

You can select one of the following options to collect core dumps from the Docker container:

Option 1

Install zing-core-pattern to dump and bundle cores while using docker:

 
# system-config-zing-core-pattern enable

Bundles will be kept under /var/crash/<docker_container_id> on the host.

Option 2

Use zing-core-bundler script to bundle the core dumped by a docker container process:

 
# zing-core-bundler -c <core> --dockercontainerid <docker_container_id>