Visit Azul.com Support

Installing Azul Zing System Tools

This page describes how to install Azul Zing System Tools (ZST)

Note
These prerequisites are only applicable to the Azul Zing Builds of OpenJDK (Zing) running with Azul Zing System Tools (ZST).

While Zing with ZST runs fine with certain default system settings, there are default OS settings that can cause response time sensitive applications to experience multi-hundred-of-millisecond stalls regardless of the JVM. Since Zing tends to eliminate or dramatically reduce JVM-related stalls, these remaining OS related stalls will tend to dominate the worst-case latency behavior of applications. It is therefore highly recommended that the following settings be applied.

To ensure that OS settings are appropriate for performance sensitive applications, apply the following settings.

If you are using a Nehalem power efficiency chip set in the latency-sensitive systems, you might need to disable the Intel idle driver.

Note
Installing ZST will cause a kernel panic if you are using Ice Lake hardware with 5-level paging. Use 4-level paging or disable ZST with this release.

Installing Azul Zing System Tools

Azul Zing System Tools installation packages are available from the Azul Platform Prime Customer Downloads page and are only available to Azul customers.

Perform the following steps to install and configure ZST:

  1. Login to your system as root administrator or use sudo.

  2. Download ZST installation package to your system <zst_install> directory. If you have installed Azul Platform Prime from the package repository, skip this step. The zing-zst package contains the kernel module and various tools. Only one version of zing-zst may be installed at a time.

    The ZST installation package is specific to the operating system version, service pack, and/or kernel on the installation machine.

    System ZST Installation Package Format

    RHEL, CentOS, CentOS Stream, Oracle Linux, Amazon Linux, Rocky Linux and Alma Linux

    zing-zst-<zst_API>.<kernel_version>-<zst_version>.<platform>.<architecture>.rpm

    For example:

    zing-zst-8c.4.18.0-5.22.8.0.4.1.el8.x86_64.rpm

    Amazon Linux

    zing-zst-<zst_API>.<kernel_version>-<zst_version>.<platform>.<architecture>.rpm

    For example:

    zing-zst-8c.2.6.32-5.22.8.0.4.el.x86_64.rpm

    SLES

    zing-zst-<zst_API>.<kernel_version>-<zst_version>.<platform>.<architecture>.rpm

    For example:

    zing-zst-8c.4.4.82-5.22.8.0.4.1.sles12.x86_64.rpm

    Ubuntu

    zing-zst_<zst_API>.<kernel_version>-<platform>-<zst_version>_<architecture>.deb

    For example:

    zing-zst_8c.5.4.0.x86.64-ub2004-5.22.8.0.4.5_amd64.deb

  3. If you have manually downloaded the ZST package, install ZST using the following commands appropriate for your operating system:

    System Commands to Install Packages

    RHEL, CentOS, CentOS Stream, Oracle Linux, Amazon Linux, Rocky Linux and Alma Linux

    sudo yum localinstall <zing zst package>

    SLES

    sudo zypper install <zing zst package>

    Ubuntu

    sudo apt install ./<zing zst package>

  4. If you are using the package repository, install ZST with the following commands appropriate for your operating system:

    System Commands to Install Packages

    RHEL, CentOS, CentOS Stream, Oracle Linux, Amazon Linux, Rocky Linux and Alma Linux

    sudo yum install zing-zst

    SLES

    sudo zypper install zing-zst

    Ubuntu

    sudo apt install zing-zst

ZST 5.21.x

ZST 5.21.x includes an installation behavior change that automatically configures Zing Memory as part of the ZST install. This eliminates the need to run system-config-zing-memory after the ZST install.

When the ZST is installed, the default for the configuration policy is reserve-at-launch – System Zing Memory will only be reserved when a Zing process starts.

If you normally configure System Zing Memory using the reserve-at-config policy to guarantee that the memory is reserved at the time of the configuration, then you will need to run system-config-zing-memory and select the reserve-at-config policy. Be certain that you select whether you want to use percentages of total system memory or specific amounts of memory because the defaults have changed.

The following table shows the list of the packages that have reserve-at-launch memory pre-configuration during the installation of ZST 5.21.x.

Operating System Package Zing Memory Configuration

RHEL6, RHEL6 MRG (RHEL6), Amazon Linux (RHEL DKMS), DEB DKMS, SLES DKMS, RHEL DKMS, OEL6, RHEL7, OEL7, SLES12, Ubuntu 14.04, Ubuntu 16.04, Ubuntu 18.04

reserve-at-launch configuration during installation

ZST 5.20.x

When installing ZST 5.20.x using RPM and DEB packages, you might need to configure Azul Platform Prime memory manually upon installation by running system-config-zing-memory. The following table shows the list of the packages that require manual memory configuration and the packages that have reserve-at-launch memory pre-configuration during the installation.

Operating System Package Zing Memory Configuration

RHEL5, OEL5 (RHEL5), RHEL6, RHEL6 MRG (RHEL6), Amazon Linux (RHEL DKMS), DEB DKMS, SLES DKMS, RHEL DKMS

manual configuration after installation

OEL6, RHEL7, OEL7, SLES11 SP4, SLES12, Ubuntu 14.04, Ubuntu 16.04

reserve-at-launch configuration during installation

To configure Azul Platform Prime memory manually, use the system-config-zing-memory command:

 
sudo system-config-zing-memory

(ZST 5.20.x-specific only) To accept default configuration settings type:y when prompted. See Configure Zing Memory for more information.

Verify the ZST installation by running the zing-ps -s command.

Installing ZST Using DKMS

This guide describes how to install ZST using the Dynamic Kernel Module Support (DKMS) framework. Use the ZST DKMS package if a ZST package is not yet available for the Linux kernel version of your system on the Software Downloads and Documents page.

Note
This information is intended for advanced users only. While this feature is tested and validated for a selected set of operating system versions, there are always risks. Test thoroughly in your staging environment before going into production. If you are installing on RHEL 8.x/7.x/6.x, Oracle Linux 7.x/6.x, SLES 12 SP5/SP4/SP3/SP2, Ubuntu 20.04/18.04/16.04 kernels, use the standard ZST installation packages.

To install Zing using DKMS, perform the steps listed below.

Complete Prerequisites

  1. Locate and install the kernel headers that match the kernel version that will be running the ZST. Install the DKMS package appropriate for your target operating system.

    DKMS is already included in Amazon Linux, Amazon Linux 2, Ubuntu, and Debian but not directly provided by CentOS, RHEL, SLES, Oracle Linux. For those systems, it can be installed from the Fedora EPEL repository. See also Using DKMS on RHEL, CentOS, or Oracle Linux to build ZST for other kernel versions.

  2. Run the commands as root or use sudo.

    System Commands to Install Packages

    RHEL, CentOS, CentOS Stream, Oracle Linux (Non-UEK Kernels), Amazon Linux, Rocky Linux and Alma Linux

    # yum install kernel-devel-$(uname -r) dkms

    SLES

    # zypper install kernel-default-devel-$(uname -r) dkms

    Oracle Linux (UEK Kernels)

    # yum install kernel-uek-devel-$(uname -r) dkms

    Ubuntu and Debian

    # apt-get install linux-headers-$(uname -r) dkms

  3. Depending upon the operating system the required packages include:

    • OpenSSL 0.9.8 or later

    • binutils required to ensure that any core files, generated when a Zing instance fails abnormally, include the shared libraries that were in use at the time

Install ZST

The ZST installation package contains a kernel loadable module and supporting scripts and tools.

To install ZST:

  1. Ensure that you have access to https://www.azul.com/software-downloads-and-documents/ to download Zing packages (contact [email protected] to obtain the required credentials).

  2. Login to your system as root or sudo.

  3. Download and install the ZST package.

  4. To install ZST, use the command appropriate for your system:

    System Commands to Install Packages

    RHEL, CentOS, CentOS Stream, Oracle Linux, Amazon Linux, Rocky Linux and Alma Linux

    # yum localinstall zing-zst-<zst-api-version>.dkms-<zst_version>.x86_64.rpm

    For example:

    # yum localinstall zing-zst-8b.dkms.2.6.32-5.22.5.0.4.el.x86_64.rpm

    SLES

    # zypper install <zst-api-version>.dkms-<zst_version>.<platform>.rpm

    For example:

    # sudo zypper install zing-zst-8c.4.4.82-5.22.13.0.2.1.sles12.x86_64.rpm

    Ubuntu and Debian

    # apt install ./zing-zst_<zst-api-version>.dkms-<zst_version>.<platform>.deb

    For example:

    # sudo apt install ./zing-zst_8b.dkms.3.2.0.x86.64-server-5.22.5.0.4_amd64.deb

A sample system response is shown below:

 
Preparing... ################################# [100%] Updating / installing... 1:zing-zst-8b.dkms.2.6.32-5.22.5.0.################################# [100%] Creating symlink /var/lib/dkms/zing_mm/5.22.5.0-4/source -> /usr/src/zing_mm-5.22.5.0-4 DKMS: add completed. Kernel preparation unnecessary for this kernel. Skipping... Building module: cleaning build area... make -j24 KERNELRELEASE=3.10.0-957.27.2.el7.x86_64 KERNELDIR=/lib/modules/3.10.0-957.27.2.el7.x86_64/build DKMS=1 INTERNAL_BUILD_NAME=zst-5.22.5.0 BUILD_SUFFIX= BUILD_NUMBER=4..... cleaning build area... DKMS: build completed. zing_mm.ko.xz: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/3.10.0-957.27.2.el7.x86_64/extra/ Adding any weak-modules depmod.... DKMS: install completed. zing-memory: INFO: Restarting... zing-memory: INFO: Stopping... zing-memory: INFO: stop successful zing-memory: INFO: Starting... INFO: Zing Memory reserved using reserve-at-launch policy INFO: az_pmem_set_device_limit to 500034437120 bytes succeeded INFO: az_pmem_set_overall_device_limit to 243259146240 bytes succeeded INFO: az_pmem_fund_limit_transfer (to 7, from 0, bytes 500034437120) succeeded INFO: az_pmem_fund_limit_transfer (to 1, from 7, bytes 135144669184) succeeded INFO: az_pmem_fund_limit_transfer (to 3, from 7, bytes 135144669184) succeeded INFO: az_pmem_fund_limit_transfer (to 0, from 7, bytes 229745098752) succeeded INFO: Set ContingencyMinimumFreeMemory to 209715200 INFO: Successfully set fund MinimumFreeMemory INFO: Set PausePreventionMinimumFreeMemory to 209715200 INFO: Successfully set fund MinimumFreeMemory INFO: Set PausePreventionMinimumFreeMemory to 209715200 INFO: Successfully set fund MinimumFreeMemory INFO: You can now run Java processes up to -Xmx219102m or -Xmx213g. INFO: Azul pmem initialized successfully. zing-memory: INFO: start successful zing-memory: INFO: restart successful

Azul Zing System Tools Post Installation Tasks

The following post-installation steps are not directly required when starting to use Azul Zing Builds of OpenJDK (Zing) with Azul Zing System Tools (ZST), but it is recommended to go through them to avoid later problems in production systems.

Transferring JDK Customizations

If you have applied customizations to the existing non-ZST Azul Platform Prime installation location, you might need to transfer those to the new installation location.

Those customizations could be, for example, JAR files added to the directory $JAVA_HOME/jre/lib/ext or changes in the security settings below $JAVA_HOME/jre/lib/security. Azul Platform Prime offers the same directories to store custom jar files and security policies.

Setting an Unlimited ulimit

Ensure the ulimit is unlimited for virtual memory, resident memory, and core file size. At the command line, type:

 
ulimit -v unlimited -m unlimited -c unlimited

Some operating systems default the ulimit to -v unlimited, others do not. The -v unlimited value is required prior to launching a Zing instance for the first time. The -m and -c options issue warnings and do not prevent a Zing instance from launching.

Set the ulimit prior to launching any Zing instances or add it to a system start command file.

Ensuring Sufficient Virtual Memory Areas

If you are running Oracle HotSpot or OpenJDK and using XX:+UseLargePages, you must ensure there is sufficient memory allocated to each: Azul Platform Prime and Linux memory.

To ensure that OS settings are appropriate for performance sensitive applications, apply the following settings:

  • Disable transparent_hugepages on your Linux system where you are running Azul Platform Prime.

  • Turn off SWAP either by removing swap or by setting swappiness to 0`. Best practice removes swap.

  • Set vm.min_free_kbytes to 1 GB or more. Best practice assigns more than 1 GB.

  • Set vm.max_map_count=1000000.

JDK Configuration Compatibility

If your Java application requires access to any javax.crypto unlimited strength policy files:

  • If they are included with your existing version of the JDK, copy them to the Linux JDK where the Zing software package is installed.

  • If you do not have an existing JDK, download the javax.crypto policy files from the Oracle website directly and store them to the Linux JDK where the Zing software package is installed.

Since Zing 17.11.0.0 the Java Cryptography Extension (JCE) jurisdiction policy files are included. In order to use encryption with key lengths more than 128 bit, no installation of JCE files is required and adding the following line in your code is sufficient:

 
Security.setProperty("crypto.policy", "unlimited");

With Zing 17.11.0.0 no additional encryption algorithm code was added compared to earlier releases, it only enabled existing code by default.

The previous approach of copying the policy files is still available and all policy files for limited and unlimited key lengths are now included. To activate the unlimited key length:

 
cp /opt/zing/zing-jdk8/jre/lib/security/policy/unlimited/* /opt/zing/zing-jdk8/jre/lib/security/

Disabling 5-level Paging on Ice Lake Hardware

If you use ZST on Ice Lake hardware, you must disable 5-level paging.

  1. Determine whether 5-level paging is enabled.

    Run cat /proc/cpuinfo | grep -m 1 la57

    If the output is empty, 5-level paging mode is either not supported or disabled. If the output is not empty, 5-level paging is enabled, and you should proceed to the next step.

  2. Switch back to 4-level paging.

    Add no5lvl to the list of command line options for the Linux boot loader.

    For example, using /etc/default/grub, update the GRUB_CMDLINE_LINUX value to include the no5lvl option.

    GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet no5lvl"

  3. Update grub

    On Ubuntu, run update-grub.

    On RHEL8 Legacy mode, run grub2-mkconfig -o /boot/grub2/grub.cfg.

    On RHEL8 UEFI mode, run grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg.

  4. Reboot the host machine.