Azul Platform Prime Errors

The error message is displayed when running Azul Zing Builds of OpenJDK (Zing) on your machine with an unsupported kernel or operating system.

Solution:

Install the ZST component. See Installing the Azul Zing System Tools for related installation details.

When running zing-licensed --configure on Ubuntu 16.04, the following error message displays:

zing-licensed --configure

Zing Error: libcurl.so.3: cannot open shared object file: No such file or directory

ERROR: Please install libcurl package with 'libcurl.so.3' or 'libcurl.so.4'

Solution:

When abrtd running zing core_pattern cannot overwrite /proc/sys/kernel/core_pattern.

Solution:

Scan the file, core_pattern, for any reference piped ( \| ) core_pattern. For example:

 
$ more /proc/sys/kernel/core_pattern

\|/usr/libexec/abrt-hook-ccpp /var/spool/abrt %s %c %p %u %g %t %h %e 636f726500

If a piped ( \| ) core_pattern is found:

The tick profiler shutdowns automatically when an attempt is made to allocate more than the configured maximum number of tick profiling buffers. The reason is printed: Max Buffers Exceeded.

Solution:

Increase the maximum number of permissible tick profiling buffers by increasing the value of EventTickBuffersMaxAllocatedBeforeShutoff in the command line arguments. To avoid data structure resizes, use EventTickBuffersAllocated to simultaneously increase the chunk size used at initialization and for expansion. For example:

java -XX:ARTAPort=9990 –XX:+UseTickProfiling -XX:EventTickBuffersAllocated=4096 -XX:EventTickBuffersMaxAllocatedBeforeShutoff=12288 <java-class>

For best performance, keep the number of threads in the application to a reasonable number based on the CPU resources available.

Application hangs, and eventually crashes with a checkpoint timeout, when real time threads run tight loops.

Background:

In these conditions, lower priority threads are not getting sufficient CPU time to reach checkpoint.

Solution:

When the parameter CheckpointBoostPriority is configured, Zing temporarily boosts the priority of real time threads to the configured value.

For example, one way to use this safely includes:

The OOP (Ordinary Object Pointer) Table - KlassTable contains a unique ID for each loaded class. Each entry is 8 bytes long. The entries will be freed when the corresponding classes are unloaded.

Solution:

The KlassTableSize option needs to be set appropriately for the number of live classes in the application, which can be monitored using PrintGCDetails.

The default value of KlassTableSize is based on the Java heap size, For example:

The flag -XX:+JNIDetachThreadsIfNotDetached is needed by any application that attaches a JNI thread to the VM but does not detach the thread before calling pthread_exit. For example with IBM WebSphere MQ Low Latency Messaging (LLM) applications.

If this error occurs, an error report file with more information is saved in, for example, /home/<user>/tmp5/hs_err_pid16620.log.

Solution:

If you receive this error, include the following flag in your Zing commands:

-XX:+JNIDetachThreadsIfNotDetached

You must stop all Zing Java processes before uninstalling or upgrading. Otherwise, you might see the following error message:

%preun(zing-zst-<version>) scriptlet failed, exit status1

Solution:

Run the zing-ps tool to list any running Zing processes. Stop the processes. Re-run zing-ps to verify all Zing processes are stopped. Reissue the uninstallation or upgrade commands.

ZST is either not loaded properly or has not been configured yet.

Solution:

Please confirm the zing-zst package is properly installed and configured.

Please run system-config-zing-memory as root. See man zing-installation for more information.

Running Linux processes are taking up a large amount of memory while attempting to configure Azul Platform Prime memory. Or, you have selected values that are too aggressive using the wizard or directly in the pmem.conf configuration file.

Solution 1:

Stop any processes that use significant resources while configuring Azul Platform Prime. You need to reevaluate your resource budget if you plan to run these concurrently with Azul Platform Prime.

Solution 2:

Try again with less aggressive memory settings.

Long after system startup (and after much memory has been consumed back and forth), ZST might fail to allocate the requested amount of memory, as it might (a) already be occupied or (b) exist in fragmented state that cannot be defragmented into 2MB pages. Transparent huge pages on RHEL 6.x can help recover 2MB pages, but might not be absolutely reliable, and on other systems (RHEL 5.x, SUSE 11, and other non-RHEL-6 kernel prior to 2.6.38) it is unlikely that you can find 75% of system memory in 2MB page form after the system has run loads for a while and cached files.

Solution:

See Enhanced Linux Memory Defragmentation.

The kernel you are running has no associated configuration file in the system /boot directory. A normal kernel installation and build will always include such a file. This is a serious error indicating that you are running a kernel in a non-standard way.

Solution:

You are likely using a non-supported system configuration. The solution is to use a standard configuration.

Kernel panic occurs when you are starting Azul Platform Prime on a paravirtualized instance.

Explanation:

Xen virtualization and in Amazon EC2 environments, paravirtualization is not supported in Azul Platform Prime. Only HVM (Hardware-assisted Virtual Machine) instances are supported. See the Azul Platform Prime System Requirements document for the complete list of the supported operating systems and kernels.

Azul Platform Prime does not support the -XX:+UseLargePages option. Azul Platform Prime pages are always large pages and they come from the ZST.

ZST and the HugePages memory conflict, as they both reserve memory for themselves. The ZST memory service reserves and manages its own memory space, so Large Memory Pages configured by the operating system make it difficult or impossible for the ZST memory service to reserve enough pages for Azul Platform Prime. Therefore, set the output from the three Huge variables to zero.

Solution:

To set the huge variables to zero:

The reason for the killed process is the memory pressure the VM Balloon Driver creates by over-expanding its virtual balloon and consuming the memory used by Azul Platform Prime.

Sample error message:

 
java invoked oom-killer
Out of memory: Kill process 6465 (java) score 38 or sacrifice child
Killed process 6465 (java) total-vm: ...

Solution Because the VM Balloon Driver is running on most standard Linux systems by default, it is recommended to disable the Balloon Driver by either of the following methods:

You might encounter the following error when running ZST on an unsupported kernel using reserve-at-launch:

 
Enter reserve-at-(c)onfig or reserve-at-(l)aunch [default 'c']: l
Zing supports only the 'reserve-at-config' policy with the OS/kernel version
on this system. To override, please refer to the Zing documentation.

This error might occur because the system-config-zing-memory script needs to identify if the kernel supports node compaction or not.

Solution:

To bypass the corresponding check:

You might encounter the following error when trying to start Azul Platform Prime memory on Azul Linux:

 
zing-memory: INFO: Starting...
Can't locate Sys/Syslog.pm

Solution:

Ensure you have installed the following packages are installed:

 
yum localinstall --nogpgcheck openssl binutils curl
yum install perl-Sys-Syslog.x86_64

You might encounter the following error when trying to start Azul Platform Prime memory:

 
zing-memory: INFO: Starting...
/usr/lib/zing/bin/tar: zm.o_<kernel>: Not found in archive
/usr/lib/zing/bin/tar: zm_linux.o_<kernel>: Not found in archive
/usr/lib/zing/bin/tar: Exiting with failure status due to previous errors
zing-memory: INFO: Zing ZST support for kernel <kernel> is unavailable. Please upgrade ZST.

Solution:

Upgrade ZST as described in Upgrading Azul Zing System Tools.

You might encounter the following error when trying to run Azul Platform Prime on CoreOS:

 
Error response from daemon: linux runtime spec devices: error gathering device information while adding custom device "/dev/zing_mm0": lstat /dev/zing_mm0: no such file or directory
Error: failed to start containers: <container ID>

Solution:

Run the following commands on privileged container:

 
ln -s /run/systemd/journal/dev-log /dev/log
service zing-memory restart

and the following command on the CoreOS host:

 
/lib/modules/zing-memory create_devices

You might encounter the following error when observing high time-to-checkpoint or event time-to-checkpoint crashes:

 
fatal error: Checkpoint sync time longer than 50000 ms detected

Before this fatal error, you must have seen several warnings when either 1 second, 20%, or 80% of CheckpointTimeout was reached:

 
Detected TTSP issue: start: nnn.nnn wait: nnn.nnn
For details on how to modify profiler configuration (for example, increase the timeout limit with -XX:CheckpointTimeoutDelay=100000 or turn off crashing with -XX:-DieOnSafepointTimeout), see Safepoint Profiler.

Solutions:

Use any from below to fix the error:

You might encounter the following error when updating information in the alternatives system:

 
update-alternatives --set java /opt/zing/zing-jdk8/bin/java
/opt/zing/zing-jdk8/bin/java has not been configured as an alternative for java

Solution:

Use the full path to the zing-jdk directory. For example:

update-alternatives --set java /opt/zing/zing-jdk1.8.0-19.04.0.0-7-x86_64/bin/java

The libjvm.so library has a dependency on the libstdc++ version 3.4.20 or higher.

Solution:

Upgrade libstdc++ to be greater than 3.4.20.

Alternatively, resolve the library dependency by using the library that is shipped with Azul Platform Prime: append the LD_LIBRARY_PATH environment variable to libstdc path in zing-jdk (e.g., export libstdc LD_LIBRARY_PATH=$LD_LIBARARY_PATH:/opt/zing/zing-jdk8//jre/lib/amd64/server/../../../../etc/libc++/) and relaunch your application.

 
$ java -version
Error: dl failure on line 1095
Error: failed /opt/zing/zing-jdk1.8.0-19.10.0.0-6-x86_64/jre/lib/amd64/server/libjvm.so, because libpam.so.0: cannot open shared object file: No such file or directory*

This error message is displayed when running Azul Platform Prime on containers due to a missing pam library.

Solution:

If you received this error, install pam.x86_64:

$ sudo yum install pam.x86_64

 
$ core-bundler c core.<pid> --docker-container <container_id>
core-bundler version 0.3
Error: 'core-bundler' failed: gpg not found
Run 'core-bundler help' for more information

In order to bundle Azul Platform Prime core files, GPG is required to be installed.

Solution:

If you received this error message, install GPG:

$ sudo apt install gpg

 
Error occurred during initialization of VM
Failed to initialize memory management system

This error can occur when migrating from a system where a JVM was configured to use static large pages.

Solution:

To check if a number of static large pages were previously configured on this system, run the following command to list the number of 2MByte pages:

sysctl vm.nr_hugepages

If no large pages are configured, remove -XX:+UseLargePages from your java command line.

If large pages are configured and performance tests show a benefit for your application, refer to Using Huge Pages with Azul Platform Prime for information about enabling transparent huge pages.

If Using Huge Pages with Azul Platform Prime is not applicable to your use case, use -XX:GPGCNoZSTBackingStoragePath. See Using Azul Zing Builds of OpenJDK Command-Line Options for details.

The error message displays for both ZST and non-ZST modes of Azul Platform Prime at startup in the following cases:

Solution:

Add more memory, reduce Xmx, or stop other running processes.

 
Error occurred during initialization of VM
Failed to initialize memory management system

This error message is displayed in the non-ZST mode if the JVM detects at startup that there is not enough memory available on the system to fund the whole of Xmx.

Solution:

Add more memory, reduce Xmx, or stop other running processes.

 
An unexpected error has been detected by Java Runtime Environment:
Internal Error (vmem_spaceManager.cpp:309), pid=63145, tid=63236
External Fatal Error: Multimap to aliases failed during multimap_from_space

Insufficient virtual memory areas (VMA) may lead to a crash in Azul Platform Prime.

Solution:

Increase the VMA as described in Ensuring Sufficient Virtual Memory Areas.

The error is generated when the open file limit for a user or system exceeds the default setting.

Solution:

Increase the open files limit on your system.

System-wide settings

User settings

Note: The <num> value varies according to the amount of RAM in the system. It is approximately 100000 files per GB of RAM.