Visit Azul.com Support

Native Memory Tracking

Table of Contents
Need help?
Schedule a consultation with an Azul performance expert.
Contact Us
Talk about OpenJDK?
Discuss with the community on the Foojay Forum.
Go to Forum
Looking for Zing?
The Azul Zing Virtual Machine is now Azul Zulu Prime Builds of OpenJDK and part of Azul Platform Prime.
Learn more

Native Memory Tracking (NMT) allows you to investigate internal memory usage of Java applications. Use NMT to monitor and diagnose memory-related issues within the JVM as it helps you to understand how the JVM allocates memory for various internal structures and other areas.

Enable Native Memory Tracking

Command-line Options

To enable Native Memory Tracking, use the command-line option -XX:NativeMemoryTracking with summary or detail:

 
java -XX:NativeMemoryTracking=summary

To print out the statistics to stdout at the end of the run, add -XX:+PrintNMTStatistics:

 
java -XX:+PrintNMTStatistics -XX:NativeMemoryTracking=summary

Extended Mode

By enabling NMT, most of the allocations inside libjvm.so are covered. As of version 23.02.0.0 of Azul Prime Builds of OpenJDK, extended NMT is available. Compared to OpenJDK that only provides HotSpot allocation coverage (inside libjvm.so), Prime provides full coverage of the malloc allocations in the process.

We strongly encourage using this extended mode where possible. The performance impact is negligible, especially in summary mode. NMT has no impact on the Falcon compile time.

To enable extended NMT, export LD_PRELOAD in addition to the regular NMT flags as shown above.

 
export LD_PRELOAD=$JAVA_HOME/etc/zing/lib/libnmt_hooks.so
Note
 The output formats and command line flags are identical for normal and extended NMT.

When the LD_PRELOAD variable is exported, all Java processes run with extended NMT. As jcmd is also a Java process, jcmd gives the following output:

 
Detected libnmt_hooks.so in LD_PRELOAD. Adding implicit flag
  -XX:NativeMemoryTracking=summary for consistency. To avoid
  this warning enable NMT explicitly or remove libnmt_hooks.so
  from LD_PRELOAD.

This means jcmd itself is running with NMT-summary and this can be misleading.

Note
 Only applications should be launched with LD_PRELOAD, not jcmd.

Use the following command to remove the exported 'LD_PRELOAD`:

 
unset LD_PRELOAD

Native Memory Tracking Pre-Header

For tracking purposes, NMT puts 16 extra bytes before and 2 after every allocation. This is used for metadata and buffer overrun/underrun checks.

 
+-----------+---------....---------+--------+
|  header   |         user         |   can  |
|           |      allocation      |   ary  |
+-----------+---------....---------+--------+
| 16 bytes  |       user size      | 2 byte |

Results of Native Memory Tracking

The NMT summary report appears in several places:

Summary Information Provided by Native Memory Tracking

The summary level of tracking provides basic information about the following native memory allocations:

Category Amount of the native memory allocated for

Java Heap

Java Heap

The committed portion displays the Xmx value.

Class

Application classes

Thread

Application threads

Code

Code being executed

GC

Garbage Collector

Compiler

Compiler operations

Compiler Runtime

Code profiles

Internal

Internal allocations

Other

Unsafe API without LD_PRELOAD

It also contains all allocations from the process intercepted by the LD_PRELOADE library (except Falcon allocations going to Compiler).

Symbol

Table of symbols

Native Memory Tracking

Native Memory Tracking overhead

Arena Chunk

Arena-managed chunk

Tracing

Profilers (like TickProfiler and JFR)

Logging

Unified and other logging

Arguments

Processing arguments

Module

Managing Java modules

Synchronization

Synchronization primitives inside VM (like semaphores, monitors, mutexes)

Unknown

Unspecified allocations
Note
 If a given category has no allocations during the run, it is not listed in the report.

Example Summary Report

Note
 When the LD_PRELOAD variable is exported, more info is provided in the Compiler and Other category, resulting in bigger numbers. 
 
Native Memory Tracking:

Total: reserved=7635375131517, committed=2925190013
    malloc: 699526013 #11264
    mmap:  reserved=7634675605504, committed=2225664000

-         Java Heap (reserved=2199023255552, committed=2147483648)
              (mmap: reserved=2199023255552, committed=2147483648)

-           Class (reserved=320262745, committed=320262745)
              (classes #458)
              ( instance classes #393, array classes #65)
              (malloc=320262745 #27)

-          Thread (reserved=207835992, committed=4649816)
              (thread #98)
              (stack: reserved=207622144, committed=4435968)
              (malloc=44600 #497)
              (arena=169248 #172)

-           Code (reserved=51113238, committed=51113238)
              (malloc=4975894 #1178)
              (mmap: reserved=46137344, committed=46137344)

-            GC (reserved=5418130183790, committed=29210222)
              (malloc=28939886 #582)
              (mmap: reserved=5418101243904, committed=270336)

-                  Compiler (reserved=14765901, committed=14765901)
                            (malloc=14477349 #395)
                            (arena=288552 #27)

-          Compiler Runtime (reserved=89196, committed=89196)
                            (malloc=89196 #72)

-                  Internal (reserved=17305170130, committed=35160274)
                            (malloc=7823570 #5286)
                            (mmap: reserved=17297346560, committed=27336704)

-                     Other (reserved=378983, committed=378983)
                            (malloc=378983 #1337)

-                    Symbol (reserved=320939304, committed=320939304)
                            (malloc=320939304 #45)

-    Native Memory Tracking (reserved=202216, committed=202216)
                            (malloc=21976 #171)
                            (tracking overhead=180240)

-               Arena Chunk (reserved=794136, committed=794136)
                            (malloc=794136)

-                   Tracing (reserved=2408, committed=2408)
                            (malloc=2408 #7)

-                   Logging (reserved=6788, committed=6788)
                            (malloc=6788 #214)

-                 Arguments (reserved=6930, committed=6930)
                            (malloc=6930 #125)

-                    Module (reserved=25424, committed=25424)
                            (malloc=25424 #3)

-           Synchronization (reserved=98784, committed=98784)
                            (malloc=98784 #1108)

When the LD_PRELOAD variable is not exported, this report starts with the following message:

 
Info: To make this report more precise add
  "export LD_PRELOAD=$JAVA_HOME/etc/zing/lib/libnmt_hooks.so"
  to the environment of Azul Prime JVM.

Detail Information Provided by Native Memory Tracking

The detail level of tracking provides the same output as the summary level, but adds a per-call site report with stack traces and info about mmaps (reserved regions and committed memory inside them).

Example Detail Report

 
Native Memory Tracking:

Total: reserved=7635374862574, committed=2924921070
       malloc: 699257070 #11168
       mmap:   reserved=7634675605504, committed=2225664000

-                 Java Heap (reserved=2199023255552, committed=2147483648)
                            (mmap: reserved=2199023255552, committed=2147483648)

# ... other categories identical to summary report

Virtual memory map:

[0x0000691d00000000 - 0x00006e0000000000] reserved 5373004087296 for GC from
[0x00007ff717f4b053] GPGC_Heap::setup_memory_for_java_heap_structures()+0x53
[0x00007ff717f4f63f] GPGC_Heap::initialize()+0x7bf
[0x00007ff7186dd57a] Universe::initialize_heap()+0x3a
[0x00007ff7186dd702] universe_init()+0x102

[0x00007ff49717b000 - 0x00007ff49737b000] reserved 2097152 for Thread Stack from
[0x00007ff718a712e5] JavaThread::run()+0x25
[0x00007ff718c69300] _start_thread+0x100
[0x00007ff71b236cd6] preRun+0x4f
[0x00007ff71ac36ea5] start_thread+0xc5

	[0x00007ff49717b000 - 0x00007ff497194000] committed 102400 from
            [0x00007ff718c7d01c] os::create_stack_guard_pages(char*, unsigned long)+0x5c
            [0x00007ff718a51991] Thread::create_stack_guard_pages()+0x91
            [0x00007ff718a712fe] JavaThread::run()+0x3e
            [0x00007ff718c69300] _start_thread+0x100

	[0x00007ff497379000 - 0x00007ff49737b000] committed 8192

[0x00007ff717dc4e86] debuginfo::builder::ScopeBuilder::make_compressed(CommonAsm*, GrowableArray<long>*)+0x3c6
[0x00007ff717dc9f5a] debuginfo::builder::DebugMapBuilder::build_debugMap(CommonAsm*)+0xf5a
[0x00007ff717dca3c6] CommonAsm::bake_into_codeOop(methodHandle, int, CodeProfile const*, bool, osr_bci_list&&, Thread*)+0x206
[0x00007ff718248927] ciEnv::register_method(ciMethod*, MacroAssembler*, CodeProfile**, ByteSize, Label&, bool, GrowableArray<int> const&)+0x267
                             (malloc=10440 type=Code #145)

[0x00007ff718b6d9b0] BasicHashtable<(MEMFLAGS)1>::new_entry(unsigned int)+0x70
[0x00007ff718b6e111] Hashtable<klassOopDesc*, (MEMFLAGS)1>::new_entry(unsigned int, klassOopDesc*)+0x11
[0x00007ff7182fde23] Dictionary::new_entry(unsigned int, klassOopDesc*, oopDesc*)+0x23
[0x00007ff7182fe2ea] Dictionary::add_klass(symbolHandle, Handle, KlassHandle)+0x12a
                             (malloc=44000 type=Class #1)

[0x00007ff717f24551] GPGC_GCManagerNewStrong::initialize()+0x351
[0x00007ff717f48c05] GPGC_Heap::post_initialize()+0x45
[0x00007ff7186e1d15] universe_post_init()+0x1c15
[0x00007ff7189a2a15] init_globals()+0xd5
                             (malloc=6912 type=GC #18)
...

Analyzing in GCLogAnalyzer

Use the GCLogAnalyzer tool to visualize and plot the NMT results for both normal and extended mode.

gcla nmt