Samples Memory Usage by Account

Need help?

Schedule a consultation with an Azul performance expert.

Examples 6 through 10 in the following subsections depict methods of using the zing-ps command to list information about Azul Zing Builds of OpenJDK (Zing) processes.

Example 1: Monitoring a Test Run.

If you run a while loop, logging it into a file every 10 seconds or so, you can then make graphs from the log data, using your favorite raw-number-to-nice-graphs conversion tool.

 # while true; do zing-ps -s >> /tmp/zing-ps_log ; sleep 10; done

Example 2: Diagnosis of Zing Fault

In the rare occurrence of a Zing crash, it can be useful to collect the output of zing-ps –s and zing-ps --acctinternal (debug option) from the system after the crash occurs, but before a system reboot or a ZST restart. If Zing crashed, please contact your Azul representative who might direct you to collect the zing-ps output and upload to the Azul FTP site along with other data, logs and cores.

Example 3: Memory statistics with zero Zing processes running.

This example shows memory funding with no Zing processes running.

 # zing-ps --acct
Fund breakdown (total 8)
                              NAME         BALANCE         MAXIMUM
       fund[0]:          Committed     22745710592     22745710592
       fund[1]:          Overdraft      1262485504      1262485504
       fund[2]:            Unknown               0               0
       fund[3]:    PausePrevention      1262485504      1262485504
       fund[4]:            Unknown               0               0
       fund[5]:            Unknown               0               0
       fund[6]:            Unknown               0               0
       fund[7]:            Unknown               0               0
Found 0 processes

Example 4: Memory account details with 1 Zing process running.

 $ zing-ps --acct2
Process level Maximum: 0
NAME                 BALANCE(now/min/max KB)  ALLOCATED(now/min/max KB)     MAXIMUM(KB) ACS_TO_CREDIT       FND  ODFND
AC[0]:Default        32768/ 30720/ 32768      2048/ 0/ 4096
32768       0xffffffffffffff00    0     1
AC[1]:Unknown         0/ 0/ 0                  0/ 0/ 0
0           0xffffffffffffff01   -1    -1
AC[2]:JavaHeap        11360/ 911360/ 1873920  2201600/ 1239040/ 2201600         8067072     0xffffffffffff0203    0      1
AC[3]:PausePrevention 0/ 0/ 0                 0/ 0/ 0
4954112     0xffffffffffff0203    3      3

Field descriptions:

Values are KB bytes.
account [0]: The maximum is equal to the Partition. The allocated plus the balance are equal to maximum.
account [2]: Java Heap. If the balance is negative, the application is using contingency memory.
account [3]: Maximum value allocated for Pause Prevention.

The value for account[2] heap ALLOCATED (used) is higher than AC[2] javaHeap ALLOCATED now (used) because the zing-ps --acct (in these examples) was followed by the zing-ps --acct2 and the application continued to allocate objects into the Java heap.

Example 5: Memory funding details with 1 Zing process running.

This example shows memory funding breakdown for ZST and allocation to the Zing process PID 15479.

 $ zing-ps --acct
Fund breakdown (total 8)
NAME BALANCEMAXIMUM
fund[0]:Committed 819357286442553311232
fund[1]:Overdraft 23634903042363490304
fund[2]:Unknown 00
fund[3]:PausePrevention23634903042363490304
fund[4]:Unknown 00
fund[5]:Unknown 00
fund[6]:Unknown 00
fund[7]:Unknown 00

Found 1 processes.

USER PID PROCESS
root 15479 /opt/zing/zing-jdk1.7.0_23-4.2.0.0.beta-14-x86_64/bin/java
NAMEBALANCEALLOCATEDMAXIMUMFNDODFND
account[0]:default341835776209715234393292801
account[1]:emergency_gc000-1-1
account[2]:heap3122659328027892121603637929574401
account[3]:pause_prevention0027487790694433
account[4]:unknown000-1-1
account[5]:unknown000-1-1
account[6]:unknown000-1-1
account[7]:unknown000-1-1

Explanation of zing-ps Fund Breakdown

The Fund breakdown in Memory funding details with 1 Zing process running, shows the memory funding as set up for the ZST. The output from the command then also shows that the zing-ps acct command has found one Zing Java process PID 15749 and gives the memory breakdown for that Java process.

The table below takes a look at the example Fund Breakdown in more detail. The figures have been converted from bytes to megabytes (because it is easier to think about megabytes than large numbers of bytes).

NAME BALANCE MAXIMUM

	NAME	BALANCE	MAXIMUM
fund[0]:	Committed	7814	40582
The Committed Maximum is the maximum Reservable Heap The Committed Balance is the maximum Available Heap size for the JVM `-Xmx` command option; and if other JVMs have already been started this will be less than the maximum Reservable value. Java Process ID 15479 has been started and set a `–Xmx` of 32768MB reducing the Available heap from 40582MB to 7814MB
fund[1]:	Overdraft	2254	2254
The Overdraft is the Contingency Memory (using the ZST defaults, this is 5% of the Zing Memory)
fund[3]:	PausePrevention	2254	2254
The PausePrevention is the Pause prevention memory (using the ZST defaults, this is 5% of the Zing Memory)

fund[0]:

Committed

7814

40582

The Committed Maximum is the maximum Reservable Heap

The Committed Balance is the maximum Available Heap size for the JVM -Xmx command option; and if other JVMs have already been started this will be less than the maximum Reservable value.

Java Process ID 15479 has been started and set a –Xmx of 32768MB reducing the Available heap from 40582MB to 7814MB

fund[1]:

Overdraft

2254

The Overdraft is the Contingency Memory (using the ZST defaults, this is 5% of the Zing Memory)

fund[3]:

PausePrevention

2254

The PausePrevention is the Pause prevention memory (using the ZST defaults, this is 5% of the Zing Memory)

The sum of the Committed Maximum and the Overdraft and Pause prevention maximums is equal to the size of the Zing Memory (using the ZST defaults this is 75% of the available Operating System memory, leaving 25% for the Linux Memory). In this case 40852 + 2254 + 2254 = 45090MB from which we can infer that if the ZST was configured with a 75/25 allocation then the available Operating System memory at the time the ZST was configured was 60120MB (58.7GB). This can be useful to know; for example, if we made another assumption that this was a 64GB RAM machine, then 5416MB has been allocated away somewhere else before the Operating System was loaded – perhaps some DMA buffers for 10GBe NIC interfaces or some such.

Example Memory Fund Allocation for a Process

Carrying on with Memory funding details with 1 Zing process running, using Process 15479 for illustration.

The table below takes a look at the example Fund allocation in more detail. The figures have been converted from bytes to megabytes (because it is easier to think about megabytes than large numbers of bytes).

PID: 15479 /opt/zing/zing-jdk1.7.0_23-4.2.0.0.beta-14-x86_64/bin/java

	NAME	BALANCE	ALLOCATED	MAXIMUM	FND	ODFND
Account 0	default	326	2	328	0	1
Account 0	The Account 0 value is in bytes, and it amounts to 328MB. Its max is equal to its funding (that is, the ALLOCATED plus the BALANCE are equal to max). Account 0 is used for some ZST-based memory stuff that is not purely in the heap (such as kernel structures for holding the kernel page tables and such). The JVM assigns 1% of `-Xmx` to Account 0, and the rest to Account 2 (both draw from Committed memory).
Account 1	emergency_gc	0	0	0	-1	-1
Account 1
Account 2	heap	29780	2660	34694	0	1
Account 2	Account 2 is the Java Heap. If its BALANCE shows as negative, your application is using contingency memory. The Account 2 funding (BALANCE + ALLOCATED) together with Account 0 funding adds up exactly to the `-Xmx` value. In this case `-Xmx ⇒ 29780 + 2660 + 328 = 32768MB` The option flag `-XX:JavaMemMax` can be used to set the maximum for Account 2 (heap). NOTE: When `-XX:JavaMemMax=<val>` equals `-Xmx<val>`, then the Zing instance is not allowed to use any Contingency memory.
Account 3	pause_prevention	0	0	262144	3	3
Account 3	The Account 3 MAXIMUM value for the pause_prevention memory computes to 256GB (exactly). Having a large MAXIMUM value doesn’t mean you have funding for that much. It just means you can draw up to that much total allocation (including dipping into overdraft accounts) if the ZST has resources to give you in the fund you are drawing from. In the case of pause prevention, a max of 256GB just means that this process can draw down the entire pause prevention memory if it wants to (it has zero funding for account 3, so all of that draw shows up in negative balances equal to the allocation level).

The MAXIMUM values for each account are not funding numbers. They are just absolute limits on how much can be allocated in that account. This matters a lot when overdraft accounts are involved, as the MAXIMUM caps the total amount of allocation, even after the balance goes negative.

The committed funding is always the sum of BALANCE + ALLOCATED, this remains true including times when Zing dips into an overdraft account, such as contingency (the BALANCE will go negative at that point, and even though ALLOCATED is greater than committed funding, the sum is still equal to committed).

Zing uses Zing Memory only for the -Xmx (Java heap). So you need to leave enough Linux memory to accommodate not only the other Linux stuff that’s there, but also Zing’s C-heap.