Visit Azul.com Support

Setting Up the Forwarder

Table of Contents
Need help?
Schedule a consultation with an Azul performance expert.
Contact Us

Setting up a Forwarder involves steps both in the Azul Intelligence Cloud Web UI and on a machine within your environment. This step-by-step guide helps you in this process.

Quick Overview

You can configure the Forwarder in different ways as described further on this page. This is a quick overview using a properties file:

  1. Configure your firewall, if needed, to make sure the Forwarder can access the required endpoints.

  2. Configure a new Forwarder in the WebUI.

  3. While still on the WebUI, make sure you download the latest Forwarder application and the forwarder-settings.properties file.

  4. Edit the downloaded properties file and replace the values starting with < and ending with > with a value of your choice, matching your system.

     
    azul.forwarder.keystore.path = <path-to-keystore>/forwarder.jks
azul.forwarder.keystore.password = <desired keystore password>

    Becomes, for example:

     
    azul.forwarder.keystore.path = /opt/forwarder/forwarder.jks
azul.forwarder.keystore.password = MySuperSecretPassword

  5. Start the Forwarder with the following command:

     
    java -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar

  6. Check the state of the Forwarder in the log, via the web interface, or with the API, as described here.

Configuring Your Firewall

The Forwarder must have access to the Azul Intelligence Cloud and AWS S3 services, through the firewall of your system. Two rules are needed with outgoing access:

  • The host address (API URL) on port 443 (HTTPS) of the Azul Intelligence Cloud service that has been set up for you by Azul. You can copy this from the Web UI Forwarder setup screen. For instance: YOUR_INSTANCE.api.ic.azul.com.

  • To be able to upload artifacts, outgoing traffic to *.amazonaws.com must also be allowed.

Configuring a New Forwarder in the Web UI

Administrators must set up a Forwarder in the Azul Intelligence Cloud Web UI to connect it to their account. Forwarders are created on a per-Azul Intelligence Cloud basis and can be managed by administrators.

Visit the Web UI of your Azul Intelligence Cloud instance.

Adding a Forwarder

  1. In the "Settings" section of the Web UI click "Forwarders".

  2. In the "Forwarders list", click the "Create forwarder" button.

  3. Fill in the fields to configure the name and domain names for the new Forwarder.

    • Forwarder name: this is a free value. It is a name to manage your Forwarder more easily, not a domain name. If your organization uses multiple Forwarders, consider a naming convention that signifies usage.

    • Domain Names: a list of domain names used by your JDKs to communicate with this Forwarder. The first entry is your certificate’s Common Name (CN) and subsequent entries are Server Alternative Names (SAN).

  4. Click the "Create" button to store the new Forwarder.

  5. The "Forwarder ID" and "Access key" are shown, together with the pre-populated settings when you click on "Show settings". You can also download them as a ready-to-use properties file with the "Download as file" button.

  6. Edit the downloaded properties file and replace the values starting with < and ending with > with a value of your choice, matching your system.

     
    azul.forwarder.keystore.path = <path-to-keystore>/forwarder.jks
azul.forwarder.keystore.password = <desired keystore password>

    Becomes, for example:

     
    azul.forwarder.keystore.path = /opt/forwarder/forwarder.jks
azul.forwarder.keystore.password = MySuperSecretPassword
Note
 For security reasons, by default the Forwarder is not allowed to run as a server in a public domain to avoid unwanted behavior like sending sensitive JVM data to the public internet and causing incorrect traffic on the domain.

If your environment requires the use of a public domain, you can request Azul support to change the configuration of your Azul Intelligence Cloud environment to add it as an approved domain.

Setting up the Forwarder in Your Environment

The remaining configuration of the Forwarder has to be done within your environment.

Server Requirements

The Forwarder can run on any server within your environment at a location where it can be reached by many JVMs and the Forwarder itself can reach Azul Intelligence Cloud.

  • Java runtime version 8 (minimum 1.8.0_252) or newer

  • CPU: 2 vCPU cores minimum

  • RAM: 8 GB minimum (deduplication uses in-memory caching)

  • Disk: 10 GB local storage (SSD recommended) for the disk-based deduplication cache

  • Capacity: up to 24K JVMs per hour

For deployments where disk storage is not available (e.g., stateless containers), the Forwarder can run with in-memory deduplication only. See Deduplication below.

For memory-constrained environments (4 GB RAM, no disk), deduplication can be disabled entirely, and the forwarder operates as a passthrough proxy with the same requirements as previous versions.

Downloading the Forwarder

You can download the Forwarder application from the Web UI if you have the Administrator role and from the Azul repository:

 
curl -O https://cdn.azul.com/forwarder/2.0.0/forwarder.jar

Configuring and Starting the Forwarder Application

Three different possibilities are supported to configure the Forwarder, and the values defined in each of them have this priority:

  1. Command line option

  2. Properties file

  3. Environment variables

Some settings are the ones filled in or retrieved from the Web UI earlier, others are related to the created certificate. The settings are used by the Forwarder to connect to Azul Intelligence Cloud. Once authenticated, the Forwarder exchanges this information with Azul Intelligence Cloud to receive cryptographic certificates trusted by the JVM.

You must use different startup commands, depending on the method used to specify the settings.

Start Forwarder With Command Line Options

When you are defining the settings for the Forwarder with system or environment variables, you can start the Forwarder with this command:

 
java -DicHost=<your-endpoint>.azul.com \
  -DazulForwarderAccessKey=<forwarder-access-key> \
  -DazulForwarderDomainNames=<domain-names-from-web-ui> \
  -DazulForwarderKeystorePath=<path-to-keystore>/forwarder.jks \
  -DazulForwarderKeystorePassword=<desired keystore password> \
  -jar forwarder.jar

Start Forwarder With Properties File

When you want to use a properties-file to configure the Forwarder, follow these steps.

  1. Create a new file forwarder-settings.properties, or download it from the Web UI > Settings > Forwarders.

     
    touch forwarder-settings.properties

  2. Edit the file and add the following content, specifying all the required settings as defined in the table with configuration options, by replacing the values starting with < and ending with > with a value of your choice.

     
    ic.host = <your-endpoint>.azul.com
azul.forwarder.access.key = <forwarder-access-key>
azul.forwarder.domain.names = <domain-names-from-web-ui>
azul.forwarder.keystore.path = <path-to-keystore>/forwarder.jks
azul.forwarder.keystore.password = <desired keystore password>

  3. Provide the file location with the commandline option -DforwarderSettings when you start the Forwarder. For example:

     
    java -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar

  4. You can use additional command line options, for example, when you need extra logging:

     
    java -DazulLogLevel=debug -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar

Start Forwarder With Environment Variables

Export each of the variables before you start the Forwarder. For example:

 
export IC_HOST=<your-endpoint>.azul.com
export AZUL_FORWARDER_ACCESS_KEY=<forwarder-access-key>
export AZUL_FORWARDER_DOMAIN_NAMES=<domain-names-from-web-ui>
export AZUL_FORWARDER_KEYSTORE_PATH=<path-to-keystore>/forwarder.jks
export AZUL_FORWARDER_KEYSTORE_PASSWORD=<desired keystore password>
java -jar forwarder.jar

All Configuration Options

This is an overview of the available configuration options.

Command Line Option (1)
Properties File (2)
Environment Variable (3)		 Description

(1): icHost
(2): ic.host
(3): IC_HOST

Required.

(1): forwarderSettings
(2): -
(3): -

File path used by the Forwarder to read the settings from a file.

The host of Azul Intelligence Cloud, copied from the Web UI Forwarder setup screen.

(1): azulForwarderAccessKey
(2): azul.forwarder.access.key
(3): AZUL_FORWARDER_ACCESS_KEY

Required.

The Forwarder Access Key copied in the Azul Intelligence Cloud Web UI after adding the Forwarder. This Access Key is used for authentication to Azul Intelligence Cloud.

(1): azulForwarderDomainNames
(2): azul.forwarder.domain.names
(3): AZUL_FORWARDER_DOMAIN_NAMES

Required.

A comma-delimited list with he Forwarder Domain Names (all or subset of them) that were used to create the Forwarder in the Azul Intelligence Cloud Web UI. These Domain Names are used as SN/SAN attributes when creating the certificates for the authorization phase between the Forwarder and Azul Intelligence Cloud.

This list can contain IP addresses, but must contain at least one domain name.

(1): azulForwarderKeystorePath
(2): azul.forwarder.keystore.path
(3): AZUL_FORWARDER_KEYSTORE_PATH

Required.

If no JKS file is found in the specified path, an empty keystore with the same name gets created. This keystore is used to store the auto-generated Forwarder certificates.

(1): azulForwarderKeystorePassword
(2): azul.forwarder.keystore.password
(3): AZUL_FORWARDER_KEYSTORE_PASSWORD

Required.

The password you want to use for the keystore.

(1): azulForwarderPort
(2): azul.forwarder.port
(3): AZUL_FORWARDER_PORT

The listening port used by the Forwarder for incoming connections from VMs. The default value is 443.

(1): azulForwarderStateReportPath
(2): azul.forwarder.state.report.path
(3): AZUL_FORWARDER_STATE_REPORT_PATH

File path used by the Forwarder to periodically store its state.

(1): azulForwarderStateCheckPeriod
(2): azul.forwarder.state.checkPeriod
(3): AZUL_FORWARDER_STATE_CHECK_PERIOD

Human-readable period specifying how often the state of the Forwarder should be checked and stored in the report path. It can be defined in the following formats:

  • {N}d

  • {N}h

  • {N}min

  • {N}sec

  • {N}ms

(1): azulCompressionEnabled
(2): azul.compression.enabled
(3): AZUL_COMPRESSION_ENABLED

Flag to define if the Forwarder must compress the data going to the backend (Intelligence Cloud). The default value is true.

(1): azulForwarderMetricsPort
(2): azul.forwarder.metricsPort
(3): AZUL_FORWARDER_METRICS_PORT

Port to retrieve the Forwarder Prometheus metrics. The default value is 9090.

See Forwarder Metrics for more info.

(1): azulForwarderMetricsPath
(2): azul.forwarder.metricsPath
(3): AZUL_FORWARDER_METRICS_PATH

Path to retrieve the Forwarder Prometheus metrics. The default value is /metrics.

See Forwarder Metrics for more info.

(1): cloudConnectionPoolSize
(2): cloud.connection.poolSize
(3): CLOUD_CONNECTION_POOL_SIZE

Connection pool size used to manage connections between the Forwarder and Azul Intelligence Cloud. The default value is 50.

(1): cloudResponseMaxContentLength
(2): cloud.response.maxContentLength
(3): CLOUD_RESPONSE_MAX_CONTENT_LENGTH

The maximum length of response content, in bytes, that the Intelligence Cloud can send to the virtual machine. The default value is 524288.

(1): http.proxyHost and http.proxyPort
(2): http.proxy.host and http.proxy.port
(3): HTTP_PROXY_HOST and HTTP_PROXY_PORT

Enables HTTP proxy.

(1): https.proxyHost and https.proxyPort
(2): https.proxy.host and https.proxy.port
(3): HTTPS_PROXY_HOST and HTTPS_PROXY_PORT

Enables HTTPS proxy (and overrides HTTP proxy if both are enabled).

(1): proxyKeystorePath and proxyKeystorePassword
(2): proxy.keystore.path and proxy.keystore.password
(3): PROXY_KEYSTORE_PATH and PROXY_KEYSTORE_PASSWORD

Custom keystore (truststore) to allow connection to proxies using custom self-signed certificates.

(1): azulForwarderDeduplicationEnabled
(2): azul.deduplication.enabled
(3): AZUL_FORWARDER_DEDUPLICATION_ENABLED

Enable or disable event deduplication. The default value is true.

See Deduplication for more info.

(1): azulForwarderDeduplicationPipeline
(2): azul.deduplication.pipeline
(3): AZUL_FORWARDER_DEDUPLICATION_PIPELINE

Dedup layers: ibf, lmdb (both) or ibf (in-memory only). The default value is ibf, lmdb.

See Deduplication for more info.

(1): azulForwarderFileCacheDirectory
(2): azul.forwarder.fileCacheDirectory
(3): AZUL_FORWARDER_FILE_CACHE_DIRECTORY

Directory for the disk-based deduplication cache. The default value is the system temp directory.

See Deduplication for more info.

(1): azulLogLevel
(2): azul.log.level
(3): AZUL_LOG_LEVEL

The default log level is INFO.

By default, when errors are caught in the Forwarder, only error messages are logged, without a stack trace. If you need to see the stack trace for debugging or troubleshooting purposes, use the TRACE level.

Possible values (case-insensitive):

  • ALL

  • TRACE

  • DEBUG

  • INFO

  • WARN

  • ERROR

  • OFF
Note
 If you are concerned about the security of the key (FORWARDER_ACCESS_KEY) and/or password (FORWARDER_KEYSTORE_PASSWORD) values, using environment variables is the recommended approach.

Event Deduplication

Starting with Forwarder 2.0.0, the forwarder automatically deduplicates events before forwarding them to Azul Intelligence Cloud. When multiple JVMs load the same classes, JARs, or methods, the forwarder detects and filters duplicate reports, typically reducing network traffic by 85-95%.

Deduplication is enabled by default and requires no changes to your JVM agents or application configuration.

How Deduplication Works

The Forwarder maintains two layers of deduplication cache:

  1. In-memory cache: A fast, lock-free cache that holds recently seen events in the JVM heap. This provides low-latency duplicate detection for active traffic.

  2. Disk-based cache (optional): A local on-disk cache that preserves deduplication state across forwarder restarts. When the forwarder restarts, it immediately resumes filtering duplicates without a warm-up period.

Both caches are self-managing: the in-memory cache automatically evicts entries under memory pressure, and the disk-based cache auto-recovers from disk errors.

Deduplication Configuration

Deduplication works out of the box with sensible defaults. The following settings are available for environments that need adjustments:

Setting (properties file) Description Default

azul.deduplication.enabled

Enable or disable event deduplication. Set to false to run the forwarder as a passthrough proxy (no filtering).

true

azul.deduplication.pipeline

Deduplication layers to use. ibf, lmdb uses both in-memory and disk-based cache (recommended). Set to ibf for in-memory only (no disk required).

ibf, lmdb

azul.forwarder.fileCacheDirectory

Directory for the disk-based deduplication cache. Must be on a local filesystem (not NFS or network storage).

System temp directory

These settings can be specified using any of the three configuration methods (command line, properties file, or environment variables).

  • Command line example:

     
    java -DazulForwarderDeduplicationEnabled=true \
     -DazulForwarderDeduplicationPipeline=ibf,lmdb \
     -jar forwarder.jar

  • Properties file example:

     
    azul.deduplication.enabled = true
azul.deduplication.pipeline = ibf, lmdb

  • Environment variables example:

     
    export AZUL_FORWARDER_DEDUPLICATION_ENABLED=true
export AZUL_FORWARDER_DEDUPLICATION_PIPELINE=ibf,lmdb

Deployment Scenarios

Scenario Configuration Resource Requirements

Standard (recommended)

Default settings

2 CPU, 8 GB RAM, 10 GB disk

In-memory only (no disk available)

azul.deduplication.pipeline = ibf

2 CPU, 4 GB RAM, no disk

Passthrough (no deduplication)

azul.deduplication.enabled = false

1 CPU, 2-4 GB RAM, no disk

JVM Settings

When deduplication is enabled, the forwarder requires additional JVM flags.

  • Required for Java 9+: --add-opens. The disk-based deduplication cache requires access to internal JVM NIO classes. Add the following flags when running on Java 9 or later:

     
    --add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED

    Without these flags, the forwarder will fail with InaccessibleObjectException at startup. You don’t need to add these flags on Java 8, or when deduplication is disabled or set to in-memory only with azul.deduplication.pipeline=ibf.

  • Heap memory (-Xmx): the Forwarder requires more heap memory for the in-memory cache. Use one of the following settings based on your deduplication configuration:

    Deployment Notes -Xmx

    Standard (in-memory + disk-based)

    Accommodates the default in-memory cache size

    6g

    In-memory only

    Smaller cache, no disk-based layer

    3g

    Passthrough (no dedup)

    Minimal memory for proxying

    1g

Full startup example (for Java 9+):

 
java -Xmx6g \
  --add-opens=java.base/java.nio=ALL-UNNAMED \
  --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
  -jar forwarder.jar

Verifying Deduplication

After starting the forwarder with deduplication enabled, verify it is working:

  1. Forwarder log: Look for the startup message:

     
    Deduplication enabled, pipeline: [IBF, LMDB]

  2. Status page: Navigate to https://<FORWARDER_IP:PORT>/. The status page displays:

    • Dedup Pipeline: GREEN, YELLOW, or RED

    • Saved Traffic: cumulative bytes saved by deduplication

    • Dedup Ratio: percentage of duplicate events filtered (typically 85-95%)

  3. Health API: Query the health endpoint:

     
    GET https://<FORWARDER_IP:PORT>/forwarder/health

    The response includes a deduplication section:

     
    {
  "state": "GREEN",
  "version": "2.0.0",
  "uptime": 86400000,
  "connectionState": "UP",
  "deduplication": {
    "state": "GREEN",
    "savedBytes": 45000000000,
    "ratio": 0.92
  }
}

    • deduplication.state: health of the dedup subsystem (GREEN = healthy)

    • deduplication.savedBytes: total bytes saved since startup

    • deduplication.ratio: fraction of events filtered as duplicates (0.92 = 92% of events were duplicates)

Disabling Deduplication

To run the forwarder as a passthrough proxy without deduplication (the same behavior as Forwarder 1.x):

 
azul.deduplication.enabled = false

When disabled, the forwarder does not allocate memory or disk for caching, and all events are forwarded directly to Azul Intelligence Cloud.

Advanced Tuning

The Forwarder includes additional configuration options for fine-tuning cache sizes, memory thresholds, and disk limits. These settings are pre-configured with production-tested defaults and typically do not need adjustment.

If you experience resource issues (high memory usage, disk space warnings, or degraded deduplication effectiveness), contact Azul Support for deployment-specific tuning guidance.

Linux Kernel Setting (Disk-Based Cache)

The disk-based deduplication cache uses memory-mapped files. On Linux, you may need to increase the maximum number of memory map areas if the forwarder fails to start with an error about memory mapping:

 
sysctl -w vm.max_map_count=262144

To make this persistent across reboots, add the following to /etc/sysctl.conf:

 
vm.max_map_count=262144

This is typically required in container environments with restrictive defaults.

Forwarder Metrics

Metrics about the usage of the Forwarder are available in the Prometheus format. By default, this is available on http://{forwarder_url}:9090/metrics, but this can be configured, see All Configuration Options above.

Some example metrics:

 
# HELP azul_forwarder_memory_committed_bytes The maximum amount memory in bytes that is committed
# TYPE azul_forwarder_memory_committed_bytes gauge
azul_forwarder_memory_committed_bytes{application="azul-forwarder",type="heap",} 1.00663296E8
azul_forwarder_memory_committed_bytes{application="azul-forwarder",type="nonHeap",} 3.9059456E7
# HELP azul_forwarder_cloud_response_total Total number of responses from the cloud, tagged by status code
# TYPE azul_forwarder_cloud_response_total counter
azul_forwarder_cloud_response_total{application="azul-forwarder",code="202",} 4.0
azul_forwarder_cloud_response_total{application="azul-forwarder",code="200",} 9.0
# HELP azul_forwarder_runtime Distribution of data uploads (event batches) proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime summary
azul_forwarder_runtime_count{application="azul-forwarder",type="artifacts",} 0.0
azul_forwarder_runtime_sum{application="azul-forwarder",type="artifacts",} 0.0
azul_forwarder_runtime_count{application="azul-forwarder",type="events",} 4.0
azul_forwarder_runtime_sum{application="azul-forwarder",type="events",} 4.0
# HELP azul_forwarder_runtime_max Distribution of data uploads (event batches) proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime_max gauge
azul_forwarder_runtime_max{application="azul-forwarder",type="artifacts",} 0.0
azul_forwarder_runtime_max{application="azul-forwarder",type="events",} 1.0
# HELP azul_forwarder_runtime_active_connections Number of active inbound connections
# TYPE azul_forwarder_runtime_active_connections gauge
azul_forwarder_runtime_active_connections{application="azul-forwarder",} 0.0
# HELP azul_forwarder_memory_max_bytes The maximum amount memory in bytes that can be used
# TYPE azul_forwarder_memory_max_bytes gauge
azul_forwarder_memory_max_bytes{application="azul-forwarder",type="heap",} 8.342470656E9
azul_forwarder_memory_max_bytes{application="azul-forwarder",type="nonHeap",} -1.0
# HELP azul_forwarder_process_cpu_usage CPU usage for Forwarder virtual machine
# TYPE azul_forwarder_process_cpu_usage gauge
azul_forwarder_process_cpu_usage{application="azul-forwarder",} 0.0033245844269466315
# HELP azul_forwarder_system_cpu_usage System CPU usage of the system where Forwarder is running
# TYPE azul_forwarder_system_cpu_usage gauge
azul_forwarder_system_cpu_usage{application="azul-forwarder",} 0.0
# HELP azul_forwarder_memory_used_bytes The amount of used memory in bytes
# TYPE azul_forwarder_memory_used_bytes gauge
azul_forwarder_memory_used_bytes{application="azul-forwarder",type="heap",} 4.120408E7
azul_forwarder_memory_used_bytes{application="azul-forwarder",type="nonHeap",} 3.5866376E7
# HELP azul_forwarder_batch_average_size The average size in bytes of outgoing batch
# TYPE azul_forwarder_batch_average_size gauge
azul_forwarder_batch_average_size{application="azul-forwarder",} 1822.0
# HELP azul_forwarder_runtime_bytes Distribution of data size proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime_bytes summary
azul_forwarder_runtime_bytes_count{application="azul-forwarder",destination="ic",} 5.0
azul_forwarder_runtime_bytes_sum{application="azul-forwarder",destination="ic",} 9111.0
azul_forwarder_runtime_bytes_count{application="azul-forwarder",destination="s3",} 0.0
azul_forwarder_runtime_bytes_sum{application="azul-forwarder",destination="s3",} 0.0
# HELP azul_forwarder_runtime_bytes_max Distribution of data size proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime_bytes_max gauge
azul_forwarder_runtime_bytes_max{application="azul-forwarder",destination="ic",} 6619.0
azul_forwarder_runtime_bytes_max{application="azul-forwarder",destination="s3",} 0.0
# HELP azul_forwarder_auth_result_total Total number of authorization result of Forwarder against the cloud
# TYPE azul_forwarder_auth_result_total counter
azul_forwarder_auth_result_total{application="azul-forwarder",authResult="SUCCESS",} 1.0
# HELP azul_forwarder_connection_state
# TYPE azul_forwarder_connection_state gauge
azul_forwarder_connection_state{application="azul-forwarder",} 1.0
# HELP azul_forwarder_runtime_connections_max Distribution of newly created connections from runtimes
# TYPE azul_forwarder_runtime_connections_max gauge
azul_forwarder_runtime_connections_max{application="azul-forwarder",} 1.0
# HELP azul_forwarder_runtime_connections Distribution of newly created connections from runtimes
# TYPE azul_forwarder_runtime_connections summary
azul_forwarder_runtime_connections_count{application="azul-forwarder",} 2.0
azul_forwarder_runtime_connections_sum{application="azul-forwarder",} 2.0

When deduplication is enabled, the following additional metrics are available at the Prometheus endpoint:

 
# HELP azul_forwarder_dedup_total_events Total events processed through deduplication
# TYPE azul_forwarder_dedup_total_events gauge
azul_forwarder_dedup_total_events 50000000

# HELP azul_forwarder_dedup_total_filtered Total duplicate events filtered
# TYPE azul_forwarder_dedup_total_filtered gauge
azul_forwarder_dedup_total_filtered 45500000

# HELP azul_forwarder_dedup_total_ratio Overall deduplication ratio
# TYPE azul_forwarder_dedup_total_ratio gauge
azul_forwarder_dedup_total_ratio 0.91

# HELP azul_forwarder_dedup_saved_bytes Cumulative bytes saved by deduplication
# TYPE azul_forwarder_dedup_saved_bytes gauge
azul_forwarder_dedup_saved_bytes 12000000000

These metrics can be used with your existing Prometheus/Grafana setup to monitor deduplication effectiveness. The` azul_forwarder_dedup_total_ratio` metric is useful for alerting. A sustained value below 0.5 may indicate that the Forwarder is not receiving enough duplicate traffic to benefit from deduplication, or that cache sizing needs adjustment. Contact Azul Support for assistance with tuning deduplication performance.

Checking the State of the Forwarder

You can check the status of the Forwarder in multiple ways.

Checking the Forwarder Log

The Forwarder log output shows if a connection can be made with Azul Intelligence Cloud.

Successful Start

A successful startup contains these info messages:

 
[INFO] c.a.c.f.ForwarderRunner - New forwarder session started: sessionId=6a964219-f7ff-4ce8-bbb7-26694e82c6b1
[WARN] c.a.c.f.a.ForwarderAuthManager - The JKS keystore does not exist at the specified path, an empty keystore will be created: path=/var/forwarder/forwarder.jks
[INFO] c.a.c.f.a.ForwarderAuthManager - Issue forwarder certificate signed by IC
[INFO] c.a.c.f.a.AuthClient - Forwarder certificate signed by IC issued successfully
[INFO] c.a.c.f.a.ForwarderAuthManager - Certificate chain successfully added to the provided keystore: alias=ic.forwarder, keystore=/var/forwarder/forwarder.jks
[INFO] c.a.c.f.a.ForwarderAuthManager - Schedule the forwarder certificate reissue: reissueDate=Wed Apr 24 13:12:13 CEST 2024, certificateNotAfter=Wed Apr 24 14:12:13 CEST 2024
[INFO] c.a.c.f.Forwarder - Start forwarding *:443 to <your-endpoint>.azul.com:443
Unsuccessful Start

If either the host of Azul Intelligence Cloud, the Forwarder Access Key, or one of the other settings is incorrect, the Forwarder displays an appropriate error message and fail to start. Keep an eye on the output of the Forwarder to check if it could start successfully. These are some possible error messages while connecting:

 
# IC_HOST is incorrect
[ERROR] c.a.c.f.a.ForwarderAuthManager - Failed to verify IC-signed certificate: error=<your-endpoint>.azul.com: nodename nor servname provided, or not known
[ERROR] c.a.c.f.ForwarderRunner - Failed to authorize the forwarder, the application will be stopped

# Authentication problem
[ERROR] c.a.c.f.a.AuthClient - Failed to init certificate challenge: responseCode=403, responseContent={"message":"Forbidden"}
[WARN] c.a.c.f.Forwarder - Please double-check your forwarder settings: IC Host URL, Forwarder Access Key, Forwarder Domain Names
[ERROR] c.a.c.f.ForwarderRunner - Failed to authorize the forwarder, the application will be stopped

Status Screen

After a successful startup, a status page is available at the address https://<FORWARDER_IP:PORT>/.

Status API

The health of the Forwarder can also be checked via an API on https://<FORWARDER_IP:PORT>/forwarder/health, which returns, for example:

 
{
  "state": "GREEN",
  "version": "2.0.0",
  "uptime": 3600000,
  "formattedUptime": "00d 01h 00m 00s",
  "connectionState": "UP",
  "forwarderId": "fwd-abc123",
  "sessionId": "sess-xyz789",
  "deduplication": {
    "state": "GREEN",
    "savedBytes": 12000000000,
    "ratio": 0.91
  }
}

Or more extended data can be retrieved by adding ?extended=true like this https://<FORWARDER_IP:PORT>/forwarder/health?extended=true, resulting in, for example:

 
{
  "state": "GREEN",
  "version": "1.0.9",
  "uptime": "411029",
  "formattedUptime": "00d 00h 06m 51s",
  "memory": {
    "heap.max": "8342470656",
    "heap.used": "60146176",
    "heap.committed": "113246208",
    "heap.init": "524288000",
    "nonHeap.max": "-1",
    "nonHeap.used": "39368184",
    "nonHeap.committed": "42008576",
    "nonHeap.init": "7667712"
  },
  "cpu": {
    "jvmCpuUsage": "0%",
    "systemCpuUsage": "3%"
  },
  "connectionState": "UP",
  "forwarderId": "cc7784b6-cb6a-44a7-a00e-efab7a776f93",
  "sessionId": "0f21b9fa-f52b-43cb-95c4-92a561c87605",
  "deduplication": {
    "state": "GREEN",
    "savedBytes": 12000000000,
    "ratio": 0.91,
    "batches": 5000,
    "events": 50000000
  }
}

The deduplication data contains the following fields:

Field Type Description

state

String

Health of the deduplication pipeline: GREEN, YELLOW, or RED. Contributes to the overall forwarder state.

savedBytes

Long

Cumulative bytes of network traffic saved by filtering duplicate events.

ratio

Double

Fraction of events identified as duplicates (0.0-1.0). A value of 0.9 means 90% of events were filtered.

batches

Long

Total event batches processed through the deduplication pipeline.

events

Long

Total individual events processed.

When deduplication is disabled (azul.deduplication.enabled=false), the deduplication section is omitted from the response.

Note
 Additional internal diagnostic fields (cache load, disk usage, error counts) are available in the extended response for use by Azul Support during troubleshooting.

List of Possible Forwarder States

The state of the Forwarder is shown in both the Status-page and the API as GREEN in normal state or one of the following:

State Metrics Details Threshold

YELLOW

OperatingSystemMXBean#ProcessCpuLoad

"recent cpu usage" for the Java Virtual Machine process

50%

YELLOW

MemoryMXBean#HeapMemoryUsage

[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management]

65%

RED

OperatingSystemMXBean#ProcessCpuLoad

"recent cpu usage" for the Java Virtual Machine process

85%

RED

MemoryMXBean#HeapMemoryUsage

[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management]

90%

YELLOW

Deduplication

In-memory cache under GC pressure

Cache array reclaimed by garbage collector

YELLOW

Deduplication

Disk-based cache errors detected

Intermittent disk read/write errors

RED

Deduplication

Disk-based cache error threshold reached

Consecutive errors exceeded limit; cache auto-recreating