File path used by the Forwarder to read the settings from a file.
The host of Azul Intelligence Cloud, copied from the Web UI Forwarder setup screen.
Setting up a Forwarder involves steps both in the Azul Intelligence Cloud Web UI and on a machine within your environment. This step-by-step guide helps you in this process.
You can configure the Forwarder in different ways as described further on this page. This is a quick overview using a properties file:
Configure your firewall, if needed, to make sure the Forwarder can access the required endpoints.
While still on the WebUI, make sure you download the latest Forwarder application and the forwarder-settings.properties file.
Edit the downloaded properties file and replace the values starting with < and ending with > with a value of your choice, matching your system.
azul.forwarder.keystore.path = <path-to-keystore>/forwarder.jks
azul.forwarder.keystore.password = <desired keystore password>
Becomes, for example:
azul.forwarder.keystore.path = /opt/forwarder/forwarder.jks
azul.forwarder.keystore.password = MySuperSecretPassword
Start the Forwarder with the following command:
java -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar
Check the state of the Forwarder in the log, via the web interface, or with the API, as described here.
The Forwarder must have access to the Azul Intelligence Cloud and AWS S3 services, through the firewall of your system. Two rules are needed with outgoing access:
The host address (API URL) on port 443 (HTTPS) of the Azul Intelligence Cloud service that has been set up for you by Azul. You can copy this from the Web UI Forwarder setup screen. For instance: YOUR_INSTANCE.api.ic.azul.com.
To be able to upload artifacts, outgoing traffic to *.amazonaws.com must also be allowed.
Administrators must set up a Forwarder in the Azul Intelligence Cloud Web UI to connect it to their account. Forwarders are created on a per-Azul Intelligence Cloud basis and can be managed by administrators.
Visit the Web UI of your Azul Intelligence Cloud instance.
In the "Settings" section of the Web UI click "Forwarders".

In the "Forwarders list", click the "Create forwarder" button.
Fill in the fields to configure the name and domain names for the new Forwarder.

Forwarder name: this is a free value. It is a name to manage your Forwarder more easily, not a domain name. If your organization uses multiple Forwarders, consider a naming convention that signifies usage.
Domain Names: a list of domain names used by your JDKs to communicate with this Forwarder. The first entry is your certificate’s Common Name (CN) and subsequent entries are Server Alternative Names (SAN).
Click the "Create" button to store the new Forwarder.
The "Forwarder ID" and "Access key" are shown, together with the pre-populated settings when you click on "Show settings". You can also download them as a ready-to-use properties file with the "Download as file" button.

Edit the downloaded properties file and replace the values starting with < and ending with > with a value of your choice, matching your system.
azul.forwarder.keystore.path = <path-to-keystore>/forwarder.jks
azul.forwarder.keystore.password = <desired keystore password>
Becomes, for example:
azul.forwarder.keystore.path = /opt/forwarder/forwarder.jks
azul.forwarder.keystore.password = MySuperSecretPassword
|
Note
|
For security reasons, by default the Forwarder is not allowed to run as a server in a public domain to avoid unwanted behavior like sending sensitive JVM data to the public internet and causing incorrect traffic on the domain. If your environment requires the use of a public domain, you can request Azul support to change the configuration of your Azul Intelligence Cloud environment to add it as an approved domain. |
The remaining configuration of the Forwarder has to be done within your environment.
The Forwarder can run on any server within your environment at a location where it can be reached by many JVMs and the Forwarder itself can reach Azul Intelligence Cloud.
Java runtime, version 8 (minimally 1.8.0_252) or newer is required.
A single Forwarder instance is designed to process data from up to 24K JVMs per hour and requires 1 to 2 vCPU cores and at least 4Gb of RAM.
The Forwarder application can be downloaded from the Web UI and from the Azul repository:
curl -O https://cdn.azul.com/forwarder/1.0.9/forwarder.jar
Three different possibilities are supported to configure the Forwarder, and the values defined in each of them have this priority:
Command line option
Properties file
Environment variables
Some settings are the ones filled in or retrieved from the Web UI earlier, others are related to the created certificate. The settings are used by the Forwarder to connect to Azul Intelligence Cloud. Once authenticated, the Forwarder exchanges this information with Azul Intelligence Cloud to receive cryptographic certificates trusted by the JVM.
You must use different startup commands, depending on the method used to specify the settings.
When you are defining the settings for the Forwarder with system or environment variables, you can start the Forwarder with this command:
java -DicHost=<your-endpoint>.azul.com \
-DazulForwarderAccessKey=<forwarder-access-key> \
-DazulForwarderDomainNames=<domain-names-from-web-ui> \
-DazulForwarderKeystorePath=<path-to-keystore>/forwarder.jks \
-DazulForwarderKeystorePassword=<desired keystore password> \
-jar forwarder.jar
When you want to use a properties-file to configure the Forwarder, follow these steps.
Create a new file forwarder-settings.properties, or download it from the Web UI > Settings > Forwarders.
touch forwarder-settings.properties
Edit the file and add the following content, specifying all the required settings as defined in the table with configuration options, by replacing the values starting with < and ending with > with a value of your choice.
ic.host = <your-endpoint>.azul.com
azul.forwarder.access.key = <forwarder-access-key>
azul.forwarder.domain.names = <domain-names-from-web-ui>
azul.forwarder.keystore.path = <path-to-keystore>/forwarder.jks
azul.forwarder.keystore.password = <desired keystore password>
Provide the file location with the commandline option -DforwarderSettings when you start the Forwarder. For example:
java -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar
You can use additional command line options, for example, when you need extra logging:
java -DlogLevel=debug -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar
Export each of the variables before you start the Forwarder. For example:
export IC_HOST=<your-endpoint>.azul.com
export AZUL_FORWARDER_ACCESS_KEY=<forwarder-access-key>
export AZUL_FORWARDER_DOMAIN_NAMES=<domain-names-from-web-ui>
export AZUL_FORWARDER_KEYSTORE_PATH=<path-to-keystore>/forwarder.jks
export AZUL_FORWARDER_KEYSTORE_PASSWORD=<desired keystore password>
java -jar forwarder.jar
This is an overview of the available configuration options.
| Command Line Option (1) Properties File (2) Environment Variable (3) |
Description |
|---|---|
(1): icHost |
Required. |
(1): forwarderSettings |
File path used by the Forwarder to read the settings from a file. The host of Azul Intelligence Cloud, copied from the Web UI Forwarder setup screen. |
(1): azulForwarderAccessKey |
Required. The Forwarder Access Key copied in the Azul Intelligence Cloud Web UI after adding the Forwarder. This Access Key is used for authentication to Azul Intelligence Cloud. |
(1): azulForwarderDomainNames |
Required. A comma-delimited list with he Forwarder Domain Names (all or subset of them) that were used to create the Forwarder in the Azul Intelligence Cloud Web UI. These Domain Names are used as SN/SAN attributes when creating the certificates for the authorization phase between the Forwarder and Azul Intelligence Cloud. This list can contain IP addresses, but must contain at least one domain name. |
(1): azulForwarderKeystorePath |
Required. If no JKS file is found in the specified path, an empty keystore with the same name gets created. This keystore is used to store the auto-generated Forwarder certificates. |
(1): azulForwarderKeystorePassword |
Required. The password you want to use for the keystore. |
(1): azulForwarderPort |
The listening port used by the Forwarder for incoming connections from VMs. The default value is |
(1): azulForwarderStateReportPath |
File path used by the Forwarder to periodically store its state. |
(1): azulForwarderStateCheckPeriod |
Human-readable period specifying how often the state of the Forwarder should be checked and stored in the report path. It can be defined in the following formats:
|
(1): azulCompressionEnabled |
Flag to define if the Forwarder must compress the data going to the backend (Intelligence Cloud). The default value is |
(1): azulForwarderMetricsPort |
Port to retrieve the Forwarder Prometheus metrics. The default value is See Forwarder Metrics for more info. |
(1): azulForwarderMetricsPath |
Path to retrieve the Forwarder Prometheus metrics. The default value is See Forwarder Metrics for more info. |
(1): cloudConnectionPoolSize |
Connection pool size used to manage connections between the Forwarder and Azul Intelligence Cloud. The default value is |
(1): cloudResponseMaxContentLength |
The maximum length of response content, in bytes, that the Intelligence Cloud can send to the virtual machine. The default value is |
(1): http.proxyHost and http.proxyPort |
Enables HTTP proxy. |
(1): https.proxyHost and https.proxyPort |
Enables HTTPS proxy (and overrides HTTP proxy if both are enabled). |
(1): proxyKeystorePath and proxyKeystorePassword |
Custom keystore (truststore) to allow connection to proxies using custom self-signed certificates. |
(1): logLevel |
The default log level is By default, when errors are caught in the Forwarder, only error messages are logged, without a stack trace. If you need to see the stack trace for debugging or troubleshooting purposes, use the Possible values (case-insensitive):
|
|
Note
|
If you are concerned about the security of the key (FORWARDER_ACCESS_KEY) and/or password (FORWARDER_KEYSTORE_PASSWORD) values, using environment variables is the recommended approach. |
Metrics about the usage of the Forwarder are available in the Prometheus format. By default, this is available on http://{forwarder_url}:9090/metrics, but this can be configured, see All Configuration Options above.
Some example metrics:
# HELP azul_forwarder_memory_committed_bytes The maximum amount memory in bytes that is committed
# TYPE azul_forwarder_memory_committed_bytes gauge
azul_forwarder_memory_committed_bytes{application="azul-forwarder",type="heap",} 1.00663296E8
azul_forwarder_memory_committed_bytes{application="azul-forwarder",type="nonHeap",} 3.9059456E7
# HELP azul_forwarder_cloud_response_total Total number of responses from the cloud, tagged by status code
# TYPE azul_forwarder_cloud_response_total counter
azul_forwarder_cloud_response_total{application="azul-forwarder",code="202",} 4.0
azul_forwarder_cloud_response_total{application="azul-forwarder",code="200",} 9.0
# HELP azul_forwarder_runtime Distribution of data uploads (event batches) proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime summary
azul_forwarder_runtime_count{application="azul-forwarder",type="artifacts",} 0.0
azul_forwarder_runtime_sum{application="azul-forwarder",type="artifacts",} 0.0
azul_forwarder_runtime_count{application="azul-forwarder",type="events",} 4.0
azul_forwarder_runtime_sum{application="azul-forwarder",type="events",} 4.0
# HELP azul_forwarder_runtime_max Distribution of data uploads (event batches) proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime_max gauge
azul_forwarder_runtime_max{application="azul-forwarder",type="artifacts",} 0.0
azul_forwarder_runtime_max{application="azul-forwarder",type="events",} 1.0
# HELP azul_forwarder_runtime_active_connections Number of active inbound connections
# TYPE azul_forwarder_runtime_active_connections gauge
azul_forwarder_runtime_active_connections{application="azul-forwarder",} 0.0
# HELP azul_forwarder_memory_max_bytes The maximum amount memory in bytes that can be used
# TYPE azul_forwarder_memory_max_bytes gauge
azul_forwarder_memory_max_bytes{application="azul-forwarder",type="heap",} 8.342470656E9
azul_forwarder_memory_max_bytes{application="azul-forwarder",type="nonHeap",} -1.0
# HELP azul_forwarder_process_cpu_usage CPU usage for Forwarder virtual machine
# TYPE azul_forwarder_process_cpu_usage gauge
azul_forwarder_process_cpu_usage{application="azul-forwarder",} 0.0033245844269466315
# HELP azul_forwarder_system_cpu_usage System CPU usage of the system where Forwarder is running
# TYPE azul_forwarder_system_cpu_usage gauge
azul_forwarder_system_cpu_usage{application="azul-forwarder",} 0.0
# HELP azul_forwarder_memory_used_bytes The amount of used memory in bytes
# TYPE azul_forwarder_memory_used_bytes gauge
azul_forwarder_memory_used_bytes{application="azul-forwarder",type="heap",} 4.120408E7
azul_forwarder_memory_used_bytes{application="azul-forwarder",type="nonHeap",} 3.5866376E7
# HELP azul_forwarder_batch_average_size The average size in bytes of outgoing batch
# TYPE azul_forwarder_batch_average_size gauge
azul_forwarder_batch_average_size{application="azul-forwarder",} 1822.0
# HELP azul_forwarder_runtime_bytes Distribution of data size proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime_bytes summary
azul_forwarder_runtime_bytes_count{application="azul-forwarder",destination="ic",} 5.0
azul_forwarder_runtime_bytes_sum{application="azul-forwarder",destination="ic",} 9111.0
azul_forwarder_runtime_bytes_count{application="azul-forwarder",destination="s3",} 0.0
azul_forwarder_runtime_bytes_sum{application="azul-forwarder",destination="s3",} 0.0
# HELP azul_forwarder_runtime_bytes_max Distribution of data size proxied from a runtime to the ic cloud.
# TYPE azul_forwarder_runtime_bytes_max gauge
azul_forwarder_runtime_bytes_max{application="azul-forwarder",destination="ic",} 6619.0
azul_forwarder_runtime_bytes_max{application="azul-forwarder",destination="s3",} 0.0
# HELP azul_forwarder_auth_result_total Total number of authorization result of Forwarder against the cloud
# TYPE azul_forwarder_auth_result_total counter
azul_forwarder_auth_result_total{application="azul-forwarder",authResult="SUCCESS",} 1.0
# HELP azul_forwarder_connection_state
# TYPE azul_forwarder_connection_state gauge
azul_forwarder_connection_state{application="azul-forwarder",} 1.0
# HELP azul_forwarder_runtime_connections_max Distribution of newly created connections from runtimes
# TYPE azul_forwarder_runtime_connections_max gauge
azul_forwarder_runtime_connections_max{application="azul-forwarder",} 1.0
# HELP azul_forwarder_runtime_connections Distribution of newly created connections from runtimes
# TYPE azul_forwarder_runtime_connections summary
azul_forwarder_runtime_connections_count{application="azul-forwarder",} 2.0
azul_forwarder_runtime_connections_sum{application="azul-forwarder",} 2.0
You can check the status of the Forwarder in multiple ways.
The Forwarder log output shows if a connection can be made with Azul Intelligence Cloud.
A successful startup contains these info messages:
[INFO] c.a.c.f.ForwarderRunner - New forwarder session started: sessionId=6a964219-f7ff-4ce8-bbb7-26694e82c6b1
[WARN] c.a.c.f.a.ForwarderAuthManager - The JKS keystore does not exist at the specified path, an empty keystore will be created: path=/var/forwarder/forwarder.jks
[INFO] c.a.c.f.a.ForwarderAuthManager - Issue forwarder certificate signed by IC
[INFO] c.a.c.f.a.AuthClient - Forwarder certificate signed by IC issued successfully
[INFO] c.a.c.f.a.ForwarderAuthManager - Certificate chain successfully added to the provided keystore: alias=ic.forwarder, keystore=/var/forwarder/forwarder.jks
[INFO] c.a.c.f.a.ForwarderAuthManager - Schedule the forwarder certificate reissue: reissueDate=Wed Apr 24 13:12:13 CEST 2024, certificateNotAfter=Wed Apr 24 14:12:13 CEST 2024
[INFO] c.a.c.f.Forwarder - Start forwarding *:443 to <your-endpoint>.azul.com:443
If either the host of Azul Intelligence Cloud, the Forwarder Access Key, or one of the other settings is incorrect, the Forwarder displays an appropriate error message and fail to start. Keep an eye on the output of the Forwarder to check if it could start successfully. These are some possible error messages while connecting:
# IC_HOST is incorrect
[ERROR] c.a.c.f.a.ForwarderAuthManager - Failed to verify IC-signed certificate: error=<your-endpoint>.azul.com: nodename nor servname provided, or not known
[ERROR] c.a.c.f.ForwarderRunner - Failed to authorize the forwarder, the application will be stopped
# Authentication problem
[ERROR] c.a.c.f.a.AuthClient - Failed to init certificate challenge: responseCode=403, responseContent={"message":"Forbidden"}
[WARN] c.a.c.f.Forwarder - Please double-check your forwarder settings: IC Host URL, Forwarder Access Key, Forwarder Domain Names
[ERROR] c.a.c.f.ForwarderRunner - Failed to authorize the forwarder, the application will be stopped
After a successful startup, a status page is available at the address https://{forwarder_endpoint}.

The health of the Forwarder can also be checked via an API on https://{forwarder_endpoint}/forwarder/health, which returns, for example:
{
"state": "GREEN",
"version": "1.0.9",
"uptime": "258751",
"formattedUptime": "00d 00h 04m 18s",
"connectionState": "UP",
"forwarderId": "cc7784b6-cb6a-44a7-a00e-efab7a776f93",
"sessionId": "0f21b9fa-f52b-43cb-95c4-92a561c87605"
}
Or more extended data can be retrieved by adding ?extended=true like this https://{forwarder_endpoint}/forwarder/health?extended=true, resulting in, for example:
{
"state": "GREEN",
"version": "1.0.9",
"uptime": "411029",
"formattedUptime": "00d 00h 06m 51s",
"memory": {
"heap.max": "8342470656",
"heap.used": "60146176",
"heap.committed": "113246208",
"heap.init": "524288000",
"nonHeap.max": "-1",
"nonHeap.used": "39368184",
"nonHeap.committed": "42008576",
"nonHeap.init": "7667712"
},
"cpu": {
"jvmCpuUsage": "0%",
"systemCpuUsage": "3%"
},
"connectionState": "UP",
"forwarderId": "cc7784b6-cb6a-44a7-a00e-efab7a776f93",
"sessionId": "0f21b9fa-f52b-43cb-95c4-92a561c87605"
}
The state of the Forwarder is shown in both the Status-page and the API as GREEN in normal state or one of the following:
| State | Metrics | Details | Threshold |
|---|---|---|---|
YELLOW |
OperatingSystemMXBean#ProcessCpuLoad |
"recent cpu usage" for the Java Virtual Machine process |
50% |
YELLOW |
MemoryMXBean#HeapMemoryUsage |
[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management] |
65% |
RED |
OperatingSystemMXBean#ProcessCpuLoad |
"recent cpu usage" for the Java Virtual Machine process |
85% |
RED |
MemoryMXBean#HeapMemoryUsage |
[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management] |
90% |