Setting Up the Forwarder

Table of Contents

Quick Overview
Configuring Your Firewall
Configuring a New Forwarder in the Web UI
- Adding a Forwarder
Setting up the Forwarder in Your Environment

Need help?

Schedule a consultation with an Azul performance expert.

Setting up a Forwarder involves steps both in the Azul Intelligence Cloud Web UI and on a machine within your environment. This step-by-step guide helps you in this process.

Quick Overview

You can configure the Forwarder in different ways as described further on this page. This is a quick overview using a properties file:

Configure your firewall, if needed, to make sure the Forwarder can access the required endpoints.
Configure a new Forwarder in the WebUI.
While still on the WebUI, make sure you download the latest Forwarder application and the forwarder-settings.properties file.

Edit the donwloaded properties file and replace the values starting with < and ending with > with a value of your choice, matching your system.

 forwarder.keystore.path = <path-to-keystore>/forwarder.jks
forwarder.keystore.password = <desired keystore password>

Becomes, for example:

 forwarder.keystore.path = /opt/forwarder/forwarder.jks
forwarder.keystore.password = MySuperSecretPassword

Start the Forwarder with the following command:

 java -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar

Check the state of the Forwarder in the log, via the web interface, or with the API, as described here.

Configuring Your Firewall

The Forwarder must have access to the Azul Intelligence Cloud and AWS S3 services, through the firewall of your system. Two rules are needed with outgoing access:

The host address (API URL) on port 443 (HTTPS) of the Azul Intelligence Cloud service that has been set up for you by Azul. You can copy this from the Web UI Forwarder setup screen. For instance: YOUR_INSTANCE.api.ic.azul.com.
To be able to upload artifacts, outgoing traffic to *.amazonaws.com must also be allowed.

Configuring a New Forwarder in the Web UI

Administrators must set up a Forwarder in the Azul Intelligence Cloud Web UI to connect it to their account. Forwarders are created on a per-Azul Intelligence Cloud basis and can be managed by administrators.

Visit the Web UI of your Azul Intelligence Cloud instance.

Adding a Forwarder

In the "Settings" section of the Web UI click "Forwarders".
In the "Forwarders list", click the "Create new" button.
Fill in the fields to configure the name and domain names for the new Forwarder.
- Forwarder name: this is a free value. It is a name to manage your Forwarder more easily, not a domain name. If your organization uses multiple Forwarders, consider a naming convention that signifies usage.
- Domain Names: a list of domain names used by your JDKs to communicate with this Forwarder. The first entry is your certificate’s Common Name (CN) and subsequent entries are Server Alternative Names (SAN).
Click the "Create" button to store the new Forwarder.
The "Forwarder ID" and "Access key" are shown, together with the pre-populated settings when you click on "Show settings". You can also download them as a ready-to-use properties file with the "Download as file" button.

Note

For security reasons, by default the Forwarder is not allowed to run as a server in a public domain to avoid unwanted behaviour like sending sensitive JVM data to the public internet and causing incorrect traffic on the domain.

If your environment requires the use of a public domain, you can request Azul support to change the configuration of your Azul Intelligence Cloud environment to add it as an approved domain.

Setting up the Forwarder in Your Environment

The remaining configuration of the Forwarder has to be done within your environment.

Server Requirements

The Forwarder can run on any server within your environment at a location where it can be reached by many JVMs and the Forwarder itself can reach Azul Intelligence Cloud. The only requirement is a Java runtime, version 8 (minimally 1.8.0_252) or newer.

Downloading the Forwarder

The Forwarder application can be downloaded from the Web UI and from the Azul repository:

 curl -O https://cdn.azul.com/forwarder/1.0.8/forwarder.jar

Configuring and Starting the Forwarder Application

Three different possibilities are supported to configure the Forwarder, and the values defined in each of them have this priority:

Command line option
Properties file
Environment variables

Some settings are the ones filled in or retrieved from the Web UI earlier, others are related to the created certificate. The settings are used by the Forwarder to connect to Azul Intelligence Cloud. Once authenticated, the Forwarder exchanges this information with Azul Intelligence Cloud to receive cryptographic certificates trusted by the JVM.

You must use different startup commands, depending on the method used to specify the settings.

Start Forwarder With Command Line Options

When you are defining the settings for the Forwarder with system or environment variables, you can start the Forwarder with this command:

 java -DicHost=<your-endpoint>.azul.com \
  -DforwarderAccessKey=<forwarder-access-key> \
  -DforwarderDomainNames=<domain-names-from-web-ui> \
  -DforwarderKeystorePath=<path-to-keystore>/forwarder.jks \
  -DforwarderKeystorePassword=<desired keystore password> \
  -jar forwarder.jar

You can use command line options in combination with a properties file, for example, when you need extra logging:

 java -DlogLevel=debug -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar

Start Forwarder With Properties File

When you want to use a properties-file to configure the Forwarder, follow these steps.

Create a new file forwarder-settings.properties, or download it from the Web UI > Settings > Forwarders.
```
 touch forwarder-settings.properties
```

Edit the file and add the following content, specifying all the required settings as defined in the table with configuration options, by replacing the values starting with < and ending with > with a value of your choice.

 ic.host = <your-endpoint>.azul.com
forwarder.access.key = <forwarder-access-key>
forwarder.domain.names = <domain-names-from-web-ui>
forwarder.keystore.path = <path-to-keystore>/forwarder.jks
forwarder.keystore.password = <desired keystore password>

Provide the file location with the commandline option -DforwarderSettings when you start the Forwarder. For example:
```
 java -DforwarderSettings=forwarder-settings.properties -jar forwarder.jar
```

Start Forwarder With Environment Variables

Export each of the variables before you start the Forwarder. For example:

 export IC_HOST=<your-endpoint>.azul.com
export FORWARDER_ACCESS_KEY=<forwarder-access-key>
export FORWARDER_DOMAIN_NAMES=<domain-names-from-web-ui>
export FORWARDER_KEYSTORE_PATH=<path-to-keystore>/forwarder.jks
export FORWARDER_KEYSTORE_PASSWORD=<desired keystore password>
java -jar forwarder.jar

All Configuration Options

This is an overview of the available configuration options.

Command Line Option (1)
Properties File (2)
Environment Variable (3) Description

Command Line Option (1) Properties File (2) Environment Variable (3)	Description
(1): icHost (2): ic.host (3): IC_HOST	Required. The host of Azul Intelligence Cloud, copied from the Web UI Forwarder setup screen.
(1): cloudConnectionPoolSize (2): cloud.connection.poolSize (3): CLOUD_CONNECTION_POOL_SIZE	Connection pool size used to manage connections between the Forwarder and Azul Intelligence Cloud. The default value is `50`.
(1): cloudResponseMaxContentLength (2): cloud.response.maxContentLength (3): CLOUD_RESPONSE_MAX_CONTENT_LENGTH	The maximum length of response content, in bytes, that the Intelligence Cloud can send to the virtual machine. The default value is `524288`.
(1): forwarderAccessKey (2): forwarder.access.key (3): FORWARDER_ACCESS_KEY	Required. The Forwarder Access Key copied in the Azul Intelligence Cloud Web UI after adding the Forwarder. This Access Key is used for authentication to Azul Intelligence Cloud.
(1): forwarderDomainNames (2): forwarder.domain.names (3): FORWARDER_DOMAIN_NAMES	Required. A comma-delimited list with he Forwarder Domain Names (all or subset of them) that were used to create the Forwarder in the Azul Intelligence Cloud Web UI. These Domain Names are used as SN/SAN attributes when creating the certificates for the authorization phase between the Forwarder and Azul Intelligence Cloud. This list can contain IP addresses, but must contain at least one domain name.
(1): forwarderKeystorePath (2): forwarder.keystore.path (3): FORWARDER_KEYSTORE_PATH	Required. If no JKS file is found in the specified path, an empty keystore with the same name gets created. This keystore is used to store the auto-generated Forwarder certificates.
(1): forwarderKeystorePassword (2): forwarder.keystore.password (3): FORWARDER_KEYSTORE_PASSWORD	Required. The password you want to use for the keystore.
(1): forwarderPort (2): forwarder.port (3): FORWARDER_PORT	The listening port used by the Forwarder for incoming connections from VMs. The default value is `443`.
(1): forwarderSettings (2): - (3): -	File path used by the Forwarder to read the settings from a file.
(1): forwarderStateReportPath (2): forwarder.state.reportPath (3): FORWARDER_STATE_REPORT_PATH	File path used by the Forwarder to periodically store its state.
(1): forwarderStateCheckPeriod (2): forwarder.state.checkPeriod (3): FORWARDER_STATE_CHECK_PERIOD	Human-readable period specifying how often the state of the Forwarder should be checked and stored in the report path. It can be defined in the following formats: {N}d {N}h {N}min {N}sec {N}ms
(1): http.proxyHost and http.proxyPort (2): HTTP_PROXY_HOST and HTTP_PROXY_PORT (3): http.proxy.host and http.proxy.port	Enables HTTP proxy.
(1): https.proxyHost and https.proxyPort (2): HTTPS_PROXY_HOST and HTTPS_PROXY_PORT (3): https.proxy.host and https.proxy.port	Enables HTTPS proxy (and overrides HTTP proxy if both are enabled).
(1): proxyKeystorePath and proxyKeystorePassword (2): PROXY_KEYSTORE_PATH and PROXY_KEYSTORE_PASSWORD (3): proxy.keystore.path and proxy.keystore.password	Custom keystore (truststore) to allow connection to proxies using custom self-signed certificates.
(1): logLevel (2): log.level (3): LOG_LEVEL	The default log level is `INFO`. By default, when errors are caught in the Forwarder, only error messages are logged, without a stack trace. If you need to see the stack trace for debugging or troubleshooting purposes, use the `TRACE` level. Possible values (case-insensitive): ALL TRACE DEBUG INFO WARN ERROR OFF

(1): icHost
(2): ic.host
(3): IC_HOST

Required.

The host of Azul Intelligence Cloud, copied from the Web UI Forwarder setup screen.

(1): cloudConnectionPoolSize
(2): cloud.connection.poolSize
(3): CLOUD_CONNECTION_POOL_SIZE

Connection pool size used to manage connections between the Forwarder and Azul Intelligence Cloud. The default value is 50.

(1): cloudResponseMaxContentLength
(2): cloud.response.maxContentLength
(3): CLOUD_RESPONSE_MAX_CONTENT_LENGTH

The maximum length of response content, in bytes, that the Intelligence Cloud can send to the virtual machine. The default value is 524288.

(1): forwarderAccessKey
(2): forwarder.access.key
(3): FORWARDER_ACCESS_KEY

Required.

The Forwarder Access Key copied in the Azul Intelligence Cloud Web UI after adding the Forwarder. This Access Key is used for authentication to Azul Intelligence Cloud.

(1): forwarderDomainNames
(2): forwarder.domain.names
(3): FORWARDER_DOMAIN_NAMES

Required.

A comma-delimited list with he Forwarder Domain Names (all or subset of them) that were used to create the Forwarder in the Azul Intelligence Cloud Web UI. These Domain Names are used as SN/SAN attributes when creating the certificates for the authorization phase between the Forwarder and Azul Intelligence Cloud.

This list can contain IP addresses, but must contain at least one domain name.

(1): forwarderKeystorePath
(2): forwarder.keystore.path
(3): FORWARDER_KEYSTORE_PATH

Required.

If no JKS file is found in the specified path, an empty keystore with the same name gets created. This keystore is used to store the auto-generated Forwarder certificates.

(1): forwarderKeystorePassword
(2): forwarder.keystore.password
(3): FORWARDER_KEYSTORE_PASSWORD

Required.

The password you want to use for the keystore.

(1): forwarderPort
(2): forwarder.port
(3): FORWARDER_PORT

The listening port used by the Forwarder for incoming connections from VMs. The default value is 443.

(1): forwarderSettings
(2): -
(3): -

File path used by the Forwarder to read the settings from a file.

(1): forwarderStateReportPath
(2): forwarder.state.reportPath
(3): FORWARDER_STATE_REPORT_PATH

File path used by the Forwarder to periodically store its state.

(1): forwarderStateCheckPeriod
(2): forwarder.state.checkPeriod
(3): FORWARDER_STATE_CHECK_PERIOD

Human-readable period specifying how often the state of the Forwarder should be checked and stored in the report path. It can be defined in the following formats:

{N}d
{N}h
{N}min
{N}sec
{N}ms

(1): http.proxyHost and http.proxyPort
(2): HTTP_PROXY_HOST and HTTP_PROXY_PORT
(3): http.proxy.host and http.proxy.port

Enables HTTP proxy.

(1): https.proxyHost and https.proxyPort
(2): HTTPS_PROXY_HOST and HTTPS_PROXY_PORT
(3): https.proxy.host and https.proxy.port

Enables HTTPS proxy (and overrides HTTP proxy if both are enabled).

(1): proxyKeystorePath and proxyKeystorePassword
(2): PROXY_KEYSTORE_PATH and PROXY_KEYSTORE_PASSWORD
(3): proxy.keystore.path and proxy.keystore.password

Custom keystore (truststore) to allow connection to proxies using custom self-signed certificates.

(1): logLevel
(2): log.level
(3): LOG_LEVEL

The default log level is INFO.

By default, when errors are caught in the Forwarder, only error messages are logged, without a stack trace. If you need to see the stack trace for debugging or troubleshooting purposes, use the TRACE level.

Possible values (case-insensitive):

ALL
TRACE
DEBUG
INFO
WARN
ERROR
OFF

Note	If you are concerned about the security of the key (FORWARDER_ACCESS_KEY) and/or password (FORWARDER_KEYSTORE_PASSWORD) values, using environment variables is the recommended approach.

Checking the State of the Forwarder

You can check the status of the Forwarder in multiple ways.

Checking the Forwarder Log

The Forwarder log output shows if a connection can be made with Azul Intelligence Cloud.

Successful Start

A successful startup contains these info messages:

 [INFO] c.a.c.f.ForwarderRunner - New forwarder session started: sessionId=6a964219-f7ff-4ce8-bbb7-26694e82c6b1
[WARN] c.a.c.f.a.ForwarderAuthManager - The JKS keystore does not exist at the specified path, an empty keystore will be created: path=/var/forwarder/forwarder.jks
[INFO] c.a.c.f.a.ForwarderAuthManager - Issue forwarder certificate signed by IC
[INFO] c.a.c.f.a.AuthClient - Forwarder certificate signed by IC issued successfully
[INFO] c.a.c.f.a.ForwarderAuthManager - Certificate chain successfully added to the provided keystore: alias=ic.forwarder, keystore=/var/forwarder/forwarder.jks
[INFO] c.a.c.f.a.ForwarderAuthManager - Schedule the forwarder certificate reissue: reissueDate=Wed Apr 24 13:12:13 CEST 2024, certificateNotAfter=Wed Apr 24 14:12:13 CEST 2024
[INFO] c.a.c.f.Forwarder - Start forwarding *:443 to <your-endpoint>.azul.com:443

Unsuccessful Start

If either the host of Azul Intelligence Cloud, the Forwarder Access Key, or one of the other settings is incorrect, the Forwarder displays an appropriate error message and fail to start. Keep an eye on the output of the Forwarder to check if it could start successfully. These are some possible error messages while connecting:

 # IC_HOST is incorrect
[ERROR] c.a.c.f.a.ForwarderAuthManager - Failed to verify IC-signed certificate: error=<your-endpoint>.azul.com: nodename nor servname provided, or not known
[ERROR] c.a.c.f.ForwarderRunner - Failed to authorize the forwarder, the application will be stopped

# Authentication problem
[ERROR] c.a.c.f.a.AuthClient - Failed to init certificate challenge: responseCode=403, responseContent={"message":"Forbidden"}
[WARN] c.a.c.f.Forwarder - Please double-check your forwarder settings: IC Host URL, Forwarder Access Key, Forwarder Domain Names
[ERROR] c.a.c.f.ForwarderRunner - Failed to authorize the forwarder, the application will be stopped

Status Screen

After a successful startup, a status page is available at the address https://{FORWARDER_ENDPOINT}.

Status API

The health of the Forwarder can also be checked via an API on https://{FORWARDER_ENDPOINT}/forwarder/health, which returns, for example:

 {
  "version":"1.0.7",
  "state":"GREEN",
  "uptime":"327454123",
  "formattedUptime":"03d 18h 57m 34s"
}

Or more extended data can be retrieved by adding ?extended=true like this https://{FORWARDER_ENDPOINT}/forwarder/health?extended=true, resulting in, for example:

 {
  "state": "RED",
  "version": "1.0.7",
  "uptime": "100610966",
  "formattedUptime": "01d 03h 56m 50s",
  "reasons": [
    "CPU usage exceeded the RED threshold: usage=86%, threshold=85%",
    "Heap memory usage (70%) has exceeded 65%"
  ],
  "memory": {
    "heap.max": "10485760",
    "heap.used": "7361768",
    "heap.committed": "10485760",
    "heap.init": "10485760",
    "nonHeap.max": "-1",
    "nonHeap.used": "28254384",
    "nonHeap.committed": "31522816",
    "nonHeap.init": "7667712"
  },
  "cpu": {
    "jvmCpuUsage": "86%",
    "systemCpuUsage": "97%"
  }
}

List of Possible Forwarder States

The state of the Forwarder is shown in both the Status-page and the API as GREEN in normal state or one of the following:

State	Metrics	Details	Threshold
YELLOW	OperatingSystemMXBean#ProcessCpuLoad	"recent cpu usage" for the Java Virtual Machine process	50%
YELLOW	MemoryMXBean#HeapMemoryUsage	[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management]	65%
RED	OperatingSystemMXBean#ProcessCpuLoad	"recent cpu usage" for the Java Virtual Machine process	85%
RED	MemoryMXBean#HeapMemoryUsage	[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management]	90%

State

Metrics

Details

Threshold

YELLOW

OperatingSystemMXBean#ProcessCpuLoad

"recent cpu usage" for the Java Virtual Machine process

50%

YELLOW

MemoryMXBean#HeapMemoryUsage

[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management]

65%

RED

OperatingSystemMXBean#ProcessCpuLoad

"recent cpu usage" for the Java Virtual Machine process

85%

RED

MemoryMXBean#HeapMemoryUsage

[amount of used memory in bytes] / [maximum amount of memory in bytes that can be used for memory management]

90%