Installing the Forwarder

Using a Properties File

In on-premises mode, the Forwarder needs from you where to reach your installation, its access key, and whether it serves your Agents and AMT over plain HTTP or HTTPS. Everything else, such as the storage backend, the path it writes to, and the limits of the delivery files, comes back from your installation when the Forwarder checks in.

Use --mode=local --dump-config to see the effective value of every setting and where it comes from, and --mode=local --help args to list the settings of the mode. Settings that don’t apply to on-premises mode, such as azul.ic.host, azul.forwarder.domain.names, and the proxy settings, are ignored with a warning in the log.

Frequently Used Options

Delivery File Limits

The Forwarder completes a delivery file when it reaches a number of lines, a size, or an age, whichever comes first. Your installation can make these limits smaller than the defaults, never larger.

Storing in a Directory

With filesystem storage, the Forwarder writes its delivery files straight into your installation’s inbox directory, so the Forwarder host needs that directory, writable: the same disk when both run on one host, otherwise a network share or a container volume. Grant the Forwarder access to the inbox only, not to the databases or the configuration of your installation.

At check-in, your installation tells the Forwarder where to write. By default, that is the installation’s own absolute path, <INBOX_DIR>/<tenant>/<forwarder>/, so mount the inbox at exactly the same path on the Forwarder host. When the Forwarder sees the inbox at another path, for example /mnt/ljvmi-inbox, set INBOX_WRITE_PREFIX on your installation to that path. Check-in then hands back <prefix>/<tenant>/<forwarder>/. See Storage Locations.

Storing in an S3-Compatible Bucket

Your installation decides the storage when the Forwarder checks in. With an S3-compatible bucket (AWS S3, Silo, RustFS, Ceph), the Forwarder uploads with a short-lived upload grant that your installation issues and renews, so it holds no bucket name and no S3 credentials.

Uploads to the bucket use HTTPS, and the Forwarder verifies the bucket’s certificate and host name. A bucket with a certificate from a public CA, such as AWS S3, needs no setting. For a private or self-signed certificate, such as a self-hosted Silo, RustFS, or Ceph, point the Forwarder at a truststore with the issuing CA before you start it: azul.local.s3.truststore.path and, when it is protected, azul.local.s3.truststore.password. This truststore replaces the default truststore of the JVM for the bucket. Your installation assigns the storage only at check-in, so prepare these settings whenever it may assign S3. Otherwise the Forwarder doesn’t start.

Reading the Log

The Forwarder starts accepting connections only after the settings are valid, your installation accepts the access key, your installation assigns a supported storage, and that storage works: the directory is writable, or the S3 endpoint is reachable. Otherwise, it logs the reason and stops, so a misconfigured Forwarder fails at once on the host where you configure it.

A successful start ends with a line like this:

 
[INFO] c.a.c.f.LjvmiForwarder - Start Forwarder in local mode: scheme=http://*:443, storageType=FS, writing to <delivery-directory>

A failed start logs the reason. When the check-in with your installation fails, the reason is followed by Failed to authorize the forwarder, the application will be stopped. These messages tell you what to fix:

With plain HTTP, the Forwarder also logs a warning at startup that it serves plain HTTP and that the traffic from your Agents is unencrypted.

Opening the Status Screen

After a successful start, the Forwarder serves a status page at http://<FORWARDER_IP:PORT>/, or at https:// when it serves HTTPS.

Querying the Health API

 
curl http://<FORWARDER_IP:PORT>/forwarder/health

When the Forwarder serves HTTPS, use https://, and when your own CA issued its certificate, add --cacert <ca-certificate>.

The response contains a localDelivery block with the state of the delivery:

 
{
  "state": "GREEN",
  "mode": "local",
  "version": "2.1.0",
  "uptime": "3600000",
  "formattedUptime": "00d 01h 00m 00s",
  "localDelivery": {
    "state": "GREEN",
    "storageType": "fs",
    "writePath": "<delivery-directory>",
    "deliverySuspended": "false",
    "droppedObjects": "0"
  },
  "connectionState": "UP",
  "sessionId": "<session-id>"
}

Add ?extended=true for memory and CPU details.

The Forwarder also publishes Prometheus metrics at http://<FORWARDER_IP>:<METRICS_PORT>/metrics. Most of them have a type label that is events or amt. A series appears once it has a value, so the S3 metrics are present only with S3 storage, and the drop counter only after a drop.

The Forwarder’s metrics are the only view of drops on the Forwarder side. They do not count data dropped while delivery is paused. The web interface of your installation also lists every Forwarder that has authenticated, with its last check-in and its last delivery. Use that list to confirm your data arrives. See Checking That Data Arrives.