Visit Azul.com Support

Monitoring and Health

Table of Contents
Need help?
Schedule a consultation with an Azul performance expert.
Contact Us

Azul Intelligence On-Premises publishes three signals, each with one job: metrics carry quantities, logs carry events, and health probes carry whether the instance may serve. No signal is reconstructed from another, so you are never asked to rebuild a counter by parsing log lines.

The Management Port

Metrics and health probes are served on a management port, 9000 by default, separate from the application port.

The separation is deliberate. The metric set describes your estate, including fleet size and vendor mix, so who can reach it is a decision you make at the network layer rather than something inherited by anyone who can reach the login page.

The management port has no authentication. It binds to 127.0.0.1:9000 by default, so only a Prometheus on the same host reaches it. For a Prometheus on another host, set MANAGEMENT_HOST to the interface it reaches, or 0.0.0.0, and firewall the port to your monitoring network.

 
curl -fs http://127.0.0.1:9000/q/health/ready
curl -fs http://127.0.0.1:9000/q/metrics

Set MANAGEMENT_PORT and MANAGEMENT_HOST in /etc/ljvmi/ljvmi.env to move it or bind it to one address. See Configuration Options.

Metrics

Metrics are published in Prometheus text format at /q/metrics. You scrape them with the Prometheus and Grafana you already run. Your installation never pushes metrics and needs no collector, so an installation with all outbound access blocked is fully observable. The one outbound connection the product can make is the optional check of the vulnerability data feed, and it exists only when you set VULN_DATA_METADATA_URL. See Security Exposure.

Eight groups are published:

Group Answers Representative series

Runtime and HTTP

Is the process healthy?

jvm_memory_used_bytes, jvm_gc_pause_seconds, http_server_requests_seconds

Ingestion

Is data still flowing?

ljvmi_ingestion_files_processed_total, ljvmi_ingestion_rows_total, ljvmi_ingestion_tick_seconds, ljvmi_ingestion_last_success_timestamp_seconds, ljvmi_inbox_backlog_files, ljvmi_inbox_tmp_swept_total (absent until the first sweep)

Query

Why is the web interface slow?

ljvmi_query_seconds, ljvmi_query_queue_seconds, ljvmi_duckdb_queries_active, ljvmi_duckdb_buffer_bytes, ljvmi_duckdb_spill_bytes, ljvmi_dataset_native_files, ljvmi_dataset_native_bytes

API

Who is automating against it, and where does a limit bite?

ljvmi_api_requests_total, ljvmi_api_rate_limited_total, labelled by whether the rate or the concurrency limit refused the call

Audit

How large is the record growing?

ljvmi_audit_entries, a gauge, since nothing removes an entry

Inventory

What is under management?

ljvmi_jvms by vendor, ljvmi_forwarders_registered, ljvmi_forwarders_silent, ljvmi_forwarder_oldest_checkin_timestamp_seconds

Health

Did the last local health check pass, and how recent is that result?

ljvmi_health_check_up, ljvmi_health_checked_at_seconds, labelled by check

Vulnerability data

Is the data behind Security Exposure loaded, current, and arriving cleanly?

ljvmi_vuln_data_generated_timestamp_seconds, ljvmi_vuln_data_activations_total, ljvmi_vuln_data_refusals_total, ljvmi_vuln_data_scan_failures_total, ljvmi_vuln_data_download_checks_total

Every tenant-scoped series carries a tenant label. ljvmi_dataset_native_files and ljvmi_dataset_native_bytes report the physical size of the one shared event database and its write-ahead log, labelled tenant="all". They are not a per-tenant estimate, and a checkpoint does not promise that the file shrinks.

The vulnerability data series behave as follows:

  • ljvmi_vuln_data_generated_timestamp_seconds is when the active dataset was built, and is NaN while none is loaded. time() - ljvmi_vuln_data_generated_timestamp_seconds is how stale the data behind Security Exposure is.

  • ljvmi_vuln_data_activations_total counts each time a dataset became active, labelled via with scan (the watched folder), download, upload, or rollback.

  • ljvmi_vuln_data_refusals_total counts datasets that failed their checks, labelled via with scan or upload. The reason is in the log line that starts with vuln-data: refused.

  • ljvmi_vuln_data_scan_failures_total counts folder scans that failed before they could judge any file.

  • ljvmi_vuln_data_download_checks_total counts finished checks of VULN_DATA_METADATA_URL, labelled outcome with up_to_date, activated, refused, or failed. It does not exist until the first check, and it never exists without the URL.

Four properties hold across the set:

  • The names are a contract. You build dashboards on them, so a rename is a breaking change announced in a release note rather than a refactor.

  • Forwarder health is aggregate. There is no per-Forwarder series, because a Forwarder-scoped label grows with your fleet and does not shrink when you decommission one. Per-Forwarder detail lives in Settings > Data Source Setup, which can show a name rather than a label value. See Checking That Data Arrives.

  • Scraping is not a source of load. Every polled gauge is read through a short-lived cache, so tightening your scrape interval cannot add load here. Gauges are accurate to within METRICS_GAUGE_TTL_MS rather than instantaneous.

  • Last-success is an absolute timestamp, not an elapsed duration, so an alert expression stays correct across a restart instead of resetting to zero with the process.

Azul ships the endpoint, not the dashboards. What you build on these series is yours.

Alerts Worth Setting

Each of these is backed by a published series:

  • No successful ingestion within the expected interval, on ljvmi_ingestion_last_success_timestamp_seconds. This is the one to set first: nothing else tells you that your estate view has quietly stopped moving.

  • Inbox backlog growing while ingestion runs, on ljvmi_inbox_backlog_files. That is the observable form of deliveries arriving faster than passes consume them, and of a spent ingestion allowance. See Quota and Usage.

  • A Forwarder gone silent, on ljvmi_forwarders_silent above zero. It counts Forwarders with no check-in or status poll for an hour, and ljvmi_forwarder_oldest_checkin_timestamp_seconds tells you how far behind the worst one is.

  • Ingestion failing or being skipped repeatedly, on ljvmi_ingestion_tick_seconds.

  • API error rate or latency departing from its baseline, on http_server_requests_seconds.

  • Vulnerability data that stopped arriving or was refused, on ljvmi_vuln_data_refusals_total, ljvmi_vuln_data_scan_failures_total, and ljvmi_vuln_data_download_checks_total{outcome=~"failed|refused"}, and data older than a few weeks, on ljvmi_vuln_data_generated_timestamp_seconds.

  • Any increase in ljvmi_inbox_tmp_swept_total, which counts abandoned in-flight files that the sweep removed. Each one is a delivery that was lost.

  • Delivery storage unusable, on ljvmi_health_check_up{check="delivery-storage"} at 0, or on its ljvmi_health_checked_at_seconds older than about 90 minutes. The check runs hourly, so give it a staleness budget above one hour rather than the readiness one.

Health Probes

Liveness and readiness are served at /q/health on the management port.

Liveness reports that the process is running. Readiness reports that this instance can serve: the application database is reachable, the query engine’s connection is open, and the event database is accessible. Schema migrations are applied during startup, before the instance serves anything. Health results are also published as ljvmi_health_check_up, sampled every 30 seconds.

Readiness never fails on a data condition. Stale, absent, or delayed ingestion is an alerting condition and is published as one. If it also failed readiness, an orchestrator would restart or remove a perfectly serviceable instance because a Forwarder upstream went quiet, turning somebody else’s outage into yours.

The delivery binding is probed separately, at startup, hourly, and on demand at /q/health/group/delivery-storage on the management port. The probe writes a marker, reads it back, and deletes it, all three, because the permission that matters most is the one to write. It answers 200 when the binding is usable and 503 when it is not, and it never fails startup or readiness: a misconfigured bucket that made the product unbootable would take away the screen that explains what is wrong.

Its result is published as ljvmi_health_check_up{check="delivery-storage"}, and that gauge is what you alert on. Use the route only to re-check right after you fix the storage, without waiting for the hour. Every request to it, and to /q/health/group, which runs all groups, writes and deletes a marker in your storage, so never scrape or poll it.

That probe is worth knowing about, because without it a mistyped key surfaces as every Forwarder getting a refusal on upload, in somebody else’s log, on somebody else’s host, as far as possible from whoever typed it.

/health on the application port stays public and answers only whether the HTTP server has come up.

Application Logs

 
journalctl -u ljvmi

In production, logs are written to stdout as one JSON object per line, carrying tenant and request identifiers, so a line can be joined to a metric spike and a log shipper needs no pattern matching.

Each ingestion pass logs, per tenant: files considered, files ingested per channel, rows written, ledger entries added, temporary files swept, and how long it took. A skipped directory logs which Forwarder it named and why. A refused call logs that a key was not recognised, never the key.

After an upgrade, filter for flyway to see which database migrations ran:

 
journalctl -u ljvmi | grep -i flyway

Two things are deliberately in the log rather than in the audit log, because nobody performed them: a quota threshold crossing, and a connection refused for an unrecognised key. See Audit Log.

Failure Modes and What They Cost

Most of these cost you nothing but time, which is worth knowing before an incident rather than during one.

Situation What happens

A Forwarder cannot reach your installation

Its connection status shows DOWN until the next successful status poll. Delivery itself does not go through your installation, so the Forwarder keeps writing files, or keeps uploading while its upload grant is valid. The Forwarder does not buffer data on disk. See Installing the Forwarder.

You disable a Forwarder

It stops delivering at its next status poll: it keeps running, but drops the data it receives, and delivers again when you enable it. Its deliveries are skipped at pickup whatever it managed to write.

An ingestion pass fails part-way

Nothing is recorded, so the next pass reads the same deliveries again.

One unreadable delivery

Skipped and logged, and recorded so it is not retried for ever. The rest of the batch is ingested, and a corrected re-export arrives as a new delivery and is picked up normally.

The object store is unreachable, or its credential is wrong

Reported by the startup and hourly probe. Ingestion finds nothing and the backlog gauge stops moving. No data is lost, because a delivery that was never read was never removed.

An ingestion allowance is spent

Deliveries stay sealed in the inbox and are taken when the allowance returns. Every screen keeps working. See Quota and Usage.

Clock drift on a Forwarder host

Avoided. Delivery timestamps come from your installation’s clock, handed back at check-in.

A lost Admin password

Recovered by an Operator with a configuration switch and a restart. See Recovering a Lost Admin Password.

Finding the Version You Run

Support questions start with the version. Any signed-in user finds it on the About page, opened from the bottom of the sidebar, as Version <version> (<build stamp>).

On the host, the start-up log line names the same version, and the VERSION file in the release directory records it with the build stamp:

 
journalctl -u ljvmi | grep 'ljvmi-server'
cat /opt/ljvmi/current/VERSION