Visit Azul.com Support

Monitoring Optimizer Hub

Table of Contents
Need help?
Schedule a consultation with an Azul performance expert.
Contact Us

You can monitor your Optimizer Hub using the standard Kubernetes monitoring tools (Prometheus and Grafana) and through log files.

Using Prometheus and Grafana

The Optimizer Hub components are already configured to expose key metrics for scraping by Prometheus. Follow Configuring Prometheus and Grafana to set up these monitoring tools and check Using the Grafana Dashboard for more info about the different sections of the dashboard.

Retrieving Optimizer Hub Logs

All Optimizer Hub components, including third-party ones, log some information to stdout. These logs are very important for diagnosing problems.

You can extract individual logs with the following command:

 
kubectl -n my-opthub logs {pod}

However by default Kubernetes keeps only the last 10 MB of logs for every container, which means that in a cluster under load the important diagnostic information can be quickly overwritten by subsequent logs.

You should configure log aggregation from all Optimizer Hub components, so that logs are moved to some persistent storage and then extracted when some issue needs to be analyzed. You can use any log aggregation One suggested way is to use Loki. You can query the Loki logs using the logcli tool.

Here are some common commands you can run to retrieve logs:

  • Find out host and port where Loki is listening

     
    export LOKI_ADDR=http://{ip-adress}:{port}

  • Get logs of all pods in the selected namespace

     
    logcli query --since 24h --forward --limit=10000 '{namespace="zvm-dev-3606"}'

  • Get logs of a single application in the selected namespace

     
    logcli query --since 24h --forward --limit=10000 '{namespace="zvm-dev-3606" app="compile-broker"}'

  • Get logs of a single pod in the selected namespace

     
    logcli query --since 24h --forward --limit=10000 '{namespace="zvm-dev-3606",pod="compile-broker-5fd956f44f-d5hb2"}'

Extracting Compilation Artifacts

Optimizer Hub uploads compiler engine logs to the blob storage. By default, only logs from failed compilations are uploaded.

You can retrieve the logs from your blob storage, which uses the directory structure <compilationId>/<artifactName>. The <compilationId> starts with the VM-Id which you can find in connected-compiler-%p.log:

 
# Log command-line option
-Xlog:concomp=info:file=connected-compiler-%p.log::filesize=500M:filecount=20

# Example:
[0.647s][info ][concomp] [ConnectedCompiler] received new VM-Id: 4f762530-8389-4ae9-b64a-69b1adacccf2

Note About gw-proxy Metrics

The gw-proxy component in Optimizer Hub uses, by default, /stats/prometheus as target HTTP endpoint to provide metrics. Most other Optimizer Hub components use /q/metrics. If you make manual changes in the configuration of the metrics for individual Kubernetes Deployments in the Optimizer Hub installation, make sure that you don’t use the /q/metrics for the gw-proxy deployment. Doing so would lead to confusion when metrics are processed.