Optimizer Hub Release Notes

Includes bug fixes for Optimizer Hub 1.10.1.

The REST API in Optimizer Hub has been improved and extended:

A new readiness endpoint /api/opthub-health/healthy is available and replaces /q/health. You can use this endpoint to determine when it is safe to route traffic to a newly started Optimizer Hub cluster. This endpoint is provided by the gateway component, instead of the other ones, which are provided by the mgmt-gateway.

Returned states:

With the new Scaling API, you can instrument Optimizer Hub to temporarily increase the minimum number of vCPUs between a start and end timestamp. Multiple calls can be made to this API and Optimizer Hub will take all given timestamps and potential overlaps into account to start and stop the extra resources.

See Sizing and Scaling your Optimizer Hub Installation for more info.

The Management Gateway component in Optimizer Hub is now installed by default. It has become an essential part of any deployment as it provides the REST APIs.

Azul Zing Builds of OpenJDK, version 25.02 and newer, provide a new command line option -XX:CNCLocalFallbackOptLevelLimit=<3,2,1,0> to define the OptLevel for local fallback. See Cloud Native Compiler JVM Options.

Secrets can be externally defined to allow you to manage Kubernetes secrets independent of the Optimizer Hub configuration. See Using Externally Defined Secrets for more info.

Optimizer Hub now requires you to specify enough vCores to provision at least one Compile Broker if you have used the full installation. The minimum number of vCores is 39. When this is not the case, one or more of the following messages can be found in the log files:

 
The compile-broker's minimum replicas must be greater than 0
The gateway's minimum replicas must be greater than 0
The cache's minimum replicas must be greater than 0

Improvement in ReadyNow Orchestrator to reduce the time before it can serve profiles during a restart of Optimizer Hub with a pre-populated storage.

Includes bug fixes for Optimizer Hub 1.10.0.

Fix for a problem where cross-region syncing of ReadyNow Orchestrator got stuck indefinitely. A better timeout implementation was added to remote calls (fetching profiles from another region), to prevent such blocks in case of network glitches.

Azul Zing Builds of OpenJDK, version 24.08 introduced the new -XX:ProfileName=<name> option that allows a JVM instance to specify a profile name to Optimizer Hub. This name allows multiple JVM instances that use the same profile name to share the Optimizer Hub functionality, such as the use of ReadyNow Orchestrator profiles and Cloud Native Compiler caching.

This new feature also introduces the new command line option -XX:+EnableRNO that enables ReadyNow read and writes in the JVMs against ReadyNow Orchestrator, using ProfileName as the name for the profile log. More info is provided in Using ReadyNow Orchestrator > ReadyNow Orchestrator JVM Options.

The previously used options (OptHubHost and ProfileLogName) is still supported to existing configurations don’t break. But we advise to use the new settings with OptHubHost, EnableRNO, and ProfileName.

Native support for Google Cloud Platform Blob Storage is added. See Installing Optimizer Hub on Google Cloud.

MinIO has been removed from the helm chart as it is mainly intended for testing and demos. For production, cloud-managed blob storage is recommended (AWS S3, Azure Blob Storage, GCP Blob Storage, or S3-compatible (e.g. for Alibaba)).

See Installing Optimizer Hub on Kubernetes > Configure blob storage for more info.

The database storage for Code Cache is deprecated because blob storage is the best production-friendly option as it is more scalable, highly available, and durable. At the same time, blob storage is simpler to maintain.

See Optional Database Pod Configuration if you want to keep using the database.

Avoid cleanup of profiles from systems with low restarts by using the last time the JVM requesting that profile was seen alive, rather than the last time the profile was requested.

Improved time-outs for Profile Sync Task. In some configurations with cross-region syncing, the sync task could get stuck because of incorrect configurations. This has been fixed with improved time-outs.

Cross-region syncing of ReadyNow Orchestrator profiles is improved to let new instances check if a later promoted generation becomes available.

Improved the handling of unloaded or unknown classes.

A large amount of stored data may require a significant grace period after a restart of Optimizer Hub to avoid issues with profile download.

Includes bug fixes for Optimizer Hub 1.9.4.

Faster down-scaling to release resources that are no longer needed.

The stabilization window in the scaling configuration changed from 1 to 2 minutes.

On the first connection to Optimizer Hub, each JVM was getting a certain about of compile-broker capacity allocated. This could cause a high number of compile-brokers when many JVMs connect but don’t have enough compilations. This feature is now disabled by default.

Includes bug fixes for Optimizer Hub 1.9.3.

When using Optimizer Hub with AWS, you can now use an K8S ServiceAccount for S3 permissions. For more info, check out the documentation at Using AWS Service Accounts.

Profile download errors, caused by any reason, are now reflected in metrics and visible in the Grafana dashboard.

Improved stability of the readiness probes for the Optimizer Hub Components.

The limit of incoming connections for single instances of the gateway (Envoy proxy) is increased from 1024 to 3072.

You can now configure the AWS region of S3 with the Helm value storage.s3.region.

Implemented stricter data consistency validation during the upload of RNO profiles to prevent BlobNotFound errors later.

Old Prime JVMs (pre-23.08), using an earlier protocol version, can send profile chunks in an incorrect order. This can lead to some chunks getting lost, e.g., due to reconnections.

With newer Prime JVMs, using the latest protocol, this same issue has been noticed very rarely, and research is ongoing.

Includes bug fixes for Optimizer Hub 1.9.2.

Because of configuration changes, --set version is no longer supported during installation using Helm.

You can now also specify Service labels, as described in the Kubernetes installation instructions

Improved cleanup policy for profiles written with continueRecordingOnPromotion to avoid profiles to grow too much.

Fixed a bug where MariaDB deployed with an empty password, potentially allowing unauthorized root connections. MariaDB is now deployed with a randomly set password.

In some configurations with cross-region syncing, the sync task can get stuck because of incorrect configurations.

Fixes an issue in 1.8.1 where the cache component is not able to scale up.

Fixes an issue that caused unexpected HTTP/1.x requests for GET /q/metrics to be reported in the logging.

Monitoring with Prometheus and Grafana is no longer included in the Optimizer Hub Helm charts, but must be configured separately as described on Monitoring Optimizer Hub.

In the past, each release was bundled with the most likely JVM compiler engine. This is no longer the cause, resulting in smaller images.

Session rebalancing has been improved with an (optional) Envoy proxy, or any other gRPC-aware load balancer/ingress in your Kubernetes cluster. More information can be found on Configuring gRPC Proxy.

Documentation has been extended with installation instructions for Google Cloud.

Profile Log Service now stores profile metadata in the blob storage. This means that you can use AWS S3 or Azure Blob Storage to persist profile metadata and no longer need to back up the database pod with persistent storage. This change also means that when you upgrade from any release prior to 1.7.1 your previously collected profiles are no longer available.

Profile Log service automatically cleans-up unused profile names when not requested for a defined time. You can configure the duration with profileLogService.cleaner.keepUnrequestedProfileNamesFor. See Configuring ReadyNow Orchestrator for more configuration information.

New version of the Grafana monitoring dashboard with additional charts, and updates related to changes in the metrics reported by CNC components.

You can define the profile log name with a Java property specified in the command line, in the format %prop={PROPERTY}%. For more info, see "Using the Profile Log Service > Substitution Macros".

Improved setup for Profile Log Service-only deployment.

CNC can automatically recover from DB pod restarts with loss of schema. To enable this feature, set the following value in values-override.yaml:

 
dbschema.auto-recreate.enabled=true

The hostPort attribute is no longer required and included for the storage pod.

Improved performance of autoscaling for the Compiler Service.

Usability improvements to the Profile Log Service Admin REST API.

Native blob storage on Azure and AWS. Extra documentation is provided on:

Added documentation of the CNC API.

The CNC helm charts now use full names for the Docker images to prevent issues in environments where a Docker Hub mirror is used.

CNC pods can now be run as non-root user. The Docker images have a non-root user and the Helm chart is instructing Kubernetes to use this non-root user for CNC pods.

To avoid restarts of the Gateway pod when a large number of clients try to write profile logs at the same time, a default limit has been configured.

Upgrade from version 1.6.0 can be done with a helm upgrade, as described on Upgrading Cloud Native Compiler.

Gateway pod gets restarted when large number of clients try to write profile simultaneously.

JVMs released before CNC 1.6.1 use HTTP for uploads of the compiler engine. Since version 1.6.1, gRPC is used and the HTTP port is disabled by default in values.yaml. Because of this, these JVMs are not able to upload their appropriate compiler engine to CNC.

When a CNC version prior to 1.6.1 already has been used and upgraded, the older JVMs keep working with CNC, because the upload is not needed anymore.

The first attempt to download a previously existing profile, after CNC upgrade to 1.6.1 can fail with a timeout.

Cloud Native Compiler has a new Profile Log Service. This service allows you to read and write ReadyNow profile logs to Cloud Native Compiler. This simplifies getting profile logs in and out of containers and other environments without persistent storage. For more information on Profile Log Service configuration, see "Using the Profile Log Service".

Introduced ReadyNow-only deployment to helm charts.

Multiple APIs failed with empty response.

Cache requests latency increased manifold resulting in an increase in wait time and overall compilation duration.

In case of heavy applications, if you see anomalies in TTCOB, the problem can be resolved by increasing the number of cache pods. For more info, see "Troubleshooting Cloud Native Compiler ".

Compiler Cache on by default.

New Time to Clear Optimization Backlog metric in Grafana dashboard.

Multiple pods can get evicted because of low ephemeral storage in a long-running Code Cache cluster.

Early access of the Compiler Cache. The Compiler Cache stores previously performed optimizations and serves them from the cache rather than recompiling whenever possible. Running your workloads with a Compiler Cache leads to lower CNC CPU usage and faster warmup time.

Compiler Cache is not scalable and too many connections overload the database.

Multiple pods can get evicted because of low ephemeral storage in a long-running Code Cache cluster.

Simplified installation and configuration with Helm charts.

ZVM-23070 - Using Cloud Native Compiler with local ReadyNow can dramatically increase the CPU required to deliver the compilations in time. Monitor your compiler output and look for connections being rejected and the JVM switching to local compilation, and scale out your CNC instance accordingly.

Fallback to local JIT compilation when Cloud Native Compiler is unreachable or underperforming.

You can now provide an existing ReadyNow profile as the input of the -XX:ProfileLogIn={file} flag. Note that generating a ReadyNow profile using the -XX:ProfileLogOut={file} is not supported with Cloud Native Compiler yet.

Built-in monitoring stack with Prometheus and Grafana.

JDK 17 support.

The CNC gateway is currently configured with one instance. Do not attempt to increase the number of gateway instances.

Extremely slow disk I/o configurations (with latencies in the multiple seconds) can lead to internal crashes and data loss within CNC (due to Artemis crashes). Avoid configuring CNC with pods using very slow HDD or network volumes.

Cloud Native Compiler server able to provide JIT compilations to Azul Zing Builds of OpenJDK 12.09.1.0 and later.

Configuration files to provision an AWS Elastic Kubernetes Service cluster for your CNC server.

A sample Grafana dashboard for monitoring your CNC server.