Visit Azul.com Support

Server-Emitted OpenTelemetry Signals

When OpenTelemetry is enabled, the server automatically emits telemetry signals without requiring any application changes. Two instrumentation scopes are used, corresponding to two OpenTelemetry SDK instances:

  • payara-runtime — the runtime (server-wide) SDK. Emits JVM runtime metrics and, when enabled, processes all telemetry including application signals. Using the runtime SDK is generally advised; see enabling OpenTelemetry.

  • fish.payara.telemetry — the per-application SDK. Scoped to a single deployed application; emits spans and metrics for application-level subsystems (HTTP, JAX-RS, IIOP, Jakarta Concurrency, JAX-WS, MicroProfile Fault Tolerance). More limited than the runtime SDK — for example, it does not publish JVM runtime metrics.

For setup and configuration, see OpenTelemetry configuration.

Resource Attributes

Every SDK instance (both the runtime SDK and per-application SDKs) is populated with the following resource attributes by default. Attributes that are already provided by the OpenTelemetry auto-configuration (for example, through OTEL_RESOURCE_ATTRIBUTES) are not overwritten.

Attribute Type Description

payara.server.domain

string

Name of the Payara domain.

payara.server.instance

string

Name of the server instance within the domain.

payara.server.runtime_type

string

Runtime type of the instance, lower-cased (e.g. server, embedded, micro).

payara.version

string

Full Payara server version string.

service.instance.id

string

A random UUID generated each time the SDK is initialised. Uniquely identifies this instance startup.

jvm.vm.name

string

JVM implementation name as reported by RuntimeMXBean.getVmName().

jvm.vm.vendor

string

JVM vendor string as reported by RuntimeMXBean.getVmVendor().

jvm.vm.version

string

JVM version string as reported by RuntimeMXBean.getVmVersion().

service.name

string

Defaults to <domain>-<instance> for the server-wide (runtime) SDK, or to the deployed application name for per-application SDKs. Overridden when otel.service.name is explicitly configured or when a standard auto-discovered value is present.

The payara.subsystem Attribute

Spans emitted by Jakarta EE subsystem instrumentation carry a payara.subsystem string attribute that identifies the subsystem that created the span. This attribute provides a stable, low-cardinality way to filter or group spans by subsystem in a monitoring backend.

Value Subsystem

jakarta-rest

JAX-RS client outbound calls and JAX-RS server endpoint spans.

jakarta-concurrency

Jakarta Concurrency waiting spans (queue latency before a task starts).

jakarta-xml-ws

JAX-WS (SOAP) client outbound call spans.

Traces (Spans)

The following spans are created automatically. All spans belong to the fish.payara.telemetry instrumentation scope unless noted otherwise.

HTTP Server Span

Property Value

Span name

HTTP method at span start (e.g. GET), updated to the low-cardinality route at span end (e.g. GET /api/users/{id}). The route is derived from Jakarta REST resource and method annotations (and equivalent framework routing metadata) once routing is resolved.

SpanKind

SERVER

Trigger

Every inbound HTTP request processed by StandardWrapper. Nested forward/include dispatches reuse the existing span without creating a child.

Attribute Always present Description

http.request.method

yes

Normalised HTTP method. Unknown methods are mapped to HTTP.

url.scheme

yes

Request scheme (http or https).

url.path

yes

Full request URI path.

server.address

yes

Server host name from the request.

server.port

yes

Server port number.

url.query

if present

Raw query string, omitted when there is no query.

user_agent.original

if present

User-Agent request header value.

client.address

if present

Remote IP address of the client.

http.route

if resolved

Low-cardinality route template (e.g. /api/users/{id}). Set only after framework routing completes.

http.response.status_code

yes

HTTP response status code.

network.protocol.name

yes

Fixed value http.

error.type

status >= 500 only

String representation of the status code (e.g. "500"). Never emitted for successful responses.

Note
The error.type attribute is only recorded in the http.server.request.duration histogram, not on the span itself; the span status is set to ERROR instead.

SOAP Fault Enrichment

When a JAX-WS response contains a SOAP fault, the active SERVER span (started by the HTTP layer) is enriched with the following additional attributes. No separate span is created.

Attribute Description

soap.fault.code

Qualified SOAP fault code (e.g. SOAP-ENV:Server or env:Receiver for SOAP 1.2).

soap.fault.string

Human-readable fault description from the endpoint (also used as the span status description).

The span status is set to ERROR with soap.fault.string as the description.

JAX-RS Client Span

Property Value

Span name

HTTP method of the outbound request (e.g. POST).

SpanKind

CLIENT

Trigger

Every outbound JAX-RS client call when OpenTelemetry is enabled for the application.

Attribute Always present Description

url.full

yes

Full URI of the outbound request.

http.request.method

yes

HTTP method.

server.address

yes

Target host.

server.port

if not -1

Target port. Omitted when the URI does not specify a port.

payara.subsystem

yes

Fixed value jakarta-rest.

http.response.status_code

yes

HTTP response status code received.

Context is propagated to the downstream service via the configured OTel propagator (W3C Trace Context by default).

IIOP/CORBA Client Span

Property Value

Span name

CORBA operation name (e.g. findByPrimaryKey).

SpanKind

CLIENT

Trigger

Every outbound IIOP call from the ORB client-side interceptor, via OpenTelemetryIiopClientInterceptor.

Attribute Always present Description

rpc.system.name

yes

Fixed value corba.

rpc.method

yes

CORBA operation name, identical to the span name.

The active OTel context is propagated to the server side as a serialised W3C carrier inside an IIOP service context.

IIOP/CORBA Server Span

Property Value

Span name

CORBA operation name received from the client.

SpanKind

SERVER

Trigger

Remote inbound IIOP calls processed by OpenTelemetryIiopServerInterceptor. Co-located same-JVM EJB calls bypass PortableInterceptors entirely and do not produce this span. Set the JVM property com.sun.corba.ee.ORBAllowLocalOptimization=false to force full IIOP dispatch for local calls during testing.

Attribute Always present Description

rpc.system.name

yes

Fixed value corba.

rpc.method

yes

CORBA operation name.

Jakarta Concurrency Waiting Span

Property Value

Span name

concurrent/<pool> wait — where <pool> is the managed executor pool name.

SpanKind

INTERNAL

Trigger

Each time a task is picked up from the executor queue, immediately before the task body runs. The span starts at task submission time and ends when the worker thread picks up the task, recording queue latency.

The waiting span ends before the parent context is made current on the worker thread, so user-created spans remain direct children of the original parent span rather than of the waiting span.

Attribute Always present Description

payara.subsystem

yes

Fixed value jakarta-concurrency.

thread.id

yes

ID of the worker thread that picked up the task.

thread.name

yes

Name of the worker thread.

scheduling.wait_ms

yes

Queue latency in milliseconds (fractional), measured from task submission to pickup.

Note
When a contextual proxy is passivated and reactivated (for example when held by a @Stateful EJB), the transient timing fields are lost and the waiting span is silently skipped; only the OTel context is restored.

@WithSpan Span

Property Value

Span name

The value attribute of @WithSpan if non-empty; otherwise the canonical class name + method name (e.g. com.example.MyBean.doWork). For anonymous/synthetic classes the simple class name is used instead.

SpanKind

Value of the kind attribute on @WithSpan. Defaults to INTERNAL.

Trigger

CDI bean methods annotated with @io.opentelemetry.instrumentation.annotations.WithSpan when OpenTelemetry is enabled. JAX-RS and JAX-WS methods are excluded — those are handled by their respective container instrumentation.

Method parameters annotated with @io.opentelemetry.instrumentation.annotations.SpanAttribute are captured as span attributes.

To disable @WithSpan tracing for a specific method via MicroProfile Config, set:

 
com.example.MyBean/doWork/WithSpan/enabled=false

The config key format is [fully.qualified.ClassName]/[methodName]/WithSpan/enabled. Disabling by method name applies to all overloads sharing that name.

Metrics

HTTP Server Metric

Instrumentation scope: fish.payara.telemetry

Property Value

Name

http.server.request.duration

Instrument type

Histogram (double)

Unit

s (seconds)

Description

Duration of HTTP server request processing, measured from the outermost server entry point to response completion. Async requests are measured until the async context completes.

Explicit histogram bucket boundaries (seconds): 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0

Attribute Always present Description

http.request.method

yes

Normalised HTTP method.

url.scheme

yes

Request scheme (http or https).

http.response.status_code

yes

HTTP response status code.

http.route

if resolved

Low-cardinality route template. Omitted when routing could not determine a template.

error.type

status >= 500 only

String representation of the status code (e.g. "500"). Never emitted for non-error responses.

JVM Runtime Metrics

Instrumentation scope: payara-runtime

These metrics are collected at server startup using JVM MXBeans and are independent of any deployed application.

Metric name Instrument type Unit Attributes

jvm.memory.used

UpDownCounter (async)

By

jvm.memory.type (heap / non_heap), jvm.memory.pool.name

jvm.memory.committed

UpDownCounter (async)

By

jvm.memory.type, jvm.memory.pool.name

jvm.memory.limit

UpDownCounter (async)

By

jvm.memory.type, jvm.memory.pool.name

jvm.memory.init

UpDownCounter (async)

By

jvm.memory.type, jvm.memory.pool.name

jvm.memory.total.used

UpDownCounter (async)

By

jvm.memory.type (heap or non_heap)

jvm.memory.total.committed

UpDownCounter (async)

By

jvm.memory.type

jvm.memory.total.limit

UpDownCounter (async)

By

jvm.memory.type

jvm.memory.total.init

UpDownCounter (async)

By

jvm.memory.type

jvm.memory.used_after_last_gc

UpDownCounter (async)

By

jvm.memory.type, jvm.memory.pool.name

jvm.thread.count

UpDownCounter (async)

{thread}

jvm.thread.state (lowercase thread state name), jvm.thread.daemon (boolean)

jvm.class.loaded

Counter (async)

{class}

jvm.class.unloaded

Counter (async)

{class}

jvm.class.count

UpDownCounter (async)

{class}

jvm.cpu.time

Counter, double (async)

s

jvm.cpu.count

UpDownCounter (async)

{cpu}

jvm.cpu.recent_utilization

Gauge, double (async)

1

jvm.buffer.memory.limit

UpDownCounter (async)

By

jvm.buffer.pool.name

jvm.buffer.count

UpDownCounter (async)

{buffer}

jvm.buffer.pool.name

jvm.gc.duration

Histogram (double)

s

jvm.gc.name — garbage collector name (e.g. G1 Young Generation)

Note
jvm.cpu.time is sourced from OperatingSystemMXBean.getProcessCpuTime() (nanoseconds) and converted to seconds. jvm.gc.duration is sourced from GC notification events (milliseconds) and converted to seconds. Both units are mandated by the MicroProfile Telemetry specification and TCK; they are not UCUM s in the conventional sense.

MicroProfile Fault Tolerance Metrics

Instrumentation scope: the application’s active meter (same scope as other application signals).

When OpenTelemetry is enabled, MicroProfile Fault Tolerance metrics are emitted as OpenTelemetry instruments. The metric names and semantics are defined by the MicroProfile Fault Tolerance — Integration with Metrics and Telemetry specification.

The following Payara-specific details apply:

  • Each metric carries the mp.scope attribute (the MicroProfile Metrics registry scope name, e.g. base) in addition to its standard MicroProfile tags.

  • Duration histograms (ft.timeout.executionDuration, ft.bulkhead.runningDuration, ft.bulkhead.waitingDuration) are recorded in seconds, as mandated by the MicroProfile Fault Tolerance specification. The unit string seconds differs from the UCUM s used by other OTel metrics.

  • ft.circuitbreaker.state.total is emitted as a monotonic counter with unit nanoseconds, recording cumulative nanoseconds spent in each circuit-breaker state (open, halfOpen, closed).