The Linkerd proxy exposes metrics that describe the traffic flowing through the
proxy. The following metrics are available at
/metrics on the proxy’s metrics
:4191) in the Prometheus format.
request_total: A counter of the number of requests the proxy has received. This is incremented when the request stream begins.
response_total: A counter of the number of responses the proxy has received. This is incremented when the response stream ends.
response_latency_ms: A histogram of response latencies. This measurement reflects the time-to-first-byte (TTFB) by recording the elapsed time between the proxy processing a request’s headers and the first data frame of the response. If a response does not include any data, the end-of-stream event is used. The TTFB measurement is used so that Linkerd accurately reflects application behavior when a server provides response headers immediately but is slow to begin serving the response body.
route_response_total: These metrics are analogous to
response_totalexcept that they are collected at the route level. This means that they do not have
grpc_status_codeor any outbound labels but instead they have:
dst: The authority of this request.
rt_route: The name of the route for this request.
control_response_total: These metrics are analogous to
response_totalbut for requests that the proxy makes to the Linkerd control plane. Instead of
direction, or any outbound labels, instead they have:
addr: The address used to connect to the control plane.
Note that latency measurements are not exported to Prometheus until the stream completes. This is necessary so that latencies can be labeled with the appropriate response classification.
Each of these metrics has the following labels:
authority: The value of the
Host(HTTP/1.1) header of the request.
inboundif the request originated from outside of the pod,
outboundif the request originated from inside of the pod.
trueif the request’s connection was secured with TLS.
The following labels are only applicable on
successif the response was successful, or
failureif a server error occurred. This classification is based on the gRPC status code if one is present, and on the HTTP status code otherwise. Only applicable to response metrics.
grpc_status_code: The value of the
grpc-statustrailer. Only applicable for gRPC responses.
status_code: The HTTP status code of the response.
The following labels are only applicable if
dst_deployment: The deployment to which this request is being sent.
dst_k8s_job: The job to which this request is being sent.
dst_replicaset: The replica set to which this request is being sent.
dst_daemonset: The daemon set to which this request is being sent.
dst_statefulset: The stateful set to which this request is being sent.
dst_replicationcontroller: The replication controller to which this request is being sent.
dst_namespace: The namespace to which this request is being sent.
dst_service: The service to which this request is being sent.
dst_pod_template_hash: The pod-template-hash of the pod to which this request is being sent. This label selector roughly approximates a pod’s
Prometheus Collector labels
The following labels are added by the Prometheus collector.
instance: ip:port of the pod.
job: The Prometheus job responsible for the collection, typically
Kubernetes labels added at collection time
Kubernetes namespace, pod name, and all labels are mapped to corresponding Prometheus labels.
namespace: Kubernetes namespace that the pod belongs to.
pod: Kubernetes pod name.
pod_template_hash: Corresponds to the pod-template-hash Kubernetes label. This value changes during redeploys and rolling restarts. This label selector roughly approximates a pod’s
Linkerd labels added at collection time
Kubernetes labels prefixed with
linkerd.io/ are added to your application at
linkerd inject time. More specifically, Kubernetes labels prefixed with
linkerd.io/proxy-* will correspond to these Prometheus labels:
daemonset: The daemon set that the pod belongs to (if applicable).
deployment: The deployment that the pod belongs to (if applicable).
k8s_job: The job that the pod belongs to (if applicable).
replicaset: The replica set that the pod belongs to (if applicable).
replicationcontroller: The replication controller that the pod belongs to (if applicable).
statefulset: The stateful set that the pod belongs to (if applicable).
Here’s a concrete example, given the following pod snippet:
The resulting Prometheus labels will look like this:
The following metrics are collected at the level of the underlying transport layer.
tcp_open_total: A counter of the total number of opened transport connections.
tcp_close_total: A counter of the total number of transport connections which have closed.
tcp_open_connections: A gauge of the number of transport connections currently open.
tcp_write_bytes_total: A counter of the total number of sent bytes. This is updated when the connection closes.
tcp_read_bytes_total: A counter of the total number of received bytes. This is updated when the connection closes.
tcp_connection_duration_ms: A histogram of the duration of the lifetime of a connection, in milliseconds. This is updated when the connection closes.
Each of these metrics has the following labels:
inboundif the connection was established either from outside the pod to the proxy, or from the proxy to the application,
outboundif the connection was established either from the application to the proxy, or from the proxy to outside the pod.
srcif the connection was accepted by the proxy from the source,
dstif the connection was opened by the proxy to the destination.
Note that the labels described above under the heading “Prometheus Collector labels” are also added to transport-level metrics, when applicable.
Connection Close Labels
The following labels are added only to metrics which are updated when a
connection closes (
successif the connection terminated cleanly,
failureif the connection closed due to a connection failure.
identity_cert_expiration_timestamp_seconds: A gauge of the time when the proxy’s current mTLS identity certificate will expire (in seconds since the UNIX epoch).
identity_cert_refresh_count: A counter of the total number of times the proxy’s mTLS identity certificate has been refreshed by the Identity service.