Metrics development¶

Define the operation, unit and aggregation scope before adding a metric at the point where that operation occurs. A shared Dispatcher distributes native UCM statistics; the default path exports them through vLLM's KV Connector to the engine's /metrics endpoint.

Default collection path¶

UCMConnector._setup_ucm_metrics() reads launch configuration. Without a custom catalog it uses built-in definitions. Inline metrics_config or metrics_config_path supplies a complete replacement.
setup_ucm_metrics() registers native Counters, Gauges and Histograms from definitions, including configured bucket boundaries.
Python or C++ updates the native statistics where operations occur.
MetricsDispatcher drains native increments into separate buffers for enabled consumers, preventing consumers from consuming one another's data.
When vLLM calls get_kv_connector_stats(), UCM retrieves the vllm_connector buffer and passes Connector statistics to vLLM's Prometheus bridge.

Default labels include engine labels and worker_rank. The optional multiproc consumer uses PrometheusStatsLogger, a periodic logger and worker_id. This is a separate path, not the default Connector registration/export process.

Define a metric¶

Put runtime custom metrics in the deployed metrics YAML. For non-KV additions to the default set, update both examples/metrics/metrics_configs.yaml and ucm/default_metrics_config.py. The existing test_default_metrics_config_matches_example_yaml checks their consistency. For KV metrics, use the generation workflow below.

This fragment illustrates definitions and belongs inside a complete metrics configuration:

counter:
  - name: "my_events_total"
    documentation: "Total number of events"

gauge:
  - name: "my_current_value"
    documentation: "Most recent value"
    multiprocess_mode: "livemostrecent"

  # A gauge split by a runtime dimension reports one Prometheus series per
  # dimension value, exposed as a real label.
  - name: "my_pool_usage_ratio"
    documentation: "Used-slot ratio per block size, exported as my_pool_usage_ratio{slot_size=<size>}"
    multiprocess_mode: "livemostrecent"
    dynamic_labels: ["slot_size"]

histogram:
  - name: "my_stage_duration_ms"
    documentation: "Stage duration in milliseconds"
    buckets: [0.1, 0.5, 1, 2, 5, 10, 20, 50, 100]

Counters take positive increments, Gauges take current values and Histograms take observations. Configure buckets in ascending order; registration adds +Inf when needed. Keep a fixed event scope and unit: interface calls, transfer shards and user requests are different quantities.

dynamic_labels declares a dimension whose value is not known at registration time. Because UCM's transport is name-keyed, the emitter reports each dimension value under the wire name <name>_<value> (my_pool_usage_ratio_4096), and both export paths split the numeric suffix back into a real Prometheus label (my_pool_usage_ratio{slot_size="4096"}). At most one label is supported and the suffix must be all digits. Metrics that do not declare the field are unaffected.

Generate and validate KV metrics¶

The source of KV definitions is kv_semantics/metrics/config/kv_metrics.yaml. Edit that file, then run these commands from the repository root:

python kv_semantics/metrics/tools/generate_kv_metrics.py --sync-ucm
python kv_semantics/metrics/tools/generate_kv_metrics.py --check

--sync-ucm updates kv_semantics/metrics/include/kv_metrics/default_metric_descriptors.h and the BEGIN/END GENERATED KV blocks in both examples/metrics/metrics_configs.yaml and ucm/default_metrics_config.py. Commit the source and all three outputs together; do not edit the generated blocks directly. --check is read-only and exits nonzero if any output is stale.

When adding, removing, or changing the type of a KV metric, also update the KV table and count summary in the English and Chinese metrics-reference.md files and docs/source/user-guide/metrics/metrics_list.md. --check rejects missing, duplicate, obsolete, or incorrectly grouped KV rows and stale summaries. The legacy reference's overall catalog counts are checked against the complete Python defaults, including non-KV metrics. Translated descriptions and units remain manually reviewed; --sync-ucm does not rewrite documentation. Update these documents before running the final --check.

For a deployment with a custom metrics catalog, synchronize its KV definitions and validate the catalog separately (requires PyYAML):

python kv_semantics/metrics/tools/generate_kv_metrics.py --ucm-config /etc/ucm/metrics_configs.yaml

Pass the metrics catalog, not the top-level serving configuration. This read-only check requires every KV definition exactly once, with matching type, documentation text and histogram buckets. Additional custom metrics are allowed. Documentation changes must therefore also be copied into deployment catalogs. --ucm-config cannot be combined with --check or --sync-ucm; run separate commands. Write buckets as unquoted decimal numbers such as 0.00001: YAML 1.1 may parse 1e-5 as a string.

KV duration histograms use seconds. Preserve that unit in observations and buckets; multiply a displayed mean or quantile by 1000 to show milliseconds. The standalone node-task histograms retain node_id; the UCM adapter aggregates these observations across nodes. See ASU / KV metrics for the catalog, store-level backend installation switch, and independent UCM initialization/export controls.

Update at the operation boundary¶

The caller measures cost_ms in this Python example:

from ucm.shared.metrics import ucmmetrics

ucmmetrics.update_stats("my_stage_duration_ms", cost_ms)
ucmmetrics.update_stats({"my_events_total": 1.0})

Link the C++ target to metrics and use the API with UCM include paths configured:

target_link_libraries(xxxstore PUBLIC storeintf metrics)

#include "metrics_api.h"

UC::Metrics::UpdateStats(NAME_TO_METRIC_ID("my_stage_duration_ms"), costMs);
UC::Metrics::UpdateStats(NAME_TO_METRIC_ID("my_events_total"), 1.0);

NAME_TO_METRIC_ID caches the ID for hot paths. Producers do not need to create/register metrics again and must not independently drain global statistics around the Dispatcher.

Update a successful-completion metric only on success. Name submission metrics accordingly. Explain whether durations or bytes include errors and cancellations in the definition or metric reference.

Load the metric configuration¶

Copy the complete metrics catalog, add the definitions above, and point the serving configuration to it. The fragment below belongs at the root of the existing UCM YAML; it does not replace the Store configuration. Restart the engine to register the updated catalog before producing values.

enable_metrics: true
metrics_config_path: /etc/ucm/metrics_configs.yaml

Verify export and presentation¶

Load a configuration containing the definition, trigger the operation and check the actual exported name, unit, labels and delta. The default prefix is ucm:; consumer configuration can change names and scales. A metric may be absent or unchanged when its operation has not run or Connector statistics have not been collected.

For example, query the Counter rate:

rate(ucm:my_events_total[5m])

Use buckets for histogram quantiles and preserve le when aggregating. See metric semantics for other queries. For dashboards, update the appropriate grafana_connector.json, grafana_store.json or grafana_vllm.json, retaining model, instance and worker selectors.

Read ucm/metrics_config.py, ucm/metrics_dispatcher.py, ucm/integration/vllm/ucm_connector.py and ucm/integration/vllm/metrics.py in order. Collection setup belongs in the user guide.

Dashboard query examples¶

The following use the Grafana model and interval variables from the original guide. For per-worker views, retain the dashboard's worker grouping in addition to le.

Counter throughput：

rate(ucm:my_events_total{model_name="$model_name"}[$__rate_interval])

Histogram average：

sum(rate(ucm:my_stage_duration_ms_sum{model_name="$model_name"}[$__rate_interval]))
/
sum(rate(ucm:my_stage_duration_ms_count{model_name="$model_name"}[$__rate_interval]))

Histogram quantile：

histogram_quantile(
  0.99,
  sum by (le) (
    rate(ucm:my_stage_duration_ms_bucket{model_name="$model_name"}[$__rate_interval])
  )
)