Skip to content

UCM Trace Mode

UCM Trace Mode is a lightweight diagnostic and evaluation mode that records per-request traces during inference without performing any actual KV cache dump/load operations. Trace Mode lets you collect real request traffic data and simulate the theoretical KV cache hit rate UCM could deliver before committing to a full UCM storage rollout.

It is recommended to first collect hit ratio statistics with Trace Mode and confirm with relevant project members whether to adopt UCM.

Overview

Enable Trace Mode with use_lite: true. UCM chooses the Lite connector from the model's KV cache layout: UCMLiteConnector for standard models, UCMHLALiteConnector for hybrid linear-attention models, and UCMFAWALiteConnector for FAWA models.

Option Default Description
use_lite false Switches to the UCM Lite Connector, which works with a Fake Store that skips all actual KV dump/load operations.

This mode records request and block metadata without real KV I/O or avoided inference compute. Analysis estimates reuse opportunities; measure actual storage performance separately.

Logged Trace Format

At startup the Lite connector writes a UCMTraceMeta: line with the model type, block dimensions and cache topology. Keep this line and the vLLM startup logs together with the request traces; the analyzer uses them to select the simulation engine and determine cache capacity.

UCMTraceMeta: type=standard, is_mla=False, vllm_hash_block_size=128, trace_hash_block_size=512, hbm_block_data_size=<bytes>
UCMTrace: timestamp: 1234567.890123, request_id: req-42, input_length: 8192, output_length: 128, block_hashes: ['a1b2...', 'c3d4...', ...]

The topology fields differ for standard, mamba and fawa models. The request line records:

Field Description
timestamp Lookup time from time.perf_counter(), used to preserve request order.
request_id The vLLM request identifier.
input_length Input token count (request.num_tokens).
output_length Maximum output token count (request.max_tokens).
block_hashes Hex-encoded block hashes at the granularity reported by trace_hash_block_size.

The standard Lite connector records requests once they contain at least one complete trace block. Timing diagnostics may appear on a separate log line.

Configuration

You can start from the sample file at unified-cache-management/examples/ucm_config_example.yaml

Enable Trace Mode with:

use_lite: true

Log Configuration (Optional)

A long request can produce a large list of block hashes. Tune the following environment variables before launching the service:

Environment Variable Default Description
UCM_LOG_PATH log Directory for per-process log files (e.g. ucm-<pid>.log).
UCM_LOG_MAX_FILES 10 Maximum number of rotated log files kept per process.
UCM_LOG_MAX_SIZE 5 Maximum size in MiB per log file before rotation. Increase this significantly when recording traces for long requests.
UCM_LOG_LEVEL info Log level. Traces are emitted at INFO level, so keep this at info (or lower for extra debug output).

Example for a trace-collection run:

export UCM_LOG_PATH=/workspace/ucm-trace-logs
export UCM_LOG_MAX_SIZE=256      # 256 MiB per file
export UCM_LOG_MAX_FILES=50     # keep up to 50 rotated files per process
export UCM_LOG_LEVEL=info

Launching the Inference Service

Trace Mode is deployed as an OpenAI-compatible vLLM server. Start it the same way as a normal UCM deployment — the only difference is the UCM config file contents.

Take the Qwen/Qwen2.5-14B-Instruct model as an example:

export ENABLE_UCM_PATCH=1
vllm serve Qwen/Qwen2.5-14B-Instruct \
  --max-model-len 32000 \
  --tensor-parallel-size 2 \
  --gpu_memory_utilization 0.87 \
  --block_size 128 \
  --trust-remote-code \
  --port 7800 \
  --enforce-eager \
  --no-enable-prefix-caching \
  --kv-transfer-config \
  '{
      "kv_connector": "UCMConnector",
      "kv_role": "kv_both",
      "kv_connector_module_path": "ucm.integration.vllm.ucm_connector",
      "kv_connector_extra_config": {"UCM_CONFIG_FILE": "/workspace/unified-cache-management/examples/ucm_config_example.yaml"}
  }'

Replace the UCM_CONFIG_FILE path with the actual path to your trace-mode config file on your machine.

For a standard model, the following startup log confirms that the Lite connector is active:

[UC][I] Init UCMLiteConnector.

You can now send production-equivalent traffic to the server. Every request will produce a trace record in the UCM log directory. No KV cache is dumped or loaded.

Trace Analysis

After collecting traces, run benchmarks/auto_trace_analysis.py to estimate KV cache hit rates. The script reads UCMTraceMeta: and the vLLM startup information to select the standard, mamba or fawa simulation engine and model HBM → DRAM → FS caching. Block sizes and model topology come from the logs.

python benchmarks/auto_trace_analysis.py \
  --log-dir /workspace/ucm-trace-logs \
  --dram-pool-size-gb <dram_gb> \
  --fs-pool-size-gb <fs_gb>

Required Arguments

Argument Description
--log-dir Directory scanned recursively for *.log, *.log.* and *.log.gz. Include the UCMTraceMeta: line and vLLM startup logs.
--dram-pool-size-gb Simulated DRAM pool capacity in GiB.
--fs-pool-size-gb Simulated filesystem pool capacity in GiB.

Optional Arguments

Argument Description
--service-url vLLM service address. Fetches /metrics to compare against the service's actual prefix-cache hit rate.
--num-nodes Physical node count; defaults to 1.
--unified-memory-pool Simulate a shared DRAM pool across nodes instead of separate per-node pools.
--output Write the analysis summary to a JSON file.
--trace-output Write the parsed trace records to a JSON Lines file.

The current analyzer no longer accepts --block-kv-cache-size or --is-mla. Logs without UCMTraceMeta: must be collected again with the matching Lite connector.

Analysis Report

The analysis produces a comprehensive report including:

  • Total request count and token count: Summary of the workload
  • Hit rate scenarios: Theoretical max, HBM only, HBM+DRAM, and HBM+DRAM+FS hit rates
  • Request lifetime: How long a request's blocks stay reusable (average, P90, P95)

These metrics help you evaluate whether UCM is suitable for your workload and how much capacity to allocate for each storage tier.