🌟 PipelineStore

PipelineStore is a composite store built by chaining multiple Store implementations together to form a data transfer pipeline.

Currently, the pipeline supports a chain composed of Cache Store and Posix Store.

In this chained pipeline:

• Cache Store handles data transfer between the Device and Host.

• Once the data flows from the Device to the Host, Posix Store is responsible for transferring the data between the Host and POSIX-compliant persistent storage, such as local disks, SSDs, or remote NFS (including NFS over RDMA) mount points.

At present, only this Store chain is supported.
Additional Store implementations will be developed in the future and chained into the pipeline to enable more flexible and extensible transfer paths.

Performance

Overview

The following are the multi-concurrency performance test results of UCM in the Prefix Cache scenario under a CUDA environment, showing the performance improvements of UCM. During the tests, HBM cache was disabled, and KV Cache was retrieved and matched only from SSD.

Here, Full Compute refers to pure VLLM inference, while SSD80% indicates that after UCM pooling, the SSD hit rate of the KV cache is 80%.

The following table shows the results on the QwQ-32B model(4 x H100 GPUs):

QwQ-32B
Input length	Concurrent	Full Compute (ms)	SSD80% (ms)	Speedup (%)
4 000	1	223.05	156.54	+42.5%
8 000	1	350.47	228.27	+53.5%
16 000	1	708.94	349.17	+103.0%
32 000	1	1512.04	635.18	+138.0%
4 000	8	908.52	625.92	+45.1%
8 000	8	1578.72	955.25	+65.3%
16 000	8	3139.03	1647.72	+90.5%
32 000	8	6735.25	3025.23	+122.6%
4 000	16	1509.79	919.53	+64.2%
8 000	16	2602.34	1480.30	+75.8%
16 000	16	5732.49	2393.54	+139.5%
32 000	16	11891.61	4790.00	+148.3%

The following table shows the results on the DeepSeek-R1-awq model (8 × H100 GPUs):

DeepSeek-R1-awq
Input length	Concurrent	Full Compute (ms)	SSD80% (ms)	Speedup (%)
4 000	1	429.30	261.34	+64.3%
8 000	1	762.23	363.37	+109.8%
16 000	1	1426.06	586.17	+143.3%
32 000	1	3086.85	1073.25	+187.6%
4 000	8	1823.55	1017.72	+79.2%
8 000	8	3214.76	1511.16	+112.7%
16 000	8	6417.81	2596.70	+147.2%
32 000	8	14278.00	5111.67	+179.3%
4 000	16	3205.22	1534.00	+108.9%
8 000	16	5813.09	2208.60	+163.2%
16 000	16	11752.48	4000.46	+193.8%
32 000	16	38643.73	19910.41	+94.1%

Configuration for Prefix Caching

Modify the UCM configuration file to specify which UCM connector to use and where KV blocks should be stored.
You may directly edit the example file at:

unified-cache-management/examples/ucm_config_example.yaml

A minimal configuration looks like this:

ucm_connectors:
  - ucm_connector_name: "UcmPipelineStore"
    ucm_connector_config:
      store_pipeline: "Cache|Posix"
      storage_backends: "/mnt/test"
      io_direct: false
      # cache_buffer_capacity_gb: 256
      # posix_capacity_gb: 1024
      use_gdr: false
enable_event_sync: true
use_layerwise: true
enable_record_traces: false
use_lite: false
persist_token_threshold: 0

Required Parameters

• ucm_connector_name:
  Specifies UcmPipelineStore as the UCM connector.

• store_pipeline: “Cache|Posix”
  Specifies a pipeline built by chaining the Cache Store and the Posix Store.
  In this chained pipeline, the Cache Store handles data transfer between the Device and Host,
  and once the data reaches the Host, the Posix Store transfers it between the Host and POSIX-compliant persistent storage.

  The pipeline must be registered in advance in
  unified-cache-management/ucm/store/pipeline/connector.py under PIPELINE_REGISTRY.

  Currently, only this Store chain is supported.

• storage_backends:
  Directory used for storing KV blocks. Can be a local path or an NFS-mounted path.
  ⚠️ Replace "/mnt/test" with your actual storage directory.

Optional Parameters

Parameters inside ucm_connector_config:

• io_direct (optional, default: false):
  Whether to enable direct I/O. For Posix Store, this bypasses the OS page cache when reading/writing files to disk.

• cache_buffer_capacity_gb (optional, default: 256)
  The capacity of the Cache Store host-side buffer in GB.
  The default value of 256 GB is sufficient in most cases.
  When using models like MLA, make sure /dev/shm has enough space,
  or set this parameter to a smaller value.
  Note: when multiple data-parallel instances run on the same node,
  each creates its own Cache Store buffer.

• posix_capacity_gb (optional, default: 0)
  The maximum storage capacity in GB for the Posix Store.
  When set to a value greater than 0, garbage collection (GC) is enabled
  and will recycle disk space when the threshold is reached.
  When set to 0 (default), no capacity limit or GC is applied.

• posix_io_engine (optional, default: “psync”)
  I/O engine type for the Posix Store. Supported values: "psync" (pread/pwrite), "aio" (libaio).

• use_gdr (optional, default: false)
  Enable GPUDirect RDMA transfers for Cache Store/Pc Store H2D and D2H copies.
  This requires an RDMA-capable NIC, a working GPUDirect RDMA environment,
  and building UCM with ENABLE_GDR=1.

• waiting_queue_depth (optional, default: 8192)
  Depth of the waiting queue for transfer tasks in the Cache Store.

• running_queue_depth (optional, default: 524288)
  Depth of the running queue for transfer tasks in the Cache Store.

• timeout_ms (optional, default: 30000)
  Timeout in milliseconds for external interfaces (applies to both Cache Store and Posix Store).

Top-level parameters (outside ucm_connector_config):

• enable_event_sync (optional, default: true)
  Whether to enable event synchronization.
  When using UcmNfsStore, this should be set to false.

• use_layerwise (optional, default: true)
  Whether to use layerwise loading/saving of KV cache blocks.
  Enabled by default for UCMConnector.

• hit_ratio (optional)
  When set, limits the number of hit tokens to hit_ratio × num_prompt_tokens.
  Useful for controlling the proportion of cached tokens that are actually loaded.

• enable_record_traces (optional, default: false)
  Whether to record request traces.
  When enabled, logs per-request traces (timestamp, input_length, output_length, hash_ids).
  Each hash_id takes 32 bytes. Enable only when needed,
  and configure UCM_LOG_MAX_FILES and UCM_LOG_MAX_SIZE accordingly.

• use_lite (optional, default: false)
  Whether to use the UCM Lite Connector.
  UCM Lite Connector works with Fake Store to skip actual KV dump/load operations,
  only collecting hit ratio statistics.
  It is suggested to first collect hit ratio statistics and confirm with relevant
  project members whether to adopt UCM.

• persist_token_threshold (optional, default: 0)
  Minimum token threshold for KV persistence.
  Requests with fewer tokens than this threshold will skip KV load and dump operations.

• metrics_config_path (optional)
  Path to a UCM metrics configuration YAML file.
  When set, enables UCM metrics that can be monitored online via Grafana and Prometheus.

Launching Inference

In this guide, we describe online inference using vLLM with the UCM connector, deployed as an OpenAI-compatible server. For best performance with UCM, it is recommended to set block_size to 128.

To start the vLLM server with the Qwen/Qwen2.5-14B-Instruct model, run:

vllm serve Qwen/Qwen2.5-14B-Instruct \
--max-model-len 20000 \
--tensor-parallel-size 2 \
--gpu_memory_utilization 0.87 \
--block_size 128 \
--trust-remote-code \
--port 7800 \
--enforce-eager \
--no-enable-prefix-caching \
--kv-transfer-config \
'{
    "kv_connector": "UCMConnector",
    "kv_role": "kv_both",
    "kv_connector_module_path": "ucm.integration.vllm.ucm_connector",
    "kv_connector_extra_config": {"UCM_CONFIG_FILE": "/vllm-workspace/unified-cache-management/examples/ucm_config_example.yaml"}
}'

You can also use the Layerwise Connector by adding "use_layerwise": true in the UCM_CONFIG_FILE. for example:

ucm_connectors:
  - ucm_connector_name: "UcmPipelineStore"
    ucm_connector_config:
      store_pipeline: "Cache|Posix"
      storage_backends: "/mnt/test"
      io_direct: false
      # cache_buffer_capacity_gb: 256
      # posix_capacity_gb: 1024
      use_gdr: false
enable_event_sync: true
use_layerwise: true
enable_record_traces: false
use_lite: false
persist_token_threshold: 0

⚠️ Make sure to replace "/vllm-workspace/unified-cache-management/examples/ucm_config_example.yaml" with your actual config file path.

If you see log as below:

INFO:     Started server process [1049932]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Congratulations, you have successfully started the vLLM server with UCM connector!

Evaluating UCM Prefix Caching Performance

After launching the vLLM server with UCMConnector enabled, the easiest way to observe the prefix caching effect is to run the built-in vllm bench CLI. Executing the following command twice in a separate terminal shows the improvement clearly.

vllm bench serve \
--backend vllm \
--model Qwen/Qwen2.5-14B-Instruct \
--host 127.0.0.1 \
--port 7800 \
--dataset-name random \
--num-prompts 12 \
--random-input-len 16000 \
--random-output-len 2 \
--request-rate inf \
--seed 123456 \
--percentile-metrics "ttft,tpot,itl,e2el" \
--metric-percentiles "90,99" \
--ignore-eos

After the first execution

The vllm bench terminal prints the benchmark result:

---------------Time to First Token----------------
Mean TTFT (ms):                           15001.64

Inspecting the vLLM server logs reveals entries like:

INFO ucm_connector.py:317: request_id: xxx, total_blocks_num: 125, hit hbm: 0, hit external: 0

This indicates that for the first inference request, UCM did not hit any cached KV blocks. As a result, the full 16K-token prefill must be computed, leading to a relatively large TTFT.

After the second execution

Running the same benchmark again produces:

---------------Time to First Token----------------
Mean TTFT (ms):                            2874.21

The vLLM server logs now contain similar entries:

INFO ucm_connector.py:317: request_id: xxx, total_blocks_num: 125, hit hbm: 0, hit external: 125

This indicates that during the second request, UCM successfully retrieved all 125 cached KV blocks from the storage backend. Leveraging the fully cached prefix significantly reduces the initial latency observed by the model, yielding an approximate 5× improvement in TTFT compared to the initial run.

Log Message Structure

If you want to view detailed transfer information, set the environment variable UC_LOGGER_LEVEL to debug.

You may see the following typical log messages in the logs.

[UC][D] Cache task({task_id},{operation},{subtask_number},{size}) dispatching. [PID,TID]

This log indicates that the Cache Store has received a load or dump task

Component	Description
task_id	Unique identifier for the Cache Ctore task
operation	DUMP: Dump to Host(Device → Host) LOAD: Load from Host (Host → Device)
subtask_number	Number of subtasks executed in this operation
size	Total size of data transferred in bytes (across all tasks)

[UC][D] Cache task({task_id},{operation},{subtask_number},{size}) finished, cost {time}ms. [PID,TID]

This log indicates that a load or dump task in the Cache Store has completed, along with its execution time in ms.

[UC][D] Posix task({task_id},{operation},{subtask_number},{size}) dispatching. [PID,TID]

This log indicates that the Posix Store has received a load or dump task

Component	Description
task_id	Unique identifier for the Posix Store task
operation	Cache2Backend: Dump data from Cache Store to Posix Store. Backend2Cache: Load data from Posix Store back to Cache Store.
subtask_number	Number of subtasks executed in this operation
size	Total size of data transferred in bytes (across all tasks)

[UC][D] Posix task({task_id},{operation},{subtask_number},{size}) finished, cost {time}ms. [PID,TID]

This log indicates that a load or dump task in the Posix Store has completed, along with its execution time in in ms.