Unified Cache Manager¶
Unified Cache Manager (UCM) is a KV Cache management and reuse system for LLM inference. It persists reusable KV Cache to external storage and shares it across requests and compatible inference instances, reducing repeated Prefill computation.
In multi-turn conversations, long-context reasoning, coding and multi-agent collaboration, project code, task instructions and conversation history often recur across successive requests. UCM reuses the KV Cache already generated for that context, reducing computation for repeated prefixes and lowering response latency.
Compared with KV Cache kept only within an inference process, UCM uses external storage to expand cache capacity. It supports reuse across requests, persistence beyond process lifetimes and sharing between compatible instances, so existing computation results remain reusable.
When integrated with vLLM, UCM achieves a 3–10x reduction in inference latency in scenarios with substantial prefix repetition, including multi-turn conversations and long-context reasoning.
Start with your task¶
Quickstart Choose artifacts, configure your engine and run your first service with UCM.
Start with a model Find official model guides and UCM examples for deployment and API calls.
Operate and measure¶
Operations explains how to check cache reuse, backend health and time spent in each stage. The toolkit provides deployment checks, storage tests and metric inspection; use the KV Cache calculator for capacity estimates.
Supported integrations¶
UCM provides integration paths for vLLM, vLLM-Ascend and SGLang. Check the support matrix for model and platform coverage and installation for released engine/backend combinations.