PagedAttention: Close-Reading a Real AI Infra Paper

The Passage#

Today’s reading comes from a paper almost every LLM-serving engineer has encountered: Efficient Memory Management for Large Language Model Serving with PagedAttention, by Woosuk Kwon et al., published at SOSP 2023. It is the paper behind vLLM, the inference engine now ubiquitous in the AI Infra world. Both excerpts are quoted from the real paper: the abstract is word-for-word, and the introduction passage has only its in-text citations removed.

Excerpt 1 — The abstract (verbatim):

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM’s source code is publicly available at https://github.com/vllm-project/vllm

Excerpt 2 — The opening of the introduction (citations removed):

The emergence of large language models (LLMs) like GPT and PaLM have enabled new applications such as programming assistants and universal chatbots that are starting to profoundly impact our work and daily routines. Many cloud companies are racing to provide these applications as hosted services. However, running these applications is very expensive, requiring a large number of hardware accelerators such as GPUs. According to recent estimates, processing an LLM request can be 10× more expensive than a traditional keyword query.

Key Vocabulary#

Idioms & Real Usage#

1. “existing systems struggle because…” — In academic English, struggle means “fail to cope”（难以应付）. Chinese learners typically write “existing systems have some problems…” or “can’t handle…”, which sounds flat and vague. Papers love struggle: “Naive approaches struggle to scale beyond…”, “Prior work struggles with long sequences.”

2. “When managed inefficiently, this memory can be significantly wasted…” — A participle clause with the subject dropped. Reading tip: when a sentence opens with a participle (when managed, given, assuming), supply “it is”: When [it is] managed inefficiently…. The implied subject must match the main clause’s subject — here, this memory.

3. “On top of it, we build vLLM” — On top of X = “built on the foundation of X”（在……的基础上）. Note the layering image: one thing is constructed upon another. A typical Chinese-English calque here is “based on this, we…” — grammatical but flat; the more idiomatic written form is “on top of which we build vLLM” (the authors simply started a new sentence instead).

4. “2-4× … compared to …” — Say it two to four times: “vLLM improves throughput by two to four times compared to FasterTransformer and Orca.” Two notes: (a) compared to and compared with both appear in papers; compared to is the common choice here; (b) keep the comparison target adjacent to compared to so the claim reads unambiguously.

5. “The improvement is more pronounced with longer sequences” — pronounced（显著的、明显的）is the refined word for how strong an effect is, preferred over obvious or big in academic prose: “The gap is most pronounced under high load.” Structure to copy: X is more pronounced with Y = 在 Y 情况下 X 更明显.

6. “such as FasterTransformer and Orca” — The written-register way to give examples. In conversation you would say like (“systems like FasterTransformer”); such as is formal prose. e.g. belongs only inside parentheses: “(e.g., FasterTransformer)”.

7. A grammar slip that survived peer review — “The emergence of large language models (LLMs) like GPT and PaLM have enabled…” should technically be has enabled, because the subject is the emergence (singular). The paper was published as-is at SOSP 2023. Takeaway: even top-tier papers are written by humans; a slip like this should not distract you from the content — though in a CET-6 grammar exam, write has.

8. “processing an LLM request can be 10× more expensive than a traditional keyword query” — A pattern worth copying: X can be N times more expensive than Y（X 比 Y 贵 N 倍）. Chinese speakers often over-engineer this (“compared to Y, the cost of X is N times”), when the simple subject-verb-complement form is what native writers actually use.

How to Read Any ML Abstract#

This abstract is a model of the four-move structure, one move per sentence group:

1. Sentence 1 — requirement: what good serving demands (batching sufficiently many requests).
2. Sentences 2–3 — problem: why existing systems fail (KV cache memory is dynamic and gets wasted).
3. Sentence 4 — method: the key idea (PagedAttention), flagged by “we propose”.
4. Sentences 5–6 — results: the numbers first (2-4×), then the conditions where the gains grow (more pronounced with…).

Almost every ML abstract you read follows this arc: Context → Problem → Method → Results. When skimming a new paper, draw that line through the abstract first — within thirty seconds you will know whether the paper is worth your time.

Source#

• W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023. arXiv:2309.06180 — https://arxiv.org/abs/2309.06180
• vLLM project — https://github.com/vllm-project/vllm