LlamaIndex OCRia-1 Inference: What the 10x Speedup Measures

"Nearly 10× faster on a single B200" is the headline of LlamaIndex's new write-up on OCRia-1 inference, meaning how it serves OCRia-1, its 4-billion-parameter document-parsing model. It's a fair headline for a hard engineering job. It also describes one request running alone on the GPU, and that's not the number a team paying for a GPU cares about most.

LlamaIndex's post on X, Nov. 3, 2026, about making OCRia-1 nearly 10x faster on one B200 (@llama_index)

View on X

The post from @llama_index, dated Nov. 3, says: "We got OCRia-1 generating tokens nearly 10× faster on a single B200," and adds that "some improvements were only 2–5%." It points to the LlamaIndex blog post by Daniel Bustamante Ospina, which gives the figure as "~285 tokens/sec to ~2800 tokens/sec." We read the blog and the OCRia-1 technical report end to end, because the two documents don't quite use the same numbers.

Ten times faster at what?

The blog never says how many requests were in flight for the 285-to-2,800 comparison. It says only that the team focused on "generation speed across concurrent requests on a single GPU." Midway through, it reports "more than 1400 tokens per second per user at batch 8 to 12, whereas the base model was around 200."

The technical report is more exact. Its headline is 4,700 tokens per second with a single request, "about nine times faster than the base model on stock vLLM." Appendix E gives the protocol: greedy decoding, pages rendered at 4.1 megapixels, and a baseline of 321 to 333 tokens per second for the BF16 model on stock vLLM. The optimized stack reaches 4,734 tokens per second on decode alone, and about 3,000 end to end for the median request.

Our reading: the blog's ~2,800 matches the report's end-to-end figure of about 3,000, and ~285 sits just under the 321-to-333 baseline. The blog doesn't say so, so treat that as our inference, not LlamaIndex's statement. Either way, the 10× is a single-stream speed.

Speed per request falls as the GPU fills

The report's Table 20 shows what happens under load. We kept the columns LlamaIndex published and picked five of its six levels.

Our arithmetic: from one request to 64, each request slows to about 6% of its solo speed (302 against 4,734), while the GPU's total output rises about 3.3× and pages parsed per second rise about 4.7× (10.21 against 2.16). The report says 10.21 pages per second at 64 requests was "the highest concurrency tested with p90 latency below 10 s," and gives 9.6 s.

So the speedup is real, but a busy production queue gets about a 4.7× page throughput gain over a single request, not 10×. The report doesn't publish pages per second for the old baseline, so a like-for-like throughput gain over stock vLLM is Unconfirmed.

Seventeen small wins and one kernel

The blog's methods read like a profiling diary, and they're the most reusable part. LlamaIndex quantized to a mix of BF16, FP8 and NVFP4, leaving the vision encoder unquantized. It added a speculative-decoding drafter called DFlash2, which the blog says gets over 12 accepted tokens per step (the report gives 12.5 to 17.5 with 31 draft tokens, depending on the page set).

Then came the unglamorous work. It adapted ReplaySSM to the model's 24 Gated Delta Network layers, which the blog says is "to date still as a series of draft PRs to vLLM." A custom CUDA kernel cut one operation from 72–80 µs per layer to 23–24 µs, about 3.3× faster. Building batch metadata once instead of for 38 cache groups removed around 290 small GPU launches per step. The fusion of SiLU and quantization alone added 2.3–2.6%. Seventeen more tweaks, each worth 2–5%, "composed together" for "more than 60%" more throughput.

What carries over to your stack

Some pieces are portable: mixed-precision quantization checked against held-out data, speculative decoding, shared metadata, kernel fusion. LlamaIndex says it plans to contribute the general ones to vLLM. Others are welded to OCRia-1's architecture, such as the DFlash2 drafter trained on its own pages and the GDN kernels.

Two caveats are worth keeping. The blog calls quantization "effectively lossless on our target tasks," and the report says scores stayed "within measurement noise," which is a claim about LlamaIndex's benchmarks, not yours. And both documents use LlamaIndex's own measurements; we found no independent replication.

Four things the posts leave open

Is this faster parsing in LlamaParse today? The blog says "you can use OCRia-1 today on LlamaParse, powering all our tiers."

Is the speculative decoding lossy? The blog says verification makes it "100% lossless" in the output distribution.

Are the vLLM changes merged? Unconfirmed. The blog says only that it plans to contribute them back.

Are OCRia-1's weights downloadable? Unconfirmed. Neither document we read says.

Ask for pages per second

If you evaluate parsing vendors, the figure to request is pages per second at your own concurrency, with the p90 latency next to it. Here that's about 10 pages a second at 64 requests and a 9.6-second p90, on one B200, with LlamaIndex's production prompt. Then run your own scanned, multilingual or table-heavy pages through it, because the report's numbers come from its own page sets.

Source: LlamaIndex Blog, "The Inference Stack Behind OCRia"

llamaindex ocria 1 llamaparse inference nvidia b200

AI

About the author

AI Briefing Daily Editorial. The editorial desk of AI Briefing Daily, an independent publication covering artificial intelligence. Every story is researched against primary sources, checked by hand and reviewed by a person before publication.