Background

As a test, I ran vLLM with the following configuration,

vllm serve Qwen/Qwen3-0.6B \
  --host 127.0.0.1 --port 8000 \
  --dtype bfloat16 --tensor-parallel-size 1 \
  --max-model-len 2048 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.94 \
  --served-model-name Qwen/Qwen3-0.6B

vLLM processed more output tokens per second as I increased concurrency on the same GPU.

Problem

A question I have been trying to answer: what if we wanted to run more than one model? To be fair, I don't have any workflows that require me to do so, but the question I am trying to answer is: can we run more than one instance, let's say, for different use cases? An example that comes to mind: companies are looking into fine-tuning specialized models on top of small general-purpose models. Now, running one model might not be sufficient since there could be other specialized models.

The solution to this problem is simple: let's run two vLLM instances, each pretending to service a specific workload by serving a specialized model. I don't have a fine-tuned specialized model, but we are pretending here.

Experiment 1

So let's run two vLLM instances, same model, same config, except half the memory available. So --gpu-memory-utilization is set to 0.47 for both instances. Both the single- and dual-engine runs used max-num-seqs 128, with prefix caching disabled.

To test, we run the same concurrency sweep benchmark.

Interesting result. The throughput didn't scale as expected. Instead, it became quite worse. I checked with Astra why that could be. No definitive answer as to why. So let's try to find where the slowness could be coming from. We have the following candidates,

• GPU contention, two inference engines with active workloads maybe contending with each other. Which is fair, but that's a huge efficiency drop.

• CPU contention, maybe workload scheduling by vLLM itself is contending for CPU cycles.

• Memory bandwidth: my laptop GPU is just not cut for this type of workload. Which is not so great when thinking about local LLM applications in the future.

• Something else,

Experiment 2

Astra mentioned NVIDIA MPS (Multi-Process Service), which lets CUDA kernels from separate processes run concurrently on the GPU. That can improve utilization, although the two vLLM engines still schedule their own batches.

So we enabled MPS and ran our concurrency sweep benchmark test again,

Definitely better results, but the gap between single-instance and multi-instance still exists.

Experiment 3

Ok so I wanted to understand where this throughput drop is coming from. I asked Astra to run the experiment again and capture some more metrics,

Important discovery: Larger decode batches moved fewer DRAM bytes per token in our isolated probes. That supports batching efficiency as one contributor to the single-engine advantage.

Now I am wondering if these regressions show up in production environments of inference providers. I can't know that because I don't work at such companies yet, and I don't have a datacenter-grade GPU to test this on. Anyway, on to our laptop GPU.

Experiment 4

Alright, I want to try to improve utilization beyond what is achievable here. I am not thinking about model quality because my workload is mostly benchmark traffic. And keeping a real production fleet in mind, with recent datacenter-grade hardware, I do not want to be able to serve only one model, especially when specialized models can be great for automation. Also, what about running multimodal inference? These days, a single general-purpose model can do multimodal inference, but I want the ability to run different types of models.

I think there is enough cause here that we can further our experiments. I decided to test a different multi-instance setup.

• One instance runs Qwen3-0.6B with BF16 weights and KV cache, reserving 1.5 GiB for KV cache.

• Another runs Whisper Small with FP16 weights and KV cache, reserving 1 GiB for KV cache.

We started by capturing single-engine throughput and latencies of both models. And then captured them running as a dual-engine multimodal setup. The goal is to understand if the regression surfaces in this setup also.

Each service lost some of its solo speed. However, their combined normalized progress exceeded a time-split reference based on the measured solo rates, so co-location still delivered useful aggregate work.

Experiment 5

Going back to finding where the slowness came from. I instructed Astra to profile GPU utilization; it used tools like Nsight Systems and Nsight Compute, and CPU scheduling delay using the bcc tool, runqlat and capturing other important metrics,

CPU run-queue waits were small in the sampled Qwen threads. GPU graph spans increased while sharing, and isolated larger-batch probes used fewer DRAM bytes per token. Memory traffic is a credible contributor, but these measurements do not isolate it as the sole bottleneck.

Astra traces noted an important finding. Our isolated decode probes showed that larger batches and lower precision could reduce DRAM bytes read per decoded token,

• Batch 16 versus batch 32: approximately 116.41 → 89.00 MB/token.

• BF16 versus FP8 KV/attention: approximately 116.38 → 85.81 MB/token.

That provides two insights: First, a dual-instance setup with the same model duplicates model weights, which could have been utilized better in single-instance setups. Second, our 256GB/s memory bandwidth could be insufficient here for the amount of memory moving per token at any time.

The second insight is a good learning point; This made me appreciate why memory bandwidth matters so much for these workloads.

Experiment 6

Memory traffic looks like a strong lead, but we haven’t isolated the whole cause. And vLLM's efficiency is really good at this point for what I am testing. Is there anything we can do to split the resource usage in exact half so that there is no contention? So we tried a few things,

1. Does this problem surface with k8s-based deployments? I had little reason to suspect that it wouldn't. It surfaced in the same way. We tried the official NVIDIA controls for the same.

2. Can we do CPU-like pinning with GPUs? We tried nvtaskset, a research tool that uses undocumented NVIDIA controls to restrict GPU execution resources. Astra fixed a CUDA 13 build incompatibility, and the tool worked, but the tested partitions didn’t improve Qwen throughput.

The only remaining exploration related to resource management was the use of MIG (Multi-Instance GPU). As per the documentation, it partitions not only GPU compute, but also its L2 caches and VRAM, to the best extent possible. The support spreadsheet says that B200 can be partitioned into 7 partitions. Unfortunately, MIG is not supported on my GPU, so I can't do much about it.

Experiment 7

What if we brought in an admission controller to manage incoming requests across two engine instances using shared request limits and queues? Astra came up with a bunch of options; we tried a RayServe-based simple controller. Our particular admission policy didn’t close the performance gap, so I stopped the experiment.

Experiment 8

Going back to understanding the contention more. I asked Astra to review the resource usage metrics and traces we had captured and do another pass at inferring what the cause of the regression we could be seeing was, apart from the batching-efficiency evidence we had already collected. We came back to the contention point very likely being the amount of memory transferred per token. Duplicated weights, yes, they are a factor, but we have plenty of memory. The earlier controlled tests had no KV-cache preemptions. The final sweep did: dual FP16 at total concurrency 64 recorded 160 preemptions across two repeats. So now we have one strong lead, and it could possibly be solved by just faster hardware, like HBM.

To test the memory-traffic idea, we compared BF16 with FP8 KV cache and attention operands, for both single and dual engines, at total concurrency 32.

• One engine, BF16 weights, BF16 KV and attention operands.

• One engine, BF16 weights, FP8 KV and attention operands.

• Two engines with MPS, BF16 weights, BF16 KV and attention operands.

• Two engines with MPS, BF16 weights, FP8 KV and attention operands.

Voila! Both topologies improved with FP8. Dual FP8 still trailed single BF16 in this first test. Isolated decode-graph reads dropped to about 86 MB/token.

Let's test with a multi-modal setup (MPS ON, concurrency 8 per model, 16 total).

• Baseline: Qwen with BF16 weights and KV; Whisper with FP16 weights and KV.

• Candidate: Qwen unchanged; Whisper with FP16 weights and FP8 KV.

Hmmm. Slightly better but still an improvement.

I think this is a good path forward. The thing with precision tuning is that it can cause a decrease in the quality of answers. And that's totally understandable.

Experiment 9

Let’s measure how answer quality changes when we reduce the precision of KV cache and attention operands. I am new to all of this, so I asked Astra to find the most common benchmarks and run them against a setup of FP16 and FP8 KV cache precisions. So, benchmarking quality with the following config,

1. Single-instance vLLM, BF16 weights, FP16 KV Cache and Attention operands, Qwen3-0.6B.

2. Single-instance vLLM, BF16 weights, FP8 KV Cache and Attention operands, Qwen3-0.6B.

The first Qwen FP8 pilot used uncalibrated scales. The later quality tests and final sweep used calibrated FP8. Keeping BF16 model weights while using literal FP16 attention required an experiment-local adapter.

There is some degradation and some improvements. Strong note though, this is not quality equivalence in the same sense, like its a small test. Some observations from the quality tests,

Notably, Instruction-following scores fell on our tested subset, while math and science scores rose. Useful information.

Experiment 10

Time to close the loop. Let's see the final result with the following configurations,

• One engine: BF16 weights, FP16 KV and attention, MPS OFF

• Two engines: BF16 weights, FP8 KV/attention engine, MPS ON

• Two engines: BF16 weights, FP16 KV and attention, MPS ON

So there it is: Dual FP8 achieved the highest throughput at the largest tested concurrency, but single FP16 retained the best latency-SLO goodput. The useful choice depends on the workload’s latency and quality requirements.

Conclusion

The quality loss could be acceptable depending on the use case. The tradeoff here is better utilization of expensive hardware, especially when planning to run multiple small specialist models. A well-balanced topology here could be one engine running at FP16 for KV cache and attention, and another at reduced precision. A smart router in front of them could send simple conversational traffic to the reduced-precision engine, whereas complicated requests could be routed to the higher-precision engine. We haven’t tested whether simple conversations actually tolerate the lower precision better.

Database systems are Retrieval engines, whereas Gen AI systems are Generative engines. Databases store and retrieve stateful data from persistent storage, with tunable access paths. On the other hand, Gen AI systems generate a response to a request based on probabilistic training of the model. So one retrieves the data it stores, and the other, quite magically, generates data.

Database systems serve a strict SQL dialect; Gen AI systems work with natural language, with a very loose structure. I would consider this a superpower of these systems. Responses are similarly natural language text, code, or something else; the important part is that it's all generated. Fascinating technology.

Models themselves are basically a set of files on your disk after you have downloaded the models. I use HuggingFace for that. The software that serves requests using the model is called an Inference Engine.

From an infrastructure engineer's perspective, I think it is fair to think of Inference Engine deployment as somewhat equivalent to Database Server deployments. One difference is the hardware. GPUs are commonly used for inference at scale; other accelerators, such as TPUs, also exist. We'll explore the hardware some other time, but the gist is that LLMs utilize parallel processing capability provided by GPUs to provide a reliable experience.

Another interesting aspect to understand is that the LLMs, when serving, are frozen. There are no updates to the model weights during inference; that's a large difference from our database analogy. Compared with the database workloads I'm used to, GPU-hosted inference puts much more emphasis on GPU compute and memory capacity and bandwidth.

The hardware

I think we have a rough mental model now on how to think about Inference engines. So getting more hands-on by actually running a model would be interesting. I have a decent laptop, Lenovo ThinkPad P1, 16-core CPU, 64GB RAM, NVIDIA RTX 3000 Ada Generation Laptop GPU, 2TB NVMe disk. Of these resources, the most important is the GPU: 4608 CUDA cores, 8GB VRAM, 256 GB/s bandwidth, ~20 TFLOPS, and 319 TOPS sparse FP8.

Some explanation of these numbers:

• 8GB VRAM stores the model weights and some buffers. For comparison, the NVIDIA B300 has 288GB, 36x more. Can run bigger models; we'll see soon why.

• 256 GB/s bandwidth; for now, understand that higher memory bandwidth helps when memory transfers are the bottleneck. Blackwell B300 has upto 8TB/s, ~31x more.

It is not fair that I compare a laptop-class GPU with a datacenter-grade accelerator, but it's important to understand the constraints before we start and what to expect in real Production Environments.

I'll start with the simplest approach; we'll use vLLM to run a small model like Qwen3-0.6B. That model name is a mouthful, so here's how I understand it,

• Qwen - It's the model family name, like GPT models or the Claude model family. For now, it's just a name. 3 is the major version. I still haven't checked what changes cause a major version. Will report that later.

• 0.6B - This model has 600 million learned parameters. Parameter count matters for deployment because it affects weight storage and computational requirements. It alone does not indicate the model's answer quality.

The parameter count is especially interesting in deployments because it helps determine how much GPU memory the weights need. The KV cache and other runtime allocations need additional space.

A 14B model at four bits looks close to my GPU’s capacity on paper. Whether it actually fits requires checking the complete runtime memory footprint.

Running the Model

Below is the command I run to start a vLLM engine to serve the said model,

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve Qwen/Qwen3-0.6B \
  --served-model-name Qwen/Qwen3-0.6B \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 2048 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.65 &

VLLM_PID=$!

I'll try to explain the flags we used,

VLLM_USE_FLASHINFER_SAMPLER=0 causes vLLM to not use the FlashInfer top-k/top-p sampler on CUDA. We did not have it available.

• --served-model-name - The name of the model exposed through the API.

• --host --port - The address where the API is hosted.

• --dtype bfloat16 - We run with BF16 precision, a 16-bit floating-point format. Given this is a small model, the memory requirement of ~1.2 GB should fit comfortably in our VRAM, before the KV cache and other runtime allocations.

• --tensor-parallel-size 1 - Let's skip this one for now.

• --max-model-len 2048 - This is the context size of a request to the model plus the output length. The same one we see in ChatGPT and other apps.

• --max-num-seqs 8 - This is an upper limit on sequences processed in an iteration. The token budget also affects scheduling.

• --max-num-batched-tokens - This is an upper limit on how many tokens would be processed in an iteration.

• --gpu-memory-utilization 0.65 - This gives vLLM a budget of 65% of GPU memory for its model executor, including weights, runtime allocations, and the KV cache.

So that runs the model.

First Request

$ curl -sS http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [
      {
        "role": "user",
        "content": "What is the most famous greeting in programming?"
      }
    ],
    "max_tokens": 128,
    "temperature": 0.7,
    "chat_template_kwargs": {
      "enable_thinking": false
    }
  }' | python3 -m json.tool

Response excerpt

"The most famous greeting in programming is often **\"Hello, world!\"**. It is a simple and universally recognized program that is used in various programming environments, from simple text editors to more complex applications. This greeting is often celebrated in programming communities and is a classic example of a programming language."

It answered the question, but then wandered into an incorrect claim about programming languages. The engine works; whether the answers are good is a separate question.

Benchmarking

To get an idea of what kind of performance to expect, I ran a benchmark. This benchmark tool is provided by vLLM; from a database analogy perspective, it is similar to sysbench. This is one synthetic serving benchmark using random 512-token inputs and fixed 128-token outputs, with a maximum concurrency of 8. It measures serving performance and not answer quality.

• vLLM version - vllm-0.30.0-nohash

• OS - Ubuntu

• Kernel - 6.8.0-146-generic

• NVIDIA driver - 580.178.04

vllm bench serve \
  --backend openai \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model Qwen/Qwen3-0.6B \
  --dataset-name random \
  --random-input-len 512 \
  --random-output-len 128 \
  --random-range-ratio 0 \
  --random-prefix-len 0 \
  --num-prompts 128 \
  --num-warmups 8 \
  --max-concurrency 8 \
  --request-rate inf \
  --ignore-eos \
  --temperature 0 \
  --seed 42 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,95,99 \
  --disable-tqdm \
  --save-result \
  --result-filename qwen3-06b-benchmark.json

Some important flags,

• --random-input-len 512 - That's the prompt size in tokens that the benchmark will fire at the engine.

• --random-output-len 128 - That's the output response size in tokens. I use tokens as a unit here because that's what LLMs work with. Token vocabulary is provided by the model.

• --max-concurrency 8 - The benchmark keeps at most 8 requests in flight. The engine can process up to 8 sequences per iteration, depending on its token budget.

• --percentile-metrics ttft, tpot, itl, e2el

  • TTFT (Time To First Token) measures the elapsed time from when the benchmark client submits a request until its first streamed output arrives. Prefill contributes to this, but TTFT does not isolate prefill time. We measure this specifically to understand prefill phase performance. Prefill is the phase where the request prompt is processed.

  • TPOT (Time Per Output Token) tells us the average time taken per output token after the first. For reference, the calculation is (end-to-end latency - TTFT) / (output tokens - 1).

  • ITL (Inter-Token Latency) measures the gap between successive streamed outputs received by the benchmark client.

  • E2EL (End-to-End Latency) is the complete request-response latency. It helps us capture phases such as tokenization, prefill, decoding, and output generation.

Understanding the Results

Now let's review the results of this benchmark run,

Model:                              Qwen/Qwen3-0.6B
Weight dtype:                       bfloat16
GPU:                                RTX 3000 Ada Laptop, 8 GB
Measured requests:                  128
Successful / failed:                128 / 0
Input / output tokens per request:  512 / 128
Configured concurrency:             8

Benchmark duration:                 18.20 s
Request throughput:                 7.03 requests/s
Output token throughput:            900.35 tokens/s
Total input + output throughput:    4,501.74 tokens/s

Mean time to first token:           84.35 ms
P95 time to first token:            121.85 ms
Mean time per output token*:        8.28 ms
Mean request latency:              1,135.82 ms
P95 request latency:               1,146.26 ms

* Excludes the first token.

I have formatted the results for display. Let's go through the important ones.

• We did a total of 128 requests, shown by Measured requests. It took 18.20s to do so.

• Across eight concurrent requests, the server produced approximately 900 output tokens per second in aggregate.

• Mean time to first token is ~85 ms with a p95 of about 122 ms. Prefill contributes to this, but this run doesn't tell me how much time was spent processing the prompt versus waiting or handling the request. Does batching hurt TTFT here? That needs another experiment.

• The mean time per output token is 8.28 ms. This is the decode phase where the model is iterating through all its layers to generate one token, excluding the first one. This model has 28 transformer layers. Likewise, TPOT is the average time per subsequent output token.

• Mean request latency and p95 for the same indicate the overall end-to-end latency of the request. With roughly eight requests in flight, about 1.14 seconds per request across 128 requests gives us ~18 seconds. So the math checks out.

Some thoughts about the important phases of inferencing.

Tokenization

• Each model ships with a token vocabulary.

• There is a tokenizer that converts your prompts into tokens that the model understands.

Prefill

• This is the phase where the tokenized input prompt is processed by the model.

• Since all the input tokens are known beforehand, the model can process many prompt positions in parallel within each layer. During prefill, the model also builds the prompt's KV cache, which will be reused during decoding.

• Prefill is therefore a major contributor to TTFT, although measured TTFT also includes time spent queueing, handling the request, and returning the first output to the client.

• Batching at this phase can improve GPU utilization, but it can also increase individual request latency; I need to measure the trade-off rather than assume it.

Decode

• This is the phase after the first output token has been selected.

• The engine feeds the newest token through the model to predict the next token, then repeats the process.

• For this Qwen3-0.6B model, each forward pass traverses all 28 transformer layers. Without caching, the model would need to recompute earlier tokens' representations repeatedly. The KV cache avoids this by retaining the key and value vectors computed for previously processed tokens in each layer.

• Multiple requests can have their decode steps batched together while keeping their attention state logically separate. They share the same model weights, so batching helps spread the cost of reading those weights across more tokens.

Detokenization

• We still have tokens from the decode phase. The tokenizer converts the generated token IDs back into text, which is either streamed or returned to the client, based on how outputs are set to be received.

So that's basically it about Inference Engines that I have learned so far. It is an interesting technology with many solutions differing in their implementation. For example, the KV Cache we talked about is prone to memory fragmentation, leading to waste. vLLM implemented PagedAttention, which borrows paging ideas from the operating system's virtual memory to GPU memory, mapping logical blocks to potentially noncontiguous physical blocks.

We also ran a small and simpler model to demonstrate how inference works; an interesting challenge would be trying to fit a larger model in our system. More on that later.