More balanced you — and works tirelessly to help you get there.
Estimate model-weight and KV-cache footprint per model, and see how many concurrent users fit before the card runs out of memory.
Parameter counts are computed from the architecture, not looked up.
Source: meta-llama/Llama-3.1-8B config.json. Published total 8.03B; computed 8.03B (+0.00%).
2 bytes / parameter.
2 bytes / element.
Each sequence keeps its own KV cache. Prompt + generated tokens must fit the context length.
gpu_memory_utilization defaults to 0.90 and covers weights + activations + CUDA context + KV cache. KV is allocated in 16-token blocks.
Memory-bandwidth roofline, not a benchmark. Real throughput is lower.
= bytes streamed per step (16.08 GB) ÷ 1,008 GB/s. Interconnect, kernel efficiency and prefill time are not modelled.
Smallest GPU count (1–16) that fits the same workload.
Llama 3.1 8B · prompt 6,144 tok · RTX 4090 (24GB)
Timed against the TTFT and decode-speed estimates above — a real generation with different, unrelated text, but at the same pace.
Exact terms are separated from modelled ones. Nothing is padded with a hidden safety multiplier.