knotmeter
Every 15 minutes, one 256-token completion against every open-weight
model hosted on digitalocean.
Tokens per second, time to first token,
availability - measured, not quoted.
method
one streaming request per model per 15-minute slot, aligned to :00/:15/:30/:45 UTC,
max_tokens=256, run sequentially so our own requests never queue behind
each other and inflate the very latency we are measuring. the prompt is frozen and
versioned; it carries a per-sweep timestamp prefix so prompt caching cannot
quietly turn a cache hit into a fast model.
- tok/s
- decode speed after the first token:
(output_tokens − 1) ÷ (t_last − t_first). reasoning tokens count, because the gpu really produced them. - ttft
- milliseconds from dispatch to the first token of any kind — including a reasoning model's thinking tokens. this is what "latency" means here; there is no second number hiding behind that word.
- availability
- successful requests ÷ requests the endpoint can be held to. timeouts, stalls and 5xx count against a model. our own gaps — a sweep that ran out of budget, a rate limit we hit by benchmarking, a broken key — never do.
- coverage
- how many of the sweeps we should have run actually ran. kept strictly apart from availability: coverage is knotmeter's reliability, availability is digitalocean's.
- p50 / p95
- median and tail over a rolling window. both tails are the slow end, but the slow end sits at opposite ends of the two distributions: for tok/s it is the bottom, so that one shows p05; for ttft it is the top, so that one shows p95.
limits, stated rather than implied: one vantage point at 15-minute resolution is an uptime estimate, not an sla measurement — an outage shorter than the interval can be missed entirely. a single sample per slot is noisy on its own; the windows are the smoothing. closed models are excluded because digitalocean proxies rather than hosts them, so measuring them would measure someone else's infrastructure.