Begin with a task and a pass condition
Choose the workload before choosing the benchmark prompt. For document extraction, define the fields and compare them with known answers. For code, run an appropriate test or compiler. For an agent that selects tools, check the tool name and arguments against the intended action.
Use a saved set of prompts and record how you selected them. Include difficult cases and failures you care about. Keep the same set when comparing models. A fast completion that fails the task should remain visible as a failure.
Record model identity and environment
Save the platform model ID, repository name and revision if available, quantization, runtime version, machine memory and chip. State the date and whether the model was already loaded. If the service does not expose a field, mark it unknown rather than guessing from a tier name.
The PROXIES.SX catalog exposes model requirements. Its September 11 response includes immutable repository revisions, but no public performance results. The provider guide describes automatic verification, but does not supply enough benchmark details to reproduce its tier assignment independently. Provider verification description
What the preliminary local result establishes
The published facts snapshot reports one isolated local test on September 10, 2026: an M4 Max with 36 GB RAM running qwen3.8-27b-4bit at approximately 19–20 output tokens per second. That is a reported result for one setup. It is not a fleet comparison, a customer rental test or an availability promise.
The snapshot does not include the prompt set, token-counting method, trial count, detailed runtime settings or raw timing logs. Those omissions prevent an independent reproduction from the snapshot alone. Use the methodology below before comparing another result. The same snapshot marks a live paid rental and a real supplier 24-hour soak as unproven. We did not run this local test or validate its speed independently.
Separate waiting from generation
Measure time to first token from sending the request to receiving the first generated content token. State whether that includes network travel, queueing and prompt processing. A timestamp taken only after response headers arrive misses some of the user's waiting time.
For output speed, record output tokens and the generation interval you used as the denominator. A client receiving streamed text chunks should not count chunks as tokens. Use returned token usage when available, or document the tokenizer and counting method. Measure whole-request duration as well so a long prompt cannot disappear from the comparison.
Repeat under the concurrency you need
Test one request first, then run the simultaneous workload the application actually produces. Keep a separate record for each concurrency level. Report the number of attempts, successful responses, timeouts and other errors.
For latency, include a median and a high percentile only with enough observations to make it meaningful. A p95 calculated from a handful of requests is unstable. Preserve failed attempts in the dataset rather than quietly removing them to improve the average.
A separate 2025 comparative study used a 192 GB M2 Ultra Mac Studio and Qwen-2.5 models to examine five local runtimes. It reported different leaders for sustained generation and first-token latency under its settings. Those results explain why both measurements matter; they do not rank the Qwen3.8 rental configurations here. Apple Silicon runtime study
Use consistent latency and throughput definitions
NVIDIA's GenAI-Perf documents distinct measurements for first response, request duration and token throughput. That separation is useful when comparing any inference endpoints; its example results are not Apple Silicon measurements. GenAI-Perf metric definitions
| Measurement | Record in your worksheet |
|---|---|
| Time to first token (TTFT) | Request start to first generated content. State if your tool instead uses the first server event. |
| Completion latency | Request start to final response. |
| Decode rate | Output tokens divided by the declared generation interval. State whether the first token is excluded. |
| Aggregate throughput | Completed useful work across all concurrent requests per wall-clock interval. |
| Task pass rate | Accepted results divided by all scored attempts. Keep errors visible separately. |
If a stream starts with metadata, timing that first event can understate the wait for readable output. Match definitions before comparing numbers from different tools.
Use several prompt lengths, with the same output cap and generation settings for each candidate. Run a separate cold-load case, then a warm sequence. Finish with expected peak concurrency and report how many cases you ran. A single fast request cannot establish sustained capacity or a stable high-percentile latency.
Download the recording worksheet
The CSV worksheet is an empty recording format. It contains no measured PROXIES.SX performance. Store a prompt-set identifier instead of confidential prompt text when sharing results.
Record at least request start, first content token and completion times; input and output token counts; concurrency; status; and the task's quality result. Add power readings if you are evaluating a provider machine. Mark requests made while loading the model separately from requests made after it is ready.
Turn the result into a capacity decision
Estimate whether the successful throughput can finish the expected work inside its deadline. Allow for bursts and interruption. An overnight classification job may tolerate slower responses that would make an interactive assistant frustrating.
Then use the pricing calculation with measured useful output. For providers, use wall power and observed rental periods in the earnings calculator. Publish the test conditions with any future speed claim, including the cases that did not pass.
No paid rental or hardware benchmark was run for this guide. It provides a test method and a recording format, not a performance ranking.
Sources and references
Reviewed September 11, 2026. Product statements come from public APIs, provider documentation and published application code. Technical references explain the evaluation methods. Authenticated rental and payout behavior has not been tested.
- Dated compute product facts. PROXIES.SX.
- Production-Grade Local LLM Inference on Apple Silicon (2025). Varun Rajesh et al..
- Compute provider documentation. PROXIES.SX.
- GenAI-Perf metrics. NVIDIA.