Imagine choosing a cloud provider if every provider measured speed with its own test, on its own schedule, and published only the results it chose. That is close to where quantum computing is today. Quantum computers are improving quickly, but the way we describe and compare them has not kept up.

Why the usual numbers mislead

The most quoted number is the qubit count, yet it says little about what a machine can do. Two devices with the same number of qubits can differ greatly in error rates, connectivity between qubits, measurement accuracy and how long qubits keep their state. Other figures, such as gate fidelities, are usually published by the device’s maker, measured with the maker’s chosen method, and averaged in ways that can hide the weakest qubits a real program may end up using.

Even a careful figure has a short shelf life. Devices are recalibrated often, and their behaviour drifts between calibrations. A result from last month was, in every sense that matters, produced by a different machine from the one available today.

What is missing

Good work on benchmarking exists, from research groups, industry consortia and standards bodies, and Australia’s National Quantum Strategy names standards and frameworks among its themes. What is still missing is a widely adopted, practical way to report results so that anyone can compare them and check them. In particular, a result rarely records:

  • the exact program as written, and the circuit the compiler actually produced for the device;
  • the device, its calibration at the time of the run, and the compiler’s settings;
  • the number of shots, the raw counts and any error mitigation applied;
  • what the result should have been, and whether the difference is larger than shot noise can explain.

Without these, a published result cannot be reproduced, a prediction cannot be tested, and a comparison between devices is an opinion.

What a better standard looks like

I think a useful benchmarking practice for quantum computing should rest on a few principles, each familiar from good science:

  • Freeze before running: capture everything that determines a result before the run, so nothing can be adjusted afterwards.
  • Keep evidence classes apart: what the compiler produced, what a model predicts and what a device measured are different things, and should never be presented as one another.
  • Judge with statistics: a check can pass, fail or be inconclusive, and saying “we can’t tell yet” is an honest answer.
  • Measure predictions, don’t assert them: if a model claims to predict a device’s behaviour, record how often it is right.
  • Publish the method: any benchmark should come with its method, software versions and data, so others can repeat it.

Where Lucivant fits

These principles are the foundation of Lucivant Technologies Pty Ltd. The platform we are building is designed to freeze each experiment into a fingerprinted record, to predict a device’s behaviour from its published calibration and measure those predictions against real results, and to judge outcomes against shot noise honestly. It is in development, and we will describe what it does only when we can show the evidence.

We also hold ourselves to a simple rule: no marketing comparisons with other quantum tools. If we publish a benchmark, it will come with its method, versions and data, so that anyone can check it. A field that asks the world to trust very unusual machines should always be ready to show its working.