Engine performance

Where the engine’s measured limits are: a roofline analysis of the current engine against the reference host’s measured memory bandwidth, then the partitioned and distributed measurements. Every number is copied from a committed source, named inline and in the footer.

Manual: Stats, memory and logging, NUMA partitions, MPI ranks

The design these numbers measure is Inside the engine; the thread-count advice they support is in Stats, memory and logging.

Layer time breakdown

Measured on the reference host (2× Xeon Gold 6244, 16c/32t) with the repo’s phase-timing probe, per layer class, as a share of summed worker busy time (gather / sort / merge):

Gather + merge dominate the sparse classes; the sort dominates only dense two-qubit PTMs. Parallel efficiency (busy time over coset-loop wall × threads) is 0.99 for su4 at 16 threads: load balance is a solved problem, and the scaling limits below are memory-system effects, not imbalance.

Per-term cost is flat in the sum size over the measured range: 30.5–30.6 ns/term for the ZZ rotation across m = 1.50e6 → 4.50e6, 138.5–141.3 for the sparse 2q unitary, 322–327 for su4 (single thread).

Memory wall

The ceiling comes from a STREAM-style probe (crates/membench via scripts/bandwidth.sh), nominal bytes with plain write-allocating stores — the same store pattern the engine uses. Best GB/s on the reference host:

Two structural facts from the ceiling measurement: hyperthreads add no bandwidth (39.0 → 39.2 GB/s), and the second socket adds only 15–25% under first-touch page placement with work-stealing, not 2×.

The modeled traffic side prices each layer from its row counts at T = 48 B/term (benchmarks/PROFILING.md §Roofline model):

bytes/layer = terms_in×T + 2×(rows_gathered−rows_id)×T + 2×rows_id×16
              + 2×rows_sorted×T + terms_out×T

Reading the ratio of measured DRAM traffic to ceiling as a classification: at or above ~70% of the measured ceiling a phase is bandwidth-bound; modeled traffic far above measured means the working set is cache-served; far below ceiling with a high LLC miss rate points at latency, not bandwidth.

Single-thread roofline

Measured on the current engine (fact sheet: research/HARDWARE.md §Engine roofline, single thread; ceilings per the 1-core row above):

The byte model over-counts DRAM traffic by 2.5–12.8× at one thread because most modeled traffic is served from cache. The sparse classes are limited by load latency (IPC ≈ 2.2–2.6 with 34–41% LLC load-miss); su4 is compute-bound in its sort (IPC 2.98, 2.1% LLC load-miss).

Multi-thread roofline

su4 at m = 1.41e7 (both-socket ceilings: read 45.0 / write 25.3 at 8–16t, 48.8 / 23.1 at 32t):

By 16 threads the dense-PTM class sits at the machine’s write ceiling — the figures above 100% are against the STREAM nominal-store ceiling, which the engine’s write-allocating stores share, so “at the ceiling” is the correct reading. 32 threads buy zero additional bandwidth (67.0 vs 67.0 GB/s attributable) and cost 12% of wall time.

The sparse classes at 32 threads stay latency-bound and keep scaling:

| cell | ns/term | speedup vs 1t | read GB/s | write GB/s | % write ceil | verdict | |—|—:|—:|—:|—:|—:|—:|—| | ZZ rotation, m = 4.50e6 | 2.7 | 11.3× | 8.9 | 11.5 | 50% | latency-bound | | sparse 2q, m = 3.0e6 | 10.8 | 13.1× | 11.7 | 11.3 | 49% | latency-bound |

They stop at half the write ceiling with 70–90% of their modeled traffic cache-served: bandwidth is not what limits them, and they take the full thread count profitably.

Partitioned: P = 2 vs P = 1

Two facts above — the second socket adding 15–25% rather than 2×, and the dense-PTM class pinned to the write ceiling — are the same effect: pages are placed by first touch and the thread pool then steals work across sockets, so roughly half of the second socket’s reads are remote. The engine can instead be run partitioned, one pinned pool and one share of the sum per NUMA domain, with only the rows a layer moves across a domain boundary exchanged (NUMA partitions).

Locality decides everything. A layer whose gates leave the partition rows fixed runs entirely inside its own domain and gains; a layer that moves any row across a boundary pays for the export pass, the copy and the receiver’s larger merge stream. Measured at P = 2 against P = 1 on the same binary, on two-socket hosts from 8 to 48 cores per socket:

The gain grows with cores per socket, as a memory-system effect should: −4% on an 8-core-per-socket workstation, −18% on a 48-core Genoa at 96 threads. Load imbalance across partitions stays within 1.000–1.004 with random rows. Random partition rows put roughly half of a dense two-qubit gate’s deltas across a boundary, so the default draw lands a mixed circuit in the bottom row of that table.

The per-host cells behind the summary, copied from research/HARDWARE.md §Partitioned engine, P=1 vs P=2 in-process: scripts/ab-compare.sh runtime-knob A/B, 5 pairs abba, --n 1000000 --qubits 128 (W = 2), random partition rows, each partition pinned to its socket with MPOL_BIND; steady-state m ≈ 1.5e6 for rotations, 1.0e6 for gu2q, 1.4e7 for su4/su4_local. Wall time per layer, P=2 vs P=1 (median Δ%, pairs agreeing):

Exchange-free rotation layer, per layer (from the probe sidecars):

Counters, ccqlin038, dense layers at 16 threads (8 per socket at P=2); scripts/perf-stat.sh, shared box:

partition_imbalance is 1.000–1.004 everywhere; exchange volume is bytes ≈ rows × 48.

Benchmarking

P = 1 versus P = N is a runtime-knob A/B of one binary, not a code A/B:

scripts/ab-compare.sh partitions-1v2 --a . --b . \
  --probe   '--n 1000000 --threads 32 --layers su4_local --partitions 1' \
  --probe-b '--n 1000000 --threads 32 --layers su4_local --partitions 2 \
             --partition-cpus "0-15,32-47;16-31,48-63"'

--threads is the total at every P. A partitioned cell runs under no placement prefix: numactl and taskset both defeat the split the engine is making. On an exclusive cluster node, scripts/slurm/ab-campaign.sbatch builds the CPU lists from the node’s own topology. The protocol and the sidecar fields are in benchmarks/PROFILING.md.

Distributed: weak scaling

The same split across processes is one partition per MPI rank (MPI ranks), and there the goal is capacity rather than bandwidth. Measured on Icelake ccq nodes (2 × 32 cores), one rank per NUMA domain, 32 threads per rank, UCX shared memory within a node and InfiniBand between nodes, at 6·10⁶ terms per rank:

Weak scaling is flat once the exchange leaves the node: the remote layer costs the same at 4 and 8 ranks, and local layers cost ~10.5 ms whatever the rank count. The step from 2 to 4 is shared memory giving way to InfiniBand. A remote layer is transfer-bound — the pipeline runs the coset loop under the transfer, so what is left is the export pass plus the bytes on the wire — which means the lever is fewer bytes (locality rows, a lower truncation), not more threads.

The full per-layer breakdown, copied from research/HARDWARE.md §Partitioned engine, MPI weak scaling: phase_breakdown --mpi, replicated input --n 4e6 × ranks → 6.0e6 terms per rank (4.0e6 for cnot), --reps 8; medians over ranks, ms per layer.

The remote layer is transfer-bound: ~6–8 ms compute, 7–8 ms export, ~30 ms transfer for 192 MB per rank (two ranks share a NIC → ~13 GB/s per node). vmhwm growth with rank count is the probe’s replicated input, not the engine; engine-side peak per rank is flat.

Noise floor

Single-shot campaign noise on the reference host is ±5–8% single-threaded and ±10–26% at 8–32 threads; untouched code moves that much between campaigns. Effects below that need the interleaved A/B protocol (scripts/ab-compare.sh), whose acceptance criterion is direction consistency across every pair.

Sources: research/HARDWARE.md (roofline tables, phase shares, bandwidth ceilings, partitioned and distributed tables, all achieved numbers and verdicts); benchmarks/PROFILING.md (byte model, interpretation rules, noise floor); the Rust API (layout and dispatch).