A new public FHIR server benchmark from Health Samurai measures Aidbox, HAPI FHIR, Medplum, and the Microsoft FHIR Server on a 1,000-patient Synthea dataset, totaling around 2 million resources. The harness, scripts, and daily rerun data are open on GitHub. The numbers are useful, but the scale of the test dataset is small enough that several production-relevant questions cannot be answered from this snapshot alone. The honest read is the one that distinguishes what the benchmark establishes from what it explicitly defers.
What the 1K Dataset Establishes Cleanly
A 1,000-patient corpus with 2 million resources gives the harness enough cardinality to exercise CRUD, bundle import, and the six FHIR search families against a non-trivial body of data. Every server under test sees the same resources in the same order, runs on identical hardware (8 vCPU and 24 GB RAM each, on a 64-core machine with 500 GB total RAM), and reports against the same k6 load profile. That is enough to produce a clean baseline.
The numbers in the report at this size are best read as a baseline. The 5,212 CRUD RPS Aidbox posted, the 3,058 HAPI posted, the 1,420 Medplum posted, and the 440 Microsoft posted are real measurements under the harness's conditions. The relative ordering across servers is the part that is most likely to survive a scale test; the absolute numbers are the part that will shift.
What the Dataset Size Hides
The 2-million-resource working set fits comfortably inside the 500 GB of RAM the benchmark machine carries. That single fact changes the shape of the test in ways that matter for production reasoning.
A FHIR server running against a fully in-memory dataset does not have to make the trade-offs a server running against a dataset larger than RAM has to make. Cold-cache reads do not show up. Index pages stay resident. Write-behind buffers never fill. Connection pool contention never reaches the levels a multi-tenant production system sees. These behaviors are exactly the ones that decide whether a server holds up at scale, and the 1K-patient test sits below the threshold where they appear.
For an EMR vendor or telemedicine platform evaluating servers for a stack that will eventually carry hundreds of thousands of patients, that gap matters. The honest signal from the benchmark is the ordering and the methodology, not the per-server number you should expect to see in your own production deployment.
What Comes Next in the Series
The Health Samurai team has stated the next post in the series tests at scale. That follow-up is where the production-relevant numbers will surface: behavior when the working set exceeds RAM, search latency under cache eviction, bundle import under sustained pressure rather than a controlled run, and storage growth curves under continuous write load. A baseline benchmark and a scale benchmark answer different questions, and the team has been clear that this snapshot is the first.
The benchmark was authored by Marat Surmashev, VP of Engineering at Health Samurai, which is also the company that develops Aidbox. The repository is open and the harness reruns daily, which makes the snapshot re-verifiable. That said, a vendor-run benchmark on a small dataset is a starting point, not a procurement input.
How to Use This Snapshot Today
The defensible read is to use the current numbers to shortlist servers and to wait for the scale follow-up before drawing operational conclusions. For the FHIR product comparison hub and adjacent comparisons on terminology and form-builder selection that face the same scale-vs-baseline trade-off, the Ontoserver vs Kodjin comparison covers a parallel decision, and the FHIR terminology servers reference guide covers the broader selection ground. The benchmark is a useful input. Treat it as one.