Benchmarks

Every library runs unmodified in the same simulator, against the same fakes of DataStore, MemoryStore, MessagingService and the engine, with the same players, faults and seeds. The table below is bench/Benchmarks.md as the last luneblox run bench/Report 20 wrote it; the methodology at its end explains every column. To run it yourself:

luneblox run bench/download       # the rivals, at their pinned versions
luneblox run bench/Report 20      # every scenario over seeds 1-20, then the cost

These are simulator figures. The KeepBlox rows did not change when the benchmarks were re-measured on 2026-09-30 with the budget model measured live. On Roblox’s real services calls take 0.3-0.6 s each, so hand-overs (4.2-4.6 s against 3.2 s) and crash takeovers (11.6-12.9 s against 9.1 s) are slower. The No library row is a game’s own GetAsync / SetAsync code, as the Roblox guides teach.

Benchmarks: KeepBlox and the libraries games use today

Generated by luneblox run bench/Report 20 on 2026-09-30. Every library runs unmodified in the same simulator (tests/sim): the same fakes of DataStore, MessagingService and the engine, the same players, the same faults, and the same seeds. See Methodology below.

Data safety

Worst case over seeds 1-20 for each scenario.

Library	Violations	Crash: progress lost (steps)	Longest stale owner (s)	Slowest open (s)	Failed opens	Library errors
KeepBlox	0	4	0.0	9.3	0 of 2584	0
ProfileStore	0	287	244.5	46.6	20 of 2485	0
ProfileService	0	29	0.0	65.8	27 of 2458	0
DocumentService	0	0 (player locked out: 20 of 40 rejoins)	0.0	15.9	389 of 2539	40
Lapis	0	0 (player locked out: 20 of 40 rejoins)	0.0	11.9	374 of 2493	0
DataStore2	61	287	900.2	8.8	0 of 2595	0
Suphi	1	0 (player locked out: 20 of 40 rejoins)	0.0	0.5	355 of 2585	0
No library (GetAsync / SetAsync)	63	51	900.3	0.2	0 of 2601	0

Cost

Requests per player per hour of play (20 players, one hour), and 50 players joining one server at once.

Library	Read	Write	Ordered write	Ordered list	MemoryStore units	Join storm p50 / p99 / max (s)	Storm failures
KeepBlox	13.1	13.1	0.0	0.0	45.2	0.1 / 0.2 / 0.2	0 of 50
ProfileStore	13.6	13.6	0.0	0.0	0.0	0.1 / 0.2 / 0.2	0 of 50
ProfileService	122.0	122.0	0.0	0.0	0.0	6.2 / 6.3 / 6.3	0 of 50
DocumentService	25.9	25.9	0.0	0.0	0.0	0.1 / 0.2 / 0.2	0 of 50
Lapis	14.0	14.0	0.0	0.0	0.0	0.3 / 0.3 / 0.3	0 of 50
DataStore2	0.0	1.0	1.0	1.0	0.0	0.3 / 7.6 / 7.6	0 of 50
Suphi	1.0	120.3	0.0	0.0	62.0	0.3 / 0.4 / 0.4	0 of 50
No library (GetAsync / SetAsync)	1.0	60.7	0.0	0.0	0.0	0.1 / 0.2 / 0.2	0 of 50

Every scenario

KeepBlox

Scenario	Progress lost (steps)	Slowest open (s)	Failed opens	Shutdown (s)	Violations
rejoin	0	1.2	0 of 40	-	none
handoff	0	2.9	0 of 40	-	none
crash	4	9.1	0 of 40	-	none
partitioned	0	4.2	0 of 40	-	none
thirdRequester	0	4.5	0 of 60	-	none
shutdown	0	0.5	0 of 1000	3.1	none
outage	0	0.5	0 of 60	-	none
poison	0	0.5	0 of 40	-	none
soup	4	9.3	0 of 1264	-	none

ProfileStore

Scenario	Progress lost (steps)	Stale owner (s)	Slowest open (s)	Failed opens	Shutdown (s)	Violations
rejoin	0	0.0	5.4	0 of 40	-	none
handoff	0	0.0	5.4	0 of 40	-	none
crash	287	0.0	46.6	0 of 40	-	none
partitioned	149	244.5	46.5	0 of 40	-	none
thirdRequester	0	0.0	5.4	20 of 60	-	none
shutdown	0	0.0	0.2	0 of 1000	3.1	none
outage	0	0.0	0.2	0 of 60	-	none
poison	258	0.0	46.5	0 of 40	-	none
soup	230	0.0	46.6	0 of 1165	-	none

ProfileService

Scenario	Progress lost (steps)	Slowest open (s)	Failed opens	Shutdown (s)	Violations
rejoin	0	7.3	0 of 40	-	none
handoff	0	14.4	0 of 40	-	none
crash	29	63.7	0 of 40	-	none
partitioned	0	35.4	0 of 40	-	none
thirdRequester	0	14.4	20 of 60	-	none
shutdown	0	7.3	0 of 1000	9.9	none
outage	0	7.3	0 of 60	-	none
poison	228	63.8	0 of 40	-	none
soup	42	65.8	7 of 1138	-	none

DocumentService

Scenario	Progress lost (steps)	Slowest open (s)	Failed opens	Shutdown (s)	Violations	Library errors
rejoin	0	1.4	0 of 40	-	none	0
handoff	0	15.9	0 of 40	-	none	0
crash	0	0.2	20 of 40	-	none	0
partitioned	0	0.2	20 of 40	-	none	0
thirdRequester	0	0.2	40 of 60	-	none	0
shutdown	0	0.2	0 of 1000	3.0	none	0
outage	0	0.2	20 of 60	-	none	0
poison	0	0.2	20 of 40	-	none	40
soup	143	15.9	269 of 1219	-	none	0

Lapis

Scenario	Slowest open (s)	Failed opens	Shutdown (s)	Violations
rejoin	1.4	0 of 40	-	none
handoff	11.9	0 of 40	-	none
crash	0.3	20 of 40	-	none
partitioned	0.3	20 of 40	-	none
thirdRequester	0.3	40 of 60	-	none
shutdown	0.3	0 of 1000	3.1	none
outage	0.3	20 of 60	-	none
poison	0.3	20 of 40	-	none
soup	0.3	254 of 1173	-	none

DataStore2

Scenario	Progress lost (steps)	Stale owner (s)	Slowest open (s)	Failed opens	Shutdown (s)	Violations
rejoin	17	0.1	0.6	0 of 40	-	ack-lost x1
handoff	103	10.2	0.3	0 of 40	-	ack-lost x20
crash	287	0.0	0.3	0 of 40	-	none
partitioned	103	900.2	0.3	0 of 40	-	ack-lost x20
thirdRequester	30	300.3	0.3	0 of 60	-	ack-lost x20
shutdown	0	0.0	8.8	0 of 1000	6.2	none
outage	89	0.0	0.3	0 of 60	-	none
poison	258	0.0	0.3	0 of 40	-	none
soup	449	0.0	0.6	0 of 1275	-	none

Suphi

Scenario	Progress lost (steps)	Slowest open (s)	Failed opens	Shutdown (s)	Violations
rejoin	0	0.4	5 of 40	-	none
handoff	0	0.4	20 of 40	-	none
crash	0	0.4	20 of 40	-	ack-lost x1
partitioned	0	0.4	20 of 40	-	none
thirdRequester	0	0.4	40 of 60	-	none
shutdown	0	0.5	0 of 1000	6.1	none
outage	0	0.4	20 of 60	-	none
poison	228	0.4	0 of 40	-	none
soup	30	0.5	230 of 1265	-	none

No library (GetAsync / SetAsync)

Scenario	Progress lost (steps)	Stale owner (s)	Slowest open (s)	Failed opens	Shutdown (s)	Violations
rejoin	40	0.3	0.2	0 of 40	-	ack-lost x3
handoff	58	10.3	0.2	0 of 40	-	ack-lost x20
crash	51	0.0	0.2	0 of 40	-	none
partitioned	58	900.3	0.2	0 of 40	-	ack-lost x20
thirdRequester	30	300.3	0.2	0 of 60	-	ack-lost x20
shutdown	0	0.0	0.2	0 of 1000	3.1	none
outage	89	0.0	0.2	0 of 60	-	none
poison	0	0.0	0.2	0 of 40	-	none
soup	60	0.0	0.2	0 of 1281	-	none

Versions

KeepBlox: this branch
ProfileStore: 1.0.3 (vendored, tests/reference)
ProfileService: master d736800 (2024-10-13)
DocumentService: 1.2.2
Lapis: 0.3.3
DataStore2: 1.4.0
Suphi: Creator Store 11671168253 (vendored, tests/reference/Suphi)
No library (GetAsync / SetAsync): GetAsync / SetAsync, as the Roblox guides teach

Methodology

One simulator for every library. Each library runs unmodified, one copy per virtual server, in tests/sim/RobloxEnv: game, task, require, the DataStore, MemoryStore and MessagingService are fakes that copy the documented engine behaviour (each fake cites its Roblox doc page), and what the live check measured where the docs are silent (tests/live). Time is virtual and every service call is a yield point; the seed decides the interleaving, so any run replays exactly from its seed. The rival libraries are downloaded at exact pins by bench/download.luau; ProfileStore and Suphi’s DataStore Module (which ships only as a Roblox asset) are vendored in tests/reference. The No library row (bench/adapters/BareAdapter.luau) is what the Roblox guides teach: GetAsync on join, SetAsync on leave and every 60 s, and everyone saved on BindToClose, with no lock.

One way to use each library. Each adapter (tests/sim/*Adapter.luau, bench/adapters/) uses its library the way its own documentation tells a game to: load on join, release on leave, kick when the session ends. An adapter translates, it never adds behaviour. Libraries with no “saved, with this data” signal are acknowledged at the service boundary: a write the library made commits while its session is live, and the data that write stored counts as acknowledged (it is exactly what the store holds).

The scenarios (tests/sim/Scenarios.luau): a player leaves and rejoins elsewhere (rejoin), arrives on another server before the old one noticed (handoff), their server crashes (crash), their server is cut off from MessagingService (partitioned), three servers want the same key (thirdRequester), 50 players on a closing server (shutdown), the data store fails for minutes (outage), the game writes a value the store cannot hold (poison), and everything at once for many players (soup).

The columns.

Violations: breaches the ledger (tests/sim/Ledger.luau) saw. ack-lost: a save the library acknowledged was later missing. foreign-write: a server changed the data while another held the lock. load-count-backwards: the session counter went back. For a library that keeps no lock in the stored value (DataStore2, and the game with no library), only ack-lost applies.
Progress lost: the most steps of play a reload did not have. It is measured only on loads that succeed, so read it with player locked out: a library whose lock outlives a crashed server (for 10 or 30 minutes) loses nothing on paper because the player cannot load at all.
Stale owner: how long a server kept a session active after another server had opened the same key. Trades and purchases made meanwhile are made on a copy that will be overwritten.
Slowest open and failed opens: how long a player waits for their data, and how many never get it. An open whose player had already left is neither: it is not counted.
Library errors: errors the library threw into the game’s threads.
Requests per player-hour: data store requests, by the budget they draw on, over an hour of play. Each server’s budget refills at 60 + 40 per player a minute for reads and for writes; a new server starts with a burst (about 600 with one player), and nothing caps what it saves up (both measured live). Above that, the experience as a whole may make 300 + 40 x players reads and 300 + 20 x players writes a minute, shared by every server; a request over it fails. Every request the game makes draws on both, so a library that spends less leaves more to the game.
MemoryStore units: request units per player-hour against MemoryStore’s own quota (1000 + 120 per user a minute, experience-wide). KeepBlox’s heartbeat and Suphi’s lock live there.

What is not compared. CPU time: the engine serializes values natively in Roblox, and in the simulator it does not, so a library’s own CPU cost cannot be separated fairly. KeepBlox’s own frame cost is in bench/Run.luau.