Benchmarking

Measure how KahunaCart performs on a store far larger than yours, on your own hardware and database engine. Use it to size a server, or to compare a change against the release before it. The results we publish are on the benchmarks page.

Before you begin

• Use a database you can throw away. bench:seed writes hundreds of thousands of rows and refuses to run on a store that already holds orders it did not write.
• Migrate the schema first. Run bin/plugin kahunacart migrate; both commands refuse to start otherwise.
• Free about 2 GB of disk for the default size on SQLite.

The two commands

Both are documented with their full option lists on the CLI reference.

Seed a benchmark store

1. Point database.type at the engine you want to measure. See Configuration → Database.
2. Run bin/plugin kahunacart migrate.
3. Run the seeder:

bin/plugin kahunacart bench:seed

The default size takes about 30 seconds on an M-series Mac with SQLite and produces a database of about 1.5 GB, roughly 4.5 million rows.

Use a smaller store while you are setting the process up:

bin/plugin kahunacart bench:seed --products=1000 --customers=5000 --orders=20000

Write a machine-readable summary instead of a report:

bin/plugin kahunacart bench:seed --json > seed.json

Take it all back out:

bin/plugin kahunacart bench:seed --purge

--purge takes about 12 seconds at the default size.

What the seeded store contains

At the default size:

The status mix is roughly 85% paid and fulfilled, 5% awaiting payment, 5% canceled and 5% refunded in part or in full.

Every row is filled the way the checkout and fulfillment code fills it: totals reconcile with lines and adjustments, fulfillment status agrees with the parcels, a refund's ledger row points at the purchase it reverses, and each status change carries its history entry.

The seeder does not decrement stock per line, queue an email or fire an order event.

Time the queries

bin/plugin kahunacart bench:queries

Raise the sample size for a figure you intend to quote:

bin/plugin kahunacart bench:queries --iterations=100 --warmup=10

Write a machine-readable document:

bin/plugin kahunacart bench:queries --json > queries.json

Each query calls the same repository and service methods the storefront and the admin call, not hand-written SQL. When a release changes what OrderRepository::completed() does, the number moves.

The command leaves the store as it found it: the cart it prices is removed and the stock it takes is put back.

What each query stands for

Read the numbers

Four figures per query, in milliseconds.

The rows column is how many rows the query returned. A timing that improved because the query stopped matching anything shows up as a row count that fell.

Measured: the order search and the report rollup

The first standard-size run found two queries that scaled with the orders table. Both were fixed, in migrations 0031 and 0032. Before-and-after figures on the default seed, SQLite 3.53, PHP 8.4, an M-series Mac, 50 iterations after 5 warm-up runs, measured while the machine was also serving a load test:

Every other query moved by less than a millisecond. Seeding costs about 17 seconds more, for the search haystack and the rollup.

Three indexes do most of that work.

• ix_kahunacart_orders_search_text, and the engine text index beside it. See Orders and the admin.
• ix_kahunacart_orders_status_completed on (status, completed_at). The live half of a report went from 762 ms to 0.1 ms.
• ix_kahunacart_product_stats_product, which covers the group-by behind top products: 23 ms instead of 200 ms.

See Reports and the rollup.

Measured: comparison reporting

The Analytics screen asks two questions the benchmark's own targets do not: a range against another range, and a filtered range cut by a dimension. Both were measured on a 250,000-order store on each engine, 11 runs each, on the machine described under Compare database engines.

Three things to read out of that table.

An unfiltered range that is entirely rolled up is free, on every engine. It is a sum over day rows under a covering index.

The comparison range costs what it costs because of the timezone. Comparison periods use calendar dates in the store's timezone, so the previous-year range starts and ends part way through a UTC day, and those two partial days are read from the orders table. That is the whole 44 ms on SQLite, 71 ms on MariaDB and 25 ms on PostgreSQL. See the timezone rule.

A filter means reading the orders table, because the rollup has no order columns to filter on. The filtered numbers are four indexed statements (orders, lines, adjustments and refund transactions) plus the breakdown's own, all driven from ix_kahunacart_orders_status_completed. Before filter sets that do not join stopped being wrapped in a semi-join, the filtered scan was 1,608 ms and the segmented report 3,454 ms on SQLite.

Compare database engines

Point database.type at each engine in turn, migrate, seed from the same --seed, and run the queries:

bin/plugin kahunacart migrate
bin/plugin kahunacart bench:seed --json > seed-sqlite.json
bin/plugin kahunacart bench:queries --json > queries-sqlite.json

Then set database.type to mysql or pgsql in user/config/plugins/kahunacart.yaml, and repeat.

The same rows land on all three engines, so any difference in the timings is the engine.

Measured: all three engines, same store, same machine

Every figure below was measured on one laptop: an Apple M4 Max with 36 GB of memory running macOS 26.6 and PHP 8.4.25, with nothing else running.

The store is the default seed on each engine: bench:seed with no flags, so seed 20260830 and --now 2026-08-30T00:00:00Z, giving 5,000 products, 25,000 customers, 250,000 orders and 511,731 order lines on all three.

bench:queries at its defaults, 50 timed runs after 5 warm-up runs, three times per engine. Each cell is the median of the three runs' p50s, in milliseconds. Every cell above 5 ms sat inside a 10% spread across its runs, most inside 3%.

There is no headline. Fourteen of the eighteen queries land inside a few milliseconds of each other on all three engines, and on a real storefront request the page, the theme and the network are what a shopper waits for. Pick the engine your hosting and your backups want. The four rows where the engines differ each have a cause.

MariaDB's order search is a fixture artefact. Every seeded order has an address at bench.kahunacart.test and a number starting BENCH-, so the words bench, kahunacart and test appear in all 250,000 search haystacks. InnoDB's FULLTEXT in boolean mode intersects a posting list per word, and a word in every row is a posting list the length of the table. On the seeded store +bench* +126001* takes 76.8 ms and +126001* alone takes 0.20 ms, for the same single order. A real store's emails and order numbers are not all the same three words, so a real store does not pay this.

*MariaDB's deep order page is `SELECT ` over an offset.** 110 ms at page 500, against 15 ms and 11 ms. The admin list reads whole order rows and skips 12,475 of them to reach 25, and an order row on MariaDB carries the search haystack and several JSON snapshots. Page 1 is 0.23 ms on the same engine. Nobody browses to page 500 of an order list; they search.

MariaDB's reports are three to four times the other two, 178 ms against 53 ms and 44 ms. Both are one GROUP BY product_id over the 469,000 rows of kahunacart_product_stats, and EXPLAIN confirms MariaDB uses the covering index like the other two. It reads that index more slowly per row, and there is no setting to change. The comparison-reporting table shows the same gap larger, because a filtered report reads orders instead of the rollup.

PostgreSQL needs its table statistics, and after a bulk seed it does not have them yet. Straight after bench:seed, catalog listing, three facets was 56.29 ms; after a VACUUM ANALYZE (2.6 seconds by hand) it was 2.02 ms, and everything else came down 10 to 20% too. Autovacuum had not caught up with a quarter of a million rows written in eighty seconds. The PostgreSQL column above is measured after it. ANALYZE TABLE on MariaDB moved nothing, and SQLite has no statistics unless you ask for them, which the plugin never does.

So on PostgreSQL, run VACUUM ANALYZE between bench:seed and bench:queries, or wait for autovacuum. On the other two, seed and measure.

The determinism contract

One integer and one instant decide everything the seeder writes: --seed and --now. Nothing in the seeder calls a random or a clock function, so:

1. The same options produce the same store. Same product slugs, same order numbers, same totals, same hashes, on SQLite, MySQL and PostgreSQL alike.
2. The engines are comparable. Seed all three from the same seed and any difference in the timings is the engine, not the data.
3. Releases are comparable. Re-seed after a release with the same --seed and --now, and the numbers either side measure the same store.

Each phase draws from its own stream derived from the seed, so raising --orders leaves the catalog byte-identical.

Both --json documents carry the plugin version, the git commit, the engine and its version, the PHP version, the seed and the instant.

What the benchmark marks as its own

There is no migration and no marker column. Every row the seeder writes carries a recognizable name:

--purge removes exactly those, children first, and leaves anything else untouched.

bench:seed refuses to run when it finds orders that are not its own, unless you pass --force.

Turn a benchmark into a decision

The answer is not the same for every store.

• SQLite needs no server, is the quickest on the small reads a storefront makes most of, and is the right answer for a store whose traffic one machine can hold.
• PostgreSQL is the most even of the three. It leads on the catalog listing and on everything that reads the whole order history. Pick it if your merchants live in the Analytics screen with filters on, and give it a VACUUM ANALYZE after a bulk import.
• MySQL and MariaDB are the pragmatic answer where a MySQL server already exists and is backed up. They keep up on fourteen of the eighteen queries; where they do not is reporting and a deep order page.

Take your own numbers before you decide. A store with 2,000 orders and a store with 250,000 do not have the same answer, and neither does a box with a slow disk. Size the database server to the data first: see Performance.

Related

• CLI: every bin/plugin kahunacart command with its flags.
• Configuration: the database options, including the three engines.
• Orders and the admin: the three-axis order status the seeded orders carry, and how the order search is indexed.
• Reports and the rollup: the daily rollup the report timings above are measured against.
• Analytics: the comparison, filter and breakdown paths the second table measures.
• Performance: the settings that decide how fast the storefront answers, and what to tune before you rerun.
• Troubleshooting: what to check when a query is slow on a real store.
• Published benchmarks: the numbers we take from these commands, rerun after every development wave.