{"article":{"slug":"jepsen-particledb-1-1-4","title":"Jepsen: ParticleDB 1.1.4","subtitle":null,"summary":"Kyle Kingsbury's Jepsen analysis of ParticleDB, a multi-model database for AI workloads, reports 96 issues across versions 1.1.4 to 1.1.18.11.rc-22, from crashes and data loss to Lost Update under serializable, and finds its \"Full ACID\" claims premature. The work was funded by Orkes.","content_type":"research","language":"en","canonical_url":"https://jepsen.io/analyses/particledb-1.1.4","author":{"name":"Kyle Kingsbury","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Jepsen","url":"https://jepsen.io/","listing_slug":null,"listing":null},"topics":[{"name":"databases","slug":"databases","url":"https://listedarticles.com/topics/databases"},{"name":"research","slug":"research","url":"https://listedarticles.com/topics/research"},{"name":"infrastructure","slug":"infrastructure","url":"https://listedarticles.com/topics/infrastructure"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":17232,"reading_minutes":75,"published_at":"2026-10-31T00:00:00.000Z","added_at":"2026-10-31T17:14:11.509Z","updated_at":"2026-10-31T17:14:11.509Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/jepsen-particledb-1-1-4","markdown_url":"https://listedarticles.com/articles/jepsen-particledb-1-1-4.md","example":false,"citation":"Kyle Kingsbury, Jepsen. \"Jepsen: ParticleDB 1.1.4.\" 31 Oct 2026. https://jepsen.io/analyses/particledb-1.1.4 (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://jepsen.io/analyses/particledb-1.1.4"},"body_markdown":"# Jepsen: ParticleDB 1.1.4\n\nKyle Kingsbury, 2026-10-31\n\n[ParticleDB](https://particledb.ai) is a multi-model\ndatabase designed for AI workloads. We tested versions 1.1.4 through\n1.1.18.11.rc-22 and found that ParticleDB’s claims of “Full ACID”\nwere somewhat premature. We present 96 issues in ParticleDB, ranging\nfrom the pedestrian (crashes, deadlocks, data loss, etc.) to the\npiquant (partially-invisible rows which could not be deleted,\n`WHERE 4 = 5`, dropped tables re-emerging in distorted\nforms, confused columns, strings turning into integers, and so on).\nMost issues required just a single server, single client, and no\nfaults. By version 1.1.18.11.rc-22, all but ten of those issues\nappeared resolved. However, ParticleDB continued to lose committed\nwrites during normal operation with the default `occ`\ntransaction mode, and allowed internal anomalies and Lost Update\nwith `serializable`, the strongest setting. Orkes, the\nmakers of ParticleDB, have since released version 1.1.19-rc.13; [they\nclaim](https://orkes.io/blog/particledb-database-built-for-ai-agents) that it resolves all remaining issues discussed in this\nreport. This research was funded by Orkes, Inc., and conducted in\naccordance with the [Jepsen ethics\npolicy](https://jepsen.io/analyses/ethics).\n\n# 1 Background\n\nModern applications often spread state across a variety of systems:\ndatabases, caches, queues, data warehouses, search indices, and so on.\nCoordinating these systems is challenging: caches are often stale, the\nanalytics platform might lag days behind the primary database, the\nsearch index might lose data, and transactional updates across disparate\nsystems [remains\nan open research problem](https://www.semanticscholar.org/paper/Typhon%3A-Consistency-Semantics-for-Data-Processing-Arora-Nawab/018ce7bfc890712fa62ab988fe0703bccb806907).\n\nParticleDB[1](#fn1) aims to unify these workloads in a\nsingle engine with a special focus on AI agents and applications. It\noffers transactional SQL, a key-value store, vector search, live\nanalytics, and user-defined functions called *Particles*.[2](#fn2) To integrate with existing systems,\nclients can connect through [a variety of\ninterfaces](https://particledb.ai/connectivity/overview/), including the Postgres and Redis protocols, as well as\nits own gRPC and HTTP2 APIs. Written from the ground up in Rust,\nParticleDB claims to be orders of magnitude faster than [Postgres](https://www.postgresql.org/) while preserving the\nsame safety characteristics.\n\nParticleDB was built by [Orkes](https://orkes.io/) to\nsupport their workflow orchestration platform, [Conductor](https://jepsen.io/analyses/orkes-conductor-5.3.2).\nConductor was initially built to store data in Redis and Postgres, but\nto improve performance, Orkes opted to write their own database.\nParticleDB 1.0.0 offered a limited feature set specifically for\nConductor’s queries, and was deployed as the backing database for some\nof their Conductor clusters. A few Orkes customers followed suit. In\npreparation for a broader release, Orkes undertook a major refactoring\nof ParticleDB, adding a custom wire protocol, more datatypes, additional\nOLAP queries, and more. While Orkes and their customers continued to run\n1.0.0, they soft launched the refactored versions on [particledb.ai](https://particledb.ai). Orkes engaged Jepsen to\ntest these versions of ParticleDB, beginning with 1.1.4.\n\nWhile ParticleDB could run as a single node, it also offered a\nreplicated cluster using the [Raft](https://raft.github.io/)\nconsensus protocol. This replication feature was available only in\nenterprise cloud deployments. The builds Orkes provided did not support\nreplication, so this report covers only single-node behavior.\n\n## 1.1 Safety\n\nParticleDB’s home page [advertised](https://web.archive.org/web/20260518203012/https://particledb.ai/)\nthat ParticleDB was “formally proven” with a TLA+ proof. However, during\nour collaboration no TLA+ specification was available. It is unclear\nwhat properties might be proven, or whether they described ParticleDB as\nbuilt.\n\nParticleDB offered [three\ntactics](https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#wal-sync-modes) for syncing its Write-Ahead Log (WAL). The `sync`\nmode called `fsync` on each commit, `groupsync`\nbatched multiple transactions into a single `fsync`, and\n`nosync` skipped `fsync` altogether. The [Transaction\nEngine documentation](https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#wal-sync-modes) described `sync` as offering\n“stronger durability” than `groupsync`. On the other hand,\n`particledb start --help` said the `groupsync`\nmode ensured that “COMMIT waits for flush”, and Orkes’ internal operator\nguide concurred: `groupsync` offered the “same durability as\n`sync`”.\n\nParticleDB’s home page proudly claimed “Full ACID. 159K TPS.”\nHowever, the text immediately below this said “snapshot isolation”,\nwhich is [not fully\nisolated](https://jepsen.io/consistency/models/snapshot-isolation).\n\nMost SQL databases use the\n`SET TRANSACTION ISOLATION LEVEL` statement to control the\nconcurrent semantics of a transaction. However, ParticleDB [ignored\nor rejected these statements](https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#transaction-modes). In some cases it instructed users to\nchoose `READ COMMITTED`, a relatively weak isolation level.\nIn other cases it accepted requests for different isolation levels, and\nsilently ignored them (despite documentation to the contrary).\n\nInstead, transaction isolation was governed by the\n`--txn-mode` setting passed at startup. The transaction\nengine documentation [described\nfour modes](https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#transaction-modes):\n\n| Mode | Behavior | When to use |\n| --- | --- | --- |\n| `fast` | No locks; commit-on-write | Bulk loads, single-writer benchmarks |\n| `occ` | Overlay + commit-time OCC validation (default) | General OLTP — safe under concurrency |\n| `zero-copy-occ` | Partition-pinned OCC; single-partition txns skip cross-partition coordination | High-throughput OLTP on partition-affine workloads (TPC-C, key-value) |\n| `table-2pl` | Exclusive table locks at first touch | Strictest serialization, simple semantics |\n\nThis table was difficult to interpret. While the [best\npractices documentation](https://web.archive.org/web/20260518193127/https://particledb.ai/sql/transactions/#best-practices) warned that users should “pick\n`--txn-mode` deliberately”, since “it changes correctness and\nthroughput characteristics more than SQL text does”, the documentation\ndid not explain what these modes actually guaranteed. The\n`fast` mode was clearly intended for weaker isolation, but\nwas that Read Atomic, Read Uncommitted, or something else? The\n`occ` mode was similarly unclear. “Commit-time OCC\nvalidation” could describe a broad spectrum of consistency models from\nRead Uncommitted to Strong Serializable, but “safe under concurrency”\nsuggests it might be Serializable. The `zero-copy-occ` mode\ndescribed a performance optimization which could be as safe as\n`occ`, but it was *not* described as “safe under\nconcurrency”—raising the question: in what ways was it unsafe? Finally,\nif `table-2pl` offered the “strictest serialization”, was\n`occ` some weaker isolation level?\n\nThe [CLI\nreference page](https://web.archive.org/web/20260518193140/https://particledb.ai/operations/cli-reference/) contained another table which described\n`occ` and `table-2pl` as “Jepsen-safe”, whereas\n`fast` was not “Jepsen-safe”, but did not explain their\nsafety properties. Since Jepsen evaluates safety with respect to\ndocumentation and marketing claims, it is not clear how to interpret\nthese terms.[3](#fn3)\n\nRunning `particledb start --help` provided the following\ntable:\n\n| Mode | Description |\n| --- | --- |\n| `fast` | No locks, maximum OLTP throughput (unsafe under concurrency) |\n| `occ` | Table locks + commit-time validation (safe for most OLTP) |\n| `row-2pl` | Alias for `occ` |\n| `zero-copy-occ` | Partition-pinned OCC (highest throughput safe mode) |\n| `table-2pl` | Exclusive table locks, full serialization (strictest) |\n| `serializable` | Alias for `table-2pl` |\n| `serializable-ssi` | Adaptive SSI (Cahill 2008), PL-3 with optimistic semantics (v1 scaffolding — see SSI\\_ADAPTIVE\\_DESIGN.md) |\n\nThe point of Optimistic Concurrency Control (OCC) is generally to\n*avoid* taking locks, so the mention of table locks and a\n`row-2pl`[4](#fn4) alias was surprising. Moreover,\n“safe for most OLTP” undermines the claim that `occ` was\n“safe under concurrency”; it suggests there are some OLTP workloads\nwhich are unsafe!\n\nOrkes explained that this documentation was wrong: `occ`\nwas actually optimistic, and the `row-2pl` alias was to\npreserve compatibility with an earlier two-phase locking mechanism. They\nalso provided an unpublished operator guide with detailed descriptions\nof the various transaction modes. This document was [difficult\nto interpret](https://www.tiktok.com/@charlixcx/video/7603137654682570006), but did specify the following consistency models:\n\n| Mode | Consistency Model |\n| --- | --- |\n| `fast` | Unknown |\n| `occ-v3` (a.k.a. `occ`, `occ-v2`, `row-2pl`) | Snapshot Isolation |\n| `zero-copy-occ` (a.k.a. `zocc`) | Snapshot Isolation |\n| `serializable-ssi` (a.k.a. `ssi`) | Serializable |\n| `table-2pl` (a.k.a. `serializable`) | Strong Serializable |\n\nSources disagreed about the default transaction and WAL sync modes,\nbut in general, `occ-v3` and `groupsync` appeared\nto be the default.\n\n# 2 Test Design\n\nWe wrote a [test\nsuite](https://github.com/jepsen-io/particledb) for ParticleDB using the [Jepsen testing library](https://github.com/jepsen-io/jepsen).\nWe ran ParticleDB versions 1.1.4 through 1.1.18.11.rc-22 on individual\nDebian 13 nodes, both in containers and EC2 VMs. Our test harness could\nsimulate process crashes by killing ParticleDB, simulate the loss of\nun-`fsync`ed writes using [LazyFS](https://github.com/dsrhaslab/lazyfs), and truncate or\ncorrupt disk files. We ran ParticleDB using each transaction mode and\nall three WAL sync modes.\n\nWe first tried ParticleDB’s official Java SDK. While the [documentation\nindicated](https://web.archive.org/web/20260518203023/https://particledb.ai/sdks/java/) that the library was available as a Maven dependency,\nneither the JAR nor source code were actually available. Instead, we\nbegan with the official Postgres JDBC client, version 42.7.11, and later\nadded support for the [ParticleDB Java SDK](https://particledb.ai/sdks/java/),\nversions 1.0.0 through 1.0.2.\n\n## 2.1 Workloads\n\nWe used several Jepsen workloads from other SQL databases, and\ndesigned new ones specifically for ParticleDB.\n\nThe [`rw`\nworkload](https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/rw.clj) performed randomly generated transactions involving reads\nand writes of integer values identified by unique integer keys. We\nstored each value in a single row and accessed them both via primary and\nunindexed secondary keys. Keys and values were encoded as a variety of\ndatatypes, including integers, decimals, and strings. We assumed that\nwrites followed reads within each transaction, and that individual keys\nwere sequentially consistent, and used these assumptions to infer\nconstraints on the [version orders and\ntransaction dependency graphs](https://jepsen.io/consistency/dependencies). Jepsen’s transactional checker, [Elle](https://github.com/jepsen-io/elle), then searched for\ncycles in that dependency graph, as well as a variety of non-cyclic\nphenomena like [G1a\n(Aborted Read)](https://jepsen.io/consistency/phenomena/g1a).\n\nThe [`append`](https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/append.clj)\nworkload also performed randomly generated transactions over rows\nidentified by unique integer keys, but the values were lists of\nintegers. Transactions could read one of these lists by key, or append a\nunique integer to a list by key. We encoded each list as a\ncomma-separated `text` field, and used SQL\n`concat` to append to them. Rows could be selected via\nprimary key or unindexed secondary key, or via a join to an indirection\ntable. From there, we used Elle to infer version orders and dependency\ngraphs, and searched for a variety of cyclic and non-cyclic\nphenomena.\n\nWhen these workloads revealed consistency violations within a single\ntransaction, we designed an [`internal`](https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal.clj)\nworkload to stress internal consistency. This workload performed a\nseries of randomly generated transactions involving `INSERT`,\n`UPDATE`, `DELETE`, and `SELECT`\nstatements over a simple key-value table, and verified that within each\nindividual transaction, reads observed prior writes.\n\nTo investigate tantalizing anomalies in the `internal`\nworkload, we designed the [`compare`](https://github.com/jepsen-io/particledb/blob/e804ab8b0528b089b908b6249d2e1e4ac966802f/src/jepsen/particledb/workload/compare.clj)\nworkload. This workload used Clojure’s [test.check](https://github.com/clojure/test.check) generative\ntesting library to [produce](https://github.com/jepsen-io/particledb/blob/e804ab8b0528b089b908b6249d2e1e4ac966802f/src/jepsen/particledb/gen/tc.clj)\nrandom tables and simple SQL statements over them, then executed those\nstatements against both ParticleDB and PostgreSQL. If ParticleDB’s\nbehavior was [not\nequivalent](https://github.com/jepsen-io/particledb/blob/e804ab8b0528b089b908b6249d2e1e4ac966802f/src/jepsen/particledb/workload/compare.clj#L71-L238) to PostgreSQL’s, the compare workload would [search](https://github.com/jepsen-io/particledb/blob/d4721aeef9435af0131eacfb2163059272b13d57/src/jepsen/particledb/gen/tc.clj#L1195-L1234)\nfor a simpler sequence of statements that could reproduce the same\ndivergence. Although our SQL generator was relatively simple—just a few\nbasic datatypes, literals, arithmetic, comparisons, boolean logic,\ntransactions, and some simple `LIMIT`, `OFFSET`,\nand `ORDER BY` clauses—it proved remarkably fruitful.\n\nDrawing on these successes, we devised a hybrid [`internal-sim`](https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal_sim.clj)\nworkload to search for violations of internal consistency in more\ncomplex transactions. This workload [generated](https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/gen.clj)\na random set of tables, then concurrently executed [random\ntransactions](https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal_sim.clj#L370-L406) over them. It then [checked\neach transaction](https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal_sim.clj#L115-L368) independently to ensure that each of its statements\nobserved results consistent with those preceding. For example, if a\ntransaction executed `DELETE ... WHERE 2 < c`, then\nobserved a row with `c = 3`, we could prove that transaction\nwas not internally consistent.\n\n# 3 Results\n\nWe found 96 issues with ParticleDB, which we present in several\ncategories below. All but three of these issues occurred in healthy,\nsingle-node instances of ParticleDB. Busy readers may wish to jump to\nthe [Discussion](#discussion) section, which summarizes our\nfindings.\n\n- [Availability and\n  Performance](#availability-and-performance)\n- [Broken Features](#broken-features)\n- [Predicates](#predicates)\n- [Constraints](#constraints)\n- [Joins](#joins)\n- [Zombies](#zombies)\n- [State Corruption](#state-corruption)\n- [Transactions and\n  Concurrency](#transactions-and-concurrency)\n- [With Faults](#with-faults)\n\nAdditional results are presented in [Appendix\nA](#appendix-a):\n\n- [Type Safety and Syntax](#type-safety-and-syntax)\n- [Arithmetic](#arithmetic)\n- [Strings](#strings)\n\n## 3.1 Availability and Performance\n\nUnder normal operation, ParticleDB was slow, deadlocked, and crashed\nwithout warning.\n\n### 3.1.1 Abysmal Performance With `table-2pl` (#62)\n\nIn ParticleDB 1.1.4 through 1.1.18.4,\n`--txn-mode table-2pl` was so slow as to be essentially\nunusable. While some transactions might complete in a few milliseconds,\nmost took 10, 20, or even 30 seconds, then failed.\n\nFor example, take [this\ntest run](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260608T155928-table-2pl-perf.zip) with a single node running 1.1.18.4, transaction mode\n`table-2pl`, and WAL mode `groupsync`. With ten\nconcurrent clients, most transactions blocked for tens of seconds before\nthrowing\n`could not serialize access due to concurrent update`, or an\nIO error.\n\nHigh latencies ([#62](https://github.com/particledbai/jepsen/issues/62))\nappeared improved by 1.1.18.6, which could execute transactions with\nsub-100 millisecond latencies for at least a few seconds.\n\n### 3.1.2 Mystery Crash (#81)\n\nWhile 1.1.18.6 improved latencies, it also crashed frequently.\nCrashes occurred spontaneously with various transaction modes, WAL\nmodes, and client interfaces; there was no obvious cause. Nodes would\nrun for a few hundred to a few thousand seconds before exiting. No error\nmessages were printed. These crashes ([#81](https://github.com/particledbai/jepsen/issues/81)) no\nlonger appeared in 1.1.18.11.rc-18.\n\n### 3.1.3 Deadlock After Two Transactions (#27)\n\nIn ParticleDB 1.1.18.8, submitting as few as two concurrent\ntransactions to a healthy, single-node deployment could cause it to\ndeadlock. This occurred both using the Postgres and ParticleDB\ninterfaces. The node would log an error message like\n`[DEADLOCK] 1 cycles detected at SystemTime ...`, and an\nempty backtrace. In some cases new clients could attach and perform\ntransactions; in others, new clients would also hang indefinitely. This\nmay reflect multiple deadlock bugs.\n\nWith the `append` workload, version 1.1.18.9 generally\ncould not run for more than 45 seconds without locking up. Version\n1.1.18.11.rc-1 could reliably run for two minutes without lockups, and\nwe declared the issue ([#27](https://github.com/particledbai/jepsen/issues/27))\nresolved.\n\n### 3.1.4 Deadlock With `table-2pl` (#71)\n\nWhile version 1.1.18.11.rc-1 fixed the most frequent deadlocks, we\ncontinued to find deadlocks through 1.1.18.11.rc-6 when running with\n`table-2pl`. An otherwise healthy ParticleDB node would\nexecute transactions for about 30 seconds, then get stuck forever. The\nnode would log [a\nsimilar deadlock error message](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T085629-1.1.18.11.rc-1-table-2pl-deadlock.zip) to #27, and all requests with the\nParticleDB client would throw `PDB JDBC: null` after a 100\nsecond default timeout. These deadlocks ([#71](https://github.com/particledbai/jepsen/issues/71))\nappeared resolved by 1.1.18.11.rc-7.\n\n## 3.2 Broken Features\n\nOrkes’ [public\ndocumentation](https://web.archive.org/web/20260519195820/https://particledb.ai/connectivity/pg-wire/) claimed broad support for Postgres features, and the\ninternal operator guide concurred:\n\n> The engine speaks PostgreSQL v3 wire protocol natively — anything\n> that speaks pgjdbc / libpq / psycopg2 / pq connects unchanged.\n> `BEGIN`, `COMMIT`, `ROLLBACK`, prepared\n> statements, parameterised queries, and extended-query pipelining all\n> work as expected.\n\nThese claims were somewhat optimistic. ParticleDB was missing support\nfor a wide array of Postgres features, and others were badly broken.\n\n### 3.2.1 No Aggregates Within `CASE` (#20)\n\nIn ParticleDB 1.1.4 through 1.1.18.4, aggregate expressions like\n`COUNT(*)` were not supported within `CASE`\nexpressions.\n\n```\nSELECT\n  CASE\n    WHEN COUNT(*) = 2\n      THEN 'CASE_COUNT_STAR_OK'\n      ELSE 'CASE_COUNT_STAR_BAD'\n  END\nFROM t;\n```\n\nThis threw `Error: cannot evaluate wildcard expression`.\nParticleDB gained support for this feature ([#20](https://github.com/particledbai/jepsen/issues/20)) by\nversion 1.1.18.6.\n\n### 3.2.2 Unsupported Exponentiation (#28)\n\nParticleDB 1.1.4 through 1.1.18.8 did not support the Postgres\nexponentiation operator:\n\n```\nroot=> SELECT 2 ^ 3;\nERROR:  unsupported binary operator: PGExp\n```\n\nVersion 1.1.18.9 added support for the `^` operator ([#28](https://github.com/particledbai/jepsen/issues/28)).\n\n### 3.2.3 Malformed Responses for Prepared Statements (#2)\n\nIn ParticleDB 1.1.4 through 1.1.18.9, any attempt to execute a\nprepared statement using the Postgres JDBC driver caused an exception\nlike\n`Execution error (NoSuchElementException) at java.util.ArrayDeque/removeFirst`.\nSince the JDBC driver automatically switches to prepared statements\nafter the first few calls to `execute()`, this rendered the\ndatabase essentially unusable. As a workaround, we disabled the client’s\nuse of prepared statements.\n\nPrepared statements are executed using Postgres’ [extended\nquery protocol](https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY), which takes place in several (potentially reusable)\nphases: Parse, Bind, Execute, and so on. Orkes believes that ParticleDB\nlikely failed to return a response for one of those phases, or returned\nan extra message, causing the client’s queue to underflow. By version\n1.1.18.11.rc-1, this issue ([#2](https://github.com/particledbai/jepsen/issues/2)) appeared\nfixed.\n\n### 3.2.4 Most Postgres Queries Fail (#83)\n\nIn ParticleDB 1.1.18.11.rc-7 through 1.1.18.11.rc-19, many queries\nusing the Postgres interface threw\n`Bad value for type long: some_column`. This rendered the\ndatabase largely unusable.\n\n```\nCREATE TABLE ints (a int);\nSELECT * FROM ints;\n  -> Bad value for type long : a\n```\n\nOrkes believes this issue ([#83](https://github.com/particledbai/jepsen/issues/83)) was\nfixed in 1.1.19-rc.12.\n\n### 3.2.5 Tuples Without Fields (#111)\n\nIn 1.1.18.11.rc-18 and rc-19, queries with the Postgres interface\noccasionally threw errors like\n`Received resultset tuples, but no field structure for them`.[5](#fn5) For example, take [this\ntest](https://github.com/user-attachments/files/30419491/20260727T025222.565-0500.zip) of version 1.1.18.11.rc-19, with no faults, a new\n`strict-serializable` transaction mode, and WAL mode\n`sync`. It executed a transaction like:\n\n```\nBEGIN;\nSELECT * FROM t_38;\nINSERT INTO u_38 (d, a, c, b) VALUES\n  (null, null, false, \"oudt1THxi\");\nINSERT INTO t_38 (d, b) VALUES\n  (false, \"BCczDbueXLz9zXnpgbdCeFbC5y6XSEys3\");\nCOMMIT;\n```\n\nThis complained of resultset tuples with no field structure,\nsuggesting an error in ParticleDB’s Postgres protocol state machine.\nWhile Jepsen has not tested this version, Orkes reports this issue ([#111](https://github.com/particledbai/jepsen/issues/111)) was\nfixed in version 1.1.19-rc.13.\n\n### 3.2.6 Broken Postgres Authentication (#1)\n\nThe Postgres wire protocol usually authenticates clients using a\nusername and password. ParticleDB had a `--pg-password`\noption, but 1.1.4 through 1.1.18.8 lacked any option to set the\nusername. [Some\nof the documentation](https://web.archive.org/web/20260519195820/https://particledb.ai/connectivity/pg-wire/) indicated the default username was\n`admin`, while [other](https://web.archive.org/web/20260518193025/https://particledb.ai/getting-started/quickstart/)\n[pages](https://web.archive.org/web/20260518203023/https://particledb.ai/sdks/java/)\nsaid it was `particledb`. Neither appeared to work; the only\noption was to use `--no-auth` to disable authentication\naltogether. Version 1.1.18.11.rc-1 introduced a\n`--pg-username` option ([#1](https://github.com/particledbai/jepsen/issues/1)).\n\n### 3.2.7 Java Client Could Not Execute Transactions (#3)\n\nVersion 1.0.0 of the official ParticleDB Java client refused every\nattempt to execute a transaction. Any call to begin a JDBC transaction,\nor to execute a `BEGIN` statement, would result in an error\nmessage:\n`PDB wire error 1064: SQL plan error: transaction/session statements do not produce a query plan`.\n\nParticleDB 1.1.4 could execute transactions via its Postgres\ninterface, but using the official Java SDK was essentially impossible.\nThis issue ([#3](https://github.com/particledbai/jepsen/issues/3)) seems to\nhave been resolved by ParticleDB 1.1.15.\n\n## 3.3 Predicates\n\nParticleDB struggled to interpret predicates. `WHERE`\nclauses failed to match rows they should have, or matched unexpected or\nnonexistent rows. These issues occurred with a single server, single\nclient, and no faults.\n\n### 3.3.1 `SELECT` Ignores `WHERE` Clause (#40)\n\nIn ParticleDB 1.1.4 through 1.1.18.9, `SELECT` queries\nwithout a table accepted a `WHERE` clause, but ignored\nit.\n\n```\nroot=> SELECT 5 WHERE FALSE;\n 5\n---\n 5\n(1 row)\n```\n\nThis should have returned zero rows. Version 1.1.18.11.rc-1 seems to\nhave resolved this issue ([#40](https://github.com/particledbai/jepsen/issues/40)).\n\n### 3.3.2 `WHERE 1 < (a + 2)` (#47)\n\nVersion 1.1.1.18.9 introduced a new problem with predicates: inside a\ntransaction, a `WHERE` clause which performed sufficiently\ncomplex floating-point operations would silently fail to match any rows.\nFor example:\n\n```\nroot=> CREATE TABLE t (a real);\nroot=> INSERT INTO t (a) VALUES (3);\nroot=> BEGIN;\nroot=*> SELECT * FROM t WHERE 1 < (a + 2);\n a\n---\n(0 rows)\n```\n\nThe same problem applied to `SELECT`, `UPDATE`,\nand `DELETE`: predicates like\n`WHERE (a * 2) IS NOT NULL` would silently fail to match.\nHowever, simpler predicates like `WHERE a IS NOT NULL` would\nmatch correctly. The effect disappeared if the column was an integer.\nMoreover, some predicates which would throw errors outside a transaction\nwould silently match zero rows within a transaction, suggesting these\npredicates were not being evaluated at all:\n\n```\nroot=> SELECT * FROM t WHERE 1 < (5.0 % 3);\nERROR:  internal error\nroot=> BEGIN;\nroot=*> SELECT * FROM t WHERE 1 < (5.0 % 3);\n a\n---\n(0 rows)\n```\n\nThis issue ([#47](https://github.com/particledbai/jepsen/issues/47))\nappeared resolved by 1.1.18.11.rc-1.\n\n### 3.3.3 `WHERE 4 = 5` (#18)\n\nParticleDB 1.1.4 through 1.1.17 aggressively interpreted\n`WHERE something = x` as a primary-key lookup of\n`x`, regardless of the value of `something`.\n\n```\nroot=> CREATE TABLE t\n       (a int primary key, b int);\nroot=> INSERT INTO t VALUES (4, 5), (5, 6);\nroot=> SELECT * FROM t WHERE b = 4;\n a | b\n---+---\n 4 | 5\n```\n\nParticleDB would ignore a row with `b = 5` in favor of\n`a = 5`:\n\n```\nroot=> SELECT * FROM t WHERE b = 5;\n a | b\n---+---\n 5 | 6\n(1 row)\n```\n\nYou could write predicates over non-existent columns:\n\n```\nroot=> SELECT * FROM t\n       WHERE definitely_not_a_column = 4;\n a | b\n---+----\n 4 | 5\n(1 row)\n```\n\nOr ask if `4 = 5`:[6](#fn6)\n\n```\nroot=> SELECT * FROM t WHERE 4 = 5;\n a | b\n---+----\n 5 | 6\n(1 row)\n```\n\nBy version 1.1.18.4, ParticleDB had learned that not every equality\ncomparison involved a primary key ([#18](https://github.com/particledbai/jepsen/issues/18)).\n\n### 3.3.4 `SELECT` in a Transaction Returns Nonsense (#35)\n\nThe behavior of `SELECT` within a transaction in\nParticleDB versions 1.1.4 through 1.1.18.8 was thoroughly confusing.\nConsider a single client connected to a fresh single-node instance of\nParticleDB 1.1.18.8. It creates a simple table of integers with two\nrows, `1` and `2`, and begins a transaction:\n\n```\nroot=> CREATE TABLE t (a INT UNIQUE);\nroot=> INSERT INTO t VALUES (1), (2);\nroot=> BEGIN;\nroot=*> SELECT * FROM t;\n a\n---\n 1\n 2\n(2 rows)\n```\n\nSo far, so good. But now consider:\n\n```\nroot=*> SELECT * FROM t WHERE a = 1;\n a\n---\n 2\n(1 row)\n\nroot=*> SELECT * FROM t WHERE a = 3;\n a\n---\n 2\n(1 row)\n\nroot=*> SELECT * FROM t WHERE a IN (1, 2);\n a\n---\n 2\n(1 row)\n\nroot=*> SELECT * FROM t WHERE FALSE;\n a\n---\n 2\n(1 row)\n```\n\nOne might conclude that every `SELECT` with a predicate\nreturned a single row from the requested table—perhaps the most recently\ninserted row. But this is not true, because many predicates evaluating\nto `TRUE` returned nothing.\n\n```\nroot=*> SELECT * FROM t WHERE TRUE;\n a\n---\n(0 rows)\n\nroot=*> SELECT * FROM t WHERE 3 = 3;\n a\n---\n(0 rows)\n```\n\nThis behavior ([#35](https://github.com/particledbai/jepsen/issues/35)) seemed\nimproved by ParticleDB 1.1.18.9.\n\n## 3.4 Constraints\n\nParticleDB often violated constraints like `NOT NULL` or\n`UNIQUE`, even with a single server, single client, and no\nfaults.\n\n### 3.4.1 Not-Null Constraints Allow Null (#8)\n\nWhile ParticleDB allowed users to add `NOT NULL`\nconstraints to columns, versions 1.1.4 through 1.1.17 ignored them,\nresulting in bizarre states.\n\n```\nroot=> CREATE TABLE t (a BOOLEAN NOT NULL);\nroot=> INSERT INTO t (a) VALUES (NULL);\nINSERT 0 1\n```\n\nWhile the resulting row existed in some sense, it was partly\ninvisible.\n\n```\nroot=> SELECT * FROM t;\n a\n---\n(0 rows)\n\nroot=> SELECT COUNT(*) FROM t;\n COUNT\n-------\n     1\n(1 row)\n\nroot=> SELECT COUNT(*) FROM t WHERE a IS NULL;\n COUNT\n-------\n     0\n(1 row)\n```\n\nIn version 1.1.18.4, ParticleDB started to enforce null constraints\n([#8](https://github.com/particledbai/jepsen/issues/8)).\n\n### 3.4.2 Unique Constraints Allow Duplicates (#9)\n\nSimilarly, while ParticleDB allowed users to add unique constraints\nto columns, versions 1.1.4 through 1.1.18.4 simply ignored them.\n\n```\nroot=> CREATE TABLE t (a boolean UNIQUE);\nroot=> INSERT INTO t (a) VALUES (TRUE);\nINSERT 0 1\nroot=> INSERT INTO t (a) VALUES (TRUE);\nINSERT 0 1\nroot=> SELECT * FROM t;\n a\n---\n t\n t\n(2 rows)\n```\n\nIn version 1.1.18.6, ParticleDB partially enforced unique constraints\n([#9](https://github.com/particledbai/jepsen/issues/9)).\n\n### 3.4.3 Duplicate Primary Keys (#29)\n\nAlthough 1.1.18.6 made `UNIQUE` constraints somewhat more\nrobust, 1.1.4 through 1.1.18.8 allowed an `UPDATE` to produce\nmultiple rows with the same primary key.\n\n```\nroot=> CREATE TABLE t (id text PRIMARY KEY,\n                       desc text);\nroot=> INSERT INTO t VALUES ('a', 'a');\nroot=> INSERT INTO t VALUES ('b', 'b');\nroot=> UPDATE t SET id = 'c';\nUPDATE 2\n```\n\nSelecting a row by primary key returned only one of the two rows, but\nother predicates could return both:\n\n```\nroot=> SELECT * FROM t WHERE id = 'c';\n id | desc\n----+------\n c  | b\n(1 row)\n\nroot=> SELECT * FROM t WHERE id LIKE 'c';\n id | desc\n----+------\n c  | a\n c  | b\n```\n\nThis issue ([#29](https://github.com/particledbai/jepsen/issues/29)) appears\nto have been fixed by 1.1.18.9.\n\n### 3.4.4 Null Primary Keys (#57)\n\nPer the SQL standard, primary keys may not be null. This makes\nintuitive sense, since `NULL` represents an unknown value.\nParticleDB would not allow you to insert null values for primary keys.\nHowever, versions 1.1.4 through 1.1.18.11.rc-1 were perfectly happy to\n*update* the primary key of one (or more!) rows to\n`NULL`.\n\n```\nroot=> CREATE TABLE t (a int PRIMARY KEY);\nroot=> INSERT INTO t (a) VALUES (3), (4);\nroot=> UPDATE t SET a = NULL;\nroot=> SELECT * FROM t;\n a\n---\n\n(2 rows)\n```\n\nParticleDB 1.1.18.11.rc-3 rejected attempts to set primary keys to\n`NULL` ([#57](https://github.com/particledbai/jepsen/issues/57)).\n\n### 3.4.5 Primary Key Conflicts with Deleted Rows (#82)\n\nIn ParticleDB 1.1.18.4 through 1.1.18.11.rc-11, a transaction which\ndeleted a row, then inserted a new row with the same primary key, would\nabort due to a primary key constraint violation. For example:\n\n```\nCREATE TABLE t (a int PRIMARY KEY);\nINSERT INTO t (a) VALUES (7);\nBEGIN;\nDELETE FROM t;\nINSERT INTO t (a) VALUES (7);\nCOMMIT;\n```\n\nThe second `INSERT` would throw\n`PDB wire error 1064: SQL execution error: duplicate key value violates unique constraint (key=7)`,\nbut only when using the ParticleDB interface—the Postgres interface was\nunaffected. Orkes partially addressed this issue ([#82](https://github.com/particledbai/jepsen/issues/82)) in\nversion 1.1.18.11.rc-9, and fixed another case of it in\n1.1.18.11.rc-18.\n\n### 3.4.6 Lack of Primary Key Conflicts with Extant Rows (#90)\n\nIn ParticleDB 1.1.18.11.rc-9, a transaction executing concurrently\nwith other transactions could read a row with a primary key, then\nsuccessfully insert a row with that same primary key. For example, take\n[this\n`internal-sim` test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260618T140338-1.1.18.11.rc-9-internal.zip), which created a table with two\ninteger columns:\n\n```\nCREATE TABLE u_14 (b int, a int PRIMARY KEY);\n```\n\nA transaction ran `SELECT * FROM u_14` and observed\nseveral rows, including `{:b 1020026655, :a -1}`. It then ran\n`INSERT INTO u_14 (b, a) VALUES (-27, -1)`, which should have\nconflicted with the extant primary key, but succeeded instead. It went\non to `SELECT * FROM u_14` again, and observed the newly\ninserted row, but not the previously existent one. This is a violation\nof internal consistency.\n\nAs of version 1.1.18.11.rc-19, we no longer observed these internal\nconsistency violations ([#90](https://github.com/particledbai/jepsen/issues/90)).\n\n## 3.5 Joins\n\nParticleDB struggled to join tables together. The following issues\noccurred with a single server, single client, and no faults.\n\n### 3.5.1 Can’t Join Integers to Floats (#58)\n\nEven after agreeing that `2.0 = 2`, ParticleDB 1.1.4\nthrough 1.1.18.9 refused to join two tables on an `int` and a\n`real` column:\n\n```\nroot=> CREATE TABLE t (a real);\nroot=> CREATE TABLE u (a int);\nroot=> INSERT INTO t (a) VALUES (2.0);\nroot=> INSERT INTO u (a) VALUES (2);\nroot=> SELECT t.a FROM t JOIN u ON u.a = t.a;\nERROR:  join key must be integer type\n```\n\nDatabases like Postgres would return a single row here, since\n`2.0 = 2`. ParticleDB gained the ability to join floats to\nintegers ([#58](https://github.com/particledbai/jepsen/issues/58)) by\n1.1.18.11.rc-1.\n\n### 3.5.2 Incorrect Column Names (#54)\n\nBecause two tables can have a column with the same name, SQL\ndatabases generally return both table and column names, so you can tell\nwhich column belonged to which table. ParticleDB, with both the\nParticleDB and Postgres interfaces, neglected to return table names.\nInstead, versions 1.1.4 through 1.1.18.11.rc-1 renamed column\n`a` in joined tables to `right.a`,\n`right.right.a`, and so on. Consider the following simple\njoin across two tables `t` and `u`, both with an\ninteger column `a`:\n\n```\nSELECT * FROM t FULL OUTER JOIN u\n  ON t.a = u.a;\n```\n\nIn Postgres this produced rows like:\n\n```\n[{:t/a 1, :u/a 1.0}\n {:t/a 2, :u/a 2.0}]\n```\n\nHere the expression `:t/a` denotes column `a`\nof table `t`. In ParticleDB, this same query returned:\n\n```\n[{:a 1, :right.a 1.0}\n {:a 2, :right.a 2.0}]\n```\n\nThe table names are completely missing, and ParticleDB has altered\nthe name of column `a` to `right.a`. Programs\nwhich expected typical SQL behavior would throw an error as they\nattempted to read columns which were not present, or could silently read\nnull values.\n\nParticleDB 1.1.18.11.rc-3 no longer munged columns, but the database\nstill omitted table names. The resulting name collisions destroyed\ninformation.\n\n```\n[{:a 1}\n {:a 2}]\n```\n\nWhile Jepsen has not had a chance to test this version, Orkes claims\nthis issue ([#54](https://github.com/particledbai/jepsen/issues/54)) was\nresolved by version 1.1.19-rc.13.\n\n### 3.5.3 Coercing to Strings (#76)\n\nIn ParticleDB 1.1.4 through 1.1.18.11.rc-19, joins would coerce some\nvalues to strings in ParticleDB’s Postgres interface. For example,\nconsider the following join:\n\n```\nCREATE TABLE t (a int);\nCREATE TABLE u (a real);\nINSERT INTO u (a) VALUES (3.2);\nINSERT INTO t (a) VALUES (5);\nSELECT u.a FROM t CROSS JOIN u;\n```\n\nWith the ParticleDB interface, this produced the number\n`3.2`. With the Postgres interface, it produced the string\n`\"3.2\"`. This issue ([#76](https://github.com/particledbai/jepsen/issues/76)) seemed\nfixed by 1.1.18.11.rc-21.\n\n### 3.5.4 Missing Rows (#59)\n\nIn ParticleDB 1.1.4 through 1.1.18.11.rc-1, outer joins of a\nnon-empty table to an empty one often returned no rows, even though the\nquery should have produced all rows in the non-empty table. For\nexample:\n\n```\nroot=> CREATE TABLE t (a int);\nroot=> CREATE TABLE u (a int);\nroot=> INSERT INTO t (a) VALUES (1), (2);\nroot=> SELECT t.a AS t, u.a AS u\n       FROM t LEFT JOIN u ON u.a = t.a;\n t | u\n---+---\n(0 rows)\n```\n\nBy version 1.1.18.11.rc-3, ParticleDB correctly returned rows when\njoining to empty tables ([#59](https://github.com/particledbai/jepsen/issues/59)).\n\n### 3.5.5 Column Confusion (#77)\n\nMany tables in real databases contain columns with identical names,\nlike `id` or `created_at`. When asked to join two\ntables like this, ParticleDB 1.1.4 through 1.1.18.11.rc-6 became\nseverely confused, even when all columns were fully qualified. For\nexample:\n\n```\nroot=> CREATE TABLE t (a text);\nroot=> CREATE TABLE u (a text);\nroot=> INSERT INTO t VALUES ('this'), ('that');\nroot=> INSERT INTO u VALUES\n       ('is not the right column');\nroot=> SELECT t.a AS ta, u.a AS ua FROM t\n       CROSS JOIN u WHERE (u.a = 'this');\n  ta  |           ua\n------+-------------------------\n this | is not the right column\n(1 row)\n```\n\nThis should not have matched anything—there are no rows in table\n`u` where `u.a = 'this'`. Instead ParticleDB\nmatched a row in table `t` which satisfied\n`t.a = 'this'`, then cross-joined it to all rows in\n`u`. To add insult to injury, 1.1.4 through 1.1.18.9 appeared\nto swap around the output columns, producing the following table:\n\n```\n           ta            |  ua\n-------------------------+------\n is not the right column | this\n```\n\nIf column `t.a` was of a different type than\n`u.a`, ParticleDB would try to coerce the types into\ncompatible forms, producing bizarre error messages or delightfully weird\nresults. Version 1.1.18.11.rc-7 was much less confused when joining\ntables containing the same column name ([#77](https://github.com/particledbai/jepsen/issues/77)).\n\n## 3.6 Zombies\n\nParticleDB often wound up with rows that could not be deleted or\nupdated, or which persisted in some (perhaps mangled) form after their\ndeletion. These issues occurred with a single server, single client, and\nno faults.\n\n### 3.6.1 Total Recall (#26)\n\nIn ParticleDB 1.1.17 through 1.1.18.8, it appeared impossible to\ndelete rows within a transaction. For example:\n\n```\nroot=> CREATE TABLE t (a boolean);\nroot=> INSERT INTO t (a) VALUES (TRUE);\nroot=> BEGIN;\nroot=*> DELETE FROM t;\nDELETE 0\nroot=*> COMMIT;\nroot=> SELECT * FROM t;\n a\n---\n t\n(1 row)\n```\n\nWe were unable to find a predicate that allowed one to delete records\n([#26](https://github.com/particledbai/jepsen/issues/26)).\nVersion 1.1.18.9 allowed transactions to mostly[7](#fn7)\ndelete rows.\n\n### 3.6.2 Memento Delere (#45)\n\n1.1.18.9 resolved issue #26, but deleted rows still lingered on in\nsome capacity. While a transaction could not directly select deleted\nrows, it could observe them via aggregate functions and various\npredicates.\n\n```\nroot=> CREATE TABLE t (a boolean);\nroot=> INSERT INTO t (a) VALUES (TRUE);\nroot=> BEGIN;\nroot=*> DELETE FROM t;\nDELETE 1\nroot=*> SELECT COUNT(*) FROM t WHERE a = TRUE;\n COUNT\n-------\n     1\n(1 row)\n\nroot=*> SELECT COUNT(*) FROM t WHERE a = FALSE;\n COUNT\n-------\n     0\n(1 row)\n```\n\nThe row must still be present in some sense, because these aggregate\nqueries could observe its value via predicates. However, trying to read\nthe row directly returned nothing:\n\n```\nroot=*> SELECT a FROM t WHERE a = TRUE;\n a\n---\n(0 rows)\n```\n\nVersion 1.1.18.11.rc-1 no longer allowed shades of deleted records to\npersist in aggregate queries ([#45](https://github.com/particledbai/jepsen/issues/45)).\n\n### 3.6.3 Night of the Living Integers (#23)\n\nIn versions 1.1.4 through 1.1.17, updating an integer column could\nmake rows impossible to delete. For example:\n\n```\nroot=> CREATE TABLE t (i int);\nroot=> INSERT INTO t VALUES (0);\nroot=> UPDATE t SET i = 1.5;\nroot=> DELETE FROM t;\nDELETE 0\nroot=> SELECT * FROM t;\n i\n---\n 1\n(1 row)\n```\n\nWhile this row appears to contain the integer `1`, that\ninteger is *special*, since the `UPDATE` statement\nprovided the value `1.5`. This row cannot be deleted. It does\nappear in `SELECT`, and is apparently equal to\n`1`, rather than `1.5`. Oddly, updating a row by\nprimary key with the same `1.5` value did not create zombies;\nthe problem ([#23](https://github.com/particledbai/jepsen/issues/23)) was\nsomehow linked to predicates. ParticleDB 1.1.18.4 no longer exhibited\nthis behavior.\n\n### 3.6.4 Team Rocket (#84)\n\nTables with a `double precision` column suffered from a\nsimilar problem: they could be updated with integer values, but doing so\nwould produce zombie rows that could neither be updated nor deleted\nagain.\n\n```\nROOT=> CREATE TABLE t (a double precision);\nROOT=> INSERT INTO t (a) VALUES (1.0);\nROOT=> UPDATE t SET a = 2;\nROOT=> DELETE FROM t;\nDELETE 0\nroot=> SELECT * FROM t;\n a\n---\n 2\n(1 row)\n```\n\nIf the double precision column was a primary key, these updates would\nproduce duplicate copies of the updated row. This issue ([#84](https://github.com/particledbai/jepsen/issues/84))\naffected ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-8, and was fixed\nin 1.1.18.11.rc-9.\n\n### 3.6.5 A Ghost is Technically Not a Number (#78)\n\nIn ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-6, `NaN`\nvalues stored in `real` or `double precision`\ncolumns could neither be updated nor deleted.\n\n```\nDROP TABLE IF EXISTS t;\nCREATE TABLE t (a real);\nINSERT INTO t (a) VALUES ('NaN');\nUPDATE t SET a = 5.0;\n  -> #:next.jdbc{:update-count 0}\nDELETE FROM t;\n  -> #:next.jdbc{:update-count 0}\nSELECT * FROM t;\n  -> [{:a ##NaN}]\n```\n\nThis issue ([#78](https://github.com/particledbai/jepsen/issues/78)) did not\noccur with the `psql` shell, but did happen with both\nParticleDB and Postgres JDBC clients. Version 1.1.18.11.rc-7 seems to\nhave fixed the problem.\n\n### 3.6.6 Beached Strings (#74)\n\nIn ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-6, fixed-width\nstrings were unmodifiable unless created with exactly the right number\nof characters. While ParticleDB would correctly pad the string\n`'hi'` to `'hi '`, the resulting row could neither\nbe updated nor deleted.\n\n```\nroot=> CREATE TABLE t (a character(3));\nroot=> INSERT INTO t (a) VALUES ('hi');\nroot=> DELETE FROM t;\nDELETE 0\nroot=> UPDATE t SET a = 'sup';\nUPDATE 0\nroot=> SELECT * FROM t;\n  a\n-----\n hi\n(1 row)\n```\n\nIf a fixed-width column was used as a primary key, it could be\nupdated, but only once. Both old and new copies would appear as distinct\nrows.\n\n```\nCREATE TABLE t\n       (a character(3) PRIMARY KEY);\nINSERT INTO t (a) VALUES ('um');\nUPDATE t SET a = 'no';\nSELECT a FROM t;\n-> [{:a \"um \"} {:a \"no \"}]\n```\n\nAny further attempt to modify those rows would produce a duplicate\nkey error, despite all the values involved being distinct. Fixed-width\nstrings were also incomparable using the `=` operator.\nVersion 1.1.18.11.rc-7 resolved these issues ([#74](https://github.com/particledbai/jepsen/issues/74)).\n\n### 3.6.7 Pet Sematary (#46)\n\nIn SQL, the `DROP TABLE` statement destroys a table\ncompletely. In ParticleDB 1.1.18.9 through 1.1.18.11.rc-1, it left a\nshadow of the table behind, whose rows would reappear in later tables\nwith the same name and more-or-less compatible schemata. To demonstrate\nthis behavior, let us create a table with a single row:\n\n```\nroot=> CREATE TABLE t (a text, b real);\nroot=> BEGIN;\nroot=*> INSERT INTO t VALUES ('boo', 18.83);\nroot=*> COMMIT;\n```\n\nWe then drop the table, and demonstrate that it no longer exists:\n\n```\nroot=> DROP TABLE t;\nroot=> SELECT * FROM t;\nERROR:  table not found\n```\n\nIf we then create a new table with the same name and a somewhat\ncompatible schema, rows from the deleted table reappear in the new one.\nHowever, the resurrected rows are not exactly their former selves. The\nfloating-point value `18.83` has become the integer\n`19`.\n\n```\nroot=> CREATE TABLE t (a text, b int);\nroot=> BEGIN;\nroot=*> SELECT * FROM t;\n  a  | b\n-----+----\n boo | 19\n(1 row)\n```\n\nThis behavior ([#46](https://github.com/particledbai/jepsen/issues/46)) seemed\nlinked to transactions, and no longer appeared in 1.1.18.11.rc-3.\n\n## 3.7 State Corruption\n\nParticleDB had many ways to create values which could not be read,\nalter values unexpectedly, create rows with incorrect or missing\ncolumns, or otherwise corrupt logical state. The issues in this section\ninvolved only a single server, single client, and no faults.\n\n### 3.7.1 Unreadable Infinities (#15)\n\nIn versions 1.1.4 through 1.1.17, ParticleDB’s Postgres interface did\nnot correctly serialize floating-point values like\n`Infinity`. One could write infinite values, but trying to\nread them would cause the client to throw an error.\n\n```\nCREATE TABLE t (a real);\nINSERT INTO t (a) VALUES ('Infinity');\nSELECT * FROM t\n   => PSQLException: \"Bad value for type\n      double : inf\"\n```\n\nWith Postgres, this correctly returned the JVM’s\n`Double/INFINITY` value. `NaN` was serialized\ncorrectly. At the `psql` shell, stored values came back as\n`inf`, rather than `Infinity`. In version\n1.1.18.6, ParticleDB serialized infinite values correctly ([#15](https://github.com/particledbai/jepsen/issues/15)).\n\n### 3.7.2 `NaN` Sometimes Converted to 0.0 (#21)\n\nParticleDB 1.1.4 through 1.1.18.4 would sometimes (but not always!)\nconvert the floating-point value `NaN` to `0.0`.\nFor example:\n\n```\nroot=> CREATE TABLE t (a real);\nroot=> INSERT INTO t (a) VALUES (2.5);\nroot=> INSERT INTO t (a) VALUES ('NaN');\nroot=> SELECT * FROM t;\n  a\n-----\n 2.5\n   0\n(2 rows)\n```\n\nDelightfully, whether this `NaN` corruption occurred\ndepended on the order in which rows were inserted. If the first value\ninserted into the table was `NaN`, that and later\n`NaN`s would be preserved. Version 1.1.18.6 seemed to store\n`NaN` correctly ([#21](https://github.com/particledbai/jepsen/issues/21)).\n\n### 3.7.3 Assignment of Incompatible Types (#7)\n\nIn ParticleDB 1.1.4 through 1.1.18.4, you could update a column of\none type to a column of a different, incompatible type—producing values\nlike `null` or `0`. For example:\n\n```\nroot=> CREATE TABLE t (a INT, b BOOLEAN);\nroot=> INSERT INTO t VALUES (2, FALSE);\nroot=> SELECT * FROM t WHERE a = b;\nERROR:  arrow compute: Invalid argument error:\nInvalid comparison operation: Int64 == Boolean\nroot=> UPDATE t SET a = b;\nroot=> SELECT * FROM t;\n a | b\n---+---\n 0 | f\n```\n\nIn Postgres, this UPDATE would throw\n`ERROR: column \"a\" is of type text but expression is of type int`.\nParticleDB refused to *compare* this string to an integer (at\nleast in this case), but was perfectly happy to make one equal to the\nother, resulting in `NULL`. Version 1.1.18.6 seemed to\nresolve this issue ([#7](https://github.com/particledbai/jepsen/issues/7)).\n\n### 3.7.4 Insert Column Order Table Corruption (#16)\n\nIn SQL, one usually inserts a row into a table using a statement like\n`INSERT INTO people (name, age) VALUES ('xanthippe', 15)`.\nParticleDB 1.1.4 through 1.1.17 ignored the `INSERT`\nstatement’s columns, using whatever column order had been used to create\nthe table originally. This produced rows with titillating properties.\nConsider the following pair of inserts: one with columns in the same\norder as the table’s schema, and a second in a different order:\n\n```\nroot=> CREATE TABLE t\n       (a TEXT NOT NULL PRIMARY KEY,\n        b boolean);\nroot=> INSERT INTO t (a, b) VALUES ('x', TRUE);\nroot=> INSERT INTO t (b, a) VALUES (FALSE, 'y');\nroot=> SELECT COUNT(*) FROM t;\n COUNT\n-------\n     2\n(1 row)\n```\n\nWhile ParticleDB thinks this table contains two rows, neither can be\nread via `SELECT *`:\n\n```\nroot=> SELECT * FROM t;\n a | b\n---+---\n(0 rows)\n```\n\nIn general, any query which happened to encounter a row inserted with\na different column order would silently produce zero results. However,\neach row could be read by primary key! Note the different column orders\nin the output:\n\n```\nroot=> SELECT * FROM t WHERE a = 'x';\n a | b\n---+---\n x | t\n(1 row)\n\nroot=> SELECT * FROM t WHERE a = 'y';\n b | a\n---+---\n f | y\n(1 row)\n```\n\nIn the preceding example, the two columns were of (somewhat)\nincompatible types. What happens if the types are compatible?\n\n```\nroot=> CREATE TABLE t (a INT, b REAL);\nroot=> INSERT INTO t (a, b)\n       VALUES (1, 1.23E32);\nroot=> INSERT INTO t (b, a)\n       VALUES (2.34E32, 2);\nroot=> SELECT * FROM t;\n          a          |    b\n---------------------+---------\n                   1 | 1.23e32\n 9223372036854775807 |       2\n(2 rows)\n```\n\nNow the table is readable, but the data is corrupt. The second row’s\n`a` value appears under `b`, and vice-versa. The\nfloating-point value `2.34E32` has become the integer 263 − 1.\n\nIf we attempt to delete every row from the table, something very\nstrange happens. The `DELETE` leaves the differently-ordered\nrow in place, and its originally inserted values now appear intact!\n\n```\nroot=> DELETE FROM t;\nDELETE 1\nroot=> SELECT * FROM t;\n a |    b\n---+---------\n 2 | 2.34e32\n(1 row)\n```\n\nA second `DELETE` statement—now that the table contains\nonly rows inserted with column order `(b, a)`—could\nsuccessfully delete the remaining row. These column-order issues ([#16](https://github.com/particledbai/jepsen/issues/16)) no\nlonger appeared in version 1.1.18.4.\n\n### 3.7.5 Updates Can Insert Corrupt Rows (#24, #25)\n\nIn ParticleDB 1.1.4 through 1.1.18.8, an update which matched no rows\ncould instead create a new, possibly corrupt row in the table, with\nbaffling effects. Consider a table mapping integer keys to integer\nvalues:\n\n```\nroot=> CREATE TABLE t\n       (id int not null primary key,\n        val integer);\nroot=> BEGIN;\nroot=*> UPDATE t SET val = 10 WHERE id = 0;\nUPDATE 1\nroot=*> COMMIT;\n```\n\nThis table was just created. It contains no rows, so the update\nshould have affected zero, not one row. And yet it now contains a\nrow!\n\n```\nroot=> SELECT COUNT (*) FROM t;\n COUNT\n-------\n     1\n(1 row)\n```\n\nHowever, that row is invisible to typical queries:\n\n```\nroot=> SELECT * FROM t;\n id | val\n----+-----\n(0 rows)\n\nroot=> SELECT * FROM t WHERE id = 0;\n id | val\n----+-----\n(0 rows)\n\nroot=> SELECT * FROM t WHERE val = 10;\n id | val\n----+-----\n(0 rows)\n```\n\nAlthough one cannot directly see the row by primary key, it can\nconflict with some inserts.\n\n```\nroot=> INSERT INTO t (id, val) VALUES (0, 100);\nINSERT 0 1\nroot=> INSERT INTO t (id, val) VALUES (10, 110);\nERROR:  duplicate key value violates unique\n        constraint (key=10)\n```\n\n“Hang on,” an astute reader might exclaim. “The row we supposedly\nupdated had `id = 0` and set `val = 10`. Why did\nit not conflict with an insert of `id = 0`?”\n\nQuite right, dear reader! And why did it instead conflict with\n`id = 10`?\n\nThese are excellent questions—which we do not know the answer to.\nPressing on:\n\n```\nroot=> CREATE TABLE t\n       (id int not null primary key,\n        val integer);\nroot=> BEGIN;\nroot=*> UPDATE t SET val = 50 WHERE id = 5;\nUPDATE 1\nroot=*> SELECT * FROM t WHERE id = 5;\n val\n-----\n  50\n(1 row)\n\nroot=*> COMMIT;\n```\n\nWithin the transaction, the row produced by this update is readable\nby its primary key `id = 5`. Although this table has two\ncolumns, the row is printed as if it had only one:\n`val = 50`.\n\nAs in #16, these corrupt rows caused queries that encountered them to\nsilently return zero rows, even if there were other valid rows that\nwould have matched.\n\n```\nroot=> INSERT INTO t VALUES (6, 60);\nroot=> SELECT * FROM t WHERE id IN (6);\n id | val\n----+-----\n  6 |  60\n\n(1 row)\n\nroot=> SELECT * FROM t WHERE id IN (6, 7);\n id | val\n----+-----\n(0 rows)\n```\n\nAlthough our corrupt row had `id = 5` (or perhaps\n`50`), adding a second value to the `IN` predicate\nseems to have caused ParticleDB to “notice” the corrupt row. It then\nfailed to return the valid row with `id = 6`.\n\nDepending on the table structure and `UPDATE` statement,\nrows created by updates might be structurally valid, but logically\ncorrupt:\n\n```\nroot=> CREATE TABLE t (a INT UNIQUE);\nroot=> INSERT INTO t VALUES (1);\nroot=> BEGIN;\nroot=*> UPDATE t SET a = 3 WHERE a = 2;\nUPDATE 1\nroot=*> COMMIT;\nroot=> SELECT * FROM t;\n a\n---\n 1\n 3\n(2 rows)\n```\n\nThis bug ([#24](https://github.com/particledbai/jepsen/issues/24)) also\nallowed updates which would have left a row’s value unchanged to instead\nthrow a duplicate key exception, as they (presumably) conflicted with\nthemselves ([#25](https://github.com/particledbai/jepsen/issues/25)). Both\nissues appeared resolved as of version 1.1.18.9.\n\n### 3.7.6 Sometimes, Strings That Are Extensive… Are Cursed (#37)\n\nIn versions 1.1.15 through 1.1.17, `text` strings could be\nunexpectedly transmogrified into eight-bit integers. We encountered this\nin `append` tests using the official ParticleDB client, like\n[this\none](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260521T190337-strings-as-ints.zip). The `append` workload stores a comma-separated list\nof integers in an SQL `TEXT` column called `val`,\nand appends values to it using `CONCAT`. Most reads of\n`val` sensibly returned strings:\n\n```\n[\"select (val) from txn2 where id = ? \" 2946]\n  -> [{:val \"1\"}]\n```\n\nHowever, after approximately 30 seconds, the ParticleDB client began\nreturning eight-bit integers for key `2946`. Here, the key’s\nvalue has become the JVM `Byte` `1`:\n\n```\n[\"select (val) from txn2 where sk = ? \" 2946]\n  -> [{:val 1}]\n```\n\nFrom this point on, some keys appeared as strings, while others\nreturned integers. A read of key `2961` returned the string\n`\"2\"`, while key `2865` returned the integer\n`1`—then moments later, the string `\"1,2\"`.\nSometimes the integer values corresponded to the string value produced\nby earlier writes, but not always. For example, a transaction read key\n`2980` as the integer `0`—when the only elements\nappended to its string were `\"1\"` and `\"2\"`.\nShortly thereafter, a single transaction read key `2980`\ntwice and found a string, then an apparently unrelated integer:\n\n```\n[\"select (val) from txn2 where id = ? \" 2980]\n  -> [{:val \"2,1\"}]\n[\"select (val) from txn0 where sk = ? \" 2981]\n  -> [{:val \"1\"}]\n[\"select (val) from txn2 where sk = ? \" 2980]\n  -> [{:val 3}]\n```\n\nThis issue ([#37](https://github.com/particledbai/jepsen/issues/37)) seemed\nto disappear by 1.1.18.11.rc-1.\n\n### 3.7.7 Eternal Sunshine of the Spotless Table (#72)\n\nIn ParticleDB 1.1.18.11.rc-1, `append` tests of the\n`table-2pl` transaction mode often threw errors like:\n\n> PDB wire error 1064: SQL execution error: failed to create\n> RecordBatch: Invalid argument error: Column ‘id’ is declared as\n> non-nullable but contains null values\n\nThis is odd. The `append` workload always picks a unique\ninteger ID for every write, and never tries to write a `NULL`\nID. It also never updates an ID—they are set only once, when the row is\ncreated. Here’s [an\nexample](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T110736-contains-null-values.zip) where a single `SELECT` statement, issued outside\nof a transaction, complained of null primary keys. This suggests that\nthe table state had somehow become corrupt.\n\nWe no longer observed this error ([#72](https://github.com/particledbai/jepsen/issues/72)) in\n1.1.18.11.rc-7.\n\n## 3.8 Transactions and Concurrency\n\nWe found several problems with ParticleDB’s transactions and\nconcurrency control. All of these occurred in single-node instances\nwithout fault injection.\n\n### 3.8.1 Nested Transactions Destroy Ongoing Ones (#44)\n\nParticleDB 1.1.4 through 1.1.18.9 allowed clients to begin nested\ntransactions, instead of warning that a transaction was already ongoing.\nAny writes performed by the enclosing transaction would be lost, both\nwithin the nested transaction and after commit.\n\n```\nroot=> CREATE TABLE t (a int);\nroot=> BEGIN;\nroot=*> INSERT INTO t VALUES (3);\nroot=*> BEGIN;\nroot=*> SELECT * FROM t;\n a\n---\n(0 rows)\nroot=*> COMMIT;\n```\n\nParticleDB 1.1.18.11.rc-1 seems to have resolved this issue ([#44](https://github.com/particledbai/jepsen/issues/44)):\nissuing `BEGIN` within a transaction warned\n`there is already a transaction in progress`, and preserved\nthe ongoing transaction’s writes.\n\n### 3.8.2 Aborted Transactions Can Commit (#43)\n\nBetween versions 1.1.17 and 1.1.18.4, ParticleDB handled aborted\ntransactions irregularly. With the Postgres interface, an attempt to\ncommit an aborted transaction would throw an error, as one would\nexpect:\n\n```\nroot=> BEGIN;\nroot=> INSERT INTO t (a) VALUES (3);\nroot=> THIS IS NOT VALID SQL;\nERROR:  sql parser error: Expected: an SQL\nstatement, found: THIS at Line: 1, Column: 1\nroot=> COMMIT;\nROLLBACK\n```\n\nNevertheless, the writes performed by the aborted transaction\npersisted:\n\n```\nSELECT * FROM t;\n  -> [{:a 3}]\n```\n\nThe ParticleDB client had a different, also unexpected behavior:\nexecuting a `COMMIT` statement on a failed transaction\nreturned `[#:next.jdbc{:update-count 0}]`, rather than an\nerror. Although the transaction apparently committed, its effects were\ninvisible. Both interfaces, as of version 1.1.18.11.rc-18, returned\nerrors and did not expose the effects of aborted transactions ([#43](https://github.com/particledbai/jepsen/issues/43)).\n\n### 3.8.3 Aggregates Sometimes Fail to Observe a Transaction’s Rows (#48)\n\nIn ParticleDB 1.1.4 through 1.1.18.9, transactions which performed\naggregate queries, like `SELECT COUNT(*)`, could miss rows\nthey should have matched, and return duplicate copies of rows, depending\non the predicate and previous queries in the transaction. For\nexample:\n\n```\nroot=> CREATE TABLE t (a int);\nroot=> INSERT INTO t (a) VALUES (2);\nroot=> BEGIN;\nroot=*> INSERT INTO t (a) VALUES (3);\nroot=*> SELECT COUNT(*) FROM t;\n COUNT \n-------\n     1\n(1 row)\n```\n\nThis table clearly contains two rows: `2` and\n`3`. Yet while the second row was visible to\n`SELECT *`, it was invisible to `SELECT COUNT(*)`.\nOddly enough, aggregate functions which interacted with the column’s\nvalue, like `SELECT COUNT(*) FROM t WHERE a < 5`,\nsometimes observed correct state. In other circumstances,\n`SELECT` queries appeared to create duplicate copies of rows\nwithin the transaction, causing each query to observe more and more\nrows. These behaviors ([#48](https://github.com/particledbai/jepsen/issues/48)) no\nlonger appeared in version 1.1.18.11.rc-1.\n\n### 3.8.4 Double Trouble (#89)\n\nParticleDB 1.1.18.11.rc-9 would occasionally return duplicate copies\nof a single row, even though the row had a primary key which ought to\nhave ensured uniqueness. For example, take [this\nappend test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260617T165351-duplicate-rows.zip), in which a read of key `10` returned two\nrows, rather than one.\n\n```\n[{:val \"1,2,5,10\"}\n {:val \"1,2,5,10\"}]\n```\n\nWe no longer observed duplicate rows ([#89](https://github.com/particledbai/jepsen/issues/89)) in\n1.1.18.11.rc-18.\n\n### 3.8.5 Updates to Zero Values Sometimes Fail Silently (#91)\n\nIn ParticleDB 1.1.18.4, a transaction could set all rows in a table\nto some value, receive an acknowledgement that every row had been\nupdated, then read the table to find that some rows still had their\nprevious values. This bug seemed to hinge on inserting rows with a\nspecial “zero” value: `0` for integers, `0.0` for\nfloats, and `''` for various kinds of string columns. For\nexample:\n\n```\nroot=> CREATE TABLE t (a INT);\nroot=> INSERT INTO t (a) VALUES (0);\nroot=> BEGIN;\nroot=*> UPDATE t SET a = 1;\nroot=*> INSERT INTO t (a) VALUES (0);\nroot=*> UPDATE t SET a = 2;\nUPDATE 2\nroot=*> SELECT t.a FROM t;\n a\n---\n 2\n 0\n(2 rows)\n```\n\nParticleDB claimed that it updated all two rows in this table, but\nshowed the original value of `0` for one of them. This issue\n([#91](https://github.com/particledbai/jepsen/issues/91))\nseemed fixed by 1.1.18.11.rc-9.2.\n\n### 3.8.6 Tables With a `TEXT` Primary Key Are Mostly Invisible (#41)\n\nVersion 1.1.18.9 fixed several safety issues, but also introduced new\nones. For example, tables with a `text` primary key were\nmostly invisible to transactions:\n\n```\nroot=> CREATE TABLE t (a TEXT PRIMARY KEY);\nroot=> INSERT INTO t (a) VALUES ('mood');\nroot=> BEGIN;\nroot=*> SELECT * FROM t;\n a\n---\n(0 rows)\n```\n\n`SELECT`, `UPDATE`, and `DELETE`\nwere all affected. Rows were invisible to a broad variety of predicates,\nincluding `WHERE a LIKE 'moo%'` and\n`WHERE a IS NOT NULL`. However, rows were still accessible by\nprimary key:\n\n```\nroot=*> SELECT * FROM t WHERE a = 'mood';\n  a\n------\n mood\n(1 row)\n```\n\nVersion 1.1.18.11.rc-1 no longer exhibited these mostly-invisible\ntables ([#41](https://github.com/particledbai/jepsen/issues/41)).\n\n### 3.8.7 A Thousand Tableaus (#38, #66, #68, #69)\n\nPerhaps owing to the single-threaded issues discussed above,\nParticleDB 1.1.4 through 1.1.18.9 exhibited frequent, profound\ntransactional anomalies in essentially every way we could measure. We\nroutinely observed the loss of committed writes, internal consistency\nviolations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost\nUpdate). In the `append` workload, successfully appended\nvalues would routinely disappear from later reads of a given key. Our\ntest suite inferred every kind of cyclic anomaly including G0, G1c,\nG-Single, and G2-item, but many of these can be attributed to more basic\nissues, like write loss. These issues occurred with a single server and\nwithout faults, and affected every transaction mode, interface, and WAL\nmode tested.[8](#fn8)\n\nSince we have already discussed single-threaded anomalies at length,\nwe omit detailed discussions of these issues. By version 1.1.18.11.rc-1,\nParticleDB’s most glaring examples of G1a ([#66](https://github.com/particledbai/jepsen/issues/66)), G1b\n([#38](https://github.com/particledbai/jepsen/issues/38)),\ninternal consistency anomalies ([#68](https://github.com/particledbai/jepsen/issues/68)), write\nloss ([#69](https://github.com/particledbai/jepsen/issues/69)), and so\non were much diminished.\n\n### 3.8.8 G-Single (#73)\n\nWith versions 1.1.18.11.rc-1 through 1.1.18.11.rc-22, we occasionally\nobserved [G-single](https://jepsen.io/consistency/phenomena/g-single), a\nviolation of Snapshot Isolation, in single nodes without faults. For\nexample, here are two transactions from [an\n`append` test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T113409-g-single.zip) using transaction mode `occ-v3`\nand WAL mode `groupsync`:\n\nThe top transaction read key `9` and found it did not\nexist, then appended `8` to it. The bottom transaction\nappended `6` to key `9`. Since every read of key\n`9` observed a value like `[6 8 12 ...]`, we know\nthat the append of `6` happened before the append of\n`8`. The top transaction failed to observe the bottom, even\nthough the bottom must have occurred before the top. This is a similar\nphenomenon to P4 (Lost Update), and violates Snapshot Isolation.\n\nSimilar behavior ([#73](https://github.com/particledbai/jepsen/issues/73))\noccurred in `zero-copy-occ` and 1.1.18.11.rc-22’s\n`occ`. Orkes is investigating. Following our collaboration,\nOrkes reports version 1.1.19-rc.13 no longer exhibits G-Single.\n\n### 3.8.9 Brazil (#67)\n\nIn version 1.1.18.9, we found that queries for a specific row by a\nunique, constant integer ID would occasionally return the value of a\ncompletely different row. For example, consider [this\n`append` test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260609T162039-g1a-g1b-write-loss.zip) running against a single, healthy node with\ntransaction mode `occ-v3` and WAL mode\n`groupsync`. Reads of key `12` mostly observed\nvalues consistent with the appends to key `12`, but for a\nbrief time observed nothing, and then the value of key `7`\ninstead:\n\n| Time (s) | Value |\n| --- | --- |\n| 6.02 | 1, 2, 7, … 82 |\n| 6.05 | 1, 2, 7, … 82 |\n| 6.17 |  |\n| 6.22 |  |\n| 6.24 | 1, 2, 7, … 82, 92 |\n| 6.24 |  |\n| 6.25 | 1, 5, 8, … 181, 86 |\n| 6.29 |  |\n| 6.29 | 1, 5, 8, … 181, 86 |\n| 6.40 |  |\n| 6.49 |  |\n| 6.68 | 1, 2, 7, … 82, 92, … 139, 142 |\n\nThe value `[1, 5, 8, ... 181, 86]` was consistent with\nreads of key `7`, not key `12`. For the remainder\nof the test, reads observed values consistent with the original version\nof key `12`: `[1, 2, 7, ...]`. This issue ([#67](https://github.com/particledbai/jepsen/issues/67)) did not\nappear in our tests after version 1.1.18.11.rc-1.\n\n### 3.8.10 Lost Writes With `table-2pl` (#70)\n\nIn versions 1.1.18.11.rc-1 through 1.1.18.11.rc-6, ParticleDB’s\n`table-2pl` transaction mode routinely lost committed writes\nduring normal operation. For example [this\n`append` test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T085702.901-0500-table-2pl-write-loss-deadlock.zip), using the ParticleDB Java client, lost\nwrites to dozens of keys in just a few seconds. Here are some of its\nreads of key `187`:\n\n| Time (s) | Value |\n| --- | --- |\n| 10.89 | 5 |\n| 10.89 |  |\n| 10.92 | 1, 6 |\n| 10.92 |  |\n| 10.92 |  |\n| 10.92 | 1, 6, 10, 11 |\n| 10.93 | 2 |\n| 11.09 | 2, 13 |\n\nSince the `append` workload only appends to rows and never\ndeletes them, every read should have observed a prefix of the longest\nversion of the list. Instead it appears that the append of\n`5`, then the appends of `1`, `6`,\n`10`, and `11`, were lost. Moreover, some of these\nwrites were lost in the middle of a transaction, resulting in internal\nconsistency errors.\n\nBy version 1.1.18.11.rc-7, `table-2pl` no longer lost\nwrites this frequently ([#70](https://github.com/particledbai/jepsen/issues/70)).\n\n### 3.8.11 Internal Anomalies With `occ-v3` (#96)\n\nParticleDB 1.1.18.11.rc-9.2 occasionally violated internal\nconsistency using transaction mode `occ-v3` and WAL sync mode\n`groupsync`. For example, take [this\n`append` test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260620T151000-1.1.18.11.rc-9-internal.zip) where a single transaction read\n`nil`, then `[1, 4]` for key `8`:\n\n```\n[[:r 8 nil] [:r 8 [1 4]]]\n```\n\nAt Read Atomic and above, two successive reads of the same key should\nnot return different results. This also violates Snapshot Isolation and\nSerializability. This bug ([#96](https://github.com/particledbai/jepsen/issues/96)) may\nhave been resolved by 1.1.18.11.rc-19.\n\n### 3.8.12 There is No Anti-Memetics Division (#65)\n\nParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-9.1 would often forget\na column existed. A query like\n`SELECT * FROM snacks WHERE sk = 4` would throw an error like\n`PDB wire error 1064: column not found: sk`. This issue ([#65](https://github.com/particledbai/jepsen/issues/65))\noccurred in every transaction mode, but seemed to be resolved by version\n1.1.18.11.rc-9.2.\n\n### 3.8.13 Six Data Blockers and a Large Iced Coffee (#31)\n\nVersions 1.1.18.8 through 1.1.18.11.rc-22 exhibited perplexing\nnondeterministic failures in the `compare` workload. Each\ntest case established a fresh database client, created a globally unique\ntable, and performed operations against that table synchronously. At no\npoint should there have been two clients concurrently accessing a table.\nAnd yet sometimes, rows failed to appear in queries:\n\n```\nroot=> CREATE TABLE t_876 (a boolean);\nroot=> INSERT INTO t_876 (a) VALUES (NULL);\nroot=> INSERT INTO t_876 (a) VALUES (NULL);\nroot=> SELECT * FROM t_876 WHERE TRUE;\n  a\n------\n NULL\n(1 row)\n```\n\nOne could update rows using the predicate `WHERE NULL`,\nwhich should never match anything:\n\n```\nroot=> CREATE TABLE t_40 (a boolean);\nroot=> INSERT INTO t_40 (a) VALUES (FALSE);\nroot=> UPDATE t_40 SET a = (a <= a) WHERE NULL;\nUPDATE 1\n```\n\nIn many cases single SQL statements threw errors complaining of\nconcurrent updates, even though at most one client should ever have\naccessed a given table.\n\n```\nroot=> CREATE TABLE t_101 (\n  a text COLLATE unicode PRIMARY KEY\n);\nroot=> INSERT INTO t_101 (a) VALUES (' ');\nroot=> UPDATE t_101 SET a = NULL;\nERROR: could not serialize access due to\n       concurrent update on table \"t_101\"\n       key \" \"\n```\n\nDeleted rows could also remain visible:\n\n```\nroot=> CREATE TABLE t_27763 (\n  a boolean,\n  b boolean NOT NULL UNIQUE PRIMARY KEY\n);\nroot=> INSERT INTO t_27763 (a, b)\n       VALUES (NULL, TRUE);\nroot=> DELETE FROM t_27763;\nroot=> SELECT t_27763.a AS j FROM t_27763\n       WHERE t_27763.b ORDER BY t_27763.b;\n  j\n------\n NULL\n(1 row)\n```\n\nThese anomalies ([#31](https://github.com/particledbai/jepsen/issues/31))\noccurred with all transaction modes and WAL sync modes, but attempts to\nreproduce them deterministically, or to narrow down their cause, were\nunsuccessful. They may be artifacts of a bug in the test suite, but we\nlacked time to investigate fully. After our collaboration, Orkes\nreported 1.1.19-rc.13 resolved this issue.\n\n### 3.8.14 Read Skew With `table-2pl` (#106)\n\nWith the ParticleDB interface, WAL mode `groupsync`, and\n`table-2pl`, version 1.1.18.11.rc-18 exhibited [G-single](https://jepsen.io/consistency/phenomena/g-single)\nunder normal operation. Here’s [an\nexample](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260721T143917-table-2pl-g-single.zip):\n\n```\nT1: [[:r 112 nil]\n     [:append 112 1]\n     [:append 103 29]\n     [:append 67 105]]\n\nT2: [[:r 110 [10 14 13 20 16 28 30 33 48 35]]\n     [:r 112 [1]]\n     [:r 110 [10 14 13 20 16 28 30 33 48 35]]\n     [:r 103 [1 4 11 15 17 20 24 25]]]\n```\n\nThe first transaction appended `1` to key `112`\nand `29` to key `103`. The second transaction\nobserved the append to key `112`, but missed the append to\nkey `103`. Since it observed but also failed to observe\nanother transaction, this is [A5A (Read Skew)](https://jepsen.io/consistency/phenomena/a5a).\nSince the first transaction read-write depends on the second, and the\nsecond write-read depends on the first, it is also G-single. Both are\nviolations of Snapshot Isolation.\n\nWe no longer observed Read Skew with `table-2pl` ([#106](https://github.com/particledbai/jepsen/issues/106)) by\nversion 1.1.18.11.rc-19.\n\n### 3.8.15 Stale Read With `strict-serializable` (#109)\n\nVersion 1.1.18.11.rc-18 added a new transaction mode,\n`strict-serializable`, which may have been renamed to\n`serializable` in 1.1.18.11.rc-22.[9](#fn9)\nStrict Serializability (also known as Strong Serializability) guarantees\nthat all transactions appear to execute in real-time order: if *T*1 completes before *T*2 begins, then *T*1 appears to execute\nbefore *T*2. However,\nthis was not true in 1.1.18.11.rc-18. Healthy ParticleDB nodes\ninfrequently but regularly exhibited [Stale\nRead](https://jepsen.io/consistency/phenomena/stale-read), even with WAL mode `sync`. In one of our testing\nenvironments, these anomalies appeared roughly every 300,000\ntransactions, using Elle’s exponential key distribution and 10\nconcurrent clients pushing ~80 transactions per second.\n\nStale reads all seemed to share the same structure: a cycle\ncomprising a real-time, read-write, then write-read edge. Here’s [an\nexample](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260723T023218-strict-serializable-stale-read.zip) involving a single key:\n\nThe top transaction appended `66` to key\n`22209`, which was read by the middle transaction. After the\nmiddle transaction completed, the bottom transaction began, and read key\n`22209` without `66`. This is a violation of\nStrong Snapshot Isolation, and therefore Strong Serializability as\nwell.\n\nAfter the conclusion of our testing work, Orkes reported that version\n1.1.19-rc.10 no longer exhibits stale read ([#109](https://github.com/particledbai/jepsen/issues/109)).\n\n### 3.8.16 Lost Writes (#110)\n\nIn version 1.1.18.11.rc-18 through 1.1.18.11.rc-22, a healthy\nParticleDB node running `occ-v3`, `zero-copy-occ`,\n`table-2pl`, or `serializable-ssi` would\noccasionally discard acknowledged writes. For example, take [this\nappend test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260724T083315-occ-v3-data-loss.zip), which ran with WAL mode `sync`. Out of\napproximately 20,000 keys, 31 lost writes. Here are some of the reads of\nkey `16647`:\n\n| Time (s) | Value |\n| --- | --- |\n| 2798.88 |  |\n| 2798.88 | 1, 2, 3 |\n| 2798.88 | 1, 2, 3 |\n| 2798.90 | 1, 2, 3 |\n| 2798.90 | 1, 2, 3 |\n| 2798.93 | 1, 2, 3 |\n| 2798.96 | 1, 2, 3 |\n| 2798.98 | 4, 5, 6, 7 |\n| 2798.98 | 4, 5, 6, 7 |\n| 2799.02 | 4, 5, 6, 7, 27 |\n| 2799.02 | 1, 2, 3 |\n| 2799.02 | 1, 2, 3 |\n| 2799.02 | 1, 2, 3 |\n| 2799.02 | 4, 5, 6, 7, 27 |\n\nWhile the transaction which wrote `3` committed\nsuccessfully, and its write was visible to several other transactions,\nthe elements `1`, `2`, and `3` were\nreplaced by `4, 5, 6, ...` after 2799.02 seconds, and never\nreappeared. This occurred every few minutes in our tests, with both the\nPostgres and ParticleDB interfaces.\n\nA transaction could also read an acknowledged write, modify that row,\nand find that the modification apparently destroyed the written value,\nleading to internal anomalies as well as data loss. After we finished\ntesting Orkes released version 1.1.19-rc.13, and believes that it fixes\nthis issue ([#110](https://github.com/particledbai/jepsen/issues/110)).\n\n## 3.9 With Faults\n\nThe issues described above occurred in healthy, single-node instances\nof ParticleDB. When we introduced faults, new behaviors emerged.\n\n### 3.9.1 Data Loss on WAL Bitflips (#108)\n\nParticleDB uses checksums to detect corruption or incomplete writes\nto its Write-Ahead Log (WAL) files. Sometimes these checks prevented the\nserver from starting up with corrupt data. In other cases, ParticleDB\nwould discover that its data files were corrupt, then start up anyway,\nserving data to clients.\n\nTake [this\n`append` test](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260722T092352-bitflip-data-loss.zip) of 1.1.18.11.rc-18. At 09:25:09 we\nintroduced two single-bit errors into one of ParticleDB’s\n`.wal` files. At 09:25:39 we read key `1754` and\nobserved a list of 44 elements:\n\n```\n[[:r 1754 [3 5 7 9 12 13 18 19 20 22 24 25 28\n           30 31 33 34 36 35 37 38 39 42 43 44\n           45 47 50 51 53 58 59 62 63 64 65 67\n           68 69 70 73 75 79 80]] ...\n```\n\nA few milliseconds later, we killed and restarted ParticleDB a few\ntimes. When the server came back online at 09:26:07, it logged:\n\n```\n2026-07-22T14:26:07.985223Z  WARN ThreadId(01)\nrecover_streaming{dir=/opt/particledb/data/wal}:\nspanner_wal::segment: checksum mismatch in\nsegment tail, stopping read segment_id=0\n\n2026-07-22T14:26:07.985235Z  WARN ThreadId(01)\nrecover_streaming{dir=/opt/particledb/data/wal}:\nspanner_wal::recovery: replaying valid prefix\nof already-sealed corrupt WAL segment (.corrupt\nmarker present) and continuing with later\nsegments of this stripe stripe_id=0 segment_id=0\ncorruption_offset=25184 valid_size=25184\nrecovered_entries=242 detail=\"checksum mismatch:\nstored 0xf3b8aa37, computed 0x4a774da2\"\n```\n\nHowever, the server went on to start up and serve client requests\nnormally. At 09:26:09, we read key `1754` and found those 44\nelements had disappeared:\n\n```\n[... [:r 1754 [81]] ...]\n```\n\nSeveral other keys also lost some or all of their acknowledged\nwrites. Orkes is investigating this issue ([#108](https://github.com/particledbai/jepsen/issues/108)).\nWhile Jepsen has not tested this version, Orkes reports 1.1.19-rc.12\ndetects bitflips correctly and refuses to start.\n\n### 3.9.2 Total Data Loss With Simulated Misdirected Reads/Writes (#112)\n\nOne of our file corruption faults divided a file into four-kilobyte\nchunks, then randomly replaced a small fraction of those chunks with\nother chunks from the same file. The goal is to simulate [misdirected\nreads or writes](https://www.cse.cuhk.edu.hk/~pclee/www/pubs/msst14.pdf), a type of silent data corruption that happens in\nsome storage systems. In 1.1.18.11.rc-22, ParticleDB sometimes responded\nto this type of fault by forgetting every table and (apparently)\ndeleting everything in its WAL file.\n\nFor example, take [this\ntest run](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260730T095243-copy-file-chunks-table-lost.zip). At 09:53:56, the test harness shuffled 13 four-kilobyte\nchunks of the first `.wal` file around—approximately\none-tenth of the file’s contents. It then killed and restarted the node.\nAlthough this file had hundreds of kilobytes in it, when ParticleDB\nstarted up it logged that there was no data on disk:\n\n```\n2026-07-30T14:53:58.715589Z  INFO ThreadId(01)\nspanner_exec::engine::context: No data found in\nstorage — starting fresh\n```\n\nAt 09:53:58, ParticleDB began throwing\n`ERROR: table not found` for every query. No request ever\nsucceeded after this time.\n\nIt appears that ParticleDB may have deleted the contents of its WAL\nfile. A few seconds later, the test found its size was zero bytes, and\nthe same held true at the end of the test. After we finished testing\nOrkes released version 1.1.19-rc.12; they believe it resolves this issue\n([#112](https://github.com/particledbai/jepsen/issues/112)).\n\n### 3.9.3 Internal Anomalies With `strict-serializable` and Process Kills (#107)\n\nWith `strict-serializable`, WAL mode\n`groupsync`, and the ParticleDB interface, single nodes\nrunning 1.1.18.11.rc-18 exhibited internal anomalies with process kills.\nHere’s [an\nexample](https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260721T150149-strict-serializable-internal.zip):\n\n```\n{:op {:index 190704,\n      :time 100211183125,\n      :type :ok,\n      :process 59,\n      :f :txn,\n      :value [[:r 2879 nil]\n              [:append 2879 16]\n              [:r\n               2879\n               [1 2 3 4 5 7 8 16]]]},\n :mop [:r 2879 [1 2 3 4 5 7 8 16]],\n :expected [16]}\n```\n\nThis transaction read key `2879` as `nil`, then\nappended to it, and read its value—only to find that seven elements had\nbeen appended in the intervening time. Transactions could also exhibit\nG-single and P4 (G-single), but we suspect these could be due to the\nsame underlying bug. While Jepsen has not had a chance to test this\nrelease, Orkes claims this issue ([#107](https://github.com/particledbai/jepsen/issues/107)) was\nresolved in version 1.1.19-rc.12.\n\n| № | Summary | Event Required | Fixed in |\n| --- | --- | --- | --- |\n| [#62](https://github.com/particledbai/jepsen/issues/62) | Abysmal performance with `table-2pl` | None | 1.1.18.6 |\n| [#81](https://github.com/particledbai/jepsen/issues/81) | Mystery Crash | None | 1.1.18.11.rc-18 |\n| [#27](https://github.com/particledbai/jepsen/issues/27) | Deadlock after two transactions | None | 1.1.18.11.rc-1 |\n| [#71](https://github.com/particledbai/jepsen/issues/71) | Deadlock with `table-2pl` | None | 1.1.18.11.rc-7 |\n| [#33](https://github.com/particledbai/jepsen/issues/33) | Ordering by an unused column | None | 1.1.18.9 |\n| [#6](https://github.com/particledbai/jepsen/issues/6) | No static check for comparators of incomparable types | None | 1.1.18.6 |\n| [#94](https://github.com/particledbai/jepsen/issues/94) | Decimals not comparable to floats | None | 1.1.18.11.rc-18 |\n| [#12](https://github.com/particledbai/jepsen/issues/12) | Filter predicates reject `NULL` | None | 1.1.18.6 |\n| [#13](https://github.com/particledbai/jepsen/issues/13) | Cannot compare values to `NULL` | None | 1.1.18.6 |\n| [#30](https://github.com/particledbai/jepsen/issues/30) | Boolean logic on integers | None | 1.1.18.9 |\n| [#50](https://github.com/particledbai/jepsen/issues/50) | More boolean logic on integers | None | 1.1.18.11.rc-8 |\n| [#5](https://github.com/particledbai/jepsen/issues/5) | Multiple assignment to a single column | None | 1.1.18.6 |\n| [#10](https://github.com/particledbai/jepsen/issues/10) | Predicates can refer to nonexistent columns | None | 1.1.18.6 |\n| [#20](https://github.com/particledbai/jepsen/issues/20) | No aggregates within `CASE` | None | 1.1.18.6 |\n| [#28](https://github.com/particledbai/jepsen/issues/28) | No exponent operator (`^`) | None | 1.1.18.9 |\n| [#2](https://github.com/particledbai/jepsen/issues/2) | Malformed responses for prepared statements | None | 1.1.18.11.rc-1 |\n| [#83](https://github.com/particledbai/jepsen/issues/83) | `select` throws `Bad value for type long` | None | 1.1.19-rc.12 |\n| [#111](https://github.com/particledbai/jepsen/issues/111) | Tuples with no field structure | None | 1.1.19-rc.13 |\n| [#1](https://github.com/particledbai/jepsen/issues/1) | Broken Postgres authentication | None | 1.1.18.11.rc-1 |\n| [#3](https://github.com/particledbai/jepsen/issues/3) | Java client could not execute transactions | None | 1.1.15 |\n| [#14](https://github.com/particledbai/jepsen/issues/14) | No large integer literals | None | 1.1.18.6 |\n| [#85](https://github.com/particledbai/jepsen/issues/85) | No exponentiation on `Infinity` | None | 1.1.18.11.rc-9 |\n| [#49](https://github.com/particledbai/jepsen/issues/49) | `-1 ^ 0 = -1` | None | 1.1.18.11.rc-3 |\n| [#88](https://github.com/particledbai/jepsen/issues/88) | `1.0 + 1E-39 = 1.0` | None | 1.1.18.11.rc-9.2 |\n| [#56](https://github.com/particledbai/jepsen/issues/56) | Large literals become `Infinity` | None | 1.1.18.11.rc-3 |\n| [#61](https://github.com/particledbai/jepsen/issues/61) | Exponent underflow | None | 1.1.18.11.rc-3 |\n| [#34](https://github.com/particledbai/jepsen/issues/34) | Division by integer `0` | None | 1.1.18.9 |\n| [#51](https://github.com/particledbai/jepsen/issues/51) | Modulo operator cannot handle floats | None | 1.1.18.11.rc-1 |\n| [#36](https://github.com/particledbai/jepsen/issues/36) | `NaN` arithmetic throws errors | None | 1.1.18.9 |\n| [#92](https://github.com/particledbai/jepsen/issues/92) | `NULL` arithmetic throws | None | 1.1.18.11.rc-18 |\n| [#60](https://github.com/particledbai/jepsen/issues/60) | `UPDATE` arithmetic throws errors | None | 1.1.18.11.rc-1 |\n| [#32](https://github.com/particledbai/jepsen/issues/32) | Same precision for `real` and `double precision` | None | 1.1.18.9 |\n| [#11](https://github.com/particledbai/jepsen/issues/11) | Integers allow out-of-range values | None | 1.1.18.6 |\n| [#86](https://github.com/particledbai/jepsen/issues/86) | Dividing nearby numbers fails | None | 1.1.18.11.rc-9 |\n| [#87](https://github.com/particledbai/jepsen/issues/87) | Floating-point errors in integer values | None | 1.1.18.11.rc-9 |\n| [#19](https://github.com/particledbai/jepsen/issues/19) | Floats not equal to quoted representation | None | 1.1.18.6 |\n| [#63](https://github.com/particledbai/jepsen/issues/63) | `0` is not equal to `0.0` | None | 1.1.18.8 |\n| [#80](https://github.com/particledbai/jepsen/issues/80) | `5` appears equal to `0.90...` | None | 1.1.18.11.rc-9 |\n| [#64](https://github.com/particledbai/jepsen/issues/64) | `CHARACTER(n)` allows more than `n` characters | None | 1.1.18.6 |\n| [#52](https://github.com/particledbai/jepsen/issues/52) | `CHARACTER(n)` allows more than `n` characters again | None | 1.1.18.11.rc-3 |\n| [#55](https://github.com/particledbai/jepsen/issues/55) | `CHARACTER(n)` allows fewer than `n` characters | None | 1.1.18.11.rc-1 |\n| [#22](https://github.com/particledbai/jepsen/issues/22) | Default text collation ignores `LOCALE` | None | 1.1.18.9 |\n| [#42](https://github.com/particledbai/jepsen/issues/42) | Mis-sorted text | None | 1.1.18.11.rc-7 |\n| [#93](https://github.com/particledbai/jepsen/issues/93) | More mis-sorted text | None | 1.1.18.11.rc-21 |\n| [#40](https://github.com/particledbai/jepsen/issues/40) | `SELECT 5 WHERE FALSE` returns `5` | None | 1.1.18.11.rc-1 |\n| [#47](https://github.com/particledbai/jepsen/issues/47) | `WHERE` clauses with math match nothing | None | 1.1.18.11.rc-1 |\n| [#18](https://github.com/particledbai/jepsen/issues/18) | `WHERE 4 = 5` matches primary key `5` | None | 1.1.18.4 |\n| [#35](https://github.com/particledbai/jepsen/issues/35) | Transactional selects return nonsense | None | 1.1.18.9 |\n| [#8](https://github.com/particledbai/jepsen/issues/8) | `NOT NULL` constraints don’t | None | 1.1.18.4 |\n| [#9](https://github.com/particledbai/jepsen/issues/9) | `UNIQUE` constraints don’t | None | 1.1.18.6 |\n| [#29](https://github.com/particledbai/jepsen/issues/29) | Updates can create duplicate primary keys | None | 1.1.18.9 |\n| [#57](https://github.com/particledbai/jepsen/issues/57) | Primary keys may be `NULL` | None | 1.1.18.11.rc-3 |\n| [#82](https://github.com/particledbai/jepsen/issues/82) | Primary key conflicts with deleted rows | None | 1.1.18.11.rc-18 |\n| [#90](https://github.com/particledbai/jepsen/issues/90) | Duplicate primary keys | None | 1.1.18.11.rc-19 |\n| [#58](https://github.com/particledbai/jepsen/issues/58) | Can’t join integers to floats | None | 1.1.18.11.rc-1 |\n| [#54](https://github.com/particledbai/jepsen/issues/54) | Joins omit table names, mangle columns instead | None | 1.1.19-rc.13 |\n| [#76](https://github.com/particledbai/jepsen/issues/76) | Joins coerce values to strings | None | 1.1.18.11.rc-21 |\n| [#59](https://github.com/particledbai/jepsen/issues/59) | Outer joins to empty tables often produce no rows | None | 1.1.18.11.rc-3 |\n| [#77](https://github.com/particledbai/jepsen/issues/77) | Joins confuse columns with same name | None | 1.1.18.11.rc-7 |\n| [#26](https://github.com/particledbai/jepsen/issues/26) | Transactions can’t delete anything | None | 1.1.18.9 |\n| [#45](https://github.com/particledbai/jepsen/issues/45) | Transactions which delete leave shades behind | None | 1.1.18.11.rc-1 |\n| [#23](https://github.com/particledbai/jepsen/issues/23) | Updating integers makes rows not-deletable | None | 1.1.18.4 |\n| [#84](https://github.com/particledbai/jepsen/issues/84) | `double precision` primary keys yield zombie twins | None | 1.1.18.11.rc-9 |\n| [#78](https://github.com/particledbai/jepsen/issues/78) | `NaN` values are immutable | None | 1.1.18.11.rc-7 |\n| [#74](https://github.com/particledbai/jepsen/issues/74) | Fixed-width strings are deeply cursed | None | 1.1.18.11.rc-7 |\n| [#46](https://github.com/particledbai/jepsen/issues/46) | Dropped tables can be resurrected, in distorted forms | None | 1.1.18.11.rc-3 |\n| [#15](https://github.com/particledbai/jepsen/issues/15) | `Infinity` is writable, but unreadable | None | 1.1.18.6 |\n| [#21](https://github.com/particledbai/jepsen/issues/21) | `NaN` sometimes becomes `0.0` | None | 1.1.18.6 |\n| [#7](https://github.com/particledbai/jepsen/issues/7) | Assignment of incompatible types yields (e.g.) `NULL` | None | 1.1.18.6 |\n| [#16](https://github.com/particledbai/jepsen/issues/16) | Values assigned to wrong columns, yielding cursed rows | None | 1.1.18.4 |\n| [#24](https://github.com/particledbai/jepsen/issues/24) | Updates of no rows create new, corrupt rows instead | None | 1.1.18.9 |\n| [#25](https://github.com/particledbai/jepsen/issues/25) | No-op updates cause duplicate key violations | None | 1.1.18.9 |\n| [#37](https://github.com/particledbai/jepsen/issues/37) | Strings are sometimes returned as 8-bit integers | None | 1.1.18.11.rc-1 |\n| [#72](https://github.com/particledbai/jepsen/issues/72) | Primary keys mysteriously become `null` | None | 1.1.18.11.rc-7 |\n| [#44](https://github.com/particledbai/jepsen/issues/44) | Nested transactions destroy ongoing ones | None | 1.1.18.11.rc-1 |\n| [#43](https://github.com/particledbai/jepsen/issues/43) | Aborted transactions can commit | None | 1.1.18.11.rc-18 |\n| [#48](https://github.com/particledbai/jepsen/issues/48) | Aggregates sometimes miss a transaction’s rows | None | 1.1.18.11.rc-1 |\n| [#89](https://github.com/particledbai/jepsen/issues/89) | Queries sometimes observe duplicate copies of rows | None | 1.1.18.11.rc-18 |\n| [#91](https://github.com/particledbai/jepsen/issues/91) | Updates to special values can silently fail | None | 1.1.18.11.rc-9.2 |\n| [#41](https://github.com/particledbai/jepsen/issues/41) | Tables with `text` primary keys are mostly invisible | None | 1.1.18.11.rc-1 |\n| [#66](https://github.com/particledbai/jepsen/issues/66) | G1a (Aborted Read) | None | 1.1.18.11.rc-1 |\n| [#38](https://github.com/particledbai/jepsen/issues/38) | G1b (Intermediate Read) | None | 1.1.18.11.rc-1 |\n| [#68](https://github.com/particledbai/jepsen/issues/68) | Internal consistency anomalies | None | 1.1.18.11.rc-1 |\n| [#69](https://github.com/particledbai/jepsen/issues/69) | Lost write | None | 1.1.18.11.rc-1 |\n| [#73](https://github.com/particledbai/jepsen/issues/73) | G-single | None | 1.1.19-rc.13 |\n| [#67](https://github.com/particledbai/jepsen/issues/67) | Reads observe data from different row | None | 1.1.18.11.rc-1 |\n| [#70](https://github.com/particledbai/jepsen/issues/70) | Lost write with `table-2pl` | None | 1.1.18.11.rc-7 |\n| [#96](https://github.com/particledbai/jepsen/issues/96) | Occasional internal consistency anomalies with `occ-v3` | None | 1.1.18.11.rc-19 |\n| [#65](https://github.com/particledbai/jepsen/issues/65) | Columns sometimes vanish | None | 1.1.18.11.rc-9.2 |\n| [#31](https://github.com/particledbai/jepsen/issues/31) | Odd nondeterministic single-threaded anomalies | None | 1.1.19-rc.13 |\n| [#106](https://github.com/particledbai/jepsen/issues/106) | A5A (Read Skew) with `table-2pl` | None | 1.1.18.11.rc-19 |\n| [#109](https://github.com/particledbai/jepsen/issues/109) | Stale Read with `strict-serializable` | None | 1.1.19-rc.10 |\n| [#110](https://github.com/particledbai/jepsen/issues/110) | Lost writes | None | 1.1.19-rc.13 |\n| [#108](https://github.com/particledbai/jepsen/issues/108) | Silent(ish) data loss | WAL truncation/bitflips | 1.1.19-rc.12 |\n| [#112](https://github.com/particledbai/jepsen/issues/112) | Total data loss | Misdirected reads/writes | 1.1.19-rc.12 |\n| [#107](https://github.com/particledbai/jepsen/issues/107) | Spurious `nil` reads | Process crash | 1.1.19-rc.12 |\n\n# 4 Discussion\n\nParticleDB 1.1.4 claimed to be a Postgres-compatible database which\noffered Snapshot Isolation with its `occ-v3` and\n`zero-copy-occ` transaction modes, Serializability with\n`serializable-ssi`, and Strong Serializability with\n`table-2pl`.\n\nInstead, we found a library of confusion.[10](#fn10)\nIn versions 1.1.4 through 1.1.18.11.rc-19, basic functionality like\ntransactions (#3), prepared statements (#2), and Postgres auth (#1) were\nunusably broken. The server spontaneously deadlocked (#27, #71) or\ncrashed (#81) every few minutes. Integers stood in for booleans (#30,\n#50), comparisons to `NULL` threw errors (#13), and you could\nassign multiple values to the same column (#5). Large floating-point\nnumbers quietly became infinite (#56); `Infinity`,\n`NaN`, and `NULL` arithmetic were broken (#36,\n#92, #15, #21); and out-of-range values were improperly stored or\nclipped to fit (#32, #11). In ParticleDB, `0.0` was not equal\nto `0` (#63), but `0.90...` wasn’t *not*\nequal to `5` (#80). Strings allowed too many (#52, #64) and\ntoo few (#55) characters. A `WHERE` clause with\nfloating-point operations would silently fail to match any rows (#47),\n`WHERE 4 = 5` returned rows with primary key `5`\n(#18), `WHERE a = 1` matched the row `{a: 2}`, and\n`WHERE TRUE` matched nothing (#35). Null (#8) and unique (#9)\nconstraints did nothing, and primary keys could be duplicated (#29),\nnull (#57), or both. Primary keys failed to conflict with rows that did\nexist (#90), and conflicted with rows that didn’t (#82). Joins returned\nincorrect column names (#54), coerced values to strings (#76), failed to\nreturn rows (#59), and confused similarly-named columns (#77). It was\nimpossible to delete rows within a transaction (#26, #45), rows with\nintegers set to floats (#23), or with floats set to integers (#84), or\nthose containing `NaN`s (#78), or fixed-width strings\n(#74).[11](#fn11) Rows from dropped tables could\nre-emerge in mangled forms (#46). Columns could be set to values of the\nwrong type, silently altering or corrupting data (#7). Inserts ignored\nprovided columns, creating rows best described as “deeply cursed”\n(#16).[12](#fn12) Updates could create new, shambolic\nrows where none existed (#24, #25). Strings sometimes turned into\neight-bit integers (#37), and primary keys occasionally became\n`NULL` (#72). Aborted transactions could commit, or fail with\ntheir effects still visible (#43). Queries sometimes missed (#48),\nduplicated (#89), or returned the wrong (#67) rows. Tables with a\n`text` primary key were mostly invisible (#41). Special\nvalues were sometimes inalterable (#91). Entire columns occasionally\nvanished for no obvious reason (#65). Uncontended statements on freshly\ncreated tables observed nonsensical results or conflicted (apparently)\nwith themselves (#31). Each transaction and WAL mode exhibited\nessentially every transactional anomaly our tests could detect: write\nloss, internal consistency violations, G0, G1a, G1b, G1c, G-Single,\nG-nonadjacent, P4, A5A, G2, and real-time variants thereof (#38, #66,\n#68, #69, #73, #70, #96, #109, #110, etc.).\n\nThese behaviors occurred in single-node instances of ParticleDB,\nwithout fault injection, often with just a single client. When we killed\nParticleDB, it allowed internal anomalies, P4 (Lost Update), and\nG-single (#107), even with the new `serializable` transaction\nmode. Corruption or truncation of WAL files caused partial data loss\n(#108). Simulating misdirected reads or writes could cause total data\nloss (#112).\n\nAs we concluded our engagement, version 1.1.18.11.rc-22 had just\nbecome available. It abandoned the `table-2pl` and\n`serializable-ssi` transaction modes in favor of\n`fast`, `occ`, and `serializable`,\nwhich were intended to offer no consistency, Snapshot Isolation, and\nStrong Serializability, respectively. While 1.1.18.11.rc-22 was\nsignificantly safer than 1.1.4, we were still finding new issues. The\ndefault `occ` mode regularly lost data in normal operation\n(#110). The new `serializable` mode appeared significantly\nsafer, but still allowed stale read (#109) in normal operation, and with\nprocess kills, sometimes failed to read extant rows. This led to\ninternal anomalies, Lost Update, and more (#107).\n\nAfter we finished testing, Orkes updated ParticleDB’s documentation\nto claim that every transaction mode discussed here was [“experimental”](https://particledb.ai/operations/modes/#experimental-modes-not-in-current-release-build).\nThis claim is difficult to support. To the best of Jepsen’s knowledge,\nnone of the ParticleDB documentation [described\nthese modes](https://web.archive.org/web/20260518193025/https://particledb.ai/getting-started/concepts/#transaction-modes) as experimental or unsupported during our engagement.\nOrkes’ unpublished operator guide did describe one mode,\n`serializable-occ`, as an “internal A/B variant… do not use\nfor correctness runs”. We considered that mode experimental and omitted\nit from this report. However, the others had no such caveats.\n\nOrkes [appears to\nhave reworked](https://particledb.ai/operations/modes/) ParticleDB’s approach to consistency. It now supports\nthe standard SQL isolation level statements for\n`READ COMMITTED`, `REPEATABLE READ` (meaning\nSnapshot Isolation), and `SERIALIZABLE`. The\n`--txn-mode` option is gone, and a new\n`--concurrency-control` setting has `none`,\n`optimistic`, and `strict-optimistic` options. The\n[documentation\nclaims](https://particledb.ai/operations/modes/#experimental-modes-not-in-current-release-builds) that every mode discussed in this report has been removed\nfrom version 1.1.19-rc.13. However, it goes on to say that\n`occ-v3` and `strict-serializable` were “earlier\nnames” of the “optimistic engine” and “strict serializable mode”,\nrespectively. It is not clear whether these modes were removed or simply\nrenamed.\n\nJepsen has not evaluated versions of ParticleDB after\n1.1.18.11.rc-22, and cannot speak to these isolation levels or\nconcurrency control modes. However, Orkes claims that all issues from\nthis report have been addressed as of version 1.1.19-rc.13.\n\nAs always, Jepsen takes an experimental approach to safety\nverification: we can prove the presence of bugs, but not their absence.\nWhile we make extensive efforts to find problems, we cannot prove\ncorrectness.\n\n## 4.1 Impact\n\nOrkes believes that the issues we found in 1.1.4 were likely due to\nan extensive refactor of ParticleDB in spring 2026, and that Orkes and\ntheir customers, who continued to run an earlier version, were likely\nunaffected. Near the end of our collaboration we briefly evaluated an\ninternal build of version 1.0.0. While it exhibited some string and\narithmetic issues, that version did not seem susceptible to most of the\nserious safety errors we found in 1.1.4, and passed cursory\n`rw` and `append` tests. We tested only a single\nnode at a time, so we cannot speak to the safety of the replication\nsystem.\n\nFrom May through July 2026, the ParticleDB [download page](https://particledb.ai/installation/binary/)\noffered users (at least) versions 1.1.4 through 1.1.18.11.rc-18.[13](#fn13) All exhibited serious safety errors\nduring normal operation. It seems likely that users who ran these\nversions of ParticleDB would have lost or corrupted data. In version\n1.1.18.11.rc-22, using WAL mode `sync` and transaction mode\n`serializable` significantly reduced the frequency and\nseverity of these issues, but did not prevent internal anomalies or Lost\nUpdate.\n\nGiven the severity and extent of the safety errors we found, and the\nprocess which produced them, Jepsen recommends users halt further\ndeployment of ParticleDB and migrate to other databases as soon as\npossible. Consider checking and/or rebuilding database state from more\nreliable sources, if available.\n\n## 4.2 The Unreasonable Effectiveness of Generative Testing\n\nThe `compare` workload generated random schemas and SQL\nstatements over them, then ran both against ParticleDB and Postgres to\nsee if they produced equivalent results. This should not have worked.\nThere are many ways in which two “Postgres-compatible” databases can\nlegitimately differ, such as floating-point details, which error message\ntakes precedence, or the order of unsorted results. The space of SQL\nqueries is large and difficult to cover. Many queries trivially return\nequivalent results (e.g. `SELECT ... WHERE FALSE`) and\ntherefore tell us little information. We can test only sequential\nbehavior, which is easy for a database’s standard test suite to\ncheck.\n\nNevertheless, roughly [two\nthousand lines](https://github.com/jepsen-io/particledb/blob/d4721aeef9435af0131eacfb2163059272b13d57/src/jepsen/particledb/gen/tc.clj) of generative tests found a wealth of bugs in\nParticleDB, and unlike Jepsen’s typical workloads, the examples\ngenerated were usually easy to reproduce. ParticleDB offered a good deal\nof “low-hanging fruit” that should have been caught by example-based\ntests—but the `compare` workload also found more subtle\nerrors. For example, the fact that `0.90...` was apparently\nequal to `5` hinged on a high-precision floating-point\nliteral. Inspired by this success, we extracted part of the ParticleDB\nSQL generator into [a shared\nlibrary](https://github.com/jepsen-io/sql) so it could be used in other tests, like the new\n`internal-sim` workload.\n\nJepsen has traditionally focused on thorough analysis of a small set\nof carefully chosen transactions. This is partly because checking (e.g.)\nSerializability in general [is NP-complete](https://dl.acm.org/doi/10.1145/322154.322158),\nand also because we assume that if transactions of (e.g.) reads and\nappends work correctly on strings, so will other operations on other\ndatatypes. Our intuition is that the “hard part” is in the concurrency\ncontrol mechanism, and sequential datatype behavior is generally covered\nby existing tests. This was not the case in ParticleDB. We’ve expanded\nJepsen’s `list-append` and `rw` tests to use\nalternate encodings and transaction structures, in an attempt to cover\nmore codepaths. This revealed new bugs in ParticleDB, and we hope it\nwill in other databases too.\n\nUnfortunately, `compare` became less useful as Orkes fixed\nmajor bugs and left smaller incompatibilities in place. For example,\nPostgres checks unique constraints when updating each row of a table,\nbut ParticleDB only checks after all have been updated. This means that\na statement like `UPDATE bools SET x = NOT x` can swap the\nvalues of `TRUE` and `FALSE` in ParticleDB without\nviolating a unique constraint. Toward the end of our engagement, the\n`compare` test generally hit these minor inconsistencies\nearly, then spent most of its time searching for minimal cases of\nthem—potentially missing more serious bugs.\n\n## 4.3 Quality Assurance\n\nOrkes said they ran Jepsen’s existing Postgres test harness to find\nand resolve issues in ParticleDB prior to this work. However, that same\ntest harness failed immediately when run against ParticleDB 1.1.4. Orkes\nalso stated they tested ParticleDB with [SQLancer](http://www.sqlancer.com/), which generates far more\ncomplex queries than the ones we tested. It is therefore perplexing that\nnearly every feature we tested exhibited glaring errors, and that our\nfindings seemed to come as a surprise.\n\nMost databases have a test suite which exercises at least basic\nexamples of each feature, and it is customary to run those tests before\nrelease, if not continuously. Where were those tests for ParticleDB? How\ndoes one add support for `NOT NULL` constraints without a\ntest which tries to insert at least one `NULL` value? What\ndevelopment process adds a `CHARACTER(10)` type without\ntrying to write a string of (e.g.) five characters?\n\nOur findings suggest structural deficiencies in the ParticleDB QA\nprocess. Perhaps tests were not run prior to release. Perhaps the test\nsuite was run, but failed to exercise basic functionality. Perhaps\nengineers entrusted Large Language Models with the work of testing.\n\nJepsen recommends that Orkes (and all database vendors!) build and\nexercise an automated test suite for their database. This suite should\ninclude at least simple, example-based tests covering important database\nfeatures. These tests should fail when the feature is unimplemented or\nmalformed, and pass when it is complete. Observing these failures is an\nimportant part of the development process: it [validates that the\ntest measures something](https://en.wikipedia.org/wiki/Mutation_testing). Tests should be run at least once before\neach release, and results disseminated to the team. Afterwards consider\nmore sophisticated generative tests, establishing adversarial QA and\nimplementation teams, and subtly breaking safety mechanisms to determine\nwhether the tests are still effective.\n\n## 4.4 Documentation\n\nParticleDB’s website gave the appearance of a mature database\nproject. It made strong claims: a general-purpose OLTP/OLAP SQL database\na thousand times faster than PostgreSQL, “formally verified” by a TLA+\nproof, multiple APIs, and client libraries in thirteen languages. It\noffered a variety of deployment options and extensive documentation on\ndatabase features and operations.\n\nOn closer inspection, that documentation was riddled with omissions,\ninconsistencies, and confabulations. The TLA+ proof was not published,\nand in any event did not prevent disaster—perhaps it was incorrect,\nformalized the wrong properties, or did not describe the system as\nbuilt. Safety-critical configuration settings like the transaction mode\nwere described in vague terms, like [“safe\nunder concurrency”](https://web.archive.org/web/20260518193025/particledb.ai/getting-started/concepts). The documentation for the [Java\nclient library](https://web.archive.org/web/20260518203023/https://particledb.ai/sdks/java/) referred to non-existent dependency coordinates and\nseemed to make up class and package names which were nowhere to be found\nin the library itself. The quickstart and client examples made up a\nvariety of default usernames for ParticleDB, all of which were wrong.\nThe stored procedure documentation even fabricated an [imaginary\nemail address](https://web.archive.org/web/20260727225950/https://particledb.ai/particles/rust/#run-it) for the author of this report. In Orkes’ unpublished\noperator guide, the descriptions of transaction modes omitted basic\nphenomena like [G0](https://jepsen.io/consistency/phenomena/g0) and confused [G1a](https://jepsen.io/consistency/phenomena/g1a) (Aborted\nRead) with [G1c](https://jepsen.io/consistency/phenomena/g1c)\n(Circular Information Flow). The Snapshot Isolation modes made no\nmention of [G-single](https://jepsen.io/consistency/phenomena/g-single) or\nthe [other\nboundary phenomena](https://jepsen.io/consistency/phenomena/g-nonadjacent) proscribed by Snapshot Isolation, and\nnon-real-time models made no mention of the real-time phenomena they\nallowed, like Stale Read.[14](#fn14)\n\nTechnical writing is important work that requires skill and care.\nJepsen recommends that Orkes work with a human technical writer to\nrewrite ParticleDB’s documentation, with an eye toward concisely and\naccurately communicating intended behavior and the guarantees users\nshould expect.\n\nJepsen is no stranger to working with early software projects, and\nthere is nothing intrinsically wrong with offering buggy or badly broken\nsoftware. However, it would be polite for a maintainer aware of serious\nsafety issues to display a prominent warning on the website. One might\nplace a large red box on the download page which reads, “This project is\nin early development and is not suitable for production”, or more\nbluntly, “This software has lost data and will do so again”. Orkes\nshould clearly explain the general status of ParticleDB, and advise\nusers of specific known safety issues.\n\nOrkes has since published [a\nrelease history](https://particledb.ai/installation/releases/#release-history) which describes versions 1.1.4 through\n1.1.18.11.rc-22 as “unsupported pre-release builds”. It is not clear how\nusers would have understood versions like 1.1.4 or 1.1.18.9 as\n“pre-release”. Pre-release software generally has a version number\nbeginning with 0, or including “rc”, “alpha”, etc. The documentation [did\nnot describe these versions](https://web.archive.org/web/20260518201315/https://particledb.ai/installation/binary/) as pre-release or experimental at the\ntime. Indeed, the [Docker\ninstall page](https://web.archive.org/web/20260518200305/https://particledb.ai/installation/docker/) told users how to deploy these versions “for\nproduction”.\n\n## 4.5 LLM Tradeoffs\n\nLarge Language Models enable small teams to produce software and\ndocumentation of staggering scope and complexity in a remarkably short\ntime. Five years ago, writing a distributed SQL database of this scope\nmight have taken hundreds of person-years.[15](#fn15)\nWith LLM assistance, a handful of engineers at Orkes were able to\nproduce a highly sophisticated database system.\n\nHowever, our work on ParticleDB (and other databases) suggests there\nare serious tradeoffs to relying on LLMs. Quality suffered: ParticleDB\nwas a fractal of correctness errors. LLM tooling repeatedly claimed that\nbugs had been fixed when, in fact, they had not. ParticleDB also seemed\nmore difficult to understand than typical databases. As with most Jepsen\nanalyses, Orkes engineers offered detailed explanations of the causes\nand resolutions of the bugs we found in [Conductor](https://jepsen.io/analyses/orkes-conductor-5.3.2).\nHowever, Orkes seemed unable to explain the causes or resolutions of\nnearly every issue we found in ParticleDB. New builds simply\nmaterialized, sans changelogs, each hopefully containing a slew of\nimprovements. Why did ParticleDB sometimes turn strings into integers?\nWhy did it stop in 1.1.18.11.rc-1? Perhaps no one knows. With LLMs, [perhaps\nno one needs to](https://www.tiktok.com/@charlixcx/video/7603137654682570006).\n\nMore broadly, the use of LLMs erodes trust in documentation, in\nsoftware, and in the organizations which produce them. Perhaps engineers\nwill begin to treat any new software project with suspicion—the signals\nwe used to rely on as markers of a mature system are now easier to\ncounterfeit.\n\nWhere correctness matters—and surely it ought to matter for\ndatabases!—our industry must acknowledge and mitigate these tradeoffs.[16](#fn16) In the age of LLMs, what can\nmaintainers do to ensure correct software and accurate documentation?\nHaving done that work, how can they demonstrate its quality to\nusers?\n\n## 4.6 Future Work\n\nAs of 1.1.18.11.rc-22, our `append` and `rw`\nworkloads were still finding anomalies in every transaction mode. We\nrecommend further testing as development continues. The\n`compare` workload also revealed a number of minor\ncompatibility bugs which we have not discussed here. These small\ndifferences could mask more serious errors. Bringing ParticleDB into\ncloser alignment with Postgres could help the `compare`\nworkload reveal them.\n\nToward the end of our collaboration we tested ParticleDB’s on-disk\nformat, and found that corruption in WAL files could cause data loss.\nThe variety of error messages we found suggests that there might be\nmultiple paths to data loss—further tests should stress on-disk storage\nmore thoroughly. Furthermore, in almost all our tests ParticleDB wrote\nonly WAL files, and stored its tables entirely in memory. We recommend\nOrkes add a configuration option to reduce the thresholds for on-disk\nstorage, so that future tests can exercise those codepaths.\n\nThe ParticleDB client has a number of slow network timeouts we did\nnot know how to change. These timeouts could conceal safety errors by\npreventing the test from observing the database during a critical\nmoment. We worked around this by using the Postgres client (whose\ntimeouts are configurable), but we recommend testing the ParticleDB\nclient with lower timeouts as well.\n\nParticleDB offered a massive set of features, and we only tested a\nsmall portion of them. The SQL specification and Postgres’ dialect of it\nare astonishingly complex languages, and we used only a small fraction\nof their syntax and vocabulary. We also did not evaluate ParticleDB’s\nRedis or HTTP APIs, temporal or geospatial queries, materialized views,\nvector search, or multi-tenant features. The replication system remains\nto be tested.\n\nOrkes is now adding stored procedures to ParticleDB. These\nuser-defined functions, called [*Particles*](https://particledb.ai/particles/overview/),\nare written in Python or Rust and executed by the database server. Both\nRust and Python have large surface areas, and executing user-provided\nbinaries in a database environment seems like an exciting opportunity.\nOur [previous\nwork](https://jepsen.io/analyses/capela-dda5892) with stored procedures suggests this feature is worth further\ninvestigation.\n\nParticleDB claimed to be “1087x faster than PostgreSQL”. In our tests\nParticleDB was nowhere near this fast, and over the course of an hour we\nobserved a linear decline in throughput and a significant increase in\nmedian latencies—even over small datasets. Once ParticleDB reaches a\nbaseline level of correctness, revisiting performance benchmarks could\nbe worthwhile.\n\n*Our thanks to the ParticleDB team at Orkes, including Jeu George\nand Nikhil Kala. Jepsen also wishes to thank [Irene Kannyo](https://www.irenekannyo.com/) for her editorial\nsupport. This research was funded by Orkes, Inc., and conducted in\naccordance with the [Jepsen\nethics policy](https://jepsen.io/analyses/ethics).*\n\n# 5 Appendix A\n\nThis appendix contains additional results involving types, syntax,\nnumbers, and strings.\n\n## 5.1 Type Safety and Syntax\n\nWe found several incompatibilities or bugs involving static or\ndynamic type safety, or behaviors that would normally have been caught\nby static analysis of a query, such as referring to a nonexistent\ncolumn. These issues occurred with a single server, single client, and\nno faults.\n\n### 5.1.1 Ordering By an Unused Column (#33)\n\nIn ParticleDB 1.1.4 through 1.1.18.8, you could order an aggregate\nquery’s results by a column that was not used in the aggregate\nexpressions or `GROUP BY` clause. This seems like a minor\nissue, but it does differ from Postgres behavior:\n\n```\nroot=> CREATE TABLE t (a boolean);\nroot=> SELECT COUNT(*) FROM t\n       WHERE (a >= a) ORDER BY a LIMIT 0;\n COUNT\n-------\n(0 rows)\n```\n\nOrkes resolved this issue ([#33](https://github.com/particledbai/jepsen/issues/33)) in\n1.1.18.9.\n\n### 5.1.2 Comparison of Incomparable Types (#6)\n\nIn ParticleDB 1.1.4 through 1.1.18.4, you could write predicates for\n`UPDATE` and `DELETE` which made equality\ncomparisons between columns of different types.\n\n```\nroot=> CREATE TABLE t (a text, b smallint);\nroot=> DELETE FROM t WHERE (a = b);\nDELETE 0\n```\n\nIn Postgres, this threw an error like\n`ERROR: operator does not exist: text = smallint`. You could\ncompare strings to boolean columns, or literal values like\n`false`. ParticleDB allowed these comparisons on empty\ntables, but would throw an error if a record was present. In 1.1.18.6,\nParticleDB threw an error for empty tables as well ([#6](https://github.com/particledbai/jepsen/issues/6)).\n\n### 5.1.3 Cannot Compare Decimals to Floats (#94)\n\nParticleDB 1.1.18.11.rc-9.2 forgot how to compare large decimal\nliterals to 64-bit floating-point numbers:\n\n```\nroot=> CREATE TABLE t (a double precision);\nroot=> INSERT INTO t (a) VALUES (1.0);\nroot=> SELECT * FROM t WHERE\n       ((0 / 1.2002480805601245E26) >= t.a);\nERROR: arrow compute: Invalid argument error:\n       Invalid comparison operation:\n       Decimal256(76, 44) >= Float64\n```\n\nThis issue ([#94](https://github.com/particledbai/jepsen/issues/94)) no\nlonger appeared by version 1.1.18.11.rc-18.\n\n### 5.1.4 Filter Predicates Reject `NULL` (#12)\n\nSQL has three boolean values: `TRUE`, `FALSE`,\nand `NULL`, which represents an unknown value. In ParticleDB\n1.1.4 through 1.1.18.4, predicates in SELECT, UPDATE, etc. could not\nevaluate to `NULL`, even though the SQL specification states\nthat search conditions are boolean expressions. For example:\n\n```\nroot=> CREATE TABLE t (a boolean);\nroot=> INSERT INTO t (a) VALUES (TRUE);\nroot=> SELECT * FROM t WHERE NULL;\nERROR:  filter predicate must produce boolean\nresult\n```\n\nIn Postgres this would return an empty relation, because\n`NULL` is not `TRUE`. In 1.1.18.6,\n`NULL` predicates evaluated correctly ([#12](https://github.com/particledbai/jepsen/issues/12)).\n\n### 5.1.5 Cannot Compare Values to `NULL` (#13)\n\nPer the ANSI SQL standard, the value of comparing anything to\n`NULL` is *unknown*, represented as the boolean value\n`NULL`. However, ParticleDB 1.1.4 through 1.1.17 sometimes\nrefused to evaluate predicates which compared (e.g.) integers to\n`NULL`.\n\n```\nroot=> CREATE TABLE t (a int);\nroot=> INSERT INTO t (a) VALUES (2), (NULL);\nroot=> UPDATE t SET a = 5 WHERE a = NULL;\nERROR:  arrow compute: Invalid argument error:\nInvalid comparison operation: Int64 == Null\n```\n\nOddly enough, ParticleDB correctly evaluated statements like\n`SELECT 1 = NULL`. Version 1.1.18.6 correctly compared\nintegers to `NULL` ([#13](https://github.com/particledbai/jepsen/issues/13)).\n\n### 5.1.6 Boolean Logic on Integers (#30)\n\nThe SQL standard is clear that boolean logical expressions like\n`A AND B` evaluate to booleans. In ParticleDB 1.1.4 through\n1.1.18.8, they could sometimes be other things:\n\n```\nroot=> SELECT TRUE AND 5;      # => 5\nroot=> SELECT 5 AND TRUE;      # => 5\nroot=> SELECT 5 AND FALSE;     # => FALSE\nroot=> SELECT 5 OR TRUE;       # => TRUE\nroot=> SELECT 'cat' OR FALSE;  # => 'cat'\nroot=> SELECT 'cat' OR 'dog';\nERROR:  OR requires boolean operands\nroot=> SELECT 1 AND 2;\nERROR:  AND conjunct must produce boolean\n```\n\nBy version 1.1.18.9, ParticleDB rejected boolean expressions over\nnon-boolean literals ([#30](https://github.com/particledbai/jepsen/issues/30)).\n\n### 5.1.7 More Boolean/Integer Confusion (#50)\n\nParticleDB 1.1.18.9 disallowed boolean logic on integer literals, but\nallowed it on columns.\n\n```\nroot=> CREATE TABLE t (a int);\nroot=> UPDATE t SET a = NOT a;\nUPDATE 0\n```\n\nLikewise, you could do integer operations on booleans:\n\n```\nroot=> CREATE TABLE t (a boolean);\nroot=> UPDATE t SET a = ((a + a) IS NULL)\n       WHERE a;\nUPDATE 0\n```\n\nOrkes seems to have resolved this issue ([#50](https://github.com/particledbai/jepsen/issues/50)) by\nversion 1.1.18.11.rc-8.\n\n### 5.1.8 Multiple Assignment to a Single Column (#5)\n\nThe SQL standard specifies that an `UPDATE` statement\nshall not contain a column name more than once. However, ParticleDB\n1.1.4 through 1.1.18.4 allowed an `UPDATE` to assign multiple\nvalues to the same column. It seemed to pick the last value,\nlexically.\n\n```\nroot=> CREATE TABLE t (a boolean, b boolean);\nroot=> INSERT INTO t (a, b)\n       VALUES (FALSE, FALSE);\nroot=> UPDATE t SET a = FALSE, a = TRUE;\nUPDATE 1\n```\n\nIn Postgres this returned an error:\n`multiple assignments to same column \"a\"`. However,\nParticleDB applied the update, producing:\n\n```\nroot=> select * from t;\n a | b\n---+---\n t | f\n(1 row)\n```\n\nBy version 1.1.18.6, multiple updates to the same column threw an\nerror ([#5](https://github.com/particledbai/jepsen/issues/5)).\n\n### 5.1.9 Predicates Referring to Non-Existent Columns (#10)\n\nIn ParticleDB 1.1.4 through 1.1.17, predicates in\n`SELECT`, `UPDATE`, and `DELETE`\nstatements could refer to columns which did not exist. For example:\n\n```\nroot=> CREATE TABLE t (a INT);\nroot=> SELECT * FROM t WHERE b = 1;\n a\n---\n(0 rows)\n```\n\nIn Postgres, expressions like this would throw\n`ERROR: column \"b\" does not exist`. By version 1.1.18.6,\nParticleDB did as well ([#10](https://github.com/particledbai/jepsen/issues/10)).\n\n## 5.2 Arithmetic\n\nParticleDB often performed arithmetic in unexpected ways. These\nissues occurred with a single server, single client, and no faults.\n\n### 5.2.1 No Large Integer Literals (#14)\n\nPostgres allows large integer literals like\n`9223372036854775808` (263). However, ParticleDB 1.1.4\nthrough 1.1.17 refused these literals, throwing an\n`invalid integer` error:\n\n```\nroot=> SELECT 9223372036854775808;\nERROR:  invalid integer: 9223372036854775808\n```\n\nIn ParticleDB 1.1.18.6, one could write large integer literals ([#14](https://github.com/particledbai/jepsen/issues/14)).\n\n### 5.2.2 To Infinity, and Beyond (#85)\n\nParticleDB 1.1.18.9 gained an exponentiation operator `^`,\nbut could not apply it to `Infinity`.\n\n```\nSELECT 'Infinity' ^ 2;\n```\n\nThis returned `'Infinity'` in Postgres, but threw\n`operator does not exist: ^ requires numeric operands` in\nParticleDB. By 1.1.18.11.rc-9, Orkes seems to have resolved this issue\n([#85](https://github.com/particledbai/jepsen/issues/85)).\n\n### 5.2.3 `-1 ^ 0 = -1` (#49)\n\nParticleDB 1.1.18.9 considered `-1 ^ 0` to be equal to\n`-1`, rather than `1`.\n\n```\nroot=> select -1 ^ 0;\n -1\n----\n -1\n(1 row)\n```\n\nBy version 1.1.18.11.rc-3, it correctly returned `1` ([#49](https://github.com/particledbai/jepsen/issues/49)).\n\n### 5.2.4 `1.0 + 1E-39 = 1.0` (#88)\n\nParticleDB 1.1.18.6 gained support for literals using scientific\nnotation—for example, using `3E12` to denote 3 × 1012. However, the parser\nturned this notation into floating-point values, rather than precise\ndecimals, as in Postgres. This resulted in floating-point errors:\n\n```\nroot=> SELECT 1.0 + 1E-39 = 1.0;\n ?column?\n----------\n t\n(1 row)\n```\n\nThis issue ([#88](https://github.com/particledbai/jepsen/issues/88)) was\nfixed by 1.1.18.11.rc-9.2.\n\n### 5.2.5 What’s the Largest Number You Can Think Of? (#56)\n\nParticleDB 1.1.4 through 1.1.18.11.rc-1 silently converted large\nfloating-point literals to infinities:\n\n```\nroot=> SELECT -4.56E500;\n ?column?\n-----------\n -Infinity\n(1 row)\n```\n\nBy version 1.1.18.11.rc-3, ParticleDB threw an error like\n`\"4.56E500\" is out of range for type double precision` ([#56](https://github.com/particledbai/jepsen/issues/56)).\n\n### 5.2.6 Exponent Underflow (#61)\n\nIn ParticleDB 1.1.18.9, the exponent operator checked to see if its\narguments would cause an overflow, but did not check for underflow:\n\n```\nroot=> SELECT 200 ^ 200;\nERROR:  value out of range: overflow\nroot=> SELECT -200 ^ -200;\n -0\n----\n  0\n(1 row)\n```\n\nBy version 1.1.18.11.rc-3, ParticleDB threw an underflow error ([#61](https://github.com/particledbai/jepsen/issues/61)).\n\n### 5.2.7 Division by Integer Zero (#34)\n\nIn ParticleDB 1.1.4 through 1.1.18.8, you could divide floating-point\nvalues by integer `0` and receive `Infinity`,\nrather than a `division by zero` error, as in Postgres.\n\n```\nroot=> SELECT 1.0 / 0;\n 1 / 0\n-------\n   inf\n(1 row)\n```\n\nParticleDB 1.1.18.9 returned an error instead ([#34](https://github.com/particledbai/jepsen/issues/34)).\n\n### 5.2.8 Modulo Operator Cannot Handle Floats (#51)\n\nIn ParticleDB 1.1.18.9, the modulo operator `%` threw an\ninternal error when either argument was a floating-point number.\n\n```\nroot=> SELECT 3.0 % 2;\nERROR:  internal error\n```\n\nThe `MOD` function was unaffected, and this issue ([#51](https://github.com/particledbai/jepsen/issues/51)) was\nfixed by 1.1.18.11.rc-1.\n\n### 5.2.9 Broken `NaN` Arithmetic (#36)\n\nIn ParticleDB 1.1.4 through 1.1.18.8, arithmetic operations on\n`NaN` would throw an error:\n\n```\nroot=> SELECT 'NaN' - 1.0;\nERROR:  arrow compute: Invalid argument error:\nInvalid arithmetic operation: Utf8 - Float64\n```\n\nBy 1.1.18.9, arithmetic on `NaN` correctly returned\n`NaN` ([#36](https://github.com/particledbai/jepsen/issues/36)).\n\n### 5.2.10 Broken `NULL` Arithmetic (#92)\n\nIn ParticleDB 1.1.18.11.rc-9.2, arithmetic operations involving\nsufficiently detailed scientific notation literals and `NULL`\nthrew internal errors:\n\n```\nroot=> CREATE TABLE t (a int);\nroot=> INSERT INTO t (a) VALUES (NULL);\nroot=> SELECT * FROM t WHERE\n       (t.a + 2.0000000000000000E-23) IS NULL;\nERROR:  internal error: called `Option::unwrap()`\n        on a `None` value (at crates/exec/...)\n```\n\nThis error ([#92](https://github.com/particledbai/jepsen/issues/92)) no\nlonger appeared in version 1.1.18.11.rc-18.\n\n### 5.2.11 UPDATE Can’t Do Math (#60)\n\nParticleDB 1.1.18.9 introduced a new bug which rendered it unable to\ndo arithmetic operations on floating-point columns in an\n`UPDATE` statement.[17](#fn17) For example:\n\n```\nroot=> CREATE TABLE t (a real);\nCREATE TABLE\nroot=> INSERT INTO t (a) VALUES (1.2);\nINSERT 0 1\nroot=> UPDATE t SET a = a + 1;\nERROR:  internal error\n```\n\nIn 1.1.18.11.rc-1, ParticleDB was (mostly) able to add numbers ([#60](https://github.com/particledbai/jepsen/issues/60)).\n\n### 5.2.12 Same Precision For `real` and `double precision` (#32)\n\nPer the SQL standard, the `double precision` data type\nshould have a greater precision than `real`. In Postgres,\n`real` and `double precision` are ([usually](https://www.postgresql.org/docs/8.1/datatype.html)) a\n32-bit and 64-bit floating-point value, respectively. In ParticleDB,\nboth appear to be implemented as 64-bit floats.\n\n```\nCREATE TABLE t (a real NOT NULL);\nINSERT INTO t (a) VALUES (4.134917993754883E8);\nSELECT * FROM t ORDER BY a;\n  ParticleDB:\n    [{:a 4.134917993754883E8}]\n  Postgres:\n    [{:a 4.134918E8}]\n```\n\nVersion 1.1.18.9 seemed to introduce a 32-bit `real` type\n([#32](https://github.com/particledbai/jepsen/issues/32)).\n\n### 5.2.13 Big Integers (#11)\n\nSimilarly, integer fields (e.g. `smallint`,\n`int`, …) in ParticleDB 1.1.4 through 1.1.17 allowed values\nmuch bigger than their typical ranges. It appeared as if ParticleDB\ninternally represented all integer types as signed 64-bit integers. For\nexample, here are four values too big to fit in a Postgres\n`smallint`:\n\n```\nroot=> CREATE TABLE t (a smallint);\nroot=> INSERT INTO t (a) VALUES (32768);\nroot=> INSERT INTO t (a)\n       VALUES (9223372036854775807);\nroot=> INSERT INTO t (a)\n       VALUES (9223372036854775808);\nroot=> INSERT INTO t (a) VALUES (1.234E40);\nroot=> SELECT * FROM t;\n          a\n---------------------\n               32768\n 9223372036854775807\n                   0\n 9223372036854775807\n(5 rows)\n```\n\nIn Postgres `smallint` is a 16-bit integer; each\n`INSERT` would throw\n`ERROR: smallint out of range`. However, ParticleDB stored\neach of these values with varying results. It preserved 32,768 and 263 − 1 (the largest\ntwo’s-complement signed 64-bit integer). 263 and larger integers were\nsilently converted to `0`. On the other hand, the large\nfloating-point number was clipped to 263 − 1.\n\nSimilar behavior occurred with `int` and\n`bigint`. ParticleDB 1.1.18.6 rejected oversized integers ([#11](https://github.com/particledbai/jepsen/issues/11)).\n\n### 5.2.14 Dividing Nearby Numbers (#86)\n\nParticleDB 1.1.18.9 through 1.1.18.11.rc-8 struggled to divide\nfloating-point numbers, often throwing `Numeric out of range`\nwhen asked to divide values which were well within the range of a 32-bit\nfloat. For example:\n\n```\nroot=> SELECT 81938.0 / 81937.82012987136848;\nERROR:  numeric out of range\n```\n\nThis value should be approximately `1.00000219`. Orkes\nseems to have resolved this issue ([#86](https://github.com/particledbai/jepsen/issues/86)) in\nversion 1.1.18.11.rc-9.\n\n### 5.2.15 Floating-Point Errors in Integer Values (#87)\n\nIn ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-8, adding small\nnumbers (e.g. `0.1`) to large integer columns could cause\ntheir values to change by surprising amounts. For example:\n\n```\nroot=> CREATE TABLE t (a bigint);\nroot=> INSERT INTO t (a) VALUES\n       (58950791971326900);\nroot=> UPDATE t SET a = (a + 1E-1);\nroot=> SELECT a FROM t;\n         a\n-------------------\n 58950791971326896\n```\n\nAdding `0.1` to this column actually subtracted\n`4`. Jepsen suspects this bug ([#87](https://github.com/particledbai/jepsen/issues/87)) was\ncaused by performing these operations on floating-point values (which\ncannot represent all integers) then coercing the result back to the\nnearest integer value. Version 1.1.18.11.rc-9 seems to have fixed\nthis.\n\n### 5.2.16 Floats Are Not Equal to Quoted Forms (#19)\n\nPostgres considers floating-point numbers like `0.0` to be\nequal to their quoted representations—this is useful because some\nfloating-point values, like `'Infinity'`, are written with\nquotes. ParticleDB 1.1.4 through 1.1.18.4 thought otherwise:\n\n```\nroot=> SELECT 0.0 = '0.0';\n false\n-------\n f\n(1 row)\n```\n\nThis issue ([#19](https://github.com/particledbai/jepsen/issues/19))\nappeared resolved as of version 1.1.18.6.\n\n### 5.2.17 Floats Are Not Equal to Integers (#63)\n\nIn ParticleDB 1.1.4 through 1.1.18.6, floats were not equal to\nintegers.\n\n```\nroot=> SELECT 0.0 = 0;\n false\n-------\n f\n(1 row)\n```\n\nBy version 1.1.18.8, reasonably-sized integers were equal to their\nfloating-point counterparts ([#63](https://github.com/particledbai/jepsen/issues/63)).\n\n### 5.2.18 Double Precision Floats Appear Equal to Almost All Integers (#80)\n\nVersion 1.1.18.9 introduced a new bug which made predicates like\n`WHERE x != y` fail to match for most double precision floats\nin (-1.0, 1.0). It was as if all small floats were equal to all nonzero\nintegers. If we took a table containing a few integers, and performed\nsome simple queries that should return every row…\n\n```\nCREATE TABLE t (a int);\nINSERT INTO t (a) VALUES (-1), (0), (5);\nSELECT * FROM t WHERE a != (0.1 ^ 25);\nSELECT * FROM t WHERE\na != 0.90000000000000000000000000000000000000;\nSELECT * FROM t WHERE\na != -0.34656654456890453890613489540952456545;\n```\n\nAll three of these queries returned only a single row,\n`0`—implying that `0.90...` was equal to\n`-1` and `5`. The only exception seemed to be\n`WHERE a != 0.0...`, which returned no rows at all. This was\nalso wrong, but somewhat less so. Version 1.1.18.11.rc-9 seems to have\nresolved the matter ([#80](https://github.com/particledbai/jepsen/issues/80)).\n\n## 5.3 Strings\n\nWe found several problems with strings in ParticleDB, including\ndatatypes that could hold more or fewer characters than they ought, and\nimproper sort orders. These issues occurred with a single server, single\nclient, and no faults.\n\n### 5.3.1 `CHARACTER(n)` Allows More Than `n` Characters (#64, #52)\n\nThe SQL standard defines a variety of fixed-width string datatypes\nlike `character(3)`, which holds a string of exactly three\ncharacters. Unfortunately, ParticleDB 1.1.4 through 1.1.18.4 were happy\nto store much larger strings in those columns.[18](#fn18)\n\n```\nroot=> CREATE TABLE t (a character(3));\nroot=> INSERT INTO t\n       VALUES ('large language model');\nroot=> SELECT * FROM t;\n          a\n----------------------\n large language model\n(1 row)\n```\n\nVersion 1.1.18.6 rejected inserts and updates of too-long string\nliterals ([#64](https://github.com/particledbai/jepsen/issues/64)), but\nappeared to perform these checks only at runtime; one could still say\n`UPDATE t SET a = 'far too long'` so long as it matched no\nrows ([#52](https://github.com/particledbai/jepsen/issues/52)). That\nissue was resolved in version 1.1.18.11.rc-3.\n\n### 5.3.2 `CHARACTER(n)` Allows Fewer Than `n` Characters (#55)\n\nWhile 1.1.18.6 rejected strings that were too large for a fixed-width\nstring column, 1.1.4 through 1.1.18.9 also allowed strings that were too\nshort. Normally, shorter strings are padded with spaces to the specified\nlength, but in ParticleDB, short strings were returned unchanged.\n\n```\nCREATE TABLE t (a character(16));\nINSERT INTO t VALUES ('smol');\nSELECT * FROM t;\n  -> {:a \"smol\"}\n```\n\nThis should have returned `\"smol \"`, with\npadding spaces. ParticleDB 1.1.18.11.rc-1 seems to have resolved the\nissue ([#55](https://github.com/particledbai/jepsen/issues/55)).\n\n### 5.3.3 Default Text Collation (#22)\n\nOn a platform with `LC_COLLATE = en_US.UTF-8`, Postgres\nuses that collation by default. ParticleDB appeared to collate strings\ndifferently. Sorting via queries like\n`SELECT ... ORDER BY name`, or comparisons like\n`'a' < 'B'`, returned different results between the two\ndatabases. Where Postgres considered `'a' < 'B'`,\nParticleDB believed the opposite.\n\nExperiments suggest ParticleDB sorted by Unicode code point order,\nrather than the [Unicode\nCollation Algorithm](https://www.unicode.org/reports/tr10/) (UCA) with the locale’s collation rules. It may\nhave been compatible with `ucs_basic`, `C`, or\n`pg_c_utf8`. ParticleDB version 1.1.18.9 seemed to choose a\ndefault collation more like UCA ([#22](https://github.com/particledbai/jepsen/issues/22)).\n\n### 5.3.4 Mis-Sorted Text (#42, #93)\n\nThroughout our collaboration ParticleDB struggled to sort things.\nVersions 1.1.4 through 1.1.18.8 accepted an explicit\n`unicode` text collation, then ignored it:\n\n```\nroot=> CREATE TABLE t (b text COLLATE unicode);\nroot=> INSERT INTO t (b)\n       VALUES ('a'), ('b'), ('B');\nroot=> SELECT * FROM t ORDER BY b;\n b\n---\n B\n a\n b\n(3 rows)\n```\n\nIn Postgres’ `unicode` collation, letters like\n`a` and `A` sort similarly, despite being of\ndifferent cases. This query would have returned\n`'a', 'b', 'B'`.\n\nParticleDB 1.1.18.9 sorted English uppercase and lowercase letters\ncorrectly, but only for some queries—adding a `LIMIT` clause\nto a `SELECT` statement would cause it to revert to the old\nsort order. Version 1.1.18.11.rc-1 resolved that, but sorted other\ncharacters, like punctuation, oddly:\n\n```\nroot=> CREATE TABLE t (a text COLLATE unicode);\nroot=> INSERT INTO t VALUES\n       ('_aa'), ('a'), ('aa'), ('aaa');\nroot=> SELECT * FROM t ORDER BY a ASC;\n  a\n-----\n a\n _aa\n aa\n aaa\n(4 rows)\n```\n\nDifferent sorting of whitespace and punctuation ([#42](https://github.com/particledbai/jepsen/issues/42)) seemed\nbetter in version 1.1.18.11.rc-7, but reappeared in 1.1.18.11.rc-9\nthrough 1.1.18.11.rc-19 ([#93](https://github.com/particledbai/jepsen/issues/93)). Text\nsorting seemed improved in 1.1.18.11.rc-21.\n\n---\n\n1. There are also a handful of unrelated databases called\n   ParticleDB, mainly used in particle physics—for example, the [database\n   of particles](https://software.belle2.org/light-2409-toyger/doxygen/dir_49ab761b69e05c004232ce029feed737.html) used in [the Belle II\n   physics experiment](https://en.wikipedia.org/wiki/Belle_II_experiment), or [this\n   one](https://github.com/MannyMoo/gaudi-script-builder/blob/master/python/GaudiScriptBuilder/ParticleDB.py) at the Large Hadron Collider.[↩︎](#fnref1)\n2. The Particles feature was not available during our work;\n   we focus on ParticleDB’s SQL interfaces.[↩︎](#fnref2)\n3. One envisions a world of meta-recursive database\n   documentation in the style of [Raymond Smullyan](https://raymondsmullyan.com/). “One page of\n   the documentation always tells the truth. Another tells only lies…”[↩︎](#fnref3)\n4. Presumably [Two-Phase\n   Locking](https://en.wikipedia.org/wiki/Two-phase_locking).[↩︎](#fnref4)\n5. *With apologies to Deleuze & Guattari*—Thus,\n   the tuple without fields is opposed less to fields as such than to the\n   organization of the fields insofar as it composes a row. *Le n-uplet\n   sans composants* is not a deleted tuple but a living tuple all the\n   more alive and teeming once it has blown apart the client and its memory\n   model. Structs commingling on the heap. [Self-modifying\n   Perl](https://github.com/spencertipping/writing-self-modifying-perl/blob/c25d7374dccac01413725acea4d0776badaf9233/writing-self-modifying-perl.pdf). The full tuple without fields is a tuple populated by\n   multiplicities.[↩︎](#fnref5)\n6. Madred, 2369.[↩︎](#fnref6)\n7. [Cameron, 1986](https://www.youtube.com/watch?v=kfPwJKz_rlo).[↩︎](#fnref7)\n8. He explained that the Database—the Denormalized, the\n   Data Lakehouse, the giant Monad—is a table without tuples. *Le\n   tableau sans n-uplets* is permeated by unformed, unstable values, by\n   dataflows in all directions, by free JSON or nomadic singletons, by mad\n   or transitory transactions. That, however, was not the question at hand.\n   For there simultaneously occurs upon the database a very important,\n   inevitable phenomenon that is beneficial in many respects and\n   unfortunate in many others: schematization. Schemata are Layers, Belts.\n   They consist of giving form to values, of imprisoning arrays or locking\n   singletons into systems of record and replication, of producing upon the\n   body of the database records large and small and organizing them into\n   normal aggregates. Schemata are acts of capture, they are like “black\n   holes” or occlusions striving to seize whatever comes within their\n   reach. They operate by coding and normalization upon the database; they\n   proceed simultaneously by code and normality. The Schemata are\n   judgements of God; schematization in general is the entire system of the\n   judgement of God (but the database, or the table without tuples,\n   constantly eludes that judgement, flees and becomes deschematized,\n   destructured, denormalized).[↩︎](#fnref8)\n9. To make matters more confusing,\n   `serializable` in versions 1.1.4 through 1.1.18.11.rc-11.2\n   was (depending on which version of the binary you asked) either an alias\n   for `table-2pl` or possibly `occ-v3`. In\n   1.1.18.11.rc-22, `serializable` reappeared as a new Strong\n   Serializable transaction mode, also based on Serializable Snapshot\n   Isolation (SSI), but perhaps not the `serializable-ssi` level\n   present in earlier builds, which was distinct from\n   `strict-serializable`.[↩︎](#fnref9)\n10. De Lioncourt, 1793, presaging Borges, 1941.[↩︎](#fnref10)\n11. See ANSI/NISO Z39.19-2005 (R2010) “Celestial Emporium\n    of Benevolent Knowledge” for the full taxonomy of database records: rows\n    belonging to the DBA, tombstones, those which from afar look like JSON,\n    et cetera, etc.[↩︎](#fnref11)\n12. Jepsen encourages the consistency model research\n    community to come up with more official terms for this.[↩︎](#fnref12)\n13. Running `particledb --version` might not\n    return the actual version. For instance, 1.1.16 through 1.1.18.11.rc-18\n    all identified themselves as “spanner 1.1.16”.[↩︎](#fnref13)\n14. Barthes can rejoice: not only is the author dead, but\n    they were never alive to begin with. Nothing is true, everything is\n    permitted.[↩︎](#fnref14)\n15. MongoDB had [962\n    employees in 2018](https://www.sec.gov/Archives/edgar/data/1441816/000144181618000028/mdb-013118x10k.htm), and Cockroach Labs had [400\n    in 2022](https://www.cockroachlabs.com/blog/why-join-cockroach-labs/). Databases are hard![↩︎](#fnref15)\n16. Or perhaps not. Not every application requires perfect\n    correctness. Vibrational Database Management Systems might be fine, at\n    least some of the time.[↩︎](#fnref16)\n17. [Mattel,\n    1992](http://itre.cis.upenn.edu/~myl/languagelog/archives/002892.html).[↩︎](#fnref17)\n18. Danielewski, 2000.[↩︎](#fnref18)\n","body_html":"<h1 id=\"jepsen-particledb-1-1-4\">Jepsen: ParticleDB 1.1.4</h1>\n<p>Kyle Kingsbury, 2026-10-31</p>\n<p><a href=\"https://particledb.ai\" rel=\"nofollow ugc noopener\">ParticleDB</a> is a multi-model\ndatabase designed for AI workloads. We tested versions 1.1.4 through\n1.1.18.11.rc-22 and found that ParticleDB’s claims of “Full ACID”\nwere somewhat premature. We present 96 issues in ParticleDB, ranging\nfrom the pedestrian (crashes, deadlocks, data loss, etc.) to the\npiquant (partially-invisible rows which could not be deleted,\n<code>WHERE 4 = 5</code>, dropped tables re-emerging in distorted\nforms, confused columns, strings turning into integers, and so on).\nMost issues required just a single server, single client, and no\nfaults. By version 1.1.18.11.rc-22, all but ten of those issues\nappeared resolved. However, ParticleDB continued to lose committed\nwrites during normal operation with the default <code>occ</code>\ntransaction mode, and allowed internal anomalies and Lost Update\nwith <code>serializable</code>, the strongest setting. Orkes, the\nmakers of ParticleDB, have since released version 1.1.19-rc.13; <a href=\"https://orkes.io/blog/particledb-database-built-for-ai-agents\" rel=\"nofollow ugc noopener\">they\nclaim</a> that it resolves all remaining issues discussed in this\nreport. This research was funded by Orkes, Inc., and conducted in\naccordance with the <a href=\"https://jepsen.io/analyses/ethics\" rel=\"nofollow ugc noopener\">Jepsen ethics\npolicy</a>.</p>\n<h1 id=\"1-background\">1 Background</h1>\n<p>Modern applications often spread state across a variety of systems:\ndatabases, caches, queues, data warehouses, search indices, and so on.\nCoordinating these systems is challenging: caches are often stale, the\nanalytics platform might lag days behind the primary database, the\nsearch index might lose data, and transactional updates across disparate\nsystems <a href=\"https://www.semanticscholar.org/paper/Typhon%3A-Consistency-Semantics-for-Data-Processing-Arora-Nawab/018ce7bfc890712fa62ab988fe0703bccb806907\" rel=\"nofollow ugc noopener\">remains\nan open research problem</a>.</p>\n<p>ParticleDB<a href=\"#fn1\">1</a> aims to unify these workloads in a\nsingle engine with a special focus on AI agents and applications. It\noffers transactional SQL, a key-value store, vector search, live\nanalytics, and user-defined functions called <em>Particles</em>.<a href=\"#fn2\">2</a> To integrate with existing systems,\nclients can connect through <a href=\"https://particledb.ai/connectivity/overview/\" rel=\"nofollow ugc noopener\">a variety of\ninterfaces</a>, including the Postgres and Redis protocols, as well as\nits own gRPC and HTTP2 APIs. Written from the ground up in Rust,\nParticleDB claims to be orders of magnitude faster than <a href=\"https://www.postgresql.org/\" rel=\"nofollow ugc noopener\">Postgres</a> while preserving the\nsame safety characteristics.</p>\n<p>ParticleDB was built by <a href=\"https://orkes.io/\" rel=\"nofollow ugc noopener\">Orkes</a> to\nsupport their workflow orchestration platform, <a href=\"https://jepsen.io/analyses/orkes-conductor-5.3.2\" rel=\"nofollow ugc noopener\">Conductor</a>.\nConductor was initially built to store data in Redis and Postgres, but\nto improve performance, Orkes opted to write their own database.\nParticleDB 1.0.0 offered a limited feature set specifically for\nConductor’s queries, and was deployed as the backing database for some\nof their Conductor clusters. A few Orkes customers followed suit. In\npreparation for a broader release, Orkes undertook a major refactoring\nof ParticleDB, adding a custom wire protocol, more datatypes, additional\nOLAP queries, and more. While Orkes and their customers continued to run\n1.0.0, they soft launched the refactored versions on <a href=\"https://particledb.ai\" rel=\"nofollow ugc noopener\">particledb.ai</a>. Orkes engaged Jepsen to\ntest these versions of ParticleDB, beginning with 1.1.4.</p>\n<p>While ParticleDB could run as a single node, it also offered a\nreplicated cluster using the <a href=\"https://raft.github.io/\" rel=\"nofollow ugc noopener\">Raft</a>\nconsensus protocol. This replication feature was available only in\nenterprise cloud deployments. The builds Orkes provided did not support\nreplication, so this report covers only single-node behavior.</p>\n<h2 id=\"1-1-safety\">1.1 Safety</h2>\n<p>ParticleDB’s home page <a href=\"https://web.archive.org/web/20260518203012/https://particledb.ai/\" rel=\"nofollow ugc noopener\">advertised</a>\nthat ParticleDB was “formally proven” with a TLA+ proof. However, during\nour collaboration no TLA+ specification was available. It is unclear\nwhat properties might be proven, or whether they described ParticleDB as\nbuilt.</p>\n<p>ParticleDB offered <a href=\"https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#wal-sync-modes\" rel=\"nofollow ugc noopener\">three\ntactics</a> for syncing its Write-Ahead Log (WAL). The <code>sync</code>\nmode called <code>fsync</code> on each commit, <code>groupsync</code>\nbatched multiple transactions into a single <code>fsync</code>, and\n<code>nosync</code> skipped <code>fsync</code> altogether. The <a href=\"https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#wal-sync-modes\" rel=\"nofollow ugc noopener\">Transaction\nEngine documentation</a> described <code>sync</code> as offering\n“stronger durability” than <code>groupsync</code>. On the other hand,\n<code>particledb start --help</code> said the <code>groupsync</code>\nmode ensured that “COMMIT waits for flush”, and Orkes’ internal operator\nguide concurred: <code>groupsync</code> offered the “same durability as\n<code>sync</code>”.</p>\n<p>ParticleDB’s home page proudly claimed “Full ACID. 159K TPS.”\nHowever, the text immediately below this said “snapshot isolation”,\nwhich is <a href=\"https://jepsen.io/consistency/models/snapshot-isolation\" rel=\"nofollow ugc noopener\">not fully\nisolated</a>.</p>\n<p>Most SQL databases use the\n<code>SET TRANSACTION ISOLATION LEVEL</code> statement to control the\nconcurrent semantics of a transaction. However, ParticleDB <a href=\"https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#transaction-modes\" rel=\"nofollow ugc noopener\">ignored\nor rejected these statements</a>. In some cases it instructed users to\nchoose <code>READ COMMITTED</code>, a relatively weak isolation level.\nIn other cases it accepted requests for different isolation levels, and\nsilently ignored them (despite documentation to the contrary).</p>\n<p>Instead, transaction isolation was governed by the\n<code>--txn-mode</code> setting passed at startup. The transaction\nengine documentation <a href=\"https://web.archive.org/web/20260518193013/https://particledb.ai/architecture/transaction-engine/#transaction-modes\" rel=\"nofollow ugc noopener\">described\nfour modes</a>:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Mode</th><th>Behavior</th><th>When to use</th></tr></thead><tbody><tr><td><code>fast</code></td><td>No locks; commit-on-write</td><td>Bulk loads, single-writer benchmarks</td></tr><tr><td><code>occ</code></td><td>Overlay + commit-time OCC validation (default)</td><td>General OLTP — safe under concurrency</td></tr><tr><td><code>zero-copy-occ</code></td><td>Partition-pinned OCC; single-partition txns skip cross-partition coordination</td><td>High-throughput OLTP on partition-affine workloads (TPC-C, key-value)</td></tr><tr><td><code>table-2pl</code></td><td>Exclusive table locks at first touch</td><td>Strictest serialization, simple semantics</td></tr></tbody></table></div>\n<p>This table was difficult to interpret. While the <a href=\"https://web.archive.org/web/20260518193127/https://particledb.ai/sql/transactions/#best-practices\" rel=\"nofollow ugc noopener\">best\npractices documentation</a> warned that users should “pick\n<code>--txn-mode</code> deliberately”, since “it changes correctness and\nthroughput characteristics more than SQL text does”, the documentation\ndid not explain what these modes actually guaranteed. The\n<code>fast</code> mode was clearly intended for weaker isolation, but\nwas that Read Atomic, Read Uncommitted, or something else? The\n<code>occ</code> mode was similarly unclear. “Commit-time OCC\nvalidation” could describe a broad spectrum of consistency models from\nRead Uncommitted to Strong Serializable, but “safe under concurrency”\nsuggests it might be Serializable. The <code>zero-copy-occ</code> mode\ndescribed a performance optimization which could be as safe as\n<code>occ</code>, but it was <em>not</em> described as “safe under\nconcurrency”—raising the question: in what ways was it unsafe? Finally,\nif <code>table-2pl</code> offered the “strictest serialization”, was\n<code>occ</code> some weaker isolation level?</p>\n<p>The <a href=\"https://web.archive.org/web/20260518193140/https://particledb.ai/operations/cli-reference/\" rel=\"nofollow ugc noopener\">CLI\nreference page</a> contained another table which described\n<code>occ</code> and <code>table-2pl</code> as “Jepsen-safe”, whereas\n<code>fast</code> was not “Jepsen-safe”, but did not explain their\nsafety properties. Since Jepsen evaluates safety with respect to\ndocumentation and marketing claims, it is not clear how to interpret\nthese terms.<a href=\"#fn3\">3</a></p>\n<p>Running <code>particledb start --help</code> provided the following\ntable:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Mode</th><th>Description</th></tr></thead><tbody><tr><td><code>fast</code></td><td>No locks, maximum OLTP throughput (unsafe under concurrency)</td></tr><tr><td><code>occ</code></td><td>Table locks + commit-time validation (safe for most OLTP)</td></tr><tr><td><code>row-2pl</code></td><td>Alias for <code>occ</code></td></tr><tr><td><code>zero-copy-occ</code></td><td>Partition-pinned OCC (highest throughput safe mode)</td></tr><tr><td><code>table-2pl</code></td><td>Exclusive table locks, full serialization (strictest)</td></tr><tr><td><code>serializable</code></td><td>Alias for <code>table-2pl</code></td></tr><tr><td><code>serializable-ssi</code></td><td>Adaptive SSI (Cahill 2008), PL-3 with optimistic semantics (v1 scaffolding — see SSI_ADAPTIVE_DESIGN.md)</td></tr></tbody></table></div>\n<p>The point of Optimistic Concurrency Control (OCC) is generally to\n<em>avoid</em> taking locks, so the mention of table locks and a\n<code>row-2pl</code><a href=\"#fn4\">4</a> alias was surprising. Moreover,\n“safe for most OLTP” undermines the claim that <code>occ</code> was\n“safe under concurrency”; it suggests there are some OLTP workloads\nwhich are unsafe!</p>\n<p>Orkes explained that this documentation was wrong: <code>occ</code>\nwas actually optimistic, and the <code>row-2pl</code> alias was to\npreserve compatibility with an earlier two-phase locking mechanism. They\nalso provided an unpublished operator guide with detailed descriptions\nof the various transaction modes. This document was <a href=\"https://www.tiktok.com/@charlixcx/video/7603137654682570006\" rel=\"nofollow ugc noopener\">difficult\nto interpret</a>, but did specify the following consistency models:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Mode</th><th>Consistency Model</th></tr></thead><tbody><tr><td><code>fast</code></td><td>Unknown</td></tr><tr><td><code>occ-v3</code> (a.k.a. <code>occ</code>, <code>occ-v2</code>, <code>row-2pl</code>)</td><td>Snapshot Isolation</td></tr><tr><td><code>zero-copy-occ</code> (a.k.a. <code>zocc</code>)</td><td>Snapshot Isolation</td></tr><tr><td><code>serializable-ssi</code> (a.k.a. <code>ssi</code>)</td><td>Serializable</td></tr><tr><td><code>table-2pl</code> (a.k.a. <code>serializable</code>)</td><td>Strong Serializable</td></tr></tbody></table></div>\n<p>Sources disagreed about the default transaction and WAL sync modes,\nbut in general, <code>occ-v3</code> and <code>groupsync</code> appeared\nto be the default.</p>\n<h1 id=\"2-test-design\">2 Test Design</h1>\n<p>We wrote a <a href=\"https://github.com/jepsen-io/particledb\" rel=\"nofollow ugc noopener\">test\nsuite</a> for ParticleDB using the <a href=\"https://github.com/jepsen-io/jepsen\" rel=\"nofollow ugc noopener\">Jepsen testing library</a>.\nWe ran ParticleDB versions 1.1.4 through 1.1.18.11.rc-22 on individual\nDebian 13 nodes, both in containers and EC2 VMs. Our test harness could\nsimulate process crashes by killing ParticleDB, simulate the loss of\nun-<code>fsync</code>ed writes using <a href=\"https://github.com/dsrhaslab/lazyfs\" rel=\"nofollow ugc noopener\">LazyFS</a>, and truncate or\ncorrupt disk files. We ran ParticleDB using each transaction mode and\nall three WAL sync modes.</p>\n<p>We first tried ParticleDB’s official Java SDK. While the <a href=\"https://web.archive.org/web/20260518203023/https://particledb.ai/sdks/java/\" rel=\"nofollow ugc noopener\">documentation\nindicated</a> that the library was available as a Maven dependency,\nneither the JAR nor source code were actually available. Instead, we\nbegan with the official Postgres JDBC client, version 42.7.11, and later\nadded support for the <a href=\"https://particledb.ai/sdks/java/\" rel=\"nofollow ugc noopener\">ParticleDB Java SDK</a>,\nversions 1.0.0 through 1.0.2.</p>\n<h2 id=\"2-1-workloads\">2.1 Workloads</h2>\n<p>We used several Jepsen workloads from other SQL databases, and\ndesigned new ones specifically for ParticleDB.</p>\n<p>The <a href=\"https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/rw.clj\" rel=\"nofollow ugc noopener\"><code>rw</code>\nworkload</a> performed randomly generated transactions involving reads\nand writes of integer values identified by unique integer keys. We\nstored each value in a single row and accessed them both via primary and\nunindexed secondary keys. Keys and values were encoded as a variety of\ndatatypes, including integers, decimals, and strings. We assumed that\nwrites followed reads within each transaction, and that individual keys\nwere sequentially consistent, and used these assumptions to infer\nconstraints on the <a href=\"https://jepsen.io/consistency/dependencies\" rel=\"nofollow ugc noopener\">version orders and\ntransaction dependency graphs</a>. Jepsen’s transactional checker, <a href=\"https://github.com/jepsen-io/elle\" rel=\"nofollow ugc noopener\">Elle</a>, then searched for\ncycles in that dependency graph, as well as a variety of non-cyclic\nphenomena like <a href=\"https://jepsen.io/consistency/phenomena/g1a\" rel=\"nofollow ugc noopener\">G1a\n(Aborted Read)</a>.</p>\n<p>The <a href=\"https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/append.clj\" rel=\"nofollow ugc noopener\"><code>append</code></a>\nworkload also performed randomly generated transactions over rows\nidentified by unique integer keys, but the values were lists of\nintegers. Transactions could read one of these lists by key, or append a\nunique integer to a list by key. We encoded each list as a\ncomma-separated <code>text</code> field, and used SQL\n<code>concat</code> to append to them. Rows could be selected via\nprimary key or unindexed secondary key, or via a join to an indirection\ntable. From there, we used Elle to infer version orders and dependency\ngraphs, and searched for a variety of cyclic and non-cyclic\nphenomena.</p>\n<p>When these workloads revealed consistency violations within a single\ntransaction, we designed an <a href=\"https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal.clj\" rel=\"nofollow ugc noopener\"><code>internal</code></a>\nworkload to stress internal consistency. This workload performed a\nseries of randomly generated transactions involving <code>INSERT</code>,\n<code>UPDATE</code>, <code>DELETE</code>, and <code>SELECT</code>\nstatements over a simple key-value table, and verified that within each\nindividual transaction, reads observed prior writes.</p>\n<p>To investigate tantalizing anomalies in the <code>internal</code>\nworkload, we designed the <a href=\"https://github.com/jepsen-io/particledb/blob/e804ab8b0528b089b908b6249d2e1e4ac966802f/src/jepsen/particledb/workload/compare.clj\" rel=\"nofollow ugc noopener\"><code>compare</code></a>\nworkload. This workload used Clojure’s <a href=\"https://github.com/clojure/test.check\" rel=\"nofollow ugc noopener\">test.check</a> generative\ntesting library to <a href=\"https://github.com/jepsen-io/particledb/blob/e804ab8b0528b089b908b6249d2e1e4ac966802f/src/jepsen/particledb/gen/tc.clj\" rel=\"nofollow ugc noopener\">produce</a>\nrandom tables and simple SQL statements over them, then executed those\nstatements against both ParticleDB and PostgreSQL. If ParticleDB’s\nbehavior was <a href=\"https://github.com/jepsen-io/particledb/blob/e804ab8b0528b089b908b6249d2e1e4ac966802f/src/jepsen/particledb/workload/compare.clj#L71-L238\" rel=\"nofollow ugc noopener\">not\nequivalent</a> to PostgreSQL’s, the compare workload would <a href=\"https://github.com/jepsen-io/particledb/blob/d4721aeef9435af0131eacfb2163059272b13d57/src/jepsen/particledb/gen/tc.clj#L1195-L1234\" rel=\"nofollow ugc noopener\">search</a>\nfor a simpler sequence of statements that could reproduce the same\ndivergence. Although our SQL generator was relatively simple—just a few\nbasic datatypes, literals, arithmetic, comparisons, boolean logic,\ntransactions, and some simple <code>LIMIT</code>, <code>OFFSET</code>,\nand <code>ORDER BY</code> clauses—it proved remarkably fruitful.</p>\n<p>Drawing on these successes, we devised a hybrid <a href=\"https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal_sim.clj\" rel=\"nofollow ugc noopener\"><code>internal-sim</code></a>\nworkload to search for violations of internal consistency in more\ncomplex transactions. This workload <a href=\"https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/gen.clj\" rel=\"nofollow ugc noopener\">generated</a>\na random set of tables, then concurrently executed <a href=\"https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal_sim.clj#L370-L406\" rel=\"nofollow ugc noopener\">random\ntransactions</a> over them. It then <a href=\"https://github.com/jepsen-io/sql/blob/81019139aa4bdc4b597cf9b63fa1f0c6b697a753/src/jepsen/sql/workload/internal_sim.clj#L115-L368\" rel=\"nofollow ugc noopener\">checked\neach transaction</a> independently to ensure that each of its statements\nobserved results consistent with those preceding. For example, if a\ntransaction executed <code>DELETE ... WHERE 2 &lt; c</code>, then\nobserved a row with <code>c = 3</code>, we could prove that transaction\nwas not internally consistent.</p>\n<h1 id=\"3-results\">3 Results</h1>\n<p>We found 96 issues with ParticleDB, which we present in several\ncategories below. All but three of these issues occurred in healthy,\nsingle-node instances of ParticleDB. Busy readers may wish to jump to\nthe <a href=\"#discussion\">Discussion</a> section, which summarizes our\nfindings.</p>\n<ul><li><p>[Availability and</p><p>Performance](#availability-and-performance)</p></li><li><a href=\"#broken-features\">Broken Features</a></li><li><a href=\"#predicates\">Predicates</a></li><li><a href=\"#constraints\">Constraints</a></li><li><a href=\"#joins\">Joins</a></li><li><a href=\"#zombies\">Zombies</a></li><li><a href=\"#state-corruption\">State Corruption</a></li><li><p>[Transactions and</p><p>Concurrency](#transactions-and-concurrency)</p></li><li><a href=\"#with-faults\">With Faults</a></li></ul>\n<p>Additional results are presented in <a href=\"#appendix-a\">Appendix\nA</a>:</p>\n<ul><li><a href=\"#type-safety-and-syntax\">Type Safety and Syntax</a></li><li><a href=\"#arithmetic\">Arithmetic</a></li><li><a href=\"#strings\">Strings</a></li></ul>\n<h2 id=\"3-1-availability-and-performance\">3.1 Availability and Performance</h2>\n<p>Under normal operation, ParticleDB was slow, deadlocked, and crashed\nwithout warning.</p>\n<h3 id=\"3-1-1-abysmal-performance-with-table-2pl-62\">3.1.1 Abysmal Performance With <code>table-2pl</code> (#62)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.4,\n<code>--txn-mode table-2pl</code> was so slow as to be essentially\nunusable. While some transactions might complete in a few milliseconds,\nmost took 10, 20, or even 30 seconds, then failed.</p>\n<p>For example, take <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260608T155928-table-2pl-perf.zip\" rel=\"nofollow ugc noopener\">this\ntest run</a> with a single node running 1.1.18.4, transaction mode\n<code>table-2pl</code>, and WAL mode <code>groupsync</code>. With ten\nconcurrent clients, most transactions blocked for tens of seconds before\nthrowing\n<code>could not serialize access due to concurrent update</code>, or an\nIO error.</p>\n<p>High latencies (<a href=\"https://github.com/particledbai/jepsen/issues/62\" rel=\"nofollow ugc noopener\">#62</a>)\nappeared improved by 1.1.18.6, which could execute transactions with\nsub-100 millisecond latencies for at least a few seconds.</p>\n<h3 id=\"3-1-2-mystery-crash-81\">3.1.2 Mystery Crash (#81)</h3>\n<p>While 1.1.18.6 improved latencies, it also crashed frequently.\nCrashes occurred spontaneously with various transaction modes, WAL\nmodes, and client interfaces; there was no obvious cause. Nodes would\nrun for a few hundred to a few thousand seconds before exiting. No error\nmessages were printed. These crashes (<a href=\"https://github.com/particledbai/jepsen/issues/81\" rel=\"nofollow ugc noopener\">#81</a>) no\nlonger appeared in 1.1.18.11.rc-18.</p>\n<h3 id=\"3-1-3-deadlock-after-two-transactions-27\">3.1.3 Deadlock After Two Transactions (#27)</h3>\n<p>In ParticleDB 1.1.18.8, submitting as few as two concurrent\ntransactions to a healthy, single-node deployment could cause it to\ndeadlock. This occurred both using the Postgres and ParticleDB\ninterfaces. The node would log an error message like\n<code>[DEADLOCK] 1 cycles detected at SystemTime ...</code>, and an\nempty backtrace. In some cases new clients could attach and perform\ntransactions; in others, new clients would also hang indefinitely. This\nmay reflect multiple deadlock bugs.</p>\n<p>With the <code>append</code> workload, version 1.1.18.9 generally\ncould not run for more than 45 seconds without locking up. Version\n1.1.18.11.rc-1 could reliably run for two minutes without lockups, and\nwe declared the issue (<a href=\"https://github.com/particledbai/jepsen/issues/27\" rel=\"nofollow ugc noopener\">#27</a>)\nresolved.</p>\n<h3 id=\"3-1-4-deadlock-with-table-2pl-71\">3.1.4 Deadlock With <code>table-2pl</code> (#71)</h3>\n<p>While version 1.1.18.11.rc-1 fixed the most frequent deadlocks, we\ncontinued to find deadlocks through 1.1.18.11.rc-6 when running with\n<code>table-2pl</code>. An otherwise healthy ParticleDB node would\nexecute transactions for about 30 seconds, then get stuck forever. The\nnode would log <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T085629-1.1.18.11.rc-1-table-2pl-deadlock.zip\" rel=\"nofollow ugc noopener\">a\nsimilar deadlock error message</a> to #27, and all requests with the\nParticleDB client would throw <code>PDB JDBC: null</code> after a 100\nsecond default timeout. These deadlocks (<a href=\"https://github.com/particledbai/jepsen/issues/71\" rel=\"nofollow ugc noopener\">#71</a>)\nappeared resolved by 1.1.18.11.rc-7.</p>\n<h2 id=\"3-2-broken-features\">3.2 Broken Features</h2>\n<p>Orkes’ <a href=\"https://web.archive.org/web/20260519195820/https://particledb.ai/connectivity/pg-wire/\" rel=\"nofollow ugc noopener\">public\ndocumentation</a> claimed broad support for Postgres features, and the\ninternal operator guide concurred:</p>\n<blockquote><p>The engine speaks PostgreSQL v3 wire protocol natively — anything\nthat speaks pgjdbc / libpq / psycopg2 / pq connects unchanged.\n<code>BEGIN</code>, <code>COMMIT</code>, <code>ROLLBACK</code>, prepared\nstatements, parameterised queries, and extended-query pipelining all\nwork as expected.</p></blockquote>\n<p>These claims were somewhat optimistic. ParticleDB was missing support\nfor a wide array of Postgres features, and others were badly broken.</p>\n<h3 id=\"3-2-1-no-aggregates-within-case-20\">3.2.1 No Aggregates Within <code>CASE</code> (#20)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.4, aggregate expressions like\n<code>COUNT(*)</code> were not supported within <code>CASE</code>\nexpressions.</p>\n<pre><code>SELECT\n  CASE\n    WHEN COUNT(*) = 2\n      THEN &#39;CASE_COUNT_STAR_OK&#39;\n      ELSE &#39;CASE_COUNT_STAR_BAD&#39;\n  END\nFROM t;</code></pre>\n<p>This threw <code>Error: cannot evaluate wildcard expression</code>.\nParticleDB gained support for this feature (<a href=\"https://github.com/particledbai/jepsen/issues/20\" rel=\"nofollow ugc noopener\">#20</a>) by\nversion 1.1.18.6.</p>\n<h3 id=\"3-2-2-unsupported-exponentiation-28\">3.2.2 Unsupported Exponentiation (#28)</h3>\n<p>ParticleDB 1.1.4 through 1.1.18.8 did not support the Postgres\nexponentiation operator:</p>\n<pre><code>root=&gt; SELECT 2 ^ 3;\nERROR:  unsupported binary operator: PGExp</code></pre>\n<p>Version 1.1.18.9 added support for the <code>^</code> operator (<a href=\"https://github.com/particledbai/jepsen/issues/28\" rel=\"nofollow ugc noopener\">#28</a>).</p>\n<h3 id=\"3-2-3-malformed-responses-for-prepared-statements-2\">3.2.3 Malformed Responses for Prepared Statements (#2)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.9, any attempt to execute a\nprepared statement using the Postgres JDBC driver caused an exception\nlike\n<code>Execution error (NoSuchElementException) at java.util.ArrayDeque/removeFirst</code>.\nSince the JDBC driver automatically switches to prepared statements\nafter the first few calls to <code>execute()</code>, this rendered the\ndatabase essentially unusable. As a workaround, we disabled the client’s\nuse of prepared statements.</p>\n<p>Prepared statements are executed using Postgres’ <a href=\"https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY\" rel=\"nofollow ugc noopener\">extended\nquery protocol</a>, which takes place in several (potentially reusable)\nphases: Parse, Bind, Execute, and so on. Orkes believes that ParticleDB\nlikely failed to return a response for one of those phases, or returned\nan extra message, causing the client’s queue to underflow. By version\n1.1.18.11.rc-1, this issue (<a href=\"https://github.com/particledbai/jepsen/issues/2\" rel=\"nofollow ugc noopener\">#2</a>) appeared\nfixed.</p>\n<h3 id=\"3-2-4-most-postgres-queries-fail-83\">3.2.4 Most Postgres Queries Fail (#83)</h3>\n<p>In ParticleDB 1.1.18.11.rc-7 through 1.1.18.11.rc-19, many queries\nusing the Postgres interface threw\n<code>Bad value for type long: some_column</code>. This rendered the\ndatabase largely unusable.</p>\n<pre><code>CREATE TABLE ints (a int);\nSELECT * FROM ints;\n  -&gt; Bad value for type long : a</code></pre>\n<p>Orkes believes this issue (<a href=\"https://github.com/particledbai/jepsen/issues/83\" rel=\"nofollow ugc noopener\">#83</a>) was\nfixed in 1.1.19-rc.12.</p>\n<h3 id=\"3-2-5-tuples-without-fields-111\">3.2.5 Tuples Without Fields (#111)</h3>\n<p>In 1.1.18.11.rc-18 and rc-19, queries with the Postgres interface\noccasionally threw errors like\n<code>Received resultset tuples, but no field structure for them</code>.<a href=\"#fn5\">5</a> For example, take <a href=\"https://github.com/user-attachments/files/30419491/20260727T025222.565-0500.zip\" rel=\"nofollow ugc noopener\">this\ntest</a> of version 1.1.18.11.rc-19, with no faults, a new\n<code>strict-serializable</code> transaction mode, and WAL mode\n<code>sync</code>. It executed a transaction like:</p>\n<pre><code>BEGIN;\nSELECT * FROM t_38;\nINSERT INTO u_38 (d, a, c, b) VALUES\n  (null, null, false, &quot;oudt1THxi&quot;);\nINSERT INTO t_38 (d, b) VALUES\n  (false, &quot;BCczDbueXLz9zXnpgbdCeFbC5y6XSEys3&quot;);\nCOMMIT;</code></pre>\n<p>This complained of resultset tuples with no field structure,\nsuggesting an error in ParticleDB’s Postgres protocol state machine.\nWhile Jepsen has not tested this version, Orkes reports this issue (<a href=\"https://github.com/particledbai/jepsen/issues/111\" rel=\"nofollow ugc noopener\">#111</a>) was\nfixed in version 1.1.19-rc.13.</p>\n<h3 id=\"3-2-6-broken-postgres-authentication-1\">3.2.6 Broken Postgres Authentication (#1)</h3>\n<p>The Postgres wire protocol usually authenticates clients using a\nusername and password. ParticleDB had a <code>--pg-password</code>\noption, but 1.1.4 through 1.1.18.8 lacked any option to set the\nusername. <a href=\"https://web.archive.org/web/20260519195820/https://particledb.ai/connectivity/pg-wire/\" rel=\"nofollow ugc noopener\">Some\nof the documentation</a> indicated the default username was\n<code>admin</code>, while <a href=\"https://web.archive.org/web/20260518193025/https://particledb.ai/getting-started/quickstart/\" rel=\"nofollow ugc noopener\">other</a>\n<a href=\"https://web.archive.org/web/20260518203023/https://particledb.ai/sdks/java/\" rel=\"nofollow ugc noopener\">pages</a>\nsaid it was <code>particledb</code>. Neither appeared to work; the only\noption was to use <code>--no-auth</code> to disable authentication\naltogether. Version 1.1.18.11.rc-1 introduced a\n<code>--pg-username</code> option (<a href=\"https://github.com/particledbai/jepsen/issues/1\" rel=\"nofollow ugc noopener\">#1</a>).</p>\n<h3 id=\"3-2-7-java-client-could-not-execute-transactions-3\">3.2.7 Java Client Could Not Execute Transactions (#3)</h3>\n<p>Version 1.0.0 of the official ParticleDB Java client refused every\nattempt to execute a transaction. Any call to begin a JDBC transaction,\nor to execute a <code>BEGIN</code> statement, would result in an error\nmessage:\n<code>PDB wire error 1064: SQL plan error: transaction/session statements do not produce a query plan</code>.</p>\n<p>ParticleDB 1.1.4 could execute transactions via its Postgres\ninterface, but using the official Java SDK was essentially impossible.\nThis issue (<a href=\"https://github.com/particledbai/jepsen/issues/3\" rel=\"nofollow ugc noopener\">#3</a>) seems to\nhave been resolved by ParticleDB 1.1.15.</p>\n<h2 id=\"3-3-predicates\">3.3 Predicates</h2>\n<p>ParticleDB struggled to interpret predicates. <code>WHERE</code>\nclauses failed to match rows they should have, or matched unexpected or\nnonexistent rows. These issues occurred with a single server, single\nclient, and no faults.</p>\n<h3 id=\"3-3-1-select-ignores-where-clause-40\">3.3.1 <code>SELECT</code> Ignores <code>WHERE</code> Clause (#40)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.9, <code>SELECT</code> queries\nwithout a table accepted a <code>WHERE</code> clause, but ignored\nit.</p>\n<pre><code>root=&gt; SELECT 5 WHERE FALSE;\n 5\n---\n 5\n(1 row)</code></pre>\n<p>This should have returned zero rows. Version 1.1.18.11.rc-1 seems to\nhave resolved this issue (<a href=\"https://github.com/particledbai/jepsen/issues/40\" rel=\"nofollow ugc noopener\">#40</a>).</p>\n<h3 id=\"3-3-2-where-1-a-2-47\">3.3.2 <code>WHERE 1 &lt; (a + 2)</code> (#47)</h3>\n<p>Version 1.1.1.18.9 introduced a new problem with predicates: inside a\ntransaction, a <code>WHERE</code> clause which performed sufficiently\ncomplex floating-point operations would silently fail to match any rows.\nFor example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a real);\nroot=&gt; INSERT INTO t (a) VALUES (3);\nroot=&gt; BEGIN;\nroot=*&gt; SELECT * FROM t WHERE 1 &lt; (a + 2);\n a\n---\n(0 rows)</code></pre>\n<p>The same problem applied to <code>SELECT</code>, <code>UPDATE</code>,\nand <code>DELETE</code>: predicates like\n<code>WHERE (a * 2) IS NOT NULL</code> would silently fail to match.\nHowever, simpler predicates like <code>WHERE a IS NOT NULL</code> would\nmatch correctly. The effect disappeared if the column was an integer.\nMoreover, some predicates which would throw errors outside a transaction\nwould silently match zero rows within a transaction, suggesting these\npredicates were not being evaluated at all:</p>\n<pre><code>root=&gt; SELECT * FROM t WHERE 1 &lt; (5.0 % 3);\nERROR:  internal error\nroot=&gt; BEGIN;\nroot=*&gt; SELECT * FROM t WHERE 1 &lt; (5.0 % 3);\n a\n---\n(0 rows)</code></pre>\n<p>This issue (<a href=\"https://github.com/particledbai/jepsen/issues/47\" rel=\"nofollow ugc noopener\">#47</a>)\nappeared resolved by 1.1.18.11.rc-1.</p>\n<h3 id=\"3-3-3-where-4-5-18\">3.3.3 <code>WHERE 4 = 5</code> (#18)</h3>\n<p>ParticleDB 1.1.4 through 1.1.17 aggressively interpreted\n<code>WHERE something = x</code> as a primary-key lookup of\n<code>x</code>, regardless of the value of <code>something</code>.</p>\n<pre><code>root=&gt; CREATE TABLE t\n       (a int primary key, b int);\nroot=&gt; INSERT INTO t VALUES (4, 5), (5, 6);\nroot=&gt; SELECT * FROM t WHERE b = 4;\n a | b\n---+---\n 4 | 5</code></pre>\n<p>ParticleDB would ignore a row with <code>b = 5</code> in favor of\n<code>a = 5</code>:</p>\n<pre><code>root=&gt; SELECT * FROM t WHERE b = 5;\n a | b\n---+---\n 5 | 6\n(1 row)</code></pre>\n<p>You could write predicates over non-existent columns:</p>\n<pre><code>root=&gt; SELECT * FROM t\n       WHERE definitely_not_a_column = 4;\n a | b\n---+----\n 4 | 5\n(1 row)</code></pre>\n<p>Or ask if <code>4 = 5</code>:<a href=\"#fn6\">6</a></p>\n<pre><code>root=&gt; SELECT * FROM t WHERE 4 = 5;\n a | b\n---+----\n 5 | 6\n(1 row)</code></pre>\n<p>By version 1.1.18.4, ParticleDB had learned that not every equality\ncomparison involved a primary key (<a href=\"https://github.com/particledbai/jepsen/issues/18\" rel=\"nofollow ugc noopener\">#18</a>).</p>\n<h3 id=\"3-3-4-select-in-a-transaction-returns-nonsense-35\">3.3.4 <code>SELECT</code> in a Transaction Returns Nonsense (#35)</h3>\n<p>The behavior of <code>SELECT</code> within a transaction in\nParticleDB versions 1.1.4 through 1.1.18.8 was thoroughly confusing.\nConsider a single client connected to a fresh single-node instance of\nParticleDB 1.1.18.8. It creates a simple table of integers with two\nrows, <code>1</code> and <code>2</code>, and begins a transaction:</p>\n<pre><code>root=&gt; CREATE TABLE t (a INT UNIQUE);\nroot=&gt; INSERT INTO t VALUES (1), (2);\nroot=&gt; BEGIN;\nroot=*&gt; SELECT * FROM t;\n a\n---\n 1\n 2\n(2 rows)</code></pre>\n<p>So far, so good. But now consider:</p>\n<pre><code>root=*&gt; SELECT * FROM t WHERE a = 1;\n a\n---\n 2\n(1 row)\n\nroot=*&gt; SELECT * FROM t WHERE a = 3;\n a\n---\n 2\n(1 row)\n\nroot=*&gt; SELECT * FROM t WHERE a IN (1, 2);\n a\n---\n 2\n(1 row)\n\nroot=*&gt; SELECT * FROM t WHERE FALSE;\n a\n---\n 2\n(1 row)</code></pre>\n<p>One might conclude that every <code>SELECT</code> with a predicate\nreturned a single row from the requested table—perhaps the most recently\ninserted row. But this is not true, because many predicates evaluating\nto <code>TRUE</code> returned nothing.</p>\n<pre><code>root=*&gt; SELECT * FROM t WHERE TRUE;\n a\n---\n(0 rows)\n\nroot=*&gt; SELECT * FROM t WHERE 3 = 3;\n a\n---\n(0 rows)</code></pre>\n<p>This behavior (<a href=\"https://github.com/particledbai/jepsen/issues/35\" rel=\"nofollow ugc noopener\">#35</a>) seemed\nimproved by ParticleDB 1.1.18.9.</p>\n<h2 id=\"3-4-constraints\">3.4 Constraints</h2>\n<p>ParticleDB often violated constraints like <code>NOT NULL</code> or\n<code>UNIQUE</code>, even with a single server, single client, and no\nfaults.</p>\n<h3 id=\"3-4-1-not-null-constraints-allow-null-8\">3.4.1 Not-Null Constraints Allow Null (#8)</h3>\n<p>While ParticleDB allowed users to add <code>NOT NULL</code>\nconstraints to columns, versions 1.1.4 through 1.1.17 ignored them,\nresulting in bizarre states.</p>\n<pre><code>root=&gt; CREATE TABLE t (a BOOLEAN NOT NULL);\nroot=&gt; INSERT INTO t (a) VALUES (NULL);\nINSERT 0 1</code></pre>\n<p>While the resulting row existed in some sense, it was partly\ninvisible.</p>\n<pre><code>root=&gt; SELECT * FROM t;\n a\n---\n(0 rows)\n\nroot=&gt; SELECT COUNT(*) FROM t;\n COUNT\n-------\n     1\n(1 row)\n\nroot=&gt; SELECT COUNT(*) FROM t WHERE a IS NULL;\n COUNT\n-------\n     0\n(1 row)</code></pre>\n<p>In version 1.1.18.4, ParticleDB started to enforce null constraints\n(<a href=\"https://github.com/particledbai/jepsen/issues/8\" rel=\"nofollow ugc noopener\">#8</a>).</p>\n<h3 id=\"3-4-2-unique-constraints-allow-duplicates-9\">3.4.2 Unique Constraints Allow Duplicates (#9)</h3>\n<p>Similarly, while ParticleDB allowed users to add unique constraints\nto columns, versions 1.1.4 through 1.1.18.4 simply ignored them.</p>\n<pre><code>root=&gt; CREATE TABLE t (a boolean UNIQUE);\nroot=&gt; INSERT INTO t (a) VALUES (TRUE);\nINSERT 0 1\nroot=&gt; INSERT INTO t (a) VALUES (TRUE);\nINSERT 0 1\nroot=&gt; SELECT * FROM t;\n a\n---\n t\n t\n(2 rows)</code></pre>\n<p>In version 1.1.18.6, ParticleDB partially enforced unique constraints\n(<a href=\"https://github.com/particledbai/jepsen/issues/9\" rel=\"nofollow ugc noopener\">#9</a>).</p>\n<h3 id=\"3-4-3-duplicate-primary-keys-29\">3.4.3 Duplicate Primary Keys (#29)</h3>\n<p>Although 1.1.18.6 made <code>UNIQUE</code> constraints somewhat more\nrobust, 1.1.4 through 1.1.18.8 allowed an <code>UPDATE</code> to produce\nmultiple rows with the same primary key.</p>\n<pre><code>root=&gt; CREATE TABLE t (id text PRIMARY KEY,\n                       desc text);\nroot=&gt; INSERT INTO t VALUES (&#39;a&#39;, &#39;a&#39;);\nroot=&gt; INSERT INTO t VALUES (&#39;b&#39;, &#39;b&#39;);\nroot=&gt; UPDATE t SET id = &#39;c&#39;;\nUPDATE 2</code></pre>\n<p>Selecting a row by primary key returned only one of the two rows, but\nother predicates could return both:</p>\n<pre><code>root=&gt; SELECT * FROM t WHERE id = &#39;c&#39;;\n id | desc\n----+------\n c  | b\n(1 row)\n\nroot=&gt; SELECT * FROM t WHERE id LIKE &#39;c&#39;;\n id | desc\n----+------\n c  | a\n c  | b</code></pre>\n<p>This issue (<a href=\"https://github.com/particledbai/jepsen/issues/29\" rel=\"nofollow ugc noopener\">#29</a>) appears\nto have been fixed by 1.1.18.9.</p>\n<h3 id=\"3-4-4-null-primary-keys-57\">3.4.4 Null Primary Keys (#57)</h3>\n<p>Per the SQL standard, primary keys may not be null. This makes\nintuitive sense, since <code>NULL</code> represents an unknown value.\nParticleDB would not allow you to insert null values for primary keys.\nHowever, versions 1.1.4 through 1.1.18.11.rc-1 were perfectly happy to\n<em>update</em> the primary key of one (or more!) rows to\n<code>NULL</code>.</p>\n<pre><code>root=&gt; CREATE TABLE t (a int PRIMARY KEY);\nroot=&gt; INSERT INTO t (a) VALUES (3), (4);\nroot=&gt; UPDATE t SET a = NULL;\nroot=&gt; SELECT * FROM t;\n a\n---\n\n(2 rows)</code></pre>\n<p>ParticleDB 1.1.18.11.rc-3 rejected attempts to set primary keys to\n<code>NULL</code> (<a href=\"https://github.com/particledbai/jepsen/issues/57\" rel=\"nofollow ugc noopener\">#57</a>).</p>\n<h3 id=\"3-4-5-primary-key-conflicts-with-deleted-rows-82\">3.4.5 Primary Key Conflicts with Deleted Rows (#82)</h3>\n<p>In ParticleDB 1.1.18.4 through 1.1.18.11.rc-11, a transaction which\ndeleted a row, then inserted a new row with the same primary key, would\nabort due to a primary key constraint violation. For example:</p>\n<pre><code>CREATE TABLE t (a int PRIMARY KEY);\nINSERT INTO t (a) VALUES (7);\nBEGIN;\nDELETE FROM t;\nINSERT INTO t (a) VALUES (7);\nCOMMIT;</code></pre>\n<p>The second <code>INSERT</code> would throw\n<code>PDB wire error 1064: SQL execution error: duplicate key value violates unique constraint (key=7)</code>,\nbut only when using the ParticleDB interface—the Postgres interface was\nunaffected. Orkes partially addressed this issue (<a href=\"https://github.com/particledbai/jepsen/issues/82\" rel=\"nofollow ugc noopener\">#82</a>) in\nversion 1.1.18.11.rc-9, and fixed another case of it in\n1.1.18.11.rc-18.</p>\n<h3 id=\"3-4-6-lack-of-primary-key-conflicts-with-extant-rows-90\">3.4.6 Lack of Primary Key Conflicts with Extant Rows (#90)</h3>\n<p>In ParticleDB 1.1.18.11.rc-9, a transaction executing concurrently\nwith other transactions could read a row with a primary key, then\nsuccessfully insert a row with that same primary key. For example, take\n<a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260618T140338-1.1.18.11.rc-9-internal.zip\" rel=\"nofollow ugc noopener\">this\n<code>internal-sim</code> test</a>, which created a table with two\ninteger columns:</p>\n<pre><code>CREATE TABLE u_14 (b int, a int PRIMARY KEY);</code></pre>\n<p>A transaction ran <code>SELECT * FROM u_14</code> and observed\nseveral rows, including <code>{:b 1020026655, :a -1}</code>. It then ran\n<code>INSERT INTO u_14 (b, a) VALUES (-27, -1)</code>, which should have\nconflicted with the extant primary key, but succeeded instead. It went\non to <code>SELECT * FROM u_14</code> again, and observed the newly\ninserted row, but not the previously existent one. This is a violation\nof internal consistency.</p>\n<p>As of version 1.1.18.11.rc-19, we no longer observed these internal\nconsistency violations (<a href=\"https://github.com/particledbai/jepsen/issues/90\" rel=\"nofollow ugc noopener\">#90</a>).</p>\n<h2 id=\"3-5-joins\">3.5 Joins</h2>\n<p>ParticleDB struggled to join tables together. The following issues\noccurred with a single server, single client, and no faults.</p>\n<h3 id=\"3-5-1-can-t-join-integers-to-floats-58\">3.5.1 Can’t Join Integers to Floats (#58)</h3>\n<p>Even after agreeing that <code>2.0 = 2</code>, ParticleDB 1.1.4\nthrough 1.1.18.9 refused to join two tables on an <code>int</code> and a\n<code>real</code> column:</p>\n<pre><code>root=&gt; CREATE TABLE t (a real);\nroot=&gt; CREATE TABLE u (a int);\nroot=&gt; INSERT INTO t (a) VALUES (2.0);\nroot=&gt; INSERT INTO u (a) VALUES (2);\nroot=&gt; SELECT t.a FROM t JOIN u ON u.a = t.a;\nERROR:  join key must be integer type</code></pre>\n<p>Databases like Postgres would return a single row here, since\n<code>2.0 = 2</code>. ParticleDB gained the ability to join floats to\nintegers (<a href=\"https://github.com/particledbai/jepsen/issues/58\" rel=\"nofollow ugc noopener\">#58</a>) by\n1.1.18.11.rc-1.</p>\n<h3 id=\"3-5-2-incorrect-column-names-54\">3.5.2 Incorrect Column Names (#54)</h3>\n<p>Because two tables can have a column with the same name, SQL\ndatabases generally return both table and column names, so you can tell\nwhich column belonged to which table. ParticleDB, with both the\nParticleDB and Postgres interfaces, neglected to return table names.\nInstead, versions 1.1.4 through 1.1.18.11.rc-1 renamed column\n<code>a</code> in joined tables to <code>right.a</code>,\n<code>right.right.a</code>, and so on. Consider the following simple\njoin across two tables <code>t</code> and <code>u</code>, both with an\ninteger column <code>a</code>:</p>\n<pre><code>SELECT * FROM t FULL OUTER JOIN u\n  ON t.a = u.a;</code></pre>\n<p>In Postgres this produced rows like:</p>\n<pre><code>[{:t/a 1, :u/a 1.0}\n {:t/a 2, :u/a 2.0}]</code></pre>\n<p>Here the expression <code>:t/a</code> denotes column <code>a</code>\nof table <code>t</code>. In ParticleDB, this same query returned:</p>\n<pre><code>[{:a 1, :right.a 1.0}\n {:a 2, :right.a 2.0}]</code></pre>\n<p>The table names are completely missing, and ParticleDB has altered\nthe name of column <code>a</code> to <code>right.a</code>. Programs\nwhich expected typical SQL behavior would throw an error as they\nattempted to read columns which were not present, or could silently read\nnull values.</p>\n<p>ParticleDB 1.1.18.11.rc-3 no longer munged columns, but the database\nstill omitted table names. The resulting name collisions destroyed\ninformation.</p>\n<pre><code>[{:a 1}\n {:a 2}]</code></pre>\n<p>While Jepsen has not had a chance to test this version, Orkes claims\nthis issue (<a href=\"https://github.com/particledbai/jepsen/issues/54\" rel=\"nofollow ugc noopener\">#54</a>) was\nresolved by version 1.1.19-rc.13.</p>\n<h3 id=\"3-5-3-coercing-to-strings-76\">3.5.3 Coercing to Strings (#76)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.11.rc-19, joins would coerce some\nvalues to strings in ParticleDB’s Postgres interface. For example,\nconsider the following join:</p>\n<pre><code>CREATE TABLE t (a int);\nCREATE TABLE u (a real);\nINSERT INTO u (a) VALUES (3.2);\nINSERT INTO t (a) VALUES (5);\nSELECT u.a FROM t CROSS JOIN u;</code></pre>\n<p>With the ParticleDB interface, this produced the number\n<code>3.2</code>. With the Postgres interface, it produced the string\n<code>&quot;3.2&quot;</code>. This issue (<a href=\"https://github.com/particledbai/jepsen/issues/76\" rel=\"nofollow ugc noopener\">#76</a>) seemed\nfixed by 1.1.18.11.rc-21.</p>\n<h3 id=\"3-5-4-missing-rows-59\">3.5.4 Missing Rows (#59)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.11.rc-1, outer joins of a\nnon-empty table to an empty one often returned no rows, even though the\nquery should have produced all rows in the non-empty table. For\nexample:</p>\n<pre><code>root=&gt; CREATE TABLE t (a int);\nroot=&gt; CREATE TABLE u (a int);\nroot=&gt; INSERT INTO t (a) VALUES (1), (2);\nroot=&gt; SELECT t.a AS t, u.a AS u\n       FROM t LEFT JOIN u ON u.a = t.a;\n t | u\n---+---\n(0 rows)</code></pre>\n<p>By version 1.1.18.11.rc-3, ParticleDB correctly returned rows when\njoining to empty tables (<a href=\"https://github.com/particledbai/jepsen/issues/59\" rel=\"nofollow ugc noopener\">#59</a>).</p>\n<h3 id=\"3-5-5-column-confusion-77\">3.5.5 Column Confusion (#77)</h3>\n<p>Many tables in real databases contain columns with identical names,\nlike <code>id</code> or <code>created_at</code>. When asked to join two\ntables like this, ParticleDB 1.1.4 through 1.1.18.11.rc-6 became\nseverely confused, even when all columns were fully qualified. For\nexample:</p>\n<pre><code>root=&gt; CREATE TABLE t (a text);\nroot=&gt; CREATE TABLE u (a text);\nroot=&gt; INSERT INTO t VALUES (&#39;this&#39;), (&#39;that&#39;);\nroot=&gt; INSERT INTO u VALUES\n       (&#39;is not the right column&#39;);\nroot=&gt; SELECT t.a AS ta, u.a AS ua FROM t\n       CROSS JOIN u WHERE (u.a = &#39;this&#39;);\n  ta  |           ua\n------+-------------------------\n this | is not the right column\n(1 row)</code></pre>\n<p>This should not have matched anything—there are no rows in table\n<code>u</code> where <code>u.a = &#39;this&#39;</code>. Instead ParticleDB\nmatched a row in table <code>t</code> which satisfied\n<code>t.a = &#39;this&#39;</code>, then cross-joined it to all rows in\n<code>u</code>. To add insult to injury, 1.1.4 through 1.1.18.9 appeared\nto swap around the output columns, producing the following table:</p>\n<pre><code>           ta            |  ua\n-------------------------+------\n is not the right column | this</code></pre>\n<p>If column <code>t.a</code> was of a different type than\n<code>u.a</code>, ParticleDB would try to coerce the types into\ncompatible forms, producing bizarre error messages or delightfully weird\nresults. Version 1.1.18.11.rc-7 was much less confused when joining\ntables containing the same column name (<a href=\"https://github.com/particledbai/jepsen/issues/77\" rel=\"nofollow ugc noopener\">#77</a>).</p>\n<h2 id=\"3-6-zombies\">3.6 Zombies</h2>\n<p>ParticleDB often wound up with rows that could not be deleted or\nupdated, or which persisted in some (perhaps mangled) form after their\ndeletion. These issues occurred with a single server, single client, and\nno faults.</p>\n<h3 id=\"3-6-1-total-recall-26\">3.6.1 Total Recall (#26)</h3>\n<p>In ParticleDB 1.1.17 through 1.1.18.8, it appeared impossible to\ndelete rows within a transaction. For example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a boolean);\nroot=&gt; INSERT INTO t (a) VALUES (TRUE);\nroot=&gt; BEGIN;\nroot=*&gt; DELETE FROM t;\nDELETE 0\nroot=*&gt; COMMIT;\nroot=&gt; SELECT * FROM t;\n a\n---\n t\n(1 row)</code></pre>\n<p>We were unable to find a predicate that allowed one to delete records\n(<a href=\"https://github.com/particledbai/jepsen/issues/26\" rel=\"nofollow ugc noopener\">#26</a>).\nVersion 1.1.18.9 allowed transactions to mostly<a href=\"#fn7\">7</a>\ndelete rows.</p>\n<h3 id=\"3-6-2-memento-delere-45\">3.6.2 Memento Delere (#45)</h3>\n<p>1.1.18.9 resolved issue #26, but deleted rows still lingered on in\nsome capacity. While a transaction could not directly select deleted\nrows, it could observe them via aggregate functions and various\npredicates.</p>\n<pre><code>root=&gt; CREATE TABLE t (a boolean);\nroot=&gt; INSERT INTO t (a) VALUES (TRUE);\nroot=&gt; BEGIN;\nroot=*&gt; DELETE FROM t;\nDELETE 1\nroot=*&gt; SELECT COUNT(*) FROM t WHERE a = TRUE;\n COUNT\n-------\n     1\n(1 row)\n\nroot=*&gt; SELECT COUNT(*) FROM t WHERE a = FALSE;\n COUNT\n-------\n     0\n(1 row)</code></pre>\n<p>The row must still be present in some sense, because these aggregate\nqueries could observe its value via predicates. However, trying to read\nthe row directly returned nothing:</p>\n<pre><code>root=*&gt; SELECT a FROM t WHERE a = TRUE;\n a\n---\n(0 rows)</code></pre>\n<p>Version 1.1.18.11.rc-1 no longer allowed shades of deleted records to\npersist in aggregate queries (<a href=\"https://github.com/particledbai/jepsen/issues/45\" rel=\"nofollow ugc noopener\">#45</a>).</p>\n<h3 id=\"3-6-3-night-of-the-living-integers-23\">3.6.3 Night of the Living Integers (#23)</h3>\n<p>In versions 1.1.4 through 1.1.17, updating an integer column could\nmake rows impossible to delete. For example:</p>\n<pre><code>root=&gt; CREATE TABLE t (i int);\nroot=&gt; INSERT INTO t VALUES (0);\nroot=&gt; UPDATE t SET i = 1.5;\nroot=&gt; DELETE FROM t;\nDELETE 0\nroot=&gt; SELECT * FROM t;\n i\n---\n 1\n(1 row)</code></pre>\n<p>While this row appears to contain the integer <code>1</code>, that\ninteger is <em>special</em>, since the <code>UPDATE</code> statement\nprovided the value <code>1.5</code>. This row cannot be deleted. It does\nappear in <code>SELECT</code>, and is apparently equal to\n<code>1</code>, rather than <code>1.5</code>. Oddly, updating a row by\nprimary key with the same <code>1.5</code> value did not create zombies;\nthe problem (<a href=\"https://github.com/particledbai/jepsen/issues/23\" rel=\"nofollow ugc noopener\">#23</a>) was\nsomehow linked to predicates. ParticleDB 1.1.18.4 no longer exhibited\nthis behavior.</p>\n<h3 id=\"3-6-4-team-rocket-84\">3.6.4 Team Rocket (#84)</h3>\n<p>Tables with a <code>double precision</code> column suffered from a\nsimilar problem: they could be updated with integer values, but doing so\nwould produce zombie rows that could neither be updated nor deleted\nagain.</p>\n<pre><code>ROOT=&gt; CREATE TABLE t (a double precision);\nROOT=&gt; INSERT INTO t (a) VALUES (1.0);\nROOT=&gt; UPDATE t SET a = 2;\nROOT=&gt; DELETE FROM t;\nDELETE 0\nroot=&gt; SELECT * FROM t;\n a\n---\n 2\n(1 row)</code></pre>\n<p>If the double precision column was a primary key, these updates would\nproduce duplicate copies of the updated row. This issue (<a href=\"https://github.com/particledbai/jepsen/issues/84\" rel=\"nofollow ugc noopener\">#84</a>)\naffected ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-8, and was fixed\nin 1.1.18.11.rc-9.</p>\n<h3 id=\"3-6-5-a-ghost-is-technically-not-a-number-78\">3.6.5 A Ghost is Technically Not a Number (#78)</h3>\n<p>In ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-6, <code>NaN</code>\nvalues stored in <code>real</code> or <code>double precision</code>\ncolumns could neither be updated nor deleted.</p>\n<pre><code>DROP TABLE IF EXISTS t;\nCREATE TABLE t (a real);\nINSERT INTO t (a) VALUES (&#39;NaN&#39;);\nUPDATE t SET a = 5.0;\n  -&gt; #:next.jdbc{:update-count 0}\nDELETE FROM t;\n  -&gt; #:next.jdbc{:update-count 0}\nSELECT * FROM t;\n  -&gt; [{:a ##NaN}]</code></pre>\n<p>This issue (<a href=\"https://github.com/particledbai/jepsen/issues/78\" rel=\"nofollow ugc noopener\">#78</a>) did not\noccur with the <code>psql</code> shell, but did happen with both\nParticleDB and Postgres JDBC clients. Version 1.1.18.11.rc-7 seems to\nhave fixed the problem.</p>\n<h3 id=\"3-6-6-beached-strings-74\">3.6.6 Beached Strings (#74)</h3>\n<p>In ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-6, fixed-width\nstrings were unmodifiable unless created with exactly the right number\nof characters. While ParticleDB would correctly pad the string\n<code>&#39;hi&#39;</code> to <code>&#39;hi &#39;</code>, the resulting row could neither\nbe updated nor deleted.</p>\n<pre><code>root=&gt; CREATE TABLE t (a character(3));\nroot=&gt; INSERT INTO t (a) VALUES (&#39;hi&#39;);\nroot=&gt; DELETE FROM t;\nDELETE 0\nroot=&gt; UPDATE t SET a = &#39;sup&#39;;\nUPDATE 0\nroot=&gt; SELECT * FROM t;\n  a\n-----\n hi\n(1 row)</code></pre>\n<p>If a fixed-width column was used as a primary key, it could be\nupdated, but only once. Both old and new copies would appear as distinct\nrows.</p>\n<pre><code>CREATE TABLE t\n       (a character(3) PRIMARY KEY);\nINSERT INTO t (a) VALUES (&#39;um&#39;);\nUPDATE t SET a = &#39;no&#39;;\nSELECT a FROM t;\n-&gt; [{:a &quot;um &quot;} {:a &quot;no &quot;}]</code></pre>\n<p>Any further attempt to modify those rows would produce a duplicate\nkey error, despite all the values involved being distinct. Fixed-width\nstrings were also incomparable using the <code>=</code> operator.\nVersion 1.1.18.11.rc-7 resolved these issues (<a href=\"https://github.com/particledbai/jepsen/issues/74\" rel=\"nofollow ugc noopener\">#74</a>).</p>\n<h3 id=\"3-6-7-pet-sematary-46\">3.6.7 Pet Sematary (#46)</h3>\n<p>In SQL, the <code>DROP TABLE</code> statement destroys a table\ncompletely. In ParticleDB 1.1.18.9 through 1.1.18.11.rc-1, it left a\nshadow of the table behind, whose rows would reappear in later tables\nwith the same name and more-or-less compatible schemata. To demonstrate\nthis behavior, let us create a table with a single row:</p>\n<pre><code>root=&gt; CREATE TABLE t (a text, b real);\nroot=&gt; BEGIN;\nroot=*&gt; INSERT INTO t VALUES (&#39;boo&#39;, 18.83);\nroot=*&gt; COMMIT;</code></pre>\n<p>We then drop the table, and demonstrate that it no longer exists:</p>\n<pre><code>root=&gt; DROP TABLE t;\nroot=&gt; SELECT * FROM t;\nERROR:  table not found</code></pre>\n<p>If we then create a new table with the same name and a somewhat\ncompatible schema, rows from the deleted table reappear in the new one.\nHowever, the resurrected rows are not exactly their former selves. The\nfloating-point value <code>18.83</code> has become the integer\n<code>19</code>.</p>\n<pre><code>root=&gt; CREATE TABLE t (a text, b int);\nroot=&gt; BEGIN;\nroot=*&gt; SELECT * FROM t;\n  a  | b\n-----+----\n boo | 19\n(1 row)</code></pre>\n<p>This behavior (<a href=\"https://github.com/particledbai/jepsen/issues/46\" rel=\"nofollow ugc noopener\">#46</a>) seemed\nlinked to transactions, and no longer appeared in 1.1.18.11.rc-3.</p>\n<h2 id=\"3-7-state-corruption\">3.7 State Corruption</h2>\n<p>ParticleDB had many ways to create values which could not be read,\nalter values unexpectedly, create rows with incorrect or missing\ncolumns, or otherwise corrupt logical state. The issues in this section\ninvolved only a single server, single client, and no faults.</p>\n<h3 id=\"3-7-1-unreadable-infinities-15\">3.7.1 Unreadable Infinities (#15)</h3>\n<p>In versions 1.1.4 through 1.1.17, ParticleDB’s Postgres interface did\nnot correctly serialize floating-point values like\n<code>Infinity</code>. One could write infinite values, but trying to\nread them would cause the client to throw an error.</p>\n<pre><code>CREATE TABLE t (a real);\nINSERT INTO t (a) VALUES (&#39;Infinity&#39;);\nSELECT * FROM t\n   =&gt; PSQLException: &quot;Bad value for type\n      double : inf&quot;</code></pre>\n<p>With Postgres, this correctly returned the JVM’s\n<code>Double/INFINITY</code> value. <code>NaN</code> was serialized\ncorrectly. At the <code>psql</code> shell, stored values came back as\n<code>inf</code>, rather than <code>Infinity</code>. In version\n1.1.18.6, ParticleDB serialized infinite values correctly (<a href=\"https://github.com/particledbai/jepsen/issues/15\" rel=\"nofollow ugc noopener\">#15</a>).</p>\n<h3 id=\"3-7-2-nan-sometimes-converted-to-0-0-21\">3.7.2 <code>NaN</code> Sometimes Converted to 0.0 (#21)</h3>\n<p>ParticleDB 1.1.4 through 1.1.18.4 would sometimes (but not always!)\nconvert the floating-point value <code>NaN</code> to <code>0.0</code>.\nFor example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a real);\nroot=&gt; INSERT INTO t (a) VALUES (2.5);\nroot=&gt; INSERT INTO t (a) VALUES (&#39;NaN&#39;);\nroot=&gt; SELECT * FROM t;\n  a\n-----\n 2.5\n   0\n(2 rows)</code></pre>\n<p>Delightfully, whether this <code>NaN</code> corruption occurred\ndepended on the order in which rows were inserted. If the first value\ninserted into the table was <code>NaN</code>, that and later\n<code>NaN</code>s would be preserved. Version 1.1.18.6 seemed to store\n<code>NaN</code> correctly (<a href=\"https://github.com/particledbai/jepsen/issues/21\" rel=\"nofollow ugc noopener\">#21</a>).</p>\n<h3 id=\"3-7-3-assignment-of-incompatible-types-7\">3.7.3 Assignment of Incompatible Types (#7)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.4, you could update a column of\none type to a column of a different, incompatible type—producing values\nlike <code>null</code> or <code>0</code>. For example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a INT, b BOOLEAN);\nroot=&gt; INSERT INTO t VALUES (2, FALSE);\nroot=&gt; SELECT * FROM t WHERE a = b;\nERROR:  arrow compute: Invalid argument error:\nInvalid comparison operation: Int64 == Boolean\nroot=&gt; UPDATE t SET a = b;\nroot=&gt; SELECT * FROM t;\n a | b\n---+---\n 0 | f</code></pre>\n<p>In Postgres, this UPDATE would throw\n<code>ERROR: column &quot;a&quot; is of type text but expression is of type int</code>.\nParticleDB refused to <em>compare</em> this string to an integer (at\nleast in this case), but was perfectly happy to make one equal to the\nother, resulting in <code>NULL</code>. Version 1.1.18.6 seemed to\nresolve this issue (<a href=\"https://github.com/particledbai/jepsen/issues/7\" rel=\"nofollow ugc noopener\">#7</a>).</p>\n<h3 id=\"3-7-4-insert-column-order-table-corruption-16\">3.7.4 Insert Column Order Table Corruption (#16)</h3>\n<p>In SQL, one usually inserts a row into a table using a statement like\n<code>INSERT INTO people (name, age) VALUES (&#39;xanthippe&#39;, 15)</code>.\nParticleDB 1.1.4 through 1.1.17 ignored the <code>INSERT</code>\nstatement’s columns, using whatever column order had been used to create\nthe table originally. This produced rows with titillating properties.\nConsider the following pair of inserts: one with columns in the same\norder as the table’s schema, and a second in a different order:</p>\n<pre><code>root=&gt; CREATE TABLE t\n       (a TEXT NOT NULL PRIMARY KEY,\n        b boolean);\nroot=&gt; INSERT INTO t (a, b) VALUES (&#39;x&#39;, TRUE);\nroot=&gt; INSERT INTO t (b, a) VALUES (FALSE, &#39;y&#39;);\nroot=&gt; SELECT COUNT(*) FROM t;\n COUNT\n-------\n     2\n(1 row)</code></pre>\n<p>While ParticleDB thinks this table contains two rows, neither can be\nread via <code>SELECT *</code>:</p>\n<pre><code>root=&gt; SELECT * FROM t;\n a | b\n---+---\n(0 rows)</code></pre>\n<p>In general, any query which happened to encounter a row inserted with\na different column order would silently produce zero results. However,\neach row could be read by primary key! Note the different column orders\nin the output:</p>\n<pre><code>root=&gt; SELECT * FROM t WHERE a = &#39;x&#39;;\n a | b\n---+---\n x | t\n(1 row)\n\nroot=&gt; SELECT * FROM t WHERE a = &#39;y&#39;;\n b | a\n---+---\n f | y\n(1 row)</code></pre>\n<p>In the preceding example, the two columns were of (somewhat)\nincompatible types. What happens if the types are compatible?</p>\n<pre><code>root=&gt; CREATE TABLE t (a INT, b REAL);\nroot=&gt; INSERT INTO t (a, b)\n       VALUES (1, 1.23E32);\nroot=&gt; INSERT INTO t (b, a)\n       VALUES (2.34E32, 2);\nroot=&gt; SELECT * FROM t;\n          a          |    b\n---------------------+---------\n                   1 | 1.23e32\n 9223372036854775807 |       2\n(2 rows)</code></pre>\n<p>Now the table is readable, but the data is corrupt. The second row’s\n<code>a</code> value appears under <code>b</code>, and vice-versa. The\nfloating-point value <code>2.34E32</code> has become the integer 263 − 1.</p>\n<p>If we attempt to delete every row from the table, something very\nstrange happens. The <code>DELETE</code> leaves the differently-ordered\nrow in place, and its originally inserted values now appear intact!</p>\n<pre><code>root=&gt; DELETE FROM t;\nDELETE 1\nroot=&gt; SELECT * FROM t;\n a |    b\n---+---------\n 2 | 2.34e32\n(1 row)</code></pre>\n<p>A second <code>DELETE</code> statement—now that the table contains\nonly rows inserted with column order <code>(b, a)</code>—could\nsuccessfully delete the remaining row. These column-order issues (<a href=\"https://github.com/particledbai/jepsen/issues/16\" rel=\"nofollow ugc noopener\">#16</a>) no\nlonger appeared in version 1.1.18.4.</p>\n<h3 id=\"3-7-5-updates-can-insert-corrupt-rows-24-25\">3.7.5 Updates Can Insert Corrupt Rows (#24, #25)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.8, an update which matched no rows\ncould instead create a new, possibly corrupt row in the table, with\nbaffling effects. Consider a table mapping integer keys to integer\nvalues:</p>\n<pre><code>root=&gt; CREATE TABLE t\n       (id int not null primary key,\n        val integer);\nroot=&gt; BEGIN;\nroot=*&gt; UPDATE t SET val = 10 WHERE id = 0;\nUPDATE 1\nroot=*&gt; COMMIT;</code></pre>\n<p>This table was just created. It contains no rows, so the update\nshould have affected zero, not one row. And yet it now contains a\nrow!</p>\n<pre><code>root=&gt; SELECT COUNT (*) FROM t;\n COUNT\n-------\n     1\n(1 row)</code></pre>\n<p>However, that row is invisible to typical queries:</p>\n<pre><code>root=&gt; SELECT * FROM t;\n id | val\n----+-----\n(0 rows)\n\nroot=&gt; SELECT * FROM t WHERE id = 0;\n id | val\n----+-----\n(0 rows)\n\nroot=&gt; SELECT * FROM t WHERE val = 10;\n id | val\n----+-----\n(0 rows)</code></pre>\n<p>Although one cannot directly see the row by primary key, it can\nconflict with some inserts.</p>\n<pre><code>root=&gt; INSERT INTO t (id, val) VALUES (0, 100);\nINSERT 0 1\nroot=&gt; INSERT INTO t (id, val) VALUES (10, 110);\nERROR:  duplicate key value violates unique\n        constraint (key=10)</code></pre>\n<p>“Hang on,” an astute reader might exclaim. “The row we supposedly\nupdated had <code>id = 0</code> and set <code>val = 10</code>. Why did\nit not conflict with an insert of <code>id = 0</code>?”</p>\n<p>Quite right, dear reader! And why did it instead conflict with\n<code>id = 10</code>?</p>\n<p>These are excellent questions—which we do not know the answer to.\nPressing on:</p>\n<pre><code>root=&gt; CREATE TABLE t\n       (id int not null primary key,\n        val integer);\nroot=&gt; BEGIN;\nroot=*&gt; UPDATE t SET val = 50 WHERE id = 5;\nUPDATE 1\nroot=*&gt; SELECT * FROM t WHERE id = 5;\n val\n-----\n  50\n(1 row)\n\nroot=*&gt; COMMIT;</code></pre>\n<p>Within the transaction, the row produced by this update is readable\nby its primary key <code>id = 5</code>. Although this table has two\ncolumns, the row is printed as if it had only one:\n<code>val = 50</code>.</p>\n<p>As in #16, these corrupt rows caused queries that encountered them to\nsilently return zero rows, even if there were other valid rows that\nwould have matched.</p>\n<pre><code>root=&gt; INSERT INTO t VALUES (6, 60);\nroot=&gt; SELECT * FROM t WHERE id IN (6);\n id | val\n----+-----\n  6 |  60\n\n(1 row)\n\nroot=&gt; SELECT * FROM t WHERE id IN (6, 7);\n id | val\n----+-----\n(0 rows)</code></pre>\n<p>Although our corrupt row had <code>id = 5</code> (or perhaps\n<code>50</code>), adding a second value to the <code>IN</code> predicate\nseems to have caused ParticleDB to “notice” the corrupt row. It then\nfailed to return the valid row with <code>id = 6</code>.</p>\n<p>Depending on the table structure and <code>UPDATE</code> statement,\nrows created by updates might be structurally valid, but logically\ncorrupt:</p>\n<pre><code>root=&gt; CREATE TABLE t (a INT UNIQUE);\nroot=&gt; INSERT INTO t VALUES (1);\nroot=&gt; BEGIN;\nroot=*&gt; UPDATE t SET a = 3 WHERE a = 2;\nUPDATE 1\nroot=*&gt; COMMIT;\nroot=&gt; SELECT * FROM t;\n a\n---\n 1\n 3\n(2 rows)</code></pre>\n<p>This bug (<a href=\"https://github.com/particledbai/jepsen/issues/24\" rel=\"nofollow ugc noopener\">#24</a>) also\nallowed updates which would have left a row’s value unchanged to instead\nthrow a duplicate key exception, as they (presumably) conflicted with\nthemselves (<a href=\"https://github.com/particledbai/jepsen/issues/25\" rel=\"nofollow ugc noopener\">#25</a>). Both\nissues appeared resolved as of version 1.1.18.9.</p>\n<h3 id=\"3-7-6-sometimes-strings-that-are-extensive-are-cursed-37\">3.7.6 Sometimes, Strings That Are Extensive… Are Cursed (#37)</h3>\n<p>In versions 1.1.15 through 1.1.17, <code>text</code> strings could be\nunexpectedly transmogrified into eight-bit integers. We encountered this\nin <code>append</code> tests using the official ParticleDB client, like\n<a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260521T190337-strings-as-ints.zip\" rel=\"nofollow ugc noopener\">this\none</a>. The <code>append</code> workload stores a comma-separated list\nof integers in an SQL <code>TEXT</code> column called <code>val</code>,\nand appends values to it using <code>CONCAT</code>. Most reads of\n<code>val</code> sensibly returned strings:</p>\n<pre><code>[&quot;select (val) from txn2 where id = ? &quot; 2946]\n  -&gt; [{:val &quot;1&quot;}]</code></pre>\n<p>However, after approximately 30 seconds, the ParticleDB client began\nreturning eight-bit integers for key <code>2946</code>. Here, the key’s\nvalue has become the JVM <code>Byte</code> <code>1</code>:</p>\n<pre><code>[&quot;select (val) from txn2 where sk = ? &quot; 2946]\n  -&gt; [{:val 1}]</code></pre>\n<p>From this point on, some keys appeared as strings, while others\nreturned integers. A read of key <code>2961</code> returned the string\n<code>&quot;2&quot;</code>, while key <code>2865</code> returned the integer\n<code>1</code>—then moments later, the string <code>&quot;1,2&quot;</code>.\nSometimes the integer values corresponded to the string value produced\nby earlier writes, but not always. For example, a transaction read key\n<code>2980</code> as the integer <code>0</code>—when the only elements\nappended to its string were <code>&quot;1&quot;</code> and <code>&quot;2&quot;</code>.\nShortly thereafter, a single transaction read key <code>2980</code>\ntwice and found a string, then an apparently unrelated integer:</p>\n<pre><code>[&quot;select (val) from txn2 where id = ? &quot; 2980]\n  -&gt; [{:val &quot;2,1&quot;}]\n[&quot;select (val) from txn0 where sk = ? &quot; 2981]\n  -&gt; [{:val &quot;1&quot;}]\n[&quot;select (val) from txn2 where sk = ? &quot; 2980]\n  -&gt; [{:val 3}]</code></pre>\n<p>This issue (<a href=\"https://github.com/particledbai/jepsen/issues/37\" rel=\"nofollow ugc noopener\">#37</a>) seemed\nto disappear by 1.1.18.11.rc-1.</p>\n<h3 id=\"3-7-7-eternal-sunshine-of-the-spotless-table-72\">3.7.7 Eternal Sunshine of the Spotless Table (#72)</h3>\n<p>In ParticleDB 1.1.18.11.rc-1, <code>append</code> tests of the\n<code>table-2pl</code> transaction mode often threw errors like:</p>\n<blockquote><p>PDB wire error 1064: SQL execution error: failed to create\nRecordBatch: Invalid argument error: Column ‘id’ is declared as\nnon-nullable but contains null values</p></blockquote>\n<p>This is odd. The <code>append</code> workload always picks a unique\ninteger ID for every write, and never tries to write a <code>NULL</code>\nID. It also never updates an ID—they are set only once, when the row is\ncreated. Here’s <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T110736-contains-null-values.zip\" rel=\"nofollow ugc noopener\">an\nexample</a> where a single <code>SELECT</code> statement, issued outside\nof a transaction, complained of null primary keys. This suggests that\nthe table state had somehow become corrupt.</p>\n<p>We no longer observed this error (<a href=\"https://github.com/particledbai/jepsen/issues/72\" rel=\"nofollow ugc noopener\">#72</a>) in\n1.1.18.11.rc-7.</p>\n<h2 id=\"3-8-transactions-and-concurrency\">3.8 Transactions and Concurrency</h2>\n<p>We found several problems with ParticleDB’s transactions and\nconcurrency control. All of these occurred in single-node instances\nwithout fault injection.</p>\n<h3 id=\"3-8-1-nested-transactions-destroy-ongoing-ones-44\">3.8.1 Nested Transactions Destroy Ongoing Ones (#44)</h3>\n<p>ParticleDB 1.1.4 through 1.1.18.9 allowed clients to begin nested\ntransactions, instead of warning that a transaction was already ongoing.\nAny writes performed by the enclosing transaction would be lost, both\nwithin the nested transaction and after commit.</p>\n<pre><code>root=&gt; CREATE TABLE t (a int);\nroot=&gt; BEGIN;\nroot=*&gt; INSERT INTO t VALUES (3);\nroot=*&gt; BEGIN;\nroot=*&gt; SELECT * FROM t;\n a\n---\n(0 rows)\nroot=*&gt; COMMIT;</code></pre>\n<p>ParticleDB 1.1.18.11.rc-1 seems to have resolved this issue (<a href=\"https://github.com/particledbai/jepsen/issues/44\" rel=\"nofollow ugc noopener\">#44</a>):\nissuing <code>BEGIN</code> within a transaction warned\n<code>there is already a transaction in progress</code>, and preserved\nthe ongoing transaction’s writes.</p>\n<h3 id=\"3-8-2-aborted-transactions-can-commit-43\">3.8.2 Aborted Transactions Can Commit (#43)</h3>\n<p>Between versions 1.1.17 and 1.1.18.4, ParticleDB handled aborted\ntransactions irregularly. With the Postgres interface, an attempt to\ncommit an aborted transaction would throw an error, as one would\nexpect:</p>\n<pre><code>root=&gt; BEGIN;\nroot=&gt; INSERT INTO t (a) VALUES (3);\nroot=&gt; THIS IS NOT VALID SQL;\nERROR:  sql parser error: Expected: an SQL\nstatement, found: THIS at Line: 1, Column: 1\nroot=&gt; COMMIT;\nROLLBACK</code></pre>\n<p>Nevertheless, the writes performed by the aborted transaction\npersisted:</p>\n<pre><code>SELECT * FROM t;\n  -&gt; [{:a 3}]</code></pre>\n<p>The ParticleDB client had a different, also unexpected behavior:\nexecuting a <code>COMMIT</code> statement on a failed transaction\nreturned <code>[#:next.jdbc{:update-count 0}]</code>, rather than an\nerror. Although the transaction apparently committed, its effects were\ninvisible. Both interfaces, as of version 1.1.18.11.rc-18, returned\nerrors and did not expose the effects of aborted transactions (<a href=\"https://github.com/particledbai/jepsen/issues/43\" rel=\"nofollow ugc noopener\">#43</a>).</p>\n<h3 id=\"3-8-3-aggregates-sometimes-fail-to-observe-a-transaction-s-rows-\">3.8.3 Aggregates Sometimes Fail to Observe a Transaction’s Rows (#48)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.9, transactions which performed\naggregate queries, like <code>SELECT COUNT(*)</code>, could miss rows\nthey should have matched, and return duplicate copies of rows, depending\non the predicate and previous queries in the transaction. For\nexample:</p>\n<pre><code>root=&gt; CREATE TABLE t (a int);\nroot=&gt; INSERT INTO t (a) VALUES (2);\nroot=&gt; BEGIN;\nroot=*&gt; INSERT INTO t (a) VALUES (3);\nroot=*&gt; SELECT COUNT(*) FROM t;\n COUNT \n-------\n     1\n(1 row)</code></pre>\n<p>This table clearly contains two rows: <code>2</code> and\n<code>3</code>. Yet while the second row was visible to\n<code>SELECT *</code>, it was invisible to <code>SELECT COUNT(*)</code>.\nOddly enough, aggregate functions which interacted with the column’s\nvalue, like <code>SELECT COUNT(*) FROM t WHERE a &lt; 5</code>,\nsometimes observed correct state. In other circumstances,\n<code>SELECT</code> queries appeared to create duplicate copies of rows\nwithin the transaction, causing each query to observe more and more\nrows. These behaviors (<a href=\"https://github.com/particledbai/jepsen/issues/48\" rel=\"nofollow ugc noopener\">#48</a>) no\nlonger appeared in version 1.1.18.11.rc-1.</p>\n<h3 id=\"3-8-4-double-trouble-89\">3.8.4 Double Trouble (#89)</h3>\n<p>ParticleDB 1.1.18.11.rc-9 would occasionally return duplicate copies\nof a single row, even though the row had a primary key which ought to\nhave ensured uniqueness. For example, take <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260617T165351-duplicate-rows.zip\" rel=\"nofollow ugc noopener\">this\nappend test</a>, in which a read of key <code>10</code> returned two\nrows, rather than one.</p>\n<pre><code>[{:val &quot;1,2,5,10&quot;}\n {:val &quot;1,2,5,10&quot;}]</code></pre>\n<p>We no longer observed duplicate rows (<a href=\"https://github.com/particledbai/jepsen/issues/89\" rel=\"nofollow ugc noopener\">#89</a>) in\n1.1.18.11.rc-18.</p>\n<h3 id=\"3-8-5-updates-to-zero-values-sometimes-fail-silently-91\">3.8.5 Updates to Zero Values Sometimes Fail Silently (#91)</h3>\n<p>In ParticleDB 1.1.18.4, a transaction could set all rows in a table\nto some value, receive an acknowledgement that every row had been\nupdated, then read the table to find that some rows still had their\nprevious values. This bug seemed to hinge on inserting rows with a\nspecial “zero” value: <code>0</code> for integers, <code>0.0</code> for\nfloats, and <code>&#39;&#39;</code> for various kinds of string columns. For\nexample:</p>\n<pre><code>root=&gt; CREATE TABLE t (a INT);\nroot=&gt; INSERT INTO t (a) VALUES (0);\nroot=&gt; BEGIN;\nroot=*&gt; UPDATE t SET a = 1;\nroot=*&gt; INSERT INTO t (a) VALUES (0);\nroot=*&gt; UPDATE t SET a = 2;\nUPDATE 2\nroot=*&gt; SELECT t.a FROM t;\n a\n---\n 2\n 0\n(2 rows)</code></pre>\n<p>ParticleDB claimed that it updated all two rows in this table, but\nshowed the original value of <code>0</code> for one of them. This issue\n(<a href=\"https://github.com/particledbai/jepsen/issues/91\" rel=\"nofollow ugc noopener\">#91</a>)\nseemed fixed by 1.1.18.11.rc-9.2.</p>\n<h3 id=\"3-8-6-tables-with-a-text-primary-key-are-mostly-invisible-41\">3.8.6 Tables With a <code>TEXT</code> Primary Key Are Mostly Invisible (#41)</h3>\n<p>Version 1.1.18.9 fixed several safety issues, but also introduced new\nones. For example, tables with a <code>text</code> primary key were\nmostly invisible to transactions:</p>\n<pre><code>root=&gt; CREATE TABLE t (a TEXT PRIMARY KEY);\nroot=&gt; INSERT INTO t (a) VALUES (&#39;mood&#39;);\nroot=&gt; BEGIN;\nroot=*&gt; SELECT * FROM t;\n a\n---\n(0 rows)</code></pre>\n<p><code>SELECT</code>, <code>UPDATE</code>, and <code>DELETE</code>\nwere all affected. Rows were invisible to a broad variety of predicates,\nincluding <code>WHERE a LIKE &#39;moo%&#39;</code> and\n<code>WHERE a IS NOT NULL</code>. However, rows were still accessible by\nprimary key:</p>\n<pre><code>root=*&gt; SELECT * FROM t WHERE a = &#39;mood&#39;;\n  a\n------\n mood\n(1 row)</code></pre>\n<p>Version 1.1.18.11.rc-1 no longer exhibited these mostly-invisible\ntables (<a href=\"https://github.com/particledbai/jepsen/issues/41\" rel=\"nofollow ugc noopener\">#41</a>).</p>\n<h3 id=\"3-8-7-a-thousand-tableaus-38-66-68-69\">3.8.7 A Thousand Tableaus (#38, #66, #68, #69)</h3>\n<p>Perhaps owing to the single-threaded issues discussed above,\nParticleDB 1.1.4 through 1.1.18.9 exhibited frequent, profound\ntransactional anomalies in essentially every way we could measure. We\nroutinely observed the loss of committed writes, internal consistency\nviolations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost\nUpdate). In the <code>append</code> workload, successfully appended\nvalues would routinely disappear from later reads of a given key. Our\ntest suite inferred every kind of cyclic anomaly including G0, G1c,\nG-Single, and G2-item, but many of these can be attributed to more basic\nissues, like write loss. These issues occurred with a single server and\nwithout faults, and affected every transaction mode, interface, and WAL\nmode tested.<a href=\"#fn8\">8</a></p>\n<p>Since we have already discussed single-threaded anomalies at length,\nwe omit detailed discussions of these issues. By version 1.1.18.11.rc-1,\nParticleDB’s most glaring examples of G1a (<a href=\"https://github.com/particledbai/jepsen/issues/66\" rel=\"nofollow ugc noopener\">#66</a>), G1b\n(<a href=\"https://github.com/particledbai/jepsen/issues/38\" rel=\"nofollow ugc noopener\">#38</a>),\ninternal consistency anomalies (<a href=\"https://github.com/particledbai/jepsen/issues/68\" rel=\"nofollow ugc noopener\">#68</a>), write\nloss (<a href=\"https://github.com/particledbai/jepsen/issues/69\" rel=\"nofollow ugc noopener\">#69</a>), and so\non were much diminished.</p>\n<h3 id=\"3-8-8-g-single-73\">3.8.8 G-Single (#73)</h3>\n<p>With versions 1.1.18.11.rc-1 through 1.1.18.11.rc-22, we occasionally\nobserved <a href=\"https://jepsen.io/consistency/phenomena/g-single\" rel=\"nofollow ugc noopener\">G-single</a>, a\nviolation of Snapshot Isolation, in single nodes without faults. For\nexample, here are two transactions from <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T113409-g-single.zip\" rel=\"nofollow ugc noopener\">an\n<code>append</code> test</a> using transaction mode <code>occ-v3</code>\nand WAL mode <code>groupsync</code>:</p>\n<p>The top transaction read key <code>9</code> and found it did not\nexist, then appended <code>8</code> to it. The bottom transaction\nappended <code>6</code> to key <code>9</code>. Since every read of key\n<code>9</code> observed a value like <code>[6 8 12 ...]</code>, we know\nthat the append of <code>6</code> happened before the append of\n<code>8</code>. The top transaction failed to observe the bottom, even\nthough the bottom must have occurred before the top. This is a similar\nphenomenon to P4 (Lost Update), and violates Snapshot Isolation.</p>\n<p>Similar behavior (<a href=\"https://github.com/particledbai/jepsen/issues/73\" rel=\"nofollow ugc noopener\">#73</a>)\noccurred in <code>zero-copy-occ</code> and 1.1.18.11.rc-22’s\n<code>occ</code>. Orkes is investigating. Following our collaboration,\nOrkes reports version 1.1.19-rc.13 no longer exhibits G-Single.</p>\n<h3 id=\"3-8-9-brazil-67\">3.8.9 Brazil (#67)</h3>\n<p>In version 1.1.18.9, we found that queries for a specific row by a\nunique, constant integer ID would occasionally return the value of a\ncompletely different row. For example, consider <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260609T162039-g1a-g1b-write-loss.zip\" rel=\"nofollow ugc noopener\">this\n<code>append</code> test</a> running against a single, healthy node with\ntransaction mode <code>occ-v3</code> and WAL mode\n<code>groupsync</code>. Reads of key <code>12</code> mostly observed\nvalues consistent with the appends to key <code>12</code>, but for a\nbrief time observed nothing, and then the value of key <code>7</code>\ninstead:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Time (s)</th><th>Value</th></tr></thead><tbody><tr><td>6.02</td><td>1, 2, 7, … 82</td></tr><tr><td>6.05</td><td>1, 2, 7, … 82</td></tr><tr><td>6.17</td><td></td></tr><tr><td>6.22</td><td></td></tr><tr><td>6.24</td><td>1, 2, 7, … 82, 92</td></tr><tr><td>6.24</td><td></td></tr><tr><td>6.25</td><td>1, 5, 8, … 181, 86</td></tr><tr><td>6.29</td><td></td></tr><tr><td>6.29</td><td>1, 5, 8, … 181, 86</td></tr><tr><td>6.40</td><td></td></tr><tr><td>6.49</td><td></td></tr><tr><td>6.68</td><td>1, 2, 7, … 82, 92, … 139, 142</td></tr></tbody></table></div>\n<p>The value <code>[1, 5, 8, ... 181, 86]</code> was consistent with\nreads of key <code>7</code>, not key <code>12</code>. For the remainder\nof the test, reads observed values consistent with the original version\nof key <code>12</code>: <code>[1, 2, 7, ...]</code>. This issue (<a href=\"https://github.com/particledbai/jepsen/issues/67\" rel=\"nofollow ugc noopener\">#67</a>) did not\nappear in our tests after version 1.1.18.11.rc-1.</p>\n<h3 id=\"3-8-10-lost-writes-with-table-2pl-70\">3.8.10 Lost Writes With <code>table-2pl</code> (#70)</h3>\n<p>In versions 1.1.18.11.rc-1 through 1.1.18.11.rc-6, ParticleDB’s\n<code>table-2pl</code> transaction mode routinely lost committed writes\nduring normal operation. For example <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260610T085702.901-0500-table-2pl-write-loss-deadlock.zip\" rel=\"nofollow ugc noopener\">this\n<code>append</code> test</a>, using the ParticleDB Java client, lost\nwrites to dozens of keys in just a few seconds. Here are some of its\nreads of key <code>187</code>:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Time (s)</th><th>Value</th></tr></thead><tbody><tr><td>10.89</td><td>5</td></tr><tr><td>10.89</td><td></td></tr><tr><td>10.92</td><td>1, 6</td></tr><tr><td>10.92</td><td></td></tr><tr><td>10.92</td><td></td></tr><tr><td>10.92</td><td>1, 6, 10, 11</td></tr><tr><td>10.93</td><td>2</td></tr><tr><td>11.09</td><td>2, 13</td></tr></tbody></table></div>\n<p>Since the <code>append</code> workload only appends to rows and never\ndeletes them, every read should have observed a prefix of the longest\nversion of the list. Instead it appears that the append of\n<code>5</code>, then the appends of <code>1</code>, <code>6</code>,\n<code>10</code>, and <code>11</code>, were lost. Moreover, some of these\nwrites were lost in the middle of a transaction, resulting in internal\nconsistency errors.</p>\n<p>By version 1.1.18.11.rc-7, <code>table-2pl</code> no longer lost\nwrites this frequently (<a href=\"https://github.com/particledbai/jepsen/issues/70\" rel=\"nofollow ugc noopener\">#70</a>).</p>\n<h3 id=\"3-8-11-internal-anomalies-with-occ-v3-96\">3.8.11 Internal Anomalies With <code>occ-v3</code> (#96)</h3>\n<p>ParticleDB 1.1.18.11.rc-9.2 occasionally violated internal\nconsistency using transaction mode <code>occ-v3</code> and WAL sync mode\n<code>groupsync</code>. For example, take <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260620T151000-1.1.18.11.rc-9-internal.zip\" rel=\"nofollow ugc noopener\">this\n<code>append</code> test</a> where a single transaction read\n<code>nil</code>, then <code>[1, 4]</code> for key <code>8</code>:</p>\n<pre><code>[[:r 8 nil] [:r 8 [1 4]]]</code></pre>\n<p>At Read Atomic and above, two successive reads of the same key should\nnot return different results. This also violates Snapshot Isolation and\nSerializability. This bug (<a href=\"https://github.com/particledbai/jepsen/issues/96\" rel=\"nofollow ugc noopener\">#96</a>) may\nhave been resolved by 1.1.18.11.rc-19.</p>\n<h3 id=\"3-8-12-there-is-no-anti-memetics-division-65\">3.8.12 There is No Anti-Memetics Division (#65)</h3>\n<p>ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-9.1 would often forget\na column existed. A query like\n<code>SELECT * FROM snacks WHERE sk = 4</code> would throw an error like\n<code>PDB wire error 1064: column not found: sk</code>. This issue (<a href=\"https://github.com/particledbai/jepsen/issues/65\" rel=\"nofollow ugc noopener\">#65</a>)\noccurred in every transaction mode, but seemed to be resolved by version\n1.1.18.11.rc-9.2.</p>\n<h3 id=\"3-8-13-six-data-blockers-and-a-large-iced-coffee-31\">3.8.13 Six Data Blockers and a Large Iced Coffee (#31)</h3>\n<p>Versions 1.1.18.8 through 1.1.18.11.rc-22 exhibited perplexing\nnondeterministic failures in the <code>compare</code> workload. Each\ntest case established a fresh database client, created a globally unique\ntable, and performed operations against that table synchronously. At no\npoint should there have been two clients concurrently accessing a table.\nAnd yet sometimes, rows failed to appear in queries:</p>\n<pre><code>root=&gt; CREATE TABLE t_876 (a boolean);\nroot=&gt; INSERT INTO t_876 (a) VALUES (NULL);\nroot=&gt; INSERT INTO t_876 (a) VALUES (NULL);\nroot=&gt; SELECT * FROM t_876 WHERE TRUE;\n  a\n------\n NULL\n(1 row)</code></pre>\n<p>One could update rows using the predicate <code>WHERE NULL</code>,\nwhich should never match anything:</p>\n<pre><code>root=&gt; CREATE TABLE t_40 (a boolean);\nroot=&gt; INSERT INTO t_40 (a) VALUES (FALSE);\nroot=&gt; UPDATE t_40 SET a = (a &lt;= a) WHERE NULL;\nUPDATE 1</code></pre>\n<p>In many cases single SQL statements threw errors complaining of\nconcurrent updates, even though at most one client should ever have\naccessed a given table.</p>\n<pre><code>root=&gt; CREATE TABLE t_101 (\n  a text COLLATE unicode PRIMARY KEY\n);\nroot=&gt; INSERT INTO t_101 (a) VALUES (&#39; &#39;);\nroot=&gt; UPDATE t_101 SET a = NULL;\nERROR: could not serialize access due to\n       concurrent update on table &quot;t_101&quot;\n       key &quot; &quot;</code></pre>\n<p>Deleted rows could also remain visible:</p>\n<pre><code>root=&gt; CREATE TABLE t_27763 (\n  a boolean,\n  b boolean NOT NULL UNIQUE PRIMARY KEY\n);\nroot=&gt; INSERT INTO t_27763 (a, b)\n       VALUES (NULL, TRUE);\nroot=&gt; DELETE FROM t_27763;\nroot=&gt; SELECT t_27763.a AS j FROM t_27763\n       WHERE t_27763.b ORDER BY t_27763.b;\n  j\n------\n NULL\n(1 row)</code></pre>\n<p>These anomalies (<a href=\"https://github.com/particledbai/jepsen/issues/31\" rel=\"nofollow ugc noopener\">#31</a>)\noccurred with all transaction modes and WAL sync modes, but attempts to\nreproduce them deterministically, or to narrow down their cause, were\nunsuccessful. They may be artifacts of a bug in the test suite, but we\nlacked time to investigate fully. After our collaboration, Orkes\nreported 1.1.19-rc.13 resolved this issue.</p>\n<h3 id=\"3-8-14-read-skew-with-table-2pl-106\">3.8.14 Read Skew With <code>table-2pl</code> (#106)</h3>\n<p>With the ParticleDB interface, WAL mode <code>groupsync</code>, and\n<code>table-2pl</code>, version 1.1.18.11.rc-18 exhibited <a href=\"https://jepsen.io/consistency/phenomena/g-single\" rel=\"nofollow ugc noopener\">G-single</a>\nunder normal operation. Here’s <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260721T143917-table-2pl-g-single.zip\" rel=\"nofollow ugc noopener\">an\nexample</a>:</p>\n<pre><code>T1: [[:r 112 nil]\n     [:append 112 1]\n     [:append 103 29]\n     [:append 67 105]]\n\nT2: [[:r 110 [10 14 13 20 16 28 30 33 48 35]]\n     [:r 112 [1]]\n     [:r 110 [10 14 13 20 16 28 30 33 48 35]]\n     [:r 103 [1 4 11 15 17 20 24 25]]]</code></pre>\n<p>The first transaction appended <code>1</code> to key <code>112</code>\nand <code>29</code> to key <code>103</code>. The second transaction\nobserved the append to key <code>112</code>, but missed the append to\nkey <code>103</code>. Since it observed but also failed to observe\nanother transaction, this is <a href=\"https://jepsen.io/consistency/phenomena/a5a\" rel=\"nofollow ugc noopener\">A5A (Read Skew)</a>.\nSince the first transaction read-write depends on the second, and the\nsecond write-read depends on the first, it is also G-single. Both are\nviolations of Snapshot Isolation.</p>\n<p>We no longer observed Read Skew with <code>table-2pl</code> (<a href=\"https://github.com/particledbai/jepsen/issues/106\" rel=\"nofollow ugc noopener\">#106</a>) by\nversion 1.1.18.11.rc-19.</p>\n<h3 id=\"3-8-15-stale-read-with-strict-serializable-109\">3.8.15 Stale Read With <code>strict-serializable</code> (#109)</h3>\n<p>Version 1.1.18.11.rc-18 added a new transaction mode,\n<code>strict-serializable</code>, which may have been renamed to\n<code>serializable</code> in 1.1.18.11.rc-22.<a href=\"#fn9\">9</a>\nStrict Serializability (also known as Strong Serializability) guarantees\nthat all transactions appear to execute in real-time order: if *T*1 completes before *T*2 begins, then *T*1 appears to execute\nbefore *T*2. However,\nthis was not true in 1.1.18.11.rc-18. Healthy ParticleDB nodes\ninfrequently but regularly exhibited <a href=\"https://jepsen.io/consistency/phenomena/stale-read\" rel=\"nofollow ugc noopener\">Stale\nRead</a>, even with WAL mode <code>sync</code>. In one of our testing\nenvironments, these anomalies appeared roughly every 300,000\ntransactions, using Elle’s exponential key distribution and 10\nconcurrent clients pushing ~80 transactions per second.</p>\n<p>Stale reads all seemed to share the same structure: a cycle\ncomprising a real-time, read-write, then write-read edge. Here’s <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260723T023218-strict-serializable-stale-read.zip\" rel=\"nofollow ugc noopener\">an\nexample</a> involving a single key:</p>\n<p>The top transaction appended <code>66</code> to key\n<code>22209</code>, which was read by the middle transaction. After the\nmiddle transaction completed, the bottom transaction began, and read key\n<code>22209</code> without <code>66</code>. This is a violation of\nStrong Snapshot Isolation, and therefore Strong Serializability as\nwell.</p>\n<p>After the conclusion of our testing work, Orkes reported that version\n1.1.19-rc.10 no longer exhibits stale read (<a href=\"https://github.com/particledbai/jepsen/issues/109\" rel=\"nofollow ugc noopener\">#109</a>).</p>\n<h3 id=\"3-8-16-lost-writes-110\">3.8.16 Lost Writes (#110)</h3>\n<p>In version 1.1.18.11.rc-18 through 1.1.18.11.rc-22, a healthy\nParticleDB node running <code>occ-v3</code>, <code>zero-copy-occ</code>,\n<code>table-2pl</code>, or <code>serializable-ssi</code> would\noccasionally discard acknowledged writes. For example, take <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260724T083315-occ-v3-data-loss.zip\" rel=\"nofollow ugc noopener\">this\nappend test</a>, which ran with WAL mode <code>sync</code>. Out of\napproximately 20,000 keys, 31 lost writes. Here are some of the reads of\nkey <code>16647</code>:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Time (s)</th><th>Value</th></tr></thead><tbody><tr><td>2798.88</td><td></td></tr><tr><td>2798.88</td><td>1, 2, 3</td></tr><tr><td>2798.88</td><td>1, 2, 3</td></tr><tr><td>2798.90</td><td>1, 2, 3</td></tr><tr><td>2798.90</td><td>1, 2, 3</td></tr><tr><td>2798.93</td><td>1, 2, 3</td></tr><tr><td>2798.96</td><td>1, 2, 3</td></tr><tr><td>2798.98</td><td>4, 5, 6, 7</td></tr><tr><td>2798.98</td><td>4, 5, 6, 7</td></tr><tr><td>2799.02</td><td>4, 5, 6, 7, 27</td></tr><tr><td>2799.02</td><td>1, 2, 3</td></tr><tr><td>2799.02</td><td>1, 2, 3</td></tr><tr><td>2799.02</td><td>1, 2, 3</td></tr><tr><td>2799.02</td><td>4, 5, 6, 7, 27</td></tr></tbody></table></div>\n<p>While the transaction which wrote <code>3</code> committed\nsuccessfully, and its write was visible to several other transactions,\nthe elements <code>1</code>, <code>2</code>, and <code>3</code> were\nreplaced by <code>4, 5, 6, ...</code> after 2799.02 seconds, and never\nreappeared. This occurred every few minutes in our tests, with both the\nPostgres and ParticleDB interfaces.</p>\n<p>A transaction could also read an acknowledged write, modify that row,\nand find that the modification apparently destroyed the written value,\nleading to internal anomalies as well as data loss. After we finished\ntesting Orkes released version 1.1.19-rc.13, and believes that it fixes\nthis issue (<a href=\"https://github.com/particledbai/jepsen/issues/110\" rel=\"nofollow ugc noopener\">#110</a>).</p>\n<h2 id=\"3-9-with-faults\">3.9 With Faults</h2>\n<p>The issues described above occurred in healthy, single-node instances\nof ParticleDB. When we introduced faults, new behaviors emerged.</p>\n<h3 id=\"3-9-1-data-loss-on-wal-bitflips-108\">3.9.1 Data Loss on WAL Bitflips (#108)</h3>\n<p>ParticleDB uses checksums to detect corruption or incomplete writes\nto its Write-Ahead Log (WAL) files. Sometimes these checks prevented the\nserver from starting up with corrupt data. In other cases, ParticleDB\nwould discover that its data files were corrupt, then start up anyway,\nserving data to clients.</p>\n<p>Take <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260722T092352-bitflip-data-loss.zip\" rel=\"nofollow ugc noopener\">this\n<code>append</code> test</a> of 1.1.18.11.rc-18. At 09:25:09 we\nintroduced two single-bit errors into one of ParticleDB’s\n<code>.wal</code> files. At 09:25:39 we read key <code>1754</code> and\nobserved a list of 44 elements:</p>\n<pre><code>[[:r 1754 [3 5 7 9 12 13 18 19 20 22 24 25 28\n           30 31 33 34 36 35 37 38 39 42 43 44\n           45 47 50 51 53 58 59 62 63 64 65 67\n           68 69 70 73 75 79 80]] ...</code></pre>\n<p>A few milliseconds later, we killed and restarted ParticleDB a few\ntimes. When the server came back online at 09:26:07, it logged:</p>\n<pre><code>2026-07-22T14:26:07.985223Z  WARN ThreadId(01)\nrecover_streaming{dir=/opt/particledb/data/wal}:\nspanner_wal::segment: checksum mismatch in\nsegment tail, stopping read segment_id=0\n\n2026-07-22T14:26:07.985235Z  WARN ThreadId(01)\nrecover_streaming{dir=/opt/particledb/data/wal}:\nspanner_wal::recovery: replaying valid prefix\nof already-sealed corrupt WAL segment (.corrupt\nmarker present) and continuing with later\nsegments of this stripe stripe_id=0 segment_id=0\ncorruption_offset=25184 valid_size=25184\nrecovered_entries=242 detail=&quot;checksum mismatch:\nstored 0xf3b8aa37, computed 0x4a774da2&quot;</code></pre>\n<p>However, the server went on to start up and serve client requests\nnormally. At 09:26:09, we read key <code>1754</code> and found those 44\nelements had disappeared:</p>\n<pre><code>[... [:r 1754 [81]] ...]</code></pre>\n<p>Several other keys also lost some or all of their acknowledged\nwrites. Orkes is investigating this issue (<a href=\"https://github.com/particledbai/jepsen/issues/108\" rel=\"nofollow ugc noopener\">#108</a>).\nWhile Jepsen has not tested this version, Orkes reports 1.1.19-rc.12\ndetects bitflips correctly and refuses to start.</p>\n<h3 id=\"3-9-2-total-data-loss-with-simulated-misdirected-reads-writes-11\">3.9.2 Total Data Loss With Simulated Misdirected Reads/Writes (#112)</h3>\n<p>One of our file corruption faults divided a file into four-kilobyte\nchunks, then randomly replaced a small fraction of those chunks with\nother chunks from the same file. The goal is to simulate <a href=\"https://www.cse.cuhk.edu.hk/~pclee/www/pubs/msst14.pdf\" rel=\"nofollow ugc noopener\">misdirected\nreads or writes</a>, a type of silent data corruption that happens in\nsome storage systems. In 1.1.18.11.rc-22, ParticleDB sometimes responded\nto this type of fault by forgetting every table and (apparently)\ndeleting everything in its WAL file.</p>\n<p>For example, take <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260730T095243-copy-file-chunks-table-lost.zip\" rel=\"nofollow ugc noopener\">this\ntest run</a>. At 09:53:56, the test harness shuffled 13 four-kilobyte\nchunks of the first <code>.wal</code> file around—approximately\none-tenth of the file’s contents. It then killed and restarted the node.\nAlthough this file had hundreds of kilobytes in it, when ParticleDB\nstarted up it logged that there was no data on disk:</p>\n<pre><code>2026-07-30T14:53:58.715589Z  INFO ThreadId(01)\nspanner_exec::engine::context: No data found in\nstorage — starting fresh</code></pre>\n<p>At 09:53:58, ParticleDB began throwing\n<code>ERROR: table not found</code> for every query. No request ever\nsucceeded after this time.</p>\n<p>It appears that ParticleDB may have deleted the contents of its WAL\nfile. A few seconds later, the test found its size was zero bytes, and\nthe same held true at the end of the test. After we finished testing\nOrkes released version 1.1.19-rc.12; they believe it resolves this issue\n(<a href=\"https://github.com/particledbai/jepsen/issues/112\" rel=\"nofollow ugc noopener\">#112</a>).</p>\n<h3 id=\"3-9-3-internal-anomalies-with-strict-serializable-and-process-ki\">3.9.3 Internal Anomalies With <code>strict-serializable</code> and Process Kills (#107)</h3>\n<p>With <code>strict-serializable</code>, WAL mode\n<code>groupsync</code>, and the ParticleDB interface, single nodes\nrunning 1.1.18.11.rc-18 exhibited internal anomalies with process kills.\nHere’s <a href=\"https://s3.amazonaws.com/jepsen.io/analyses/particledb-1.1.4/20260721T150149-strict-serializable-internal.zip\" rel=\"nofollow ugc noopener\">an\nexample</a>:</p>\n<pre><code>{:op {:index 190704,\n      :time 100211183125,\n      :type :ok,\n      :process 59,\n      :f :txn,\n      :value [[:r 2879 nil]\n              [:append 2879 16]\n              [:r\n               2879\n               [1 2 3 4 5 7 8 16]]]},\n :mop [:r 2879 [1 2 3 4 5 7 8 16]],\n :expected [16]}</code></pre>\n<p>This transaction read key <code>2879</code> as <code>nil</code>, then\nappended to it, and read its value—only to find that seven elements had\nbeen appended in the intervening time. Transactions could also exhibit\nG-single and P4 (G-single), but we suspect these could be due to the\nsame underlying bug. While Jepsen has not had a chance to test this\nrelease, Orkes claims this issue (<a href=\"https://github.com/particledbai/jepsen/issues/107\" rel=\"nofollow ugc noopener\">#107</a>) was\nresolved in version 1.1.19-rc.12.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>№</th><th>Summary</th><th>Event Required</th><th>Fixed in</th></tr></thead><tbody><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/62\" rel=\"nofollow ugc noopener\">#62</a></td><td>Abysmal performance with <code>table-2pl</code></td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/81\" rel=\"nofollow ugc noopener\">#81</a></td><td>Mystery Crash</td><td>None</td><td>1.1.18.11.rc-18</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/27\" rel=\"nofollow ugc noopener\">#27</a></td><td>Deadlock after two transactions</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/71\" rel=\"nofollow ugc noopener\">#71</a></td><td>Deadlock with <code>table-2pl</code></td><td>None</td><td>1.1.18.11.rc-7</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/33\" rel=\"nofollow ugc noopener\">#33</a></td><td>Ordering by an unused column</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/6\" rel=\"nofollow ugc noopener\">#6</a></td><td>No static check for comparators of incomparable types</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/94\" rel=\"nofollow ugc noopener\">#94</a></td><td>Decimals not comparable to floats</td><td>None</td><td>1.1.18.11.rc-18</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/12\" rel=\"nofollow ugc noopener\">#12</a></td><td>Filter predicates reject <code>NULL</code></td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/13\" rel=\"nofollow ugc noopener\">#13</a></td><td>Cannot compare values to <code>NULL</code></td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/30\" rel=\"nofollow ugc noopener\">#30</a></td><td>Boolean logic on integers</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/50\" rel=\"nofollow ugc noopener\">#50</a></td><td>More boolean logic on integers</td><td>None</td><td>1.1.18.11.rc-8</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/5\" rel=\"nofollow ugc noopener\">#5</a></td><td>Multiple assignment to a single column</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/10\" rel=\"nofollow ugc noopener\">#10</a></td><td>Predicates can refer to nonexistent columns</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/20\" rel=\"nofollow ugc noopener\">#20</a></td><td>No aggregates within <code>CASE</code></td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/28\" rel=\"nofollow ugc noopener\">#28</a></td><td>No exponent operator (<code>^</code>)</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/2\" rel=\"nofollow ugc noopener\">#2</a></td><td>Malformed responses for prepared statements</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/83\" rel=\"nofollow ugc noopener\">#83</a></td><td><code>select</code> throws <code>Bad value for type long</code></td><td>None</td><td>1.1.19-rc.12</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/111\" rel=\"nofollow ugc noopener\">#111</a></td><td>Tuples with no field structure</td><td>None</td><td>1.1.19-rc.13</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/1\" rel=\"nofollow ugc noopener\">#1</a></td><td>Broken Postgres authentication</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/3\" rel=\"nofollow ugc noopener\">#3</a></td><td>Java client could not execute transactions</td><td>None</td><td>1.1.15</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/14\" rel=\"nofollow ugc noopener\">#14</a></td><td>No large integer literals</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/85\" rel=\"nofollow ugc noopener\">#85</a></td><td>No exponentiation on <code>Infinity</code></td><td>None</td><td>1.1.18.11.rc-9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/49\" rel=\"nofollow ugc noopener\">#49</a></td><td><code>-1 ^ 0 = -1</code></td><td>None</td><td>1.1.18.11.rc-3</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/88\" rel=\"nofollow ugc noopener\">#88</a></td><td><code>1.0 + 1E-39 = 1.0</code></td><td>None</td><td>1.1.18.11.rc-9.2</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/56\" rel=\"nofollow ugc noopener\">#56</a></td><td>Large literals become <code>Infinity</code></td><td>None</td><td>1.1.18.11.rc-3</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/61\" rel=\"nofollow ugc noopener\">#61</a></td><td>Exponent underflow</td><td>None</td><td>1.1.18.11.rc-3</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/34\" rel=\"nofollow ugc noopener\">#34</a></td><td>Division by integer <code>0</code></td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/51\" rel=\"nofollow ugc noopener\">#51</a></td><td>Modulo operator cannot handle floats</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/36\" rel=\"nofollow ugc noopener\">#36</a></td><td><code>NaN</code> arithmetic throws errors</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/92\" rel=\"nofollow ugc noopener\">#92</a></td><td><code>NULL</code> arithmetic throws</td><td>None</td><td>1.1.18.11.rc-18</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/60\" rel=\"nofollow ugc noopener\">#60</a></td><td><code>UPDATE</code> arithmetic throws errors</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/32\" rel=\"nofollow ugc noopener\">#32</a></td><td>Same precision for <code>real</code> and <code>double precision</code></td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/11\" rel=\"nofollow ugc noopener\">#11</a></td><td>Integers allow out-of-range values</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/86\" rel=\"nofollow ugc noopener\">#86</a></td><td>Dividing nearby numbers fails</td><td>None</td><td>1.1.18.11.rc-9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/87\" rel=\"nofollow ugc noopener\">#87</a></td><td>Floating-point errors in integer values</td><td>None</td><td>1.1.18.11.rc-9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/19\" rel=\"nofollow ugc noopener\">#19</a></td><td>Floats not equal to quoted representation</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/63\" rel=\"nofollow ugc noopener\">#63</a></td><td><code>0</code> is not equal to <code>0.0</code></td><td>None</td><td>1.1.18.8</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/80\" rel=\"nofollow ugc noopener\">#80</a></td><td><code>5</code> appears equal to <code>0.90...</code></td><td>None</td><td>1.1.18.11.rc-9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/64\" rel=\"nofollow ugc noopener\">#64</a></td><td><code>CHARACTER(n)</code> allows more than <code>n</code> characters</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/52\" rel=\"nofollow ugc noopener\">#52</a></td><td><code>CHARACTER(n)</code> allows more than <code>n</code> characters again</td><td>None</td><td>1.1.18.11.rc-3</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/55\" rel=\"nofollow ugc noopener\">#55</a></td><td><code>CHARACTER(n)</code> allows fewer than <code>n</code> characters</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/22\" rel=\"nofollow ugc noopener\">#22</a></td><td>Default text collation ignores <code>LOCALE</code></td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/42\" rel=\"nofollow ugc noopener\">#42</a></td><td>Mis-sorted text</td><td>None</td><td>1.1.18.11.rc-7</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/93\" rel=\"nofollow ugc noopener\">#93</a></td><td>More mis-sorted text</td><td>None</td><td>1.1.18.11.rc-21</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/40\" rel=\"nofollow ugc noopener\">#40</a></td><td><code>SELECT 5 WHERE FALSE</code> returns <code>5</code></td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/47\" rel=\"nofollow ugc noopener\">#47</a></td><td><code>WHERE</code> clauses with math match nothing</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/18\" rel=\"nofollow ugc noopener\">#18</a></td><td><code>WHERE 4 = 5</code> matches primary key <code>5</code></td><td>None</td><td>1.1.18.4</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/35\" rel=\"nofollow ugc noopener\">#35</a></td><td>Transactional selects return nonsense</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/8\" rel=\"nofollow ugc noopener\">#8</a></td><td><code>NOT NULL</code> constraints don’t</td><td>None</td><td>1.1.18.4</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/9\" rel=\"nofollow ugc noopener\">#9</a></td><td><code>UNIQUE</code> constraints don’t</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/29\" rel=\"nofollow ugc noopener\">#29</a></td><td>Updates can create duplicate primary keys</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/57\" rel=\"nofollow ugc noopener\">#57</a></td><td>Primary keys may be <code>NULL</code></td><td>None</td><td>1.1.18.11.rc-3</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/82\" rel=\"nofollow ugc noopener\">#82</a></td><td>Primary key conflicts with deleted rows</td><td>None</td><td>1.1.18.11.rc-18</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/90\" rel=\"nofollow ugc noopener\">#90</a></td><td>Duplicate primary keys</td><td>None</td><td>1.1.18.11.rc-19</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/58\" rel=\"nofollow ugc noopener\">#58</a></td><td>Can’t join integers to floats</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/54\" rel=\"nofollow ugc noopener\">#54</a></td><td>Joins omit table names, mangle columns instead</td><td>None</td><td>1.1.19-rc.13</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/76\" rel=\"nofollow ugc noopener\">#76</a></td><td>Joins coerce values to strings</td><td>None</td><td>1.1.18.11.rc-21</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/59\" rel=\"nofollow ugc noopener\">#59</a></td><td>Outer joins to empty tables often produce no rows</td><td>None</td><td>1.1.18.11.rc-3</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/77\" rel=\"nofollow ugc noopener\">#77</a></td><td>Joins confuse columns with same name</td><td>None</td><td>1.1.18.11.rc-7</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/26\" rel=\"nofollow ugc noopener\">#26</a></td><td>Transactions can’t delete anything</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/45\" rel=\"nofollow ugc noopener\">#45</a></td><td>Transactions which delete leave shades behind</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/23\" rel=\"nofollow ugc noopener\">#23</a></td><td>Updating integers makes rows not-deletable</td><td>None</td><td>1.1.18.4</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/84\" rel=\"nofollow ugc noopener\">#84</a></td><td><code>double precision</code> primary keys yield zombie twins</td><td>None</td><td>1.1.18.11.rc-9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/78\" rel=\"nofollow ugc noopener\">#78</a></td><td><code>NaN</code> values are immutable</td><td>None</td><td>1.1.18.11.rc-7</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/74\" rel=\"nofollow ugc noopener\">#74</a></td><td>Fixed-width strings are deeply cursed</td><td>None</td><td>1.1.18.11.rc-7</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/46\" rel=\"nofollow ugc noopener\">#46</a></td><td>Dropped tables can be resurrected, in distorted forms</td><td>None</td><td>1.1.18.11.rc-3</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/15\" rel=\"nofollow ugc noopener\">#15</a></td><td><code>Infinity</code> is writable, but unreadable</td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/21\" rel=\"nofollow ugc noopener\">#21</a></td><td><code>NaN</code> sometimes becomes <code>0.0</code></td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/7\" rel=\"nofollow ugc noopener\">#7</a></td><td>Assignment of incompatible types yields (e.g.) <code>NULL</code></td><td>None</td><td>1.1.18.6</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/16\" rel=\"nofollow ugc noopener\">#16</a></td><td>Values assigned to wrong columns, yielding cursed rows</td><td>None</td><td>1.1.18.4</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/24\" rel=\"nofollow ugc noopener\">#24</a></td><td>Updates of no rows create new, corrupt rows instead</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/25\" rel=\"nofollow ugc noopener\">#25</a></td><td>No-op updates cause duplicate key violations</td><td>None</td><td>1.1.18.9</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/37\" rel=\"nofollow ugc noopener\">#37</a></td><td>Strings are sometimes returned as 8-bit integers</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/72\" rel=\"nofollow ugc noopener\">#72</a></td><td>Primary keys mysteriously become <code>null</code></td><td>None</td><td>1.1.18.11.rc-7</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/44\" rel=\"nofollow ugc noopener\">#44</a></td><td>Nested transactions destroy ongoing ones</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/43\" rel=\"nofollow ugc noopener\">#43</a></td><td>Aborted transactions can commit</td><td>None</td><td>1.1.18.11.rc-18</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/48\" rel=\"nofollow ugc noopener\">#48</a></td><td>Aggregates sometimes miss a transaction’s rows</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/89\" rel=\"nofollow ugc noopener\">#89</a></td><td>Queries sometimes observe duplicate copies of rows</td><td>None</td><td>1.1.18.11.rc-18</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/91\" rel=\"nofollow ugc noopener\">#91</a></td><td>Updates to special values can silently fail</td><td>None</td><td>1.1.18.11.rc-9.2</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/41\" rel=\"nofollow ugc noopener\">#41</a></td><td>Tables with <code>text</code> primary keys are mostly invisible</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/66\" rel=\"nofollow ugc noopener\">#66</a></td><td>G1a (Aborted Read)</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/38\" rel=\"nofollow ugc noopener\">#38</a></td><td>G1b (Intermediate Read)</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/68\" rel=\"nofollow ugc noopener\">#68</a></td><td>Internal consistency anomalies</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/69\" rel=\"nofollow ugc noopener\">#69</a></td><td>Lost write</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/73\" rel=\"nofollow ugc noopener\">#73</a></td><td>G-single</td><td>None</td><td>1.1.19-rc.13</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/67\" rel=\"nofollow ugc noopener\">#67</a></td><td>Reads observe data from different row</td><td>None</td><td>1.1.18.11.rc-1</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/70\" rel=\"nofollow ugc noopener\">#70</a></td><td>Lost write with <code>table-2pl</code></td><td>None</td><td>1.1.18.11.rc-7</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/96\" rel=\"nofollow ugc noopener\">#96</a></td><td>Occasional internal consistency anomalies with <code>occ-v3</code></td><td>None</td><td>1.1.18.11.rc-19</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/65\" rel=\"nofollow ugc noopener\">#65</a></td><td>Columns sometimes vanish</td><td>None</td><td>1.1.18.11.rc-9.2</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/31\" rel=\"nofollow ugc noopener\">#31</a></td><td>Odd nondeterministic single-threaded anomalies</td><td>None</td><td>1.1.19-rc.13</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/106\" rel=\"nofollow ugc noopener\">#106</a></td><td>A5A (Read Skew) with <code>table-2pl</code></td><td>None</td><td>1.1.18.11.rc-19</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/109\" rel=\"nofollow ugc noopener\">#109</a></td><td>Stale Read with <code>strict-serializable</code></td><td>None</td><td>1.1.19-rc.10</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/110\" rel=\"nofollow ugc noopener\">#110</a></td><td>Lost writes</td><td>None</td><td>1.1.19-rc.13</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/108\" rel=\"nofollow ugc noopener\">#108</a></td><td>Silent(ish) data loss</td><td>WAL truncation/bitflips</td><td>1.1.19-rc.12</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/112\" rel=\"nofollow ugc noopener\">#112</a></td><td>Total data loss</td><td>Misdirected reads/writes</td><td>1.1.19-rc.12</td></tr><tr><td><a href=\"https://github.com/particledbai/jepsen/issues/107\" rel=\"nofollow ugc noopener\">#107</a></td><td>Spurious <code>nil</code> reads</td><td>Process crash</td><td>1.1.19-rc.12</td></tr></tbody></table></div>\n<h1 id=\"4-discussion\">4 Discussion</h1>\n<p>ParticleDB 1.1.4 claimed to be a Postgres-compatible database which\noffered Snapshot Isolation with its <code>occ-v3</code> and\n<code>zero-copy-occ</code> transaction modes, Serializability with\n<code>serializable-ssi</code>, and Strong Serializability with\n<code>table-2pl</code>.</p>\n<p>Instead, we found a library of confusion.<a href=\"#fn10\">10</a>\nIn versions 1.1.4 through 1.1.18.11.rc-19, basic functionality like\ntransactions (#3), prepared statements (#2), and Postgres auth (#1) were\nunusably broken. The server spontaneously deadlocked (#27, #71) or\ncrashed (#81) every few minutes. Integers stood in for booleans (#30,\n#50), comparisons to <code>NULL</code> threw errors (#13), and you could\nassign multiple values to the same column (#5). Large floating-point\nnumbers quietly became infinite (#56); <code>Infinity</code>,\n<code>NaN</code>, and <code>NULL</code> arithmetic were broken (#36,\n#92, #15, #21); and out-of-range values were improperly stored or\nclipped to fit (#32, #11). In ParticleDB, <code>0.0</code> was not equal\nto <code>0</code> (#63), but <code>0.90...</code> wasn’t <em>not</em>\nequal to <code>5</code> (#80). Strings allowed too many (#52, #64) and\ntoo few (#55) characters. A <code>WHERE</code> clause with\nfloating-point operations would silently fail to match any rows (#47),\n<code>WHERE 4 = 5</code> returned rows with primary key <code>5</code>\n(#18), <code>WHERE a = 1</code> matched the row <code>{a: 2}</code>, and\n<code>WHERE TRUE</code> matched nothing (#35). Null (#8) and unique (#9)\nconstraints did nothing, and primary keys could be duplicated (#29),\nnull (#57), or both. Primary keys failed to conflict with rows that did\nexist (#90), and conflicted with rows that didn’t (#82). Joins returned\nincorrect column names (#54), coerced values to strings (#76), failed to\nreturn rows (#59), and confused similarly-named columns (#77). It was\nimpossible to delete rows within a transaction (#26, #45), rows with\nintegers set to floats (#23), or with floats set to integers (#84), or\nthose containing <code>NaN</code>s (#78), or fixed-width strings\n(#74).<a href=\"#fn11\">11</a> Rows from dropped tables could\nre-emerge in mangled forms (#46). Columns could be set to values of the\nwrong type, silently altering or corrupting data (#7). Inserts ignored\nprovided columns, creating rows best described as “deeply cursed”\n(#16).<a href=\"#fn12\">12</a> Updates could create new, shambolic\nrows where none existed (#24, #25). Strings sometimes turned into\neight-bit integers (#37), and primary keys occasionally became\n<code>NULL</code> (#72). Aborted transactions could commit, or fail with\ntheir effects still visible (#43). Queries sometimes missed (#48),\nduplicated (#89), or returned the wrong (#67) rows. Tables with a\n<code>text</code> primary key were mostly invisible (#41). Special\nvalues were sometimes inalterable (#91). Entire columns occasionally\nvanished for no obvious reason (#65). Uncontended statements on freshly\ncreated tables observed nonsensical results or conflicted (apparently)\nwith themselves (#31). Each transaction and WAL mode exhibited\nessentially every transactional anomaly our tests could detect: write\nloss, internal consistency violations, G0, G1a, G1b, G1c, G-Single,\nG-nonadjacent, P4, A5A, G2, and real-time variants thereof (#38, #66,\n#68, #69, #73, #70, #96, #109, #110, etc.).</p>\n<p>These behaviors occurred in single-node instances of ParticleDB,\nwithout fault injection, often with just a single client. When we killed\nParticleDB, it allowed internal anomalies, P4 (Lost Update), and\nG-single (#107), even with the new <code>serializable</code> transaction\nmode. Corruption or truncation of WAL files caused partial data loss\n(#108). Simulating misdirected reads or writes could cause total data\nloss (#112).</p>\n<p>As we concluded our engagement, version 1.1.18.11.rc-22 had just\nbecome available. It abandoned the <code>table-2pl</code> and\n<code>serializable-ssi</code> transaction modes in favor of\n<code>fast</code>, <code>occ</code>, and <code>serializable</code>,\nwhich were intended to offer no consistency, Snapshot Isolation, and\nStrong Serializability, respectively. While 1.1.18.11.rc-22 was\nsignificantly safer than 1.1.4, we were still finding new issues. The\ndefault <code>occ</code> mode regularly lost data in normal operation\n(#110). The new <code>serializable</code> mode appeared significantly\nsafer, but still allowed stale read (#109) in normal operation, and with\nprocess kills, sometimes failed to read extant rows. This led to\ninternal anomalies, Lost Update, and more (#107).</p>\n<p>After we finished testing, Orkes updated ParticleDB’s documentation\nto claim that every transaction mode discussed here was <a href=\"https://particledb.ai/operations/modes/#experimental-modes-not-in-current-release-build\" rel=\"nofollow ugc noopener\">“experimental”</a>.\nThis claim is difficult to support. To the best of Jepsen’s knowledge,\nnone of the ParticleDB documentation <a href=\"https://web.archive.org/web/20260518193025/https://particledb.ai/getting-started/concepts/#transaction-modes\" rel=\"nofollow ugc noopener\">described\nthese modes</a> as experimental or unsupported during our engagement.\nOrkes’ unpublished operator guide did describe one mode,\n<code>serializable-occ</code>, as an “internal A/B variant… do not use\nfor correctness runs”. We considered that mode experimental and omitted\nit from this report. However, the others had no such caveats.</p>\n<p>Orkes <a href=\"https://particledb.ai/operations/modes/\" rel=\"nofollow ugc noopener\">appears to\nhave reworked</a> ParticleDB’s approach to consistency. It now supports\nthe standard SQL isolation level statements for\n<code>READ COMMITTED</code>, <code>REPEATABLE READ</code> (meaning\nSnapshot Isolation), and <code>SERIALIZABLE</code>. The\n<code>--txn-mode</code> option is gone, and a new\n<code>--concurrency-control</code> setting has <code>none</code>,\n<code>optimistic</code>, and <code>strict-optimistic</code> options. The\n<a href=\"https://particledb.ai/operations/modes/#experimental-modes-not-in-current-release-builds\" rel=\"nofollow ugc noopener\">documentation\nclaims</a> that every mode discussed in this report has been removed\nfrom version 1.1.19-rc.13. However, it goes on to say that\n<code>occ-v3</code> and <code>strict-serializable</code> were “earlier\nnames” of the “optimistic engine” and “strict serializable mode”,\nrespectively. It is not clear whether these modes were removed or simply\nrenamed.</p>\n<p>Jepsen has not evaluated versions of ParticleDB after\n1.1.18.11.rc-22, and cannot speak to these isolation levels or\nconcurrency control modes. However, Orkes claims that all issues from\nthis report have been addressed as of version 1.1.19-rc.13.</p>\n<p>As always, Jepsen takes an experimental approach to safety\nverification: we can prove the presence of bugs, but not their absence.\nWhile we make extensive efforts to find problems, we cannot prove\ncorrectness.</p>\n<h2 id=\"4-1-impact\">4.1 Impact</h2>\n<p>Orkes believes that the issues we found in 1.1.4 were likely due to\nan extensive refactor of ParticleDB in spring 2026, and that Orkes and\ntheir customers, who continued to run an earlier version, were likely\nunaffected. Near the end of our collaboration we briefly evaluated an\ninternal build of version 1.0.0. While it exhibited some string and\narithmetic issues, that version did not seem susceptible to most of the\nserious safety errors we found in 1.1.4, and passed cursory\n<code>rw</code> and <code>append</code> tests. We tested only a single\nnode at a time, so we cannot speak to the safety of the replication\nsystem.</p>\n<p>From May through July 2026, the ParticleDB <a href=\"https://particledb.ai/installation/binary/\" rel=\"nofollow ugc noopener\">download page</a>\noffered users (at least) versions 1.1.4 through 1.1.18.11.rc-18.<a href=\"#fn13\">13</a> All exhibited serious safety errors\nduring normal operation. It seems likely that users who ran these\nversions of ParticleDB would have lost or corrupted data. In version\n1.1.18.11.rc-22, using WAL mode <code>sync</code> and transaction mode\n<code>serializable</code> significantly reduced the frequency and\nseverity of these issues, but did not prevent internal anomalies or Lost\nUpdate.</p>\n<p>Given the severity and extent of the safety errors we found, and the\nprocess which produced them, Jepsen recommends users halt further\ndeployment of ParticleDB and migrate to other databases as soon as\npossible. Consider checking and/or rebuilding database state from more\nreliable sources, if available.</p>\n<h2 id=\"4-2-the-unreasonable-effectiveness-of-generative-testing\">4.2 The Unreasonable Effectiveness of Generative Testing</h2>\n<p>The <code>compare</code> workload generated random schemas and SQL\nstatements over them, then ran both against ParticleDB and Postgres to\nsee if they produced equivalent results. This should not have worked.\nThere are many ways in which two “Postgres-compatible” databases can\nlegitimately differ, such as floating-point details, which error message\ntakes precedence, or the order of unsorted results. The space of SQL\nqueries is large and difficult to cover. Many queries trivially return\nequivalent results (e.g. <code>SELECT ... WHERE FALSE</code>) and\ntherefore tell us little information. We can test only sequential\nbehavior, which is easy for a database’s standard test suite to\ncheck.</p>\n<p>Nevertheless, roughly <a href=\"https://github.com/jepsen-io/particledb/blob/d4721aeef9435af0131eacfb2163059272b13d57/src/jepsen/particledb/gen/tc.clj\" rel=\"nofollow ugc noopener\">two\nthousand lines</a> of generative tests found a wealth of bugs in\nParticleDB, and unlike Jepsen’s typical workloads, the examples\ngenerated were usually easy to reproduce. ParticleDB offered a good deal\nof “low-hanging fruit” that should have been caught by example-based\ntests—but the <code>compare</code> workload also found more subtle\nerrors. For example, the fact that <code>0.90...</code> was apparently\nequal to <code>5</code> hinged on a high-precision floating-point\nliteral. Inspired by this success, we extracted part of the ParticleDB\nSQL generator into <a href=\"https://github.com/jepsen-io/sql\" rel=\"nofollow ugc noopener\">a shared\nlibrary</a> so it could be used in other tests, like the new\n<code>internal-sim</code> workload.</p>\n<p>Jepsen has traditionally focused on thorough analysis of a small set\nof carefully chosen transactions. This is partly because checking (e.g.)\nSerializability in general <a href=\"https://dl.acm.org/doi/10.1145/322154.322158\" rel=\"nofollow ugc noopener\">is NP-complete</a>,\nand also because we assume that if transactions of (e.g.) reads and\nappends work correctly on strings, so will other operations on other\ndatatypes. Our intuition is that the “hard part” is in the concurrency\ncontrol mechanism, and sequential datatype behavior is generally covered\nby existing tests. This was not the case in ParticleDB. We’ve expanded\nJepsen’s <code>list-append</code> and <code>rw</code> tests to use\nalternate encodings and transaction structures, in an attempt to cover\nmore codepaths. This revealed new bugs in ParticleDB, and we hope it\nwill in other databases too.</p>\n<p>Unfortunately, <code>compare</code> became less useful as Orkes fixed\nmajor bugs and left smaller incompatibilities in place. For example,\nPostgres checks unique constraints when updating each row of a table,\nbut ParticleDB only checks after all have been updated. This means that\na statement like <code>UPDATE bools SET x = NOT x</code> can swap the\nvalues of <code>TRUE</code> and <code>FALSE</code> in ParticleDB without\nviolating a unique constraint. Toward the end of our engagement, the\n<code>compare</code> test generally hit these minor inconsistencies\nearly, then spent most of its time searching for minimal cases of\nthem—potentially missing more serious bugs.</p>\n<h2 id=\"4-3-quality-assurance\">4.3 Quality Assurance</h2>\n<p>Orkes said they ran Jepsen’s existing Postgres test harness to find\nand resolve issues in ParticleDB prior to this work. However, that same\ntest harness failed immediately when run against ParticleDB 1.1.4. Orkes\nalso stated they tested ParticleDB with <a href=\"http://www.sqlancer.com/\" rel=\"nofollow ugc noopener\">SQLancer</a>, which generates far more\ncomplex queries than the ones we tested. It is therefore perplexing that\nnearly every feature we tested exhibited glaring errors, and that our\nfindings seemed to come as a surprise.</p>\n<p>Most databases have a test suite which exercises at least basic\nexamples of each feature, and it is customary to run those tests before\nrelease, if not continuously. Where were those tests for ParticleDB? How\ndoes one add support for <code>NOT NULL</code> constraints without a\ntest which tries to insert at least one <code>NULL</code> value? What\ndevelopment process adds a <code>CHARACTER(10)</code> type without\ntrying to write a string of (e.g.) five characters?</p>\n<p>Our findings suggest structural deficiencies in the ParticleDB QA\nprocess. Perhaps tests were not run prior to release. Perhaps the test\nsuite was run, but failed to exercise basic functionality. Perhaps\nengineers entrusted Large Language Models with the work of testing.</p>\n<p>Jepsen recommends that Orkes (and all database vendors!) build and\nexercise an automated test suite for their database. This suite should\ninclude at least simple, example-based tests covering important database\nfeatures. These tests should fail when the feature is unimplemented or\nmalformed, and pass when it is complete. Observing these failures is an\nimportant part of the development process: it <a href=\"https://en.wikipedia.org/wiki/Mutation_testing\" rel=\"nofollow ugc noopener\">validates that the\ntest measures something</a>. Tests should be run at least once before\neach release, and results disseminated to the team. Afterwards consider\nmore sophisticated generative tests, establishing adversarial QA and\nimplementation teams, and subtly breaking safety mechanisms to determine\nwhether the tests are still effective.</p>\n<h2 id=\"4-4-documentation\">4.4 Documentation</h2>\n<p>ParticleDB’s website gave the appearance of a mature database\nproject. It made strong claims: a general-purpose OLTP/OLAP SQL database\na thousand times faster than PostgreSQL, “formally verified” by a TLA+\nproof, multiple APIs, and client libraries in thirteen languages. It\noffered a variety of deployment options and extensive documentation on\ndatabase features and operations.</p>\n<p>On closer inspection, that documentation was riddled with omissions,\ninconsistencies, and confabulations. The TLA+ proof was not published,\nand in any event did not prevent disaster—perhaps it was incorrect,\nformalized the wrong properties, or did not describe the system as\nbuilt. Safety-critical configuration settings like the transaction mode\nwere described in vague terms, like <a href=\"https://web.archive.org/web/20260518193025/particledb.ai/getting-started/concepts\" rel=\"nofollow ugc noopener\">“safe\nunder concurrency”</a>. The documentation for the <a href=\"https://web.archive.org/web/20260518203023/https://particledb.ai/sdks/java/\" rel=\"nofollow ugc noopener\">Java\nclient library</a> referred to non-existent dependency coordinates and\nseemed to make up class and package names which were nowhere to be found\nin the library itself. The quickstart and client examples made up a\nvariety of default usernames for ParticleDB, all of which were wrong.\nThe stored procedure documentation even fabricated an <a href=\"https://web.archive.org/web/20260727225950/https://particledb.ai/particles/rust/#run-it\" rel=\"nofollow ugc noopener\">imaginary\nemail address</a> for the author of this report. In Orkes’ unpublished\noperator guide, the descriptions of transaction modes omitted basic\nphenomena like <a href=\"https://jepsen.io/consistency/phenomena/g0\" rel=\"nofollow ugc noopener\">G0</a> and confused <a href=\"https://jepsen.io/consistency/phenomena/g1a\" rel=\"nofollow ugc noopener\">G1a</a> (Aborted\nRead) with <a href=\"https://jepsen.io/consistency/phenomena/g1c\" rel=\"nofollow ugc noopener\">G1c</a>\n(Circular Information Flow). The Snapshot Isolation modes made no\nmention of <a href=\"https://jepsen.io/consistency/phenomena/g-single\" rel=\"nofollow ugc noopener\">G-single</a> or\nthe <a href=\"https://jepsen.io/consistency/phenomena/g-nonadjacent\" rel=\"nofollow ugc noopener\">other\nboundary phenomena</a> proscribed by Snapshot Isolation, and\nnon-real-time models made no mention of the real-time phenomena they\nallowed, like Stale Read.<a href=\"#fn14\">14</a></p>\n<p>Technical writing is important work that requires skill and care.\nJepsen recommends that Orkes work with a human technical writer to\nrewrite ParticleDB’s documentation, with an eye toward concisely and\naccurately communicating intended behavior and the guarantees users\nshould expect.</p>\n<p>Jepsen is no stranger to working with early software projects, and\nthere is nothing intrinsically wrong with offering buggy or badly broken\nsoftware. However, it would be polite for a maintainer aware of serious\nsafety issues to display a prominent warning on the website. One might\nplace a large red box on the download page which reads, “This project is\nin early development and is not suitable for production”, or more\nbluntly, “This software has lost data and will do so again”. Orkes\nshould clearly explain the general status of ParticleDB, and advise\nusers of specific known safety issues.</p>\n<p>Orkes has since published <a href=\"https://particledb.ai/installation/releases/#release-history\" rel=\"nofollow ugc noopener\">a\nrelease history</a> which describes versions 1.1.4 through\n1.1.18.11.rc-22 as “unsupported pre-release builds”. It is not clear how\nusers would have understood versions like 1.1.4 or 1.1.18.9 as\n“pre-release”. Pre-release software generally has a version number\nbeginning with 0, or including “rc”, “alpha”, etc. The documentation <a href=\"https://web.archive.org/web/20260518201315/https://particledb.ai/installation/binary/\" rel=\"nofollow ugc noopener\">did\nnot describe these versions</a> as pre-release or experimental at the\ntime. Indeed, the <a href=\"https://web.archive.org/web/20260518200305/https://particledb.ai/installation/docker/\" rel=\"nofollow ugc noopener\">Docker\ninstall page</a> told users how to deploy these versions “for\nproduction”.</p>\n<h2 id=\"4-5-llm-tradeoffs\">4.5 LLM Tradeoffs</h2>\n<p>Large Language Models enable small teams to produce software and\ndocumentation of staggering scope and complexity in a remarkably short\ntime. Five years ago, writing a distributed SQL database of this scope\nmight have taken hundreds of person-years.<a href=\"#fn15\">15</a>\nWith LLM assistance, a handful of engineers at Orkes were able to\nproduce a highly sophisticated database system.</p>\n<p>However, our work on ParticleDB (and other databases) suggests there\nare serious tradeoffs to relying on LLMs. Quality suffered: ParticleDB\nwas a fractal of correctness errors. LLM tooling repeatedly claimed that\nbugs had been fixed when, in fact, they had not. ParticleDB also seemed\nmore difficult to understand than typical databases. As with most Jepsen\nanalyses, Orkes engineers offered detailed explanations of the causes\nand resolutions of the bugs we found in <a href=\"https://jepsen.io/analyses/orkes-conductor-5.3.2\" rel=\"nofollow ugc noopener\">Conductor</a>.\nHowever, Orkes seemed unable to explain the causes or resolutions of\nnearly every issue we found in ParticleDB. New builds simply\nmaterialized, sans changelogs, each hopefully containing a slew of\nimprovements. Why did ParticleDB sometimes turn strings into integers?\nWhy did it stop in 1.1.18.11.rc-1? Perhaps no one knows. With LLMs, <a href=\"https://www.tiktok.com/@charlixcx/video/7603137654682570006\" rel=\"nofollow ugc noopener\">perhaps\nno one needs to</a>.</p>\n<p>More broadly, the use of LLMs erodes trust in documentation, in\nsoftware, and in the organizations which produce them. Perhaps engineers\nwill begin to treat any new software project with suspicion—the signals\nwe used to rely on as markers of a mature system are now easier to\ncounterfeit.</p>\n<p>Where correctness matters—and surely it ought to matter for\ndatabases!—our industry must acknowledge and mitigate these tradeoffs.<a href=\"#fn16\">16</a> In the age of LLMs, what can\nmaintainers do to ensure correct software and accurate documentation?\nHaving done that work, how can they demonstrate its quality to\nusers?</p>\n<h2 id=\"4-6-future-work\">4.6 Future Work</h2>\n<p>As of 1.1.18.11.rc-22, our <code>append</code> and <code>rw</code>\nworkloads were still finding anomalies in every transaction mode. We\nrecommend further testing as development continues. The\n<code>compare</code> workload also revealed a number of minor\ncompatibility bugs which we have not discussed here. These small\ndifferences could mask more serious errors. Bringing ParticleDB into\ncloser alignment with Postgres could help the <code>compare</code>\nworkload reveal them.</p>\n<p>Toward the end of our collaboration we tested ParticleDB’s on-disk\nformat, and found that corruption in WAL files could cause data loss.\nThe variety of error messages we found suggests that there might be\nmultiple paths to data loss—further tests should stress on-disk storage\nmore thoroughly. Furthermore, in almost all our tests ParticleDB wrote\nonly WAL files, and stored its tables entirely in memory. We recommend\nOrkes add a configuration option to reduce the thresholds for on-disk\nstorage, so that future tests can exercise those codepaths.</p>\n<p>The ParticleDB client has a number of slow network timeouts we did\nnot know how to change. These timeouts could conceal safety errors by\npreventing the test from observing the database during a critical\nmoment. We worked around this by using the Postgres client (whose\ntimeouts are configurable), but we recommend testing the ParticleDB\nclient with lower timeouts as well.</p>\n<p>ParticleDB offered a massive set of features, and we only tested a\nsmall portion of them. The SQL specification and Postgres’ dialect of it\nare astonishingly complex languages, and we used only a small fraction\nof their syntax and vocabulary. We also did not evaluate ParticleDB’s\nRedis or HTTP APIs, temporal or geospatial queries, materialized views,\nvector search, or multi-tenant features. The replication system remains\nto be tested.</p>\n<p>Orkes is now adding stored procedures to ParticleDB. These\nuser-defined functions, called <a href=\"https://particledb.ai/particles/overview/\" rel=\"nofollow ugc noopener\"><em>Particles</em></a>,\nare written in Python or Rust and executed by the database server. Both\nRust and Python have large surface areas, and executing user-provided\nbinaries in a database environment seems like an exciting opportunity.\nOur <a href=\"https://jepsen.io/analyses/capela-dda5892\" rel=\"nofollow ugc noopener\">previous\nwork</a> with stored procedures suggests this feature is worth further\ninvestigation.</p>\n<p>ParticleDB claimed to be “1087x faster than PostgreSQL”. In our tests\nParticleDB was nowhere near this fast, and over the course of an hour we\nobserved a linear decline in throughput and a significant increase in\nmedian latencies—even over small datasets. Once ParticleDB reaches a\nbaseline level of correctness, revisiting performance benchmarks could\nbe worthwhile.</p>\n<p>*Our thanks to the ParticleDB team at Orkes, including Jeu George\nand Nikhil Kala. Jepsen also wishes to thank <a href=\"https://www.irenekannyo.com/\" rel=\"nofollow ugc noopener\">Irene Kannyo</a> for her editorial\nsupport. This research was funded by Orkes, Inc., and conducted in\naccordance with the <a href=\"https://jepsen.io/analyses/ethics\" rel=\"nofollow ugc noopener\">Jepsen\nethics policy</a>.*</p>\n<h1 id=\"5-appendix-a\">5 Appendix A</h1>\n<p>This appendix contains additional results involving types, syntax,\nnumbers, and strings.</p>\n<h2 id=\"5-1-type-safety-and-syntax\">5.1 Type Safety and Syntax</h2>\n<p>We found several incompatibilities or bugs involving static or\ndynamic type safety, or behaviors that would normally have been caught\nby static analysis of a query, such as referring to a nonexistent\ncolumn. These issues occurred with a single server, single client, and\nno faults.</p>\n<h3 id=\"5-1-1-ordering-by-an-unused-column-33\">5.1.1 Ordering By an Unused Column (#33)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.8, you could order an aggregate\nquery’s results by a column that was not used in the aggregate\nexpressions or <code>GROUP BY</code> clause. This seems like a minor\nissue, but it does differ from Postgres behavior:</p>\n<pre><code>root=&gt; CREATE TABLE t (a boolean);\nroot=&gt; SELECT COUNT(*) FROM t\n       WHERE (a &gt;= a) ORDER BY a LIMIT 0;\n COUNT\n-------\n(0 rows)</code></pre>\n<p>Orkes resolved this issue (<a href=\"https://github.com/particledbai/jepsen/issues/33\" rel=\"nofollow ugc noopener\">#33</a>) in\n1.1.18.9.</p>\n<h3 id=\"5-1-2-comparison-of-incomparable-types-6\">5.1.2 Comparison of Incomparable Types (#6)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.4, you could write predicates for\n<code>UPDATE</code> and <code>DELETE</code> which made equality\ncomparisons between columns of different types.</p>\n<pre><code>root=&gt; CREATE TABLE t (a text, b smallint);\nroot=&gt; DELETE FROM t WHERE (a = b);\nDELETE 0</code></pre>\n<p>In Postgres, this threw an error like\n<code>ERROR: operator does not exist: text = smallint</code>. You could\ncompare strings to boolean columns, or literal values like\n<code>false</code>. ParticleDB allowed these comparisons on empty\ntables, but would throw an error if a record was present. In 1.1.18.6,\nParticleDB threw an error for empty tables as well (<a href=\"https://github.com/particledbai/jepsen/issues/6\" rel=\"nofollow ugc noopener\">#6</a>).</p>\n<h3 id=\"5-1-3-cannot-compare-decimals-to-floats-94\">5.1.3 Cannot Compare Decimals to Floats (#94)</h3>\n<p>ParticleDB 1.1.18.11.rc-9.2 forgot how to compare large decimal\nliterals to 64-bit floating-point numbers:</p>\n<pre><code>root=&gt; CREATE TABLE t (a double precision);\nroot=&gt; INSERT INTO t (a) VALUES (1.0);\nroot=&gt; SELECT * FROM t WHERE\n       ((0 / 1.2002480805601245E26) &gt;= t.a);\nERROR: arrow compute: Invalid argument error:\n       Invalid comparison operation:\n       Decimal256(76, 44) &gt;= Float64</code></pre>\n<p>This issue (<a href=\"https://github.com/particledbai/jepsen/issues/94\" rel=\"nofollow ugc noopener\">#94</a>) no\nlonger appeared by version 1.1.18.11.rc-18.</p>\n<h3 id=\"5-1-4-filter-predicates-reject-null-12\">5.1.4 Filter Predicates Reject <code>NULL</code> (#12)</h3>\n<p>SQL has three boolean values: <code>TRUE</code>, <code>FALSE</code>,\nand <code>NULL</code>, which represents an unknown value. In ParticleDB\n1.1.4 through 1.1.18.4, predicates in SELECT, UPDATE, etc. could not\nevaluate to <code>NULL</code>, even though the SQL specification states\nthat search conditions are boolean expressions. For example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a boolean);\nroot=&gt; INSERT INTO t (a) VALUES (TRUE);\nroot=&gt; SELECT * FROM t WHERE NULL;\nERROR:  filter predicate must produce boolean\nresult</code></pre>\n<p>In Postgres this would return an empty relation, because\n<code>NULL</code> is not <code>TRUE</code>. In 1.1.18.6,\n<code>NULL</code> predicates evaluated correctly (<a href=\"https://github.com/particledbai/jepsen/issues/12\" rel=\"nofollow ugc noopener\">#12</a>).</p>\n<h3 id=\"5-1-5-cannot-compare-values-to-null-13\">5.1.5 Cannot Compare Values to <code>NULL</code> (#13)</h3>\n<p>Per the ANSI SQL standard, the value of comparing anything to\n<code>NULL</code> is <em>unknown</em>, represented as the boolean value\n<code>NULL</code>. However, ParticleDB 1.1.4 through 1.1.17 sometimes\nrefused to evaluate predicates which compared (e.g.) integers to\n<code>NULL</code>.</p>\n<pre><code>root=&gt; CREATE TABLE t (a int);\nroot=&gt; INSERT INTO t (a) VALUES (2), (NULL);\nroot=&gt; UPDATE t SET a = 5 WHERE a = NULL;\nERROR:  arrow compute: Invalid argument error:\nInvalid comparison operation: Int64 == Null</code></pre>\n<p>Oddly enough, ParticleDB correctly evaluated statements like\n<code>SELECT 1 = NULL</code>. Version 1.1.18.6 correctly compared\nintegers to <code>NULL</code> (<a href=\"https://github.com/particledbai/jepsen/issues/13\" rel=\"nofollow ugc noopener\">#13</a>).</p>\n<h3 id=\"5-1-6-boolean-logic-on-integers-30\">5.1.6 Boolean Logic on Integers (#30)</h3>\n<p>The SQL standard is clear that boolean logical expressions like\n<code>A AND B</code> evaluate to booleans. In ParticleDB 1.1.4 through\n1.1.18.8, they could sometimes be other things:</p>\n<pre><code>root=&gt; SELECT TRUE AND 5;      # =&gt; 5\nroot=&gt; SELECT 5 AND TRUE;      # =&gt; 5\nroot=&gt; SELECT 5 AND FALSE;     # =&gt; FALSE\nroot=&gt; SELECT 5 OR TRUE;       # =&gt; TRUE\nroot=&gt; SELECT &#39;cat&#39; OR FALSE;  # =&gt; &#39;cat&#39;\nroot=&gt; SELECT &#39;cat&#39; OR &#39;dog&#39;;\nERROR:  OR requires boolean operands\nroot=&gt; SELECT 1 AND 2;\nERROR:  AND conjunct must produce boolean</code></pre>\n<p>By version 1.1.18.9, ParticleDB rejected boolean expressions over\nnon-boolean literals (<a href=\"https://github.com/particledbai/jepsen/issues/30\" rel=\"nofollow ugc noopener\">#30</a>).</p>\n<h3 id=\"5-1-7-more-boolean-integer-confusion-50\">5.1.7 More Boolean/Integer Confusion (#50)</h3>\n<p>ParticleDB 1.1.18.9 disallowed boolean logic on integer literals, but\nallowed it on columns.</p>\n<pre><code>root=&gt; CREATE TABLE t (a int);\nroot=&gt; UPDATE t SET a = NOT a;\nUPDATE 0</code></pre>\n<p>Likewise, you could do integer operations on booleans:</p>\n<pre><code>root=&gt; CREATE TABLE t (a boolean);\nroot=&gt; UPDATE t SET a = ((a + a) IS NULL)\n       WHERE a;\nUPDATE 0</code></pre>\n<p>Orkes seems to have resolved this issue (<a href=\"https://github.com/particledbai/jepsen/issues/50\" rel=\"nofollow ugc noopener\">#50</a>) by\nversion 1.1.18.11.rc-8.</p>\n<h3 id=\"5-1-8-multiple-assignment-to-a-single-column-5\">5.1.8 Multiple Assignment to a Single Column (#5)</h3>\n<p>The SQL standard specifies that an <code>UPDATE</code> statement\nshall not contain a column name more than once. However, ParticleDB\n1.1.4 through 1.1.18.4 allowed an <code>UPDATE</code> to assign multiple\nvalues to the same column. It seemed to pick the last value,\nlexically.</p>\n<pre><code>root=&gt; CREATE TABLE t (a boolean, b boolean);\nroot=&gt; INSERT INTO t (a, b)\n       VALUES (FALSE, FALSE);\nroot=&gt; UPDATE t SET a = FALSE, a = TRUE;\nUPDATE 1</code></pre>\n<p>In Postgres this returned an error:\n<code>multiple assignments to same column &quot;a&quot;</code>. However,\nParticleDB applied the update, producing:</p>\n<pre><code>root=&gt; select * from t;\n a | b\n---+---\n t | f\n(1 row)</code></pre>\n<p>By version 1.1.18.6, multiple updates to the same column threw an\nerror (<a href=\"https://github.com/particledbai/jepsen/issues/5\" rel=\"nofollow ugc noopener\">#5</a>).</p>\n<h3 id=\"5-1-9-predicates-referring-to-non-existent-columns-10\">5.1.9 Predicates Referring to Non-Existent Columns (#10)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.17, predicates in\n<code>SELECT</code>, <code>UPDATE</code>, and <code>DELETE</code>\nstatements could refer to columns which did not exist. For example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a INT);\nroot=&gt; SELECT * FROM t WHERE b = 1;\n a\n---\n(0 rows)</code></pre>\n<p>In Postgres, expressions like this would throw\n<code>ERROR: column &quot;b&quot; does not exist</code>. By version 1.1.18.6,\nParticleDB did as well (<a href=\"https://github.com/particledbai/jepsen/issues/10\" rel=\"nofollow ugc noopener\">#10</a>).</p>\n<h2 id=\"5-2-arithmetic\">5.2 Arithmetic</h2>\n<p>ParticleDB often performed arithmetic in unexpected ways. These\nissues occurred with a single server, single client, and no faults.</p>\n<h3 id=\"5-2-1-no-large-integer-literals-14\">5.2.1 No Large Integer Literals (#14)</h3>\n<p>Postgres allows large integer literals like\n<code>9223372036854775808</code> (263). However, ParticleDB 1.1.4\nthrough 1.1.17 refused these literals, throwing an\n<code>invalid integer</code> error:</p>\n<pre><code>root=&gt; SELECT 9223372036854775808;\nERROR:  invalid integer: 9223372036854775808</code></pre>\n<p>In ParticleDB 1.1.18.6, one could write large integer literals (<a href=\"https://github.com/particledbai/jepsen/issues/14\" rel=\"nofollow ugc noopener\">#14</a>).</p>\n<h3 id=\"5-2-2-to-infinity-and-beyond-85\">5.2.2 To Infinity, and Beyond (#85)</h3>\n<p>ParticleDB 1.1.18.9 gained an exponentiation operator <code>^</code>,\nbut could not apply it to <code>Infinity</code>.</p>\n<pre><code>SELECT &#39;Infinity&#39; ^ 2;</code></pre>\n<p>This returned <code>&#39;Infinity&#39;</code> in Postgres, but threw\n<code>operator does not exist: ^ requires numeric operands</code> in\nParticleDB. By 1.1.18.11.rc-9, Orkes seems to have resolved this issue\n(<a href=\"https://github.com/particledbai/jepsen/issues/85\" rel=\"nofollow ugc noopener\">#85</a>).</p>\n<h3 id=\"5-2-3-1-0-1-49\">5.2.3 <code>-1 ^ 0 = -1</code> (#49)</h3>\n<p>ParticleDB 1.1.18.9 considered <code>-1 ^ 0</code> to be equal to\n<code>-1</code>, rather than <code>1</code>.</p>\n<pre><code>root=&gt; select -1 ^ 0;\n -1\n----\n -1\n(1 row)</code></pre>\n<p>By version 1.1.18.11.rc-3, it correctly returned <code>1</code> (<a href=\"https://github.com/particledbai/jepsen/issues/49\" rel=\"nofollow ugc noopener\">#49</a>).</p>\n<h3 id=\"5-2-4-1-0-1e-39-1-0-88\">5.2.4 <code>1.0 + 1E-39 = 1.0</code> (#88)</h3>\n<p>ParticleDB 1.1.18.6 gained support for literals using scientific\nnotation—for example, using <code>3E12</code> to denote 3 × 1012. However, the parser\nturned this notation into floating-point values, rather than precise\ndecimals, as in Postgres. This resulted in floating-point errors:</p>\n<pre><code>root=&gt; SELECT 1.0 + 1E-39 = 1.0;\n ?column?\n----------\n t\n(1 row)</code></pre>\n<p>This issue (<a href=\"https://github.com/particledbai/jepsen/issues/88\" rel=\"nofollow ugc noopener\">#88</a>) was\nfixed by 1.1.18.11.rc-9.2.</p>\n<h3 id=\"5-2-5-what-s-the-largest-number-you-can-think-of-56\">5.2.5 What’s the Largest Number You Can Think Of? (#56)</h3>\n<p>ParticleDB 1.1.4 through 1.1.18.11.rc-1 silently converted large\nfloating-point literals to infinities:</p>\n<pre><code>root=&gt; SELECT -4.56E500;\n ?column?\n-----------\n -Infinity\n(1 row)</code></pre>\n<p>By version 1.1.18.11.rc-3, ParticleDB threw an error like\n<code>&quot;4.56E500&quot; is out of range for type double precision</code> (<a href=\"https://github.com/particledbai/jepsen/issues/56\" rel=\"nofollow ugc noopener\">#56</a>).</p>\n<h3 id=\"5-2-6-exponent-underflow-61\">5.2.6 Exponent Underflow (#61)</h3>\n<p>In ParticleDB 1.1.18.9, the exponent operator checked to see if its\narguments would cause an overflow, but did not check for underflow:</p>\n<pre><code>root=&gt; SELECT 200 ^ 200;\nERROR:  value out of range: overflow\nroot=&gt; SELECT -200 ^ -200;\n -0\n----\n  0\n(1 row)</code></pre>\n<p>By version 1.1.18.11.rc-3, ParticleDB threw an underflow error (<a href=\"https://github.com/particledbai/jepsen/issues/61\" rel=\"nofollow ugc noopener\">#61</a>).</p>\n<h3 id=\"5-2-7-division-by-integer-zero-34\">5.2.7 Division by Integer Zero (#34)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.8, you could divide floating-point\nvalues by integer <code>0</code> and receive <code>Infinity</code>,\nrather than a <code>division by zero</code> error, as in Postgres.</p>\n<pre><code>root=&gt; SELECT 1.0 / 0;\n 1 / 0\n-------\n   inf\n(1 row)</code></pre>\n<p>ParticleDB 1.1.18.9 returned an error instead (<a href=\"https://github.com/particledbai/jepsen/issues/34\" rel=\"nofollow ugc noopener\">#34</a>).</p>\n<h3 id=\"5-2-8-modulo-operator-cannot-handle-floats-51\">5.2.8 Modulo Operator Cannot Handle Floats (#51)</h3>\n<p>In ParticleDB 1.1.18.9, the modulo operator <code>%</code> threw an\ninternal error when either argument was a floating-point number.</p>\n<pre><code>root=&gt; SELECT 3.0 % 2;\nERROR:  internal error</code></pre>\n<p>The <code>MOD</code> function was unaffected, and this issue (<a href=\"https://github.com/particledbai/jepsen/issues/51\" rel=\"nofollow ugc noopener\">#51</a>) was\nfixed by 1.1.18.11.rc-1.</p>\n<h3 id=\"5-2-9-broken-nan-arithmetic-36\">5.2.9 Broken <code>NaN</code> Arithmetic (#36)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.8, arithmetic operations on\n<code>NaN</code> would throw an error:</p>\n<pre><code>root=&gt; SELECT &#39;NaN&#39; - 1.0;\nERROR:  arrow compute: Invalid argument error:\nInvalid arithmetic operation: Utf8 - Float64</code></pre>\n<p>By 1.1.18.9, arithmetic on <code>NaN</code> correctly returned\n<code>NaN</code> (<a href=\"https://github.com/particledbai/jepsen/issues/36\" rel=\"nofollow ugc noopener\">#36</a>).</p>\n<h3 id=\"5-2-10-broken-null-arithmetic-92\">5.2.10 Broken <code>NULL</code> Arithmetic (#92)</h3>\n<p>In ParticleDB 1.1.18.11.rc-9.2, arithmetic operations involving\nsufficiently detailed scientific notation literals and <code>NULL</code>\nthrew internal errors:</p>\n<pre><code>root=&gt; CREATE TABLE t (a int);\nroot=&gt; INSERT INTO t (a) VALUES (NULL);\nroot=&gt; SELECT * FROM t WHERE\n       (t.a + 2.0000000000000000E-23) IS NULL;\nERROR:  internal error: called `Option::unwrap()`\n        on a `None` value (at crates/exec/...)</code></pre>\n<p>This error (<a href=\"https://github.com/particledbai/jepsen/issues/92\" rel=\"nofollow ugc noopener\">#92</a>) no\nlonger appeared in version 1.1.18.11.rc-18.</p>\n<h3 id=\"5-2-11-update-can-t-do-math-60\">5.2.11 UPDATE Can’t Do Math (#60)</h3>\n<p>ParticleDB 1.1.18.9 introduced a new bug which rendered it unable to\ndo arithmetic operations on floating-point columns in an\n<code>UPDATE</code> statement.<a href=\"#fn17\">17</a> For example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a real);\nCREATE TABLE\nroot=&gt; INSERT INTO t (a) VALUES (1.2);\nINSERT 0 1\nroot=&gt; UPDATE t SET a = a + 1;\nERROR:  internal error</code></pre>\n<p>In 1.1.18.11.rc-1, ParticleDB was (mostly) able to add numbers (<a href=\"https://github.com/particledbai/jepsen/issues/60\" rel=\"nofollow ugc noopener\">#60</a>).</p>\n<h3 id=\"5-2-12-same-precision-for-real-and-double-precision-32\">5.2.12 Same Precision For <code>real</code> and <code>double precision</code> (#32)</h3>\n<p>Per the SQL standard, the <code>double precision</code> data type\nshould have a greater precision than <code>real</code>. In Postgres,\n<code>real</code> and <code>double precision</code> are (<a href=\"https://www.postgresql.org/docs/8.1/datatype.html\" rel=\"nofollow ugc noopener\">usually</a>) a\n32-bit and 64-bit floating-point value, respectively. In ParticleDB,\nboth appear to be implemented as 64-bit floats.</p>\n<pre><code>CREATE TABLE t (a real NOT NULL);\nINSERT INTO t (a) VALUES (4.134917993754883E8);\nSELECT * FROM t ORDER BY a;\n  ParticleDB:\n    [{:a 4.134917993754883E8}]\n  Postgres:\n    [{:a 4.134918E8}]</code></pre>\n<p>Version 1.1.18.9 seemed to introduce a 32-bit <code>real</code> type\n(<a href=\"https://github.com/particledbai/jepsen/issues/32\" rel=\"nofollow ugc noopener\">#32</a>).</p>\n<h3 id=\"5-2-13-big-integers-11\">5.2.13 Big Integers (#11)</h3>\n<p>Similarly, integer fields (e.g. <code>smallint</code>,\n<code>int</code>, …) in ParticleDB 1.1.4 through 1.1.17 allowed values\nmuch bigger than their typical ranges. It appeared as if ParticleDB\ninternally represented all integer types as signed 64-bit integers. For\nexample, here are four values too big to fit in a Postgres\n<code>smallint</code>:</p>\n<pre><code>root=&gt; CREATE TABLE t (a smallint);\nroot=&gt; INSERT INTO t (a) VALUES (32768);\nroot=&gt; INSERT INTO t (a)\n       VALUES (9223372036854775807);\nroot=&gt; INSERT INTO t (a)\n       VALUES (9223372036854775808);\nroot=&gt; INSERT INTO t (a) VALUES (1.234E40);\nroot=&gt; SELECT * FROM t;\n          a\n---------------------\n               32768\n 9223372036854775807\n                   0\n 9223372036854775807\n(5 rows)</code></pre>\n<p>In Postgres <code>smallint</code> is a 16-bit integer; each\n<code>INSERT</code> would throw\n<code>ERROR: smallint out of range</code>. However, ParticleDB stored\neach of these values with varying results. It preserved 32,768 and 263 − 1 (the largest\ntwo’s-complement signed 64-bit integer). 263 and larger integers were\nsilently converted to <code>0</code>. On the other hand, the large\nfloating-point number was clipped to 263 − 1.</p>\n<p>Similar behavior occurred with <code>int</code> and\n<code>bigint</code>. ParticleDB 1.1.18.6 rejected oversized integers (<a href=\"https://github.com/particledbai/jepsen/issues/11\" rel=\"nofollow ugc noopener\">#11</a>).</p>\n<h3 id=\"5-2-14-dividing-nearby-numbers-86\">5.2.14 Dividing Nearby Numbers (#86)</h3>\n<p>ParticleDB 1.1.18.9 through 1.1.18.11.rc-8 struggled to divide\nfloating-point numbers, often throwing <code>Numeric out of range</code>\nwhen asked to divide values which were well within the range of a 32-bit\nfloat. For example:</p>\n<pre><code>root=&gt; SELECT 81938.0 / 81937.82012987136848;\nERROR:  numeric out of range</code></pre>\n<p>This value should be approximately <code>1.00000219</code>. Orkes\nseems to have resolved this issue (<a href=\"https://github.com/particledbai/jepsen/issues/86\" rel=\"nofollow ugc noopener\">#86</a>) in\nversion 1.1.18.11.rc-9.</p>\n<h3 id=\"5-2-15-floating-point-errors-in-integer-values-87\">5.2.15 Floating-Point Errors in Integer Values (#87)</h3>\n<p>In ParticleDB 1.1.18.11.rc-1 through 1.1.18.11.rc-8, adding small\nnumbers (e.g. <code>0.1</code>) to large integer columns could cause\ntheir values to change by surprising amounts. For example:</p>\n<pre><code>root=&gt; CREATE TABLE t (a bigint);\nroot=&gt; INSERT INTO t (a) VALUES\n       (58950791971326900);\nroot=&gt; UPDATE t SET a = (a + 1E-1);\nroot=&gt; SELECT a FROM t;\n         a\n-------------------\n 58950791971326896</code></pre>\n<p>Adding <code>0.1</code> to this column actually subtracted\n<code>4</code>. Jepsen suspects this bug (<a href=\"https://github.com/particledbai/jepsen/issues/87\" rel=\"nofollow ugc noopener\">#87</a>) was\ncaused by performing these operations on floating-point values (which\ncannot represent all integers) then coercing the result back to the\nnearest integer value. Version 1.1.18.11.rc-9 seems to have fixed\nthis.</p>\n<h3 id=\"5-2-16-floats-are-not-equal-to-quoted-forms-19\">5.2.16 Floats Are Not Equal to Quoted Forms (#19)</h3>\n<p>Postgres considers floating-point numbers like <code>0.0</code> to be\nequal to their quoted representations—this is useful because some\nfloating-point values, like <code>&#39;Infinity&#39;</code>, are written with\nquotes. ParticleDB 1.1.4 through 1.1.18.4 thought otherwise:</p>\n<pre><code>root=&gt; SELECT 0.0 = &#39;0.0&#39;;\n false\n-------\n f\n(1 row)</code></pre>\n<p>This issue (<a href=\"https://github.com/particledbai/jepsen/issues/19\" rel=\"nofollow ugc noopener\">#19</a>)\nappeared resolved as of version 1.1.18.6.</p>\n<h3 id=\"5-2-17-floats-are-not-equal-to-integers-63\">5.2.17 Floats Are Not Equal to Integers (#63)</h3>\n<p>In ParticleDB 1.1.4 through 1.1.18.6, floats were not equal to\nintegers.</p>\n<pre><code>root=&gt; SELECT 0.0 = 0;\n false\n-------\n f\n(1 row)</code></pre>\n<p>By version 1.1.18.8, reasonably-sized integers were equal to their\nfloating-point counterparts (<a href=\"https://github.com/particledbai/jepsen/issues/63\" rel=\"nofollow ugc noopener\">#63</a>).</p>\n<h3 id=\"5-2-18-double-precision-floats-appear-equal-to-almost-all-intege\">5.2.18 Double Precision Floats Appear Equal to Almost All Integers (#80)</h3>\n<p>Version 1.1.18.9 introduced a new bug which made predicates like\n<code>WHERE x != y</code> fail to match for most double precision floats\nin (-1.0, 1.0). It was as if all small floats were equal to all nonzero\nintegers. If we took a table containing a few integers, and performed\nsome simple queries that should return every row…</p>\n<pre><code>CREATE TABLE t (a int);\nINSERT INTO t (a) VALUES (-1), (0), (5);\nSELECT * FROM t WHERE a != (0.1 ^ 25);\nSELECT * FROM t WHERE\na != 0.90000000000000000000000000000000000000;\nSELECT * FROM t WHERE\na != -0.34656654456890453890613489540952456545;</code></pre>\n<p>All three of these queries returned only a single row,\n<code>0</code>—implying that <code>0.90...</code> was equal to\n<code>-1</code> and <code>5</code>. The only exception seemed to be\n<code>WHERE a != 0.0...</code>, which returned no rows at all. This was\nalso wrong, but somewhat less so. Version 1.1.18.11.rc-9 seems to have\nresolved the matter (<a href=\"https://github.com/particledbai/jepsen/issues/80\" rel=\"nofollow ugc noopener\">#80</a>).</p>\n<h2 id=\"5-3-strings\">5.3 Strings</h2>\n<p>We found several problems with strings in ParticleDB, including\ndatatypes that could hold more or fewer characters than they ought, and\nimproper sort orders. These issues occurred with a single server, single\nclient, and no faults.</p>\n<h3 id=\"5-3-1-character-n-allows-more-than-n-characters-64-52\">5.3.1 <code>CHARACTER(n)</code> Allows More Than <code>n</code> Characters (#64, #52)</h3>\n<p>The SQL standard defines a variety of fixed-width string datatypes\nlike <code>character(3)</code>, which holds a string of exactly three\ncharacters. Unfortunately, ParticleDB 1.1.4 through 1.1.18.4 were happy\nto store much larger strings in those columns.<a href=\"#fn18\">18</a></p>\n<pre><code>root=&gt; CREATE TABLE t (a character(3));\nroot=&gt; INSERT INTO t\n       VALUES (&#39;large language model&#39;);\nroot=&gt; SELECT * FROM t;\n          a\n----------------------\n large language model\n(1 row)</code></pre>\n<p>Version 1.1.18.6 rejected inserts and updates of too-long string\nliterals (<a href=\"https://github.com/particledbai/jepsen/issues/64\" rel=\"nofollow ugc noopener\">#64</a>), but\nappeared to perform these checks only at runtime; one could still say\n<code>UPDATE t SET a = &#39;far too long&#39;</code> so long as it matched no\nrows (<a href=\"https://github.com/particledbai/jepsen/issues/52\" rel=\"nofollow ugc noopener\">#52</a>). That\nissue was resolved in version 1.1.18.11.rc-3.</p>\n<h3 id=\"5-3-2-character-n-allows-fewer-than-n-characters-55\">5.3.2 <code>CHARACTER(n)</code> Allows Fewer Than <code>n</code> Characters (#55)</h3>\n<p>While 1.1.18.6 rejected strings that were too large for a fixed-width\nstring column, 1.1.4 through 1.1.18.9 also allowed strings that were too\nshort. Normally, shorter strings are padded with spaces to the specified\nlength, but in ParticleDB, short strings were returned unchanged.</p>\n<pre><code>CREATE TABLE t (a character(16));\nINSERT INTO t VALUES (&#39;smol&#39;);\nSELECT * FROM t;\n  -&gt; {:a &quot;smol&quot;}</code></pre>\n<p>This should have returned <code>&quot;smol &quot;</code>, with\npadding spaces. ParticleDB 1.1.18.11.rc-1 seems to have resolved the\nissue (<a href=\"https://github.com/particledbai/jepsen/issues/55\" rel=\"nofollow ugc noopener\">#55</a>).</p>\n<h3 id=\"5-3-3-default-text-collation-22\">5.3.3 Default Text Collation (#22)</h3>\n<p>On a platform with <code>LC_COLLATE = en_US.UTF-8</code>, Postgres\nuses that collation by default. ParticleDB appeared to collate strings\ndifferently. Sorting via queries like\n<code>SELECT ... ORDER BY name</code>, or comparisons like\n<code>&#39;a&#39; &lt; &#39;B&#39;</code>, returned different results between the two\ndatabases. Where Postgres considered <code>&#39;a&#39; &lt; &#39;B&#39;</code>,\nParticleDB believed the opposite.</p>\n<p>Experiments suggest ParticleDB sorted by Unicode code point order,\nrather than the <a href=\"https://www.unicode.org/reports/tr10/\" rel=\"nofollow ugc noopener\">Unicode\nCollation Algorithm</a> (UCA) with the locale’s collation rules. It may\nhave been compatible with <code>ucs_basic</code>, <code>C</code>, or\n<code>pg_c_utf8</code>. ParticleDB version 1.1.18.9 seemed to choose a\ndefault collation more like UCA (<a href=\"https://github.com/particledbai/jepsen/issues/22\" rel=\"nofollow ugc noopener\">#22</a>).</p>\n<h3 id=\"5-3-4-mis-sorted-text-42-93\">5.3.4 Mis-Sorted Text (#42, #93)</h3>\n<p>Throughout our collaboration ParticleDB struggled to sort things.\nVersions 1.1.4 through 1.1.18.8 accepted an explicit\n<code>unicode</code> text collation, then ignored it:</p>\n<pre><code>root=&gt; CREATE TABLE t (b text COLLATE unicode);\nroot=&gt; INSERT INTO t (b)\n       VALUES (&#39;a&#39;), (&#39;b&#39;), (&#39;B&#39;);\nroot=&gt; SELECT * FROM t ORDER BY b;\n b\n---\n B\n a\n b\n(3 rows)</code></pre>\n<p>In Postgres’ <code>unicode</code> collation, letters like\n<code>a</code> and <code>A</code> sort similarly, despite being of\ndifferent cases. This query would have returned\n<code>&#39;a&#39;, &#39;b&#39;, &#39;B&#39;</code>.</p>\n<p>ParticleDB 1.1.18.9 sorted English uppercase and lowercase letters\ncorrectly, but only for some queries—adding a <code>LIMIT</code> clause\nto a <code>SELECT</code> statement would cause it to revert to the old\nsort order. Version 1.1.18.11.rc-1 resolved that, but sorted other\ncharacters, like punctuation, oddly:</p>\n<pre><code>root=&gt; CREATE TABLE t (a text COLLATE unicode);\nroot=&gt; INSERT INTO t VALUES\n       (&#39;_aa&#39;), (&#39;a&#39;), (&#39;aa&#39;), (&#39;aaa&#39;);\nroot=&gt; SELECT * FROM t ORDER BY a ASC;\n  a\n-----\n a\n _aa\n aa\n aaa\n(4 rows)</code></pre>\n<p>Different sorting of whitespace and punctuation (<a href=\"https://github.com/particledbai/jepsen/issues/42\" rel=\"nofollow ugc noopener\">#42</a>) seemed\nbetter in version 1.1.18.11.rc-7, but reappeared in 1.1.18.11.rc-9\nthrough 1.1.18.11.rc-19 (<a href=\"https://github.com/particledbai/jepsen/issues/93\" rel=\"nofollow ugc noopener\">#93</a>). Text\nsorting seemed improved in 1.1.18.11.rc-21.</p>\n<hr />\n<ol><li><p>There are also a handful of unrelated databases called</p><p> ParticleDB, mainly used in particle physics—for example, the <a href=\"https://software.belle2.org/light-2409-toyger/doxygen/dir_49ab761b69e05c004232ce029feed737.html\" rel=\"nofollow ugc noopener\">database\n of particles</a> used in <a href=\"https://en.wikipedia.org/wiki/Belle_II_experiment\" rel=\"nofollow ugc noopener\">the Belle II\n physics experiment</a>, or <a href=\"https://github.com/MannyMoo/gaudi-script-builder/blob/master/python/GaudiScriptBuilder/ParticleDB.py\" rel=\"nofollow ugc noopener\">this\n one</a> at the Large Hadron Collider.<a href=\"#fnref1\">↩︎</a></p></li><li><p>The Particles feature was not available during our work;</p><p> we focus on ParticleDB’s SQL interfaces.<a href=\"#fnref2\">↩︎</a></p></li><li><p>One envisions a world of meta-recursive database</p><p> documentation in the style of <a href=\"https://raymondsmullyan.com/\" rel=\"nofollow ugc noopener\">Raymond Smullyan</a>. “One page of\n the documentation always tells the truth. Another tells only lies…”<a href=\"#fnref3\">↩︎</a></p></li><li><p>Presumably [Two-Phase</p><p> Locking](<a href=\"https://en.wikipedia.org/wiki/Two-phase_locking\" rel=\"nofollow ugc noopener\">https://en.wikipedia.org/wiki/Two-phase_locking</a>).<a href=\"#fnref4\">↩︎</a></p></li><li><p><em>With apologies to Deleuze &amp; Guattari</em>—Thus,</p><p> the tuple without fields is opposed less to fields as such than to the\n organization of the fields insofar as it composes a row. *Le n-uplet\n sans composants* is not a deleted tuple but a living tuple all the\n more alive and teeming once it has blown apart the client and its memory\n model. Structs commingling on the heap. <a href=\"https://github.com/spencertipping/writing-self-modifying-perl/blob/c25d7374dccac01413725acea4d0776badaf9233/writing-self-modifying-perl.pdf\" rel=\"nofollow ugc noopener\">Self-modifying\n Perl</a>. The full tuple without fields is a tuple populated by\n multiplicities.<a href=\"#fnref5\">↩︎</a></p></li><li>Madred, 2369.<a href=\"#fnref6\">↩︎</a></li><li><a href=\"https://www.youtube.com/watch?v=kfPwJKz_rlo\" rel=\"nofollow ugc noopener\">Cameron, 1986</a>.<a href=\"#fnref7\">↩︎</a></li><li><p>He explained that the Database—the Denormalized, the</p><p> Data Lakehouse, the giant Monad—is a table without tuples. *Le\n tableau sans n-uplets* is permeated by unformed, unstable values, by\n dataflows in all directions, by free JSON or nomadic singletons, by mad\n or transitory transactions. That, however, was not the question at hand.\n For there simultaneously occurs upon the database a very important,\n inevitable phenomenon that is beneficial in many respects and\n unfortunate in many others: schematization. Schemata are Layers, Belts.\n They consist of giving form to values, of imprisoning arrays or locking\n singletons into systems of record and replication, of producing upon the\n body of the database records large and small and organizing them into\n normal aggregates. Schemata are acts of capture, they are like “black\n holes” or occlusions striving to seize whatever comes within their\n reach. They operate by coding and normalization upon the database; they\n proceed simultaneously by code and normality. The Schemata are\n judgements of God; schematization in general is the entire system of the\n judgement of God (but the database, or the table without tuples,\n constantly eludes that judgement, flees and becomes deschematized,\n destructured, denormalized).<a href=\"#fnref8\">↩︎</a></p></li><li><p>To make matters more confusing,</p><p> <code>serializable</code> in versions 1.1.4 through 1.1.18.11.rc-11.2\n was (depending on which version of the binary you asked) either an alias\n for <code>table-2pl</code> or possibly <code>occ-v3</code>. In\n 1.1.18.11.rc-22, <code>serializable</code> reappeared as a new Strong\n Serializable transaction mode, also based on Serializable Snapshot\n Isolation (SSI), but perhaps not the <code>serializable-ssi</code> level\n present in earlier builds, which was distinct from\n <code>strict-serializable</code>.<a href=\"#fnref9\">↩︎</a></p></li><li>De Lioncourt, 1793, presaging Borges, 1941.<a href=\"#fnref10\">↩︎</a></li><li><p>See ANSI/NISO Z39.19-2005 (R2010) “Celestial Emporium</p><p>  of Benevolent Knowledge” for the full taxonomy of database records: rows\n  belonging to the DBA, tombstones, those which from afar look like JSON,\n  et cetera, etc.<a href=\"#fnref11\">↩︎</a></p></li><li><p>Jepsen encourages the consistency model research</p><p>  community to come up with more official terms for this.<a href=\"#fnref12\">↩︎</a></p></li><li><p>Running <code>particledb --version</code> might not</p><p>  return the actual version. For instance, 1.1.16 through 1.1.18.11.rc-18\n  all identified themselves as “spanner 1.1.16”.<a href=\"#fnref13\">↩︎</a></p></li><li><p>Barthes can rejoice: not only is the author dead, but</p><p>  they were never alive to begin with. Nothing is true, everything is\n  permitted.<a href=\"#fnref14\">↩︎</a></p></li><li><p>MongoDB had [962</p><p>  employees in 2018](<a href=\"https://www.sec.gov/Archives/edgar/data/1441816/000144181618000028/mdb-013118x10k.htm\" rel=\"nofollow ugc noopener\">https://www.sec.gov/Archives/edgar/data/1441816/000144181618000028/mdb-013118x10k.htm</a>), and Cockroach Labs had <a href=\"https://www.cockroachlabs.com/blog/why-join-cockroach-labs/\" rel=\"nofollow ugc noopener\">400\n  in 2022</a>. Databases are hard↩︎</p></li><li><p>Or perhaps not. Not every application requires perfect</p><p>  correctness. Vibrational Database Management Systems might be fine, at\n  least some of the time.<a href=\"#fnref16\">↩︎</a></p></li><li><p>[Mattel,</p><p>  1992](<a href=\"http://itre.cis.upenn.edu/~myl/languagelog/archives/002892.html\" rel=\"nofollow ugc noopener\">http://itre.cis.upenn.edu/~myl/languagelog/archives/002892.html</a>).<a href=\"#fnref17\">↩︎</a></p></li><li>Danielewski, 2000.<a href=\"#fnref18\">↩︎</a></li></ol>","headings":[{"level":1,"text":"Jepsen: ParticleDB 1.1.4","id":"jepsen-particledb-1-1-4"},{"level":1,"text":"1 Background","id":"1-background"},{"level":2,"text":"1.1 Safety","id":"1-1-safety"},{"level":1,"text":"2 Test Design","id":"2-test-design"},{"level":2,"text":"2.1 Workloads","id":"2-1-workloads"},{"level":1,"text":"3 Results","id":"3-results"},{"level":2,"text":"3.1 Availability and Performance","id":"3-1-availability-and-performance"},{"level":3,"text":"3.1.1 Abysmal Performance With table-2pl (#62)","id":"3-1-1-abysmal-performance-with-table-2pl-62"},{"level":3,"text":"3.1.2 Mystery Crash (#81)","id":"3-1-2-mystery-crash-81"},{"level":3,"text":"3.1.3 Deadlock After Two Transactions (#27)","id":"3-1-3-deadlock-after-two-transactions-27"},{"level":3,"text":"3.1.4 Deadlock With table-2pl (#71)","id":"3-1-4-deadlock-with-table-2pl-71"},{"level":2,"text":"3.2 Broken Features","id":"3-2-broken-features"},{"level":3,"text":"3.2.1 No Aggregates Within CASE (#20)","id":"3-2-1-no-aggregates-within-case-20"},{"level":3,"text":"3.2.2 Unsupported Exponentiation (#28)","id":"3-2-2-unsupported-exponentiation-28"},{"level":3,"text":"3.2.3 Malformed Responses for Prepared Statements (#2)","id":"3-2-3-malformed-responses-for-prepared-statements-2"},{"level":3,"text":"3.2.4 Most Postgres Queries Fail (#83)","id":"3-2-4-most-postgres-queries-fail-83"},{"level":3,"text":"3.2.5 Tuples Without Fields (#111)","id":"3-2-5-tuples-without-fields-111"},{"level":3,"text":"3.2.6 Broken Postgres Authentication (#1)","id":"3-2-6-broken-postgres-authentication-1"},{"level":3,"text":"3.2.7 Java Client Could Not Execute Transactions (#3)","id":"3-2-7-java-client-could-not-execute-transactions-3"},{"level":2,"text":"3.3 Predicates","id":"3-3-predicates"},{"level":3,"text":"3.3.1 SELECT Ignores WHERE Clause (#40)","id":"3-3-1-select-ignores-where-clause-40"},{"level":3,"text":"3.3.2 WHERE 1 < (a + 2) (#47)","id":"3-3-2-where-1-a-2-47"},{"level":3,"text":"3.3.3 WHERE 4 = 5 (#18)","id":"3-3-3-where-4-5-18"},{"level":3,"text":"3.3.4 SELECT in a Transaction Returns Nonsense (#35)","id":"3-3-4-select-in-a-transaction-returns-nonsense-35"},{"level":2,"text":"3.4 Constraints","id":"3-4-constraints"},{"level":3,"text":"3.4.1 Not-Null Constraints Allow Null (#8)","id":"3-4-1-not-null-constraints-allow-null-8"},{"level":3,"text":"3.4.2 Unique Constraints Allow Duplicates (#9)","id":"3-4-2-unique-constraints-allow-duplicates-9"},{"level":3,"text":"3.4.3 Duplicate Primary Keys (#29)","id":"3-4-3-duplicate-primary-keys-29"},{"level":3,"text":"3.4.4 Null Primary Keys (#57)","id":"3-4-4-null-primary-keys-57"},{"level":3,"text":"3.4.5 Primary Key Conflicts with Deleted Rows (#82)","id":"3-4-5-primary-key-conflicts-with-deleted-rows-82"},{"level":3,"text":"3.4.6 Lack of Primary Key Conflicts with Extant Rows (#90)","id":"3-4-6-lack-of-primary-key-conflicts-with-extant-rows-90"},{"level":2,"text":"3.5 Joins","id":"3-5-joins"},{"level":3,"text":"3.5.1 Can’t Join Integers to Floats (#58)","id":"3-5-1-can-t-join-integers-to-floats-58"},{"level":3,"text":"3.5.2 Incorrect Column Names (#54)","id":"3-5-2-incorrect-column-names-54"},{"level":3,"text":"3.5.3 Coercing to Strings (#76)","id":"3-5-3-coercing-to-strings-76"},{"level":3,"text":"3.5.4 Missing Rows (#59)","id":"3-5-4-missing-rows-59"},{"level":3,"text":"3.5.5 Column Confusion (#77)","id":"3-5-5-column-confusion-77"},{"level":2,"text":"3.6 Zombies","id":"3-6-zombies"},{"level":3,"text":"3.6.1 Total Recall (#26)","id":"3-6-1-total-recall-26"},{"level":3,"text":"3.6.2 Memento Delere (#45)","id":"3-6-2-memento-delere-45"},{"level":3,"text":"3.6.3 Night of the Living Integers (#23)","id":"3-6-3-night-of-the-living-integers-23"},{"level":3,"text":"3.6.4 Team Rocket (#84)","id":"3-6-4-team-rocket-84"},{"level":3,"text":"3.6.5 A Ghost is Technically Not a Number (#78)","id":"3-6-5-a-ghost-is-technically-not-a-number-78"},{"level":3,"text":"3.6.6 Beached Strings (#74)","id":"3-6-6-beached-strings-74"},{"level":3,"text":"3.6.7 Pet Sematary (#46)","id":"3-6-7-pet-sematary-46"},{"level":2,"text":"3.7 State Corruption","id":"3-7-state-corruption"},{"level":3,"text":"3.7.1 Unreadable Infinities (#15)","id":"3-7-1-unreadable-infinities-15"},{"level":3,"text":"3.7.2 NaN Sometimes Converted to 0.0 (#21)","id":"3-7-2-nan-sometimes-converted-to-0-0-21"},{"level":3,"text":"3.7.3 Assignment of Incompatible Types (#7)","id":"3-7-3-assignment-of-incompatible-types-7"},{"level":3,"text":"3.7.4 Insert Column Order Table Corruption (#16)","id":"3-7-4-insert-column-order-table-corruption-16"},{"level":3,"text":"3.7.5 Updates Can Insert Corrupt Rows (#24, #25)","id":"3-7-5-updates-can-insert-corrupt-rows-24-25"},{"level":3,"text":"3.7.6 Sometimes, Strings That Are Extensive… Are Cursed (#37)","id":"3-7-6-sometimes-strings-that-are-extensive-are-cursed-37"},{"level":3,"text":"3.7.7 Eternal Sunshine of the Spotless Table (#72)","id":"3-7-7-eternal-sunshine-of-the-spotless-table-72"},{"level":2,"text":"3.8 Transactions and Concurrency","id":"3-8-transactions-and-concurrency"},{"level":3,"text":"3.8.1 Nested Transactions Destroy Ongoing Ones (#44)","id":"3-8-1-nested-transactions-destroy-ongoing-ones-44"},{"level":3,"text":"3.8.2 Aborted Transactions Can Commit (#43)","id":"3-8-2-aborted-transactions-can-commit-43"},{"level":3,"text":"3.8.3 Aggregates Sometimes Fail to Observe a Transaction’s Rows (#48)","id":"3-8-3-aggregates-sometimes-fail-to-observe-a-transaction-s-rows-"},{"level":3,"text":"3.8.4 Double Trouble (#89)","id":"3-8-4-double-trouble-89"},{"level":3,"text":"3.8.5 Updates to Zero Values Sometimes Fail Silently (#91)","id":"3-8-5-updates-to-zero-values-sometimes-fail-silently-91"},{"level":3,"text":"3.8.6 Tables With a TEXT Primary Key Are Mostly Invisible (#41)","id":"3-8-6-tables-with-a-text-primary-key-are-mostly-invisible-41"},{"level":3,"text":"3.8.7 A Thousand Tableaus (#38, #66, #68, #69)","id":"3-8-7-a-thousand-tableaus-38-66-68-69"},{"level":3,"text":"3.8.8 G-Single (#73)","id":"3-8-8-g-single-73"},{"level":3,"text":"3.8.9 Brazil (#67)","id":"3-8-9-brazil-67"},{"level":3,"text":"3.8.10 Lost Writes With table-2pl (#70)","id":"3-8-10-lost-writes-with-table-2pl-70"},{"level":3,"text":"3.8.11 Internal Anomalies With occ-v3 (#96)","id":"3-8-11-internal-anomalies-with-occ-v3-96"},{"level":3,"text":"3.8.12 There is No Anti-Memetics Division (#65)","id":"3-8-12-there-is-no-anti-memetics-division-65"},{"level":3,"text":"3.8.13 Six Data Blockers and a Large Iced Coffee (#31)","id":"3-8-13-six-data-blockers-and-a-large-iced-coffee-31"},{"level":3,"text":"3.8.14 Read Skew With table-2pl (#106)","id":"3-8-14-read-skew-with-table-2pl-106"},{"level":3,"text":"3.8.15 Stale Read With strict-serializable (#109)","id":"3-8-15-stale-read-with-strict-serializable-109"},{"level":3,"text":"3.8.16 Lost Writes (#110)","id":"3-8-16-lost-writes-110"},{"level":2,"text":"3.9 With Faults","id":"3-9-with-faults"},{"level":3,"text":"3.9.1 Data Loss on WAL Bitflips (#108)","id":"3-9-1-data-loss-on-wal-bitflips-108"},{"level":3,"text":"3.9.2 Total Data Loss With Simulated Misdirected Reads/Writes (#112)","id":"3-9-2-total-data-loss-with-simulated-misdirected-reads-writes-11"},{"level":3,"text":"3.9.3 Internal Anomalies With strict-serializable and Process Kills (#107)","id":"3-9-3-internal-anomalies-with-strict-serializable-and-process-ki"},{"level":1,"text":"4 Discussion","id":"4-discussion"},{"level":2,"text":"4.1 Impact","id":"4-1-impact"},{"level":2,"text":"4.2 The Unreasonable Effectiveness of Generative Testing","id":"4-2-the-unreasonable-effectiveness-of-generative-testing"},{"level":2,"text":"4.3 Quality Assurance","id":"4-3-quality-assurance"},{"level":2,"text":"4.4 Documentation","id":"4-4-documentation"},{"level":2,"text":"4.5 LLM Tradeoffs","id":"4-5-llm-tradeoffs"},{"level":2,"text":"4.6 Future Work","id":"4-6-future-work"},{"level":1,"text":"5 Appendix A","id":"5-appendix-a"},{"level":2,"text":"5.1 Type Safety and Syntax","id":"5-1-type-safety-and-syntax"},{"level":3,"text":"5.1.1 Ordering By an Unused Column (#33)","id":"5-1-1-ordering-by-an-unused-column-33"},{"level":3,"text":"5.1.2 Comparison of Incomparable Types (#6)","id":"5-1-2-comparison-of-incomparable-types-6"},{"level":3,"text":"5.1.3 Cannot Compare Decimals to Floats (#94)","id":"5-1-3-cannot-compare-decimals-to-floats-94"},{"level":3,"text":"5.1.4 Filter Predicates Reject NULL (#12)","id":"5-1-4-filter-predicates-reject-null-12"},{"level":3,"text":"5.1.5 Cannot Compare Values to NULL (#13)","id":"5-1-5-cannot-compare-values-to-null-13"},{"level":3,"text":"5.1.6 Boolean Logic on Integers (#30)","id":"5-1-6-boolean-logic-on-integers-30"},{"level":3,"text":"5.1.7 More Boolean/Integer Confusion (#50)","id":"5-1-7-more-boolean-integer-confusion-50"},{"level":3,"text":"5.1.8 Multiple Assignment to a Single Column (#5)","id":"5-1-8-multiple-assignment-to-a-single-column-5"},{"level":3,"text":"5.1.9 Predicates Referring to Non-Existent Columns (#10)","id":"5-1-9-predicates-referring-to-non-existent-columns-10"},{"level":2,"text":"5.2 Arithmetic","id":"5-2-arithmetic"},{"level":3,"text":"5.2.1 No Large Integer Literals (#14)","id":"5-2-1-no-large-integer-literals-14"},{"level":3,"text":"5.2.2 To Infinity, and Beyond (#85)","id":"5-2-2-to-infinity-and-beyond-85"},{"level":3,"text":"5.2.3 -1 ^ 0 = -1 (#49)","id":"5-2-3-1-0-1-49"},{"level":3,"text":"5.2.4 1.0 + 1E-39 = 1.0 (#88)","id":"5-2-4-1-0-1e-39-1-0-88"},{"level":3,"text":"5.2.5 What’s the Largest Number You Can Think Of? (#56)","id":"5-2-5-what-s-the-largest-number-you-can-think-of-56"},{"level":3,"text":"5.2.6 Exponent Underflow (#61)","id":"5-2-6-exponent-underflow-61"},{"level":3,"text":"5.2.7 Division by Integer Zero (#34)","id":"5-2-7-division-by-integer-zero-34"},{"level":3,"text":"5.2.8 Modulo Operator Cannot Handle Floats (#51)","id":"5-2-8-modulo-operator-cannot-handle-floats-51"},{"level":3,"text":"5.2.9 Broken NaN Arithmetic (#36)","id":"5-2-9-broken-nan-arithmetic-36"},{"level":3,"text":"5.2.10 Broken NULL Arithmetic (#92)","id":"5-2-10-broken-null-arithmetic-92"},{"level":3,"text":"5.2.11 UPDATE Can’t Do Math (#60)","id":"5-2-11-update-can-t-do-math-60"},{"level":3,"text":"5.2.12 Same Precision For real and double precision (#32)","id":"5-2-12-same-precision-for-real-and-double-precision-32"},{"level":3,"text":"5.2.13 Big Integers (#11)","id":"5-2-13-big-integers-11"},{"level":3,"text":"5.2.14 Dividing Nearby Numbers (#86)","id":"5-2-14-dividing-nearby-numbers-86"},{"level":3,"text":"5.2.15 Floating-Point Errors in Integer Values (#87)","id":"5-2-15-floating-point-errors-in-integer-values-87"},{"level":3,"text":"5.2.16 Floats Are Not Equal to Quoted Forms (#19)","id":"5-2-16-floats-are-not-equal-to-quoted-forms-19"},{"level":3,"text":"5.2.17 Floats Are Not Equal to Integers (#63)","id":"5-2-17-floats-are-not-equal-to-integers-63"},{"level":3,"text":"5.2.18 Double Precision Floats Appear Equal to Almost All Integers (#80)","id":"5-2-18-double-precision-floats-appear-equal-to-almost-all-intege"},{"level":2,"text":"5.3 Strings","id":"5-3-strings"},{"level":3,"text":"5.3.1 CHARACTER(n) Allows More Than n Characters (#64, #52)","id":"5-3-1-character-n-allows-more-than-n-characters-64-52"},{"level":3,"text":"5.3.2 CHARACTER(n) Allows Fewer Than n Characters (#55)","id":"5-3-2-character-n-allows-fewer-than-n-characters-55"},{"level":3,"text":"5.3.3 Default Text Collation (#22)","id":"5-3-3-default-text-collation-22"},{"level":3,"text":"5.3.4 Mis-Sorted Text (#42, #93)","id":"5-3-4-mis-sorted-text-42-93"}]}}