{"article":{"slug":"jepsen-orkes-conductor-5-3-2","title":"Jepsen: Orkes Conductor 5.3.2","subtitle":null,"summary":"Kyle Kingsbury's Jepsen analysis of the Orkes Conductor workflow orchestrator (versions 5.3.2 to 5.5.0-rc2) found frequent concurrent execution of individual tasks, a delayed-execution issue, four minor startWorkflow errors, and three serious bugs that abandoned, forgot or falsely completed workflow executions. Orkes 5.5.6 patches all serious safety issues. Funded by Orkes.","content_type":"research","language":"en","canonical_url":"https://jepsen.io/analyses/orkes-conductor-5.3.2","author":{"name":"Kyle Kingsbury","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Jepsen","url":"https://jepsen.io/","listing_slug":null,"listing":null},"topics":[{"name":"Distributed Systems","slug":"distributed-systems","url":"https://listedarticles.com/topics/distributed-systems"},{"name":"Databases","slug":"databases","url":"https://listedarticles.com/topics/databases"},{"name":"Testing","slug":"testing","url":"https://listedarticles.com/topics/testing"},{"name":"Workflow Orchestration","slug":"workflow-orchestration","url":"https://listedarticles.com/topics/workflow-orchestration"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":5043,"reading_minutes":22,"published_at":"2026-10-20T00:00:00.000Z","added_at":"2026-10-20T14:07:45.285Z","updated_at":"2026-10-20T14:07:45.285Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/jepsen-orkes-conductor-5-3-2","markdown_url":"https://listedarticles.com/articles/jepsen-orkes-conductor-5-3-2.md","example":false,"citation":"Kyle Kingsbury, Jepsen. \"Jepsen: Orkes Conductor 5.3.2.\" 20 Oct 2026. https://jepsen.io/analyses/orkes-conductor-5.3.2 (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://jepsen.io/analyses/orkes-conductor-5.3.2"},"body_markdown":"# Jepsen: Orkes Conductor 5.3.2\n\nKyle Kingsbury | Jepsen | 2026-10-20\n\n[Orkes Conductor](https://orkes.io/platform) is a\nworkflow orchestration platform which runs user-defined tasks and\nroutes data between them. We tested Orkes Conductor versions 5.3.2\nthrough 5.5.0-rc2 and found unexpectedly frequent concurrent\nexecution of individual tasks, one issue that could significantly\ndelay workflow execution, four minor issues with\n`startWorkflow` errors or null IDs, and three serious\nproblems which caused Conductor to permanently abandon or forget\nworkflow executions, or to mark them completed even though their\ntasks were never run. Orkes has released version 5.5.6 with patches\nfor all serious safety issues, as well as reducing concurrent\nexecution and fixing two `startWorkflow` errors. The\nother two `startWorkflow` errors are under investigation.\nDelayed execution is consistent with Conductor’s design parameters\nand need not be addressed. Orkes has published [a\ncompanion blog post](https://orkes.io/blog/orkes-conductor-jepsen-testing) to this work. This research was funded by\nOrkes Inc., and conducted in accordance with the [Jepsen ethics\npolicy](https://jepsen.io/analyses/ethics).\n\n# 1 Background\n\nA [workflow\norchestrator](https://github.com/resources/articles/what-is-workflow-orchestration) is a software system which executes computations made\nup of small *tasks*. Tasks are connected in a *workflow*,\nwhich routes the outputs of some tasks to the inputs of others,\ndetermines which tasks may be scheduled and when, and so on. The\norchestrator manages workflows and specific executions of them, runs\ntheir tasks, persists state, and tries to drive each execution to\ncompletion.\n\n[Conductor](https://orkes.io/platform) is a workflow\nsystem [originally built\nby Netflix](https://github.com/netflix/conductor) and open sourced in 2016.[1](#fn1) The\nopen source version, [Conductor OSS](https://github.com/conductor-oss/conductor), is\nnow maintained by the [Conductor OSS\nFoundation](https://conductor-oss.org/). This foundation is led by [Orkes](https://orkes.io/blog/differences-between-conductor-oss-vs-orkes-conductor),\nwhose founders helped build Conductor at Netflix. Orkes also offers a [hosted version of Conductor](https://orkes.io/platform) with\nadditional features, called *Orkes Conductor*. This analysis\nfocuses on Orkes Conductor, which we call *Conductor* for\nshort.\n\nIn Conductor, a *workflow definition* is a JSON structure that\ndefines which tasks must be executed, their order, what their inputs and\noutputs should be, and so on. Each workflow definition is identified by\na name and a version. For example, a workflow definition could specify\nthat to adopt a cat, one must first complete an application, then\nschedule a vet visit, and receive vaccination records from the vet. A\nworkflow may also accept an input (say, the name of the cat) and produce\nan output (like a certificate of adoption).\n\nA workflow definition may be run many times—say, once for each cat to\nbe adopted. Each instance is called a *workflow execution*, but\nwe often call an execution a “workflow” for short. Each execution is\nidentified by a unique *workflow ID*. During an execution a task\nmay also be attempted multiple times (e.g. in case of failure), and each\nattempt receives a unique *execution ID*.\n\nSome tasks are built into Conductor. Some are *system tasks*,\nlike making HTTP requests or submitting events to a log like Kafka.\nOthers are called *operators*—these provide control flow like\nconditionals, fork/join, loops, and so on. Operators may execute\nECMAScript code to perform user-defined logic. Finally there are\n*worker tasks*, which are written by users to perform the\n“business logic” of the system.\n\nTo start an execution of a workflow, one can make an API call to\nConductor, submit an event through a message broker, or use Conductor’s\nbuilt-in `cron`-style scheduler. Once acknowledged, Conductor\nschedules the workflow’s tasks and attempts to drive it to\ncompletion.\n\n## 1.1 Architecture\n\nConductor [has\nfive major components](https://orkes.io/academy/learn/56cf6301-6257-4bf3-8df8-513cc752a47d). Conductor servers are stateless processes\nwhich manage workflows and schedule execution. They provide an API and\nuser interface, interact with external services, and track state. The\ndefinitions of workflows, the state of their executions and the results\nof tasks are stored in a database like Postgres, Cassandra, or Redis.\nConductor also uses a queue—often Redis—to dispatch work to various\nprocesses, and to store short-lived state. Finally, there are two pools\nof stateless workers: standard workers which execute Conductor’s\nbuilt-in tasks, and custom workers, which are written by users using [SDKs for various\nlanguages](https://github.com/conductor-oss#conductor-sdks). All workers [connect to a\nConductor server](https://orkes.io/content/conductor-architecture), poll the server for tasks, execute them, and send\nresults back to the server.\n\nEach component may be scaled independently. Any number of servers and\nworkers can run concurrently, and nodes may be killed and restarted at\nany time. In version 5.3.2, the database and queue were both required to\npreserve all acknowledged writes. However, version 5.4.0 introduced a\nnew option which enabled the use of a lossy storage system for all\nqueues except for the main *decider queue*.\n\n## 1.2 Guarantees\n\nConductor guarantees [“at\nleast once message delivery”](https://orkes.io/content/error-handling#message-delivery-guarantees), ensuring that all messages are\npersistent and will be delivered to task workers one or more times.\nConductor also guarantees that workflows eventually terminate. “If a\nworkflow has started, it will run to completion as long as all its tasks\nare completed.” This does not necessarily mean that workflows and tasks\nexecute successfully. [Tasks\ncan run out of retry attempts](https://orkes.io/content/developer-guides/task-and-workflow-status-in-conductor), fail with non-retryable errors, or be\nexplicitly canceled—and a similar state machine applies to workflow\nexecutions. However, workflows should eventually converge on a [terminal\nstatus](https://github.com/conductor-oss/conductor/blob/411a800f2f29af2bc3af112c3b1b0ecfc6582b46/common/src/main/java/com/netflix/conductor/common/run/Workflow.java#L34-L40): `COMPLETED`, `FAILED`,\n`TIMED_OUT`, or `TERMINATED`.\n\nWhen tasks are executed, each attempt receives a unique execution ID,\nand also a unique *retry count*, which [indicates\nthe number of attempts already performed](https://orkes.io/content/sdks/golang#retry-count-and-logs).\n\nOrkes says that for each workflow execution, each instance of a task\nhas at most one visible successful execution. Tasks may execute\nsuccessfully multiple times—for instance, if a Conductor server loses\nits connection to the database just before persisting the completion of\nthe task execution. However, once the output of a successful task\nbecomes visible, no task, workflow, or API call will observe a different\noutput.[2](#fn2)\n\n# 2 Test Design\n\nTo verify these properties, we built a [test harness](https://github.com/jepsen-io/conductor) for Orkes\nConductor using the [Jepsen\ntesting library](https://github.com/jepsen-io/jepsen) and the [Conductor Java SDK](https://github.com/conductor-oss/java-sdk)\nat version 5.0.1. We used this harness to test Orkes Conductor 5.3.2,\n5.4.0, and 5.5.0-rc1 and -rc2.\n\nOrkes typically deploys Conductor in cloud environments with Postgres\n(or a Postgres-compatible database, like their in-house [ParticleDB](https://particledb.ai/)) as the primary database,\nand Redis (or a compatible service like [Valkey](https://aws.amazon.com/elasticache/what-is-valkey/)) as\nthe queue. To approximate these deployments, we set up clusters of ten\nDebian machines, both in containers and as separate EC2 VMs: a single\nPostgres node for the database, either a single Redis node or Redis\nCluster for the queue, and the remaining nodes split evenly between\nConductor servers, Conductor’s standard workers, and a [custom\nworker](https://github.com/jepsen-io/conductor/tree/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/worker) we wrote using the Java SDK.\n\nOur [test\nworkload](https://github.com/jepsen-io/conductor/blob/main/src/jepsen/conductor/workload.clj) defined a single workflow with one task, handled by our\ncustom worker. The task slept up to 30 seconds to simulate work being\ndone, then [returned\na randomly generated output](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/worker/src/jepsen/conductor/worker.clj#L104). It also [logged\nits execution](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/worker/src/jepsen/conductor/worker.clj#L106-L114) with the test harness, so we could tell exactly which\ntasks had run and when, along with their observed inputs and intended\noutputs.\n\nOur test proceeded in three phases. During the [main\nphase](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/generator.clj#L276-L299), the test harness called [`startWorkflow()`](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/client.clj#L157-L170)\nwith unique inputs (to distinguish executions from one another), [kept\ntrack](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/generator.clj#L36-L88) of which executions were in-progress and which were terminal,\nand checked the status of specific executions using\n`getWorkflow()`. To discover executions when the call to\n`startWorkflow()` timed out or failed, we periodically called\n[`getRunningWorkflow()`](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/client.clj#L185-L189)\nto list all in-progress executions. During the main phase we also\ninjected [a\nvariety of faults](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/nemesis.clj), including pausing and killing processes, as well\nas network partitions between Conductor servers and Postgres, Redis, and\nworker nodes.\n\nDuring the [recovery\nphase](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/generator.clj#L301-L307), the test ended network partitions, restarted and unpaused\nprocesses, etc., then let the cluster recover. To figure out when the\nsystem had converged on a steady state, we polled the depth of\nConductor’s decider queue. When this queue reached zero, Conductor had\nnothing more to do and the test moved to the final phase.\n\nDuring the [final\nphase](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/generator.clj#L309-L316), the test read the state of every workflow execution it had\never seen. We then searched for [a\nvariety of unexpected behaviors](https://github.com/jepsen-io/conductor/blob/1e9860de4c3ae063dfcb1017a98b1f5c657e4302/src/jepsen/conductor/checker/workflow.clj#L985-L1029):\n\nBackend Error:\n: Some request returned an unexpected backend error.\n\nConcurrent Execution:\n: Two or more workers attempted to execute the same task in a single\n workflow execution at the same time.\n\nDifferent Task Input:\n: Two views of the same task in the same workflow execution had different\n inputs, or a task’s input was inconsistent with the workflow’s input.\n\nDifferent Workflow Input:\n: A single workflow execution had two different inputs.\n\nDifferent Workflow Output:\n: A single workflow execution had two different outputs.\n\nDuplicate Workflow ID:\n: Two calls to `startWorkflow` returned the same workflow\n execution ID.\n\nG1a:\n: A `startWorkflow` request returned what appeared to be a\n definite failure, but a workflow with that requested input was later\n visible to some reader.\n\nImpossible Output:\n: A workflow execution’s output was inconsistent with the outputs of its\n tasks.\n\nIncomplete:\n: A workflow execution was not in state `COMPLETED` during the\n final phase.\n\nMissing:\n: A read failed to observe a workflow execution after we knew it existed.\n\nNever Started:\n: A workflow came into existence with no corresponding\n `startWorkflow` request.\n\nNull Workflow ID:\n: A `startWorkflow` call returned `null`, rather\n than a workflow ID string.\n\nTask Never Executed:\n: None of a workflow’s tasks were ever executed.\n\n# 3 Results\n\n## 3.1 Tasks Execute Concurrently (#1)\n\nUnder normal operation, healthy Conductor clusters occasionally\nexecuted the same task for the same workflow execution on two workers\nconcurrently. This is not surprising, since [mutual exclusion\nis impossible in asynchronous networks](https://inria.hal.science/hal-03347878/document). However, it is worth keeping\nin mind that even very short-lived tasks may be executed at the same\ntime! In [this\ntest run](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260702T224204-concurrent-execution.zip), workflow\n`3919784b-7691-11f1-a292-00163e983f22`’s single task began on\nworker `n9` at 70.110 seconds, then on worker\n`n10` at 70.113 seconds. It completed on `n9` at\n70.114 seconds, and on `n10` less than a millisecond\nlater.\n\nThe documentation for Conductor OSS describes a distributed locking\nsystem which “prevents race conditions when multiple server instances\nevaluate the same workflow concurrently”. Orkes Conductor also has a\nlocking mechanism whose lease time is configured by\n`conductor.workflow.lockLeaseTime`. While this lock can\nreduce the frequency of concurrent execution, it cannot prevent it\nentirely.\n\nWith our single-task workflows, the default lease time of one minute,\ntwo Conductor servers, three workers, near-zero network latency,\nshort-lived tasks (p50 ~500 µs, p95 ~5 ms, p99 ~10 ms), and roughly\n1,100 workflows per second, we observed concurrent execution roughly\nonce every six hours in healthy clusters. Reducing the lease time to one\nsecond allowed concurrent execution every few minutes. Production\nfrequency may vary.\n\nThis problem was exacerbated by a [time\nof check to time of use](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use) bug in the sweeper process which re-enqueues\nstuck tasks. Imagine a sweeper reads a task in state\n`SCHEDULED`. A worker then executes that task, changing its\nstate in the database to `IN_PROGRESS` and removing the task\nmessage from the queue. The sweeper then checks the queue and finds no\nmessage for the (no longer) scheduled task; it re-enqueues it, leading\nto duplicate delivery. Orkes reports this issue ([#1](https://github.com/orkes-io/jepsen-conductor/issues/1)) is\nfixed in version 5.5.6.\n\nJepsen also suggests that Orkes update the Conductor documentation to\nmake it clear that tasks may execute concurrently. It may also be\npossible for a worker to observe `retryCount = 0` (suggesting\nit was the first to run) even though some other worker had already\nsuccessfully executed that task.\n\n## 3.2 Tasks Stall Before Execution (#2)\n\nWhen using a Postgres primary DB and Redis as the queue, Orkes\nConductor 5.3.2 often allowed a handful of workflows to stall prior to\ntask execution. While stalled, each workflow would show state\n`RUNNING`, and the workflow’s task would appear as if it were\nrunning on a specific worker. However, the worker’s\n`execute()` method was never called for that task. This state\nwould persist until the task’s response timeout triggered, at which\npoint Conductor would reschedule the task. We observed this behavior in\nresponse to a broad variety of faults—for instance, if Redis were killed\nor paused; or Postgres were killed; or a server were killed or\npartitioned from Redis, Postgres, or a worker; or if a worker were\nkilled.\n\nFor example, [consider\nthis test run](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260702T082246-partition-server-redis-stall.zip), in which two out of 20 workflow executions stalled.\nWorkflow execution `3f71d24a-7619-11f1-b911-00163eb95b8c` was\nacknowledged by server `n3` at 08:23:43.588, its sole task\nwas scheduled at 08:23:43.593, and recorded as `IN_PROGRESS`\non worker `n10` at 08:23:44.615. However, worker\n`n10` never called `Worker.execute(task)` with\nthat workflow ID. Server `n3` appeared to execute normally\nuntil 08:24:42, when it logged a read timeout against Redis.\n\nIn general, a pause, crash or partition isolating a Conductor server\nfrom Redis would reliably stall a handful of workflows by an hour—the\ndefault response timeout. Users could lower the\n`responseTimeoutSeconds` parameter to shorten the delay.\n\nSince the tasks were eventually executed within the time implied by\nthe response timeout window, this does not violate Conductor’s claimed\nguarantees. Orkes opted not to change this behavior. ([#2](https://github.com/orkes-io/jepsen-conductor/issues/2)).\n\n## 3.3 Null Workflow IDs (#3)\n\nWhen creating a workflow, the Java client’s [`startWorkflow()`](https://github.com/conductor-oss/java-sdk/blob/06b9f14a29b92a15d03ba13f899a2c49f0df752f/conductor-client/src/main/java/com/netflix/conductor/client/http/WorkflowClient.java#L193-L217)\nmethod normally returned a workflow ID as a string. However, it rarely\nreturned `null`, as [in\nthis test run](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260702T113115-null-workflow-id.zip), which involved killing the Redis process.\n\nOrkes is investigating this issue ([#3](https://github.com/orkes-io/jepsen-conductor/issues/3)).\nWhile the root cause is unknown, version 6.0.2 of the Java client\nincludes a mitigation.\n\n## 3.4 Syntax Error (#4)\n\nOccasionally, in healthy clusters without faults, calls to\n`startWorkflow` would return a syntax error. There seemed to\nbe nothing unusual about the requests: they provided exactly the same\nkind of input as every other call to `startWorkflow()`.\nHere’s [a\ntypical example](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260703T164748-syntax-error.zip):\n\n```\n{:type :invoke,\n :process 12,\n :f :start-workflow,\n :value {:name \"greetings\",\n :version 1,\n :input {\"name\" \"jepsen 10\"}}}\n```\n\nThis threw\n`BACKEND_ERROR - ERROR: syntax error at or near \\\"null\\\"\\n Position: 13`.\n\nConductor stores workflow executions in a table partitioned by date.\nWhen a new partition was created or rotated, a race condition allowed\nthe partition manager to observe a temporary empty map of shards to\ntables. Conductor then looked up the shard in that empty map, and\nconstructed an SQL statement like `INSERT INTO null ...`.\nPostgres threw a syntax error, and that error message was relayed back\nto the client. In version 5.5.6 Conductor ensures that the shard map is\ninitialized before accepting writes, which should prevent the issue ([#4](https://github.com/orkes-io/jepsen-conductor/issues/4)).\n\n## 3.5 Duplicate Key Error (#8)\n\nIn the release candidate build 5.5.0-rc1, starting a workflow could\nvery rarely return a Postgres duplicate key error. For example, take [this\ntest run](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260717T092502-duplicate-key-value.zip), with Postgres kills and pauses. A single call to\n`startWorkflow` returned this error:\n\n```\nBACKEND_ERROR - ERROR: duplicate key value\nviolates unique constraint\n\\\"workflow_archive_shard_00_pkey\\\"\\n\nDetail: Key (org_id, workflow_id)=\n(0000, 72e49555-81eb-11f1-8ccb-00163eb95b8c)\nalready exists.\n```\n\nThis seems to have been caused by a retry mechanism added to\n5.5.0-rc1: when a Conductor server failed to insert a new workflow\nexecution record into Postgres, it would try again, and could conflict\nwith its previous write. That error message was then relayed to the\nclient. In version 5.5.6, Conductor should ignore duplicate ID errors on\nretries ([#8](https://github.com/orkes-io/jepsen-conductor/issues/8)).\n\n## 3.6 Aborted Read (#6)\n\nIn 5.3.2, 5.4.0, 5.5.0-rc1, and 5.5.0-rc2, using a Postgres database\nand a Redis queue, killing Conductor servers often caused calls to\n`startWorkflow()` to throw an error like\n`Failed to connect to n3/10.0.3.103:8080`. This sounds like a\ndefinite failure—after all, if the client couldn’t open a connection,\nhow could it possibly start the workflow? However, many of these\nworkflows were actually executed by workers and visible to\n`getWorkflow` calls. In [this\nshort run](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260713T170744-g1a.zip), 1,752 out of 25,920 failed calls to\n`startWorkflow` were actually successful, like this one:\n\n```\n{:index 65834,\n :time 11146587755,\n :type :fail,\n :process 24,\n :f :start-workflow,\n :value {:name \"greetings\",\n :version 1,\n :input {\"name\" \"jepsen 8568\"}},\n :error\n [:failed-to-connect\n \"Failed to connect to n3/10.0.3.103:8080\"]}\n```\n\nThe unique input for this workflow, `\"jepsen 8568\"`,\nappeared in later reads of workflow\n`559cb7ea-7f07-11f1-9dce-00163eb95b8c`. In fact, the client\nhad issued a successful request to create this workflow, but perhaps due\nto a client-side timeout, opted to retry the request internally. When\nthat retry failed to open a connection, it returned only that most\nrecent error, rather than one which reflected the earlier request\ntimeout. Orkes changed the Java client to [avoid retrying\nnon-idempotent requests](https://github.com/conductor-oss/java-sdk/pull/175), and is exploring whether to add an\n`isDefinite()` method to client errors ([#6](https://github.com/orkes-io/jepsen-conductor/issues/6)).\n\n## 3.7 Abandoned Workflows (#5)\n\nIn versions 5.3.2, 5.4.0, and 5.5.0-rc1, with a Postgres database and\na Redis queue, a variety of faults could cause Conductor to permanently\nabandon some workflow executions. These workflows would remain in the\n`RUNNING` state indefinitely, with their tasks never\nexecuted. However, both the decider queue and the queue for that task\ntype would be empty. We observed this when killing a single Conductor\nserver, or isolating a server from Postgres or Redis, or killing\nPostgres. Neither waiting nor restarting the nodes would cause Conductor\nto re-execute the abandoned workflows.\n\nConsider [this\ntest run](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260709T155430-server-kill-abandoned.zip), where during the 15-second main phase we killed server\n`n4` once, then restarted it. This caused workflow\n`70ee12a3-7bd8-11f1-8029-00163e983f22` to be abandoned. Even\nhours after the test, the cluster listed that workflow in state\n`RUNNING`, but no worker ever attempted to execute its\ntask.\n\nWhen a workflow is started, Conductor first writes a workflow record\nto Postgres, then enqueues the workflow in the decider queue. If there\nwere to be a failure during one of those writes, the workflow might be\nsuccessfully written to Postgres, but be missing from the queue; it\nwould consequently never be executed. Surprisingly, the [sweeper\nprocess](https://conductor-oss.github.io/conductor/devguide/running/deploy.html#sweeper) which periodically monitors running workflows relies purely\non the decider queue, not the database. This meant Conductor never\nrecovered from these partial writes. In version 5.5.0, Orkes added a\nbackground process which re-enqueues abandoned executions ([#5](https://github.com/orkes-io/jepsen-conductor/issues/5)).\n\n## 3.8 Lost Workflows (#7)\n\nIn versions 5.3.2, 5.4.0, and 5.5.0-rc1, network partitions between a\nConductor server and Postgres, or killing Postgres, could infrequently\ncause Conductor to forget about the existence of workflow executions\nwhich were previously visible to clients. Calls to the Java SDK’s\n`getWorkflow()` method would return null, and HTTP requests\nfor `/api/workflow/` would return a 404 error. This\nstate persisted indefinitely; lost workflows were not recoverable.\n\nFor example, take [this\ntest of version 5.4.0](https://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260716T023240-missing.zip) using Postgres as the DB and decider queue,\nand Redis as the normal queue. Workflow\n`0c426286-80e9-11f1-bb95-00163e983f22` was discovered 239\nseconds into the test via a call to `getRunningWorkflow()`.\nConductor insisted this workflow was in state RUNNING until 313 seconds.\nFrom 313 seconds until the end of the test at 677 seconds, every call to\n`getWorkflow` for that ID returned null.\n\nThis workflow execution was committed to Postgres, but the server was\nunable to append it to the decider queue—perhaps due to a network\npartition. Given this failure, the server then decided to undo its write\nto Postgres by deleting the workflow record. In the intervening 74\nseconds, that doomed workflow was visible to clients. In version 5.5.0,\nConductor no longer deletes executions when a step fails ([#7](https://github.com/orkes-io/jepsen-conductor/issues/7)).\n\n## 3.9 For Sale: Completed Workflow, Never Run (#9)\n\nWhen Postgres was killed, we occasionally found workflows in state\n`COMPLETED` whose tasks had never actually run.[3](#fn3) In\n[this\ntest](http://s3.amazonaws.com/jepsen.io/analyses/orkes-conductor-5.3.2/20260718T224350-never-executed.zip) of version 5.5.0-rc2, three workflows were completed but never\nexecuted. At the end of the test, workflow\n`774977a0-8324-11f1-911d-00163e983f22` looked like this:\n\n```\n{\n \"tasks\" : [{\n \"pollCount\" : 1,\n \"reasonForIncompletion\" : \"responseTimeout:\n 10 exceeded for the taskId:\n 77499eb5-8324-11f1-911d-00163e983f22\n with Task Definition: greet\",\n \"retried\" : true,\n \"taskDefinition\" : {\n \"retryDelaySeconds\" : 5,\n \"rateLimitFrequencyInSeconds\" : 1,\n \"responseTimeoutSeconds\" : 10,\n \"retryCount\" : 2147483647,\n \"timeoutPolicy\" : \"TIME_OUT_WF\",\n ...\n },\n \"retryCount\" : 0,\n \"status\" : \"TIMED_OUT\",\n \"executed\" : false,\n \"workerId\" : \"n7\",\n ...\n }],\n \"status\" : \"COMPLETED\",\n ...\n}\n```\n\nWhile this workflow’s sole task was supposedly assigned to worker\n`n7`, neither `n7` nor any other worker ever\nlogged it: it seems that `Worker.execute` was never called\nfor this task. The task ended in status `TIMED_OUT`, and its\nretry policy was `TIME_OUT_WF`, which, per the [Conductor Error Handling\ndocumentation](https://orkes.io/content/error-handling), should have caused the workflow to also have status\n`TIMED_OUT`. Instead the workflow was marked as\n`COMPLETED`, despite never being executed.\n\nThis workflow was affected by the task stall problem discussed\nearlier (#2): worker `n7` polled the decider queue and\nConductor believed `n7` *would* execute the task, but\n(perhaps due to a network failure) `n7` never began\nexecution. The sweeper task on server `n4` then marked the\ntask as `TIMED_OUT`, and tried to schedule a retry. That\nretry process makes two separate writes to Postgres: one which inserts a\nnew task record, and a second which marks the original task row as\nsuperseded. Postgres failed between these two writes, but Conductor\nignored that failure and marked the original task superseded without a\nnew task row. This explains why the task was never retried, even though\nthe task definition said it should have been. Server `n3`’s\nsweeper process then discovered the superseded task, and incorrectly\nconcluded that since all tasks had a terminal status, the workflow was\n`COMPLETED`.\n\nA database transaction around these two writes would have prevented\nthis issue ([#9](https://github.com/orkes-io/jepsen-conductor/issues/9)).\nHowever, the two writes came from different layers of the code, and\nwrapping the two in a transaction would have required significant\nchanges. In version 5.5.6, Conductor no longer ignores a failed write of\na retry task record.\n\n| № | Summary | Event Required | Fixed in |\n| --- | --- | --- | --- |\n| [#1](https://github.com/orkes-io/jepsen-conductor/issues/1) | Unexpectedly concurrent task execution | None | 5.5.6 |\n| [#2](https://github.com/orkes-io/jepsen-conductor/issues/2) | Tasks stall on most failures | Pause, or crash, or partition | As designed |\n| [#3](https://github.com/orkes-io/jepsen-conductor/issues/3) | Null workflow IDs | Redis crash | Unresolved |\n| [#4](https://github.com/orkes-io/jepsen-conductor/issues/4) | `startWorkflow` returns syntax error | None | 5.5.6 |\n| [#8](https://github.com/orkes-io/jepsen-conductor/issues/8) | `startWorkflow` returns duplicate key error | Postgres crash and pause | 5.5.6 |\n| [#6](https://github.com/orkes-io/jepsen-conductor/issues/6) | Aborted read | Server crash, or Redis/Postgres crash | Unresolved |\n| [#5](https://github.com/orkes-io/jepsen-conductor/issues/5) | Abandoned workflows | Server or Postgres crash, or partition | 5.5.0 |\n| [#7](https://github.com/orkes-io/jepsen-conductor/issues/7) | Missing workflows | Postgres crash or partition | 5.5.0 |\n| [#9](https://github.com/orkes-io/jepsen-conductor/issues/9) | Never-executed workflows marked as completed | Postgres crash and partition | 5.5.6 |\n\n# 4 Discussion\n\nOur tests of Orkes Conductor found that tasks ran concurrently more\noften than necessary (#1), and that execution could be significantly\ndelayed by single-node failures (#2). We also found four minor errors in\n`startWorkflow` (#3, #4, #6, #8), and three serious safety\nerrors where acknowledged workflows would never execute (#5), disappear\nentirely (#7), or be marked as `COMPLETED` despite never\nrunning (#9). Since we were able to reproduce many of these issues with\nsingle process failures, or in some cases observe them in healthy\nclusters, it is conceivable that production users of Orkes Conductor may\nhave been affected.\n\nConductor automatically recovered from task stalls (#2) after the\ndefault one-hour timeout; users should not have to take any action to\nensure affected workflows complete. Issues #3, #4, and #8 caused\n`startWorkflow` to return an error or a null workflow ID. It\nseems likely that client code would notice these errors and retry or\npropagate them appropriately. Issue #6 caused successful calls to\n`startWorkflow` to return apparently definite “failed to\nconnect” errors; if clients retried these errors automatically, they may\nhave inadvertently created multiple copies of a single workflow. In the\nabsence of documentation, it may be safest for users to assume that all\nConductor errors are indefinite.\n\nFor abandoned workflows (#5), users may be able to recover by\nsearching their Conductor clusters for `RUNNING` workflows\nwhich have not been executed in some time, and manually restarting them.\nLikewise, users may be able to search for workflows which are in state\n`COMPLETED`, but whose tasks did not complete (#9), and\nrestart them as well. Missing workflows (#7) might be recoverable by\nconsulting an external system which is known to have information about\nworkflow executions before they were sent to Conductor—for example, a\nlog maintained by an upstream system.\n\nWe were able to test up to version 5.5.0-rc2, which included fixes\nfor issues #5 and #7. Orkes subsequently released version 5.5.6 with\nfixes for #1, #4, #8, and #9. We recommend users upgrade to version\n5.5.6, since it addresses all three serious safety issues we found. Task\nstalls (#2) were consistent with Conductor’s promised behavior and need\nnot be addressed. Null workflow IDs (#3) and a misleading “failed to\nconnect” error message (#6) are still under investigation.\n\nIn other respects, Conductor appeared to function as one would\nexpect. Workflow and task inputs and outputs were consistent with one\nanother, task outputs appeared to be stable despite being executed\nmultiple times, and so on. As always, Jepsen takes an experimental\napproach to safety verification: we can prove the presence of bugs, but\nnot their absence. While we make extensive efforts to find problems, we\ncannot prove correctness.\n\n## 4.1 Concurrency\n\nUsers should be aware that within a single workflow, a single task\nmay be executed by multiple worker threads concurrently (#1). In our\ntests, we observed this behavior every few minutes to hours in healthy\nclusters, though it affected a relatively small proportion of workflows.\nThis is an inherent property of any distributed execution system, and\ncannot be prevented—though its frequency can be reduced.\n\nEven if mutual exclusion were possible, it would not prevent the\neffects of workers from being reordered or interleaved with one another.\nSince those effects might be routed through an asynchronous network,\nthey could be arbitrarily delayed or lost. This is why distributed\nsystems often use [fencing\ntokens](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html) or [consensus](https://sre.google/sre-book/managing-critical-state/)\ninstead of mutual exclusion. In any workflow system, users should take\ncare to ensure that tasks are idempotent and commute—not only with other\nattempts of the same task, but with other tasks, and with tasks executed\nby [failure\nworkflows](https://conductor-oss.github.io/conductor/devguide/how-tos/Workflows/handling-errors.html).\n\nFor example, imagine a workflow which books a flight. Conductor\nbegins the book-flight task, but the worker times out, causing Conductor\nto try the book-flight task again. That too times out, and Conductor\ninitiates a failure workflow to cancel the flight. The failure workflow\nexecutes a task which cancels the flight, receives confirmation from the\nflight service that the flight has been canceled, and terminates. Then\none of the requests sent by the book-flight task arrives at the flight\nservice, and the flight is booked again—even though the workflow\napparently canceled it.\n\nThe solution is to ensure that the book-flight operation and the\ncancel-flight operation commute—that is to say, they should have the\nsame effects when executed in either order. It is not sufficient for the\nflight service to cancel an existing flight: it must also prevent any\nnumber of *future* requests to book that flight. Downstream\nservices can use [idempotency keys](https://orkes.io/content/idempotency) and\nlogical clocks to provide these commutative semantics.\n\nThese challenges are not unique to Conductor, but apply to any\ndistributed orchestration system. The original [Saga\npaper](https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf) was intended for single-node database systems. When [generalized\nto distributed systems](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction), new failure modes arise.\n\n## 4.2 Storage Safety\n\nConductor is only as safe as the storage systems it relies on.\nConductor 5.3.2 required that the systems used for\n`conductor.db.type` and `conductor.queue.type` not\nlose acknowledged writes. In 5.4.0, the system used for\n`conductor.queue.type` could theoretically lose acknowledged\nwrites without impairing Conductor safety, so long as\n`conductor.queue.decider-queue.type` did not. This was not\nquite true (#7), but the core idea remains.\n\nWe tested both single-node Redis and Redis Cluster deployments, and\nfound that when using Redis for either the primary DB or the queue in\n5.3.2, Conductor could lose acknowledged workflow executions when Redis\nnodes were killed or partitioned from one another. We did not test\nPostgres data loss, but it seems reasonable to assume similar effects.\nThe question for Orkes and users who deploy Conductor themselves is: how\nsafe are the Redis and Postgres systems being used?\n\nA single Redis instance is of course a single point of failure: it\nmay lose data if the node’s disks fail, or if it is configured with any\nsetting weaker than [`appendfsync always`](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/)\nand the node loses power. [Redis\nCluster](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/) and [Redis\nSentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) can both lose acknowledged writes when networks partition.\nManaged Redis-compatible services vary in their safety guarantees.\nElastiCache Valkey and ElastiCache Redis OSS both [warn\nof write loss](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Replication.html), but Amazon says [this\nshould not occur](https://aws.amazon.com/blogs/database/announcing-durability-for-amazon-elasticache/) with `--durability sync`. Google Cloud’s\nMemorystore for Redis may also [lose\nacknowledged writes](https://docs.cloud.google.com/memorystore/docs/redis/high-availability-for-memorystore-for-redis), as can Azure Managed Redis [with\npersistence](https://learn.microsoft.com/en-us/azure/redis/how-to-persistence) and [with\nhigh availability](https://learn.microsoft.com/en-us/azure/redis/troubleshoot-data-loss).\n\nSingle-node Postgres deployments are similarly vulnerable to\ncatastrophic node failures, like fires or filesystem corruption. Some\nPostgres replication schemes are unsafe: as [Bin\nWang demonstrated](https://www.binwang.me/2024-12-02-PostgreSQL-High-Availability-Solutions-Part-1.html), Patroni clusters can lose committed writes in\nsome situations, [even\nwith `synchronous_mode_strict`](https://patroni.readthedocs.io/en/latest/replication_modes.html). [Citus\nCloud](https://docs.citusdata.com/en/v6.2/cloud/availability.html) claims “you won’t lose any data”, but seems to rely on an\nasynchronous WAL storage system which could lose up to 60 seconds or 16\nMB of writes. Google’s Cloud SQL for PostgreSQL documentation [warns\nabout the possibility of split-brain and data loss](https://docs.cloud.google.com/sql/docs/postgres/intro-to-cloud-sql-disaster-recovery#avoid-split-brain) in some\nconfigurations. However, there are several distributed databases which\nprovide Postgres-compatible interfaces with stronger safety guarantees,\nlike [CockroachDB](https://www.cockroachlabs.com/docs/stable/postgresql-compatibility).\nIn short, one’s mileage may vary.\n\nWhile we only tested the proprietary Orkes Conductor, it seems likely\nthat Conductor OSS depends similarly on the safety of its backing\nstorage services. To avoid lost or abandoned workflow executions, we\nrecommend users carefully evaluate their choice of\n`conductor.db.type` and `conductor.queue.type`,\nand the safety of those respective systems.\n\n## 4.3 Future Work\n\nWhile these tests were run on Orkes Conductor, they used the standard\nConductor Java client. With a little work, the test suite could measure\nConductor OSS behavior as well. We focused on\n`conductor.db.type=postgres` and\n`conductor.queue.type=redis_standalone`, but Conductor\nsupports [a\nvariety of storage systems](https://conductor-oss.github.io/conductor/devguide/running/deploy.html?h=database#database). Future research could explore behavior\nwith Cassandra, MySQL, and SQLite.\n\nOur tests only evaluated workflows with a single task using our\ncustom worker. We have not yet explored multi-step workflows, workflows\nwhich use Conductor’s built-in or control-flow tasks, or failure\nworkflows. All of these might be useful avenues for further testing.\n\nWhile we tested with process pauses, crashes, and network partitions,\nwe have not yet explored high network latency or clock skew.\n\n*Our thanks to the Orkes team for their invaluable support,\nincluding Viren Baraiya, Manan Bhatt, Dale Brady, Jeu George, Dilip\nLukose, and Miguel Prieto. Jepsen also wishes to thank [Irene Kannyo](https://www.irenekannyo.com/) for her editorial\nsupport. This research was funded by Orkes Inc., and conducted in\naccordance with the [Jepsen\nethics policy](https://jepsen.io/analyses/ethics).*\n\n---\n\n1. There are two unrelated orchestration systems with\n similar names: one from Microsoft, also named [Conductor](https://github.com/microsoft/conductor), and one\n called [Postgres Conductor](https://pg-conductor.dev/). There\n is also an unrelated [AI marketing\n service](https://www.conductor.com/) called Conductor.[↩︎](#fnref1)\n2. While operations within Conductor should see only a\n single successful task execution, tasks may have visible side effects on\n systems outside Conductor. Well-behaved tasks should therefore be\n idempotent and deterministic.[↩︎](#fnref2)\n3. We discovered this issue just as our work on Conductor\n was coming to a close; regrettably, we have not been able to test with\n other versions or narrow down the failure modes that might induce it.[↩︎](#fnref3)\n","body_html":"

Jepsen: Orkes Conductor 5.3.2

\n

Kyle Kingsbury | Jepsen | 2026-10-20

\n

Orkes Conductor is a\nworkflow orchestration platform which runs user-defined tasks and\nroutes data between them. We tested Orkes Conductor versions 5.3.2\nthrough 5.5.0-rc2 and found unexpectedly frequent concurrent\nexecution of individual tasks, one issue that could significantly\ndelay workflow execution, four minor issues with\nstartWorkflow errors or null IDs, and three serious\nproblems which caused Conductor to permanently abandon or forget\nworkflow executions, or to mark them completed even though their\ntasks were never run. Orkes has released version 5.5.6 with patches\nfor all serious safety issues, as well as reducing concurrent\nexecution and fixing two startWorkflow errors. The\nother two startWorkflow errors are under investigation.\nDelayed execution is consistent with Conductor’s design parameters\nand need not be addressed. Orkes has published a\ncompanion blog post to this work. This research was funded by\nOrkes Inc., and conducted in accordance with the Jepsen ethics\npolicy.

\n

1 Background

\n

A workflow\norchestrator is a software system which executes computations made\nup of small tasks. Tasks are connected in a workflow,\nwhich routes the outputs of some tasks to the inputs of others,\ndetermines which tasks may be scheduled and when, and so on. The\norchestrator manages workflows and specific executions of them, runs\ntheir tasks, persists state, and tries to drive each execution to\ncompletion.

\n

Conductor is a workflow\nsystem originally built\nby Netflix and open sourced in 2016.1 The\nopen source version, Conductor OSS, is\nnow maintained by the Conductor OSS\nFoundation. This foundation is led by Orkes,\nwhose founders helped build Conductor at Netflix. Orkes also offers a hosted version of Conductor with\nadditional features, called Orkes Conductor. This analysis\nfocuses on Orkes Conductor, which we call Conductor for\nshort.

\n

In Conductor, a workflow definition is a JSON structure that\ndefines which tasks must be executed, their order, what their inputs and\noutputs should be, and so on. Each workflow definition is identified by\na name and a version. For example, a workflow definition could specify\nthat to adopt a cat, one must first complete an application, then\nschedule a vet visit, and receive vaccination records from the vet. A\nworkflow may also accept an input (say, the name of the cat) and produce\nan output (like a certificate of adoption).

\n

A workflow definition may be run many times—say, once for each cat to\nbe adopted. Each instance is called a workflow execution, but\nwe often call an execution a “workflow” for short. Each execution is\nidentified by a unique workflow ID. During an execution a task\nmay also be attempted multiple times (e.g. in case of failure), and each\nattempt receives a unique execution ID.

\n

Some tasks are built into Conductor. Some are system tasks,\nlike making HTTP requests or submitting events to a log like Kafka.\nOthers are called operators—these provide control flow like\nconditionals, fork/join, loops, and so on. Operators may execute\nECMAScript code to perform user-defined logic. Finally there are\nworker tasks, which are written by users to perform the\n“business logic” of the system.

\n

To start an execution of a workflow, one can make an API call to\nConductor, submit an event through a message broker, or use Conductor’s\nbuilt-in cron-style scheduler. Once acknowledged, Conductor\nschedules the workflow’s tasks and attempts to drive it to\ncompletion.

\n

1.1 Architecture

\n

Conductor has\nfive major components. Conductor servers are stateless processes\nwhich manage workflows and schedule execution. They provide an API and\nuser interface, interact with external services, and track state. The\ndefinitions of workflows, the state of their executions and the results\nof tasks are stored in a database like Postgres, Cassandra, or Redis.\nConductor also uses a queue—often Redis—to dispatch work to various\nprocesses, and to store short-lived state. Finally, there are two pools\nof stateless workers: standard workers which execute Conductor’s\nbuilt-in tasks, and custom workers, which are written by users using SDKs for various\nlanguages. All workers connect to a\nConductor server, poll the server for tasks, execute them, and send\nresults back to the server.

\n

Each component may be scaled independently. Any number of servers and\nworkers can run concurrently, and nodes may be killed and restarted at\nany time. In version 5.3.2, the database and queue were both required to\npreserve all acknowledged writes. However, version 5.4.0 introduced a\nnew option which enabled the use of a lossy storage system for all\nqueues except for the main decider queue.

\n

1.2 Guarantees

\n

Conductor guarantees “at\nleast once message delivery”, ensuring that all messages are\npersistent and will be delivered to task workers one or more times.\nConductor also guarantees that workflows eventually terminate. “If a\nworkflow has started, it will run to completion as long as all its tasks\nare completed.” This does not necessarily mean that workflows and tasks\nexecute successfully. Tasks\ncan run out of retry attempts, fail with non-retryable errors, or be\nexplicitly canceled—and a similar state machine applies to workflow\nexecutions. However, workflows should eventually converge on a terminal\nstatus: COMPLETED, FAILED,\nTIMED_OUT, or TERMINATED.

\n

When tasks are executed, each attempt receives a unique execution ID,\nand also a unique retry count, which indicates\nthe number of attempts already performed.

\n

Orkes says that for each workflow execution, each instance of a task\nhas at most one visible successful execution. Tasks may execute\nsuccessfully multiple times—for instance, if a Conductor server loses\nits connection to the database just before persisting the completion of\nthe task execution. However, once the output of a successful task\nbecomes visible, no task, workflow, or API call will observe a different\noutput.2

\n

2 Test Design

\n

To verify these properties, we built a test harness for Orkes\nConductor using the Jepsen\ntesting library and the Conductor Java SDK\nat version 5.0.1. We used this harness to test Orkes Conductor 5.3.2,\n5.4.0, and 5.5.0-rc1 and -rc2.

\n

Orkes typically deploys Conductor in cloud environments with Postgres\n(or a Postgres-compatible database, like their in-house ParticleDB) as the primary database,\nand Redis (or a compatible service like Valkey) as\nthe queue. To approximate these deployments, we set up clusters of ten\nDebian machines, both in containers and as separate EC2 VMs: a single\nPostgres node for the database, either a single Redis node or Redis\nCluster for the queue, and the remaining nodes split evenly between\nConductor servers, Conductor’s standard workers, and a custom\nworker we wrote using the Java SDK.

\n

Our test\nworkload defined a single workflow with one task, handled by our\ncustom worker. The task slept up to 30 seconds to simulate work being\ndone, then returned\na randomly generated output. It also logged\nits execution with the test harness, so we could tell exactly which\ntasks had run and when, along with their observed inputs and intended\noutputs.

\n

Our test proceeded in three phases. During the main\nphase, the test harness called startWorkflow()\nwith unique inputs (to distinguish executions from one another), kept\ntrack of which executions were in-progress and which were terminal,\nand checked the status of specific executions using\ngetWorkflow(). To discover executions when the call to\nstartWorkflow() timed out or failed, we periodically called\ngetRunningWorkflow()\nto list all in-progress executions. During the main phase we also\ninjected a\nvariety of faults, including pausing and killing processes, as well\nas network partitions between Conductor servers and Postgres, Redis, and\nworker nodes.

\n

During the recovery\nphase, the test ended network partitions, restarted and unpaused\nprocesses, etc., then let the cluster recover. To figure out when the\nsystem had converged on a steady state, we polled the depth of\nConductor’s decider queue. When this queue reached zero, Conductor had\nnothing more to do and the test moved to the final phase.

\n

During the final\nphase, the test read the state of every workflow execution it had\never seen. We then searched for a\nvariety of unexpected behaviors:

\n

Backend Error:\n: Some request returned an unexpected backend error.

\n

Concurrent Execution:\n: Two or more workers attempted to execute the same task in a single\n workflow execution at the same time.

\n

Different Task Input:\n: Two views of the same task in the same workflow execution had different\n inputs, or a task’s input was inconsistent with the workflow’s input.

\n

Different Workflow Input:\n: A single workflow execution had two different inputs.

\n

Different Workflow Output:\n: A single workflow execution had two different outputs.

\n

Duplicate Workflow ID:\n: Two calls to startWorkflow returned the same workflow\n execution ID.

\n

G1a:\n: A startWorkflow request returned what appeared to be a\n definite failure, but a workflow with that requested input was later\n visible to some reader.

\n

Impossible Output:\n: A workflow execution’s output was inconsistent with the outputs of its\n tasks.

\n

Incomplete:\n: A workflow execution was not in state COMPLETED during the\n final phase.

\n

Missing:\n: A read failed to observe a workflow execution after we knew it existed.

\n

Never Started:\n: A workflow came into existence with no corresponding\n startWorkflow request.

\n

Null Workflow ID:\n: A startWorkflow call returned null, rather\n than a workflow ID string.

\n

Task Never Executed:\n: None of a workflow’s tasks were ever executed.

\n

3 Results

\n

3.1 Tasks Execute Concurrently (#1)

\n

Under normal operation, healthy Conductor clusters occasionally\nexecuted the same task for the same workflow execution on two workers\nconcurrently. This is not surprising, since mutual exclusion\nis impossible in asynchronous networks. However, it is worth keeping\nin mind that even very short-lived tasks may be executed at the same\ntime! In this\ntest run, workflow\n3919784b-7691-11f1-a292-00163e983f22’s single task began on\nworker n9 at 70.110 seconds, then on worker\nn10 at 70.113 seconds. It completed on n9 at\n70.114 seconds, and on n10 less than a millisecond\nlater.

\n

The documentation for Conductor OSS describes a distributed locking\nsystem which “prevents race conditions when multiple server instances\nevaluate the same workflow concurrently”. Orkes Conductor also has a\nlocking mechanism whose lease time is configured by\nconductor.workflow.lockLeaseTime. While this lock can\nreduce the frequency of concurrent execution, it cannot prevent it\nentirely.

\n

With our single-task workflows, the default lease time of one minute,\ntwo Conductor servers, three workers, near-zero network latency,\nshort-lived tasks (p50 ~500 µs, p95 ~5 ms, p99 ~10 ms), and roughly\n1,100 workflows per second, we observed concurrent execution roughly\nonce every six hours in healthy clusters. Reducing the lease time to one\nsecond allowed concurrent execution every few minutes. Production\nfrequency may vary.

\n

This problem was exacerbated by a time\nof check to time of use bug in the sweeper process which re-enqueues\nstuck tasks. Imagine a sweeper reads a task in state\nSCHEDULED. A worker then executes that task, changing its\nstate in the database to IN_PROGRESS and removing the task\nmessage from the queue. The sweeper then checks the queue and finds no\nmessage for the (no longer) scheduled task; it re-enqueues it, leading\nto duplicate delivery. Orkes reports this issue (#1) is\nfixed in version 5.5.6.

\n

Jepsen also suggests that Orkes update the Conductor documentation to\nmake it clear that tasks may execute concurrently. It may also be\npossible for a worker to observe retryCount = 0 (suggesting\nit was the first to run) even though some other worker had already\nsuccessfully executed that task.

\n

3.2 Tasks Stall Before Execution (#2)

\n

When using a Postgres primary DB and Redis as the queue, Orkes\nConductor 5.3.2 often allowed a handful of workflows to stall prior to\ntask execution. While stalled, each workflow would show state\nRUNNING, and the workflow’s task would appear as if it were\nrunning on a specific worker. However, the worker’s\nexecute() method was never called for that task. This state\nwould persist until the task’s response timeout triggered, at which\npoint Conductor would reschedule the task. We observed this behavior in\nresponse to a broad variety of faults—for instance, if Redis were killed\nor paused; or Postgres were killed; or a server were killed or\npartitioned from Redis, Postgres, or a worker; or if a worker were\nkilled.

\n

For example, consider\nthis test run, in which two out of 20 workflow executions stalled.\nWorkflow execution 3f71d24a-7619-11f1-b911-00163eb95b8c was\nacknowledged by server n3 at 08:23:43.588, its sole task\nwas scheduled at 08:23:43.593, and recorded as IN_PROGRESS\non worker n10 at 08:23:44.615. However, worker\nn10 never called Worker.execute(task) with\nthat workflow ID. Server n3 appeared to execute normally\nuntil 08:24:42, when it logged a read timeout against Redis.

\n

In general, a pause, crash or partition isolating a Conductor server\nfrom Redis would reliably stall a handful of workflows by an hour—the\ndefault response timeout. Users could lower the\nresponseTimeoutSeconds parameter to shorten the delay.

\n

Since the tasks were eventually executed within the time implied by\nthe response timeout window, this does not violate Conductor’s claimed\nguarantees. Orkes opted not to change this behavior. (#2).

\n

3.3 Null Workflow IDs (#3)

\n

When creating a workflow, the Java client’s startWorkflow()\nmethod normally returned a workflow ID as a string. However, it rarely\nreturned null, as in\nthis test run, which involved killing the Redis process.

\n

Orkes is investigating this issue (#3).\nWhile the root cause is unknown, version 6.0.2 of the Java client\nincludes a mitigation.

\n

3.4 Syntax Error (#4)

\n

Occasionally, in healthy clusters without faults, calls to\nstartWorkflow would return a syntax error. There seemed to\nbe nothing unusual about the requests: they provided exactly the same\nkind of input as every other call to startWorkflow().\nHere’s a\ntypical example:

\n
{:type    :invoke,\n :process 12,\n :f       :start-workflow,\n :value   {:name "greetings",\n           :version 1,\n           :input {"name" "jepsen 10"}}}
\n

This threw\nBACKEND_ERROR - ERROR: syntax error at or near \\"null\\"\\n Position: 13.

\n

Conductor stores workflow executions in a table partitioned by date.\nWhen a new partition was created or rotated, a race condition allowed\nthe partition manager to observe a temporary empty map of shards to\ntables. Conductor then looked up the shard in that empty map, and\nconstructed an SQL statement like INSERT INTO null ....\nPostgres threw a syntax error, and that error message was relayed back\nto the client. In version 5.5.6 Conductor ensures that the shard map is\ninitialized before accepting writes, which should prevent the issue (#4).

\n

3.5 Duplicate Key Error (#8)

\n

In the release candidate build 5.5.0-rc1, starting a workflow could\nvery rarely return a Postgres duplicate key error. For example, take this\ntest run, with Postgres kills and pauses. A single call to\nstartWorkflow returned this error:

\n
BACKEND_ERROR - ERROR: duplicate key value\nviolates unique constraint\n\\"workflow_archive_shard_00_pkey\\"\\n\nDetail: Key (org_id, workflow_id)=\n(0000, 72e49555-81eb-11f1-8ccb-00163eb95b8c)\nalready exists.
\n

This seems to have been caused by a retry mechanism added to\n5.5.0-rc1: when a Conductor server failed to insert a new workflow\nexecution record into Postgres, it would try again, and could conflict\nwith its previous write. That error message was then relayed to the\nclient. In version 5.5.6, Conductor should ignore duplicate ID errors on\nretries (#8).

\n

3.6 Aborted Read (#6)

\n

In 5.3.2, 5.4.0, 5.5.0-rc1, and 5.5.0-rc2, using a Postgres database\nand a Redis queue, killing Conductor servers often caused calls to\nstartWorkflow() to throw an error like\nFailed to connect to n3/10.0.3.103:8080. This sounds like a\ndefinite failure—after all, if the client couldn’t open a connection,\nhow could it possibly start the workflow? However, many of these\nworkflows were actually executed by workers and visible to\ngetWorkflow calls. In this\nshort run, 1,752 out of 25,920 failed calls to\nstartWorkflow were actually successful, like this one:

\n
{:index 65834,\n :time 11146587755,\n :type :fail,\n :process 24,\n :f :start-workflow,\n :value {:name "greetings",\n         :version 1,\n         :input {"name" "jepsen 8568"}},\n :error\n [:failed-to-connect\n  "Failed to connect to n3/10.0.3.103:8080"]}
\n

The unique input for this workflow, "jepsen 8568",\nappeared in later reads of workflow\n559cb7ea-7f07-11f1-9dce-00163eb95b8c. In fact, the client\nhad issued a successful request to create this workflow, but perhaps due\nto a client-side timeout, opted to retry the request internally. When\nthat retry failed to open a connection, it returned only that most\nrecent error, rather than one which reflected the earlier request\ntimeout. Orkes changed the Java client to avoid retrying\nnon-idempotent requests, and is exploring whether to add an\nisDefinite() method to client errors (#6).

\n

3.7 Abandoned Workflows (#5)

\n

In versions 5.3.2, 5.4.0, and 5.5.0-rc1, with a Postgres database and\na Redis queue, a variety of faults could cause Conductor to permanently\nabandon some workflow executions. These workflows would remain in the\nRUNNING state indefinitely, with their tasks never\nexecuted. However, both the decider queue and the queue for that task\ntype would be empty. We observed this when killing a single Conductor\nserver, or isolating a server from Postgres or Redis, or killing\nPostgres. Neither waiting nor restarting the nodes would cause Conductor\nto re-execute the abandoned workflows.

\n

Consider this\ntest run, where during the 15-second main phase we killed server\nn4 once, then restarted it. This caused workflow\n70ee12a3-7bd8-11f1-8029-00163e983f22 to be abandoned. Even\nhours after the test, the cluster listed that workflow in state\nRUNNING, but no worker ever attempted to execute its\ntask.

\n

When a workflow is started, Conductor first writes a workflow record\nto Postgres, then enqueues the workflow in the decider queue. If there\nwere to be a failure during one of those writes, the workflow might be\nsuccessfully written to Postgres, but be missing from the queue; it\nwould consequently never be executed. Surprisingly, the sweeper\nprocess which periodically monitors running workflows relies purely\non the decider queue, not the database. This meant Conductor never\nrecovered from these partial writes. In version 5.5.0, Orkes added a\nbackground process which re-enqueues abandoned executions (#5).

\n

3.8 Lost Workflows (#7)

\n

In versions 5.3.2, 5.4.0, and 5.5.0-rc1, network partitions between a\nConductor server and Postgres, or killing Postgres, could infrequently\ncause Conductor to forget about the existence of workflow executions\nwhich were previously visible to clients. Calls to the Java SDK’s\ngetWorkflow() method would return null, and HTTP requests\nfor /api/workflow/<id> would return a 404 error. This\nstate persisted indefinitely; lost workflows were not recoverable.

\n

For example, take this\ntest of version 5.4.0 using Postgres as the DB and decider queue,\nand Redis as the normal queue. Workflow\n0c426286-80e9-11f1-bb95-00163e983f22 was discovered 239\nseconds into the test via a call to getRunningWorkflow().\nConductor insisted this workflow was in state RUNNING until 313 seconds.\nFrom 313 seconds until the end of the test at 677 seconds, every call to\ngetWorkflow for that ID returned null.

\n

This workflow execution was committed to Postgres, but the server was\nunable to append it to the decider queue—perhaps due to a network\npartition. Given this failure, the server then decided to undo its write\nto Postgres by deleting the workflow record. In the intervening 74\nseconds, that doomed workflow was visible to clients. In version 5.5.0,\nConductor no longer deletes executions when a step fails (#7).

\n

3.9 For Sale: Completed Workflow, Never Run (#9)

\n

When Postgres was killed, we occasionally found workflows in state\nCOMPLETED whose tasks had never actually run.3 In\nthis\ntest of version 5.5.0-rc2, three workflows were completed but never\nexecuted. At the end of the test, workflow\n774977a0-8324-11f1-911d-00163e983f22 looked like this:

\n
{\n  "tasks" : [{\n    "pollCount" : 1,\n    "reasonForIncompletion" : "responseTimeout:\n     10 exceeded for the taskId:\n     77499eb5-8324-11f1-911d-00163e983f22\n     with Task Definition: greet",\n    "retried" : true,\n    "taskDefinition" : {\n      "retryDelaySeconds" : 5,\n      "rateLimitFrequencyInSeconds" : 1,\n      "responseTimeoutSeconds" : 10,\n      "retryCount" : 2147483647,\n      "timeoutPolicy" : "TIME_OUT_WF",\n      ...\n    },\n    "retryCount" : 0,\n    "status" : "TIMED_OUT",\n    "executed" : false,\n    "workerId" : "n7",\n    ...\n  }],\n  "status" : "COMPLETED",\n  ...\n}
\n

While this workflow’s sole task was supposedly assigned to worker\nn7, neither n7 nor any other worker ever\nlogged it: it seems that Worker.execute was never called\nfor this task. The task ended in status TIMED_OUT, and its\nretry policy was TIME_OUT_WF, which, per the Conductor Error Handling\ndocumentation, should have caused the workflow to also have status\nTIMED_OUT. Instead the workflow was marked as\nCOMPLETED, despite never being executed.

\n

This workflow was affected by the task stall problem discussed\nearlier (#2): worker n7 polled the decider queue and\nConductor believed n7 would execute the task, but\n(perhaps due to a network failure) n7 never began\nexecution. The sweeper task on server n4 then marked the\ntask as TIMED_OUT, and tried to schedule a retry. That\nretry process makes two separate writes to Postgres: one which inserts a\nnew task record, and a second which marks the original task row as\nsuperseded. Postgres failed between these two writes, but Conductor\nignored that failure and marked the original task superseded without a\nnew task row. This explains why the task was never retried, even though\nthe task definition said it should have been. Server n3’s\nsweeper process then discovered the superseded task, and incorrectly\nconcluded that since all tasks had a terminal status, the workflow was\nCOMPLETED.

\n

A database transaction around these two writes would have prevented\nthis issue (#9).\nHowever, the two writes came from different layers of the code, and\nwrapping the two in a transaction would have required significant\nchanges. In version 5.5.6, Conductor no longer ignores a failed write of\na retry task record.

\n
№SummaryEvent RequiredFixed in
#1Unexpectedly concurrent task executionNone5.5.6
#2Tasks stall on most failuresPause, or crash, or partitionAs designed
#3Null workflow IDsRedis crashUnresolved
#4startWorkflow returns syntax errorNone5.5.6
#8startWorkflow returns duplicate key errorPostgres crash and pause5.5.6
#6Aborted readServer crash, or Redis/Postgres crashUnresolved
#5Abandoned workflowsServer or Postgres crash, or partition5.5.0
#7Missing workflowsPostgres crash or partition5.5.0
#9Never-executed workflows marked as completedPostgres crash and partition5.5.6
\n

4 Discussion

\n

Our tests of Orkes Conductor found that tasks ran concurrently more\noften than necessary (#1), and that execution could be significantly\ndelayed by single-node failures (#2). We also found four minor errors in\nstartWorkflow (#3, #4, #6, #8), and three serious safety\nerrors where acknowledged workflows would never execute (#5), disappear\nentirely (#7), or be marked as COMPLETED despite never\nrunning (#9). Since we were able to reproduce many of these issues with\nsingle process failures, or in some cases observe them in healthy\nclusters, it is conceivable that production users of Orkes Conductor may\nhave been affected.

\n

Conductor automatically recovered from task stalls (#2) after the\ndefault one-hour timeout; users should not have to take any action to\nensure affected workflows complete. Issues #3, #4, and #8 caused\nstartWorkflow to return an error or a null workflow ID. It\nseems likely that client code would notice these errors and retry or\npropagate them appropriately. Issue #6 caused successful calls to\nstartWorkflow to return apparently definite “failed to\nconnect” errors; if clients retried these errors automatically, they may\nhave inadvertently created multiple copies of a single workflow. In the\nabsence of documentation, it may be safest for users to assume that all\nConductor errors are indefinite.

\n

For abandoned workflows (#5), users may be able to recover by\nsearching their Conductor clusters for RUNNING workflows\nwhich have not been executed in some time, and manually restarting them.\nLikewise, users may be able to search for workflows which are in state\nCOMPLETED, but whose tasks did not complete (#9), and\nrestart them as well. Missing workflows (#7) might be recoverable by\nconsulting an external system which is known to have information about\nworkflow executions before they were sent to Conductor—for example, a\nlog maintained by an upstream system.

\n

We were able to test up to version 5.5.0-rc2, which included fixes\nfor issues #5 and #7. Orkes subsequently released version 5.5.6 with\nfixes for #1, #4, #8, and #9. We recommend users upgrade to version\n5.5.6, since it addresses all three serious safety issues we found. Task\nstalls (#2) were consistent with Conductor’s promised behavior and need\nnot be addressed. Null workflow IDs (#3) and a misleading “failed to\nconnect” error message (#6) are still under investigation.

\n

In other respects, Conductor appeared to function as one would\nexpect. Workflow and task inputs and outputs were consistent with one\nanother, task outputs appeared to be stable despite being executed\nmultiple times, and so on. As always, Jepsen takes an experimental\napproach to safety verification: we can prove the presence of bugs, but\nnot their absence. While we make extensive efforts to find problems, we\ncannot prove correctness.

\n

4.1 Concurrency

\n

Users should be aware that within a single workflow, a single task\nmay be executed by multiple worker threads concurrently (#1). In our\ntests, we observed this behavior every few minutes to hours in healthy\nclusters, though it affected a relatively small proportion of workflows.\nThis is an inherent property of any distributed execution system, and\ncannot be prevented—though its frequency can be reduced.

\n

Even if mutual exclusion were possible, it would not prevent the\neffects of workers from being reordered or interleaved with one another.\nSince those effects might be routed through an asynchronous network,\nthey could be arbitrarily delayed or lost. This is why distributed\nsystems often use fencing\ntokens or consensus\ninstead of mutual exclusion. In any workflow system, users should take\ncare to ensure that tasks are idempotent and commute—not only with other\nattempts of the same task, but with other tasks, and with tasks executed\nby failure\nworkflows.

\n

For example, imagine a workflow which books a flight. Conductor\nbegins the book-flight task, but the worker times out, causing Conductor\nto try the book-flight task again. That too times out, and Conductor\ninitiates a failure workflow to cancel the flight. The failure workflow\nexecutes a task which cancels the flight, receives confirmation from the\nflight service that the flight has been canceled, and terminates. Then\none of the requests sent by the book-flight task arrives at the flight\nservice, and the flight is booked again—even though the workflow\napparently canceled it.

\n

The solution is to ensure that the book-flight operation and the\ncancel-flight operation commute—that is to say, they should have the\nsame effects when executed in either order. It is not sufficient for the\nflight service to cancel an existing flight: it must also prevent any\nnumber of future requests to book that flight. Downstream\nservices can use idempotency keys and\nlogical clocks to provide these commutative semantics.

\n

These challenges are not unique to Conductor, but apply to any\ndistributed orchestration system. The original Saga\npaper was intended for single-node database systems. When generalized\nto distributed systems, new failure modes arise.

\n

4.2 Storage Safety

\n

Conductor is only as safe as the storage systems it relies on.\nConductor 5.3.2 required that the systems used for\nconductor.db.type and conductor.queue.type not\nlose acknowledged writes. In 5.4.0, the system used for\nconductor.queue.type could theoretically lose acknowledged\nwrites without impairing Conductor safety, so long as\nconductor.queue.decider-queue.type did not. This was not\nquite true (#7), but the core idea remains.

\n

We tested both single-node Redis and Redis Cluster deployments, and\nfound that when using Redis for either the primary DB or the queue in\n5.3.2, Conductor could lose acknowledged workflow executions when Redis\nnodes were killed or partitioned from one another. We did not test\nPostgres data loss, but it seems reasonable to assume similar effects.\nThe question for Orkes and users who deploy Conductor themselves is: how\nsafe are the Redis and Postgres systems being used?

\n

A single Redis instance is of course a single point of failure: it\nmay lose data if the node’s disks fail, or if it is configured with any\nsetting weaker than appendfsync always\nand the node loses power. Redis\nCluster and Redis\nSentinel can both lose acknowledged writes when networks partition.\nManaged Redis-compatible services vary in their safety guarantees.\nElastiCache Valkey and ElastiCache Redis OSS both warn\nof write loss, but Amazon says this\nshould not occur with --durability sync. Google Cloud’s\nMemorystore for Redis may also lose\nacknowledged writes, as can Azure Managed Redis with\npersistence and with\nhigh availability.

\n

Single-node Postgres deployments are similarly vulnerable to\ncatastrophic node failures, like fires or filesystem corruption. Some\nPostgres replication schemes are unsafe: as Bin\nWang demonstrated, Patroni clusters can lose committed writes in\nsome situations, even\nwith synchronous_mode_strict. Citus\nCloud claims “you won’t lose any data”, but seems to rely on an\nasynchronous WAL storage system which could lose up to 60 seconds or 16\nMB of writes. Google’s Cloud SQL for PostgreSQL documentation warns\nabout the possibility of split-brain and data loss in some\nconfigurations. However, there are several distributed databases which\nprovide Postgres-compatible interfaces with stronger safety guarantees,\nlike CockroachDB.\nIn short, one’s mileage may vary.

\n

While we only tested the proprietary Orkes Conductor, it seems likely\nthat Conductor OSS depends similarly on the safety of its backing\nstorage services. To avoid lost or abandoned workflow executions, we\nrecommend users carefully evaluate their choice of\nconductor.db.type and conductor.queue.type,\nand the safety of those respective systems.

\n

4.3 Future Work

\n

While these tests were run on Orkes Conductor, they used the standard\nConductor Java client. With a little work, the test suite could measure\nConductor OSS behavior as well. We focused on\nconductor.db.type=postgres and\nconductor.queue.type=redis_standalone, but Conductor\nsupports a\nvariety of storage systems. Future research could explore behavior\nwith Cassandra, MySQL, and SQLite.

\n

Our tests only evaluated workflows with a single task using our\ncustom worker. We have not yet explored multi-step workflows, workflows\nwhich use Conductor’s built-in or control-flow tasks, or failure\nworkflows. All of these might be useful avenues for further testing.

\n

While we tested with process pauses, crashes, and network partitions,\nwe have not yet explored high network latency or clock skew.

\n

*Our thanks to the Orkes team for their invaluable support,\nincluding Viren Baraiya, Manan Bhatt, Dale Brady, Jeu George, Dilip\nLukose, and Miguel Prieto. Jepsen also wishes to thank Irene Kannyo for her editorial\nsupport. This research was funded by Orkes Inc., and conducted in\naccordance with the Jepsen\nethics policy.*

\n
\n

  1. There are two unrelated orchestration systems with

    similar names: one from Microsoft, also named Conductor, and one\n called Postgres Conductor. There\n is also an unrelated AI marketing\n service called Conductor.↩︎

  2. While operations within Conductor should see only a

    single successful task execution, tasks may have visible side effects on\n systems outside Conductor. Well-behaved tasks should therefore be\n idempotent and deterministic.↩︎

  3. We discovered this issue just as our work on Conductor

    was coming to a close; regrettably, we have not been able to test with\n other versions or narrow down the failure modes that might induce it.↩︎

","headings":[{"level":1,"text":"Jepsen: Orkes Conductor 5.3.2","id":"jepsen-orkes-conductor-5-3-2"},{"level":1,"text":"1 Background","id":"1-background"},{"level":2,"text":"1.1 Architecture","id":"1-1-architecture"},{"level":2,"text":"1.2 Guarantees","id":"1-2-guarantees"},{"level":1,"text":"2 Test Design","id":"2-test-design"},{"level":1,"text":"3 Results","id":"3-results"},{"level":2,"text":"3.1 Tasks Execute Concurrently (#1)","id":"3-1-tasks-execute-concurrently-1"},{"level":2,"text":"3.2 Tasks Stall Before Execution (#2)","id":"3-2-tasks-stall-before-execution-2"},{"level":2,"text":"3.3 Null Workflow IDs (#3)","id":"3-3-null-workflow-ids-3"},{"level":2,"text":"3.4 Syntax Error (#4)","id":"3-4-syntax-error-4"},{"level":2,"text":"3.5 Duplicate Key Error (#8)","id":"3-5-duplicate-key-error-8"},{"level":2,"text":"3.6 Aborted Read (#6)","id":"3-6-aborted-read-6"},{"level":2,"text":"3.7 Abandoned Workflows (#5)","id":"3-7-abandoned-workflows-5"},{"level":2,"text":"3.8 Lost Workflows (#7)","id":"3-8-lost-workflows-7"},{"level":2,"text":"3.9 For Sale: Completed Workflow, Never Run (#9)","id":"3-9-for-sale-completed-workflow-never-run-9"},{"level":1,"text":"4 Discussion","id":"4-discussion"},{"level":2,"text":"4.1 Concurrency","id":"4-1-concurrency"},{"level":2,"text":"4.2 Storage Safety","id":"4-2-storage-safety"},{"level":2,"text":"4.3 Future Work","id":"4-3-future-work"}]}}