{"article":{"slug":"concurrent-shell-command-queues","title":"Concurrent Shell Command Queues","subtitle":null,"summary":"Christian Emmer shows how to run shell commands in parallel while limiting their concurrency, comparing Bash job-control approaches with xargs -P and GNU parallel.","content_type":"tutorial","language":"en","canonical_url":"https://emmer.dev/blog/concurrent-shell-command-queues/","author":{"name":"Christian Emmer","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Christian Emmer","url":"https://emmer.dev/","listing_slug":null,"listing":null},"topics":[{"name":"linux","slug":"linux","url":"https://listedarticles.com/topics/linux"},{"name":"tutorials","slug":"tutorials","url":"https://listedarticles.com/topics/tutorials"},{"name":"developer-tools","slug":"developer-tools","url":"https://listedarticles.com/topics/developer-tools"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":1723,"reading_minutes":7,"published_at":"2026-11-07T05:05:00.000Z","added_at":"2026-11-13T20:13:35.566Z","updated_at":"2026-11-13T20:13:35.566Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/concurrent-shell-command-queues","markdown_url":"https://listedarticles.com/articles/concurrent-shell-command-queues.md","example":false,"citation":"Christian Emmer, Christian Emmer. \"Concurrent Shell Command Queues.\" 7 Nov 2026. https://emmer.dev/blog/concurrent-shell-command-queues/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://emmer.dev/blog/concurrent-shell-command-queues/"},"body_markdown":"# Concurrent Shell Command Queues\n\n![Christian Emmer](https://www.gravatar.com/avatar/68035821d5420b058bbd2699312c276c?r=g&d=mp&s=56)\n\nChristian Emmer\n\nNov 6, 2026\\* · 10 min read\n\n![Concurrent Shell Command Queues](https://emmer.dev/static/img/blog/concurrent-shell-command-queues.jpg)\n\nHow to run shell commands in parallel while limiting their concurrency.\n\nI had a use case recently where I wanted to speed up some Bash processing that I was doing, but still keep concurrency constrained. So I wondered: *is there a straightforward way to create thread pools in Bash?* The answer is: yes, but it takes some work to be semi-resilient and portable.\n\nYou may ask: *why do this in Bash at all? It seems silly to use a language with poor concurrency controls.* You'd be right to ask this question, and my answer is: because [I have a fondness for Bash](https://emmer.dev/blog/tag/shell), and I wanted the challenge. But you're still probably better off using another language, such as [Python](https://emmer.dev/blog/tag/python).\n\nHere is code that creates a command queue that accepts shell commands to execute. You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.\n\nbash\n\n```\n1# Start this shell's command queue.\n2# @param {number} $1 Maximum number of jobs to run at once\n3queue_start() {\n4  # Validate that the parallelism is a valid limit for xargs\n5  if [[ ! ${1:-} =~ ^[1-9][0-9]*$ ]]; then\n6    echo \"queue_start: size must be a positive integer, got '${1:-}'\" >&2\n7    return 1\n8  fi\n9\n\n10  # Close this shell's copy (not the parent's) of an inherited or earlier\n11  # command queue's file descriptor\n12  if [[ -n ${_queue_fd:-} ]]; then\n13    eval \"exec ${_queue_fd}>&-\"\n14    unset _queue_fd _queue_pid\n15  fi\n16\n\n17  # Find the first unused fd from 3 to 9 (zsh's redirect max)\n18  local fd=3\n19  while { : <&\"${fd}\"; } 2>/dev/null; do fd=$(( fd + 1 )); done\n20  if [[ ${fd} -gt 9 ]]; then\n21    echo \"queue_start: no free fd from 3 to 9\" >&2\n22    return 1\n23  fi\n24\n\n25  # Make a FIFO queue file in a private temporary directory\n26  local dir\n27  dir=$(mktemp -d \"${TMPDIR:-/tmp}/queue.XXXXXX\")\n28  mkfifo \"${dir}/queue\"\n29\n\n30  # Find the shell to execute commands (the current shell)\n31  local shell\n32  shell=$(ps -o comm= -p \"$$\")\n33  shell=${shell#-}\n34\n\n35  # Start xargs in the background, reading NUL-separated commands from the\n36  # queue file. Job messages are hidden by turning off job control and\n37  # discarding the block's stderr. xargs gets the real stderr through a copy on\n38  # the queue files's fd, until the file is opened.\n39  eval \"exec ${fd}>&2\"\n40  local opts=$-\n41  set +m\n42  {\n43    # shellcheck disable=SC2016\n44    xargs -0 -n 1 -P \"$1\" \"${shell}\" -c '\n45      # Output is captured rather than streamed so that command output stays separate\n46      out=$(eval -- \"$1\" </dev/null 2>&1); rc=$?\n47      [ -z \"${out}\" ] || printf \"%s\\n\" \"${out}\"\n48      # Coerce non-zero exit codes to 1, because a 255 code would stop xargs\n49      [ \"${rc}\" -eq 0 ] || { printf \"exit %d: %s\\n\" \"${rc}\" \"$1\" >&2; exit 1;\n50    }' _ <\"${dir}/queue\" 2>&\"${fd}\" &\n51    _queue_pid=$!\n52    disown || :\n53  } 2>/dev/null\n54  if [[ ${opts} == *m* ]]; then set -m; fi\n55\n\n56  # Open the queue file for writing, which waits until xargs has opened it for reading\n57  # With both ends open, the file can be safely deleted\n58  eval \"exec ${fd}>\\\"\\${dir}/queue\\\"\"\n59  _queue_fd=${fd}\n60  rm -rf \"${dir}\"\n61\n\n62  # Register an exit handler to terminate this PID's children, ensuring zsh behaves like bash\n63  setopt localoptions posixtraps 2>/dev/null || :\n64  trap _queue_stop EXIT\n65  if [[ $- != *i* ]]; then\n66    trap 'exit 129' HUP\n67    trap 'exit 130' INT\n68    trap 'exit 143' TERM\n69  fi\n70}\n71\n\n72# Queue a command on this shell's command queue.\n73# @param {string} $1 Command to run\n74queue_submit() {\n75  if [[ -z ${_queue_fd:-} ]]; then\n76    echo \"queue_submit: no command queue started in this shell\" >&2\n77    return 1\n78  fi\n79  # Write the command to the queue file, blocking if it is full\n80  # Use a printf process so that if xargs has exited, printf will get killed and not this shell\n81  if [[ $- == *i* ]]; then\n82    # Write the command in the foreground, because an interactive shell won't\n83    # have any TERM/HUP/INT traps that could be delayed by a full queue\n84    env printf '%s\\0' \"$1\" >&\"${_queue_fd}\"\n85  else\n86    # Write the command in the background, so that a full queue doesn't delay any TERM/HUP/INT traps\n87    env printf '%s\\0' \"$1\" >&\"${_queue_fd}\" &\n88    wait \"$!\"\n89  fi || {\n90    echo \"queue_submit: couldn't queue the command; the command queue may have stopped\" >&2\n91    return 1\n92  }\n93}\n94\n\n95# Close the queue file so xargs finishes the jobs it has, then wait for it to exit\n96queue_wait() {\n97  if [[ -z ${_queue_fd:-} ]]; then\n98    echo \"queue_wait: no command queue started in this shell\" >&2\n99    return 1\n100  fi\n101  eval \"exec ${_queue_fd}>&-\"\n102  while kill -0 \"${_queue_pid}\" 2>/dev/null; do sleep 0.1; done\n103  unset _queue_fd _queue_pid\n104}\n105\n\n106# Stop every process below the given PID.\n107# @param {number=} $1 PID whose descendants to stop, defaulting to this shell's PID\n108_queue_stop() {\n109  # Default to this shell's PID, found from a child's parent PID because $$ is\n110  # the parent's PID in a subshell\n111  local parent=${1:-$(exec sh -c 'echo \"${PPID}\"')}\n112  local pid\n113  for pid in $(exec pgrep -P \"${parent}\"); do\n114    # Pause the process so it can't start any new children\n115    kill -STOP \"${pid}\" 2>/dev/null || continue\n116    # Terminate the process' descendants, then terminate the process\n117    _queue_stop \"${pid}\"\n118    kill -TERM \"${pid}\" 2>/dev/null || :\n119    # Resume the process so it can respond to the termination\n120    kill -CONT \"${pid}\" 2>/dev/null || :\n121  done\n122}\n123\n\n124# Stop all background processes on exit, because they ignore Ctrl-C and don't receive TERM\n125if [[ $- != *i* ]]; then\n126  trap _queue_stop EXIT\n127  trap 'exit 129' HUP\n128  trap 'exit 130' INT\n129  trap 'exit 143' TERM\n130fi\n```\n\n`queue_start` initializes the queue (managed by [`mkfifo`](https://man7.org/linux/man-pages/man1/mkfifo.1.html)) and starts the executor (concurrency managed by `xargs -P`), `queue_submit` submits commands to the queue, and `queue_wait` waits for the queue to drain before cleaning up. Calling `queue_wait` is optional, command processing will start as soon as you queue it with `queue_submit`.\n\nYou can copy and paste this code into a script, or it's also safe to put in your dotfiles, as it only sets some of the exit traps for interactive shells. It works in Bash and Zsh, and it has as much defensive programming as I know how to add, but I'm confident it has failure modes that are difficult to predict or avoid.\n\n## Example usage\n\nMy original use case had to do with making a large number of web requests, so here's an example using [Postman Echo](https://www.postman.com/postman/published-postman-templates/documentation/631643-f695cab7-6878-eb55-7943-ad88e1ccfd65) :\n\n● ● ●\n\nshell\n\n```\n$ queue_start 3\n$ for n in 1 2 3 4 5; do\n    queue_submit \"set -o pipefail; \\\n      curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \\\n        -d '{\\\"request\\\": ${n}}' \\\n        https://postman-echo.com/post | \\\n      jq -c .json\"\n  done\n$ queue_wait\n{\"request\":1}\n{\"request\":2}\n{\"request\":3}\n{\"request\":4}\n{\"request\":5}\n```\n\nA more deterministic scenario demonstrating that concurrency is respected, only starting command three after command two has completed:\n\n● ● ●\n\nshell\n\n```\n$ queue_start 2\n$ queue_submit 'sleep 1; echo first'\n$ queue_submit 'echo second'\n$ queue_submit 'echo third'\n$ queue_wait\nsecond\nthird\nfirst\n```\n\nCommand queues are inherited by subshells:\n\n● ● ●\n\nshell\n\n```\n$ queue_start 2\n$ ( queue_submit 'echo \"submitted from a subshell\"' )\n$ queue_submit 'echo \"submitted from the main shell\"'\n$ queue_wait\nsubmitted from a subshell\nsubmitted from the main shell\n```\n\nDifferent subshells can have their own command queue:\n\n● ● ●\n\nshell\n\n```\n$ (\n    queue_start 2\n    for n in 1 2 3 4; do queue_submit \"sleep 1; echo first queue, job ${n}\"; done\n    queue_wait\n  ) &\n$ first=$!\n$ (\n    queue_start 1\n    for n in 1 2; do queue_submit \"sleep 1; echo second queue, job ${n}\"; done\n    queue_wait\n  ) &\n$ second=$!\n$ wait \"${first}\"\n$ wait \"${second}\"\nsecond queue, job 1\nfirst queue, job 1\nfirst queue, job 2\nsecond queue, job 2\nfirst queue, job 3\nfirst queue, job 4\n```\n\nQueueing each line of a command's output as a command to execute:\n\n● ● ●\n\nshell\n\n```\n$ queue_start 2\n$ while IFS= read -r line; do\n    queue_submit \"${line}\"\n  done < <(printf 'echo \"from command output, line %d\"\\n' 1 2 3)\n$ queue_wait\nfrom command output, line 1\nfrom command output, line 2\nfrom command output, line 3\n```\n\nQueueing each line of a file as a command:\n\n● ● ●\n\nshell\n\n```\n$ queue_start 2\n$ while IFS= read -r line; do\n    queue_submit \"${line}\"\n  done < commands.txt\n$ queue_wait\n```\n\n## Known issues\n\nHere are some reasons why you might just want a different programming language:\n\n- **A shell can only have one queue.**\n\n  The amount of additional code needed to support multiple is complicated and likely isn't needed in most cases. Subshells can still have their own queues.\n- **Command output is intentionally not streamed.**\n\n  This prevents interleaving the output of most commands. Output larger than `PIPE_BUF` may still get interleaved.\n- `queue_wait` doesn't return a value indicating any job failure.\\*\\*\n\n  This would require a temporary file or a second file descriptor and some complicated code.\n- **`queue_wait` has to wait for running processes started after `queue_start`.**\n\n  Processes started after `queue_start` will inherit the queue write file descriptor (which is what allows subshells to work), and `xargs` won't see the end of the queue until every copy of the write descriptor is closed.\n- **`_queue_stop` indiscriminately terminates descendants.**\n\n  Any other backgrounded jobs will be terminated, including ones started before `queue_start`.\n\n## Why not just `xargs -P`?\n\nIf you know all commands that you want to execute up-front, then the solution gets *far* simpler:\n\n● ● ●\n\nshell\n\n```\n$ printf 'echo \"from command output, line %d\"\\0' 1 2 3 | \\\n    xargs -0 -n 1 -P 2 bash -c\nfrom command output, line 1\nfrom command output, line 2\nfrom command output, line 3\n```\n\nor\n\n● ● ●\n\nshell\n\n```\n$ tr '\\n' '\\0' < commands.txt | \\\n    xargs -0 -n 1 -P 2 bash -c\n```\n\n[GNU `parallel`](https://www.gnu.org/software/parallel/)  works similarly, though macOS doesn't ship with it:\n\n● ● ●\n\nshell\n\n```\n$ printf 'echo \"from command output, line %d\"\\n' 1 2 3 | \\\n    parallel -j 2\nfrom command output, line 1\nfrom command output, line 2\nfrom command output, line 3\n```\n\n[shell](https://emmer.dev/blog/tag/shell \"shell\")\n","body_html":"<h1 id=\"concurrent-shell-command-queues\">Concurrent Shell Command Queues</h1>\n<figure><img src=\"https://www.gravatar.com/avatar/68035821d5420b058bbd2699312c276c?r=g&amp;d=mp&amp;s=56\" alt=\"Christian Emmer\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>Christian Emmer</p>\n<p>Nov 6, 2026* · 10 min read</p>\n<figure><img src=\"https://emmer.dev/static/img/blog/concurrent-shell-command-queues.jpg\" alt=\"Concurrent Shell Command Queues\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>How to run shell commands in parallel while limiting their concurrency.</p>\n<p>I had a use case recently where I wanted to speed up some Bash processing that I was doing, but still keep concurrency constrained. So I wondered: <em>is there a straightforward way to create thread pools in Bash?</em> The answer is: yes, but it takes some work to be semi-resilient and portable.</p>\n<p>You may ask: <em>why do this in Bash at all? It seems silly to use a language with poor concurrency controls.</em> You&#39;d be right to ask this question, and my answer is: because <a href=\"https://emmer.dev/blog/tag/shell\" rel=\"nofollow ugc noopener\">I have a fondness for Bash</a>, and I wanted the challenge. But you&#39;re still probably better off using another language, such as <a href=\"https://emmer.dev/blog/tag/python\" rel=\"nofollow ugc noopener\">Python</a>.</p>\n<p>Here is code that creates a command queue that accepts shell commands to execute. You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.</p>\n<p>bash</p>\n<pre><code>1# Start this shell&#39;s command queue.\n2# @param {number} $1 Maximum number of jobs to run at once\n3queue_start() {\n4  # Validate that the parallelism is a valid limit for xargs\n5  if [[ ! ${1:-} =~ ^[1-9][0-9]*$ ]]; then\n6    echo &quot;queue_start: size must be a positive integer, got &#39;${1:-}&#39;&quot; &gt;&amp;2\n7    return 1\n8  fi\n9\n\n10  # Close this shell&#39;s copy (not the parent&#39;s) of an inherited or earlier\n11  # command queue&#39;s file descriptor\n12  if [[ -n ${_queue_fd:-} ]]; then\n13    eval &quot;exec ${_queue_fd}&gt;&amp;-&quot;\n14    unset _queue_fd _queue_pid\n15  fi\n16\n\n17  # Find the first unused fd from 3 to 9 (zsh&#39;s redirect max)\n18  local fd=3\n19  while { : &lt;&amp;&quot;${fd}&quot;; } 2&gt;/dev/null; do fd=$(( fd + 1 )); done\n20  if [[ ${fd} -gt 9 ]]; then\n21    echo &quot;queue_start: no free fd from 3 to 9&quot; &gt;&amp;2\n22    return 1\n23  fi\n24\n\n25  # Make a FIFO queue file in a private temporary directory\n26  local dir\n27  dir=$(mktemp -d &quot;${TMPDIR:-/tmp}/queue.XXXXXX&quot;)\n28  mkfifo &quot;${dir}/queue&quot;\n29\n\n30  # Find the shell to execute commands (the current shell)\n31  local shell\n32  shell=$(ps -o comm= -p &quot;$$&quot;)\n33  shell=${shell#-}\n34\n\n35  # Start xargs in the background, reading NUL-separated commands from the\n36  # queue file. Job messages are hidden by turning off job control and\n37  # discarding the block&#39;s stderr. xargs gets the real stderr through a copy on\n38  # the queue files&#39;s fd, until the file is opened.\n39  eval &quot;exec ${fd}&gt;&amp;2&quot;\n40  local opts=$-\n41  set +m\n42  {\n43    # shellcheck disable=SC2016\n44    xargs -0 -n 1 -P &quot;$1&quot; &quot;${shell}&quot; -c &#39;\n45      # Output is captured rather than streamed so that command output stays separate\n46      out=$(eval -- &quot;$1&quot; &lt;/dev/null 2&gt;&amp;1); rc=$?\n47      [ -z &quot;${out}&quot; ] || printf &quot;%s\\n&quot; &quot;${out}&quot;\n48      # Coerce non-zero exit codes to 1, because a 255 code would stop xargs\n49      [ &quot;${rc}&quot; -eq 0 ] || { printf &quot;exit %d: %s\\n&quot; &quot;${rc}&quot; &quot;$1&quot; &gt;&amp;2; exit 1;\n50    }&#39; _ &lt;&quot;${dir}/queue&quot; 2&gt;&amp;&quot;${fd}&quot; &amp;\n51    _queue_pid=$!\n52    disown || :\n53  } 2&gt;/dev/null\n54  if [[ ${opts} == *m* ]]; then set -m; fi\n55\n\n56  # Open the queue file for writing, which waits until xargs has opened it for reading\n57  # With both ends open, the file can be safely deleted\n58  eval &quot;exec ${fd}&gt;\\&quot;\\${dir}/queue\\&quot;&quot;\n59  _queue_fd=${fd}\n60  rm -rf &quot;${dir}&quot;\n61\n\n62  # Register an exit handler to terminate this PID&#39;s children, ensuring zsh behaves like bash\n63  setopt localoptions posixtraps 2&gt;/dev/null || :\n64  trap _queue_stop EXIT\n65  if [[ $- != *i* ]]; then\n66    trap &#39;exit 129&#39; HUP\n67    trap &#39;exit 130&#39; INT\n68    trap &#39;exit 143&#39; TERM\n69  fi\n70}\n71\n\n72# Queue a command on this shell&#39;s command queue.\n73# @param {string} $1 Command to run\n74queue_submit() {\n75  if [[ -z ${_queue_fd:-} ]]; then\n76    echo &quot;queue_submit: no command queue started in this shell&quot; &gt;&amp;2\n77    return 1\n78  fi\n79  # Write the command to the queue file, blocking if it is full\n80  # Use a printf process so that if xargs has exited, printf will get killed and not this shell\n81  if [[ $- == *i* ]]; then\n82    # Write the command in the foreground, because an interactive shell won&#39;t\n83    # have any TERM/HUP/INT traps that could be delayed by a full queue\n84    env printf &#39;%s\\0&#39; &quot;$1&quot; &gt;&amp;&quot;${_queue_fd}&quot;\n85  else\n86    # Write the command in the background, so that a full queue doesn&#39;t delay any TERM/HUP/INT traps\n87    env printf &#39;%s\\0&#39; &quot;$1&quot; &gt;&amp;&quot;${_queue_fd}&quot; &amp;\n88    wait &quot;$!&quot;\n89  fi || {\n90    echo &quot;queue_submit: couldn&#39;t queue the command; the command queue may have stopped&quot; &gt;&amp;2\n91    return 1\n92  }\n93}\n94\n\n95# Close the queue file so xargs finishes the jobs it has, then wait for it to exit\n96queue_wait() {\n97  if [[ -z ${_queue_fd:-} ]]; then\n98    echo &quot;queue_wait: no command queue started in this shell&quot; &gt;&amp;2\n99    return 1\n100  fi\n101  eval &quot;exec ${_queue_fd}&gt;&amp;-&quot;\n102  while kill -0 &quot;${_queue_pid}&quot; 2&gt;/dev/null; do sleep 0.1; done\n103  unset _queue_fd _queue_pid\n104}\n105\n\n106# Stop every process below the given PID.\n107# @param {number=} $1 PID whose descendants to stop, defaulting to this shell&#39;s PID\n108_queue_stop() {\n109  # Default to this shell&#39;s PID, found from a child&#39;s parent PID because $$ is\n110  # the parent&#39;s PID in a subshell\n111  local parent=${1:-$(exec sh -c &#39;echo &quot;${PPID}&quot;&#39;)}\n112  local pid\n113  for pid in $(exec pgrep -P &quot;${parent}&quot;); do\n114    # Pause the process so it can&#39;t start any new children\n115    kill -STOP &quot;${pid}&quot; 2&gt;/dev/null || continue\n116    # Terminate the process&#39; descendants, then terminate the process\n117    _queue_stop &quot;${pid}&quot;\n118    kill -TERM &quot;${pid}&quot; 2&gt;/dev/null || :\n119    # Resume the process so it can respond to the termination\n120    kill -CONT &quot;${pid}&quot; 2&gt;/dev/null || :\n121  done\n122}\n123\n\n124# Stop all background processes on exit, because they ignore Ctrl-C and don&#39;t receive TERM\n125if [[ $- != *i* ]]; then\n126  trap _queue_stop EXIT\n127  trap &#39;exit 129&#39; HUP\n128  trap &#39;exit 130&#39; INT\n129  trap &#39;exit 143&#39; TERM\n130fi</code></pre>\n<p><code>queue_start</code> initializes the queue (managed by <a href=\"https://man7.org/linux/man-pages/man1/mkfifo.1.html\" rel=\"nofollow ugc noopener\"><code>mkfifo</code></a>) and starts the executor (concurrency managed by <code>xargs -P</code>), <code>queue_submit</code> submits commands to the queue, and <code>queue_wait</code> waits for the queue to drain before cleaning up. Calling <code>queue_wait</code> is optional, command processing will start as soon as you queue it with <code>queue_submit</code>.</p>\n<p>You can copy and paste this code into a script, or it&#39;s also safe to put in your dotfiles, as it only sets some of the exit traps for interactive shells. It works in Bash and Zsh, and it has as much defensive programming as I know how to add, but I&#39;m confident it has failure modes that are difficult to predict or avoid.</p>\n<h2 id=\"example-usage\">Example usage</h2>\n<p>My original use case had to do with making a large number of web requests, so here&#39;s an example using <a href=\"https://www.postman.com/postman/published-postman-templates/documentation/631643-f695cab7-6878-eb55-7943-ad88e1ccfd65\" rel=\"nofollow ugc noopener\">Postman Echo</a> :</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ queue_start 3\n$ for n in 1 2 3 4 5; do\n    queue_submit &quot;set -o pipefail; \\\n      curl -sS --fail-with-body --max-time 10 -X POST -H &#39;Content-Type: application/json&#39; \\\n        -d &#39;{\\&quot;request\\&quot;: ${n}}&#39; \\\n        https://postman-echo.com/post | \\\n      jq -c .json&quot;\n  done\n$ queue_wait\n{&quot;request&quot;:1}\n{&quot;request&quot;:2}\n{&quot;request&quot;:3}\n{&quot;request&quot;:4}\n{&quot;request&quot;:5}</code></pre>\n<p>A more deterministic scenario demonstrating that concurrency is respected, only starting command three after command two has completed:</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ queue_start 2\n$ queue_submit &#39;sleep 1; echo first&#39;\n$ queue_submit &#39;echo second&#39;\n$ queue_submit &#39;echo third&#39;\n$ queue_wait\nsecond\nthird\nfirst</code></pre>\n<p>Command queues are inherited by subshells:</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ queue_start 2\n$ ( queue_submit &#39;echo &quot;submitted from a subshell&quot;&#39; )\n$ queue_submit &#39;echo &quot;submitted from the main shell&quot;&#39;\n$ queue_wait\nsubmitted from a subshell\nsubmitted from the main shell</code></pre>\n<p>Different subshells can have their own command queue:</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ (\n    queue_start 2\n    for n in 1 2 3 4; do queue_submit &quot;sleep 1; echo first queue, job ${n}&quot;; done\n    queue_wait\n  ) &amp;\n$ first=$!\n$ (\n    queue_start 1\n    for n in 1 2; do queue_submit &quot;sleep 1; echo second queue, job ${n}&quot;; done\n    queue_wait\n  ) &amp;\n$ second=$!\n$ wait &quot;${first}&quot;\n$ wait &quot;${second}&quot;\nsecond queue, job 1\nfirst queue, job 1\nfirst queue, job 2\nsecond queue, job 2\nfirst queue, job 3\nfirst queue, job 4</code></pre>\n<p>Queueing each line of a command&#39;s output as a command to execute:</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ queue_start 2\n$ while IFS= read -r line; do\n    queue_submit &quot;${line}&quot;\n  done &lt; &lt;(printf &#39;echo &quot;from command output, line %d&quot;\\n&#39; 1 2 3)\n$ queue_wait\nfrom command output, line 1\nfrom command output, line 2\nfrom command output, line 3</code></pre>\n<p>Queueing each line of a file as a command:</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ queue_start 2\n$ while IFS= read -r line; do\n    queue_submit &quot;${line}&quot;\n  done &lt; commands.txt\n$ queue_wait</code></pre>\n<h2 id=\"known-issues\">Known issues</h2>\n<p>Here are some reasons why you might just want a different programming language:</p>\n<ul><li><p><strong>A shell can only have one queue.</strong></p><p>The amount of additional code needed to support multiple is complicated and likely isn&#39;t needed in most cases. Subshells can still have their own queues.</p></li><li><p><strong>Command output is intentionally not streamed.</strong></p><p>This prevents interleaving the output of most commands. Output larger than <code>PIPE_BUF</code> may still get interleaved.</p></li><li><p><code>queue_wait</code> doesn&#39;t return a value indicating any job failure.**</p><p>This would require a temporary file or a second file descriptor and some complicated code.</p></li><li><p><strong><code>queue_wait</code> has to wait for running processes started after <code>queue_start</code>.</strong></p><p>Processes started after <code>queue_start</code> will inherit the queue write file descriptor (which is what allows subshells to work), and <code>xargs</code> won&#39;t see the end of the queue until every copy of the write descriptor is closed.</p></li><li><p><strong><code>_queue_stop</code> indiscriminately terminates descendants.</strong></p><p>Any other backgrounded jobs will be terminated, including ones started before <code>queue_start</code>.</p></li></ul>\n<h2 id=\"why-not-just-xargs-p\">Why not just <code>xargs -P</code>?</h2>\n<p>If you know all commands that you want to execute up-front, then the solution gets <em>far</em> simpler:</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ printf &#39;echo &quot;from command output, line %d&quot;\\0&#39; 1 2 3 | \\\n    xargs -0 -n 1 -P 2 bash -c\nfrom command output, line 1\nfrom command output, line 2\nfrom command output, line 3</code></pre>\n<p>or</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ tr &#39;\\n&#39; &#39;\\0&#39; &lt; commands.txt | \\\n    xargs -0 -n 1 -P 2 bash -c</code></pre>\n<p><a href=\"https://www.gnu.org/software/parallel/\" rel=\"nofollow ugc noopener\">GNU <code>parallel</code></a>  works similarly, though macOS doesn&#39;t ship with it:</p>\n<p>● ● ●</p>\n<p>shell</p>\n<pre><code>$ printf &#39;echo &quot;from command output, line %d&quot;\\n&#39; 1 2 3 | \\\n    parallel -j 2\nfrom command output, line 1\nfrom command output, line 2\nfrom command output, line 3</code></pre>\n<p><a href=\"https://emmer.dev/blog/tag/shell\" title=\"shell\" rel=\"nofollow ugc noopener\">shell</a></p>","headings":[{"level":1,"text":"Concurrent Shell Command Queues","id":"concurrent-shell-command-queues"},{"level":2,"text":"Example usage","id":"example-usage"},{"level":2,"text":"Known issues","id":"known-issues"},{"level":2,"text":"Why not just xargs -P?","id":"why-not-just-xargs-p"}]}}