Manual - aaronriekenberg/rust-parallel GitHub Wiki

Manual for rust-parallel 1.24.0

  1. Command line options
  2. Commands from arguments
    1. Automatic variables
  3. Commands from stdin
  4. Command and initial arguments on command line
  5. Reading multiple inputs
  6. Pipe Mode
  7. Parallelism
  8. Keep Output Order
  9. Dry run
  10. Debug logging
  11. Error handling
  12. Timeout
  13. Path cache
  14. Progress bar
  15. Regular Expression
    1. Named Capture Groups
    2. Numbered Capture Groups
    3. Capture Group Special Characters
  16. Shell Commands
  17. Bash Function
    1. Function Setup
    2. Demo of command line arguments
    3. Demo of function and command line arguments from stdin
    4. Demo of function and initial arguments on command line, additional arguments from stdin
  18. Shell Quote

Command line options

$ rust-parallel --help
Execute commands in parallel

By Aaron Riekenberg <[email protected]>

https://github.com/aaronriekenberg/rust-parallel
https://crates.io/crates/rust-parallel

Usage: rust-parallel [OPTIONS] [COMMAND_AND_INITIAL_ARGUMENTS]...

Arguments:
  [COMMAND_AND_INITIAL_ARGUMENTS]...
          Optional command and initial arguments.
          
          If this contains 1 or more ::: delimiters the cartesian product of arguments from all groups are run.

Options:
  -d, --discard-output <DISCARD_OUTPUT>
          Discard output for commands

          Possible values:
          - stdout: Redirect stdout for commands to /dev/null
          - stderr: Redirect stderr for commands to /dev/null
          - all:    Redirect stdout and stderr for commands to /dev/null

  -i, --input-file <INPUT_FILE>
          Input file or - for stdin.  Defaults to stdin if no inputs are specified

  -j, --jobs <JOBS>
          Maximum number of commands to run in parallel, defaults to num cpus. Accepts an integer (e.g. 4) or a percentage of available CPUs (e.g. 200%)
          
          [default: 8]

  -0, --null-separator
          Use null separator for reading input files instead of newline

  -p, --progress-bar
          Display progress bar

      --progress-bar-style <PROGRESS_BAR_STYLE>
          Progress bar style

  -r, --regex <REGEX>
          Apply regex pattern to inputs

  -s, --shell
          Use shell mode for running commands.
          
          Each command line is passed to "<shell-path> <shell-argument>" as a single argument.

      --shell-quote
          Use shell quoting for command arguments.
          
          Each command line is passed to "<shell-path> <shell-argument>" as a single argument with shell quoting applied to each argument.

  -t, --timeout-seconds <TIMEOUT_SECONDS>
          Timeout seconds for running commands.  Defaults to infinite timeout if not specified

      --channel-capacity <CHANNEL_CAPACITY>
          Input and output channel capacity, defaults to num cpus * 2
          
          [default: 16]

      --disable-path-cache
          Disable command path cache

      --dry-run
          Dry run mode
          
          Do not actually run commands just log.

      --exit-on-error
          Exit on error mode
          
          Exit immediately when a command fails.

      --no-run-if-empty
          Do not run commands for empty buffered input lines

  -k, --keep-order
          Keep output in the same order as input

      --pipe
          Use pipe input mode

      --block-size <BLOCK_SIZE>
          Block size for pipe input mode, defaults to 1MiB
          
          [default: 1MiB]

      --shell-path <SHELL_PATH>
          Path to shell to use for shell mode
          
          [default: bash]

      --shell-argument <SHELL_ARGUMENT>
          Argument to shell for shell mode
          
          [default: -c]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Commands from arguments

The ::: separator can be used to run the Cartesian Product of command line arguments. This is similar to the ::: behavior in GNU Parallel.

$ rust-parallel echo ::: A B ::: C D ::: E F G
A D G
B C E
B C F
A C F
A C E
A C G
A D F
A D E
B D E
B C G
B D F
B D G

$ rust-parallel echo hello ::: larry curly moe
hello larry
hello moe
hello curly

# run gzip -k on all *.html files in current directory
$ rust-parallel gzip -k ::: *.html

Automatic Variables

When using commands from arguments, numbered variables {0}, {1}, etc are automatically available based on the number of arguments. {0} will be replaced by the entire input line, and other groups match individual argument groups. {} is the same as {0}. This is useful for building more complex command lines. For example:

$ rust-parallel echo group0={0} group1={1} group2={2} group3={3} group2again={2} ::: A B ::: C D ::: E F G
group0=A C F group1=A group2=C group3=F group2again=C
group0=A D G group1=A group2=D group3=G group2again=D
group0=B C E group1=B group2=C group3=E group2again=C
group0=A D F group1=A group2=D group3=F group2again=D
group0=A C E group1=A group2=C group3=E group2again=C
group0=A D E group1=A group2=D group3=E group2again=D
group0=A C G group1=A group2=C group3=G group2again=C
group0=B C F group1=B group2=C group3=F group2again=C
group0=B D F group1=B group2=D group3=F group2again=D
group0=B D E group1=B group2=D group3=E group2again=D
group0=B C G group1=B group2=C group3=G group2again=C
group0=B D G group1=B group2=D group3=G group2again=D

$ rust-parallel echo entireline={} group1={1} group2={2} group3={3} group2again={2} ::: A B ::: C D ::: E F G
entireline=A C F group1=A group2=C group3=F group2again=C
entireline=A D G group1=A group2=D group3=G group2again=D
entireline=A D F group1=A group2=D group3=F group2again=D
entireline=A C G group1=A group2=C group3=G group2again=C
entireline=A D E group1=A group2=D group3=E group2again=D
entireline=B C E group1=B group2=C group3=E group2again=C
entireline=B C F group1=B group2=C group3=F group2again=C
entireline=A C E group1=A group2=C group3=E group2again=C
entireline=B C G group1=B group2=C group3=G group2again=C
entireline=B D G group1=B group2=D group3=G group2again=D
entireline=B D E group1=B group2=D group3=E group2again=D
entireline=B D F group1=B group2=D group3=F group2again=D

Internally these variables are implemented using an auto-generated regular expression. If a regular expression is manually specified this will override the auto-generated one.

Commands from stdin

Run complete commands from stdin.

$ cat >./test <<EOL
echo hi
echo there
echo how
echo are
echo you
EOL

$ cat test | rust-parallel
hi
how
are
there
you

Command and initial arguments on command line

Here md5 -s will be prepended to each input line to form a command like md5 -s aal

$ head -100 /usr/share/dict/words | rust-parallel md5 -s | head -10
7fc56270e7a70fa81a5935b72eacbe29
88571e5d5e13a4a60f82cea7802f6255
35c2d90f7c06b623fe763d0a4e5b7ed9
4124bc0a9335c27f086f24ba207a4912
ff45e881572ca2c987460932660d320c
0cc175b9c0f1b6a831c399e269772661
e9b22dd6213c3d29648e8ad7a8642f2f
0a1ea2a8d75d02ae052f8222e36927a5
1c0a11cc4ddc0dbd3fa4d77232a4e22e
00fde3db4e2f61e04831a968a0c13108

Reading multiple inputs

By default rust-parallel reads input from stdin only. The -i option can be used 1 or more times to override this behavior. -i - means read from stdin, -i ./test means read from the file ./test:

$ cat >./test <<EOL
foo
bar
baz
EOL

$ head -5 /usr/share/dict/words | rust-parallel -i - -i ./test echo
bar
aal
A
a
aa
baz
foo
aalii

Pipe Mode

The --pipe option can be used to enable pipe mode.

In pipe mode input from stdin is split into blocks and each block is passed to a separate instance of the command via stdin. Command instances are run in parallel.

The default block size is 1 MiB, which can be changed with the --block-size option.

By default input blocks are split on new line boundaries. This can be changed to split on null boundaries with the -0/--null-separator option.

Here we use --pipe to run wc -l

$ cat /usr/share/dict/words | rust-parallel --pipe wc -l
   99857
   98763
   37356

Here we use pipe mode with with a smaller block size of 500 KiB:

$ cat /usr/share/dict/words | rust-parallel --pipe --block-size=500KiB wc -l
   49400
   48066
   48406
   48237
   41867

Parallelism

By default the number of parallel jobs to run simultaneously is the number of cpus detected at run time.

This can be overridden with the -j/--jobs option.

With -j5 all echo commands below run in parallel.

With -j1 all jobs run sequentially.

A percentage of available CPUs can also be specified using the % suffix. For example -j200% runs twice as many jobs as there are CPUs, and -j50% runs half as many. The computed value is always at least 1.

$ rust-parallel -j5 echo ::: hi there how are you
hi
there
you
are
how

$ rust-parallel -j1 echo ::: hi there how are you
hi
there
how
are
you

$ rust-parallel -j200% echo ::: hi there how are you
are
how
hi
you
there

Keep Output Order

By default, command outputs are displayed as soon as each command completes, which may not be in the same order as the input.

Use option -k/--keep-order to ensure outputs are displayed in the same order as the input.

With -k all outputs will be displayed in the same order as the input, regardless of when commands complete.

$ rust-parallel -k echo ::: hi there how are you
hi
there
how
are
you

$ rust-parallel -j1 -k echo ::: hi there how are you
hi
there
how
are
you

Dry run

Use option --dry-run for dry run mode.

In this mode the commands that would be run are ouput as info level logs.

No commands are actually run - this is useful for testing before running a job.

$ rust-parallel --dry-run echo ::: hi there how are you
2026-08-15T16:27:39.874153Z  INFO rust_parallel::command: cmd="/bin/echo",args=["hi"],stdin=None,line=command_line_args:1
2026-08-15T16:27:39.874182Z  INFO rust_parallel::command: cmd="/bin/echo",args=["there"],stdin=None,line=command_line_args:2
2026-08-15T16:27:39.874197Z  INFO rust_parallel::command: cmd="/bin/echo",args=["how"],stdin=None,line=command_line_args:3
2026-08-15T16:27:39.874209Z  INFO rust_parallel::command: cmd="/bin/echo",args=["are"],stdin=None,line=command_line_args:4
2026-08-15T16:27:39.874217Z  INFO rust_parallel::command: cmd="/bin/echo",args=["you"],stdin=None,line=command_line_args:5

Debug logging

Set environment variable RUST_LOG=debug to see debug output.

This logs structured information about command line arguments and commands being run.

Recommend enabling debug logging for all examples to understand what is happening in more detail.

$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args | head -1
2026-08-15T16:27:39.877990Z DEBUG try_main: rust_parallel::command_line_args: command_line_args = CommandLineArgs { discard_output: None, input_file: [], jobs: 8, null_separator: false, progress_bar: false, progress_bar_style: None, regex: None, shell: false, shell_quote: false, timeout_seconds: None, channel_capacity: 16, disable_path_cache: false, dry_run: false, exit_on_error: false, no_run_if_empty: false, keep_order: false, pipe: false, block_size: 1048576, shell_path: "bash", shell_argument: "-c", command_and_initial_arguments: ["echo", ":::", "hi", "there", "how", "are", "you"] }

$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args:1
2026-08-15T16:27:39.885296Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None}: rust_parallel::command: begin run
2026-08-15T16:27:39.885515Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None child_pid=48039}: rust_parallel::command: spawned child process, awaiting completion
2026-08-15T16:27:39.886716Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None child_pid=48039}: rust_parallel::command: command exit status = exit status: 0
2026-08-15T16:27:39.886729Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None child_pid=48039}: rust_parallel::command: end run

If pipe mode is being used debug logs will show stdin line ranges and block sizes for each command:

$ cat /usr/share/dict/words| RUST_LOG=debug rust-parallel --pipe wc -l | grep spawned
2026-08-15T16:27:39.932768Z DEBUG Command::run{cmd="/usr/bin/wc" args=["-l"] line=stdin:1-99857 stdin=1.0 MiB child_pid=48047}: rust_parallel::command: spawned child process, awaiting completion
2026-08-15T16:27:39.973023Z DEBUG Command::run{cmd="/usr/bin/wc" args=["-l"] line=stdin:99858-198620 stdin=1.0 MiB child_pid=48048}: rust_parallel::command: spawned child process, awaiting completion
2026-08-15T16:27:39.988268Z DEBUG Command::run{cmd="/usr/bin/wc" args=["-l"] line=stdin:198621-235976 stdin=387.4 KiB child_pid=48049}: rust_parallel::command: spawned child process, awaiting completion

Error handling

The following are considered command failures and error will be logged:

  • Spawn error
  • Timeout
  • I/O error
  • Command exits with non-0 status

By default rust-parallel runs all commands even if failures occur.

When rust-parallel terminates, if any command failed it logs failure metrics and exits with status 1.

Here we try to use cat to show non-existing files A, B, and C, so each command exits with status 1:

$ rust-parallel cat ::: A B C
cat: B: No such file or directory
2026-08-15T16:27:39.996260Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["B"],stdin=None,line=command_line_args:2 exit_status=1
cat: A: No such file or directory
2026-08-15T16:27:39.996337Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],stdin=None,line=command_line_args:1 exit_status=1
cat: C: No such file or directory
2026-08-15T16:27:39.996392Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["C"],stdin=None,line=command_line_args:3 exit_status=1
2026-08-15T16:27:39.996427Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=3 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=3

$ echo $?
1

The --exit-on-error option can be used to exit after one command fails.

rust-parallel waits for in-progress commands to finish before exiting and then exits with status 1.

$ head -100 /usr/share/dict/words | rust-parallel --exit-on-error cat
cat: A: No such file or directory
2026-08-15T16:27:40.002225Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],stdin=None,line=stdin:1 exit_status=1
cat: a: No such file or directory
2026-08-15T16:27:40.002297Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["a"],stdin=None,line=stdin:2 exit_status=1
cat: aardvark: No such file or directory
2026-08-15T16:27:40.002414Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aardvark"],stdin=None,line=stdin:8 exit_status=1
cat: aal: No such file or directory
2026-08-15T16:27:40.002484Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aal"],stdin=None,line=stdin:4 exit_status=1
cat: aam: No such file or directory
2026-08-15T16:27:40.002733Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aam"],stdin=None,line=stdin:6 exit_status=1
cat: Aani: No such file or directory
2026-08-15T16:27:40.002786Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["Aani"],stdin=None,line=stdin:7 exit_status=1
cat: aalii: No such file or directory
2026-08-15T16:27:40.002836Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aalii"],stdin=None,line=stdin:5 exit_status=1
cat: aa: No such file or directory
2026-08-15T16:27:40.003106Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aa"],stdin=None,line=stdin:3 exit_status=1
2026-08-15T16:27:40.003138Z ERROR rust_parallel: fatal error in main: command failures: commands_run=8 total_failures=8 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=8

$ echo $?
1

Timeout

The -t/--timeout-seconds option can be used to specify a command timeout in seconds. If any command times out this is considered a command failure (see error handling).

$ rust-parallel -t 0.5 sleep ::: 0 3 5
2026-08-15T16:27:40.510011Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["3"],stdin=None,line=command_line_args:2 error: timeout: deadline has elapsed
2026-08-15T16:27:40.510006Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["5"],stdin=None,line=command_line_args:3 error: timeout: deadline has elapsed
2026-08-15T16:27:40.510208Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=2 spawn_errors=0 timeouts=2 io_errors=0 exit_status_errors=0

$ echo $?
1

Path Cache

By default as commands are run the full paths are resolved using which. Resolved paths are stored in a cache to prevent duplicate resolutions. This is generally good for performance.

The path cache can be disabled using the --disable-path-cache option.

Progress bar

The -p/--progress-bar option can be used to enable a graphical progress bar.

This is best used for commands which are running for at least a few seconds, and which do not produce output to stdout or stderr. In the below commands -d all is used to discard all output from commands run.

Progress styles can be chosen with the PROGRESS_STYLE environment variable. If PROGRESS_STYLE is not set it defaults to light_bg.

The following progress styles are available:

  • PROGRESS_STYLE=light_bg good for light terminal background with colors, spinner, and steady tick enabled: light_bg

  • PROGRESS_STYLE=dark_bg good for dark terminal background with colors, spinner, and steady tick enabled: dark_bg

  • PROGRESS_STYLE=simple good for simple or non-ansi terminals/jobs with colors, spinner, and steady tick disabled: simple

Regular Expression

Regular expressions can be specified by the -r or --regex command line argument.

Named or numbered capture groups are expanded with data values from the current input before the command is executed.

Named Capture Groups

In these examples using command line arguments {url} and {filename} are named capture groups. {} is a variable meaning the entire input line.

$ rust-parallel -r '(?P<url>.*),(?P<filename>.*)' echo got url={url} filename={filename} ::: URL1,filename1 URL2,filename2
got url=URL1 filename=filename1
got url=URL2 filename=filename2

$ rust-parallel -r '(?P<url>.*) (?P<filename>.*)' echo got url={url} filename={filename} full input={} ::: URL1 URL2 ::: filename1 filename2
got url=URL2 filename=filename2 full input=URL2 filename2
got url=URL2 filename=filename1 full input=URL2 filename1
got url=URL1 filename=filename2 full input=URL1 filename2
got url=URL1 filename=filename1 full input=URL1 filename1

Numbered Capture Groups

In the next example input file arguments {1} {2} {3} are numbered capture groups, {} is a variable meaning the entire input line. The input is a csv file:

$ cat >./test <<EOL
foo,bar,baz
foo2,bar2,baz2
foo3,bar3,baz3
EOL

$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo got arg1={1} arg2={2} arg3={3} full input={}
got arg1=foo2 arg2=bar2 arg3=baz2 full input=foo2,bar2,baz2
got arg1=foo arg2=bar arg3=baz full input=foo,bar,baz
got arg1=foo3 arg2=bar3 arg3=baz3 full input=foo3,bar3,baz3

Capture Group Special Characters

All occurrences of capture groups are replaced as exact strings. Surrounding characters have no effect on this.

This means capture groups can be nested with other { or } characters such as when building json:

$ cat >./test <<EOL
1,2,3
4,5,6
7,8,9
EOL

$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo '{"one":{1},"two":{2},"nested_object":{"three":{3}}}'
{"one":4,"two":5,"nested_object":{"three":6}}
{"one":1,"two":2,"nested_object":{"three":3}}
{"one":7,"two":8,"nested_object":{"three":9}}

Shell Commands

Shell commands can be written using -s shell mode.

Multiline commands can be written using ;.

Environment variables, $ characters, nested commands and much more are possible:

$ rust-parallel -s -r '(?P<arg1>.*) (?P<arg2>.*)' 'FOO={arg1}; BAR={arg2}; echo "FOO = ${FOO}, BAR = ${BAR}, shell pid = $$, date = $(date)"' ::: A B ::: CAT DOG
FOO = A, BAR = DOG, shell pid = 48092, date = Sat Aug 15 11:27:40 CDT 2026
FOO = B, BAR = DOG, shell pid = 48094, date = Sat Aug 15 11:27:40 CDT 2026
FOO = B, BAR = CAT, shell pid = 48093, date = Sat Aug 15 11:27:40 CDT 2026
FOO = A, BAR = CAT, shell pid = 48091, date = Sat Aug 15 11:27:40 CDT 2026

Bash Function

-s shell mode can be used to invoke an arbitrary bash function.

Similar to normal commands bash functions can be called using stdin, input files, or from command line arguments.

Function Setup

Define a bash fuction logargs that logs all arguments and make visible with export -f:

$ logargs() {
  echo "logargs got $@"
}

$ export -f logargs

Demo of command line arguments:

$ rust-parallel -s logargs ::: A B C ::: D E F
logargs got A D
logargs got C E
logargs got B F
logargs got A F
logargs got B D
logargs got A E
logargs got B E
logargs got C D
logargs got C F

Demo of function and command line arguments from stdin:

$ cat >./test <<EOL
logargs hello alice
logargs hello bob
logargs hello charlie
EOL

$ cat test | rust-parallel -s
logargs got hello charlie
logargs got hello alice
logargs got hello bob

Demo of function and initial arguments on command line, additional arguments from stdin:

$ cat >./test <<EOL
alice
bob
charlie
EOL

$ cat test | rust-parallel -s logargs hello
logargs got hello charlie
logargs got hello alice
logargs got hello bob

Shell Quote

When using shell mode (-s), the --shell-quote option applies shell escaping to each argument before passing it to the shell.

This is useful when arguments contain special shell characters like $, backticks, single quotes, or double quotes that should be treated as literal strings and not interpreted by the shell.

Without --shell-quote, special characters can cause command injection or unexpected shell behavior. With --shell-quote enabled, arguments are properly escaped.

Consider a bash function that uses special characters:

$ demonstrate_shell_quote() {
  for arg in "$@"; do
    echo "arg: $arg"
  done
}

$ export -f demonstrate_shell_quote

With --shell-quote, special characters in arguments are properly escaped:

$ cat >./test <<'EOL'
hello$world
foo`cmd`
bar'baz'
EOL

$ cat test | rust-parallel -s --shell-quote demonstrate_shell_quote
arg: bar'baz'
arg: foo`cmd`
arg: hello$world

Without --shell-quote, special characters are interpreted by the shell, causing errors:

$ cat test | rust-parallel -s demonstrate_shell_quote
arg: hello
arg: foo
/opt/homebrew/bin/bash: line 1: cmd: command not found
arg: barbaz
⚠️ **GitHub.com Fallback** ⚠️