Manual - aaronriekenberg/rust-parallel GitHub Wiki

Manual for rust-parallel 1.22.0

Command line options
Commands from arguments
1. Automatic variables
Commands from stdin
Command and initial arguments on command line
Reading multiple inputs
Pipe Mode
Parallelism
Keep Output Order
Dry run
Debug logging
Error handling
Timeout
Path cache
Progress bar
Regular Expression
Shell Commands
Bash Function

Command line options

$ rust-parallel --help
Execute commands in parallel

By Aaron Riekenberg <[email protected]>

https://github.com/aaronriekenberg/rust-parallel
https://crates.io/crates/rust-parallel

Usage: rust-parallel [OPTIONS] [COMMAND_AND_INITIAL_ARGUMENTS]...

Arguments:
  [COMMAND_AND_INITIAL_ARGUMENTS]...
          Optional command and initial arguments.
          
          If this contains 1 or more ::: delimiters the cartesian product of arguments from all groups are run.

Options:
  -d, --discard-output <DISCARD_OUTPUT>
          Discard output for commands

          Possible values:
          - stdout: Redirect stdout for commands to /dev/null
          - stderr: Redirect stderr for commands to /dev/null
          - all:    Redirect stdout and stderr for commands to /dev/null

  -i, --input-file <INPUT_FILE>
          Input file or - for stdin.  Defaults to stdin if no inputs are specified

  -j, --jobs <JOBS>
          Maximum number of commands to run in parallel, defauts to num cpus
          
          [default: 8]

  -0, --null-separator
          Use null separator for reading input files instead of newline

  -p, --progress-bar
          Display progress bar

      --progress-bar-style <PROGRESS_BAR_STYLE>
          Progress bar style

  -r, --regex <REGEX>
          Apply regex pattern to inputs

  -s, --shell
          Use shell mode for running commands.
          
          Each command line is passed to "<shell-path> <shell-argument>" as a single argument.

  -t, --timeout-seconds <TIMEOUT_SECONDS>
          Timeout seconds for running commands.  Defaults to infinite timeout if not specified

      --channel-capacity <CHANNEL_CAPACITY>
          Input and output channel capacity, defaults to num cpus * 2
          
          [default: 16]

      --disable-path-cache
          Disable command path cache

      --dry-run
          Dry run mode
          
          Do not actually run commands just log.

      --exit-on-error
          Exit on error mode
          
          Exit immediately when a command fails.

      --no-run-if-empty
          Do not run commands for empty buffered input lines

  -k, --keep-order
          Keep output in the same order as input

      --pipe
          Use pipe input mode

      --block-size <BLOCK_SIZE>
          Block size for pipe input mode, defaults to 1MiB
          
          [default: 1MiB]

      --shell-path <SHELL_PATH>
          Path to shell to use for shell mode
          
          [default: bash]

      --shell-argument <SHELL_ARGUMENT>
          Argument to shell for shell mode
          
          [default: -c]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Commands from arguments

The ::: separator can be used to run the Cartesian Product of command line arguments. This is similar to the ::: behavior in GNU Parallel.

$ rust-parallel echo ::: A B ::: C D ::: E F G
B C E
A C G
A D F
A C E
A C F
A D E
B C F
A D G
B C G
B D E
B D F
B D G

$ rust-parallel echo hello ::: larry curly moe
hello curly
hello larry
hello moe

# run gzip -k on all *.html files in current directory
$ rust-parallel gzip -k ::: *.html

Automatic Variables

When using commands from arguments, numbered variables {0}, {1}, etc are automatically available based on the number of arguments. {0} will be replaced by the entire input line, and other groups match individual argument groups. {} is the same as {0}. This is useful for building more complex command lines. For example:

$ rust-parallel echo group0={0} group1={1} group2={2} group3={3} group2again={2} ::: A B ::: C D ::: E F G
group0=A C F group1=A group2=C group3=F group2again=C
group0=A C E group1=A group2=C group3=E group2again=C
group0=B C F group1=B group2=C group3=F group2again=C
group0=A D G group1=A group2=D group3=G group2again=D
group0=A C G group1=A group2=C group3=G group2again=C
group0=B C E group1=B group2=C group3=E group2again=C
group0=A D E group1=A group2=D group3=E group2again=D
group0=A D F group1=A group2=D group3=F group2again=D
group0=B D E group1=B group2=D group3=E group2again=D
group0=B C G group1=B group2=C group3=G group2again=C
group0=B D F group1=B group2=D group3=F group2again=D
group0=B D G group1=B group2=D group3=G group2again=D

$ rust-parallel echo entireline={} group1={1} group2={2} group3={3} group2again={2} ::: A B ::: C D ::: E F G
entireline=A D E group1=A group2=D group3=E group2again=D
entireline=A C F group1=A group2=C group3=F group2again=C
entireline=A C E group1=A group2=C group3=E group2again=C
entireline=A D G group1=A group2=D group3=G group2again=D
entireline=A C G group1=A group2=C group3=G group2again=C
entireline=B C F group1=B group2=C group3=F group2again=C
entireline=B C E group1=B group2=C group3=E group2again=C
entireline=A D F group1=A group2=D group3=F group2again=D
entireline=B C G group1=B group2=C group3=G group2again=C
entireline=B D E group1=B group2=D group3=E group2again=D
entireline=B D F group1=B group2=D group3=F group2again=D
entireline=B D G group1=B group2=D group3=G group2again=D

Internally these variables are implemented using an auto-generated regular expression. If a regular expression is manually specified this will override the auto-generated one.

Commands from stdin

Run complete commands from stdin.

$ cat >./test <<EOL
echo hi
echo there
echo how
echo are
echo you
EOL

$ cat test | rust-parallel
are
how
hi
you
there

Command and initial arguments on command line

Here md5 -s will be prepended to each input line to form a command like md5 -s aal

$ head -100 /usr/share/dict/words | rust-parallel md5 -s | head -10
e9b22dd6213c3d29648e8ad7a8642f2f
0cc175b9c0f1b6a831c399e269772661
ff45e881572ca2c987460932660d320c
7fc56270e7a70fa81a5935b72eacbe29
0a1ea2a8d75d02ae052f8222e36927a5
4124bc0a9335c27f086f24ba207a4912
35c2d90f7c06b623fe763d0a4e5b7ed9
88571e5d5e13a4a60f82cea7802f6255
0390cf1718c4f2d76f770c7c35b40c50
1c0a11cc4ddc0dbd3fa4d77232a4e22e

Reading multiple inputs

By default rust-parallel reads input from stdin only. The -i option can be used 1 or more times to override this behavior. -i - means read from stdin, -i ./test means read from the file ./test:

$ cat >./test <<EOL
foo
bar
baz
EOL

$ head -5 /usr/share/dict/words | rust-parallel -i - -i ./test echo
A
aa
foo
aal
a
aalii
bar
baz

Pipe Mode

The --pipe option can be used to enable pipe mode.

In pipe mode input from stdin is split into blocks and each block is passed to a separate instance of the command via stdin. Command instances are run in parallel.

The default block size is 1 MiB, which can be changed with the --block-size option.

By default input blocks are split on new line boundaries. This can be changed to split on null boundaries with the -0/--null-separator option.

Here we use --pipe to run wc -l

$ cat /usr/share/dict/words | rust-parallel --pipe wc -l
   99857
   98763
   37356

Here we use pipe mode with with a smaller block size of 500 KiB:

$ cat /usr/share/dict/words | rust-parallel --pipe --block-size=500KiB wc -l
   49400
   48066
   48406
   48237
   41867

Parallelism

By default the number of parallel jobs to run simulatenously is the number of cpus detected at run time.

This can be override with the -j/--jobs option.

With -j5 all echo commands below run in parallel.

With -j1 all jobs run sequentially.

$ rust-parallel -j5 echo ::: hi there how are you
there
are
how
hi
you

$ rust-parallel -j1 echo ::: hi there how are you
hi
there
how
are
you

Keep Output Order

By default, command outputs are displayed as soon as each command completes, which may not be in the same order as the input.

Use option -k/--keep-order to ensure outputs are displayed in the same order as the input.

With -k all outputs will be displayed in the same order as the input, regardless of when commands complete.

$ rust-parallel -k echo ::: hi there how are you
hi
there
how
are
you

$ rust-parallel -j1 -k echo ::: hi there how are you
hi
there
how
are
you

Dry run

Use option --dry-run for dry run mode.

In this mode the commands that would be run are ouput as info level logs.

No commands are actually run - this is useful for testing before running a job.

$ rust-parallel --dry-run echo ::: hi there how are you
2026-02-28T17:48:57.287167Z  INFO rust_parallel::command: cmd="/bin/echo",args=["hi"],stdin=None,line=command_line_args:1
2026-02-28T17:48:57.287194Z  INFO rust_parallel::command: cmd="/bin/echo",args=["there"],stdin=None,line=command_line_args:2
2026-02-28T17:48:57.287203Z  INFO rust_parallel::command: cmd="/bin/echo",args=["how"],stdin=None,line=command_line_args:3
2026-02-28T17:48:57.287211Z  INFO rust_parallel::command: cmd="/bin/echo",args=["are"],stdin=None,line=command_line_args:4
2026-02-28T17:48:57.287219Z  INFO rust_parallel::command: cmd="/bin/echo",args=["you"],stdin=None,line=command_line_args:5

Debug logging

Set environment variable RUST_LOG=debug to see debug output.

This logs structured information about command line arguments and commands being run.

Recommend enabling debug logging for all examples to understand what is happening in more detail.

$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args | head -1
2026-02-28T17:48:57.290792Z DEBUG try_main: rust_parallel::command_line_args: command_line_args = CommandLineArgs { discard_output: None, input_file: [], jobs: 8, null_separator: false, progress_bar: false, progress_bar_style: None, regex: None, shell: false, timeout_seconds: None, channel_capacity: 16, disable_path_cache: false, dry_run: false, exit_on_error: false, no_run_if_empty: false, keep_order: false, pipe: false, block_size: 1048576, shell_path: "bash", shell_argument: "-c", command_and_initial_arguments: ["echo", ":::", "hi", "there", "how", "are", "you"] }

$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args:1
2026-02-28T17:48:57.297956Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None}: rust_parallel::command: begin run
2026-02-28T17:48:57.298127Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None child_pid=43659}: rust_parallel::command: spawned child process, awaiting completion
2026-02-28T17:48:57.299258Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None child_pid=43659}: rust_parallel::command: command exit status = exit status: 0
2026-02-28T17:48:57.299272Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 stdin=None child_pid=43659}: rust_parallel::command: end run

If pipe mode is being used debug logs will show stdin line ranges and block sizes for each command:

$ cat /usr/share/dict/words| RUST_LOG=debug rust-parallel --pipe wc -l | grep spawned
2026-02-28T17:48:57.335539Z DEBUG Command::run{cmd="/usr/bin/wc" args=["-l"] line=stdin:1-99857 stdin=1.0 MiB child_pid=43667}: rust_parallel::command: spawned child process, awaiting completion
2026-02-28T17:48:57.366062Z DEBUG Command::run{cmd="/usr/bin/wc" args=["-l"] line=stdin:99858-198620 stdin=1.0 MiB child_pid=43668}: rust_parallel::command: spawned child process, awaiting completion
2026-02-28T17:48:57.378034Z DEBUG Command::run{cmd="/usr/bin/wc" args=["-l"] line=stdin:198621-235976 stdin=387.4 KiB child_pid=43669}: rust_parallel::command: spawned child process, awaiting completion

Error handling

The following are considered command failures and error will be logged:

Spawn error
Timeout
I/O error
Command exits with non-0 status

By default rust-parallel runs all commands even if failures occur.

When rust-parallel terminates, if any command failed it logs failure metrics and exits with status 1.

Here we try to use cat to show non-existing files A, B, and C, so each command exits with status 1:

$ rust-parallel cat ::: A B C
cat: C: No such file or directory
2026-02-28T17:48:57.385729Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["C"],stdin=None,line=command_line_args:3 exit_status=1
cat: B: No such file or directory
2026-02-28T17:48:57.385817Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["B"],stdin=None,line=command_line_args:2 exit_status=1
cat: A: No such file or directory
2026-02-28T17:48:57.385886Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],stdin=None,line=command_line_args:1 exit_status=1
2026-02-28T17:48:57.385917Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=3 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=3

$ echo $?
1

The --exit-on-error option can be used to exit after one command fails.

rust-parallel waits for in-progress commands to finish before exiting and then exits with status 1.

$ head -100 /usr/share/dict/words | rust-parallel --exit-on-error cat
cat: A: No such file or directory
2026-02-28T17:48:57.391611Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],stdin=None,line=stdin:1 exit_status=1
cat: aalii: No such file or directory
2026-02-28T17:48:57.391749Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aalii"],stdin=None,line=stdin:5 exit_status=1
cat: Aani: No such file or directory
2026-02-28T17:48:57.391894Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["Aani"],stdin=None,line=stdin:7 exit_status=1
cat: aardvark: No such file or directory
2026-02-28T17:48:57.392054Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aardvark"],stdin=None,line=stdin:8 exit_status=1
cat: aam: No such file or directory
2026-02-28T17:48:57.392127Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aam"],stdin=None,line=stdin:6 exit_status=1
cat: a: No such file or directory
2026-02-28T17:48:57.392213Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["a"],stdin=None,line=stdin:2 exit_status=1
cat: aa: No such file or directory
2026-02-28T17:48:57.392281Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aa"],stdin=None,line=stdin:3 exit_status=1
cat: aal: No such file or directory
2026-02-28T17:48:57.392333Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aal"],stdin=None,line=stdin:4 exit_status=1
cat: aardwolf: No such file or directory
2026-02-28T17:48:57.392892Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aardwolf"],stdin=None,line=stdin:9 exit_status=1
2026-02-28T17:48:57.392924Z ERROR rust_parallel: fatal error in main: command failures: commands_run=9 total_failures=9 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=9

$ echo $?
1

Timeout

The -t/--timeout-seconds option can be used to specify a command timeout in seconds. If any command times out this is considered a command failure (see error handling).

$ rust-parallel -t 0.5 sleep ::: 0 3 5
2026-02-28T17:48:57.899354Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["3"],stdin=None,line=command_line_args:2 error: timeout: deadline has elapsed
2026-02-28T17:48:57.899549Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["5"],stdin=None,line=command_line_args:3 error: timeout: deadline has elapsed
2026-02-28T17:48:57.899724Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=2 spawn_errors=0 timeouts=2 io_errors=0 exit_status_errors=0

$ echo $?
1

Path Cache

By default as commands are run the full paths are resolved using which. Resolved paths are stored in a cache to prevent duplicate resolutions. This is generally good for performance.

The path cache can be disabled using the --disable-path-cache option.

Progress bar

The -p/--progress-bar option can be used to enable a graphical progress bar.

This is best used for commands which are running for at least a few seconds, and which do not produce output to stdout or stderr. In the below commands -d all is used to discard all output from commands run.

Progress styles can be chosen with the PROGRESS_STYLE environment variable. If PROGRESS_STYLE is not set it defaults to light_bg.

The following progress styles are available:

PROGRESS_STYLE=light_bg good for light terminal background with colors, spinner, and steady tick enabled:
PROGRESS_STYLE=dark_bg good for dark terminal background with colors, spinner, and steady tick enabled:
PROGRESS_STYLE=simple good for simple or non-ansi terminals/jobs with colors, spinner, and steady tick disabled:

Regular Expression

Regular expressions can be specified by the -r or --regex command line argument.

Named or numbered capture groups are expanded with data values from the current input before the command is executed.

Named Capture Groups

In these examples using command line arguments {url} and {filename} are named capture groups. {} is a variable meaning the entire input line.

$ rust-parallel -r '(?P<url>.*),(?P<filename>.*)' echo got url={url} filename={filename} ::: URL1,filename1 URL2,filename2
got url=URL1 filename=filename1
got url=URL2 filename=filename2

$ rust-parallel -r '(?P<url>.*) (?P<filename>.*)' echo got url={url} filename={filename} full input={} ::: URL1 URL2 ::: filename1 filename2
got url=URL1 filename=filename1 full input=URL1 filename1
got url=URL2 filename=filename1 full input=URL2 filename1
got url=URL2 filename=filename2 full input=URL2 filename2
got url=URL1 filename=filename2 full input=URL1 filename2

Numbered Capture Groups

In the next example input file arguments {1} {2} {3} are numbered capture groups, {} is a variable meaning the entire input line. The input is a csv file:

$ cat >./test <<EOL
foo,bar,baz
foo2,bar2,baz2
foo3,bar3,baz3
EOL

$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo got arg1={1} arg2={2} arg3={3} full input={}
got arg1=foo arg2=bar arg3=baz full input=foo,bar,baz
got arg1=foo2 arg2=bar2 arg3=baz2 full input=foo2,bar2,baz2
got arg1=foo3 arg2=bar3 arg3=baz3 full input=foo3,bar3,baz3

Capture Group Special Characters

All occurrences of capture groups are replaced as exact strings. Surrounding characters have no effect on this.

This means capture groups can be nested with other { or } characters such as when building json:

$ cat >./test <<EOL
1,2,3
4,5,6
7,8,9
EOL

$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo '{"one":{1},"two":{2},"nested_object":{"three":{3}}}'
{"one":1,"two":2,"nested_object":{"three":3}}
{"one":7,"two":8,"nested_object":{"three":9}}
{"one":4,"two":5,"nested_object":{"three":6}}

Shell Commands

Shell commands can be written using -s shell mode.

Multiline commands can be written using ;.

Environment variables, $ characters, nested commands and much more are possible:

$ rust-parallel -s -r '(?P<arg1>.*) (?P<arg2>.*)' 'FOO={arg1}; BAR={arg2}; echo "FOO = ${FOO}, BAR = ${BAR}, shell pid = $$, date = $(date)"' ::: A B ::: CAT DOG
FOO = B, BAR = DOG, shell pid = 43715, date = Sat Feb 28 11:48:57 CST 2026
FOO = A, BAR = CAT, shell pid = 43712, date = Sat Feb 28 11:48:57 CST 2026
FOO = B, BAR = CAT, shell pid = 43714, date = Sat Feb 28 11:48:57 CST 2026
FOO = A, BAR = DOG, shell pid = 43713, date = Sat Feb 28 11:48:57 CST 2026

Bash Function

-s shell mode can be used to invoke an arbitrary bash function.

Similar to normal commands bash functions can be called using stdin, input files, or from command line arguments.

Function Setup

Define a bash fuction logargs that logs all arguments and make visible with export -f:

$ logargs() {
  echo "logargs got $@"
}

$ export -f logargs

Demo of command line arguments:

$ rust-parallel -s logargs ::: A B C ::: D E F
logargs got A E
logargs got C D
logargs got B E
logargs got A F
logargs got A D
logargs got B F
logargs got B D
logargs got C E
logargs got C F

Demo of function and command line arguments from stdin:

$ cat >./test <<EOL
logargs hello alice
logargs hello bob
logargs hello charlie
EOL

$ cat test | rust-parallel -s
logargs got hello bob
logargs got hello charlie
logargs got hello alice

Demo of function and initial arguments on command line, additional arguments from stdin:

$ cat >./test <<EOL
alice
bob
charlie
EOL

$ cat test | rust-parallel -s logargs hello
logargs got hello alice
logargs got hello bob
logargs got hello charlie