bash xargs - ghdrako/doc_snipets GitHub Wiki

• xargs man page
• xargs vs. exec {}
• find exec vs find xargs

xargs [options] [command [initial-arguments]]

Some of the commonly used options include:

• -a file: read items from a file instead of standard input.
• -I replace-str: replace occurrences of replace-str in the initial-arguments with the input item.
• -n num: use at most num arguments per command line.
• -P max-procs: run up to max-procs processes at once.

ls *.png | xargs -I {} convert {} {}.jpg

The xargs command uses the -I option to replace occurrences of {} in the convert command with the input items. The convert command then converts each file to the .jpg format.

By default xargs reads items from standard input as separated by blanks and executes a command once for each argument.

$ echo 'one two three' | xargs mkdir
$ ls
one two three

When filenames contains spaces you need to use -d option to change delimiter

$ ls 
one two three.txt' 'four.txt
$ find . -name '*.txt' | xargs -d '\n' rm

find . -name "*.log" | xargs rm

use xargs with find

The most common usage of xargs is to use it with the find command. This uses find to search for files or directories and then uses xargs to operate on the results.

find /tmp -mtime +14 | xargs rm

xargs v exec

The find command supports the -exec option that allows arbitrary commands to be performed on found files.

find ./foo -type f -name "*.txt" -exec rm {} \; 
find ./foo -type f -name "*.txt" | xargs rm

Use xargs to pass all three file names to the rm command, which deletes them all at once.

echo "file1.txt file2.txt file3.txt" | xargs rm

Clearly using xargs is far more efficient. In fact several benchmarks suggest using xargs over exec {} is six times more efficient.

print commands that are executed

The -t option prints each command that will be executed to the terminal. This can be helpful when debugging scripts.

echo 'one two three' | xargs -t rm
rm one two three

view the command and prompt for execution

The -p command will print the command to be executed and prompt the user to run it. This can be useful for destructive operations where you really want to be sure on the command to be run. l

echo 'one two three' | xargs -p touch
touch one two three ?...

run multiple commands with xargs

It is possible to run multiple commands with xargs by using the -I flag. This replaces occurrences of the argument with the argument passed to xargs. The following echos a string and creates a folder.

cat foo.txt
one
two
three

cat foo.txt | xargs -I % sh -c 'echo %; mkdir %'
one 
two
three

ls 
one two three

xargs -P allows you to run commands in parallel.

ls *.txt | xargs -P 4 -n 1 gzip

The xargs command uses the -P option to specify the maximum number of processes to run at once (in this case, 4), and the -n option to specify the number of arguments per command line

time xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF

The xargs command itself won't return until all commands have finished, but you can execute it in the background by terminating it with control operator & and then using the wait builtin to wait for the entire xargs command to finish.

{
  xargs -P 3 -I {} sh -c 'eval "$1"' - {} <<'EOF'
sleep 1; echo 1
sleep 2; echo 2
sleep 3; echo 3
echo 4
EOF
} &

# Script execution continues here while `xargs` is running 
# in the background.
echo "Waiting for commands to finish..."

# Wait for `xargs` to finish, via special variable $!, which contains
# the PID of the most recently started background process.
wait $!

Note:

• BSD/macOS xargs requires you to specify the count of commands to run in parallel explicitly, whereas GNU xargs allows you to specify -P 0 to run as many as possible in parallel.
• Output from the processes run in parallel arrives as it is being generated, so it will be unpredictably interleaved.

find . -type f -print0 | xargs -0 -I % bsdtar -cf %.tar %

• ``print0'' - find: It separates the paths to found objects by the null character. Without the option, the separator is a line break.
• -0 - xargs: By default, the utility separates the strings on the input stream by line breaks. The -0 option changes this behavior. xargs applies the null character as the separator. In this way, we have reconciled the output and input formats of the utilities

You can change the output format of the greputility in the same manner. It allows you to pass its output through the pipeline. The -Z option does that. The option separates filenames with the null character. Here is an example:

grep -RlZ "GNU" . | xargs -0 -I % bsdtar -cf %.tar %

The command searches files that contain the “GNU” pattern. Then it passes their names to the xargs utility. The utility constructs the bsdtar call for archiving the files.

Note Always use -0 option when process filenames by xargs. Pass null separated names to the utility.