Search this site


Metadata

Articles

Projects

Presentations

xargs - Week of Unix Tools; Day 5

Intro

This week-of-unix-tools is intended to be a high concentration of information with little fluff. I'll be covering only GNU versions of the tools, for the sake of choosing only one version for sanity sake.

xargs?

xargs is used to take input to generate a list of arguments and run a program with those arguments. GNU xargs is extremely useful. You can:
  • Use xargs as a 'threadpool' for parallel processing of input.
  • Take a list of things on input, and run a command with one thing as the argument, for all things input.
  • Reformat word lists.

And then... ?

No "and then".

Usage

xargs [flags] [command [args]]
'command' is optional. If unspecified, it defaults to '/bin/echo' or something very similar.

Flags that matter

-t
trace. Show the commands being run before they are executed
-P{N}
Specify number (where {N} is) of forked processes for parallelism; ie, -P5 for 5 processes.
-n{N}
Call the command with {N} arguments
-I{REPLSTR}
Specify that '{REPLSTR}' will be replaced with the argument

First example

Find some files and delete them.
% mkdir i; touch i/a.png i/b.png i/c.png i/otherfile
% find ./i/ -name '*.png' | xargs
./i/a.png ./i/b.png ./i/c.png

# Default action above was to echo, so it output as if you had typed:
# echo ./i/a.png ./i/b.png ./i/c.png

# Actually delete the png files:
% find ./i/ -name '*.png' | xargs rm
% ls ./i
otherfile

Advanced xargs

Sometimes the "thing" you want to do with this argument list is not just one command. You know how express what you need in shell, but you don't know how to get xargs to play nice? Not to fear. Simply use "sh" as your command, like this example:

Rename *.foo -> *.bar
% touch a.foo b.foo c.foo
% ls *.foo | xargs -n1 -I@ sh -c 'x="@"; mv $x ${x%.foo}.bar'
# ${x%.foo} means remove '.foo' from the end of $x. 

% ls *.bar
a.bar   b.bar   c.bar
This example shows using replacement with the character '@'. This means any instance of '@' in the command string will be replaced with the current value (a.foo, for example).

A better example is this, using sh's natural argument handling:
# A quick refresher for you:
% sh -c 'echo $0,$1,$2,$3' hello one two three
hello,one,two,three

# Now let's use it:
% ls *.foo | xargs -n1 sh -c 'mv $1 ${1%.foo}.bar' -
# the '-' is to pass '-' as $0 to sh, so we can use $* and $@ like normal if we wanted.
% ls
a.bar   b.bar   c.bar
This lets you use $@, $*, and $1, etc, and is more natural to shell scripting.

Parallel ssh'ing

This example expects 'somehosts' to a line-delimited list of hosts to connect to. Please note that I am using ssh keys (with ssh-agent) so ssh will not prompt for a password when logging in.
% echo kenya scorn | tr ' ' '\n' > /tmp/somehosts
% cat /tmp/somehosts \
  | xargs -P10 -I"HOST" -n1 ssh HOST uptime
 1:28AM  up 11 days,  3:44, 4 users, load averages: 0.13, 0.08, 0.02
 4:28AM  up 2 days, 18:54, 10 users, load averages: 0.10, 0.05, 0.05
There's a slight problem with the above invocation. You don't know what host is outputting what data! This becomes much more clear of a problem when you aren't executing 'uptime' but instead are doing something that outputs many lines.

An easy solution is to use sed (or awk, et al) to prefix everything with the hostname.
% cat /tmp/somehosts \
  | xargs -P10 -I"HOST" -n1 sh -c 'ssh HOST uptime | sed -e "s/^/HOST: /"'
kenya:  1:36AM  up 11 days,  3:52, 4 users, load averages: 0.07, 0.07, 0.03
scorn:  4:36AM  up 2 days, 19:03, 10 users, load averages: 0.01, 0.02, 0.02