View source for Software:Xargs

{{short description|Standard UNIX utility}}
{{lowercase title|title=xargs}}
{{Infobox software
| name                   = xargs
| logo                   = 
| screenshot             = 
| screenshot size        = 
| caption                = 
| author                 = 
| developer              = Various [[Open-source software|open-source]] and [[Commercial software|commercial]] developers
| released               = 
| latest release version = 
| latest release date    = 
| operating system       = [[Software:Unix|Unix]], [[Unix-like]], [[Software:Plan 9 from Bell Labs|Plan 9]], [[Software:IBM i|IBM i]]
| platform               = Cross-platform
| genre                  = [[Command (computing)|Command]]
| license                = 
| website                = 
}}
'''xargs''' (short for "e'''x'''tended '''arg'''ument'''s'''" <ref>{{Cite web|url=http://www.roesler-ac.de/wolfram/acro/all.htm|title=The Unix Acronym List: The Complete List|website=www.roesler-ac.de|access-date=2020-04-12}}</ref>) is a [[Command (computing)|command]] on [[Software:Unix|Unix]] and most [[Unix-like]] [[Operating system|operating system]]s used to build and execute commands from [[Standard streams|standard input]]. It converts input from standard input into arguments to a command.

Some commands such as <code>[[Software:Grep|grep]]</code> and <code>awk</code> can take input either as command-line arguments or from the standard input. However, others such as <code>[[Software:Cp (Unix)|cp]]</code> and <code>[[Software:Echo (command)|echo]]</code> can only take input as arguments, which is why '''xargs''' is necessary.

A port of an older version of GNU {{Mono|xargs}} is available for [[Software:Microsoft Windows|Microsoft Windows]] as part of the UnxUtils collection of [[Native (computing)|native]] [[Software:Windows API|Win32]] [[Porting|ports]] of common GNU Unix-like utilities.<ref>{{Cite web|url=http://unxutils.sourceforge.net/|title=Native Win32 ports of some GNU utilities|website=unxutils.sourceforge.net}}</ref> A ground-up rewrite named {{Mono|wargs}} is part of the open-source TextTools<ref>{{Cite web|url=https://github.com/idigdoug/TextTools|title=Text processing tools for Windows}}</ref> project. The {{Mono|xargs}} command has also been ported to the [[Software:IBM i|IBM i]] operating system.<ref>{{cite web |title=IBM System i Version 7.2 Programming Qshell |language=en |author=IBM |url=https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/rzahz/rzahzpdf.pdf?view=kc |access-date=2020-09-05 }}</ref>

==Examples==
One use case of the '''xargs''' command is to remove a list of files using the [[Software:Rm (Unix)|rm]] command. [[POSIX]] systems have an {{tt|ARG_MAX}} for the maximum total length of the command line,<ref>{{cite web|url=https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Argument-list-too-long|title=GNU Core Utilities Frequently Asked Questions|access-date=December 7, 2015}}</ref><ref>{{cite web |title=The maximum length of arguments for a new process |url=https://www.in-ulm.de/~mascheck/various/argmax/ |website=www.in-ulm.de}}</ref> so the command may fail with an error message of "Argument list too long" (meaning that the exec system call's limit on the length of a command line was exceeded): <source lang="bash" inline>rm /path/*</source> or <source lang="bash" inline>rm $(find /path -type f)</source>. (The latter invocation is incorrect, as it may expand globs in the output.)

This can be rewritten using the <code>xargs</code> command to break the list of arguments into sublists small enough to be acceptable:

<source lang="console">
$ find /path -type f -print | xargs rm
</source>
In the above example, the [[Software:Find (Unix)|<code>find</code> utility]] feeds the input of <code>xargs</code> with a long list of file names. <code>xargs</code> then splits this list into sublists and calls <code>rm</code> once for every sublist.

Some implementations of '''xargs''' can also be used to parallelize operations with the <code>-P maxprocs</code> argument to specify how many parallel processes should be used to execute the commands over the input argument lists. However, the output streams may not be synchronized. This can be overcome by using an <code>--output file</code> argument where possible, and then combining the results after processing. The following example queues 24 processes and waits on each to finish before launching another.
<source lang="console">
$ find /path -name '*.foo' | xargs -P 24 -I '{}' /cpu/bound/process '{}' -o '{}'.out
</source>

'''xargs''' often covers the same functionality as the ''command substitution'' feature of many [[Unix shell|shells]], denoted by the backquote notation (<code>`...`</code> or <code>$(...)</code>). '''xargs''' is also a good companion for commands that output long lists of files such as <code>[[Software:Find (Unix)|find]]</code>, <code>locate</code> and <code>[[Software:Grep|grep]]</code>, but only if one uses <code>-0</code> (or equivalently <code>--null</code>), since <code>xargs</code> without <code>-0</code> deals badly with file names containing <code>'</code>, <code>"</code> and space. GNU Parallel is a similar tool that offers better compatibility with [[Software:Find (Unix)|find]], locate and [[Software:Grep|grep]] when file names may contain <code>'</code>, <code>"</code>, and space (newline still requires <code>-0</code>).

==Placement of arguments==
==={{code|-I}} option: single argument===
The '''xargs''' command offers options to insert the listed arguments at some position other than the end of the command line. The <code>-I</code> option to '''xargs''' takes a string that will be replaced with the supplied input before the command is executed. A common choice is <code>%</code>.
<source lang="shell-session">
$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 -I % cp -a % ~/backups
</source>

The string to replace may appear multiple times in the command part. Using {{code|-I}} at all limits the number of lines used each time to one.

===Shell trick: any number===
Another way to achieve a similar effect is to use a shell as the launched command, and deal with the complexity in that shell, for example:
<source lang="shell-session">
$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 sh -c 'for filename; do cp -a "$filename" ~/backups; done' sh
</source>

The word {{code|sh}} at the end of the line is for the POSIX shell {{code|sh -c}} to fill in for {{code|$0}}, the "executable name" part of the positional parameters (argv). If it weren't present, the name of the first matched file would be instead assigned to <code>$0</code> and the file wouldn't be copied to <code>~/backups</code>. One can also use any other word to fill in that blank, {{code|my-xargs-script}} for example.

Since {{code|cp}} accepts multiple files at once, one can also simply do the following:
<source lang="shell-session">
$ find /path -type f -name '*~' -print0 | xargs -0 sh -c 'if [ $# -gt 0 ]; then cp -a "$@" ~/backup; fi' sh
</source>
This script runs {{code|cp}} with all the files given to it when there are any arguments passed. Doing so is more efficient since only one invocation of {{code|cp}} is done for each invocation of {{code|sh}}.

==Separator problem==
Many Unix utilities are line-oriented. These may work with <code>xargs</code> as long as the lines do not contain <code>'</code>, <code>"</code>, or a space. Some of the Unix utilities can use [[Null character|NUL]] as record separator (e.g. [[Perl]] (requires <code>-0</code> and <code>\0</code> instead of <code>\n</code>), <code>[[Software:Locate (Unix)|locate]]</code> (requires using <code>-0</code>), <code>[[Software:Find (Unix)|find]]</code> (requires using <code>-print0</code>), <code>[[Software:Grep|grep]]</code> (requires <code>-z</code> or <code>-Z</code>), <code>[[Software:Sort (Unix)|sort]]</code> (requires using <code>-z</code>)). Using <code>-0</code> for <code>xargs</code> deals with the problem, but many Unix utilities cannot use NUL as separator (e.g. <code>[[Software:Head (Unix)|head]]</code>, <code>[[Software:Tail (Unix)|tail]]</code>, <code>[[Software:Ls|ls]]</code>, <code>[[Software:Echo (command)|echo]]</code>, <code>[[Sed|sed]]</code>, <code>tar -v</code>, <code>[[Software:Wc (Unix)|wc]]</code>, <code>which</code>).

But often people forget this and assume <code>xargs</code> is also line-oriented, which is '''not''' the case (per default <code>xargs</code> separates on newlines '''and''' blanks within lines, substrings with blanks must be single- or double-quoted).

The separator problem is illustrated here:
<source lang="bash">
# Make some targets to practice on
touch important_file
touch 'not important_file'
mkdir -p '12" records'

find . -name not\* | tail -1 | xargs rm
find \! -name . -type d | tail -1 | xargs rmdir
</source>
Running the above will cause <code>important_file</code> to be removed but will remove neither the directory called <code>12" records</code>, nor the file called <code>not important_file</code>.

The proper fix is to use the GNU-specific <code>-print0</code> option, but <code>tail</code> (and other tools) do not support NUL-terminated strings:
<source lang="bash">
# use the same preparation commands as above
find . -name not\* -print0 | xargs -0 rm
find \! -name . -type d -print0 | xargs -0 rmdir
</source>

When using the <code>-print0</code> option, entries are separated by a null character instead of an end-of-line. This is equivalent to the more verbose command:<source lang="bash" inline>find . -name not\* | tr \\n \\0 | xargs -0 rm</source> or shorter, by switching <code>xargs</code> to (non-POSIX) '''line-oriented mode''' with the <code>-d</code> (delimiter) option: <source lang="bash" inline>find . -name not\* | xargs -d '\n' rm</source>

but in general using <code>-0</code> with <code>-print0</code> should be preferred, since newlines in filenames are still a problem.

GNU <code>[[Software:GNU parallel|parallel]]</code> is an alternative to <code>xargs</code> that is designed to have the same options, but is line-oriented. Thus, using GNU Parallel instead, the above would work as expected.<ref>[https://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel Differences Between xargs and GNU Parallel]. [http://www.gnu.org GNU.org]. Accessed February 2012.</ref>

For Unix environments where <code>xargs</code> does not support the <code>-0</code> nor the {{code|-d}} option (e.g. Solaris, AIX), the POSIX standard states that one can simply backslash-escape every character:<source lang="bash" inline>find . -name not\* | sed 's/\(.\)/\\\1/g' | xargs rm</source>.<ref>{{man|1|xargs|SUS}}</ref> Alternatively, one can avoid using xargs at all, either by using GNU parallel or using the {{code|-exec ... +}} functionality of {{code|find}}.

==Operating on a subset of arguments at a time==
One might be dealing with commands that can only accept one or maybe two arguments at a time. For example, the <code>diff</code> command operates on two files at a time. The <code>-n</code> option to <code>xargs</code> specifies how many arguments at a time to supply to the given command. The command will be invoked repeatedly until all input is exhausted. Note that on the last invocation one might get fewer than the desired number of arguments if there is insufficient input. Use <code>xargs</code> to break up the input into two arguments per line:
<source lang="shell-session">
$ echo {0..9} | xargs -n 2
0 1
2 3
4 5
6 7
8 9
</source>

In addition to running based on a specified number of arguments at a time, one can also invoke a command for each line of input with the <code>-L 1</code> option. One can use an arbitrary number of lines at a time, but one is most common. Here is how one might <code>diff</code> every git commit against its parent.<ref>{{cite web|url=http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html|title=Things you (probably) didn't know about xargs|author=Cosmin Stejerean|access-date=December 7, 2015}}</ref>
<source lang="shell-session">
$ git log --format="%H %P" | xargs -L 1 git diff
</source>

==Encoding problem==
The argument separator processing of <code>xargs</code> is not the only problem with using the <code>xargs</code> program in its default mode. Most Unix tools which are often used to manipulate filenames (for example <code>sed</code>, <code>basename</code>, <code>sort</code>, etc.) are text processing tools. However, Unix path names are not really text. Consider a path name /aaa/bbb/ccc. The /aaa directory and its bbb subdirectory can in general be created by different users with different environments. That means these users could have a different locale setup, and that means that aaa and bbb do not even necessarily have to have the same character encoding. For example, aaa could be in UTF-8 and bbb in Shift JIS. As a result, an absolute path name in a Unix system may not be correctly processable as text under a single character encoding. Tools which rely on their input being text may fail on such strings.

One workaround for this problem is to run such tools in the C locale, which essentially processes the bytes of the input as-is. However, this will change the behavior of the tools in ways the user may not expect (for example, some of the user's expectations about case-folding behavior may not be met).

==References==
{{Reflist}}

==External links==
* {{man|cu|xargs|SUS|construct argument lists and invoke utility}}
* [https://www.poftut.com/linux-xargs-command-tutorial-with-examples/ Linux Xargs Command Tutorial With Examples]

===Manual pages===
* {{man/format|1|xargs|https://www.gnu.org/software/findutils/manual/html_node/find_html/Invoking-xargs.html||[[Software:GNU|GNU]] Findutils reference}}
* {{man|1|xargs|FreeBSD|construct argument list(s) and execute utility}}
* {{man|1|xargs|NetBSD|construct argument list(s) and execute utility}}
* {{man|1|xargs|OpenBSD|construct argument list(s) and execute utility}}
* {{man|1|xargs|Solaris|construct argument lists and invoke utility}}

{{Unix commands}}

[[Category:Unix text processing utilities]]

{{Sourceattribution|Xargs}}