, 1 min read

UNIX Process Substitution

Process substitution is available in ksh, bash, and zsh. More examples can be found here: Process Substitution.

Process substitution uses /dev/fd/<n> files to send the results of the process(es) within parentheses to another process.

Piping the output of one program to the input of another program is a very powerful way to run multiple programs at once without any auxiliary temporary files.

1. Sequential flow. Simple case:

program1 | program2 | program3 > file1
flowchart LR A(program1) B(program2) C(program3) U@{ shape: flag, label: "file1" } A --> B --> C --> U

2. Read from multiple programs. This is a quite common case.

program1 <(program2) <(program3)
flowchart TD A(program1) B(program2) C(program3) B --> A C --> A

It looks very similar to the usual command line:

program1 file1 file2

A real-word example of multiple programs started:

cat <(cut -d\| -f1 WOLA_EXP_WOLA_ADRESSE_CZSWADTB.csv \
                   WOLA_EXP_WOLA_PARTYINFOS_CZSWPTTB.csv \
                   WOLA_EXP_WOLA_PARTY_BEZ_CZSWPBTB.csv \
                   WOLA_EXP_WOLA_VERFUEGUNGEN_CZSWVFTB.csv) \
    <(cut -d\| -f7 WOLA_EXP_WOLA_GESCHAEFTS_VM_CZSWGVTB.csv | cut -b2-17) \
  | sort -u > bpkennWOLA

3. Pipe to multiple programs. One program, here program2, takes multiple files as command-line arguments, i.e., it has multiple outputs.

program1 | program2 >(program4 > file2) >(program5 > file3) \
         | program3 > file1
flowchart LR A(program1) B(program2) C(program3) D(program4) E(program5) U@{ shape: flag, label: "file1" } V@{ shape: flag, label: "file2" } W@{ shape: flag, label: "file3" } A --> B --> C --> U B --> D --> V B --> E --> W

Here is a real-world example for generating the statistics for this web-server:

time blogconcatlog 57 | pv | tee /tmp/a2 | accesslogFilter -o >(blogstatcnt > /srv/http/statcnt.html) | tee /tmp/a1 | blogurlcnt -m70 > /srv/http/urlstat2-m100.html