iterpipes API

A library for running shell pipelines using shell-like syntax.

iterpipes is a thin wrapper around the standard subprocess module. It represents a shell command pipeline as a function over iterables that maps its stdin to stdout. As iterpipes deals with iterables, it plays nicely with itertools from the standard library, as well as with list comprehensions and generator expressions.

To invoke a shell command pipeline, you should:

  1. Create it using a command constructor
  2. Execute it directly or via a helper function

Overview

Command Constructors

Commands are created using command constructors. Command constructors take a string command to run as their first argument. Special values '{}' in the command are replaced with constructors’ positional arguments using safe shell escaping.

Keyword arguments of constructors are almost identical to the keyword arguments of subprocess.Popen from subprocess. This allows redirecting stdout to a file, merging stdout and stderr, etc.

A few examples of using command constructors:

cmd('rm -fr {}', dirname)

linecmd('find {} -name {} -print0 | xargs -0 wc -l',
        dirname,
        '\*.py')

cmd(r'ls -d .* | tr \n \0 | xargs -0 rm -f')

Command constructors summary:

bincmd():
a binary command, that works with str iterables
cmd():
a string command, that works with unicode iterables and performs necessary encoding conversions
linecmd():
a line-oriented command, that returns buffered unicode lines delimited by the newline character '\n'

Execution Helpers

As a command is an ordinary function over iterables, you can run it by passing an stdin iterable as its input and iterating over its stdout result:

for line in linecmd('gunzip | head')(zipped_data):
    print line.rstrip('\\n')

If a command returns a non-zero code, then the CalledProcessError exception is raised.

It is often the case that a command doesn’t require any stdin data or doesn’t write anything useful to stdout. There are several helper functions for such cases.

If a command doesn’t need any stdin data, you may run it using None or [] as its input or use the run() helper function to get a little bit more readable syntax:

for line in run(linecmd('ls -a')):
    print line.rstrip('\\n')

If a command delivers no useful data to stdout, then you may use call() or check_call() helpers. If you need a return code of the command, use the call(): helper:

retcode = call(cmd('rm -f {}', filename))

otherwise use the check_call() helper that raises CalledProcessError on errors:

check_call(cmd('rm -f {}', filename))

Execution helpers summary:

run():
run a command with None as the default input value
call():
run a command and return its return code
check_call():
run a command and raise an exception if it returned a non-zero code

All the execution helpers accept input as their second argument. The default value for input is None.

Other Functions

format():
format a shell command using safe escapes and argument substitutions
compose():
function composition from the functional programming world

Function Definitions

iterpipes.bincmd(command, *args, **kwargs)

Create a binary command.

command argument is a shell pipeline string. Special '{}' values in command are replaced with positional arguments args using safe shell escaping. The keyword arguments are identical to the keyword arguments of subprocess.Popen.

Returns a binary command that works with str iterables. It is a function from stdin iterable to stdout iterable. It also may accept a single str value or None.

iterpipes.cmd(command, *args, **kwargs)

Create a string command.

It is an extension of bincmd() that performs necessary encoding conversions for unicode values.

command argument is a shell pipeline string. Special '{}' values in command are replaced with positional arguments args using safe shell escaping. The encoding keyword argument is a string encoding for unicode values. If not specified, the locale-specific encoding will be used. The other keyword arguments are identical to the keyword arguments of subprocess.Popen.

Returns a string command that works with unicode iterables and performs necessary encoding conversions. It is a function from stdin iterable to stdout iterable. It also may accept a single unicode value or None.

iterpipes.linecmd(command, *args, **kwargs)

Create a line-oriented command.

It is an extension of cmd() that returns buffered unicode lines.

command argument is a shell pipeline string. Special '{}' values in command are replaced with positional arguments args using safe shell escaping. The encoding keyword argument is a string encoding for unicode values. If not specified, the locale-specific encoding will be used. The other keyword arguments are identical to the keyword arguments of subprocess.Popen.

Returns a line-oriented command that returns buffered unicode lines delimited by the newline character '\n'. It works with unicode iterables and performs necessary encoding conversions. It is a function from stdin iterable to stdout iterable. It also may accept a single class:unicode value or None.

iterpipes.run(cmd, input=None)

Run a command with None as the default input value.

cmd is a command to run. It is a function over iterables. input is the stdin data. It may be an iterable, a single value or None.

The return value is the cmd‘s resulting stdout iterable. If the process running cmd returns a non-zero code, then a CalledProcessError is raised.

iterpipes.call(cmd, input=None)

Run a command and return its return code.

cmd is a command to run. It is a function over iterables. input is the stdin data. It may be an iterable, a single value or None.

The return value is the return code of the process running cmd.

iterpipes.check_call(cmd, input=None)

Run a command and raise an exception if it returned a non-zero code.

cmd is a command to run. It is a function over iterables. input is the stdin data. It may be an iterable, a single value or None.

There is no return value. If the process running cmd returns a non-zero code, then a CalledProcessError is raised.

iterpipes.format(command, args)

Format a shell command using safe escapes and argument substitutions.

An example:

>>> format('ls -l {} | grep {} | wc', ['foo 1', 'bar$baz'])
'ls -l foo\\ 1 | grep bar\\$baz | wc'
iterpipes.compose(*fs)

Function composition from the functional programming world.

Table Of Contents

Previous topic

Introduction

Next topic

Changelog

This Page