consume

The consume program splits input, feeds it into the given utility and consumes it if successful.

Features

• Quality
  • Compiled with security hardening flags.
  • Static analysis integrated using clang's scan-build using checkers alpha.security, alpha.core.CastSize, alpha.core.CastToStruct, alpha.core.IdenticalExpr, alpha.core.PointerArithm, alpha.core.PointerSub, alpha.core.SizeofPtr, alpha.core.TestAfterDivZero, alpha.unix.
  • Follows FreeBSD coding style.
• Portable
  • C99 compliant and may be built in an environment which provides POSIX.1-2008 system interfaces.
  • Self-contained, no external dependencies.
  • Easy to compile and uses POSIX make.

Limitations

• Input lines are limited to LINE_MAX bytes in lenght.
• Input file names are limited to BUFSIZ bytes in length.

Build dependencies

The only dependency is a toolchain supporting the following flags:

CFLAGS = -std=c99 -O2 -Wall -Wextra -Wpedantic \
        -Walloca -Wcast-qual -Wconversion -Wformat=2 -Wformat-security \
        -Wnull-dereference -Wstack-protector -Wvla -Warray-bounds \
        -Wbad-function-cast -Wconversion -Wshadow -Wstrict-overflow=4 -Wundef \
        -Wstrict-prototypes -Wswitch-default -Wfloat-equal -Wimplicit-fallthrough \
        -Wpointer-arith -Wswitch-enum \
        -D_FORTIFY_SOURCE=2 \
        -fstack-protector-strong -fPIE -fstack-clash-protection

LDFLAGS = -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack -Wl,-z,separate-code

Otherwise you can just remove the security flags and compile it with

CFLAGS = -std=c99 -O2 -Wall -Wextra -Wpedantic
LDFLAGS =

or pass your own flags to make

make CC=gcc CFLAGS=... LDFLAGS=...

Installation

Clone this repository then

$ make PREFIX=/usr install

This will install the compiled binary under PREFIX (/usr/bin) in this case, if not specified PREFIX will default to /usr/local. For staged installs, DESTDIR is also supported. As the binary does not have any dependency it does not have to be installed before use.

Usage

consume [-0ach] [-i file] [-v] [utility [argument ...]]

The consume program reads input from the file named by the file operand or from stdin if absent, and splits it when encountering the given delimiter (defaults to newline). It then invokes the utility repeatedly, feeding it the generated data as input. On success the given data is consumed (removed) from the input file, if any.

The options are as follows:

• -a input items are fed as a command-line argument to utility.
• -0 input items are teminated by a null character instead of by newline. The GNU find -print0 option produces input suitable for this mode.
• -c continue even if utility failed.
• -h print usage information and exit.
• -i file read items from file instead of standard input.
• -v print the command line on the standard error ouput before executing it.

Examples

Count the number of words in each line of 'La Divina Commedia':

$ curl -s https://dmf.unicatt.it/~della/pythoncourse18/commedia.txt | consume wc -w

Find files named core in or below the directory /tmp and delete them. Before deleting each file the full command is printed to stderr:

$ find /tmp -name core -type f -print | consume -a -v /bin/rm -f

Consume all lines delimited by 0 in biglistoflinks.txt that wget has downloaded successfully:

$ consume -0 -a -i biglistoflinks.txt wget -qc

Static analysis

Static analysis on the code base is done by using clang's static analyzer run through scan-build.sh which wraps the scan-build utility. The checkers used are part of the Experimental Checkers (aka alpha checkers):

• alpha.security
• alpha.core.CastSize
• alpha.core.CastToStruct
• alpha.core.IdenticalExpr
• alpha.core.PointerArithm
• alpha.core.PointerSub
• alpha.core.SizeofPtr
• alpha.core.TestAfterDivZero
• alpha.unix

Contributing

Send patches on the mailing list, report bugs on the issue tracker.

License

BSD 2-Clause FreeBSD License, see LICENSE.