fex - flexible token-field extraction


What is fex?

Fex is a powerful field extraction tool. Fex provides a very concise language for tokenizeing strings and extracting fields.

The basic usage model is that you provide a series of delimiter and field selection pairs. Tokens can be any character, while field selections have a specific syntax.

Download

Releases are available on GitHub.

Sometimes simpler than cut and awk

cut(1) from GNU coreutils (on Linux) does not support negative offsets, so you cannot ask cut to only show you the Nth field from the end. Additionally, to cut by multiple fields, you end up having to write cut ... | cut ... | cut ...

awk lets you select negative offsets using the NF variable. Get the 2nd to last field with $(NF - 1). However, cutting multiple times still requires multiple awk invocations or using awk’s split() function multiple times.

Field selection

There are a few ways to specify field selections.

Notes:

Tokenizing behavior

The default behavior is to ignore empty fields. That is, a string “foo…bar” would only have two fields when split by “.” rather than four. If you want fex to not ignore empty fields, you should prefix your field selection with “?”

Greedy (default)

% echo "foo.....bar..baz.fizz" | fex .2
bar

Nongreedy

% echo "foo.....bar..baz.fizz" | fex '.{?6}'
bar

Command line arguments

You can specify multiple, independent field selectors on the command line. Each argument is treated as a standalone field selector. Selectors are split by spaces on output (though I am open to changing this).

For example, output the IP and URL from an apache request log:

echo '208.36.144.8 - - [22/Aug/2007:23:39:05 -0400] "GET /svnweb/logwatch/tags/?pathrev=420 HTTP/1.0" 200 3595' \
| fex 1 '"2 2'

208.36.144.8 /svnweb/logwatch/tags/?pathrev=420 

Usage Examples

Simple splitting

With input text: /usr/local/bin/firefox

Command Result
fex /1 “usr”
fex /{2:3} “local/bin”
fex /{1,-1} “usr/firefox”
fex /-1 “firefox”
fex /{:} “/usr/local/bin/firefox/
fex /0 “/usr/local/bin/firefox/

Greedy vs nongreedy splitting

Input text: a:b::c:::d

Command Result
fex :{1:3} “a:b:c”
fex :{?1:3} “a:b:”
fex :{3} “c”
fex :{?3} "” (empty result)

Real world uses

Parse IP and URL from apache logs

% fex 1 '"2 2' < /b/logs/access| head
65.57.245.11 /
65.57.245.11 /icons/blank.gif
65.57.245.11 /icons/folder.gif
65.57.245.11 /favicon.ico
65.57.245.11 /semicomplete/
65.57.245.11 /style.css
65.57.245.11 /images/semicomplete.png
65.57.245.11 /
65.57.245.11 /style.css
65.57.245.11 /images/semicomplete.png

Show paths from strace

Show files found with strace:

% strace -e trace=file cat /etc/motd |& fex '"2'
/bin/cat
/etc/ld.so.nohwcap
/etc/ld.so.preload
/etc/ld.so.cache
/etc/ld.so.nohwcap
/lib/libc.so.6
/etc/motd

Show home directory root paths

Here’s a simple example, to find which root directories contain home directories:

% fex ':-2/1' < /etc/passwd | sort | uniq -c
      3 bin
      1 dev
      4 home
      2 nonexistent
      1 root
      2 usr
     14 var

The string ‘0:-2/1’ means:

The output is essentially the root directory for everyone’s home directories. Doing this in awk, cut, perl, or any other tool would be much more typing.

You can also specify multiple field extractions on a single invocation:

# Take the first and 2nd to last token split by colon
% fex :1 :-2 < /etc/passwd
root /root
daemon /usr/sbin
bin /bin

# Alternatively, {x,y,z,...} syntax selects multiple tokens
# note that the output is joined by colons.
# Again, this is a feature unavailable in xapply's subfield extraction
% fex ':{1,-2}' < /etc/passwd
root:/root
daemon:/usr/sbin
bin:/bin