New xdotool for your enjoyment.

Download: http://semicomplete.googlecode.com/files/xdotool-2.20100524.2888.tar.gz

Major changes:

• xdotool getmouselocation now supports --shell for outputting something you can simply eval into your shell
• Added 'sync' support to several commands. The --sync flag, if supported, will wait until the requested operation has completed before exiting.
• Added --polar (polar coordinate) support to mousemove
• Fixed some keymap bugs

As usual, if you find problems or have feature requests, please file bugs or send an email to the list.

Changelist since previous announcement:

2.20100524.*:
  - Add --sync flag to many commands. This flag will make any supporting command
    block until the operation completes. For example, a 'windowmove --sync' will
    only exit once the window has moved. Commands supported:
    * windowactivate
    * windowfocus
    * windowmap
    * windowmove
    * windowunmap
  - Fix http://code.google.com/p/semicomplete/issues/detail?id=21
    Multiple keyboard layouts are now supported better. Added unit tests
    to ensure consistency in typing across releases.
  - Added polar coordinate support to mouse movement. 'mousemove' and
    'mousemove_relative' support this. Flag is --polar. 'x' becomes angle (degrees)
    and 'y' becomes distance. The default origin is screen center. If you specify
    a window (to mousemove) then the origin is the center of that window.
    Requested by Paul S on the xdotool-users mailing list.
  - internal: Reworked testing to be less flakey
    http://code.google.com/p/semicomplete/issues/detail?id=30
  - internal: split commands into one per C file
  - API changes that required a version bump.
    * xdo_keysequence and friends now take a delay.
  - New libxdo functions:
    * xdo_window_wait_for_map_state
    * xdo_window_wait_for_active
    * xdo_get_window_location
    * xdo_window_wait_for_focus


1.20100415.*:
  - Add support for searching classnames (not just class)
  - Support switched xkbmaps. We do this by checking if the keycode found for a keysym
    has that keysym as the first or second (normal or with shift-key) mapping, if not
    we pretend we didn't find it and request it be bound as if the keysym wasn't mapped
    to any key.

    This is a hack to avoid knowing what other modifiers are required to type
    keysyms mapped at indexes 2 or beyond (0 is normal, 1 is shift, 2 and beyond
    are unknown).
    Verified fixes this: http://code.google.com/p/semicomplete/issues/detail?id=13
    Should fix some of this: http://code.google.com/p/semicomplete/issues/detail?id=21
  - Add --shell flag to getmouselocation. This will output data you can safely
    eval in your shell.
  - Add platform detection for Darwin when building.
    Should fix: http://code.google.com/p/semicomplete/issues/detail?id=28

1.20100325.*:
  - Fix bug in search where --name wasn't working. Test written to cover this.
    http://code.google.com/p/semicomplete/issues/detail?id=25

1.20100318.*:
  - Bug fixes only this release
  - Rewrote tests to use ruby and will now test against Xvfb and Xephyr using:
    no window manager, openbox, and then gnome.
  - Fix build problems on Arch linux (or gentoo?)
    Fixes: http://code.google.com/p/semicomplete/issues/detail?id=20
  - Fix searching for only visible windows.
    Fixes: http://code.google.com/p/semicomplete/issues/detail?id=22
  - Fix xdotool search flags: --title, --name, --class
  - Add include for keysym.h to fix compile errors reported on the mailing
    list.

Logstash uses EventMachine, which is an event-driven library for Ruby. Part of logstash's requirements is the ability to watch logfiles like 'tail -f' would. Previously, I was using File::Tail, but this was not EventMachine friendly.

Additionally, it's pretty common for applications to write to files with generated names that include timestamps, etc, so it was clear logstash would need a way to watch a pattern of files, like a glob such as /var/log/*.log

Thus was born eventmachine-tail.

You can install it with:

gem install eventmachine-tail

And try the 'rtail' tool that comes with it:

rtail -x "*.gz" "/var/log/**/*"

The project is hosted on github: jordansissel/eventmachine-tail.

This is the first project I've tried to use git with. I'm not really happy with git as it only seems to complicate my workflow, but I'll try to stick with it.

While I wait for a mysql backup to replay against a test server, I was wondering how far along the replay was.

I am building the database using this command:

% gzip -dc admin.sql.20100511.060001.gz| mysql -uroot -proot

Pipes have a finite buffer. That is, the mysql command is busy reading stuff from stdin while gzip is busy writing to stdout. If gzip outputs faster than mysql can consume, gzip will end up filling the pipe's buffer and its next write will block, pausing it momentarily until mysql can catch up.

If we can inspect what position gzip is currently at, we can use that data along with the input file size to give us a progress indicator.

In linux 2.6.22 and beyond, there is /proc/<pid>/fdinfo which will tell you the current seek position of any open file in a process, so let's use that. First we'll need to find what file descriptor number is the input file, then ask what position it is at.

% pgrep -f 'gzip -dc admin'
6261
% PID=6261
% cd /proc/$PID/
% ls -l fd
total 0
lrwx------ 1 jsissel jsissel 64 May 11 15:04 0 -> /dev/pts/12
l-wx------ 1 jsissel jsissel 64 May 11 15:04 1 -> pipe:[31912237]
lrwx------ 1 jsissel jsissel 64 May 11 15:04 2 -> /dev/pts/12
lr-x------ 1 jsissel jsissel 64 May 11 15:04 3 -> /home/jsissel/admin.sql.20100511.060001.gz 
% cat fdinfo/3
pos:    149028864
flags:  0104000

# Get the file size
% size=$(stat -L fd/3 -c "%s")

# Get the position
% pos=$(awk '/pos:/ { print $2 }' fdinfo/3)

% echo "$pos/$size" | bc -l
.25203137570698241973

The above output says we are 25% complete.

You could go a step further and include the process uptime to show an estimate of time remaining.

cd /proc/$PID
size=$(stat -L fd/3 -c "%s")
pos=$(awk '/pos:/ { print $2 }' fdinfo/3)
start=$(date -d "$(ps -p $PID -o lstart=)" +%s)
now=$(date +%s)
echo "Minutes elapsed:"
echo "($now - $start) / 60" | bc -l
echo "Minutes remaining (estimated):"
echo "((($now - $start) / ($pos / $size)) - ($now - $start)) / 60" | bc -l

# Output
Minutes elapsed:
55.81666666666666666666
Minutes remaining (estimated):
110.58584668301847161105

Update: Some commenters pointed out pv as a solution here. It looks pretty good.

For various projects I'm doing right now, I need an easy way to automatically run code in an X server that may not necessarily be the active display. This code may even run on servers in production that don't have video cards or monitors attached.

For some history on this, check out this post on xvfb and firefox. You can solve the problem in that post by simply launching firefox with the tool below, and your X server will exit when firefox exits.

To solve my need for automated headless Xserver shenanigans, I wrote a script that will wrap execution with a temporary X server of your choosing. For xdotool tests I generally use Xephyr and Xvfb.

What's so special about this script? It picks an unused display number reliably and runs your command with that set as DISPLAY. This is a major win because you don't have to pre-allocate X servers (headless or otherwise) before you start your programs. ephemeral-x.sh will pick the first unused display, just as binding a socket to port 0 will pick an unused port. Finally, when your process exits, the xserver and windowmanagers are killed. Example usage:

% ./ephemeral-x.sh firefox
Trying :0

Fatal server error:
Server is already active for display 0
        If this server is no longer running, remove /tmp/.X0-lock
        and start again.

Trying :1
Xvfb looks healthy. Moving on.
Using display: :1
Running: firefox

It does health checks to make sure the X server is actually up and running before continuing.

You can also specify a window manager to run:

% ./ephemeral-x.sh -w openbox-session firefox
...
Trying :1
Xvfb looks healthy. Moving on.
Using display: :1
Starting window manager: openbox-session
Waiting for window manager 'openbox-session' to be healthy.
... startup messages from openbox ...
openbox-session looks healthy. Moving on.
Running: firefox

The default X server is Xvfb. You can use Xvnc, Xephyr, or any X server. Here we run Xephyr with some options:

% sh ephemeral-x.sh -x "Xephyr -ac -screen 1280x720x24" -w openbox-session firefox www.google.com
...
Xephyr looks healthy. Moving on.
Using display: :1
Starting window manager: openbox-session
Waiting for window manager 'openbox-session' to be healthy.
...
openbox-session looks healthy. Moving on.
Running: firefox www.google.com

Screenshot of what this looks like is here: xephyr-ephemeral-x-example.png

I use this script to run my xdotool tests. Additionally, parallelizing test execution can often lead to faster tests. Wrapping each test run with ephemeral-x.sh ensure that each test suite runs in a clean environment, untainted by any previous test, allowing me to run all test suites in parallel. Prior to writing this script, I would run each test suite in serial on the same X server instance.

I use a similar technique at work to start ephemeral X servers for running WebDriver tests in hadoop. Each mapper starts its own X server, safely, and kills it when it is completed. This is implemented in java, instead of shell, since the mappers all launch from java and mapreduce launch differently than a standard command would in your shell.

Code lives here: ephemeral-x.sh
xdotool tests: xdotool/t

I spent this weekend working on implementing automatic pattern discovery in the C version of grok. I updated the C and Ruby API to be able to use this discovery:

% irb -rrubygems -rgrok
>> grok = Grok.new
>> grok.add_patterns_from_file("/usr/local/share/grok/patterns/base")
>> pattern = grok.discover("There's no place like 127.0.0.1")
=> "\\QThere's no place like \\E%{IP}\\Q\\E"
>> grok.compile(pattern)
>> grok.match("There's no place like 24.55.33.55").captures
=> {"IP"=>["24.55.33.55"]}
>> grok.match("There's no place like 234.345.12.4")
=> false

This will most likely get used in logstash to provide a helping hand in building log patterns. It could also be used on events that have no pre-defined pattern so we can at least attempt to parse the log line.

On a more important note, this was one of the few remaining features the perl grok implementation had that the C version did not.

You can download the latest version of grok here. You can also install the ruby grok gem with gem install jls-grok.

On another project node, I pushed a new fex release that includes some small fixes and also an rpm spec.

Continuing the flashback series. Here's 2004.

Still in school. Hacking on countless projects in perl.

While writing a command-line interpreter, I wanted you to be able to type 'st' or 'sta' instead of 'status' and have the code find a best match for your command. This is often called 'ambiguous commands' or somesuch. I came up with a clever way to do this using regular expressions (May 2004).

I also posted about using regular expression to match quoted and nonquoted strings (June 2004).

Finally, I experiment with using regular expressions to find word boundaries near the cursor for my command line interpreter (Oct 2004).

Are you drowning in perl and regular expressions, yet? ;)

I'm working on rolling out iptables rules to all of our servers at work. It's not a totally simple task, as many things can go wrong.

The first problem is the one where you can shoot yourself in the foot. Install a new set of rules for testing on a remote server, and suddenly your ssh session stops responding. I covered how to work around that in a previous post.

Another problem is ensuring you make your firewall changes atomically. All rules pushed in a single step. In linux, if you have a script with many lines of 'iptables' invocations, running it will make one rule change per iptables command. And what if you write your rules like this?

# Flush rules so we can install our new ones.
iptables -F

# First rule, drop input by default
iptables -P INPUT DROP

# Other rules here...
iptables -A INPUT ... -j ACCEPT
iptables -A INPUT ... -j ACCEPT

If your server is highly trafficked, then the delay between the 'DROP' default and accept rules can mean dropped traffic. That sucks. This is an example of a race condition. Additionally, there's a second race condition earlier in the script where, depending on the default rule for INPUT, we may drop or accept all traffic for a very short period. Bad.

One other problem I thought could occur was a state tracking problem with conntrack. If previously we weren't using conntrack, what would happen to existing connections when I set default deny and only allowed connections that were established? Something like this:

iptables -P INPUT DROP
iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 --syn -j ACCEPT

I did some testing with this, and I may be wrong here, but it does not drop my existing sessions as I had predicted. This is a good thing. Turns out, when this runs, the 'conntrack' table is populated with existing connections from the network stack. This further helps us not drop traffic when pushing new firewall rules. You can view the current conntrack table in the file /proc/net/ip_conntrack.

What options do we have for atomically applying a bunch of rules so we don't drop traffic? The iptables tool set comes with 'iptables-save' which lets you save your existing iptables rules to a file. I was unable to find any documentation on the exact format of this file, but it seems easy enough to read. The output includes rules and counters for each table and chain. Counters are optional.

All the documentation I've read indicates that using 'iptables-restore' will apply all of the rules atomically. This lets us set a pile of rules all at once without any race conditions.

So I generate an iptables-restore file and use iptables-restore to install it. No traffic dropped. I'm generating it with a shell script, so there was one gotcha - I basically take iptables commands and output them to a file. I do this with a shell function I wrote, called 'addrule'. However, I have some rules like this:

addrule -A INPUT -p tcp -m limit --limit 5/min -j LOG --log-prefix "Denied TCP: " --log-level debug

I quoted the argument in the addrule invocation, but we need to also produce a quoted version in our iptables-restore rule file, otherwise --log-prefix will get set to 'Denied' and we'll also fail because 'TCP:' is not an option iptables expects. It appears to be safe to quote all arguments in the iptables-restore files except for lines declaring chain counters (like ':INPUT ACCEPT [12345:987235]'), defining tables (like '*filter'), or the 'COMMIT' command. Instead of quoting everything, I just quote everything with spaces in an argument.

The fix makes my 'addrule' function look like this:

rulefile="$(mktemp)"

addrule() {
  while [ $# -gt 0 ] ; do
    # If the current arg has a space in it, output "arg"
    if echo "$1" | grep -q ' '  ; then
      echo -n "\"$1\""
    else
      echo -n "$1"
    fi
    [ $# -gt 1 ] && echo -n " "
    shift
  done >> $rulefile
  echo >> $rulefile
}

# So this:
#   addrule -A INPUT -j LOG --log-prefix "Hello World"
# will output this to the $rulefile
#   -A INPUT -j LOG --log-prefix "Hello World"

So now the quoted arguments stay quoted. All of that madness is in the name of being able to simple replace 'iptables' with 'addrule' and you're good to go. No extra formatting changes necessary.

One last thing I did was to make sure iptables-restore didn't reject my file, and if it did, to tell me:

if iptables-restore -t $rulefile ; then
  echo "iptables restore test successful, applying rules..."
  iptables-restore -v $rulefile
  rm $rulefile
else
  echo "iptables test failed. Rule file:" >&2
  echo "---" >&2
  cat $rulefile >&2
  rm $rulefile
  exit 1
fi

Throw this script into puppet and we've got automated firewall rule management that won't accidentally drop traffic on rule changes.

It's time for a new keynav release. The main push for this release was to fix problems with building against the new libxdo (xdotool) versions.

Additionally, there is one new feature: "record"

Recording lets you record a series of actions in keynav and replay them later much like you would in vim. Only keynav actions are recorded.

0.20100302.*:
  - Started using glib for dynamic arrays (GPtrArray)
  - Uses new versioning scheme major.date.svnrev
  - Added 'version' (or -v or --version) to output the keynav version
  - Now requires libxdo.so.1 (via xdotool)
  - Add ability to record keynav actions with the 'record' command. Optional argument
    is a filename to save the recordings to for persistence across keynav runs.
    Example in keynavrc to bind 'q' to record to ~/.keynav_macros:
      q record ~/.keynav_macros
    Works similar to vim recording.
      1) Hit record once
      2) Type the key you want to record to
      3) Do things in keynav you want recorded.
      4) Any 'end' operation or hitting record again will terminate and save
         this recording.
    Recordings only persist if you specify a file to save to, otherwise they are lost
    across keynav restarts.

20091231.04:
  - Try repeatedly to grab the keyboard with XGrabKeyboard on 'start' commands.
    This loop is a necessary workaround for programs like xbindkeys that could
    launch keynav but at the time of launch still hold a keyboard grab
    themselves. Reported by Colin Shea.

20091231.02:
  - Nonfunctional bug fixes and other code cleanup from patches by Russell Harmon

Thanks to some early testing and feedback from the previous xdotool release, I've put together a new release that fixes a few minor problems.

Download: xdotool-1.20100302.2708.tar.gz

As usual, if you find problems or have feature requests, please file bugs or send an email to the list.

Changelist since previous announcement:

1.20100302.*:
  - Some test fixes suggested by Daniel Kahn Gillmor
  - Don't use isatty(0) anymore as it isn't totally reliable in determining if
    we should try to read from stdin.

1.20100301.*:
  - Minor libxdo patch to avoid exporting unnecessary symbols (Daniel Kahn Gillmor)
  - Minor Makefile fix to only link libxdo (and not xdotool) against libX11 and
    libXtst (also Daniel Kahn Gillmor)
  - Some test cleanup patches by Daniel Kahn Gillmor
  - Fixed the 'type hello' test that would fail sometimes when run in Xephyr.
  - Fix a minor bug in xdo_window_setclass that incorrectly aborts when
    XGetClassHint fails.

Matt Simmons suggested that this week be a flashback week. That's easy enough to do, so I'm doing one per year starting in 2003.

A common feature request of vim is to be able to split window and have a shell be there. Well, as it turns out, that's easy to do if you run vim inside screen (gvim not applicable here).

The original post details a simple vim keybinding that will split the screen and attach a shell and close the split when the shell exits.

August 2003: Neat vim/screen trick

However, I've got a better version now. You'll want this script, "screener.sh":

#!/bin/sh

screen -X split
screen -X focus down
screen -X screen sh -c "cd $PWD; $SHELL; screen -X remove"

Then in your .vimrc:

:map \s :silent !screener.sh<CR>

The new version mostly puts the screen invocations into a shell script. Another improvement is to change to $PWD so the new shell will be in the same working directory as vim started in.