handy makefile script

I frequently find myself looking for default make rules, and inevitably how to print their values.  Here's a script to help.

Adapted from https://blog.melski.net/2010/11/30/makefile-hacks-print-the-value-of-any-variable/

$ cat ~/bin/make-vars 
#!/bin/bash

_vars=
for v in $@; do
  _vars="${_vars} print-${v}"
done

make -f- ${_vars} <<'EOF'
print-%:
	@echo '$*=$($*)'
	@echo '  origin = $(origin $*)'
	@echo '  flavor = $(flavor $*)'
	@echo '   value = $(value  $*)'
EOF

Here it is in use

$ make-vars LINK.cc
LINK.cc=c++    
  origin = default
  flavor = recursive
   value = $(CXX) $(CXXFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)

New home for joinery

As of version 1.10, joinery is now hosted in the central repo using the sh.joinery groupId.

As part of the move to central, joinery also has a shiny new domain https://joinery.sh.

pipelines and functional java

Mr. Fowler has written a nice series on using pipelines instead of iteration for processing collections.

On a similar note, I recently came across this presentation on reactive programming in Java which also notes the benefits of composing operations as pipelines (though with a particular focus on processing sequences of observable events since this is reactive java after all).  If you read through the deck to the last slide, you'll notice a humorous meme which represents more concisely Fowler's main point.

I'm so glad this exists.

Obviously, as shown in my Joinery project, I'm a big fan of this programming style.  It provides a powerful mechanism for expressing the kinds of operations that are required when operating on large amounts of data while yielding a very high signal:noise ratio when reading the code.

apache zookeeper status with bash

Most Apache ZooKeeper users are familiar with ZooKeeper's four letter words. And many bash users know that recent versions of the popular shell can redirect to network ports. But I've yet to see the two used together. The other day I found myself without netcat and looking for a quick way to get stats out of ZooKeeper. A short bash function later and I can easily check on ZooKeeper from any shell (so much quicker than spinning up a JVM).

zk4 () 
{ 
    echo "${2-localhost}:${3-2181}> ${1-ruok}"
    exec 3<> /dev/tcp/${2-localhost}/${3-2181}
    echo ${1-ruok} 1>&3
    cat 0<&3
    [[ ${1-ruok} == "ruok" ]] && echo
}

Usage is simple, by default it will issue the ruok command to localhost on the default port 2181. But you can specify alternate values for each of the host, port, and command parameters.

zk4 stat
zk4 srvr
zk4 ruok zk-server1.myco.com 20181

sweet irony

No matter which political side you take on the HBGary story, you have to admit... the irony could not be more sweet.

http://arstechnica.com/tech-policy/news/2011/02/anonymous-speaks-the-inside-story-of-the-hbgary-hack.ars

a quick review

In lieu of actual new content... a quick recap of some good posts from several of my favourite sources:

1. Steve Yegge's excellent post on code, code base size, and perhaps a reason to use rhino - http://steve-yegge.blogspot.com/2007/12/codes-worst-enemy.html

2. Reginald Braithwaite's writings on code style and why clarity is king - http://weblog.raganwald.com/2007/12/golf-is-good-program-spoiled.html

3. Hamlet D'Arcy's humorous article on the detriments of new found functional programming knowledge - http://hamletdarcy.blogspot.com/2009/08/how-to-tell-your-co-workers-have.html

There you have it, just in case you missed 'em the first time around.

getting reacquainted with javascript

Ok, I'm admittedly a little late to the party here. Lots of people are doing really, really cool things with javascript these days.

So, I've been using rhino to whip together my java experiments lately. Setup is trivial, you just need java and js.jar.

1. download rhino 
2. extract js.jar (at least)
3. for an interactive shell, run java -jar js.jar
4. or run a script using java -jar js.jar some-javascript-file.js

Once you have rhino, all sorts of java-ish things can be whipped up as quick hacks.
A quick http server, for example, goes something like this:

function main() {
  var s = new java.net.ServerSocket(8080)
  while (true) {
    var client = s.accept()
    var sc = new java.util.Scanner(client.getInputStream())
    var method = sc.next()
    var path = '.' + sc.next()
    var out = new java.io.PrintWriter(
            new java.io.OutputStreamWriter(client.getOutputStream()))
    try {
      var f = new java.io.FileInputStream(path)
      out.println("HTTP/1.1 200 Success")
      out.println("Content-Type: text/html")
      out.println()
      for (var c = f.read(); c != -1; c = f.read())
        out.write(c)
    } catch (e if e.javaException
        instanceof java.io.FileNotFoundException) {
      out.println("HTTP/1.1 404 File not found")
      out.println("Content-Type: text/html")
      out.println()
      out.println("<html><body>")
      out.println("<h1>File not found</h1>")
      out.println("</body></html>")
    }
    out.flush()
    out.close()
  }
}

main()

Wait, what about a threaded server you say? Try this on for size.

function main() {
  var s = new java.net.ServerSocket(8080)
  while (true) {
    var client = s.accept()
    var t = java.lang.Thread(function() {
          var sc = new java.util.Scanner(client.getInputStream())
          var method = sc.next()
          var path = '.' + sc.next()
          var out = new java.io.PrintWriter(
                  new java.io.OutputStreamWriter(client.getOutputStream()))
          try {
            var f = new java.io.FileInputStream(path)
            out.println("HTTP/1.1 200 Success")
            out.println("Content-Type: text/html")
            out.println()
            for (var c = f.read(); c != -1; c = f.read())
              out.write(c)
          } catch (e if e.javaException
              instanceof java.io.FileNotFoundException) {
            out.println("HTTP/1.1 404 File not found")
            out.println("Content-Type: text/html")
            out.println()
            out.println("<html><body>")
            out.println("<h1>File not found</h1>")
            out.println("</body></html>")
          }
          out.flush()
          out.close()
        }
      )
    t.start()
  }
}

main()

For more reading about rhino and javascript including how to structure, organize, and manage your code for larger projects, try these great posts.

http://steve-yegge.blogspot.com/2008/06/rhinos-and-tigers.html
http://www.adequatelygood.com/2010/3/JavaScript-Module-Pattern-In-Depth
http://peter.michaux.ca/articles/javascript-widgets-without-this
http://www.jspatterns.com/

reading files in java

Just came across this aging, but still surprisingly relevant analysis of reading files in java. Basically compares the relative performance of several common (and some newer, lesser known) file reading methods... quite interesting.

emergency lisp

Keep this excellent lisp post handy. It is perhaps a little dated, but only in terms of the actual date of the post. It provides a good introduction to lisp programming, especially when transitioning from another, less functional language. Even though it deals specifically with emacs lisp, the concepts are close enough.  Definitely useful the next time a co-worker needs convincing using clojure.

your test suite is broken

Take the quiz and fess up if your test suite is broken. Remember, test suites are software too, and as software, they need maintenance and upkeep just like the rest of the code your write.

looking forward to 2.6.35

Based on the changelog and this writeup, the latest linux kernel release appears to be pretty power-packed. I'm particularly looking forward to the performance improvements around spreading network load across cpu cores.

On a slightly related note, I find the whole kernel code shepherding process amazing. How many managers have the ability to rattle off the number of post-release-candidate commits (with averages and comparisons against previous release candidates)? Never mind the whole staging and merging process. I think many software teams could learn quite a bit about managing the development process across distributed teams by letting go of centralized source control and watching the linux kernel developers.

nice perl tip

I just came across this excellent perl tip. I'm not sure I'd use it exactly as written, but the idea of being able to open perl modules by module name instead of file name is certainly appealing. Now, to figure out how to apply this to other languages too.

autonomous agents for software testing

I've been thinking about software testing quite a bit lately. I've been thinking about the artificial intelligence/algorithms used in game programming too. And so naturally, at some point it occurred to me that the same algorithms used to make all those characters, monsters, cars, or whatever meander in and out of the background scenery of our favorite games might be useful in software testing scenarios.

Imagine you need to test a particularly hard-to-pin down bug - one that only happens during periods of high usage. It seems to me that you have a couple of options. You can force your users to pitch in and help with testing (not likely, and certainly not popular). You can beg your fellow programmers to do their best to mimic users (also not likely or popular). Or lastly (wishful thinking) you could fire up some program which will spawn an army of autonomous agents to meander through common use cases and behave as normal users creating the necessary "background traffic" needed for testing.

This is currently just an idea in its infancy, I don't know of any actual implementations. But it does seem like an interesting area for some research and coding. One could imagine initial versions would need lists of steps that could be invoked at random intervals and sequences by many agents through some programming interface. However, far off future versions might know how to inspect user interfaces for menus, buttons, and dialog boxes - simulating mouse clicks and keyboard entry (valid or invalid) until something happens in response.

Although I haven't found anything this sophisticated yet, it occurs to me that areas of security research might already have made progress on similar tools. Black-box penetration testing usually involves sending random (or perhaps not so random) payloads in an attempt to illicit an unexpected error or otherwise interesting response.

So, any takers want to whip something like this up for me?

simple overrides for quick mocking in perl

A more concrete example (and one I use too frequently) of overriding for testing similar to what is described by Sawyer X at blogs.perl.org: Simple symbol overriding for tests. Of course, my example uses inheritance and re-blessing rather than symbol overriding, but the result is basically the same. I like to use the snippet below to test mail sending functionality by simply printing the resulting mail to stdout.

my $smtp = Net::SMTP->new($mailhost);
...
if ($i_dont_want_to_send_mail) {
  package Mock::SMTP;
  our @ISA = qw(Net::SMTP);

  no strict qw(refs);
  for (qw(mail to data dataend quit)) {
    *$_ = sub { };
  }

  for (qw(datasend)) {
    *$_ = sub { shift; print @_, "\n" }
  }

  # re-bless my Net::SMTP reference to a Mock reference
  $smtp = bless $smtp || {};
}

screen for cluster management

After my parallel ssh roundup and hacking my own solution...

I just realized screen can do this too using "at" and "stuff".  Here's how:

1. open a bunch of screens and ssh to various hosts
2. get to a screen command prompt (control key followed by colon, ^C-a : )
3. at the prompt, enter the command "at \#", this tells screen to run command that follows on all windows (yes, include the backslash and don't press enter yet)
4. continue with the command "stuff", which tells screen to quite literally stuff whatever follows it into the input of a screen terminal (don't press enter quite yet)
5. follow stuff with the command you want to run, you'll need to put it in quotes if the command has arguments (not yet)
6. lastly, you need to end the line with a return so the shell(s) will run the command, so end the line with "\012" which is octal for a newline character

Now, press enter and what your command get run across all of your screen sessions.  This is particular cool for long running commands where you can switch back and forth between screen windows and check on the progress of many machines or also useful if you have a couple of quick maintenance task to run on a bunch of hosts.

So, putting it all together, the whole sequence looks like this:

^C-a : at \# stuff ls \012
^C-a : at \# stuff 'ls -alR' \012

a survey of web visualization toolkits

I'm always on the lookout for new tools to create graphs, charts, and other visualizations. It is an obsession that dates back to my days slaving over gnuplot and R scripts (or even *gasp* metapost). Here are some more modern toolkits I've come across which lend themselves to visualizations for the web.

• Processing has to be one of my favorites, probably because I've been using it the longest.
• And if Processing isn't web-2.0-enough for you, there is always the Javascript implementation.
• Protovis - any visualization toolkit that can be used to create an interactive version of Minard’s Napoleon has to get good marks.
• Google has Chart Tools.
• Degrafa has been used to create some beautiful visualizations.
• Or there is the slightly higher-level Axiis if you prefer (which is built on top of Degrafa).
• Should you be religiously opposed to flash, you might like dygraphs
• Prefuse is another Java toolkit with some great graph and chart types
• And Flare (also from the Prefuse guys) you post those interactive visualizations to the web in flash.

Ok everyone, which ones haven't I seen yet?

broadcast ssh - my entry into the world of parallel or cluster ssh tools

I threatened before when writing up my review of the parallel or cluster ssh tools out there that I would write my own. And after a quick review of paramiko, especially the demo scripts included, this turned out to be a pretty quick hack.

The basics of my ssh client are this:

• prompt for username and password - ssh keys not required (although I would like to add support for using them if available in the future)
• interactive use - I often want to look at things, which leads me to want to look at other things; in other words, I don't want to be constrained by having a list of commands to execute up front
• parallel - I have a bunch of commands and a bunch of machines, a simple loop executing each command on each machine isn't going to cut it

Those last two together make things a little tricky, the interactive bit means you have two choices.  You can be line oriented, prompt for a command and send it to each host or you can be completely interactive and send each character.

I started with the former, but soon found that if you only send commands, you have practically no environment setup (that is all done in the shell, remember...).  So, despite the difficulties in dealing with the terminal, that is what this implementation does.

A terminal based solution has its own issues, the line buffering of output from each host has to be dealt with to avoid interleaved garbage as the results of each command.  But in the end that wasn't too hard.

In the end, usage is simple, cut-n-paste the code, save it as bssh.py and run something like:

./bssh.py host1 host2 host3 host4

Enter a username and password at the prompts, and away you go, just enter commands as you would with any other shell. You will of course need paramiko and its dependency pycrypto installed and available.

dnode - javascript remote invocation

dnode is just the latest in a serious of cool things people are doing with javascript that I find amazing. processingjs is another (albeit not very new anymore).

on lunch and bike sheds

Today, Seth Godin hits on a pet peeve of mine, arguing about unimportant things.  Although he casts the problem as a discussion from the perspective of deciding on lunch, I like the programmers version better.

We all know the type, whether sitting in a code review or discussing a database schema, there is always one person willing to spend hours discussing minutia.  Either it is the indentation style of a particular comment instead of the correctness of the algorithm.  Or it is the virtues of camel case names for database objects as opposed to the performance trade off of complete normalization.

operator brings semantics to firefox

I just came across operator, a firefox add-on for recognizing microformats and I have to say I'm impressed. I think this is a pretty big step in the right direction for the semantic web.

As far as the semantic web goes, I've always been kind of on-the-fence regarding the human annotation versus automated statistical approaches. I'm still not convinced either will win out. It seems unlikely that every web author will annotate all their pages -- and yet, I'm not sure machine learning approaches will ever be able to accurately annotate things automatically.

But with all of the generated pages out there, having just a few of the big ones support microformats, and browser support for recognizing those pages is a really big step. For example, linkedin already returns profile pages in hresume format. With operator, you can extract names, phone numbers, addresses, and event information easily.

Update: operator doesn't seem to work in firefox 4 beta, but I think we can forgive that for now.

FDA52EB2TTEK