broadcast ssh - my entry into the world of parallel or cluster ssh tools

I threatened before when writing up my review of the parallel or cluster ssh tools out there that I would write my own. And after a quick review of paramiko, especially the demo scripts included, this turned out to be a pretty quick hack.

The basics of my ssh client are this:

• prompt for username and password - ssh keys not required (although I would like to add support for using them if available in the future)
• interactive use - I often want to look at things, which leads me to want to look at other things; in other words, I don't want to be constrained by having a list of commands to execute up front
• parallel - I have a bunch of commands and a bunch of machines, a simple loop executing each command on each machine isn't going to cut it

Those last two together make things a little tricky, the interactive bit means you have two choices.  You can be line oriented, prompt for a command and send it to each host or you can be completely interactive and send each character.

I started with the former, but soon found that if you only send commands, you have practically no environment setup (that is all done in the shell, remember...).  So, despite the difficulties in dealing with the terminal, that is what this implementation does.

A terminal based solution has its own issues, the line buffering of output from each host has to be dealt with to avoid interleaved garbage as the results of each command.  But in the end that wasn't too hard.

In the end, usage is simple, cut-n-paste the code, save it as bssh.py and run something like:

./bssh.py host1 host2 host3 host4

Enter a username and password at the prompts, and away you go, just enter commands as you would with any other shell. You will of course need paramiko and its dependency pycrypto installed and available.

parallel ssh tool roundup

So I'm in the market for a good "parallel" ssh tool.  Basically, I want to ssh and type some commands like I always do except that instead of one command output, I want the command to be run on a bunch of machines and I want responses from each.  I've used things like mpiexec in the past, but I was hoping for something more ad-hoc.  I just want to specify hosts on the command line (with no prior setup).  I really don't even want to required ssh keys or having the same password if I can avoid it.  Like I said, just ssh as usual, run some commands, get multiple results.

Anyway, these are the tools I've come across (in no particular order).  I'll outline what I regard as the advantages and disadvantages of each below.

1. pdsh - "a high-performance, parallel remote shell utility" from Lawrence Livermore National Laboratory
2. pssh an implementation of some parallel ssh tools in python
3. dsh - dancer's / distributed shell
4. pydsh - a python version of dancer's shell
5. clusterssh - "a tool for making the same change on multiple servers"
6. mussh - "a shell script ... to execute a command or script over ssh on multiple hosts"
7. sshpt - "SSH Power Tool (sshpt) enables you to execute commands and upload files to many servers simultaneously via SSH"
8. multixterm - part of the expect project
9. clusterit - "a collection of clustering tools, to turn your ordinary everyday pile of UNIX workstations into a speedy parallel beast"
10. dish - "The diligence shell 'dish' executes commands via ssh/rsh/telnet/mysql simultaneously on several systems"

go python!

Python 2.7 was released yesterday - and it looks like some cool python 3 features were back-ported.

django ssl decorator

I recently found myself needing a way to require some django views be accessed only via a secure channel, as in ssl. A quick search led me to this post. I was able to quickly get up and running with this snippet and a couple of standard view functions, but ran into problems when trying to decorate a FormWizard view. The problem lied in the fact that FormWizards are implemented as classes with overridden __call__ methods. Long story short, I reworked the decorator as a class to work with functions and classes implementing __call__.

try:
  from functools import update_wrapper
except ImportError:
  from django.utils.functional import update_wrapper

class ssl_required(object):
  '''Decorator class to force ssl'''
  def __init__(self, func):
    self.func = func
    update_wrapper(self, func)

  def __get__(self, obj, cls=None):
    func = self.func.__get__(obj, cls)
    return ssl_required(func)

  def __call__(self, req, *args, **kwargs):
    if req.is_secure() or \
        not getattr(settings, 'HTTPS_SUPPORT', True):
      return self.func(req, *args, **kwargs)
    url = req.build_absolute_uri(req.get_full_path())
    url = url.replace('http://', 'https://')
    return HttpResponseRedirect(url)

organizing django unit tests

I've had a revelation regarding my unit test organization.

Until recently, I was creating test case subclasses along module boundaries, and defining large numbers of tests within each case to exercise the functionality of that module. For example:

class ModelTestCase(TestCase):
    def test_some_model_behavior(self):
        MyModel.objects.create(name='foo')

    def test_some_other_model_behavior(self):
        foo = MyModel.objects.get(pk=1)
        MyOtherModel.objects.create(other=foo)

Now, we've ensured that all of the setup for ModelTestCase is performed before OtherModelTestCase, and we can safely sequence each step of a process while testing each step without a lot of copy and paste code. Admittedly, super is kind of tedious, but that is another post on another day.

This worked ok for a while, but eventually, my applications would grow to the point where the cases are not simple gets and creates, but rather entire stories about how a potential user navigates the site.

Consider unit testing a user registration and login, for example. Each of the steps, from getting the registration form to activating the account, and finally logging in needs testing. The real headache comes with trying to organize all those tests so they run in the correct order. One, certainly valid, approach would be to simply put the entire story in a single test. However, this left me repeating certain setup code, and if there's one thing for which I have no patience, it is violations of the DRY principle.

My solution, subclassing test cases to enforce order.

class ModelTestCase(TestCase):
    def setUp(self):
        self.foo=MyModel.objects.create(name='foo')

    def runTest(self):
        self.assertEqual(self.foo.name, 'foo')

class OtherModelTestCase(ModelTestCase):
    def setUp(self):
        super(OtherModelTestCase, self).setUp()
        self.bar=self.foo.bar.latest()

    def runTest(self):
        self.failUnless(self.bar)

Now, I am assured that the ModelTestCase setup has been performed before OtherModelTestCase is run. This allows me to easily sequence steps of a process (like user registration for example, view, submit, email, activate) without having to repeat a lot of code. Admittedly, having to call super all over the place is tedious, but that is another post on another day.

simple loops in django templates

Every once in a while, I need to repeat some block of html in a django template a fixed number of times. Ideally, I'd just be able to use python's range function. But, django templates don't allow raw python (probably for the best, once you start coding in your template, its a slippery slope). Anyway, for these simple cases, I finally figured out an easy solution:

{% for i in 123|make_list %}
    {{ i }}
{% endfor %}