Using Ansible to install Bittorrent Sync on all my PCs.

There are so many PCs in my house, and they are becoming a pain to keep up to date and consistent. Recently I’ve been writing Ansible scripts to automate all the configuration I used to do by hand. It’s been my new season’s resolution not to do any more manual configuration, especially for my home PCs.

Ansible makes installing software, SSH keys and setting up user accounts really easy.

It can even configure some aspects of better-behaved packages. For example, here’s my Ansible playbook for installing Bittorrent Sync – a fabulous alternative to rsync.

- hosts: btsync
  sudo: True
  tasks:
  - name: Add repository
    apt_repository: repo='ppa:tuxpoldo/btsync' update_cache=yes state=present

  - name: Install packages
    apt: pkg={{item}} state=installed
    with_items:
    - btsync
    - debconf-utils
    - btsync-gui
    - transmission-daemon

  - name: Configure BT Sync
    debconf: name='btsync' question='btsync/runas' value='sal' vtype='string'
    debconf: name='btsync' question='btsync/directory_root' value='/home/sal' vtype='string'
    debconf: name='btsync' question='btsync/webgui-bindaddr' value='0.0.0.0' vtype='string'
    debconf: name='btsync' question='btsync/folder_defaults-use_lan_broadcast' value='true' vtype='boolean'
    debconf: name='btsync' question='btsync/log_size' value='1' vtype='string'
    debconf: name='btsync' question='btsync/folder_defaults-use_dht' value='true' vtype='boolean'

  - name: Regenerate BT Sync's config file
    command: dpkg-reconfigure -f noninteractive btsync

  - name: Restart BT Sync service
    service: name='btsync' state=restarted

First of all it ensures that the PPA containing the Bittorrent Sync software is installed. Setting update_cache to “yes” is the equivalent of an apt-get update.

Next it installs a bunch of packages required to run and configure the applications I need. Ansible provides an interface to allow me to set debconf variables but does not seem to provide anthing to recompile the configuration, hence I have to remotely execute the dpkg-reconfigure command.

Once it’s all done I can simply restart the service.

To get all this working easily I need a few extra things. First of all, a hosts file – that defines the group of hosts that I want to install this package to. I’ve defined a group of hosts called “btsync” – these are all of the machines I want to install the Bittorrent software onto.

[btsync]
bobnit
ahfang
tinyenid
halob

It’s just a list of host-names in an ini-style text-file.

And I also need a handy way to run it. To save me from remembering the command-line syntax I wrote a small shall-script:

#!/usr/bin/env bash
source venv/bin/activate
export ANSIBLE_HOSTS=./hosts
ansible-playbook -s --ask-sudo-pass --ask-pass btsync.yml

The -s option forces everything to run as if from a sudo command. The –ask-sudo-pass and –ask-pass command line arguments make the playbook-runner ask the user for passwords before starting. That saves me from having to store any passwords in any of the configuration files.

Running the script should produce an output a little like this:

ansible_output

Advertisement

Advertise a new Python job by making a github push

My colleagues Steve Stagg has created a wonderful new thing – an entirely free Python jobs board.

Anybody can add a new job simply by making a GitHub pull request. It’s a work of genius because it’s the absolute simplest possible solution to a problem that’s been bothering me for weeks – and the beauty of this solution is that they’ve figured out a way to make a service that we’d normally have to pay for entirely free.

Not so long ago we used to advertise on the Python.org job board – it was a very effective way to find good people. Unfortunately the old board relied on the good will of a very small pool of volounteers and nobody within the Python Foundation seems to have the time to work on it. It’s not been updated since last summer. We wanted to create an alternative that wouldn’t be such a burden for the community that runs it – and that means we needed something that would be cheap to host and cheap to administer. The solution wasto run this jobs board like an open-source software project.

If you want to submit a new job to this site all you have to do is make a GitHub pull request to the jobs repo. There are no forms to fill in. There’s no money to pay. Advertising for a developer job is no longer something you get an HR department to do – it becomes part of your development workflow.

Certainly submitting a job on this site is going to seem a bit technically challenging for the kinds of people who normally work in recruiting – but I think that’s a feature and not a bug: We hope that by taking advantage of GitHub’s well proven code-review mechanism and it’s superb scalability we can make a community driven job board that helps developers find their perfect jobs – and we can make it entirely free of charge forever.

These are the principles we want to aspire to:

  • Absolutely Free, Forever: We don’t want to charge for posting a job advert – ever.
  • Anybody can post: From multinational to mom & pop business you are welcome to advertise your jobs.
  • Agents, IT Consultancies and Headhunters – you are welcome too: We just want you to be open about exactly who you are recruiting for.
  • Everything is open source: The code we use to build the static site, even our collection of jobs. Feel free to take it all, or better still help us to improve.

We just launched the site yesterday – we are really keen to know how you feel about this new service. If you have any feedback please log an issue via our GitHub tracker.

My newest project: Getting code onto Pyboards the easy way

My latest python project is a way of making Mycro Python “pyboards” easier to use.

Pyboards are small microcomputers which can execute code written in Micropython, which is a Python3 compatible language. It has the same grammar as cPython3 and includes a small subset of the standard library.

They are an amazing invention because they open up the world of microprocessor development to just about anybody with basic computing knowledge. They don’t require a compiler or any special SDK. You can get projects running on a Pyboard with just a text-editor.

DSC_2060~2

The suggest way to deploy code to a PyBoard is to mount it as a USB Mass Storage device, and then edit files directly on the device, or simply copy over the relevant files from a PC. This approach can be problematic because if the pyboard is reset, it can cause the Mass Storage drive to be suddenly unmounted. At best this will cause confusion in the operating system when an open file-system just disappears.

Worse problems can occur if the device needs to be factory reset – this will wipe the code on the device and will require everything to be re-copied. If you were editing directly on the pyboard then you’ve just lost all your code. Clearly this way of working is not optimal.

Using USB Mass Storage mode is a real pain. It would be much simpler to have something that was able to use the Serial mode connection to quickly copy code across without any kind of stateful connection. Achieving that kind of developer friendly connection is the goal of this project.

PybKick 0.0.1 was released just yesterday. My first attempt can quickly copy a directory of python code onto pyboard and not much else. Right now it does not know how to handle heirarchies of code, or how to put code anywhere other than the root directory of the pyboard.

Low-hassle PEP-396

PEP-396 suggests that all python packages should provide a __version__ attribute. This should contain a string in the form x.y.z which gives you the package’s version number.

That seems simple enough, except that Python does not provide an obvious way to set this attribute. It would be a tremendous waste of time to set this manually before each script.

The obvious (and in my opinion wrong) way to fix this problem is to add something to the release process which automatically writes some text to the package’s __init__ file. Personally I don’t like self-modifying code. Python is dynamic enough already!

A much simpler way to do it is to exploit the pkg_resource module. In your package’s main __init__.py file, simply add the following lines:

import pkg_resources
__version__ = pkg_resources.working_set.by_key['<your package name>'].version

Find the most common item in a sequence of items

Here’s a puzzler posed to me by Chris Laffra, one of the Bank of America Python gurus: Given a finite sequence of objects which contain some repetition find the most common object.

Here’s an example followed by my own naive solution:


input = 'aaabaccabdaabacccabdagatacaccacba'

counts = [(input.count(x), x) for x in set(input)]
sortedCounts = sorted(counts, key=lambda x:-x[0])
print sortedCounts[0][1]

This is pretty basic stuff, and the only thing I can say in defence of this first solution is that I bashed it out quickly. But after a few minutes of pondering I managed to find the optimal solution:


print max(input, key=input.count)

That’s only 18 bytes, not including whitespace and the print statement. This is a lovely example of what makes Python truly expressive – the ability to say a great deal in very few keystrokes.

Using SublimeText2 with VirtualEnv

I recently switched from using Eclipse as my main Python development environment to SublimeText2. The main reason to switch was that Sublime seems to work better on smaller screens: I figured that a little investment in learning this package might make me more productive when I work on my laptop.

The problem with Sublime is that when first installed it does not do a whole lot other than edit text. You need to configure your project or build-settings before it can actually launch code you’ve written. And while it does come with some pre-set ‘Build Systems’ (that’s Sublime’s jargon for a run-configuration), none of them are ideally suitable for Python development with VirtualEnv.

After a great deal of trial and error I’ve come across a formula for setting things up which seems to work rather well. First of all, you need to set up each of your workspaces in a standardized way.

  • Begin by making a new virtualenv – use whatever configuration you like, for example:

virtualenv myproject
  • Next checkout whatever code you want to work on into the /src folder of your new virtualenv dir. This folder should be the location of your setup.py file
  • Activate the virtual environment in the usual way:

source myproject/bin/activate
  • Build your project, install any test-dependencies and verify that everything works from the command-prompt.

Now that your basic environment is provisioned, all you need to do is provide a sublime-project file. Here is one I made earlier. Save that file into your main project folder (not the /src folder). You might have to edit some of the paths to point to the actual locations of your tests. Since all my projects use the same configuration layout I can recycle most of this configuration from one project to another. The only things I ever need to change are the locations of specific things in my projects.

The important section to take note of is build_systems – these are the various configurations for how to run my project. You will notice that I don’t actually use the activate or deactivate commands. Rather than modify the environment it turned out to be a great deal easier just to ensure that everything is invoked using the virtualenv’s copy of Python rather than the system version.

Yo Dawg, I heard you like Jenkins

These past few days I’ve been busy re-factoring my JenkinsAPI project. One of my goals was to greatly increase test-coverage. It’s somewhat ironic that a project that was all about testing had very few tests.

Most of the tests in this package try to do something to a Jenkins server and then check for consistency:

For example you can start with a new, clear Jenkins. If you add a new Jenkins job and then read-back the list of jobs you might expect to see one more job in that list. This sort of test is easy to get working on a PC, you simply start up Jenkins before running the tests. But what of the situation where these tests are run on a remote CI server, for example on Jenkins. Any tests that connect back to the server and mess about with that server’s configuration are bound not to work reliably.

I heard you like Jenkins

In order to get this kind of test working in a Jenkins environment the solution I found was to launch another Jenkins.

I wrote a threaded Jenkins launcher which starts a background sub-process and then monitors the Stdout and Stderr streams for error conditions. It returns, keeping the sub-process alive once the Jenkins server has fully started. We keep a reference to the subprocess so that when all the tests are complete I can shut the whole thing down.