adoption curve dot net

One of the – ahem – undocumented features of Movable Type is a certain estotericness about the way it interprets Boolean logic within template tags. That’s a polite way of saying it’s more-or-less completely unpredictable, which can lead to much wailing and gnashing of teeth during template development.

Fortunately, there’s a workaround, an example of which is below. It comes at a computational cost, because you’re using PHP to fill in for MT’s deficiencies – but if you’re using static templates, then that cost is only really incurred as the templates are rebuild

This code assumes that you’re checking for two conditions – the presence of a custom field called “magazine_issue” that contains “Issue 19″, and a second custom field called “entrydatasource” which should contain the value “DVD”.

There’s a PHP loop inside an MT loop here – lines 1 and 2 set up the MT part, and iterates across the entries in the blog looking for those which have the field “magazine_issue” set to “Issue 19″.  (The MT:Entries tag is split across two lines for readability here, rather than any other reason.) Then those entries are passed one by one to the inner PHP code.

Line 5 places whatever’s in <mt:entrydatasource> into $source, relying on the fact that MT tags are parsed first, so by the time the PHP code gets executed that tag has been replaced by the value inside it.

Then lines 7 to 11 run a PHP if statement to check that the content contains “DVD”, and executes the <mt:include> tag if that’s the case.  The positioning of the ?> and <?php braces allow you to place MT code within the PHP loop, but not to have the PHP parser attempt to interpret it.

If the if statement doesn’t evaluate to true, then the contents won’t be executed and the include statement won’t be executed.   Obviously the include could be replaced by pretty much any MT tags that are valid within the context of an <MT:Entries> loop, or whatever arbitrary PHP you wanted.

It’s slightly hacky, and does come with a processing overhead – but sometimes a hack is the only way to get out of the problem.

With all the controversy over MPs’ expenses, I wasn’t unduly surprised to see that Parliament have released the raw data in PDF form. Nice for presentational purposes, but bugger-all use for analysis. Funny, that?

I briefly toyed with attempting to convert from PDF to spreadsheet, but the Guardian got there first – and they’ve released the raw data in Google spreadsheet form so anyone can mash it up and see what happens. Cue some hacking…

MySociety’s TheyWorkForYou site has an API that allows you to do all kinds of intriguing things, including retrieving geographical data about constituencies. A few Ruby scripts later, I’ve got the raw data pulled down – the latitude and longitude of the constituency centre point, from which it’s a very quick job to calculate the distance from Westminster using a great circle calculation. I figured that as I was mainly interested in travel, using Westminster tube station as the centre point from which distances were calculated would be a good-enough approximation. Of course, it’s a bit of a rough calculation – the centre of the constituency isn’t necessarily the point from which the Member travels, and if the constituency has an odd shape it’ll distort the result slightly. But no more than they’ve been distorting the system…

The results are here –> http://spreadsheets.google.com/ccc?key=pX5xUAGpnOKXBe3rY77M9AA&hl=en_GB There are one or two glitches – for some reason, the Northern Ireland constituencies don’t have lat/long data so those are broken, and there’s the possibility that things got slightly mixed up for MPs that share surnames. But a quick scan suggests it’s OK.

The next stage would be to start correlating between distances and expenditure on travel, and plotting this in interesting ways. Cue a rapid learning curve on Google Maps.   This is something of an overnight hack project, but it does raise some interesting questions about the way data can be interpreted in ways that the originators probably didn’t expect – I’d be amazed if Google Maps mashups ever crossed the minds of the Commons Fees Office when they (reluctantly) released the information.   Given the raw data, some controversy and geeks with time on their hands, it’s unsurprising that interesting things result.

To someone who’s not familiar with the concept of open-source development, it can be a difficult one to get their head around. “You mean people do this for *free*? And give their code *away*?” “Well, yes, they do.” If your default frame of reference is copyright and closed-source, this can be something of a head-spinner.

This video visualises the “many hands make light work” paradigm at the heart of open source. It’s a visual representation of contributions to the Ruby on Rails project, from the inception back in 2004 through to the end of last year. It doesn’t seem too significant until about 5 minutes in, when the activity explodes as Rails really takes off. As individuals contribute more code, they gravitate towards the centre of the mesh, so you can track the magnitude of people’s involvement as they fly in and out.

Ruby on Rails from Ilya Grigorik on Vimeo.

The underlying point is that Rails – like many, many other open-source projects – isn’t a commercial undertaking, yet it has a community of active developers which would be almost impossible to replicate in a commercially-oriented environment. And sometimes you need a picture or two to bring the point home.

Charles Arthur (the Grauniad journalist) recently put up a blog post passing on some advice to would-be journalists to “learn to code”.

Not suprisingly, that kicked off a bit of a discussion around various other blogs: Chris Applegate, who describes himself as a “wannabe polymath” chipped in with recommendations to get involved with regular expressions, comma-separated variables and Yahoo Pipes. And Tom Armitage – who *IS* a programmer, even if he hotly denies it – suggested thinking like a programmer rather than concentrating too much on the specifics.

It struck me that here’s one debate where I am actually qualified to take part – I’m not a programmer or developer as such, but I do need to herd them on a daily basis. And I also need to sit between them – who generally prefer specific, down-to-earth codeable specifications – and clients, who instead prefer to talk about their requirements in terms of “we want it do, you know, STUFF“.

That’s partly because I originally trained as an electronics engineer, which is right at the crossover point between hardware and software, and also partly because I’ve got a bad case of ADD when it comes to all things technical and geeky. I was the kind of irritating child who liked nothing better than taking clocks to bits to see what made them work, and that’s something that’s stuck.

On reflection, this also makes me something of a pain in the ass to work FOR, because although I don’t know enough to necessarily do the development job myself, I do know enough to know when someone is trying to bullshit me on a technical level. Fortunately I work with a bunch of the finest geeks available who don’t do this kind of thing, but it’s happened plenty of times in the past.

Both Tom and Chris have got it right – you do need some specific skills, and you do need some conceptual ones. I’ve got a list of things that I think are pretty much indispensible if you want to make yourself into the kind of person who can do lots of things for lots of people – they might not be the best grounding for a pure programming career, but they’re a good starting point if you’re at that meeting ground where things just need to Get Done.

The first skill – and I’m surprised this didn’t get mentioned explicitly – is HTML and CSS.

What I emphatically DON’T mean is the kind of CSS wrangling that produces beautifully fluid works of semantically-correct markup art – that’s expert territory. But if you work from the assumption that you’re no-one unless you’re on the web, then that means having some kind of web presence.

While you can get a long way with “canned” services like Typepad and Wordpress.com, sooner or later you’re going to bump up against the limitations – and knowing enough HTML and CSS to be dangerous will be a big help in these moments. The purists may turn pale at the thought of HTML tables, but if it’s that or looking like a MySpace refugee, then go for it. And it’s easy to pick up, because the web is the ultimate open-source guide – “View Source” on a page, and you’ll be able to figure it out eventually.

The second skill – which was mentioned by everyone – is the ability to understand, and ideally manipulate, data and databases.

Pretty much every interactive site has a database of some sort behind it, and if you need to manipulate large quantities of data for any reason then a simple database will generally get you a lot further than munging it in Excel or whatever. Understanding one-to-many relationships also prepares you to ask questions about the underlying nature of the data as well, which can often be as important as the data itself.

While knowing enough SQL to be a database administrator might be overkill, at least understanding a simple “select * from table where…” statement can be the key to unlocking some real insight from otherwise overwhelming quantities of text and numbers.

And then finally – and personally, I think most importantly – is something that’s actually far less concrete than anything Charles or Chris suggested, and more along Tom’s lines. It’s the ability to have some diagramming techniques in your personal toolbox.

Words are incredibly bad at explaining complex concepts unambiguously, something that this blog is a case study for. Words are even worse at explaining relationships between concepts or stakeholders or actors – once you get beyond a simple “he knows her and she knows them”, it breaks down completely – whereas a simple diagram is something even a six year old can pick up.

There’s any number of techniques out there, from flowcharting to process flows to mindmaps to entity relationship diagrams. Some are more complex than others, and not every technique is applicable to every situation. But once you’ve started trying to reduce a situation to a diagram, you’re applying analysis to it – which means that you’re guarding against problems like doublethink and reductio ad absurdum.

It frustrates me enormously that in 10 years of full-time education, no teacher has ever taught my kids how to draw a simple mindmap, which has to be one of the most powerful techniques for controlling complex concepts that’s out there. That’s been left to me to do, with the (mixed) results you’d expect from an amateur.

I’m not pretending that this is advice from anything other than my own personal experience and perspective, and the people that know me may disagree about how well I manage to cope with complex situations and hack around with HTML and so on. But it does strike me that Charles, Chris and Tom are all onto something – that there ARE skills that would previously have been regarded as the preserve of the professional geek which are actually incredibly useful to the rest of us, if only we took the time to pick them up.

My stock market Twitter bots (@ftse100, @nikkei225 and @dowjonesindex) are tweeting away nicely, but they’re all a bit – well, online, really.   I think it would be far cooler if it was more visual and tangible.  This is what I’ve got in mind.

A Cron job kicks off a Ruby script, which grabs the current FTSE (or whatever) price and feeds that to Processing sketch which runs an Arduino controller.  The controller is hooked up to a three-way valve – one outlet is attached to a compressed air cylinder, while the other is vented to the atmosphere.  On the third outlet there’s a party balloon – ideally one of the long, thin, sausagey types.

As the market goes up, the compressed air valve opens to inflate the balloon, and as the market drops the vent opens to deflate the balloon.  A quick glance at the balloon gives you an instant feel for the state of the market – flacid, and it’s all doom and gloom; priapic, and the IPOs are going well.

The other advantage is that it would be easy to detect a stock market bubble because the balloon would either burst or pop off the valve and fly across the room.  Although I’ve probably chosen the wrong time in the economic cycle for this to happen very often.

The cron and the Ruby and the Arduino and the Processing bit are all reasonably straight-forward, but I’m struggling to find suitable air valves.  I was wondering about using fuel injectors from a car engine, because they’re controlled by a pulse-width modulated signal, but I’m not sure whether that would work or not.   Anyone else got any ideas?

[CC-licensed photo by Jakob E]

One of my usual London commuting routes has me walking along the riverside past City Hall in the morning, and back the same way in the evening.  It’s a fairly uninspiring inland view, because it’s home to a succession of utterly forgettable glass boxes housing the likes of Ernst & Young and Norton Rose – which does at least have the benefit of making me thankful that I work in an environment where people actually do wear space cadet uniforms to meetings.

The river itself, though, more than makes up for the corporate drone hutches.   From this side, you can see the Tower of London and Tower Bridge, and just along from HMS Belfast there’s moorings for the MV Tidy Thames, which shuttles up and down the river as a floating dustcart removing the odd dead whale now and again.   This, and all the other routine traffic like the police launches and the ferries make the River Thames seem like a real, live entity that is a pale shadow of its former self – but still alive if often overlooked.

One of the things that intrigues me about the Thames is that it’s tidal, with about 6 to 7 metres of rise and fall between high and low tides.  You don’t think of rivers as being this dynamic, and it seems to change the personality of the Thames every six hours or so.   At low tide, the mud and the shingle is exposed, and it looks like a sluggish stream – but six hours later, it’s less than a metre from the top of the embankments and there are ominous whirlpools that suggest that falling in would be a Very Bad Idea indeed.

Although I walk along it most mornings, once I’m in work I can’t see the river unless I hang out of a window and annoy the bosses – so the connection gets lost.   My connection to the web is pretty much constant during working hours on the other hand, which gave me the idea of making the river twitter.  After all, if Tower Bridge can twitter when it’s going up and down, then why not the river itself?

The Port of London Authority publish the tide tables online, but in a format that’s pretty much useless (or far, far too complicated) to grab and parse into a Twitter feed.  The BBC, on the other hand, have a nicely-formatted version which lends itself rather well to being scraped.  So every hour my server grabs the page and parses it to check if there’s a high or low tide predicted within the next 60 minutes, then tweets the time and the depth if there is.  It’s fairly simple stuff with a cron job, the Hpricot parser and some Ruby glue.

Taking a cue from Tower Bridge, I think it’s important that it’s done in the first person – after all, the river’s known as Old Father Thames, and it’s easy to anthropomorphise.   I also tweaked the wording of the tweets so that if the tide is due within the next 10 minutes, the river urges you to hurry up so as not to miss it.

It’s a very trivial example of making real world objects interactive, but I think it’s indicative of something quite interesting – something that Russell “Interesting” Davies has explained far better than I can.   Buildings and other inanimate objects *do* get human traits associated with them, and I’ve always been intrigued by the concept of the company – something that doesn’t physically exist, but yet has tangible manifestations.  One of my absolute favourites is Laphroaig whisky, who allow you to “claim” a 1 square foot patch of land alongside the distillery.  Laughable if you think about it too hard, but a great way of establishing some kind of deeper “bond” with your customers (or drinkers).

And the possibilities are quite literally out of this world, at least if Lowflyingrocks has anything to do with it.

One of the aspects of Twitter I like the most is the fact that it has an API. That makes it hackable in a way that appeals to my inner geek, so something I’ve been playing with for a while is creating Twitter bots. One of the first I made was @shippingcast, which takes the UK Shipping Forecast and squirts it out onto the web in a Twitter stream.

Unless you’re British, the Shipping Forecast probably doesn’t mean a lot. It’s one of those curiously English anachronisms – four times a day, the BBC broadcasts weather forecasts for offshore shipping areas [RealAudio link] on Radio 4 (the BBC’s main “highbrow” speech station). The waters around the British Isles are divided into areas, and the whole thing follows a strangely poetic format. There’s an entire generation of Brits who have never left dry land, but can recite “Dover, Wight, Portland, Plymouth” as if they know what it means.

Slightly eccentric, syntactically-regular? Must be ripe for a Twitter bot. Here’s how I did it.

First, a quick digression/rant about the source of the data. The Shipping Forecast is published twice a day by the Met Office and then reposted on the BBC’s website. This causes no end of hassle for the aspiring Twitterer, the reason for which usually comes as something of a shock for non-Brits. Despite the fact that it’s the US that is meant to be the land of capitalism and free enterprise, the Americans enjoy a strangely socialist attitude to data and information that the taxpayer has funded and provide it free for all-comers. We Brits, on the other hand, have been lumbered with what are known as “trading funds” – quasi-public organisations like the Met Office instructed by government to behave like private enterprises. They tend to guard their data and information as if it were a scarce, rivalrous monopoly good, and will seek every opportunity to charge for it. Never mind that there are far greater benefits to society and the economy as a whole if the data were available for reuse; and never mind that I as a taxpayer have already funded the production of it. No, these organisations have all the safety and security of being public sector, but like to play (and pay) as if they were shareholder-funded. The upshot of this is that – technically – I’m breaking the law by Twittering the Shipping Forecast.

So, back to the “how”. The Met Office have made it fairly difficult to grab the data from their site, although I’m in two minds as to whether that’s because of their aforementioned grasping nature, or the fact that they’re yet another public sector Microsoft monoculture who knows little and cares less about web standards and playing nicely online. This means I have to grab the data from the BBC site, and they format it for viewing rather than parsing. Every six hours, a cron job kicks off which runs a Ruby script to grab the HTML page. This then gets squirted through Hpricot, a Ruby-based XML parser that discards the extraneous stuff and extracts the data that I’m after. The BBC use tables to display the forecast, which makes it relatively easy to grab the “meat” and drop the “sugar”.

Once I’ve got the data itself, it needs formatting. This involves iterating across the table to grab the forecast data for each area (or group of areas), and then stripping out extraneous content to squeeze it into 140 characters. Then finally it gets Tweeted using the Twitter4R library to talk directly to the Twitter API.

So, how well does it work, and what’s worth learning from all of this? Firstly, as a proof of concept for publishing microcontent, I think it works quite well. The process is quite simple, and the information pops up into a Twitter feed on a nicely regular basis. There’s a useful side effect of the cron job, as well, which is to act as a “heartbeat” for the server that it’s running on.

As far as the information being useful, though, it’s less successful. The Shipping Forecast is optimised for broadcast – it’s written to be read aloud rather than read from the page. This means that although there’s a standardised syntax, the format is less standardised – if several sea areas are going to experience the same weather, then they’ll be lumped together. Which is incidentally why phrases like “Dover, Wight, Portland, Plymouth” have such resonance to a Brit of a certain age. While it’s undeniably poetic, it’s a bugger to squeeze into 140 characters.

By contrast, aviation forecasts are very much more standardised, particularly automated services like METAR. “EGLL 130750Z 21006KT 7000 FEW011 BKN014 07/06 Q1009″ manages to convey “13th January at 0750 UTC – wind speed 6 knots at a direction of 210 degrees, 7km visibility, a few clouds at 11,000 feet and broken cloud at 14,000 feed, air pressure at ground level 1009mb, temperature 7 degrees Celsius, dewpoint 6 degrees Celcius” in 50 characters. And once you know what it means, it’s actually no less readable. And writing Twitter bots like @EGLL_METAR is comparatively simple.

Then there’s the sheer quantity of information to convey. Each area forecast covers wind, sea state, weather and visibility information – and uses descriptive terms rather than wholly-numeric values. While this makes it a lot more readable – “squally showers” is somewhat more descriptive than having to look up “#437″ or something in a table – it doesn’t make the Twitter bot’s life any easier. So in order to compress it all in, I’ve had to convert the forecasts into something rather less verbose than txt-speak – which somewhat defeats the object.

The key for decoding the content is here.

Another limitation is that the bot squirts out the whole forecast for all areas, rather than individual ones. This does tend to flood your timeline, particularly if you’re using a mobile device that will only display a limited number of Tweets per page. It’s mainly my fault, because I set up the @shippingcast account rather than a number of smaller areas. But then @dogger or @germanbight don’t have quite the same ring to them as @shippingcast. What I should do, if I ever get around to it, is to set up the bot so that you can “subscribe” to specific areas and then be DM’d with the relevant area forecasts.

Of course, this was mainly done for my own entertainment and education, so I’m not too bothered by the limitations. What’s more interesting is the way that an online service with an API can get exploited for purposes that would never have been envisaged at the outset – I’m pretty sure that the creators of Twitter wouldn’t have thought of the Shipping Forecast as a use case. It’s also been the starting point for some more “successful” bots (for a given definition of success) – my favourite is @riverthames, which tweets high and low tides to keep @towerbridge company. And there are many more “things” waiting to be animated – @thepips and @bigbenclock for a few.