adoption curve dot net

I spent yesterday in Liverpool at Howduino – billed as a “one day event about connecting the internet to the real world, breathing life into inanimate objects and creating new ways to interact with things”, it took place at FACT (somewhere that’s worthy of a blog post in itself).

Imagine a room full of geeks, artists and hackers who’d been let loose with soldering irons and toys, and then throw in the capabilities of Arduino boards. The end results ranged from painting with bristlebots through to circuit-bent Furbies.

I took my panda along, together with a box of analogue meters. Ebay is a fantastic source of that kind of retro kit, and I’ve picked up several moving-needle meters from the late 50s for less than a tenner each. The main plan, though, was to fit an old aircraft fuel-flow meter into the panda so he could display trends and changes in an easier-to-visualise way than the LEDs.

The meter itself is just a basic voltmeter with a dial calibrated from 0 to 800 lbs per hour – presumably it came out of some kind of jet, although I’m not sure how big an engine that would imply. There’s an XLR-style bayonet connection on the back, but I drilled two holes into the back of the case and soldered flyleads onto the internal terminals to make things easier. Then the panda got hacked open, and the meter was hot-glued into the wound.

Actually getting it to work was a bit of a challenge, however. The first problem is that the meter is *incredibly* sensitive – full-scale deflection takes less than 0.5 volts. The Arduino’s analogue outputs range from 0 to 5V, so I had to connect the Arduino output to a voltage divider (having first tried to figure out the right resistor values with a rotary potentiometer.) The PWM output from the Arduino runs from 0 (off) to 1023 (full on) with the duty cycle ranging roughly linearly between the two, so it’s just a case of adjusting the deflection to full-scale at 1023, and the rest falls into place.

The biggest problem, though, was with the Arduino itself. I’ve got a Ethernet shield which fits on top of the main board, and had that linked to an old ADSL router with a 4-port hub. Getting to the outside world took longer than I could be bothered with tinkering, so I ended up running a server locally on my Mac with MAMP to give the Arduino something to talk to.

Getting it talking needs the Ethernet library, and that’s buggy. It would connect fine the first time, but then crashed immediately you tried to establish a second connection. Adrian McEwan (who organised the whole thing) pointed me in the direction of a patched version, but I didn’t R the FM closely enough and only installed the library file. You actually have to remove the (old) compiled version as well – once I’d figured that out, the code started to function as it was supposed to.

The Arduino sketch establishes a connection to a server and grabs an XML file using the Ethernet methods. That then gets parsed using the unfeasibly low-tech string library and finally converts the XML value into a PWM value which controls the duty cycle of the analogue output. Because of the inertia of the needle, the pulses are smoothed out and the display reads steady.

The XML file could come from virtually anywhere – Pachube for example – but I’ve got a few Ruby scripts running on one of my servers that does some manipulation before creating the final values as XML. The challenge now is going to be the data source – and ideally something with a range from 0 to 800, and units of pounds per hour. Any suggestions?

Signing up for Howduino in Liverpool on May 23rd has been a bit of a kick up the proverbial for my otherwise somewhat-moribund Market Bear project.   The panda is now vivisected and has LEDs for eyes, with a rather alarming trail of wires leading from his tail.   Getting the LEDs to sit in place was something of a challenge, but I eventually hit on the idea of threading two legs of the LED through a four-hole button, and using the other two holes of the button to secure it with thread to the inside of the panda’s “eye sockets”.

Codewise, I’d originally planned on putting up a simple Ruby script that spat out a webpage which could be ingested by the Arduino code (via the Ethernet shield).  The problem with that approach is that an HTML page comes with an awful lot of cruft that the Arduino has to dump, which is expensive on memory.   So after a bit of head-scratching, inspiration struck and I’m now generating XML from the Ruby script, which is a) a much cleaner format to have to parse; and b) altogether smaller.

I’ve also used the project as a reason for bootstrapping my Git-fu, so the code is now up on Github for the world to laugh at.   Up until now I’d always thought that the “social coding” strapline of Github was a bit – well, possibly pretentious, and certainly abstract – after all, there aren’t many activities LESS social than coding (unless you’re pair programming.)   But taking the plunge and committing code that other people might see is suprisingly nerve-racking – what if it’s rubbish?  What if people laugh at it?  What if they realise that you’re actually doing more adapting of other people’s code than creation of your own?   All of the above are true in my case, of course.

In any case, the panda continues to be sacrificed in a noble cause, and I’ll be interested to find out what people make of it in a couple of weeks…

My stock market Twitter bots (@ftse100, @nikkei225 and @dowjonesindex) are tweeting away nicely, but they’re all a bit – well, online, really.   I think it would be far cooler if it was more visual and tangible.  This is what I’ve got in mind.

A Cron job kicks off a Ruby script, which grabs the current FTSE (or whatever) price and feeds that to Processing sketch which runs an Arduino controller.  The controller is hooked up to a three-way valve – one outlet is attached to a compressed air cylinder, while the other is vented to the atmosphere.  On the third outlet there’s a party balloon – ideally one of the long, thin, sausagey types.

As the market goes up, the compressed air valve opens to inflate the balloon, and as the market drops the vent opens to deflate the balloon.  A quick glance at the balloon gives you an instant feel for the state of the market – flacid, and it’s all doom and gloom; priapic, and the IPOs are going well.

The other advantage is that it would be easy to detect a stock market bubble because the balloon would either burst or pop off the valve and fly across the room.  Although I’ve probably chosen the wrong time in the economic cycle for this to happen very often.

The cron and the Ruby and the Arduino and the Processing bit are all reasonably straight-forward, but I’m struggling to find suitable air valves.  I was wondering about using fuel injectors from a car engine, because they’re controlled by a pulse-width modulated signal, but I’m not sure whether that would work or not.   Anyone else got any ideas?

[CC-licensed photo by Jakob E]

One of my usual London commuting routes has me walking along the riverside past City Hall in the morning, and back the same way in the evening.  It’s a fairly uninspiring inland view, because it’s home to a succession of utterly forgettable glass boxes housing the likes of Ernst & Young and Norton Rose – which does at least have the benefit of making me thankful that I work in an environment where people actually do wear space cadet uniforms to meetings.

The river itself, though, more than makes up for the corporate drone hutches.   From this side, you can see the Tower of London and Tower Bridge, and just along from HMS Belfast there’s moorings for the MV Tidy Thames, which shuttles up and down the river as a floating dustcart removing the odd dead whale now and again.   This, and all the other routine traffic like the police launches and the ferries make the River Thames seem like a real, live entity that is a pale shadow of its former self – but still alive if often overlooked.

One of the things that intrigues me about the Thames is that it’s tidal, with about 6 to 7 metres of rise and fall between high and low tides.  You don’t think of rivers as being this dynamic, and it seems to change the personality of the Thames every six hours or so.   At low tide, the mud and the shingle is exposed, and it looks like a sluggish stream – but six hours later, it’s less than a metre from the top of the embankments and there are ominous whirlpools that suggest that falling in would be a Very Bad Idea indeed.

Although I walk along it most mornings, once I’m in work I can’t see the river unless I hang out of a window and annoy the bosses – so the connection gets lost.   My connection to the web is pretty much constant during working hours on the other hand, which gave me the idea of making the river twitter.  After all, if Tower Bridge can twitter when it’s going up and down, then why not the river itself?

The Port of London Authority publish the tide tables online, but in a format that’s pretty much useless (or far, far too complicated) to grab and parse into a Twitter feed.  The BBC, on the other hand, have a nicely-formatted version which lends itself rather well to being scraped.  So every hour my server grabs the page and parses it to check if there’s a high or low tide predicted within the next 60 minutes, then tweets the time and the depth if there is.  It’s fairly simple stuff with a cron job, the Hpricot parser and some Ruby glue.

Taking a cue from Tower Bridge, I think it’s important that it’s done in the first person – after all, the river’s known as Old Father Thames, and it’s easy to anthropomorphise.   I also tweaked the wording of the tweets so that if the tide is due within the next 10 minutes, the river urges you to hurry up so as not to miss it.

It’s a very trivial example of making real world objects interactive, but I think it’s indicative of something quite interesting – something that Russell “Interesting” Davies has explained far better than I can.   Buildings and other inanimate objects *do* get human traits associated with them, and I’ve always been intrigued by the concept of the company – something that doesn’t physically exist, but yet has tangible manifestations.  One of my absolute favourites is Laphroaig whisky, who allow you to “claim” a 1 square foot patch of land alongside the distillery.  Laughable if you think about it too hard, but a great way of establishing some kind of deeper “bond” with your customers (or drinkers).

And the possibilities are quite literally out of this world, at least if Lowflyingrocks has anything to do with it.

One of the aspects of Twitter I like the most is the fact that it has an API. That makes it hackable in a way that appeals to my inner geek, so something I’ve been playing with for a while is creating Twitter bots. One of the first I made was @shippingcast, which takes the UK Shipping Forecast and squirts it out onto the web in a Twitter stream.

Unless you’re British, the Shipping Forecast probably doesn’t mean a lot. It’s one of those curiously English anachronisms – four times a day, the BBC broadcasts weather forecasts for offshore shipping areas [RealAudio link] on Radio 4 (the BBC’s main “highbrow” speech station). The waters around the British Isles are divided into areas, and the whole thing follows a strangely poetic format. There’s an entire generation of Brits who have never left dry land, but can recite “Dover, Wight, Portland, Plymouth” as if they know what it means.

Slightly eccentric, syntactically-regular? Must be ripe for a Twitter bot. Here’s how I did it.

First, a quick digression/rant about the source of the data. The Shipping Forecast is published twice a day by the Met Office and then reposted on the BBC’s website. This causes no end of hassle for the aspiring Twitterer, the reason for which usually comes as something of a shock for non-Brits. Despite the fact that it’s the US that is meant to be the land of capitalism and free enterprise, the Americans enjoy a strangely socialist attitude to data and information that the taxpayer has funded and provide it free for all-comers. We Brits, on the other hand, have been lumbered with what are known as “trading funds” – quasi-public organisations like the Met Office instructed by government to behave like private enterprises. They tend to guard their data and information as if it were a scarce, rivalrous monopoly good, and will seek every opportunity to charge for it. Never mind that there are far greater benefits to society and the economy as a whole if the data were available for reuse; and never mind that I as a taxpayer have already funded the production of it. No, these organisations have all the safety and security of being public sector, but like to play (and pay) as if they were shareholder-funded. The upshot of this is that – technically – I’m breaking the law by Twittering the Shipping Forecast.

So, back to the “how”. The Met Office have made it fairly difficult to grab the data from their site, although I’m in two minds as to whether that’s because of their aforementioned grasping nature, or the fact that they’re yet another public sector Microsoft monoculture who knows little and cares less about web standards and playing nicely online. This means I have to grab the data from the BBC site, and they format it for viewing rather than parsing. Every six hours, a cron job kicks off which runs a Ruby script to grab the HTML page. This then gets squirted through Hpricot, a Ruby-based XML parser that discards the extraneous stuff and extracts the data that I’m after. The BBC use tables to display the forecast, which makes it relatively easy to grab the “meat” and drop the “sugar”.

Once I’ve got the data itself, it needs formatting. This involves iterating across the table to grab the forecast data for each area (or group of areas), and then stripping out extraneous content to squeeze it into 140 characters. Then finally it gets Tweeted using the Twitter4R library to talk directly to the Twitter API.

So, how well does it work, and what’s worth learning from all of this? Firstly, as a proof of concept for publishing microcontent, I think it works quite well. The process is quite simple, and the information pops up into a Twitter feed on a nicely regular basis. There’s a useful side effect of the cron job, as well, which is to act as a “heartbeat” for the server that it’s running on.

As far as the information being useful, though, it’s less successful. The Shipping Forecast is optimised for broadcast – it’s written to be read aloud rather than read from the page. This means that although there’s a standardised syntax, the format is less standardised – if several sea areas are going to experience the same weather, then they’ll be lumped together. Which is incidentally why phrases like “Dover, Wight, Portland, Plymouth” have such resonance to a Brit of a certain age. While it’s undeniably poetic, it’s a bugger to squeeze into 140 characters.

By contrast, aviation forecasts are very much more standardised, particularly automated services like METAR. “EGLL 130750Z 21006KT 7000 FEW011 BKN014 07/06 Q1009″ manages to convey “13th January at 0750 UTC – wind speed 6 knots at a direction of 210 degrees, 7km visibility, a few clouds at 11,000 feet and broken cloud at 14,000 feed, air pressure at ground level 1009mb, temperature 7 degrees Celsius, dewpoint 6 degrees Celcius” in 50 characters. And once you know what it means, it’s actually no less readable. And writing Twitter bots like @EGLL_METAR is comparatively simple.

Then there’s the sheer quantity of information to convey. Each area forecast covers wind, sea state, weather and visibility information – and uses descriptive terms rather than wholly-numeric values. While this makes it a lot more readable – “squally showers” is somewhat more descriptive than having to look up “#437″ or something in a table – it doesn’t make the Twitter bot’s life any easier. So in order to compress it all in, I’ve had to convert the forecasts into something rather less verbose than txt-speak – which somewhat defeats the object.

The key for decoding the content is here.

Another limitation is that the bot squirts out the whole forecast for all areas, rather than individual ones. This does tend to flood your timeline, particularly if you’re using a mobile device that will only display a limited number of Tweets per page. It’s mainly my fault, because I set up the @shippingcast account rather than a number of smaller areas. But then @dogger or @germanbight don’t have quite the same ring to them as @shippingcast. What I should do, if I ever get around to it, is to set up the bot so that you can “subscribe” to specific areas and then be DM’d with the relevant area forecasts.

Of course, this was mainly done for my own entertainment and education, so I’m not too bothered by the limitations. What’s more interesting is the way that an online service with an API can get exploited for purposes that would never have been envisaged at the outset – I’m pretty sure that the creators of Twitter wouldn’t have thought of the Shipping Forecast as a use case. It’s also been the starting point for some more “successful” bots (for a given definition of success) – my favourite is @riverthames, which tweets high and low tides to keep @towerbridge company. And there are many more “things” waiting to be animated – @thepips and @bigbenclock for a few.