adoption curve dot net

[Crossposted from the Headshift blog]

The news this morning is full of a report from Morgan Stanley on teenage media habits – written by a 15-year-old intern, it dismisses Twitter and describes online advertising as pointless.

Morgan Stanley seem to be promoting the report heavily, although I’m not entirely clear whether this is from the viewpoint of it being a lucid piece of analysis or more of “look! a little person! how quaint!”

The report has turned up on the Guardian website, and while it might be a fairly impressive piece of work for a 15 year old, most of the conclusions are not overwhelmingly suprising.  Teenagers don’t read newspapers, they’re “very reluctant” to pay for music and they see adverts as annoying distractions.   This is hardly earth-shattering, and wouldn’t have been earth-shattering at any point since the word “teenager” was invented.   It’s a long time since I was that age, but me and my contemporaries were hardly huge newspaper consumers and were the generation that was allegedly killing music with home taping.

The more suprising conclusion was about Twitter.   “Teenagers do not use Twitter,” Robson wrote. “Most have signed up to the service, but then just leave it as they realise that they are not going to update it (mostly because texting Twitter uses up credit, and they would rather text friends with that credit). They realise that no one is viewing their profile, so their tweets are pointless.”

This directly contradicts my experience of teenagers using Twitter.   The teens of my aquaintance are voracious Twitter users – a quick random sample of 5 taken this morning show them having produced over 20,000 tweets between them in a 3-month period, and far from using texts to update the service, they’re using the web on both browsers and phones as well as API-based clients like Tweetdeck.   In fact, their use is more akin to public IM – there’s a huge amount of direct conversation between individuals going on, which suprised me given that the asynchronous nature of Twitter doesn’t lend itself to that kind of usage particularly.   They’re also sophisticated enough to be integrating Twitter into other services such as Facebook and Tumblr – which is where you need to look if you *really* want to see the kind of content-creating behaviour that this demographic gets up to.

Where Morgan Stanley’s “analysis” falls short from my point of view is that they’re taking the experience of one particular individual, and extrapolating from that.  It would be an interesting starting point for future research, but these aren’t evidential findings any more than my anecdote above is.  And in any case, I suspect that a 15-year-old who spends the summer writing reports for Morgan Stanley is far from a representative sample of typical teenagers – so while there’s some interesting anecdotal findings here, I’m not sure it fully-deserves the breathless praise that’s been showered on it.   Full marks to Morgan Stanley’s PR people, though – there’s been acres of free publicity which is ultimately worth far more than the 15 minutes of fame that’s come Matthew Robson’s way.

One of my usual London commuting routes has me walking along the riverside past City Hall in the morning, and back the same way in the evening.  It’s a fairly uninspiring inland view, because it’s home to a succession of utterly forgettable glass boxes housing the likes of Ernst & Young and Norton Rose – which does at least have the benefit of making me thankful that I work in an environment where people actually do wear space cadet uniforms to meetings.

The river itself, though, more than makes up for the corporate drone hutches.   From this side, you can see the Tower of London and Tower Bridge, and just along from HMS Belfast there’s moorings for the MV Tidy Thames, which shuttles up and down the river as a floating dustcart removing the odd dead whale now and again.   This, and all the other routine traffic like the police launches and the ferries make the River Thames seem like a real, live entity that is a pale shadow of its former self – but still alive if often overlooked.

One of the things that intrigues me about the Thames is that it’s tidal, with about 6 to 7 metres of rise and fall between high and low tides.  You don’t think of rivers as being this dynamic, and it seems to change the personality of the Thames every six hours or so.   At low tide, the mud and the shingle is exposed, and it looks like a sluggish stream – but six hours later, it’s less than a metre from the top of the embankments and there are ominous whirlpools that suggest that falling in would be a Very Bad Idea indeed.

Although I walk along it most mornings, once I’m in work I can’t see the river unless I hang out of a window and annoy the bosses – so the connection gets lost.   My connection to the web is pretty much constant during working hours on the other hand, which gave me the idea of making the river twitter.  After all, if Tower Bridge can twitter when it’s going up and down, then why not the river itself?

The Port of London Authority publish the tide tables online, but in a format that’s pretty much useless (or far, far too complicated) to grab and parse into a Twitter feed.  The BBC, on the other hand, have a nicely-formatted version which lends itself rather well to being scraped.  So every hour my server grabs the page and parses it to check if there’s a high or low tide predicted within the next 60 minutes, then tweets the time and the depth if there is.  It’s fairly simple stuff with a cron job, the Hpricot parser and some Ruby glue.

Taking a cue from Tower Bridge, I think it’s important that it’s done in the first person – after all, the river’s known as Old Father Thames, and it’s easy to anthropomorphise.   I also tweaked the wording of the tweets so that if the tide is due within the next 10 minutes, the river urges you to hurry up so as not to miss it.

It’s a very trivial example of making real world objects interactive, but I think it’s indicative of something quite interesting – something that Russell “Interesting” Davies has explained far better than I can.   Buildings and other inanimate objects *do* get human traits associated with them, and I’ve always been intrigued by the concept of the company – something that doesn’t physically exist, but yet has tangible manifestations.  One of my absolute favourites is Laphroaig whisky, who allow you to “claim” a 1 square foot patch of land alongside the distillery.  Laughable if you think about it too hard, but a great way of establishing some kind of deeper “bond” with your customers (or drinkers).

And the possibilities are quite literally out of this world, at least if Lowflyingrocks has anything to do with it.

This is possibly the most pointless Twitter bot ever, but @thepips now Tweets on the hour.

It’s a British thing.

One of the aspects of Twitter I like the most is the fact that it has an API. That makes it hackable in a way that appeals to my inner geek, so something I’ve been playing with for a while is creating Twitter bots. One of the first I made was @shippingcast, which takes the UK Shipping Forecast and squirts it out onto the web in a Twitter stream.

Unless you’re British, the Shipping Forecast probably doesn’t mean a lot. It’s one of those curiously English anachronisms – four times a day, the BBC broadcasts weather forecasts for offshore shipping areas [RealAudio link] on Radio 4 (the BBC’s main “highbrow” speech station). The waters around the British Isles are divided into areas, and the whole thing follows a strangely poetic format. There’s an entire generation of Brits who have never left dry land, but can recite “Dover, Wight, Portland, Plymouth” as if they know what it means.

Slightly eccentric, syntactically-regular? Must be ripe for a Twitter bot. Here’s how I did it.

First, a quick digression/rant about the source of the data. The Shipping Forecast is published twice a day by the Met Office and then reposted on the BBC’s website. This causes no end of hassle for the aspiring Twitterer, the reason for which usually comes as something of a shock for non-Brits. Despite the fact that it’s the US that is meant to be the land of capitalism and free enterprise, the Americans enjoy a strangely socialist attitude to data and information that the taxpayer has funded and provide it free for all-comers. We Brits, on the other hand, have been lumbered with what are known as “trading funds” – quasi-public organisations like the Met Office instructed by government to behave like private enterprises. They tend to guard their data and information as if it were a scarce, rivalrous monopoly good, and will seek every opportunity to charge for it. Never mind that there are far greater benefits to society and the economy as a whole if the data were available for reuse; and never mind that I as a taxpayer have already funded the production of it. No, these organisations have all the safety and security of being public sector, but like to play (and pay) as if they were shareholder-funded. The upshot of this is that – technically – I’m breaking the law by Twittering the Shipping Forecast.

So, back to the “how”. The Met Office have made it fairly difficult to grab the data from their site, although I’m in two minds as to whether that’s because of their aforementioned grasping nature, or the fact that they’re yet another public sector Microsoft monoculture who knows little and cares less about web standards and playing nicely online. This means I have to grab the data from the BBC site, and they format it for viewing rather than parsing. Every six hours, a cron job kicks off which runs a Ruby script to grab the HTML page. This then gets squirted through Hpricot, a Ruby-based XML parser that discards the extraneous stuff and extracts the data that I’m after. The BBC use tables to display the forecast, which makes it relatively easy to grab the “meat” and drop the “sugar”.

Once I’ve got the data itself, it needs formatting. This involves iterating across the table to grab the forecast data for each area (or group of areas), and then stripping out extraneous content to squeeze it into 140 characters. Then finally it gets Tweeted using the Twitter4R library to talk directly to the Twitter API.

So, how well does it work, and what’s worth learning from all of this? Firstly, as a proof of concept for publishing microcontent, I think it works quite well. The process is quite simple, and the information pops up into a Twitter feed on a nicely regular basis. There’s a useful side effect of the cron job, as well, which is to act as a “heartbeat” for the server that it’s running on.

As far as the information being useful, though, it’s less successful. The Shipping Forecast is optimised for broadcast – it’s written to be read aloud rather than read from the page. This means that although there’s a standardised syntax, the format is less standardised – if several sea areas are going to experience the same weather, then they’ll be lumped together. Which is incidentally why phrases like “Dover, Wight, Portland, Plymouth” have such resonance to a Brit of a certain age. While it’s undeniably poetic, it’s a bugger to squeeze into 140 characters.

By contrast, aviation forecasts are very much more standardised, particularly automated services like METAR. “EGLL 130750Z 21006KT 7000 FEW011 BKN014 07/06 Q1009″ manages to convey “13th January at 0750 UTC – wind speed 6 knots at a direction of 210 degrees, 7km visibility, a few clouds at 11,000 feet and broken cloud at 14,000 feed, air pressure at ground level 1009mb, temperature 7 degrees Celsius, dewpoint 6 degrees Celcius” in 50 characters. And once you know what it means, it’s actually no less readable. And writing Twitter bots like @EGLL_METAR is comparatively simple.

Then there’s the sheer quantity of information to convey. Each area forecast covers wind, sea state, weather and visibility information – and uses descriptive terms rather than wholly-numeric values. While this makes it a lot more readable – “squally showers” is somewhat more descriptive than having to look up “#437″ or something in a table – it doesn’t make the Twitter bot’s life any easier. So in order to compress it all in, I’ve had to convert the forecasts into something rather less verbose than txt-speak – which somewhat defeats the object.

The key for decoding the content is here.

Another limitation is that the bot squirts out the whole forecast for all areas, rather than individual ones. This does tend to flood your timeline, particularly if you’re using a mobile device that will only display a limited number of Tweets per page. It’s mainly my fault, because I set up the @shippingcast account rather than a number of smaller areas. But then @dogger or @germanbight don’t have quite the same ring to them as @shippingcast. What I should do, if I ever get around to it, is to set up the bot so that you can “subscribe” to specific areas and then be DM’d with the relevant area forecasts.

Of course, this was mainly done for my own entertainment and education, so I’m not too bothered by the limitations. What’s more interesting is the way that an online service with an API can get exploited for purposes that would never have been envisaged at the outset – I’m pretty sure that the creators of Twitter wouldn’t have thought of the Shipping Forecast as a use case. It’s also been the starting point for some more “successful” bots (for a given definition of success) – my favourite is @riverthames, which tweets high and low tides to keep @towerbridge company. And there are many more “things” waiting to be animated – @thepips and @bigbenclock for a few.