Welcome to my website!

Manchester Hackathon, Open Data and Data Crossfader

UPDATE: If you’d like to keep updated about the project as it develops, please follow datacrossfader on Twitter.

It’s certainly no secret that I’m a flag waver for Newcastle’s forward thinking Culture Code hack earlier this year, so I was intrigued when I heard about Manchester Hackathon- the first organised opportunity for dataphiles to play with Manchester’s Open Data sets.

I was also keen to get Ashleigh involved. We’ve been playing with ideas for data visualisations on our Nerd Sundays (yes, thanks… I know) over the last few months, and it seemed like an ideal event to apply ourselves to some large, real-world data sets.

Hello Manchester!

The hack was hosted in the MadLab building (Edge St, Northern Quarter). We built stuff between 9am and 5pm and then each of the 16 teams gave two minute presentations before the judging, prizes and socialising. There had also been a get-together the previous evening, but we unfortunately couldn’t attend since we were lorry-dodging southbound on the A1.

The data sets had been released prior to the hack, so we’d discussed a few concept ideas and came armed with a broad plan for what we could build.

Our project was ‘Data Crossfader’.

I shan’t talk about the day – it was mostly heads down / eyes to screen – albeit with a break for a bacon sandwich – so instead, I shall start with my presentation.

Pass Me That Petard Hoist, Please

I felt I did us a big disservice with the pitch- I’m a bit disappointed with myself, because I’ve had this stuff hammered in to me over the years. So, live and learn… (again). I didn’t contextualise with a problem, spent half my time showing the live demo with my back to the room, and certainly left some confused about what we had made. Some of the feedback and Twitter mentions, along with this Manchester Evening News article decided the hack was about road safety or mixing of traffic data. I’m sure our eventual prize placing bewildered some of our fellow developers and, given the presentation, I expect my eyebrows might also be raised too. Thankfully for us, from around 3pm, the judges had wandered round casually chatting with the teams. I suspect we did a *far* better job of conveying the heart of the hack, which is this:

Government is asking their departments to open data. This is a great thing. It’s a recent initiative, and that’s part of the reason why these such hacks are valuable- they’re an opportunity to work out what we can build, what power is in the data, where the foundations could lie, and ultimately (at this stage) feed back to source to improve how and what data is collected. By incremental steps, we can reach a point where those who collect data know what is good to collect, and those who build can hit the ground running. As things mature, we’ll also get to a point where hacks are built on hacks and we’ll start to see a second-order power of data… but I’m getting ahead of myself. Fundamentally, we (the hackers) need also to demonstrate to those who make decisions that opening data can be both holistically fruitful, and sustainable.

This relationship between data providers and builders is in early infancy- as it is for Culture Code. I’m sure many of my fellow hackers would agree that the provided data is patchy or lumpy. This is no slight at all on the data providers. Providing anything an excellent, and essential first step on the journey – and there have been some smart people in the middle, who spent time massaging data to homogenise it for easy use by us (thanks to them too!).

One of the issues that raised a flag for me about Culture Code was that (necessarily) only a small proportion of the data providers had their data applied to a hack. One might hope that those providers would take a look at what was produced, think about how they could collect shinier data to pique the interest of the magpie hackers, and return next year armed with a set that will be built upon. I think it a more likely outcome that the data providers (especially those making financial decisions) would not punt on it twice- at the least, this is a risk.

Give Me Sight Beyond Sight

Stepping from this, we were intrigued by the idea of building something that was applicable to as many data sets as possible. To do so is fairly high-concept, and runs the very real knife-edge of being so general as to be useless.

We noticed that many of the provided data sets contained located data. I believe this predisposes hacks towards being flags on maps – i.e. information visualisation. We do something similar with Data Crossfader as our first step. We assume that most of the locational data sets can be boiled down to a height-contour map. In an example I presented, we transform one of the GM Data sets (a list of around 45k catalogued trees) into a 3D overlay – with areas containing a lot of trees being peaks, and those that are sparse being troughs.

Superficially, that becomes something akin to the Intelligence Hub Analysis Tool, albeit with some 3D sparkle, but missing a lot of the nice data drilling they allow you to do.

To yank this up a level, we wanted to attempt to build an instrument that moved beyond information visualisation to semantic visualisation- that is, visualisation of meaning.

We do this by allowing the user to select two of these height-contour sets of data, and providing them with a slider to crossfade between them. The theory is that when the crossfader lies at 50%, it highlights correlations (e.g. tree density vs respiratory problems) or potential black spots in need of attention (e.g. population of pensionable age vs provision of accessible transport). I’ve wrestled a little with the idea of pulling the crossfader around (as opposed to just giving a 50% mix of two datasets) – it can feel gimmicky rather than insightful… but I’m swaying towards retaining it, since I think it does actually help highlight the dynamics between the two sets.

So, in summary, I gave an example of two complementary data sets (traffic camera positions vs accident frequency), but that’s not really the point of the hack- it’s intended as:

“a strategic tool to compare any two of a large collection of data sets, in a way that is generic but not diluted”.

I think the main drive of this for me is to add something that can accelerate the process of making data collection more appropriate, by giving those who collect it better insight into what their data can mean – chipping away a little towards the power of second-order data.

Opening the Toy Box

From another side of the table, one of the targets of Manchester’s open data is to make information accessible to the public, and one of the key aims of this hack was to explore tools to make that easy- I think this is the side that Ashleigh is more interested in. Imagine a tool that allowed you to explore features across the city – households with children vs leisure activities, council tax band vs public transport provision, crime incidence vs time since last in employment, this vs that… – an exploratory tool to discover how things fit together… and possibly also to help you make decisions such as where you might like to live, or to see how your area compares to others.

The Result

So, the big news is that we scooped the Grand Prize! We were overjoyed, rather obviously :)

We left with:
1) A £1,000 prize.
2) A promise of an additional £3,600 to develop the idea with Manchester City Council.
3) A cute little golden robot (MadLab’s totem, I believe).

The £1,000 prize has already been spent. Last week, Ashleigh’s car gave up the ghost, and she hasn’t got the floating cash to repair it, so it’s sitting off the road. It’s a lifeline for her- and for the cat rescue charity she’s heavily involved with, so – to spur her on for the hack – I made the promise that we’d put any prize money towards getting her car fixed. That accounted for a petrol-fired-moneypit-guzzling 80% of the spot prize. Still, that’s £800-worth of happy kitten faces smiling back at us. It’s karmic. I hope. The remainder will be swallowed in fuel for the journey, a victory curry, some apposite domain names and a long-overdue haircut for me.

The £3,600 is *very* welcome. Some of my time is spent doing freelance work for startups and getting paid, some is spent the incubating ideas I have, which is obviously unpaid. This will give us the chance to spend some proper time on development of our own project without having worries about balancing external work. More than that, I think MCC are keen to offer appropriate other support to get this running- I had a few chats during the day with their Comms representatives, and they seemed genuinely very interested and switched on. So I’m looking forward to this development on an intellectual level too.

And the little golden fella?… Chums already. I clutched him all the way home. I may investigate adding laser diodes behind its eyes. Can he burn a hole in the wall every time I have mail?

(Photo: Ashleigh).

Oh, You…

Platitudes seem appropriate at this point, though I can assure you they’re heart-felt rather than simpering:

  • Future Everything / MadLab / Open Data – I’m not sure where the boundaries between these orgs lay, but the event felt both relaxed and slick, so congratulations to all. MadLab feels like a really progressive place. Thanks Julian, Tom, Stephen and extended company.
  • The Judges – It was a great idea for them to wander round the tables and chat, and also to receive some insights from them for future development.
  • Council Reps – from Manchester, Trafford and Salford (possibly others?). Approachable, interested and available all day. It greatly benefits the cohesion of the communities.
  • Hackers – I think everyone ‘got’ the point of the hack and ran with it. There were some really nice prototypes developed and the atmosphere was friendly. Special mentions to our desk buddy, John Rees who hacked SatLav (which *is* what it sounds like)- a really nice guy and deservedly left with two cash prizes for developing something quirky. I also liked the ‘Cornerhouse Explorer’ – a virtual ‘coffee table’ discovery tool, warm-looking and left-field, and ‘Virtual Lampposts’ – for propagating local news on Twitter (the developer correctly acknowledged that there were holes in the idea, but credit where it’s due for shooting somewhere different… there’s a neat, workable idea in there somehow, I’m sure).
  • Special mention should also go to the under-21s. Some great talent. I heard that one of the entrants was just fifteen years old, and he presented with the best of them.

I was asked if there was anything I would change. On reflection there were just a couple of things:

  1. Schedule – time sheets on the wall suggested we had until 5pm to code, then half an hour to develop a pitch. At about 4:15 we were told that we should finish coding and have our pitch in for 5. That certainly spooked me :)
  2. Demographics – the significant majority of participants were male. Coding certainly is male-dominated, and maybe a competitive hack pulls a certain type- but that’s stacked against a proactive Girl Geeks movement, and an even (if not female skewed) designer demographic – so it’d be great to have a more even balance. Certainly, designers were sought after for Culture Code teams, so there’s definitely an opportunity to get involved.

Further References

UPDATE: If you’d like to keep updated about the project as it develops, please follow datacrossfader on Twitter.


I’d welcome any media interest, either now or as the project develops. Please get in touch and help collect as much momentum for Manchester’s open data as possible!

Categories: Front Page and tags: , , , .
Bookmark the permalink.

6 Responses to Manchester Hackathon, Open Data and Data Crossfader

  1. Mad skillz you both! Catch up soon and will be fascinated by lessons learned :-)

  2. richwestcoast

    Congrats :)

  3. Pingback: Hackathon Extravaganza at MadLab | Manchester Digital Laboratory

  4. Pingback: The Manchester Hackathon not bad for No.1 « Open Data Manchester

  5. Pingback: What the hell is a “data crossfader”? ‹ Betarocket

  6. Pingback: Creative Nucleus - web, logo & brand design - Newcastle upon Tyne

Other Posts