Generative visuals @ HOME

Back in November last year Nick and I took part in the Manchester Hackathon, creating a rapid prototype visualisation piece using some of the data provided by Cornerhouse. A while later, Cornerhouse invited us (‘us’ being myself, Nick, and Amanda) in to discuss more ideas, and generally hear our thoughts on what can be done with their open data, and generally new uses for creative coding as a whole. While this was happening, they also mentioned they were looking for something interesting to feature on the new HOME website, based on a range of related data and visuals. No real firm brief, and not a lot of time: just how we like it! Of course we jumped at the chance.

The data was comprised of architectural plans for the new HOME building, structural specification documents, lots of written research feedback material, videos of projected footfall, branding guidelines, and CGI imagery. Our initial plan was to do something as data-driven as the hackathon project, but it quickly grew into a much more visual idea.

Our initial concept was to try and visually represent the ‘anatomy’ of HOME; the stuff that makes it tick, represented graphically. This took the form of an exploration into generating new physical forms from the data, comprised mainly of typographical elements and components from the architectural building plans. This generative output could then be framed with more traditional typographic or visual elements to create finished pieces.

We started by manually creating a few test images, using randomised elements from the plans to form some rough typographic experiments, just to see how things looked:

The next stage was to put together a quick Flash prototype that generated these visuals for us, with configurable parameters. The code simply took a starting shape (a letterform for example), and iterated over shape placement until it had filled the space as much as possible. Below are a few sample images (ignore the item repetition, this was mainly a proof of concept):

At this stage I wasn’t super happy with the final imagery: it was interesting, but not as visually impactful as I was hoping for. However, I realised that actually the part I was really happy about wasn’t the final product at all: it was the process. The code is generating these images, not me, and I quickly realised that actually without showing the process to the viewer they were missing the point entirely.

So, we evolved the idea slightly, focusing much more on the generative aspects. We loved the typography and images, and wanted to create the same interesting forms, but we wanted to do it while still showing the generative steps to the viewer. Inspired by the work of Roger Alsing, I started experimenting with my own interpretation of genetic algorithms to create more visuals. 

The idea is pretty simple:

1. Give the program a target image (i.e. the one we’re trying to create)
2. Create a blank ‘working’ image
3. Draw a completely random shape onto the current working image
4. Decide if the new working image is closer to the target than it was before we added the shape
5. If it’s closer, save the image and go to step 3
6. If not closer, discard the new shape and go to step 3

What this gave us was a program that over time would generate images that got closer and closer to the target image. There was no limit to how many times the program would run, it would just get closer and closer until I decided to stop it (or my laptop exploded). More importantly, this also allowed us to capture snapshots of the working image at regular intervals, allowing us to not just have a final image, but also an idea of steps that the program took to get there.

We then produced a series of experimental letterforms, and spent a while quickly running them through the algorithm's first few thousand cycles to find out which were most visually interesting when processed:

As we expected, the most effective ones had lots of color and contrast, so we went in that direction and produced four distinct typographical forms built on HOME imagery:

Once we had our final imagery, we needed to run them fully through the genetic algorithm code. First, we took our base image, which is then used as the target for the generative process to aim for:

The code was then left running until it had reached an interesting approximation of the image, capturing ‘in progress’ images at user-defined intervals. The final generated image in this instance was comprised of about sixty random overlapping shapes, had made around 40,000 cycles of generation, and took about ten minutes or so to create:

From here, we were able to produce ‘filmstrips’ of the process images for each letterform, both in a web-friendly resolution and also in ridiculously high res (this would allow them to be printed, or even just downloaded for a closer look):

We also put together a video showing the initial typographical images, along with animated versions of the letterforms quickly constructing and deconstructing. This really gives you a nice sense of what the program is doing in terms of it’s process, you can really see the shape emerging from what looks like randomness. 

(big thanks to David Fenn for providing the music, check his work out here)

A note on the technical implementation: Roger Alsing’s experiments and open-sourced code formed the basic idea, which I then simplified greatly and re-built using C++ via OpenFL. We'll be open-sourcing the whole thing shortly so others can build on it too. 

Manchester Hackathon 2012

Last Saturday saw the second of the three back-to-back jams I took part in, the Manchester Hackathon. Organised by FuturEverything and Open Data Manchester, the hackathon was a wide-open contest to design and build content based on Manchester's ever-increasing wealth of open data.

The hackathon was over one day at the ever-awesome Madlab, who I know pretty well through my work on the Manchester game jam. I've definitely never seen it anywhere near that packed-out though, the turnout was fantastic. I teamed up with digital whizz Nick Donovan, and together we managed to turn up a bit late and hideously under-prepared. Great start!

When the contest was first announced, the main source of data was DataGM.org.uk, which is already a pretty comprehensive (and constantly growing) site that covers everything from census data through to crime statistics, or the density of trees and lampposts in Manchester. I'd had a few ideas in advance based on checking out these datasets, and thought I'd end up going with one of them on the day. However in a late turn of events, a ton more data sources were added at the last minute, on the Manchester DDA site. This new data included some that the Cornerhouse had decided to release, which immediately piqued our interest, so we built from there.

The Cornerhouse data took the form of a huge movie screening dataset dating back to 1999, a well-organised and properly tagged Flickr stream, and an outrageously comprehensive record of books that the Cornerhouse has sold (and by 'comprehensive', I mean down to the millimeter measurements and weight of each book, of which there were thousands).

Taken as a whole, the data represented a really nice cross-section of what the Cornerhouse is all about. The Flickr photos were particularly interesting to us, especially the Exhibition photosets that had obviously been really well-chosen and were all really well shot (and crucially, well-categorised and labelled). Each photoset had one or two images that really drew the eye, even with knowledge of the exhibit context, and we quickly decided that this visual 'flicking through' exploration was one of the really exciting concepts that we could utilise. The breadth of visually-oriented mixed media is one of the big underpinning elements of what the Cornerhouse do, so our ideas took us down this route pretty quickly.

The idea we constructed was quite simple in the end: a visual tactile way to browse the Cornerhouse's past and present media, presented in a manner that was purely visual primarily, allowing the user to dig deeper into individual elements that catch their attention, be that image, quote, book or movie. This seemingly random approach to exploring media follows the same reasoning that makes browsing a bookshop or a gallery interesting, which we felt worked well with the Cornerhouse's ethos. In fact, during development we continually ran across interesting parts that one or both of us commented on and felt like exploring in more detail.

We went for a two-part approach: the first was an interactive Flash prototype that queried the Cornerhouse's photo stream, and pulled out images specifically from the Exhibitions sets. We felt these images represented a strong and well-chosen set of visuals that weren't made up of random groups of people etc. The second part was to visually mock up how we'd like to progress the idea further, with quotes and imagery from the movies and books that the Cornerhouse had curated in the past. Fake it 'til you make it!

The actual project itself takes the form of a virtual table surface literally covered in this mixed media, allowing the user to physically move content around and explore it. The user can pick up and move anything, dismissing content they're not into by simply pushing it off the screen. Users can also expand and explore content they find engaging, to read more concrete information on it. Users can also mark pieces as 'liked', allowing the app the continually re-assess their preferences and serve them increasingly tailored media.

We're really happy with what we managed to get nailed down, and I think our decision to go with something a little more visual and tactile was the right one for us: there were a lot of amazing projects for example that were more geographical representations or raw data sets visualised, whereas our was a little more visuals-first.

We had a couple of Cornerhouse folks in attendance too, they seemed quite positive about our idea, but by the point we were both pretty frazzled so I'm not sure what kind of impression we made... I also think that two minutes isn't *quite* enough to explain and demo things properly, it was really hard making sure we'd covered everything and the time pressure threw everyone off a bit I think.

The overall and well-deserved winner was the Data Crossfader, which turned out absolutely brilliantly. It was one of the stand-out projects where even in a few hours of development they'd produced a piece that already allowing users to spot patterns in data, which I think is an astounding achievement in the timeframe we had. They won a giant pile of money to develop it into a real thing, and I really do think it's going to turn out to b a fantastic piece of work.

Honourable mention also goes to John Rees, who decided that winning the developer prize wasn't enough (we both I voted for him), and narrowly beat us the visualisation prize too. Jammy git. John's app was a geo-locating toilet finder that not only looked brilliant, but more importantly had the most pun-tastic title ever: SatLav. He even did a making-of timelapse which turned out great too.

Huge thanks to the organisers and hosts, and Nick and I would particularly like to thank the Cornerhouse for giving us the opportunity to build something a little different. We had a blast.

(the event also made the Manchester Evening News and the Guardian, the latter of which I managed to sneak my face into. Neat!)