Visualising Emotion: Using A.I. to Paint with Your Mind

Using A.I. to Paint with Our Minds

The Genesis of an Idea

What ignites an idea?

Is our stream of consciousness something that flows without direction, or can it be steered and guided?

Can thoughts be more than fleeting moments in our minds?

What if we could digitally capture thoughts and re-interpret them through an algorithm?

Can we convert thoughts into structured data… and data into art?

These were the types of questions bouncing around in our minds when our team @ Wunderman Thompson joined up with Murdoch University to create a piece of digital art to be displayed at the 2023 grand opening of their new Boola Katitjin building (which is a thing of freaking beauty btw).

These questions were huge, and we had no idea if we could actually answer them.

Our aim was to create something more than just a pretty picture on the wall, but rather, something that morphed and changed by allowing the student body to input into it and collaborate. We wanted to make a collective canvas.

It was such a cool brief… we loved it from the start. Like for real, how cool is that!? A collective interactive artwork installation!? WHAT!? How often do those projects come along? Very rarely. And I am so grateful we had the opportunity to be a part of it.

Needless to say, we immediately went to work.

We started by asking ourselves all the big questions above, plus more like:

How could we use tech? Can we find an idea that takes art “up a notch”, beyond just colour and form? And tech can be so cold, is it even possible to use tech to make art that speaks to the human soul?

And all the chats that ensued from those questions were so much fun. We were locked in, focused in flow and trying to find ideas that could transcend conventional digital art forms.

We wanted to find a better way, a way to elevate the medium. We wanted to add new layers of meaning and communication. We wanted to make a mark. It was awesome.

The hunt for “the big idea” was on.

When Sparks Fly

One day, while in the early stages of ideation, I was staring at a blank canvas, and a peculiar thought crossed my mind: What if a painting could communicate with the person looking at it, and not in a static way…but an interactive one?

What if my canvas could read my mind and visually play my feelings back to me? What if it could come alive and mirror my inner emotions, or shit… even understand them? Is that even possible?

And even further, what if two people were looking at it? Could all of out thoughts mix together on the canvas? What would that look like?

At first I dismissed the thought as crazy and put it in the “too hard” bucket, but for some strange reason at the next team catch-up, I brought up the silly idea thinking we’d all surely laugh it off…but the opposite happened. Sparks started to fly.

Teaming Up with IBM Watson

It just so happened a couple days later, one of my colleagues shared a cool data project she was geeking out on that explored using IBM Watson’s Natural Language Understanding to process and understand the emotional effects of COVID on the public.

Yeah… exactly… that’s how my friends roll. 😜

The project is called Yesterday, Today and Tomorrow. In my book, it’s the definition of cool. It works by analysing public tweets (or x’s?) and extracting the author’s emotional context from them and then maps those extracts to a data viz.

But let’s back up, how does the NLP work? Let’s pretend I tweeted something like:

“Holy smokes, I love my job 🤓”

If I ran that thru IBM Watson’s NLP, I’d get some data back that looks like this:

Based on that text, here’s how Travis feels:

Happy: 75.99%
Angry: 0.00%
Joy: 20.00%
Sad: 0.00%
Fear: 0.01%

And boom, that’s when it hit us… this was exactly what we needed to make the painting. This was the raw emotional data we were looking for. But instead of making an info graphic out of it like YTT did, we could paint with it. 🤯

We found our missing link: IBM Watson.

For us, this was the big breakthrough we were looking for. We realized, we could now make something that wasn’t just about robots and algorithms. We could transcend the ones and zeros to make something truly emotive. We could make art.

Our idea was starting to take shape and it felt like we had found a new form of dialogue between human emotion and technology, like an emotional interface.

Our minds were blown. At least for a little while…before the next wave of questions came pouring in.

What Colour is Happy?

The visualization “Colours in Culture” by David McCandless and AlwaysWithHonor.com summarizes the meaning of colors across the world in a visually appealing way.

So yeah, we had the tech and we had the idea… but there was still a few problems. How were we going to collect people’s thoughts? And assuming we had them, how would we visualise them?

For example, if someone communicated they were happy, what colour would we put on the painting? And where? And what shape would it be? Or how about sad? Or angry? What does curious look like?

Luckily, the colour question was easy. It has been researched a lot. Like here. Turns out emotional colour theory is a pretty hot topic, who knew? Who does this stuff man!?

And as for collecting people’s thoughts, we initially wanted to create a passive way for people to send their thoughts to the painting, like thru computer vision, but due to privacy concerns at the university, we had to pivot to using a simple text interface for the input of thoughts.

Mind Over Matter — Meet the prototype.

Now for the fun part: turning the idea into an actual thing. Here’s a video my partner, Annika, lovingly recorded of me using the first prototype (version 0.1).

And yes, when you watch it, it looks like a bunch of squiggly worms crawling all over the TV. At this stage of the POC, I hadn’t done any serious work linking emotions to the right colours, but the most important part, is that you can see that it worked!

As I’m sending messages to the TV, new “worms” appear in different colours. Each “worm” represents a different thought. The yellow one was a happy worm, the red one was an agry one, the blue one a sad one...etc.

And it was at this moment when I felt a great sense of pride. The idea was actually working!!!

From here, the project went thru a bunch more rounds of design and perfecting… but I’ll never forget that first time when I saw the TV painting my thoughts.

Final Thoughts

Looking back now, I realized this project was less about creating a piece of art to hang on a wall and more about experimenting with what art could be in our increasingly interconnected world. It was both super techy, and yet, still raw and unfiltered.

Yes, there were lines of code and algorithms involved, but at its core, this was way more than that; it was an exploration of human connection. It was real, tangible, and offered a fresh perspective on how we communicate and express ourselves.

In the end, we didn’t just create a painting; we opened up a new way to experience art and emotions, unbounded by traditional methods and mediums. It’s not about grand statements or poetic reflections but a simple, honest connection between humans and technology, bridged by colors and crawly worms.

And lastly, I’ll leave you with one more video of the prototype. In this version, I experimented more with the colours while Annika and I threw loads of messages of hope and positivity into the thing.

For context, it was a beautiful sunny summer day in Perth. Our windows were open. The breeze was flowing. We were jammin’ Bob Marley & the Wailers on the radio and our three little girls were playing happily in yard.

Here’s what that looked like:

One More Thing

Oh wait, one more thing… I WANT TO SAY THIS AGAIN… this was just the silly prototype. There’s a waaaaay better & cooler version currently on display at the Murdoch Boola Katitjin building on campus. Definitely worth checking out. This little prototype went thru a few more rounds of visualisation and tweaking before we were done. We ended up going with a 3D design involving rings. The final result is a thing of beauty. I don’t have a video of that final version yet, but I’ll get it.

And on that, I wanna give a big shout out to my friends & co-workers at Wunderman Thompson / Murdoch for all their hard work in making such a beautiful project come to life. I’d mention my colleagues by name, but privacy being what it is these days… I’ll leave them incognito for now.

Lastly, if this sorta thing floats your boat, and you’re interested in learning more about how we did it… I’m an open book. I’d love to talk about it anytime.

So hit me up. You can find me on LinkedIn, or Instagram.

Happy Trails, ✌️
trav