The Beatles, Unanticipated Uses of AR

Issue 11 of my newsletter – subscribe here

Despite growing up near Liverpool, I never visited any Beatles attractions other than popping into the Cavern a couple of times. It’s like living in London and never visiting The Tower: there’s always something more exciting going on.

I broke the habit of a lifetime by finally going to The Beatles Story, a museum on the Albert Dock. It was just about fine given entry cost £17. The whole place felt worn down and cramped, as if it hadn’t been properly refurbished since opening in 1990.

Back then, it would’ve been much more impressive to the hundreds of thousands it attracted every year, but nowadays it comes across as dated and irrelevant, for fans only. Even so, it attracted 266,614 visitors in 2017 – numbers that most attractions would kill for.

The odd thing was just how little Beatles stuff it had; there were a few nice pieces like Lennon’s glasses and piano, but I gather that a lot of impressive objects have been sold off to richer collectors and US institutions.

Not that a Beatles museum needs a lot of objects to be successful; The Beatles Story clearly has the band’s blessing, along with a few band-adjacent voices on the pretty reasonable audio guide. If they found a new space that was quadruple the size and hired the V&A designers behind the visually stunning and immersive Bowie and McQueen exhibitions, they could easily make their money back. Throw in a better restaurant and bar, and they’d print money.

Then again, this is an obvious idea and I can only assume the rights holders just don’t care. Paul McCartney is a billionaire, getting a few million extra a year from a museum would hardly register on his bank balance. Maybe he isn’t interested in that kind of legacy.

I had yet another call this week with a major tech company about augmented reality and gamification. It’s pretty clear that all the big companies have AR glasses coming out in the next 1-3 years and so they’re thinking very hard about possible applications, because, well, they don’t have any.

So here’s a free idea, albeit one that won’t be feasible for a few years: selling used goods.

I’ve started selling a lot of stuff on Facebook Marketplace lately. I used to sell things on eBay but after being burned a couple of times by buyers who claimed their item didn’t arrive, I gave up; but with Facebook Marketplace, you can get people nearby to meet you at a location of your choice. The listing process is also free and extremely fast.

It’s mostly been games I’ve finished, along with tech that I’m replacing or not using – old TVs, Kindles, iPads, that sort of thing. It turns out you can get a really good price for solid brands, and I feel good about this stuff being used rather than gathering dust in a closet. And yeah, the money helps.

I am temperamentally suited to selling my stuff in this way because I like not owning many physical things. Other people are different, and that’s where I think AR could help. Imagine glasses that tracked your activity, and could say, “Hey Adrian, you haven’t used this exercise bike in six months! I can have someone pick it up tomorrow for £180, what do you say?”

Is this creepy? For sure – especially if it was operated by Facebook. But if it was under your control, I think it’d be helpful for a lot of people. Right now, there’s a huge amount of friction that prevents community tool lending libraries from being set up – few people enjoy keeping a database of items up to date – but if it was completely hands-free and your glasses could instantly identify the make and usage levels of a particular item, I think most would be much more willing to opt-in and share or sell their items.

In other words, AR glasses could introduce a true sharing economy – not Uber, not Airbnb, but a true non-profit version – by dramatically reducing the friction involved in data entry and maintenance.

Of course, this is the optimistic view. AR glasses will also be used to sell us all sorts of useless shit in ways we can’t possibly imagine now, like putting virtual products in your wardrobe and virtual burgers on your plate, and saying “this burger could be yours in only ten minutes for just £5!”

Playing

🎮 Valleys Between on iOS. Imagine if Monument Valley and Alto’s Adventure had a baby that wasn’t as fun as it’s parents. It’s a paid game (which I like!) so I wanted it to be good, but it was devoid of challenge, and frankly, of gameplay.

Watching

📺 Stranger Things Season 3 on Netflix. I’m only halfway through but it’s hitting the perfect 80s nostalgia family action adventure spot. This season, I feel like they’re relying on the old Mad Men trick of making fun of olden times a bit more heavily, but there’s nothing wrong with that.

📺 Legion Season 3 is a welcome improvement on the previous season, being a soft reboot and also the final season of the show. I have high hopes.

🎞️ Yesterday. I was hoping it’d be a 7/10, but instead it was a 6/10. The concept remains gold, but the romantic relationship was poorly written and unbelievable. Although now I think HBO should make a prestige series about The Beatles.

🎞️ Spider-Man: Far From Home wasn’t as good as Homecoming, which had Marvel’s best villain after Black Panther, but I enjoyed the lighthearted teen comedy bits. Can’t wait to see what they do with Spidey once they get over their Iron Man fixation.

Reading

📖 Dealers of Lightning by Michael Hiltzik. I knew the legend of Steve Jobs and Bill Gates ‘stealing’ the idea of modern windowed interfaces from Xerox PARC, but this book documents the far more impressive and interesting truth behind Xerox’s research labs. It turns out PARC didn’t just invent the modern windows UI but also Ethernet, laser printers, VLSI, and (sort of) Postscript, SGI, and TCP/IP.

The lessons of PARC may not be widely known in detail but you can be sure they are familiar to Silicon Valley founders from the 90s and 2000s (e.g. Google) because they would have been advised by people who lived through it (e.g. Eric Schmidt, who worked at PARC from 1979-83). As such, you wonder if Google’s penchant for releasing every product they can think of is a response to Xerox’s releasing nothing due to internal politics. Of course, now we can see the problems with Google’s approach…

📖 The Shock of the Old by David Edgerton. Glowingly reviewed, the author is either wilfully obtuse or downright dim – you choose. I’m planning to write a longer review, but this book essentially boils down saying “Haha, technologists haven’t figured out a better way of killing whales than the harpoon – so much for iPhones, right?”

Listening

📻 Something For Your M.I.N.D. by Superorganism. Suitably weird.

The Mapping Problem

Issue 2 of my newsletter – subscribe here

There’s the book I’d like to write. And there’s the book I ought to write.

Last year, someone from Samsung’s strategy team asked to talk to me about whether “gamification” could be a possible use (if not a killer app) for a hypothetical augmented reality heads-up display.

I admitted, yes, Zombies, Run! was an example of gamification in action, although I quickly explained that I detested the term. Gamification “experts” will tell you that merely applying a coating of points, achievements, levels, and progress bars to otherwise-boring activities is not the way to turn it into a fun or engaging game – but promptly ignore their own advice when designing games for big companies that want to improve productivity, games that invariably are only about points andz achievements and leaderboards. So no, I didn’t think that gamification – as commonly understood and practiced – was a killer app for anything.

“But isn’t Zombies, Run! popular? Couldn’t you do that for other things?”

I sighed inwardly, wondering if I would save all the time I spent talking about why gamification sucked by writing an article or book instead. I wasn’t that simple, I explained, because of the Mapping Problem.

The Mapping Problem came to me at a games conference when I was asked if I could make mopping the floor more fun. Now, I don’t mind doing chores, but mopping is hard. It makes a mess, it requires more effort than vacuuming, and you don’t have the satisfaction of seeing bits of dirt disappear. It’s also something you know you ought to do more often, so it’s really a very good candidate for gamification because you’re intrinsically motivated to get it done.

The standard playbook of gamifying mopping would be an app that gave you points for each time you mopped the floor, and to give extra points and badges if you mopped regularly. Maybe you’d even give more points for mopping different parts of the house!

But this is a shit game design. Getting points and badges after your first session of mopping might make you feel slightly better, but after the third or fourth or eighth or ninth time you tap the “I’ve mopped, yo!” button, you realise that it’s meaningless. You’re just doing this for points. Meaningless points. The app doesn’t really care whether whether you did a good job; for all it knows, you’re just jabbing the button without doing any mopping at all.

So here’s what I would do. I would try to make the act of mopping more fun. I don’t want to reward you after you’ve finished mopping, I want you to be excited while you’re mopping. Ideally, I want you to be excited to start mopping because you know it’s going to be fun. This cannot be achieved through points and badges alone. It requires real-time motivation, either through gameplay or storytelling, or both.

I’m not sure storytelling quite fits mopping, so I’ll go with gameplay. And what I’m looking for is a kind of gameplay that maps onto mopping in a natural way, in a way that will help you achieve your ultimate goal: a thoroughly-cleaned floor. This will require the game to ‘know’ the cleanliness of the floor, which is probably most easily done with computer vision. Likewise, we need a feedback mechanism so the game can respond to what you’re doing, ideally using some flavour of augmented reality, as you are manipulating the real world. And this is the point where we get into the limitations and opportunities of technology, in the sense that this game isn’t possible unless a large number of potential players have either:

a) Multiple static home cameras that can see a majority of your floor

b) Body or head-mounted cameras

c) Periodically waving your phone’s camera at the floor No! This is a terrible idea.

Basically we need to wait for an affordable augmented reality heads-up display, which is a few years out. I honestly don’t think it’s possible to make an effective mopping game without this technology, any more than it’d be possible to make Zombies, Run! without GPS-enabled smartphones. Without it, the game has no idea whether you have mopped a particular part of the floor. Maybe you can use accelerometers to sense a back-and-forth motion (yes, haha) but that doesn’t tell you about the position of the mop. So cameras it must be.

It’s unusual for technology to so overtly be the limiting factor in game design. Most designers sensibly, and unconsciously, filter out game designs that are plainly not possible with current technology. Sure, you might have games that push graphical or processing boundaries, but few games push beyond existing interface technologies. Whenever they do – like Nintendo Labo or Guitar Hero or Pokémon Go or Beat Saber – we all get tremendously excited.

I’d argue that any gamification that extends to the physical world (as opposed to entirely digital activities like, say, gamifying online language learning) falls into this category where practically every solution requires some kind of custom interface technology. That’s just one of the reasons why gamification is hard. If you’re creating the next Fortnite or Mario or Grand Theft Auto, you don’t need to invent or harness an entirely new interface; but if you want to make the best game to teach you the violin, you have some very basic interface questions to answer.

(Incidentally, this is why touchscreens are so central to the success of smartphones: the technology is so versatile, it can solve entire swathes of interface problems. The same screen used for a first-person shooter like Fortnite can also be used for a match-three puzzler like Candy Crush. And that’s why effective AR heads-up displays have the potential to solve an order of magnitude more interface problems: you can use the technology for helping people to mop, or translating street signs, or making learning the violin more fun. So the Samsung guy wasn’t wrong, per se.)

OK, enough about technology. What’s the gameplay for our mopping game? Take your pick! Maybe it’s wiping away bugs crawling around the floor; or it’s a colouring game; or a game of Snake; or you’re sculpting a 3D landscape. There are lots of possibilities, the important thing is for the game to organically encourage you to cover the entire floor in a way that isn’t too rote or tiring. The point is, the game doesn’t just reward you for having mopped, rather, the act of mopping is the game.

But here’s the sting in the tail: the Mapping Problem means that even if we’ve solved the Mopping Problem, we haven’t solved the Violin Problem, or the Ironing Problem, or the Bin-Emptying Problem. These are all completely different kinds of activities that need to be made entertaining in different ways. So my objection to “gamification” is not that I hate the idea of making ironing more fun. I would love for such a game to exist. It’s that the term has come to mean the application not merely of a standard process, but a standard set of game mechanics, to wildly different problems – and any so-called “gamification expert” who claims otherwise is being wilfully blind to the reality of the shitty industry they have helped build.

Unsurprisingly, I have a whole lot more to say about gamification. An entire book outline and proposal, in fact. It’s a book I’m uniquely qualified to write, with a background in experimental psychology and neuroscience, plus the badge of creating one of the most successful examples of gamification ever. I even know it would sell decently well, since gamification never seems to die (“What is dead…” indeed).

It’s just… I want to write a novel! Or a short story collection. Something beautiful and creative. But my book outline is for something completely different.

I ought to write about gamification. And yet.

Through an iPhone, Squinting

If you want a vision of the future, imagine your arm holding up your iPhone — forever

It has been truly delightful to see all the imaginative augmented reality prototypes made by developers playing around with Apple’s new ARKit framework. It’s only been available for a couple of months, but developers have already gone to town with all sorts of fun ideas:

Amid the excitement, it’s easy to forget that we’ve been here before, many, many times. Back in 2015, Apple acquired Metaio, a German company that provided an SDK (software development kit) that allowed other developers to easily make augmented reality experiences. Two years on, ARKit is no doubt far more powerful and far easier to use than Metaio’s SDK, but the principle is the same.

And so are many of the applications. We’ve seen 3D objects superimposed on the real world on iOS device before, powered by Metaio:

And we’ve seen people plopping down inexpensive furniture into their homes before:

I don’t mean to rain on ARKit’s parade. The ease of use, lack of licensing fees, and sheer power means that we’ll be seeing a greater variety of ideas than we ever did in Metaio’s time, and so it’s entirely possible that someone will figure out an idea that makes phone-based augmented reality really take off. This ARKit-powered measuring tape prototype is actually very neat and useful:

But I don’t think this time is different.

All of these videos you see are incredibly misleading — not because they’re faked (they aren’t), but because they make it seem as if your field of view will be filled with the digital reality.

We usually don’t have to worry about this for videos taken by smartphones because in those cases, it’s actually true: when you watch a video on Snapchat or Instagram, you’re seeing what you’d really see if you were there. But when you watch an augmented reality video, you’re definitely not: instead, you have to imagine you’re holding up a phone at arm’s length, and seeing the video on that phone.

Looking at the world through a 5″ window is never going to be comfortable for longer than a minute. I’m sure there will be specific, short experiences like the measuring tape app that do well, along with some tourism and instructional apps, but I very much doubt we’ll see experiences even as long as 5 minutes, let alone 15 minutes.

As for games, so many of the prototypes are basically placing 3D objects on top of real world 2D planes, like your coffee table:

Forget about accessibility or comfort. I fail to see how this is more fun than a non-AR game that isn’t tied to a flat plane. It would be more innovative for AR games to involve manipulating of objects in the real world to influence the digital world, but that’s decidedly tricky when you’re holding up a phone or tablet.

Before you say “Pokémon Go”, let’s be clear — literally the first thing any decent player does in that game is turn off the augmented reality layer (where the monsters are superimposed on top of the real world camera view) because it eats up your battery and makes the game harder to play.

I don’t buy it. And I don’t think Apple does, either. Unlike Google’s shameful boosting of the dreadful Google Glass, Apple has thus far been comparatively quiet about ARKit. I’m sure they realise that most people don’t want to develop their shoulder muscles by using AR all the time.

No, this is all preparation for their future heads-up display — one that really will fill up your field of vision, be perfectly comfortable to use, utterly desirable, and only barely affordable.