Superpowers

This is me, meeting President Obama at the White House. To be clear, I’m not in the photo; I’m the camera. This is the first ever presidential portrait taken by Google Glass^[1].

My wife and I had been invited to the White House for its 2nd Annual Pride Celebration^[2] with the president and Vice President Biden. At the time, I was wearing Google Glass every day. For more than two years I attached a bright blue computer to my face and pretended it was normal. Imagine the kind of product loyalty it takes to get someone to willingly look like a jackass (or “glasshole,” as people would mutter under their breath as I walked by). In fact, when I received the invitation to a formal ball at the White House, it never occurred to me that I wouldn’t wear my Glass. The reason for my fanaticism wasn’t Google or Glass, however; I had built myself a superpower.

Now, I swear that I asked the Secret Service if I could wear my face-computer with built-in live camera inside the White House. I showed them to the fellow at the door, and he said, “That’s fine.”^[3] So, my ridiculous headgear and I entered the East Wing to mingle with both the current and future presidents^[4].

My wife and I are not antisocial—asocial might be a better description. I was honored to be invited, but the idea of making small talk with a standing-room-only crowd of gay and lesbian political operators has a similar appeal to me as being forced to watch a 24-hour marathon of Love Island^[5]. So once inside, we made a beeline for the banquet. The White House lays out quite a spread, and a big purse full of cream puffs and petit fours means the party doesn’t have to stop just because you leave^[6].

As we surreptitiously lardered hors d'oeuvres and desserts in preparation for a sexy night back at the hotel, a gigantic man with the classic spirally ear wire that says, “I can disappear you” walked up. “Oh shit, we’ve been busted,” was all I could think. I imagined the Secret Service operating like store security at a Walmart, dragging us into some basement office and threatening to call our mothers.

Instead he said, “Is that Google Glass?”

“Ummm…yeah.”

“It takes photos?”

“Yes.”

“And it takes videos?”

Now I was wondering if he just wanted me to hook him up with a pair as a party favor. “Yes?”

The giant who held my entire future in his hands simply responded, “Please don’t livestream the president.”

What an incredibly measured and reasonable response to having a slightly deranged cyborg-wannabe in the same room as the President. I thought of how many times I’ve waited through long TSA lines to board a plane and imagined them simply saying, “Please don’t blow up the plane. Now on your way.”

It turns out all of that gentle reasonableness hid a chaotic reality. When the Secret Service saw my Glass-wearing splendor wandering through the halls, they freaked the fuck out. Years later, I ran into the woman that organized the entire event and was told that, behind the scenes, the Secret Service had a total meltdown when they saw me in the security cameras^[7]. My Terminator impression caused a change in White House policy the very next day, but in the moment, they ended up asking President Obama whether or not to kick me out of the White House. My fashion choices wasted five minutes of the most powerful man in the world’s time. In an admirable show of nerdy curiosity, he decided to come in for a closer look.

After escaping the ignominy of having our purses turned out on the carpet in front of the crowd, my wife and I drift through the party until stumbling across the famous portrait of JFK. I think, “How better to honor the gravitas of the greatest presidential portrait^[8] than with a selfie!” (I have no idea what we were thinking, beyond proving it was all real). As we line up the shot, the young officer in the elaborate finery pictured above walks up and says rather mysteriously, “Take one step back off the carpet and stay right here.”

We look up from selfie-induced myopia^[9] to see the Secret Service pushing nearly everyone else back, forming the mob of fans and groupies in the background of my masterwork above. At this point, it’s obvious that only one thing could happen next, and so my wife begins hurriedly trying to figure out how to flip her phone’s camera out of selfie mode.

President Obama pops out of a tiny concealed door, immediately adjacent to JFK’s portrait, to stand directly in front of us. My wife’s attention is completely invested in rescuing her phone from selfie mode—she doesn’t realize that the President of the United States is standing directly in front of us. I flail at her to put the phone away, but she doesn’t see the slow shake of his head that says to me, “Jesus, Lady, this is how you dress at my party.” It’s essentially the same head shake I give my 8-year-old daughter when I discover she’s been drawing permanent marker tattoos on herself. I grin like an idiot.

My wife is still engrossed with her phone’s impenetrable UI, and so the president moves on to shake other hands. I whisper-shout at her, “Put it away. Put it away!” And finally, we’re ready. And he returned, now with a big, wily smile. To prove my nerdish credentials, my first words to the President of the United States are, “OK Glass. Take a picture.”

This does not improve his opinion of me.

It did, however, produce the image above, confirmed to me by the Glass team as the first and only presidential portrait taken by Glass^[10].

It also allowed me to tell President Obama why I had a fluorescent blue computer on my face. While I was grateful to be there^[11], I was willing to look foolish because I’d built a superpower.

First, let’s imagine the superpowers I could have built into Glass. Given the research I’ve done on facial analysis for real-time lie detection, my word of honor to the Secret Service notwithstanding, I could have built a face and expression recognition system using Glass and scanned the room in an instant to flag false smiles^[12].

I could pull up everyone’s LinkedIn profiles or their credit scores…or their Grindr accounts^[13]. The scene could someday play out like an episode of Black Mirror with Glass cuing my actions to exploit the emotional weaknesses of others^[14]. In short, having Glass understand facial emotions could lead to a great many questionable and downright terrifying applications.

I built a program for autistic kids instead.

Imagine if everyone around you spoke a language you didn’t understand. It’s not a foreign language; it’s been spoken around you since the day you were born. But where everyone else seems to understand it immediately, for you it means nothing. People use it constantly and become frustrated when you don’t understand. This is the language of facial expression for many people with autism. For them, reading facial expressions doesn’t come for free as it does for the rest of us. It’s like learning a foreign language. For them, the ability to read facial expressions is quite literally a superpower.

Powered by my work at UCSD’s Machine Perception Lab, I decided to use Google Glass to implement a significantly enhanced version of the expression recognition algorithm I had implemented years before. The system combined two models: one for learning about faces and another for learning about how faces are perceived^[15]. It processed faces that were streamed through Glass’s camera. The network extracted subtle combinations of muscle movements called facial action codes (still the standard at the time). From patterns of those codes, the system inferred the subject’s expression. Then, the emotion was displayed on the little heads-up screen as text or simply as an emoji or spoken into the wearer’s earpiece.

The system allowed autistic children to learn how to read facial expressions in natural interactions with other people. The standard for teaching this today is still flashcards printed with cartoon faces, but our brains process social, contextual cues differently than static cardstock. Tapping into the multimodal richness of real-time social interaction enhances learning for both neurotypical and some autistic individuals. And even if I’m not getting it^[16], the system can still tell me you’re pissed off, playing the role of augmented intelligence advisor while learning progresses^[17].

While I hoped that a Glass-based expression learning system might open the secret language of facial expressions to more kids, it turned out to do one better: It improved their theory of mind, their empathy. Much like deficits in expression recognition, impairment in empathy were often seen as common and independent symptoms of Autism. The simultaneous improvement in both expression recognition and perspective taking in kids using Glass-based learning suggested an alternative. If the common language of emotion—expressions, prosody, posture—is hidden, then some are the opportunities to learn why others’ moods change. If “happiness” is just a smiling cartoon face on a flashcard, then you lose the opportunity to learn how a kindness might create happiness in others.

Unfortunately, Google would not allow developers to release any sort of face processing apps for Glass; they were (understandably) concerned that it would scare potential users. Throughout this writing, I talk about the trade-offs between the costs and benefits of artificial intelligence. I genuinely understand the fear of deploying these technologies, but if the fear of abuse means a blanket “no” on all face recognition systems like mine, we cannot attempt to help kids with Autism either. Policy must always be about maximizing the benefits while learning how to mitigate the costs; when we understand that tradeoff, then we can make rational decisions. With Glass, there were so many amazing things that could have been done if Google had focused on superpowers instead of “lightweight social fun.” We need to make these hard choices, not pretend they don’t exist.

Taking a picture of the president with my voice is a pretty mediocre superpower^[18] compared with opening up a world of emotional expression to everyone.

By the way, this wasn’t even the superpower that had me wearing Glass at the White House. My social skills may be dreadful, but I wasn’t tracking the president’s emotions. I built a superpower to track my son’s diabetes. And suddenly, I had the chance to share that story with the president…