The story of Cortana, Microsoft's Siri killer (2024)

Technically, Cortana isn’t supposed to exist for at least another 500 years, but that’s not stopping Microsoft from bringing her to life this week. While Apple has Siri and Google has Google Now — both digital assistants that run on smartphones — Microsoft is taking an approach that mixes the best of the competition with its own unique take. Based on a 26th-century artificially intelligent character in the Halo video game series, Cortana will debut as part of Windows Phone 8.1, the next big update for Microsoft’s mobile operating system.

By learning your habits and interests continuously, Cortana is positioned as a personal digital assistant that helps you organize your day-to-day activities, alongside regular web searches for information. Cortana will act as the primary way to discover and search for information on Windows Phone 8.1, or just an assistant to manage your meetings, reminders, and daily life. She’s smart and witty, all while being designed to closely resemble a human assistant. With the competition already years ahead, Cortana arrives at a time when Microsoft is focused on catching up in mobile. Cortana is a significant new feature for Windows Phone 8.1, one that has been in development for more than two years. In many ways, Microsoft’s bold new mobile efforts rest on her virtual shoulders. This is the story of Cortana, and how she came to be.

The name started from a simple suggestion from Windows Phone program manager Robert Howard in an early planning meeting. "It was just a codename, it stuck," explains Marcus Ash, group program manager of Windows Phone. "We didn't intend for it to be the actual product name from the beginning." The fact Cortana exists simply as Cortana, and not some marketing buzz like "Microsoft Personal Digital Assistant Home Premium" is surprising given Microsoft’s history of naming products. Up until a few weeks ago, it was hit and miss whether Cortana would be the final name. It could have been Naomi, Alyx, or a number of other suggestions, but leaks and a petition to use the Cortana name helped sway Microsoft’s decision.

The Cortana naming and background is linked directly to Halo, and meshes well with Microsoft’s main goal for the product: recreate a real personal assistant without being too creepy. Cortana was always there for Master Chief in the Halo games, and now she’s always there for you on your phone, but only if you want her to be. Rival services like Google Now dig deep into data from devices, and while that’s often useful it can also be irritating in the form of non-stop notifications, or just scary that the system knows so much about you. To avoid this, Microsoft spoke to a number of high-level personal assistants — yes, actual humans — and found one that kept a notebook with all the key information and interests of the person they had to look after.

Cortana has a notebook, just like a real assistant

That simple idea inspired Microsoft to create a virtual "Notebook" for Cortana which stores personal information and anything that’s approved for Cortana to see and use. It’s not a privacy control panel, per se, but a list of everything Cortana knows about you. "It’s her view of you, but clearly you can just snatch it from her at any time and say ‘That’s not right, I don’t want you to know this’ or ‘I’m not comfortable with you reading my email,’" explains Ash. "So you have complete control over what she knows and she’s transparent about it." Entries in the Notebook are stored in the cloud, and you can share contact information with it, as well as your interests, home and work locations, and more. The notion of Cortana acting as a personal assistant with a notebook— as opposed to a creepy stalker — has been drilled into the team from the beginning, they say. She also operates and functions by learning your habits and interests from your phone use, location, and communications. You can speak to Cortana or just input text, but she’ll always ask you before she stores any information she finds in her Notebook.

When you first launch Cortana, she runs through basic questions to learn about you — your name, your food preferences, what category of movie you like, and so on. After that, when the service is activated with Windows Phone's search button, you can swipe down to see a "proactive view" of information. It’s very similar to Google Now’s cards, with information on flights, sports results, stocks, and anything else Cortana has learned and jotted down in her Notebook. You can improve the Notebook by manually adding your personal interests, reminders, news, and other important data. It’s really a hub of information that turns into cards, and parts of it can be pinned as Live Tiles on the Start Screen or used to generate notifications in Windows Phone 8.1’s new Action Center, a notification hub similar to those found in iOS and Android. If, for example, your favorite football team just scored, Cortana can alert you. If you visit a foreign country you’ll be greeted with weather information, currency conversions, and maps. If you’re in a text or email message, Cortana will underline elements like "let’s meet at 8PM" to make it easy to set reminders or calendar appointments.

One of the most useful features of Cortana is its ability to trigger actions based on events, a little bit like the popular web service IFTTT. For instance, saying "Remind me the next time I call my wife that we need to talk about Kevin" will create a reminder that is triggered when you next go to call your wife or she calls you. It’s powerful, but Cortana even impresses during basic search queries. If you search for "What’s the best restaurant near me" you won’t get a big list of results like you do with Siri, you’ll get a single restaurant that’s rated the best in the area by Yelp users. "If you asked a real assistant, 'What is the best restaurant' and she held a page up to you, you would fire her and try to find another one," jokes Rob Chambers, principal group program manager of Bing. The difference is that if you had said "the best restaurants," plural, then you’d get a list thanks to Cortana’s understanding of the voice queries and their context. The truly impressive moment is when you’re simply able to say "call it" after asking for the best restaurant, or "give me directions" and Cortana understands you mean directions to the restaurant in your previous part of the query. It’s true multistep search, a way to layer query upon query to accomplish complex tasks by voice alone. It feels like the future.

In Windows Phone 8.1, Cortana appears as little more than an animated circle, but that doesn’t mean she doesn’t have a personality. Like any good assistant, and Apple’s Siri, Cortana’s personality shines through in daily use. Ask her, "Who’s your father?" and Cortana will reply, "Technically speaking, that’d be Bill Gates. No big deal." Other queries produce witty responses, and some answers make the circular character spring to life and animate with one of 16 emotions. Cortana won’t always respond with emotion and animations, but Microsoft envisions a future where she reacts visually to sports scores or other events — any good assistant knows to be pumped when a football team wins and furious when they lose, and so does Cortana now. "There's just more stuff we're gonna be able to do with the shape as we progress along this journey," explains Ash.

Microsoft has also worked closely with Halo developer 343 Industries on the eyelike visual elements and voice actress Jen Taylor for the sound of Cortana. Taylor is the voice behind Princess Peach, Toad, and Toadette in various Mario games, but she’s best known for her role as Cortana in the Halo series. For Halo fans — and there are a lot of them — having Jen Taylor as the voice of Cortana in Windows Phone is a big deal, and for Microsoft it’s equally significant. "She's gonna play a pretty big part in how we roll this out and how we evolve this speech technology," explains Ash. Initially, Taylor will be used primarily for what Microsoft calls "chit chat" responses, queries where the company can use original audio. If you ask "‘What’s up with Master Chief,’" or anything related to the Covenant, then you’ll get a Taylor response. Other interactions, meanwhile, use a synthesized voice that’s similar to Taylor’s. (If you want some more Halo-related fun with Cortana, you can just set your nickname as "Master Chief" in the settings.)

Voice actress Jen Taylor is the voice of Cortana

Microsoft didn’t just magically build a digital assistant in two years — the company is leveraging investments in data gathering that it has been making for half a decade. Cortana relies on Bing’s backend services for the majority of its features, and that’s backed up by thousands of servers crunching data in the background. Microsoft’s Windows Phone team worked closely with engineers from Bing to bring Cortana to life. Just as Google Now is indelibly connected with its namesake search engine, it’s impossible to imagine Cortana would ever exist without Bing.

I met with a number of the Bing architects behind Cortana, and it’s clear they’re excited to see their work represented in a single product. While Microsoft is transforming Bing into a platform and service, it’s typically viewed as just another search engine; Cortana, on the other hand, is a true showcase. Over the past several years, Microsoft’s Bing engineers have been working on several services that play a crucial part in Cortana. Foundational technologies like natural language processing and the linking of real-world objects to web data are key, but they were built without a specific end product in mind. If Bing is the house, Cortana is the shiny red sports car in the garage.

To glue all these bits of Bing together, Microsoft’s Mike Calcagno, partner development manager at Bing, joined the search side of Microsoft 18 months ago, and his first big project at Bing was Cortana. "My assessment of the approach when I got here though is … someone needs to actually pull these services together in a way that's coherent," he says. "Everybody who was working on it had a little Cortana doll, and we all put them in our office and when you walked around you saw like….’Oh….he’s in, he has the Cortana doll.’"

This bond carried on throughout the Cortana project. The Bing team spent so much personal time with the Windows Phone group that Cortana eventually thought Calcagno’s work address was a local bar in Bellevue, Washington. "We just really got along with that team. We lived the project with those guys and really worked as one team, and what came out of it really is this first version of Cortana." You could call it a good example of the "One Microsoft" philosophy that former CEO Steve Ballmer unveiled shortly before his departure, where teams work closely together instead of competing internally.

Cortana is the first big test of a giant mind-meld between Bing services, and it’s one reason Microsoft is placing a beta tag on the feature initially. The system needs to learn and improve over time, especially on the voice recognition side, and Microsoft is only launching it in the US for now. In the days leading up to its unveiling, the team was still fixing bugs in the background. Bing principal program manager Vish Vadlamani recalls spending countless days working from 7AM until 11:30PM on Satori — a self-learning system that voraciously chews through thousands of gigabytes of content for Bing’s indexes every day — and he’s hopeful the work is really going to pay off with Cortana’s launch. "It’s in some ways exhilarating, and some ways scary," he admits.

Cortana would not exist without the power of Microsoft's Bing platform

"The vision behind what we’re doing here is that this intelligence can expand beyond Windows Phone," explains Bing director Stefan Weitz. But where exactly Microsoft will take Cortana in the future is still largely a mystery. Third-party apps will be able to integrate with the service, allowing users to simply say, "Hulu, show me the latest episode of Modern Family" and the app will launch with the latest episode, rather like the way voice search works on Xbox. Combined with the reminders, it’s an example of how useful and powerful speech is when it’s done right.

Microsoft has seen what Apple and Google have done, wrapping some of the best ideas from Siri and Google Now into one attractive, easy-to-use interface — but now, the real trick will be to leverage Xbox, Windows, and Microsoft’s other products to get Cortana everywhere. The Bing home page will be updated in the coming weeks with notifications and information displayed in Live Tiles, personalized for each user, perhaps a small sign of things to come. The company has an always-on microphone in millions of houses through Kinect, hundreds of millions of computers running Windows, and a healthy new attitude toward iOS. For now, Cortana lives in your pocket, but her voice might soon be everywhere.