Fixing Glitchy Games

Too many video games are beset by show-stopping glitches. Donna Hesterman visits an engineer with a new way to catch those bugs. Illustrated by Alexis Demetriades.

Illustration: Alexis Demetriades

An American soldier in full combat gear runs towards a pre-planned helicopter extraction point outside a small Afghan village. He begins to fatigue and gasps for breath as he runs full-tilt toward a gate that leads to the landing zone. Pursued by a band of angry villagers, he lunges for the gate. It's stuck. He struggles again to lift the handle. No joy. The soldier turns back to find his partner, a combat hardened special-ops commando, hopelessly hung up in a small scrappy bush the size of a toy poodle. The soldier can neither move forward nor retreat. Trapped between the enemy and a jammed gate, he does the only thing he can do.

Reboot.

To the dismay of thousands of video game fanatics who bought Electronic Art's 2010 release of Medal of Honor, that is how hard-fought battles often ended—foiled by crazy software glitches that defied the laws of physics. Despite rigorous testing before its release, the game was riddled with bugs that stopped the game cold.

Medal of Honor received a thorough panning in The New York Times and other game review venues like Metacritic.com. Sales of the game lagged despite an epic marketing campaign to hype its release. The flurry of negative press left the colossal game maker with a virtual black eye.

Old-school testing tactics deserve the blame for glitch-lousy games like Medal of Honor, says Chris Lewis, a software engineer at UC Santa Cruz. Games are far more complex than they were 30 years ago, yet game producers rely on the same archaic testing procedures they used when Pac Man and Pong were all the rage.

“There’s a big problem coming,” Lewis says of the video-game industry. “The current strategy for testing isn’t scalable. Twenty years ago, it took a team of ten or so to test a game. Now it takes up to 200 people.” And game producers, even powerful companies like Electronic Arts with multi-million-dollar budgets, are missing a lot of bugs that foul up their creations.

Lewis thinks he has a solution, and it won't cost much. In fact, developers may even save money. But it's a change in business as usual—and it's not clear the industry is ready to give it a go.

Bug zapper

Lewis frets about the future of video games like some people worry about spotted owls and old-growth forest. The walls of his cubicle in UCSC's Expressive Intelligence Studio are covered with “bits of tat”—that’s Brit for “odds and ends”—about his favorite games.

“Games have emergent properties baked into them,” he says, describing their ability to take players in wholly unexpected directions. “That’s what makes them exciting.” At his wall-sized whiteboard he wipes away small, indiscriminate circles of his labmates’ scribbling to make way for his own. A new generation of bug plagues games like Medal of Honor, he says, because artificial intelligence and physics engines give characters in the game virtual minds of their own. Each element is functioning properly: the player, the game engine, the physics engine. But they come together in a way that the programmer never anticipated.

“And now your army guy can’t open the gate to get to the next layer of play,” Lewis says. “You’re stuffed!”

Photo: Donna Hesterman UC Santa Cruz software engineer Chris Lewis.

Lewis looks like a gamer. He’s in his late twenties, wearing blue jeans and a Run DMC T-shirt, hunched over his computer keyboard, scrolling through code. However, his time at the workstation has gone not into playing, but creating a radically new video game bug-zapper he calls Zenet.

Compare Zenet to the current testing regime, where a developer sends a glossy, late-stage production version of the game for testing. The test team then works feverishly over the course of many months to put the game through its paces and ferret out the bugs. “It’s actually a lot of work,” says Kai Hsu, a programmer for Activision. He was a tester when Deer Hunter was released in 1998. “You aren’t just passively enjoying a game. You’re trying to break it.”

When a tester exposes a problem, he reports it to the programmers. “But the programmers only want to hear about the problems you can reproduce reliably,” says Hsu. If it's just an occasional problem or one that you cannot re-create on demand, the glitch is never fixed, because they don’t know where to look for a problem in the code.

Not so with Zenet. Zenet doesn’t worry about the cause of the glitch. It simply notes that a problem has occurred and corrects the situation. No amount of game testing can perform every possible combination of events in a sophisticated game like Medal of Honor. Some aspects of the game will remain unexplored even after thousands of hours of play. And how do you repair a glitch that you cannot find? “You don’t,” says Lewis. “You accept it.”

Lewis says that game makers must be prepared for more glitches to pop up once the game has left the store shelves. Thousands of players pushing the limits of a game will undoubtedly manage to create situations that the testing team never encountered. To compensate for these potential failings in foresight, he has designed Zenet to operate as a post-market, real-time virtual overlord, to ride shotgun alongside a game’s software. Using a checklist of rules created by the game’s designer, Zenet monitors the game and makes repairs on the fly even after the game is in the consumer’s hands.

“In the Medal of Honor scenario, the soldier stuck on the bush is an artificial intelligence character,” Lewis explains. The game is supposed to proceed to the next level after the player’s special operations partner arrives and assists the player with opening the gate. But since his partner is stuck in the shrub, the game cannot proceed. If Zenet was installed, “it would see the problem and open the gate within a pre-determined amount of time—with or without the partner’s assistance,” says Lewis.

AI ups the ante

An innovation like Zenet might embolden developers to incorporate artificial intelligence (AI) into their game at a higher level, says UCSC computer scientist Michael Mateas, who co-directs the Expressive Intelligence Studio.

“While you see some high-water marks for the use of artificial intelligence as a game narrative driver, most developers only use it in a supporting role,” Mateas says. That is, AI typically is a means to activate characters that respond to the player briefly throughout the game. Shooter games like Call of Duty use it to power the bad guys. As a result, their attacks are more challenging than they would be if they only responded to a player’s choices with tightly scripted, predictable actions.

“It’s common to see artificial intelligence used for these supporting roles, but artificial intelligence as a driver for story progression has made almost no inroads in the industry,” Mateas says. “Quest lines are still mostly linear, and all dialogue is pre-written.” In other words, games are still pre-ordained progressions with intermittent interludes of creative interactive elements.

Rather, Mateas wants to see games move away from their scripted structures toward a narrative driven by the player’s imagination. “But when I talk to the industry about using artificial intelligence to create non-linear game experiences,” he says, “the big fear that always comes up is, 'How do we test that?'”

Mateas specializes in using AI to manage socially interactive games. One of his first AI-driven games, Façade, became famous in 2005 as the first 3-D rendered, completely interactive storyworld game. Reviewers hailed it as a “genre-defining leap forward” and “the future of video games.” Indeed, Façade inspired Lewis to leave his home in the U.K. and move to Santa Cruz for his doctoral research.

Lewis first discovered Mateas’s online game during his undergraduate years at the University of Bristol in England. It was an interactive drama that he couldn’t leave alone. In Façade, the player is invited to visit old friends, Trip and Grace, a thirty-something couple, in their swank uptown apartment. Trip and Grace are artificially intelligent characters who carry on a fully interactive conversation with their guest, the game’s player. Characters in video games usually communicate with a canned collection of phrases, but not Trip and Grace. They respond to dialogue typed on the screen by the player. Slowly and inevitably they draw their guest, the player, into an intractable marital spat.

“I started trying to find the edge cases,” says Lewis. “What happens if you swear at them? What if you make moves on Grace while Trip is watching?” It was his first liaison with AI, and he was hooked. Nothing about the exchange was pre-scripted. The characters responded directly to his cues, and the effect was exhilarating.

Years later, as Lewis sorted through the handful of universities that offer doctorate degrees in video game development, he saw that Façade’s creator was doing research at UC Santa Cruz. “I knew that’s where I needed to be,” he says.

Ready for market

The expressive intelligence team, housed at UCSC’s Baskin School of Engineering, has embraced Lewis’s vision. “Zenet feels to us like it’s ready for the next step,” says Jim Whitehead, Lewis’s thesis adviser and chair of the computer engineering department. “It works in the lab, and it could undoubtedly improve a game’s quality.” But there are obstacles to overcome within the industry before game makers will invest in bug-busting technology like Zenet, Whitehead notes.

The biggest obstacle is a doozy: “Video game producers are notoriously conservative,” says Michael Zyda, director of the Gamepipe Lab at the University of Southern California. “They prefer a brute force approach to problem solving—a go for what you know mentality,” he says. Finding a game maker willing to experiment with a new production procedure will be tough because the financial stakes are high, according to Zyda.

Lewis agrees that the industry will be slow to accept his innovation. “That’s precisely why this is an appropriate engineering problem for a university researcher,” he says. Game developers, with their marketing-driven timelines, would be suicidal to stop and invest time to develop better testing technology.

For gamers, the allure is more in flashy graphics and fantastical, otherworldly experiences, rather than a game that performs reliably, says Whitehead. In fact, some gamers appear to have a strange affection for certain types of glitches. They thrive on finding obscure video game bugs and outing them to the public as hacker-produced YouTube gems set to music or comedic commentary. An Internet search of “Tiger Woods walk on water,” “Donkey faced woman from Red Dead Redemption,” or “Best video game glitches” will produce an extensive catalog of the home-spun genre to which Whitehead refers.

But Lewis argues that a dependable playing experience can pull rank over flashiness: “There is a certain invisible threshold of how many bugs a gamer can tolerate before it becomes too much.” Clearly, Medal of Honor crossed a line. “According to The New York Times article,” Lewis observes, “the game just wasn’t playable.” As for the glitch lovers on YouTube, “One or two bugs may be funny if they aren’t show stoppers,” he says. “But there is good evidence that poor reviews will lead to poor sales.”

Sexy features and stunning graphics may drive marketing and production timelines for entertainment games. However, a different part of the industry moves at a slower, less competitive pace: educational games. That could be fertile ground for Zenet to make its commercial debut.

Still, Lewis has gained some notice among the gaming elite. “There’s a good chance the idea is brilliant,” says Bob Mitchell, a senior programmer at Ojai Games in Redwood City, near San Francisco. “Zenet catches many of the same bugs that we test for now, only it finds them more often. It would also get designers to think about problems ahead of time,” Mitchell adds.

Since Zenet communicates in machine code and in English, it can help close the gap between designers and programmers. As it stands, designers dream up a game concept, communicate it to the artists and programmers, and then their work is done. Once a game is in production, there is little a designer can contribute to debugging unless he reads code, which is seldom the case. But if Zenet identifies problems in readable English, a designer can weigh in on possible solutions that meld well with the game’s original concept.

Considering the multi-million-dollar budget developers have at their disposal, Zenet is not expensive. “It took me, a single person, not very much in terms of man-hours to put it together,” says Lewis. “I could have it going in a small game over a summer internship. Big games cost over $100 million to produce. A graduate student for three months is not going to break the bank,” he says.

“It’s not that we’re putting forth the proposition, it’s expensive but it's worth it. It’s cheap and it’s worth it.”

Story © 2011 by Donna Hesterman. For reproduction requests, contact the Science Communication Program.

Top

Biographies

Donna Hesterman
B.A. (history) University of Florida
Master of Natural Resources (wildlife science) Auburn University
Internship: Scripps Institution of Oceanography news office, La Jolla, CA

“So, do you think you’ll stick with this?” my sister asked. Over the years she's grown weary of my career shifts: military combat pilot, bird biologist, and now, burgeoning science writer. She thinks I moved every three years when I was a Marine because of some insatiable wanderlust. I still live my life in three-year increments because I married a Marine. Any career I choose must be portable and borderline addictive if I am to stay focused year after chaotic year. Reporting science news and writing about what other scientists do is bound to be more interesting than tracking, capturing, and relocating thousands of birds. Besides, for a mother of two, it's a far more sustainable lifestyle than flying combat helicopters. So yes, I think I will stick with this.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Alexis Demetriades
B.A. (fine art) University of Southern California
Internship: Smithsonian Museum of Natural History, Anthropology Department

When I graduated with a degree in fine art, I was surprised to find that I wasn’t quite ready to let go of the various other interests I had developed over the years. I had a strong desire to produce artwork that would contribute something functional and useful to the world. When I discovered the scientific illustration program at CSUMB, I felt an immediate sense that this is what I wanted to do. My time at CSUMB has been incredibly rewarding, and I am looking forward to becoming a scientific illustrator with the freedom to incorporate my other interests. My ultimate goal is to assist with the documentation of the crafts of fading cultures with the hope that I can play some part in preserving these traditions.

Top