A new iPad game may help kids learn proper speech after cleft palate surgery. Thomas Sumner sounds it out. Illustrated by Sage Ayler and Eléonore Dixon-Roche.

Alexis sits on a blue chair in the middle of the room. “Hey, look at this book with us,” she parrots back to a therapist, squirming nervously. As the eight-year-old struggles to make the “b” sound in “book,” her voice trails off. Alexis was born with an opening between the roof of her mouth and her left nostril. Along with one in 700 newborns, she came into the world with a cleft palate. The scar from her reconstructive surgery has faded away, but her speech impediment is a constant reminder of how Alexis was born disfigured.

Without successful therapy, children like Alexis can become distraught about their speech and struggle socially and academically. However, speech therapy requires practicing at home. Many children simply don’t want to read sentences from a worksheet. Now, UC Santa Cruz engineers are trying to make speech practice more fun and help kids like Alexis find their voice.

The engineers are crafting an iPad application that replaces boring worksheets with entertaining, interactive speech games. Creating the game isn’t easy; the engineers must fine-tune software not only to recognize human speech, but also to determine whether it is spoken correctly. The team hopes the adventurous app will encourage children to practice, shortening their recovery and restoring their self-confidence.

“Accelerated speech therapy can drastically improve a child’s life,” says lead researcher and computer engineer Sri Kurniawan. “We all really believe in the big impact this project can have for these kids.”

Cleft distress

A dozen children born with cleft palates are the clients today at the UC Davis Medical Center in Sacramento, where a craniofacial team is meeting to discuss each patient. For 25 years, expert plastic surgeons, dentists, social workers, speech pathologists, audiologists, pediatricians, nurses and otolaryngologists have gathered here to support cleft palate patients from birth to adulthood. The team meets once a month to talk with patients and create a plan of care.

Cleft palates are a gap between the roof of the mouth and the nose, caused by a mix of genetic factors and complications in the womb. Clefts can run into the ear canal and cause hearing problems, make teeth come in crooked and create trouble breathing at night. A cleft that only affects the lip and not the palate is called a cleft lip. Surgeons can repair these defects through plastic surgery within a few years after birth.

Many children born with cleft palates, usually in developing countries, will never access corrective surgery. In Indonesia, for example, only about one-quarter of those born with cleft palates receive treatment. Nearly all such children in the U.S. undergo surgery. But there can be complications — and difficult physiological challenges along the way.

As the team discusses each patient, a projector shows the child’s picture. Many of the children here aren’t from the area; they travel long distances to meet with the specialists. One patient had several unsuccessful plastic surgeries for his cleft palate at a different hospital. “That never would have happened had they come to us at the start,” says one of the plastic surgeons, shaking his head.

The team discusses everything from a patient’s speech to his or her school performance. So many experts are needed for the cleft palate panel because so many issues might crop up.

“It’s sometimes a Catch-22 with cleft patients,” says team member Travis Tollefson, a plastic surgeon. “They might be behind in class because they need speech therapy, but pulling them out of class for therapy just puts them further behind.”

All of the children seen today have some speech impediment; a few still have severe problems. Children with cleft palates cannot properly control airflow in their mouths. A child’s “cleft speech,” as it is sometimes called, is characterized by them replacing or skipping certain sounds they have difficulty making. For instance, a child might substitute a whispered cough, called a glottal stop, for consonant sounds like b, c, g, k, p, and t. These challenging sounds are known as “stop-plosives.” Ordinarily, speakers form these sounds by using the tongue to block their airflow when enunciating. But if a young child can’t do it, she develops bad habits. The habits linger even after surgery.

	Illustrations: Eléonore Dixon-Roche
Air escapes through the roof of the mouth of a child with a cleft palate (left), preventing her from speaking certain sounds. After the cleft is repaired (right), speech therapy helps the child learn to use her tongue to create "stop-plosive" consonant sounds like b, c, g, k, p, and t. (Click on image to see larger view.)

Speech therapy is a struggle for many kids born with cleft palates. Among the patients undergoing therapy at the hospital, doctors have seen their speech degrade unless they learn to speak slowly and deliberately. In some cases, it takes years of multiple surgeries to repair a severely cleft palate. Such children find themselves with ever-changing mouths — and they struggle to keep up.

“They have leftover habits from how they used to talk,” says Susan Goodrich, a speech pathologist at the UC Davis Medical Center. “Their muscles are trained to it.”

Such speech problems can damage kids psychologically, says Louise Dalton, a consulting clinical psychologist at Oxford University. Studies have shown higher rates of depression, anxiety and social difficulties in children struggling with compromised speech. The root of these social problems is how the child perceives his or her own voice.

Dalton notes that teachers often expect less of a student with poor verbal communication skills, even though they are otherwise on track developmentally. “If the teacher has lower expectations for a child, then the teacher won’t stretch the child enough and the child will have less of a target to work toward,” she says.

The impact of speech difficulties on a child’s social and academic life makes effective speech therapy that much more important. “They need to get positive feedback that says 'my speech is getting better' as opposed to 'my speech is a complete disaster area,'” Dalton says.

Speak to the screen

Video games work on a similar cycle of building confidence. Games for children are broken down into a series of tasks: jump over this gorge, smash this block, grab this coin. At each step, a player gets rewarded with sounds, animations and points.

For Sri Kurniawan, the idea of combining speech therapy and video games started during a short conversation at an engineering meeting. A plastic surgeon approached Kurniawan and asked whether she was interested in developing a game for children with cleft palates and lips.

“The idea stuck with me for the next few months, and I realized it was an exciting way to use games to help people,” says Kurniawan, whose lilting accent combines her Indonesian childhood and British education.

Kurniawan began the speech therapy project with no funding and no staff. She found it hard to find grants for a young academic venture in such new territory. She found her first team member by chance: Zachary Rubin, a computer engineering graduate student at UC Santa Cruz, heard Kurniawan talk about her plans and knew it was a perfect fit.

Rubin, 24, had worked as a sound technician at a music studio since he was 13. He also was a software developer for Blizzard Entertainment, makers of the popular World of Warcraft video game franchise, before returning to graduate school. His long dark hair is died green apple on the left side, turquoise on the right side and poppy red in the back. Rubin gave the project a jump-start, but Kurniawan recalls that for the first few months he had to work pro bono. Now, the team has some funding from UC Davis to support its game, called “Speech Adventure.”

As we sit in his shared office, Rubin pulls up the game’s first test page on an iPad simulator running on his laptop. It’s structured like an interactive storybook, with different scenes requiring unique actions. The game begins with its protagonist, a banana slug named Sam, trying to cross a bridge covered with balloons. The game asks in a pleasant male voice, “Sam wants to cross the bridge, but there is a balloon in the way. Can you help him clear it?” Rubin speaks the magic phrase, “Pop a balloon.” Pow! The balloon disappears from the bridge with a loud bang. “Good job, you helped Sam clear the bridge. Now he can continue on his journey.”

“You might want to turn the volume down on that popping sound,” proposes Rubin’s startled officemate. “Just saying, you might scare the kid.”

The game is made up of multiple scenes like this one, each focusing on a different speech practice phrase like “take a toy” or “stop the bus.” One proposed scene involves a house in need of repair. The child says, “Daddy can do it” (designers didn’t use “Mommy” since m’s are easier to say than d’s).

Rubin then opens a screen window with the computer code that forms the heart of the speech recognition software. After typing a few commands, he suggests I try one of the pre-programmed phrases. “Pop a balloon,” I project into the laptop’s microphone. After thinking for about two seconds, the program prints the phrase “pop a balloon.” The phrase reminds me of my own speech therapy in elementary school, when my r’s sounded like w’s. I don’t recall doing my speech homework then; maybe I would have if it meant playing a game instead of reading sentences from a piece of paper.

As I speak, the computer records my voice. It then slices the full sound wave into smaller segments and analyzes them. The software searches each segment for the smallest unit of human speech, called a phoneme. Words are composed of syllables. For instance, the word “balloon” is made up of two syllables, “buh” and “loon.” Each syllable, in turn, comprises one or more of the 44 phonemes in the English language. The syllable “buh” is composed of two phonemes: “b” and “uh.” As the computer parses my speech, it transforms the sound wave into a sequence of nine phonemes: “p-aw-p A b-uh-l-OO-n.”

The software then searches through a custom dictionary to find the closest match to the series of phonemes. The dictionary doesn’t just include the correct way of saying the phrase, but also some of the ways a child could mispronounce it. That’s a steep challenge, and other researchers haven’t yet tackled it. Indeed, according to Rubin, this is the first medical device geared toward patients to use speech recognition software.

To mimic cleft speech, Rubin pinches his upper lip between his thumb and forefinger. His voice makes almost exclusively vowel sounds. After a brief moment, the computer reads back the closest dictionary item, “hah a haloon.” If a child struggles for too long on a certain part of the game, the software can quickly adapt and lower the difficulty level to keep the player from getting discouraged.

“We’re trying to find the balance between learning and the fun parts that keep the kid playing,” says Rubin.

The size of the dictionary is a challenge, too. A larger dictionary provides more finesse, but if it’s too large, accuracy drops. The new entries crowd out the correct pronunciation, making it tough for the software to identify a phrase as spoken correctly. Ultimately, the team needs more than 95 percent accuracy for the game to be fun and frustration-free. “We can’t make the game at all annoying to use,” says Rubin. “The kids will just throw it down and say, ‘I don’t want to play that.’”

The game is designed to be fun for ages 2 through 12, the typical ages of children undergoing cleft palate speech therapy. Each age-appropriate story will include random elements to keep the game fresh through multiple sessions. Younger children might be content popping balloons on bridges, but Rubin admits they will need to add more challenging tasks for older kids to stay engaged. Right now, children in the craniofacial waiting room at the UC Davis Medical Center can play "Speech Adventure" on several iPads provided by the UC Santa Cruz team. After playtime, the children and their parents can fill out feedback cards with suggestions on how to make the game more fun and effective.

Patient power

“Speech Adventure” is designed to encourage patients to follow through with their speech homework. These practices are usually lists of sentences the children should repeat throughout the week with his or her parents — like scales or chord assignments from a piano teacher.

“Parents do try to help, but they aren’t trained speech pathologists so they can’t hear the tiny changes in speech,” says Kurniawan.

Rubin is designing the system to store samples of the child’s spoken phrases as she plays the game. The speech therapist can then play back the recordings at the next session to let the child hear the difference between good and bad speech in her own voice. Alternatively, the game can be loaded with samples of a child’s own correct pronunciation from the therapist’s office to help the patient recognize when she has made progress.

The application will hold the children accountable for following through on their practice. As someone plays “Speech Adventure,” the game measures and tucks away information for later use by the speech therapist. Using this data, the therapist can see the amount of time played and the practices completed since the last office visit.

“The whole point is to make dull repetition fun. You need that repetition to build muscle memory,” says speech pathologist Goodrich, who consulted with the team. “Our hope with this game is to let kids spend fewer years of their childhood working with their speech therapist.”

But not every child has regular access to a trained speech pathologist. Money can be a serious constraint for parents. Some healthcare plans will only pay for one speech session a week, if at all. Children living in rural areas may need to travel hours for speech therapy, making regularly planned sessions a burden.

“We hope this app can supplement or even replace regular speech therapy sessions,” says Kurniawan. “We want to automate the professional speech therapist.”

Eventually the team plans to translate the game into Spanish and Chinese. In low-income countries, the game could become the first professional speech therapy available for many children.

Soon the team will distribute iPads preloaded with the game to speech therapy patients around northern California. The researchers will survey parents, kids and therapists to fine-tune the game. The new speech recognition and adaptive software technologies should prove useful for other educational efforts too, Kurniawan says.

This trend in computer software is sometimes called “games with a purpose,” combining the instant gratification of video games with a teaching goal. “When you learn something difficult in a game, it doesn’t feel that hard,” says Kurniawan. “Learning how to slingshot the birds on ‘Angry Birds’ is a lot more fun than if you were to try to calculate projectile trajectories on a computer.”

Kurniawan and her colleagues are curious about how children will react to their games after years of use. She refers to a “honeymoon period” of about three months when games are still new and interesting to players. The challenge, she says, is keeping them fresh and engaging over the long haul.

One commercial example they would like to emulate is indeed “Angry Birds.” Despite having been released in 2009, the app has maintained a loyal following due to regular content updates. The team plans a similar technique for “Speech Adventure,” which will add new stories for kids to explore.

“It can take years for a kid’s voice to fully recover, and we have to keep them motivated,” says Rubin. “We’re trying to make a game that’s fun enough so that the kids don’t realize they’re taking their ‘medicine.’ If we can do that, then their recovery times will be dramatically shorter.”

T is for tiger

Alexis has been at the UC Davis Medical Center for several hours. She was shuttled from room to room, and she’s a bit bored and grouchy. When her dad returns to greet her with a tattered but beloved book about tigers, she smiles and takes the book from him.

“Do you like tigers? I could definitely make a game about tigers,” Rubin says, bending over to look at the book with her. “Tiger is a great word, too. Can you say tiger?” Alexis giggles and says “tiger” perfectly.

Story ©2013 by Thomas Sumner. For reproduction requests, contact the Science Communication Program office.

Top

Biographies

Thomas Sumner
B.S. (physics) University of California, Santa Cruz
Internship: American Geophysical Union (Washington, D.C.)

I never understood how people could say science is boring. Wall-scaling robots, battalions of white blood cells, Europa’s hidden oceans — science is where all the cool things are! Growing up, I ached to become a scientist, to explore the unknown and make a cool new discovery of my own.

In college I learned about the quirky quantum world and the particle zoo. I studied particles blasted from the sun that whiz around—and sometimes into—our planet. But I also worked with fellow students on social justice, and I realized science goes far beyond merely being cool. As I saw how scientific advancements can truly better people’s lives, my passion became helping others understand and enjoy science. So I packed up my magnetometer and set off to share this passion with the world.

Thomas Sumner web site

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Sage Ayler
B.S. (biology) Colorado State University
Internship: Rocky Mountain Biological Laboratory

As the granddaughter of a passionate biology professor who nurtured my curiosity, I grew up with a love and deep appreciation for all living things. My intention was to follow in my grandfather’s footsteps and pursue a Ph.D. in some biological field that captured my interest and ultimately to teach, but when it came time for that, I felt a need to take a different path. Having always been artistic, I decided to pursue a career that would incorporate both my love of science and my love of art. I can only hope that on this path I can bring attention to all the beautiful things that are encompassed in the natural world.

Sage Ayler web site

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Eléonore Marie Dixon-Roche
B.A. (studio art) Carleton College
Internships: Smithsonian Museum of Natural History, Botany, Division of Fishes, Paleontology

My goal in life is to get people excited about and help in understanding science through visual aids. When I was in secondary school I ended up doing exclusively science but it was not fulfilling enough. In college, although I majored in art, I did almost one-third of my courses as independent studies pairing science and art. This led to a year in Washington D.C. where I did many kinds of work at the Natural History Museum. I look forward to years to come illustrating science.

Eléanore Marie Dixon-Roche web site

Top