“I don’t like writing, and for two or
three weeks I had been working on a
paper and managed to submit it with 15
minutes to go on the deadline. I woke
up the next morning and said, ‘let’s do
something fun,’” Carlini explains.
The target was the DeepSpeech engine published as open-source code by
Mozilla. “Fifteen hours of work later, I
had broken it,” Carlini claims.
Rather than using noise to confuse
the system, he had found the engine
was susceptible to slightly modified
recordings of normal speech or music.
The system could be forced to recognize a phrase as something completely
different to what a human would hear.
The attacks buried subtle glitches and
clicks in the speech or music at a level
that makes it hard for a human hearing
the playback to detect. Some glitches
buried in normal phrases convinced
the network it was hearing silence.
“I was incredibly surprised it worked
so easily. You don’t expect things to
break so easily. However, much of it was
because I had spent a year and a half on
developing attacks to break neural networks in general,” Carlini explains.
However, as a practical attack, the
method did not work on audio played
through a speaker and into a microphone. Distortions caused by amplifiers
and microphones altered the glitches
enough to cause the attacks to fail. In
Carlini’s version, the adversarial exam-
DEEP NEURAL NETWORKS (DNNs) have advanced to the point where they underpin online servicesfromimagesearchto speech recognition, and are
now moving into the systems that control robots. Yet numerous experiments
have demonstrated that it is relatively
easy to force these systems to make
mistakes that seem ridiculous, but with
potentially catastrophic results. Recent
tests have shown autonomous vehicles
could be made to ignore stop signs, and
smart speakers could turn seemingly
benign phrases into malware.
Five years ago, as DNNs were beginning to be deployed on a large scale
by Web companies, Google researcher Christian Szegedy and colleagues
showed making tiny changes to many
of the pixels in an image could cause
DNNs to change their decisions radically; a bright yellow school bus became,
to the automated classifier, an ostrich.
But the changes made were imperceptible to humans.
At the time, researchers questioned
whether such adversarial examples
would translate into the physical domain because cameras would smooth
out the high-frequency noise mixed into
the digitized images that Szegedy and
others were presenting directly to their
DNNs. Within several years, examples
of real-world attacks appeared. In one
case, stickers attached to a stop sign
made a DNN interpret it as a 45 m.p.h.
(miles per hour) sign even though the
word ‘stop’ remained clearly visible.
Although most of the research into
subverting DNNs using adversarial
examples has been within the realm
of image recognition and classifica-
tion, similar vulnerabilities have been
found in networks trained for other
applications, from malware classifica-
tion to robot control. Audio systems
such as smart speakers seem just as
susceptible to attack using the same
concepts. Similar to the effects of cam-
era processing on images, the low-pass
filtering of microphones and speakers
make some attacks more feasible than
others in the real world.
As a Ph.D. student working with
David Wagner at the University of California at Berkeley, Nicholas Carlini
started looking at fooling speech engines in 2015 as part of a project to examine the vulnerabilities of wearable
devices. The UC Berkeley researchers
thought practical wearable devices
would rely on speech recognition for
their user interfaces.
Their focus switched to in-home
systems when products such as Amazon’s Echo started to become popular.
“We were able to construct audio
that to humans sounded like white
noise, that could get the device to
perform tasks such as open up Web
pages,“ says Carlini, now a research
scientist at Google Brain. “It was effective, but it was very clear to anyone who
heard it that something was going on:
you could hear that there was noise.”
In 2017, a team from Facebook AI
Research and Bar-Ilan University in
Israel showed it was possible to hide
messages in normal speech, though
a limitation of their so-called Houdini method was that it needed to use
replacement phrases, the spoken
versions of which were phonetically
similar to those being targeted. In
November of that year, Carlini found
it was possible to push attacks on
speech-based systems much further.
Hidden Messages Fool AI
Forced errors focus attention on neural network quirks.
Technology | DOI: 10.1145/3290412 Chris Edwards