Denial of service attacks can shut down internet access and leave IT teams powerless
When Jeff McCune noticed that his district’s 500 Mbps internet connection was full, he knew something was amiss. When he investigated further and saw that the Internet protocol (IP) addresses were coming in from China, Australia, and the Netherlands, McCune realized that the problem was more than just a random overload or ISP outage.
“I was seeing 550 Mbps of traffic coming from a single link and that pushed our usage up over the 10 percent cushion” allowed by its main service provider, said McCune, a network analyst with St. Charles Community Unit School District (CUSD) 303 in St. Charles, Ill. “There was no way anyone from China would surf the website of a school district in Midwestern America that hard.”
To McCune, it appeared the CUSD was being hit by a full-blown Distributed Denial of Service (DDoS) attack. The hackers cut off the entire district’s internet access for four hours at a time and then repeated the process 10 more times over the following six weeks during the fall of 2014.
A frustrated McCune hit dead ends when trying to get internet providers to address the problem for his 13,000-student, 17-school district. Using a combination of one major carrier plus a smaller, local provider, he was able to manually transfer traffic and usage between the two providers in order to cut off part of the attack and eventually restore service—at least until the next attack.
“It was pretty frustrating because we couldn’t get anyone to help us,” recalled McCune. “Even the major carrier didn’t have anything in place at that time to deal effectively with DDoS attacks.”
As CUSD and other K-12 districts have learned firsthand over the last few years, DDoS attacks can quickly bring down what has become an educational mainstay for students, teachers, and administrators alike: access to the internet. Defined as an attack in which multiple compromised systems hit a single target, thereby creating a “denial of service” for anyone who is trying to use the targeted system, DDoS prevents legitimate users from accessing information, services, and anything else that’s driven or supported by the internet.
In the CUSD’s case, the entire district was without internet access. “If you could get a web page to open, you were lucky,” recalls McCune. That instantly created problems in the classroom, where teachers relied on Google Apps for Education tools like Google Classroom for instruction. “It’s cloud-based, so no one had access to their information or lessons,” says McCune. Basic communications were also put on hold or handled via conventional telephone lines because email was down. “Instruction was impacted greatly,” says McCune. “Teachers had to scrap their plans for the day—because those plans were in Google—and go to Plan B.”
McCune said the district’s IT team jumped on the problem immediately, but notes that it took time for it to come up with a workable solution. Initially, it took a one-off approach by simply blocking the IP addresses that were causing the disruption. When the hackers figured out that strategy, they started using IP addresses that appeared to be coming in from Microsoft or Google. “We couldn’t use the same strategy and block our access to those sites,” said McCune, “so we gave up on that.”
Working with the smaller ISP, the district was able to cut off the impacted access upstream in its network until the DDoS attacks stopped—at which point access was then restored. “We called various consultants for help and even had some parents who had IT experience reach out and try to help, but nothing was working,” said McCune. “At the time, the services available to help address this type of issue cost upwards of $100,000 a year, which was way more than were paying for internet access.”
While looking for solutions to the recurring DDoS attack problem, McCune learned about the built-in intrusion detection and prevention firewall that the district had at its avail, but wasn’t using, from a company called Juniper Networks. McCune got someone from the company on the phone and was walked through the process of installing and configuring it. “At that point, we could see in detail what was happening and we were able to deal with it in a smarter way,” he said.
So while the DDoS attacks continued, they didn’t necessarily bring the district’s iternet access to its knees—nor did they overwhelm its firewall. And while the hackers continued to maximize CUSD’s 500 Mbps connection, McCune was able to partition the network in a way that would quickly show whether the culprit was an internal or an external IP address. “I narrowed it down to a group of buildings, and the I chopped it up even more and figured out it was coming from one of our high schools,” said McCune, who then traced the problem right down to a single study hall room within that high school.
“We suspected that students were responsible because word was going around school about it,” said McCune. “They were using their cell phones and a cloud/hosted service to get around our filtering and monitoring and then attacking us that way.” (Two students were later expelled and charged with the attack.)
Today, CUSD is using a solution called Border Gateway Protocol (BGP) that allows the district to handle its web traffic and automatically “flip over” to another connection when one has been compromised.
“It does that without any manual intervention on my part,” said McCune. In the event of an attack, the network will “black hole” the IP address in question and stop passing traffic to that address. The new setup has already been tested: In January 2015, middle school students orchestrated their own DDoS attack. This time, McCune says he was alerted immediately and able to take care of the problem within 20 minutes.
“I cross-referenced the information and knew right away that the attack was coming from the wireless network in a specific section of one of our middle schools,” McCune said . “We asked the principal there to make a poignant announcement over the loudspeaker to students and that put an end to it. It was a much faster and better resolution than our first go-round with DDoS.”