After lunch on Friday, October 21st, I sat down to do a code review— and couldn’t connect to Github, the web site that hosts our code. I wasn’t alone. Most of the east coast of the United States found itself unable to connect to popular web sites like the BBC, CNN, Netflix, and Yelp. As the day went on the outages spread to the west coast and beyond.
What Happened
A popular DNS provider, Dyn, was flooded by devices sending them more traffic than they could handle. This was not an accident: this was a malicious, deliberate attack designed to cripple the internet.
Most of the traffic was generated by smart devices: webcams, smart TVs, DVRs, thermostats, and tablets that connect to the internet. These devices were infected with a virus that created a botnet, a massive network that criminals can use for whatever they want.
Previously botnets have been implicated in taking down single web sites. They did this by flooding the web site with more traffic than it could handle. It’s a little like if a house were to receive a thousand or ten thousand letters in a single day, the water bill would be lost in the pile.
On Friday, a botnet dubbed Mirai was directed at Dyn, one of the largest DNS providers in the United States. DNS providers are the services that translate web sites, like www.mutuallyhuman.com, into an IP address, the address that computers can use to connect to each other (The old analogy for this is a phone book, but I’m dating myself if I use that reference). By attacking Dyn, my computer wasn’t able translate www.github.com in to the address it needed. That was also true of any web site that used Dyn for their DNS. The ever changing nature of the internet means that this list can’t be stored and computers must have a place to look it up.
The engineers at Dyn did an admirable job fighting off this attack. Their job isn’t going to get easier, however. With more devices being connected to the internet every day there are more devices for criminals to infect with their code. In fact, popular security researcher Brian Krebs of Krebs on Security commented on this just this month. Internet providers like Level 3 have taken up the call to warn that attacks like these may increase. Even the source code behind the Mirai botnet is publicly available, and refined versions of Mirai already exist.
We’re all in this together. To prevent outages like this in the future we all need to take steps to secure our devices. Fortunately, there are things all of us can do to help.
What You Can Do: Level Beginner
Change Default Passwords
The primary way the Mirai botnet and its imitators spread is by infecting devices with default passwords. Many Android devices, like smart TVs, DVRs, tablets, and thermostats, have default passwords set to make configuration easier. The botnet makers know the default passwords and use them to take control of the devices.
Internet routers are especially vulnerable to this kind of attack. Whether it was purchased from a store or provided by an ISP like a cable or phone company, routers almost always come with a default administrative password. Since these devices are designed to route traffic and connect to the internet, they make an ideal device for botnet writers to target.
Sometimes ignored are all of those devices with convenient apps to control them. When people watch their pets over internet connected web cams, it’s easy to miss that they are also computers connected to the internet.
Simply by changing default passwords in all of these places we can reduce the number of devices that can be infected.
Return Insecure Devices
Changing passwords only works on devices that allow the default password to be changed. That is not true for many of the devices infected by Marai. Rebooting them will remove the Marai code, but the botnet is so widespread that they are reinfected within minutes.
Use the store’s return policy if you discover that your device’s passwords cannot be changed. It may seem like a convenience, but it helps attackers more than it helps you.
Keep Your Software Up to Date
Botnets also scan for exploits or holes in common devices that will let them insert themselves into their code, like the biological viruses that they are named after. Software and device manufactures regularly patch the exploits that they know about. The only way these fixes can be distributed is if their owners regularly check for updates and install them.
Fewer exploits means fewer infected devices and less traffic that can be generated. Again, routers are especially vulnerable to this kind of attack and it’s worth checking for updates to them regularly.
Use Your Firewall and Antivirus Software
Computers are not immune to being infected by botnet code. Windows and macOS come with firewalls that provide a first line of defense against being taken over.
What You Can Do: Level Intermediate
Switch to a DNS Provider That Caches Responses
Some DNS providers like OpenDNS have have started to cache their responses. When everything is working as it should, these servers will act like any other DNS provider and pass queries along. However, they also save the response. If later the query can’t be resolved in the normal way, these servers will use the saved response instead.
Caching is an interesting tactic to combat attacks on DNS providers. More providers may start to implement caching as a hedge against these attacks. It’s not free, however, and the cost may prove to be prohibitive.
Caching is also not foolproof. The saved responses may no longer be correct. Also, while some computers may be able to find the correct destinations for traffic, those destinations may not be able to connect to the servers they depend on.
Whitelist Incoming Traffic at Your Router
Most routers allow traffic to be filtered and blocked. Further most home devices do not need to receive inbound internet traffic. Blocking inbound traffic at the router cuts off a vector to infection.
Web browsing, Netflix, Hulu, and other streaming services continue to work when all inbound traffic is blocked. Some services may require special router configuration, usually special port forwarding setup. If port forwarding was never configured, then this is probably a safe setting to turn on. At the very least it won’t hurt to test turning this setting on for a short time. If all of the devices work, then it can be left alone.
This won’t protect the router itself. That is another reason it is important to keep the router’s firmware up to date and to make sure it is using a strong, non-default password.
What You Can Do: Level Advanced
Disable Universal Plug and Play
In a follow up to a Marai attack, Krebs pointed out that many devices will open holes in firewalls on Windows networks. They do this through a Microsoft protocol called Universal Plug and Play.
This is confusingly named. Universal Plug and Play is not required to connect add-ons to your computer. It is a protocol for network devices, and it’s functionality most people don’t need.
Krebs recommends Steve Gibson’s tools for disabling Universal Plug and Play.
Find Infected Devices on Your Network
I haven’t found a good tool for doing this yet. I’ll update this post if I do find one. Sign up to be notified when we update this post.
Before an attack you can scan your network to see if it is open to UPnP attacks. Again, Steve Gibson has helpful scanner.
If an attack is ongoing, you can use typical traffic analyzers like Wireshark or router logs to see if your network is participating.
Spotting this traffic may be tricky. A lot of devices can generate traffic to sites that are difficult to recognize. However, multiple requests to a single, unrecognized site are worth investigation. I like using the Unix text processing tools for finding sites that I’m sending a lot of traffic to:
cat <log> | cut -f <IP address column> | sort | uniq -c | sort
You can apply that command to a traffic log from your router. The sites at the top of this list are worth investigating.
Set up Static Routes for Critical Services
If there are a few, critical web sites or services that you need access to, you can have your computer bypass the DNS lookup by setting up a static route.
Window, OS X, and Linux still all have a hosts file. For individual computers this can be a way to bypass the failing DNS query.
If you maintain your own internal DNS, you can set temporary aliases for the web services you need. It is important to get the IP addresses from authoritative sources. Calling the people hosting the web services may be a good way, or you may be able to find a good cached response.
The aliases and host file entries should be removed when the attack has been mitigated. This will allow normal DNS resolution to resume.
Do All of the Above For Your Friends and Family
Again, we’re all in this together. The more devices we can protect from being infected the less power that can be co-opted for this kind of attack.
