Phone: 1-800-862-5965

Milwaukee Datacenter (MKE1) Network Outage Post-mortem – April 8th, 2014

On Tuesday April 8th, 2014 CyberLynk’s Milwaukee Datacenter (MKE1) suffered a large scale distributed denial-of-service attack (DDoS). This attack’s sole purpose was to block legitimate traffic to our network. There were no breaches of data or customer information. This large scale attack saturated all (5) five bandwidth providers we have in our Milwaukee Datacenter (MKE1). These network providers include: ATT, TW Telecom, Cogent, Level3 and Time Warner Cable.

At 3:37PM CDT (GMT-5) CyberLynk’s monitoring systems detected the attacks and within minutes notified our NOC engineers. The attack ramped up very fast and it became apparent that our internal mitigation protocols would not match up with the size of this attack. Engineers began working on mitigation at the upstream level and disconnecting from providers that could simply not mitigate the attack. We don’t have exact data from all of the upstream provider’s network utilization. With the data that we do have, we know the attack reached over 60 Gbps in network traffic. We will announce a network maintenance window early next week in which the disconnected providers will be added back to our BGP peering.

DDoS attacks themselves are fairly common place. We believe strongly in preventative management. We usually find that targets of large scale attacks are often also in violation of our AUP or ToS policies and no longer remain hosted in our data center. More common attacks are quickly mitigated by our engineers. We also scan for unsecured services on our networks that could be used for DDoS purposes (NTP, DNS, etc). We work with those customers to configure their servers properly to prevent any such occurrence.

At 6:52pm CDT (GMT-5) connectivity to our Milwaukee Datacenter (MKE1) was partially restored. At this time customers were able to reach their servers again. Certain parts of Milwaukee Datacenter still had intermittent routing issues while our engineers resolved issues pertaining to our access layer switches.

As of 11:17pm CDT (GMT-5) on April 8th, 2014 connectivity to our Milwaukee Datacenter (MKE1) was stable again. CyberLynk will be scheduling a network maintenance event within the next week to implement a few new routing policies and to add the rest of the bandwidth providers back into the BGP peering group. This announcement will be made via Facebook, Twitter and our blog.

Moving forward, we’ve been working with upstream providers and adjusting our policies to lessen the time it takes to mitigate such a large attack at the upstream level.

Again, we would like to thank all of our customers for their patience and loyalty to CyberLynk during this network outage. We understand you have many choices when it comes to your Internet connectivity, hosting and datacenter services. Because of that we will never stop working to improve our network and provide feature rich hosting and datacenter services for years to come.

If you have specific concerns please contact sales@cyberlynk.net and a ticket will automatically be opened.

Over the past few days we have been asked many questions regarding this attack and our plans moving forward. Here are a few of those questions/answers:

QUESTION: What is the purpose of a DDOS attack?

ANSWER: Here is the official answer. In simple terms a lot of communication requests are sent from hundreds or in most cases thousands of compromised computers around the world to one or more targets. Those targets are then flooded with so many requests that render that computer unavailable since all the resources are used to try and respond to flood of requests.

QUESTION: What is the point of a DDOS attack?

ANSWER: Normally these attacks are motivated by individuals or groups of people that are trying to further their agenda politically or feel they have been offended and want to strike back at the offender.

Digital Attack Map – Top daily DDoS attacks worldwide (More Info: What is a DDoS Attack?)

QUESTION: How stable is CyberLynk’s network?

ANSWER: CyberLynk has been in business since 1995 and while our team of engineers works to make our Datacenters more resilient you can be assured that our primary focus is keeping our network up and our customers happy. Our knowledgeable staff is always available to answer your questions whether they are related to the services we offer or not. We will provide honest and straight forward answers each and every time. We appreciate all of our customers and work diligently every day to ensure your hosting and datacenter experience exceeds your expectations.

QUESTION: What will CyberLynk change in response to this attack?

ANSWER: While CyberLynk cannot prevent a DDOS attack from happening we are taking steps internally to review existing customer accounts that might be prone to DDOS attacks and we are working with our bandwidth providers to implement DDOS mitigation services. While not every bandwidth provider offers such a service we are working with the providers that do. As of this notice a few have already been implemented. We have also increased the number of Netflow servers on our network in an effort to provide more detailed information to aid in mitigating possible future DDOS attacks.

QUESTION: Would having more bandwidth providers have prevented this?

ANSWER: The number of providers is not as important as the available bandwidth. If we had 100+ Gbps of available bandwidth with multiple connections from (5) providers we could mitigate much larger attacks from within our network. Very large data centers usually build out with a high number of providers. The facility our PHX1 data center is in, has over 20 providers for example. These facilities offer bandwidth as a “blended product” where they bundle 3 to 5 providers in a BGP mix and offer it to customers hosting in the data center. If a hosting provider is in a facility with 20+ providers, it doesn’t mean their traffic goes over all of them (or even most of them). Note: We operate PHX1 with it’s own AS# and IP’s that we own. We manage the peering directly for our network at PHX1.

QUESTION: Has CyberLynk been attacked before?

ANSWER: CyberLynk’s monitoring systems detect and mitigate DDOS attacks and certain known software vulnerabilities on a weekly basis. These attacks are normally smaller in scale and are many times stopped within minutes of our monitoring system detecting them. Again, the DDOS attack from April 8th, 2014 was larger than the bandwidth capacity to our Milwaukee Datacenter and because of that no matter what actions CyberLynk engineers implemented on the edge of our network it would not have mattered.

QUESTION: Other hosting providers claim 100% uptime.. Why doesn’t CyberLynk?

ANSWER: Every provider has down time. Equipment fails, human mistakes happen, redundancy isn’t always redundant and the list goes on. Companies that refer to 100% uptime are usually only monitoring a specific device or they are using it as a marketing ploy and will give you a service credit for every minute of downtime each month. It is possible to achieve 100% uptime in a given month and CyberLynk regularly achieves that goal. CyberLynk has servers and networking equipment in our Milwaukee Datacenter that have been up and running for well over a year without being rebooted.

Below are some examples of other larger well known hosting companies that have had outages recently:

Bluehost, HostGator Among EIG Brands Hit by Massive New Year’s Eve Outage

HostGator Continues to Restore Hosting Services After Data Center Outage

Microsoft blames overheating datacenter for 16-hour Outlook outage

WP Engine System Status Page (Niche WordPress Hosting Company)

Note: The links above are to simply show that down time affects all hosting and datacenter companies both big and small.

QUESTION: Does CyberLynk provide a website or portal to receive information in the event of a network issue?

ANSWER: We encourage all of our customers to follow us on Facebook and/or Twitter for outage/maintenance updates, promotions and general company information:

Follow us on Twitter: https://twitter.com/cyberlynk
Follow us on Facebook: https://www.facebook.com/cyberlynknetwork