Security Investigation Retrospective

February 19, 2016 3:30 pm

On January 5, 2016, we issued a password reset for all Linode customers during our investigation into the unauthorized access of three customer accounts. We have been working with federal authorities on these matters and their criminal investigations are ongoing. Today we are sharing our findings and those of the third-party security firm we retained to assist us with this investigation.

Before diving in we’d like to reassure you that your account information is safe. We found only three customers affected by this incident and have resolved these issues with them directly.

What Happened

This is a complex retrospective of two separate investigations, one in July and another in December. While the cases share similarities, we have no evidence to support the two being related. Nevertheless, here is a full timeline of the events as we have come to understand them.

On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notification confirming a root password reset for one of their Linodes. Our initial investigation showed the unauthorized login was successful on the first attempt and resembled normal activity.

On July 12, in anticipation of law enforcement’s involvement, the customer followed up with a preservation request for a Linode corresponding to an IP address believed to be involved in the unauthorized access. We honored the request and asked the customer to provide us with any additional evidence (e.g., log files) that would support the Linode being the source of malicious activity. Neither the customer nor law enforcement followed up and, because we do not examine customer data without probable cause, we did not analyze the preserved image.

On the same day, the customer reported that the user whose account was accessed had lost a mobile device several weeks earlier containing the 2FA credentials required to access the account, and explained that the owner attempted to remotely wipe the device some time later. In addition, this user employed a weak password. In light of this information, and with no evidence to support that the credentials were obtained from Linode, we did not investigate further.

On December 9 an independent security researcher contacted us. The researcher claimed to be tracking an individual who had stolen credentials from numerous other service providers. The researcher wanted to make us aware that the individual may have made attempts to use these stolen credentials to log in to some of our customers’ accounts.

Our initial investigation concluded that the IPs provided had, in fact, been used to log into three accounts on the first attempt. In other words, the user arrived to the Linode Manager login page with the credentials necessary to log in, just as any regular user would. That same day we contacted the customers and received confirmation from each that the activities were suspicious. We also confirmed that none of these accounts had multi-factor authentication enabled and all had employed weak passwords.

On December 13 we started necessary fleet-wide Xen Security Advisory (XSA) maintenance, rebooting servers in their local nighttime hours around the clock. Although unrelated to the investigation, this continued through December 18 and was a significant resource constraint.

On December 14, although we had discovered no evidence of an intrusion into our infrastructure, we began interviewing third-party security firms and contacted multiple law enforcement agencies. We also dedicated all available internal resources to the effort and began scrutinizing our environment to identify any evidence of abuse or misuse.

On December 17, because of the similarities between this case and the one from July, we reopened the July case and concluded we now had sufficient reason to examine the image retained from that investigation.

Linode uses TOTP to provide two-factor authentication. This is an algorithm that uses a secret key stored on, and shared between, our server and the customer’s two-factor authentication app (such as Google Authenticator). The algorithm generates a time-sensitive code that the user provides during login as an additional authentication component. We encrypt these secret keys when storing them in our database.

After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code. Though the credentials found were unrelated to any of the unauthorized Linode Manager logins made in December, the discovery of this information significantly changed the seriousness of our investigation.

On December 21 our third-party security partner joined the investigation. This team proceeded with a forensic analysis to identify any unauthorized system-level activity that may have permitted an intruder to access customer credentials contained in our database. The team also searched for evidence of web application misuse that would have provided lateral movement from one Linode Manager account to another. Additionally, the team initiated a targeted vulnerability assessment with the objective of identifying a possible attack vector for obtaining access to the Linode database.

On December 25 DDoS attacks against our infrastructure began. Although we do not have any evidence to support these attacks being related to the incidents of unauthorized access, the attacks necessitated resources be pulled away from our investigation. This, combined with employees being away for the holidays, created additional challenges for our support and operations teams.

On January 5 our security partner concluded its investigation and we issued the password reset. Our internal security team continued to review our infrastructure for the next several weeks and developed a detailed plan for improving our overall security.

Findings

The findings of our security partner’s investigation concluded there was no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. Furthermore, the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access.

Linode’s security team did discover a vulnerability in Lish’s SSH gateway that potentially could have been used to obtain information discovered on December 17, although we have no evidence to support this supposition. We immediately fixed the vulnerability.

Other theories considered that could explain the unauthorized access include an external compromise, such as the weak passwords previously mentioned being used with other online services, or phishing attacks against those users.

What We’re Doing About it

We are using what we’ve learned to make comprehensive improvements to our infrastructure, including areas unrelated to the incident. Here are a few of the things we’ve been working toward:

Authentication microservice: Several of our applications (such as the Linode Manager and Lish) perform user authentication. Previously, these applications performed this function by having access to the credential information directly within our database, and then performing comparisons themselves. We’ve developed a new approach that involves a carefully secured and monitored microservice that maintains ownership of all customer credentials. Under this method, when an application requires user authentication, the microservice is able to validate the credentials by returning a simple “yes” or “no.” Applications won’t have access to the credential information. In fact, the databases that power our infrastructure won’t contain credential information at all when the rollout of this microservice is complete. Also, customer passwords, previously stored as salted SHA-256 hashes with thousands of rounds, will be stored using bcrypt and will be upgraded seamlessly on a subsequent login.

Linode Manager notifications: We will be working to enhance notifications our customers receive about their respective account activity, including alerts to login attempts from new IP addresses and failed logins.

CC Tokenization: Although our investigation yielded no evidence of credit card information being accessed, we are taking advantage of our payment processor’s tokenization feature to remove the risk associated with storing credit card information.

Policy: We’ve been developing multiple policies derived from the NIST framework on topics ranging from clean-desk to password standards. A significant new policy is the creation of “security zones” for sensitive elements of infrastructure, like our database and authentication servers. The results of those efforts have greatly reduced the number of employees that have access to sensitive systems and data.

Hiring: In addition to the changes outlined above, we are hiring a senior-level security expert to join our company and lead a larger team of engineers focused full-time on security. This team will not only ensure we are following current best practices but will also expand our written policies, formalize our provisioning procedures and fundamentally ensure our policies are supported by process and accountability.

A New Linode API: Our most important long-term strategy is a rewrite of our legacy ColdFusion codebase that gives us the opportunity for a fresh start and to apply the lessons we have learned over the past 13 years. To do this, we’ve been building a new Linode API that’s stateless, RESTful, and implemented in Python. We’ve been working on this for the past many months and will announce a public alpha of the new API in the next few weeks.

Open-source Linode Manager: This new API will be the foundation for all things coming in the future, including an open-source Linode Manager which will replace the current manager.

Looking Ahead

We recognize we have room for improvement when it comes to communication and transparency. XSAs and persistent DDoS attacks throughout December notwithstanding, we should have communicated the nature and extent of the DDoS attacks and this security incident to our customers sooner. To say we were resource constrained at the time would be a fair assessment. Still, we could have done better and have since made procedural changes to ensure a team member is appointed to important events like these in the future to facilitate frequent and transparent communication with our customers.

We are incredibly grateful for the customers who have supported us throughout these events. We’ve heard your recommendations and felt the support you’ve provided over the past few months. Know that we continue listening and acting on your feedback.

We’ll conclude by saying how sorry we are if we’ve let you down. We value the trust you’ve placed in us as your hosting provider and are committed to earning that trust every day. We hope the details provided here clear up some misinformation and demonstrate our willingness to address opportunities for improvement, do the right thing, and increase communication and transparency with you, our customers.

17 Responses

  1. *”…we are incredibly grateful for the customers who have supported us throughout these events… [we] felt the support you’ve provided over the past few months… We value the trust you’ve placed in us…”*

    Nice sentiments, but words are cheap. You could have shown your appreciation by giving people some money off their bill for the period in question

  2. If there is to be a new Linode Manager, please please PLEASE do not try and “modernize” or “streamline” the interface like Namecheap did. I like my utilitarian and functional Manager.

  3. “We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code.”

    How do you explain the presence of software using the decryption method you use to secure TOTP keys, along with the secret key you use to encrypt them?

  4. There are always some bumps on the road, sometimes these are big and hard to pass thru, but I think that you’re on the right track.

    Keep up the good work 🙂

  5. The most significant take away for me is that Linode is becoming more transparent. Please continue down this path.

  6. Good news for the new Linode manager, is there any ETA?

  7. Long-term customer here. I appreciate all the effort made during the difficult time over the holidays. However, I have asked for 2FA by SMS rather than google authenticator (or in addition to), and was told it will not be happening. It seems that this breach could have been avoided at least in part if 2FA by SMS had been used. The user who lost his phone would have moved his phone number to his new phone as soon as it was replaced. The entire compromise of the TOTP algorithm would not have been possible. So my request, and recommendation still stands; give us 2FA by SMS if we want it.

  8. I know how difficult these situations can be and wish you all the best in your continued growth and improvements. Unfortunately, I’ll be switching to another provider after reading this. I stuck around waiting for an explanation that I hoped would satisfy my concerns and this certainly doesn’t do that… There is reasonable evidence that your story regarding the cell phone is incorrect. In my opinion you should have had a senior security director years ago. Given the nature of the July incident, that image going unexamined is mind boggling and no explanation is given about how the 2FA crypto keys could have ended up on that system. It sounds entirely plausible your infrastructure got breached in July or earlier by a skilled attacker and the evidence is simply not there months later.

    It sounds like you’re moving in the right direction, but those are some seriously poor decisions by those in charge and I’ve lost confidence that further mistakes won’t be made.

  9. I like the current Linode manager. It’s simple and fast and clean. Please don’t change it too much. (Don’t go all Namecheap on us)

  10. I’m a longtime Linode customer as well. After reading up on this latest incident I am seriously considering moving my infrastructure to another provider. I’ve been through a number of these cases with Linode and so far have been unaffected and given them the benefit of the doubt. I completely understand that threats are uncovered and exploited and that’s a risk you take with any provider. This isn’t about me taking a knee jerk reaction. I’m fully aware that other providers have been exploited as well but Linode have had multiple exploits and that is reason for concern.

    In this post you state that you found software on a client’s VPS generating login credentials using a very private piece of your data and a piece of data that can be used to decrypt other customers data and you have no idea how it arrived on a clients machine and you don’t really seem bothered by that or at the very least you gloss over it. Have you audited every other instance to ensure they’re not also capable of generating keys/decrypting data? From my knowledge of this, the client in question was PagerDuty who alerted you after their own intrusion detection system picked up on the login. By your own account the initial illegal access looked like a legitimate login to your systems and didn’t raise any alarms. You also state that you don’t inspect customers instances without probable cause. So we have to assume that there might be other malicious software running in your infrastructure that you are not aware of and it’s allowing access that isn’t raising alerts.

    You say that only three customers were accessed under suspicious circumstances. On that face of it that looks like a small number, considering you have a large client base. However, from what I can gather these were large and notable customers PagerDuty and WPEngine, I believe. That, to me sounds less like a percentage risk and more like plucking high value targets. I’m only a small customer so it’s probably more out of luck that I haven’t been affected rather than my details not actually being available.

    Linode has offered a product that’s very valuable to me and my business. It’s provided reliable hosting and features. I chose them because I felt they would be able to support any application that I need to grow without the fuss of moving providers. I no longer have that confidence. The previous attack against Linode resulted in credit card loss and it’s only now that you are looking at tokenizing credit cards and moving to bcrypt for hashing?

    I do appreciate that resources get stretched and trying to pick out relevant data and act on it is really difficult in a growing business. But the lack of security awareness at Linode has got to a point not where I feel I can’t trust it with my business which is sad because my experience with them has been positive, well if you put aside data breaches, credit card loss and their private keys ending up on clients’ VPSs and inability to communicate.

  11. “We also found commands in the bash history that successfully generated a one-time code”

    I don’t get how you saw this and the investigation concluded that “the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access.”

    I mean, I don’t understand why someone have a bash on your systems, how he achieved that and what changed so he can’t do it in the future

  12. Just FYI, I’ve been forced to spend considerable ongoing time and money investigating other services and finally choosing a viable non-Linode. This is money that could have been spent on better things, even if it had just been more Linode servers. Outside 3-4 work weeks redirected, my computing budget doubled even after whittling down my 4 Linodes to two.

    You might consider joining forces with a cooperative and reputable industry counterpart in an effort to provide more viable business recovery scenarios for your mutual customers. In my case, now I have both a dedicated server vendor and Linode, a solution that is not neither pleasant nor economical. Each has complementary benefits.

    I hope that your security efforts prove successful and I concur with the comments that should you change the inside workings of the Linode manager, it should remain simple and shun any feature that is only cosmetic.

  13. I agree with the poster above that Linode appears to be going down a path of greater transparency. I’m extremely happy to see this.

    I also believe that full disclosure and admitting fault where necessary is a very honorable thing, and you are to be commended for what you’ve said here.

  14. Odd, all this transparency is making some people think there is a lack of security knowledge at Linode. Guys, you’d have this or worse at practically any other company.

    Sounds like almost this entire thing boiled down to a few customers who have no clue about security to begin with. Weak passwords and a mobile device that doesn’t have a screen code to lock the phone. Sigh, sorry Linode had to go through all of that because of a few lazy people, but it does look like they’ve learned a lot and are improving and moving forward.

    Kneejerk reacting will only put you with someone else, that probably isn’t doing the same thing as Linode.

    Keep up the great works guys and I’m glad to see the improvements.

  15. Well done, Linode!

  16. If you are going to update the Linode Manager API, *please, please, please* give us the ability to do programmatic snapshots.

    Thank you.

  17. I greatly appreciate the explanation and transparency. So very different than other companies. This is the kind of thing that creates customer loyalty.

Leave a Reply