Security incident update
Yesterday, a group named HTP claimed responsibility for accessing Linode Manager web servers, we believe by exploiting a previously unknown zero-day vulnerability in Adobe’s ColdFusion application server. The vulnerabilities have only recently been addressed in Adobe’s APSB13-10 hotfix (CVE-2013-1387 and CVE-2013-1388) which was released less than a week ago.
As a result of the vulnerability, this group gained access to a web server, parts of our source code, and ultimately, our database. We have been working around the clock since discovering this vulnerability. Our investigation reveals that this group did not have access to any other component of the Linode infrastructure, including access to the host machines or any other server or service that runs our infrastructure.
Credit card numbers in our database are stored in encrypted format, using public and private key encryption. The private key is itself encrypted with passphrase encryption and the complex passphrase is not stored electronically. Along with the encrypted credit card, the last four digits are stored in clear text to assist in lookups and for display on things like your Account tab and payment receipt emails. We have no evidence decrypted credit card numbers were obtained.
Linode Manager user passwords are not stored in our database, but their salted and cryptographically hashed representations are. Despite the uselessness of these hashes, as you know we expired Linode Manager passwords on Friday.
There were occurrences of Lish passwords in clear text in our database. We have corrected this issue and have invalidated all affected Lish passwords effective immediately. If you need access to the Lish console, you can reset a new Lish password under the Remote Access sub-tab of your Linode.
For users who have set an API key, we’re also taking action to expire those keys. We’ll be emailing API-enabled users with that information.
We take your trust and confidence in us very seriously, and we truly apologize for the inconvenience that these individuals caused. Our entire team has been affected by this, leaving all of us, like you, feeling violated. We care deeply about the integrity of Linode and are proud of the work that we accomplish here for you. This unfortunate incident has only strengthened our commitment to you, our customer.
Please feel free to contact customer service via our ticket system or support@linode.com if you have any questions or concerns.
Filed under: announcements by caker
A satisfactory response.
Wonder why Linode does not just completely store CC info with a 3rd party and charge with a token. PCI compliance is such a pain when you are storing CC info on your own servers.
These things happen. Keep up the good work and keep shoring up your security!
Thanks for the update, and good luck with the remaining cleanup work.
I’m less concerned about the specifics of this actual breach than I am about the security practices which turned something less serious (a web server zero-day) into one that was connected to information for both payment data, and plaintext passwords for server access.
Can we expect to hear how your security policies and audit procedures will ensure this doesn’t happen again?
Also, would you consider handling payments through a third-party processor so that Linode is only sending and receiving transaction tokens, instead of storing sensitive financial information on servers accessible through your web frontend?
Thanks for keeping your users informed. As usual great customer service.
Knowing that username, password, api key and credit cards were accessed by hackers (encripted or not) makes anyone nervous and paranoid.
Although, knowing that you’re doing your best to ensure our safety thru legal and technical means, just reasures my love for your services.
Whatever you write, people will make more increasingly technical moans in the comments here.
Thanks for the update. I went ahead and revoked my credit card as a security measure. I use a randomly generated password for all my sites so not worried about that. Please let us know in the future if you have made additional changes to prevent this from happening again.
Not to continue to moan too technically or anything, but the question about Lish passwords hasn’t been addressed at all.
– Do you know which accounts had plaintext Lish passwords in the DB?
– Do you intend to let those people know?
So you straight up lied to us in the email you sent:
LIE # 1: ” We have found no evidence that any Linode data of any other customer was accessed.” — BLATANTLY FALSE.
LIE # 2: ” In addition, we have found no evidence that payment information of any customer was accessed.” — BLATANTLY FALSE
Even if you did know your statements were untrue when you sent the email, where is the follow-up email retracting these statements?
This is unacceptable.
I don’t know that I have a choice but to start looking for another provider — NOT because of the compromise, but because of how it has been handled thus far in a dishonest manner.
@Skavoovie they didn’t lie with this though. At the time the email was sent they had no evidence. That changed since the email. Hence the followup with this blog post and alert.
Based on my interpretation of both the e-mail and the blog posts, “any Linode data of any other customer” would be the data on each individual VPS, not Linode’s customer data. One host was supposedly compromised, but not others. This appears separate from the Linode Manager compromise.
As for “payment information”… well, that’s a bit harder to debate. If the e-mail does indeed refer to a separate compromise (of an individual VPS), no “payment information” may have been lost in that breach. Of course, the loss of credit card data, even if encrypted, would definitely count as “payment information” in my book, although I’m sure someone could get pedantic on semantics. If we’re not talking about two separate breaches… well…
And a quick reminder to those getting heated and calling for answers: If law enforcement is already involved, Linode may not be ALLOWED to be completely transparent, at least until any criminal investigation is complete. While I want the same answers you do, bear in mind that once the police and/or lawyers get involved, companies have to be very careful in how they respond publicly.
As for me, yeah, I’ve killed my CC on file. First time I’ve ever had to do that.
But for now I have no plans to move anywhere else. Overall, I think Linode has handled this well and has been much more transparent than most other companies would be. That said, how much of that confidence will remain will depend heavily on what comes out of this in the next days and weeks.
You guys at Linode have offered such an exemplary service over the years that I was definitely waiting to hear your side of things before forming any opinions of this situation. And I’m satisfied with your response. It’s unfortunate that such things occur, but I think we can all agree that shaking our fists at Adobe would be better served. Just make sure to place all Mountain Dews, Dr. Peppers, energy drinks, and similar carbonated beverages into your opposite hand first.
Don’t put your trust re: your credit card number in Linode (or any other vendor you use it with); put it in your bank or (better, at least in the US) credit union, which is on the hook for fraudulent charges.
You are in *much* more danger of identity theft from a restaurant employee (again, in the US) who takes your card away from the table to run your tab. (Not a knock against restaurant employees, just stating a fact; many/most? places now in Europe for example use wireless readers that they bring to the table, so your card never leaves your sight, but who’s to say where the reader is sending the data…)
Honestly, you ID theft virgins… It only hurts the first (few) time(s).
As a long-time Linode user I applaud their integrity and efforts on *our* behalf. As previous posters have said: (a) zero-day vulnerability – hard to defend against; (b) they are only human, just like the rest of us; (c) I’m 100% confident they will learn from this experience and take steps to insure appropriate improvements are made.
Linode rules!
—
drsteve
Those of you who don’t understand public key cryptography, can you please take a break on the rants and go read what it is and how it works before you continue going on about how it makes no sense. It’s actually a great design for PCI compliant credit card storage, allowing the cards to be encrypted and tokenized by the website without the website keeping any useful information to decrypt the database. This is far superior to people who just use symmetrical encryption on the card number and attempt to control access to the key with ACLs, passwords, etc.
Based on their more detailed explanation of the storage of payment information, I am confident moving forward with my card on file at Linode. Considering the fact that most cards don’t hold you liable for fraud charges, and the significant number of cash cards and online use cards out there, we all have plenty of options to keep our money secure. I just like to know that they put the forethought into using PKI to store the cards, and it paid off when they got hit with this 0-day.
As far as using ColdFusion, I don’t care what web platform they employ, there will always be 0-days. That’s why a good architecture is the real defense, which again paid off for them here. The only thing I found disturbing was the LISH passwords in plaintext, which they say they’ve addressed.
With that my confidence in Linode is restored, stuff happens, life moves on. Now if we can get our memory upgrades in Fremont that would be great! 😉
Like quite a few other people here I’m quite confused… is this a second incident that was unrelated? That doesn’t seem to be the case because this message refers to the forced password reset last week.
The message has a lot of grey areas as pointed out by others… we need clarity. It does no good to know that the database or CC information was encrypted if the key was also lost, or if the key didn’t follow best practices.
In this day and age security is just as important if not more so than host/network reliability and availability.
Thanks for the update. An unfortunate incident, but we appreciate the additional detail about what happened.
Our main concern is the security of the access to the Linode Manager – we don’t want to wake up one morning to find our Linodes deleted. Please consider implementing some form of two-factor authentication in the future.
Now that this unfortunately happened.
One of these two things, preferably both should properly be implemented.
1) Two factor authentication. We’ve been asking for awhile :/
2) Another PayPal method, EX PayPal. Storing CC’s is a dangerous thing these days.
So I asked about a lot of these questions a while back. Mainly, I’m curious how far the systems were actually penetrated and whether or not the fact that I could purchase upgrades and get them automagically charged while my card numbers were somehow protected by a password only Linode knew but the panel figured out by means of magic. I was told that it was under investigation, so I ask again, more than a week later. What is going on?
Also linode, Cold Fusion? WTF. You’re about a decade or so behind the times. Maybe even two.
@Sorressean: That’s a bit harsh. CF is still a supported product (and it wasn’t always Adobe ;). While other mistakes may have been made, IMO not moving away from CF was not one of them – there is no web framework that has never had a security flaw. (Of course the open/closed source debate is another matter).
I’ve just reviewed all material regarding this breach. And I must say it does make me feel very uncomfortable. I don’t know what to trust anymore, if we are going to trust what the ryan(hacker) says from chat. Everything is compromised. Credit card data is decrypted (from chat log: ” They did try to encrypt them, but using public key encryption doesn’t work if you have the public and private key in the same directory”) and they can also access the content of your virtual machine (for example if you had your mail or private code repositories configured). Its totally beyond me that Linode staff could configure their system so badly, when even I could do it better?!! And my skill level is very low… Actually it seems the system was so poorly configured that it even makes me suspicious that the story of the hacker is fake. Then again something definitely happened :-/
If nothing else I would really like only one thing from Linode.
Please say to us:
Do we have to invalidate our credit cards?
It is not a problem to do it. It can become a problem if we don’t do it but should! Please Linode I hope you are not giving us a sense of false security!
Next time this happens, please be consistent about your method of communication. I received an email on the 12th stating that nothing was f*cked, then never got a notification about the update / blog post. I was under the impression nothing was compromised, but am currently rebuilding all servers due to new information that wasn’t received in the same manner.
I do appreciate the transparency, but attention to detail and consistentcy is very important in these situations.
I am dying to use linode.com from years, I do have credit card. But I use my credit card on internet only via paypal. I can not give CVV numbers etc to each and every site on number I purchase from; if in years to come they get hacked.
Its far better for me, add another virtualization layer upon my creditg card by using paypal.
I am just unable to understand why on earth linode.com does not accept paypal or netbanking from India etc. It will vastly expand their user base in an instant.
A real worry for anyone doing online trading.
Basic one would be keep anything away from billing that does not need to be there.
Eg install the machines according to NSA/CSS standards.
http://www.nsa.gov/ia/mitigation_guidance/security_configuration_guides/operating_systems.shtml
Or at least use that as a guide point to improve your companies own security.
If in doubt chuck it out.
In most cases if it’s not there they can’t steal it.
Even if it means re inventing the wheel slightly keep recurring billing separate and offline local access only.
What ever it takes it’s worth doing to keep customers safe.
Authentication servers is another interesting subject.
Not a huge issue in small scale operations but you can see with programs like wireshark and even your logs what ports the smash and grabbers are going for.
I was thinking SELinux may help here (as if I know what I’m talking about), perhaps the database can be running on another (ahem) linode so that the data is isolated.
You guys are awesome. EVERYONE gets broken into but not everyone reports it, you guys are on the short list!!
Please keep up the vigilance and innovation.
It seems a lot of people are missing the other side of this coin here.
Are there a lot of unanswered questions? Yes. Are you going to get answers to most of them? Probably not.
Answering a lot of the questions in these comments would involve divulging a lot of how their infrastructure is designed, and what security measures are in place; not a very smart thing to just drop into public-facing comments. Security breaches are embarrassing to everyone – most companies reporting breaches where caught not even using salt / hash, so props to them.
Like i saw in another comment here – give them time to get all their ducks in a row. In the meantime, do the sensible thing – assume all data is compromised, and change your passwords / encryption keys, and pay attention to your bank account.