Security Notification and Linode Manager Password Reset

January 5, 2016 1:53 pm

Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution.

A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.

This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data.

The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings.

You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be.

The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.

We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode.

84 Responses

  1. To clarify, when you say “user credentials on an external machine”, are we talking about hashed passwords? or plaintext passwords? And is the external machine a system operated by Linode, that was not supposed to have these credentials on it, or a system operated by a 3rd party?

  2. Does this issue impact API keys?

  3. “The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service.”

    I understand the goal of this is to convey your sincerity and attempt to connect with the audience, but it’s starting to ring hollow. This announcement is light on details, and this paragraph makes it feel even more like hand-waving.

    I understand you may not be ready to provide a full breakdown at this moment, and I look forward with optimism that such a breakdown will be provided in the future, but until then, I think it may be more appropriate to trim some of the PR spin and get back to the frank, up-front communication that got people interested in Linode in the first place. The people reading this page are smart, they’re technical, and they know PR spin when they see it: remind them that the folks at Linode are smart and technical as well.

  4. Is the vulnerability the attacker used to read the DB fixed now? Or do you know how he got the credentials for your database?

  5. @Les Aker “The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds.”

    It looks like they are not plain text passwords but rather properly salted/hashed ones as would be expected.

    We’re patiently riding this out. Luckily our linodes were not in Atlanta so our downtime has been quite minimal so far. (all things considered)

  6. I do not work for Linode — but Linode has been a tremendous benefit to our business, clients and colleagues and we hope for a speedy recovery and we are supportive. That said — most likely this is now in the hands of the FBI and other agencies, it is not a good idea to comment in detail during active investigations and especially security related ones. I know this very well and hope that the perpetrators are caught and hope that developed countries do their duty to finally consider this activity criminal and lately an act of economic warfare.

    If you want an idea of issues going on right now there are massive NTP amplification attacks coming from China and it is still ongoing right as I type this — I do not know if these are related to the Linode issues, but it is awfully suspicious.

    If you want an idea of how bad it is — research tools that visualize attacks in progress, there are several and also subscribe to notifications from respected security firms and/or stay in touch with respected individuals that work in security with large defense firms.

  7. I believe Les is right to ask for clarification. I’m sure this hasn’t been written in a deliberately obscure way but it certainly reads that way.

  8. I just reset my password, but now I can not log in again (at manager.linode.com). The websites on our VM seem to be functioning. Our DNS entries seem to be okay at this point too. And blog.linode.com seems to work, but not linode.com nor manager.linode.com.

  9. Okay, my previous post was written @ 8:30PM; I hadn’t saved it. But now I just checked and manager.linode.com seems to be okay again.

  10. Would it be possible to share timeline details that lead up to today’s password + 2FA credentials expiry? When did the unauthorized login of three accounts happen, when was it detected, and when were the two Linode.com user credentials on an external machine found?

  11. There are accusations on Twitter that you sat on this attack vector for several months. Can you please elaborate?

    Tim Heckman ‏@theckman 9 hours ago
    It took @linode until now to disclose. I told them this was the vector for the @pagerduty attack back in July.

  12. Linode password have salt key? If not, I have to reset all of my password.

  13. Thanks for the update and the details you’ve provided. Looking forward to hearing more as investigation and everything continues.

  14. Here is an article about recent attacks I just read :

    https://securityintelligence.com/news/ddos-attacks-storm-linode-servers-worldwide/

    I was wondering why service was so slow. Now I know.

  15. It’s obvious that they have no clue how the DB got breached… Resetting the passwords makes sense only if they found and plugged the hole, which they haven’t!

  16. @Dave L I think Les was asking about the credentials they found, not the database they’re stored in at Linode. I’m curious too since the description indicated that the original complaint from the customer that triggered the investigation involved an ex-employee attempting to access their own employment-related account. Finding encrypted/hashed credentials would be concerning, but finding cleartext passwords and 2FA secrets for a user’s account on their own machines is unsurprising (at least to me).

    I’m not too worried, Linode isn’t downplaying the significance and the forced reset should make any compromised credentials useless for active accounts (inactive accounts are another matter, but they can and should simply be deleted).

  17. What about credit card details? Are they safe?

  18. What about credit card details? Are they safe?

  19. Where was my email notification about this? Happy customer but not pleased that I have to read about this on a tech blog somewhere else to learn that I need to reset my password.

  20. If the passwords might have been compromised, wouldn’t it had made more sense to immediately invalidate all passwords and send us all an email each with a secure link to reset it from there?

    That way, you know, people who’ve obtained some passwords illegally won’t have the chance to change it to something else themselves, before the legitimate users do.

    BTW, I was looking through the linode manager and found Account > Users and Permissions > edit. Am I correct to assume that through that form a user can change their email address? I was thinking that it wouldn’t matter “as much” if someone changed my password as I could regain access by correctly resetting the password myself, but I guess that might not work if they have the ability to change my account’s associated email address, too.

  21. Hi Linode. You have my utmost sympathy and support for everything you have endured and continue to endure. I have now changed my password and two factor seed. I do have one suggestion. I found out about the above only by attempting to login by chance after months of inactivity. Wouldn’t it be better to email this post out so everyone can proactively change their passwords? Keep up the great Linode ethos and stay strong. All the best.

  22. With concerns about this, I believe I remember similar issues happening before. Totally would like more specifics about what has occurred, and I’m sure a full RCA will come in time after the initial based investigation. I would expect a full time-line and fact sheet based on this incident.

  23. Yet another security breach at Linode that could have been prevented with 2FA. We’ve been asking for this FOR YEARS. Total lack of basic security measures makes this no surprise. Linode cannot be trusted for serious applications… only hobbies.

  24. I’ve been in the same position with a compromise that I was responsible for… it’s very uncomfortable and I don’t envy your position.

    Linode has always provided me with a great value and top notch service and support. I plan to continue using Linode for a long time. Thanks for your transparency in this matter.

  25. I find it difficult to blame the attackers or Linode for the security situation. Everything we’ve built up is insecure, poorly understood, etc. Developers aren’t expected to implement good security practices. The implementers are also no less responsible to blame. I’m sure Linode could dump a significant amount of money into projects that are trying to fix some of these issues. Unfortunately I’ve not been terribly happy with Linode over the years in a number of non-service related areas like this. They don’t seem to understand or care enough about there reputation/image/security despite otherwise offering a quite terrific service.

    I’d like to see Linode offer a implement a completely libre software stack and start funding work to free hardware (removing proprietary code for which we can’t even begin to audit). Without taking such steps companies such as Intel and a variety of manufacturers in the chain and/or the companies they license code from (BIOS, Western Digital, etc) will make us all vulnerable.

    I’d also like to see them work with the Tor project, Tails, and similar privacy minded organizations.

    Lastly Linode’s PR guys really need to get rid of the Macs they’re using at conferences! Every time I bump into Linode’s reps at a conference all I can think is Really? Really? If your targeting a technical audience with a GNU/Linux/FOSS-based product do you really think it’s a good idea to show it off on a Mac/Apple/Microsoft product? You might get away with that at a corporate event which technical types don’t actually attend, but the GNU/Linux/FOSS/server-oriented events?

  26. Hi
    I have been trying to reset my password for the past week. Send 2 Emails , you wanted from me the last 6 digits of my credit card which I replayed back as you requested .
    Till now I heard nothing from you useful how to do that.
    Thank you
    Eyad

  27. Emails have been rolling out in batches today.

  28. Emails have been rolling out in batches today.

  29. @Linodecustomer. Linode does offer two factor authentication.

  30. How “securely” are the passwords hashed? That is, what hashing algorithm are you using?

    Given the power of GPU’s, i’m not at all confident that Linode’s “secure” hash is secure unless they specifically tell us what hash it is and what work factor.

  31. If we used 2-factor auth for the Linode management site and key-based ssh authentication, is it (even theoretically) possible for a malicious user to have gained access to our node? Presumably Lish/console/backdoor access requires management app access, which hopefully would be stopped by 2-factor requirement. I suppose the only safe answer is “anything is possible” but I’d like to think 2-factor ruled out access even if they obtained my Linode site and VM user passwords.

  32. @Chris > Everything we’ve built up is insecure, poorly understood, etc.

    I sure hope you’re use of “we” here is a royal inclusion of all “developers” and not that you are some rogue Linode employee complaining about their insecure practices… 🙁

  33. I’m sure the DDoS was simply a diversion to mask the true nefarious activity of breaching data, just like what happened with Steam and BBC over the same time period. We’re bound to find out a great deal more about additional data compromise. The DDoS diversion tactic is becoming fairly standard.

  34. I believe you need to get your facts straight @linodecustomer before you post false accusations like that. I have been using 2FA on Linode for 3 – 4 years now.

  35. Most of these comments are hilariously stupid. Go neckbeards!

  36. I’ve already got two factor authentication. Why are you resetting my password?

  37. We’re all vulnerable to attack. I think you guys are doing a great job! Thank you for being open about security and taking care of business. I wish you guys the best this year!

  38. @Jim. Where do I enable 2FA? Can’t find it or an announcement about it.

  39. @Jim I would also wonder how two factor authentication would have prevented a customer’s username from appearing in a public place. It would, however, likely prevent them from using the username to any great effect.

  40. Hi Linode,
    thanks for info. I like your service so let me know if you need a help with this issue. I happen to work for a firm, which replaces passwords with user-centric private key based identity/authentication. Our solution completely prevents all hacks on ID databases, it’s immune to phishing, MITM and it has instant breach detection among others. As long as you run your security based on “shared secrets”, you will always be vulnerable… I am sorry if this sounds like a sale, the purpose is to help you though…

  41. Dammit Linode! I have been a happy customer so far and I have recommended you very often, but these things make it harder for me to stick with you. I truly hope this does not happen again and you strengthen your security A LOT, otherwise I will cancel my account try an alternative.

  42. @Chris:
    What a stupid comment and FUD about Macs. Have you ever seen Google employees? I would say 80-90% use Macs for their desktops and their core products are all running on Linux. And I do the same…. I use Linux every day on my job on my server but I use a Mac (with their sort of BSD) for my desktop which has the better desktop experience for me.

  43. You are addressing technical customers here. Stop with the press release format.

    Your mention of our “securely hashed passwords” is laughable. There are developers who think running md5 on the plaintext is secure. Is this the type of “securely hashed passwords” you are referring to? Or are you doing only slightly better by salting an MD or SHA hash? Or perhaps you’re actually using bcrypt or similar? Let us know so we can decide for ourselves how likely it is that an attacker was able to brute force our plaintexts.

    This reset procedure only guarantees my account’s safety moving forward. Since I was prompted to reset my password on first login, obviously this means an attacker did not perform this reset themselves.

    Yet how am I supposed to be confident that someone did not access my account prior to your initiation of the reset protocol? You do not send an email or SMS when a login comes from a new IP address for the first time. I have no authentication log in the manager that shows a history of all logins to my account. You need to add these features, because right now every single one of your customers has no idea whether their account has been accessed by a 3rd party. You cannot be certain of the criteria for where the logins were coming from. You might have only found X accounts matching a specific unauthorized login pattern (ie: source IP), but the attackers may have used additional login patterns that you have not detected.

    *I* know where my logins should be coming from. Where are the features that notify me and let me track the logins to my account? It is not up to you to “detect” unauthorized access based on whatever pattern you believe the attackers used. I need to be able to review the login logs for my own account. This is a pretty basic damn feature, even my gmail account has it.

  44. Why was TLSv1.0 disabled? Neither the main website, http://www.linode.com, nor manager.linode.com or api.linode.com work anymore; it been a week now, and they don’t even respond to emails.

  45. With the information provided, there’s little reason to believe the database was actually breached. Assuming they are salted, brute forcing them is unrealistic unless they insanely simple passwords. In my mind, what’s much more likely is some folks w/linode accounts were compromised and a keystroke logger etc snagged their passwords as they were typed. Granted however, linode may know more about the manner in which the passwords were compromised, and are not publicizing it, but have valid reasons to trigger a global reset.

  46. My website was on Linode and was defaced a day before NYE, hacker gained access to my server and started defacing all my websites and I could not access Linode to restore backups for while, so I started migrating it. As I share the server with a couple other friends, I just assumed one of them may have done something stupid and and was ashamed to come forward.

    But now, having read this blog post now, I wonder if my site was hacked along with other Linode customers or if it was an isolated incident…..

    If Linode contact me, I can provide screenshots. The defacer had something in French, identified himself as S4N70S | ANON4AFRICA or something like that…..

  47. I’ve reset my password via https://manager.linode.com. Thanks for the announcement!

  48. It seems like there are a few people that have been unhappy for several reasons for quite some time. I am curious what would compell you to continue to choose Linode?

    No one should be happy about this attack, but if you really feel that you are getting terrible service from Linode it’s not like we are forced to use Linode. Unless you are employed/contracted by a Linode customer then you have no reason to stay.

    Personally, I am happy with the overall package. I do a lot of small things with Linode and maybe if my needs changed I wouldn’t be, but if that day comes I would start to move services to another company.

  49. It’s hilarious watching all of these armchair experts criticize Linode for the actions of another. Linode is doing everything and more that any company can be reasonably expected to do in a situation like this. From the details we’ve gotten so far, it appears that they’ve followed all of the standards plus some in terms of protecting us.

    These people clearly care deeply about all of our security, because our security is their security. We are their customers, we are the reason they get in the morning and, especially lately, we’re the reason that they’re sitting up all night at the office.

    At the end of the day, if someone big and bad and scummy enough tries, they’re going to be able to do some damage. No matter how much of a ninja you are, you’re going to take hits. That’s the facts. What you don’t do is run to a fellow victim and shake your fist at them, shaming them for taking a blow. We’re all being attacked, together. Absolutely nothing has come out that demonstrates that Linode deserves anything but praise.

  50. I’ve been trying to reset my password now for several hours and the email never turns up. Nothing in spam either – not sure what to do as I can’t log in at all now until I get that email.

  51. Appreciate the open honesty about this issue.

  52. Last time I changed my password here I was able to use all lowercase letters. Now I can’t. Please can you revert this restriction. We all know that being forced to use other characters is fake security and forcing a user to style their password in a certain way only encourages them to use a WEAKER password! Haven’t you read XKCD 936?

  53. I think some of the comments express a fair concern. It doesn’t seem well thought through and the post is not detailed enough.

    One interesting thing I noticed – although I changed my password as requested a few hours ago, the linode android application did not ask for the new password, it just kept working. it is probably using some token/sessionid and it’s valid for… how long? I hadnt started that app for days so that token must have been pretty old.

  54. Just received email about this. I still don’t know what happened but I’ll reset my password, thanks for the notification.

  55. @Nate : Linode sends me an email every time I log in from a different IP and won’t let me log in until that IP address is confirmed via a link in the email. It’s a standard feature in the Manager settings – look in My Profile -> Password & Authentication

  56. FYI Linode offers extra account protection :

    1. IP whitelisting
    2. 2-factor authentication

    https://www.linode.com/docs/security/linode-manager-security-controls

    Anybody who uses just a password to protect their Linode account, is asking for trouble. Instead of sharing your expert opinion on what may or may not have happened, harden your account and have a backup plan.

    Hacks can and WILL happen.

  57. I’d like to second TechnikEmpire’s comments and thank Linode’s staff for their work in mitigating these attacks in the last 2 weeks. I’m sure it has been very stressful.

    I also note that the post says the following: “You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.” To me, that indicates that they may suspect, as a first hypothesis, that these accounts were compromised by sloppy password handling on the part of a client, at least that’s how I took the post as a whole, but out of an abundance of caution, especially in heat of the DDoS attacks and having incomplete information, they decided to reset all passwords. To me, a less professional team would have decided to let it lay until more evidence came to light. Instead, Linode decided to take a potential public relations hit and prioritize the security of our accounts.

    I note also that the post says “two Linode.com user credentials on an external machine” were found. Why only two if the entire database has been hacked? Once I have access, it takes me a few seconds to pull down an entire database table of users. Either I have access to all the records, or I have access to none. Why only two? The post does not say they found their entire user table on an external machine.

    Still, a Linode engineer argued for caution and won over the consensus – we don’t know how those two records got there – let’s reset all passwords now. Seems quite responsible to me.

  58. As Nate points out, this is a press release, not a technical discussion. It says “The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds” As Josh points out even if the “securely hashed passwords” (including salt) were compromised there is little risk to the rest of us.

    There needs to be a good reason to force everyone to reset their passwords. This article does not give a good reason.

    If Linode has forced a password reset thrash for no good reason they have opened a serious security risk as anyone can send out a password reset links that point to a spoofed linode admin site. I should detect that spoof with my email client but there are no guarantees.

  59. “A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine”

    This appears to be yet another security breach of the Linode users database. What’s the expression about once bitten, twice shy?

  60. @Nate ” You do not send an email or SMS when a login comes from a new IP address for the first time.”

    Yes, you can enable that feature in the manager, go to “my profile” > Passwords & Authentication and at the bottom enable whitelisting of ip addresses. You’ll be sent an email when you or someone else is logging in from an unknown IP address.

    Also go to Account > Users & Permissions and you can see when an account was last logged into, but of course it’s not a full log, just the last login. And you can’t see where the login was from, but if you have multiple user accounts accessing your Linode account you can at least get some information on those other accounts.

    @Augusto, no your websites were not defaced because of this hack. Your websites were hacked because of the way your server is configured or the software you’re running on it or because of your friends.

  61. Nate: You can already set up an IP address whitelist and get notifications when anyone tries to log in from any IP address not on that whitelist. It’s in your profile under Passwords & Authentication. It also shows you what computers are set up to skip 2FA, if you’re using it. The only thing missing would be adding Linode Manager logins to the RSS feed and e-mail notifications.

    I’d imagine that if Linode’s password database really had been compromised we’d see someone crowing about it. Right now I’m thinking that this was a case of access by an ex-employee of a customer to their own account after they were terminated combined with the usual cases of people having backup copies of their credentials on their own machines (without encryption, naturally), and Linode’s just being properly paranoid about it. If there was a problem the resets will cut off unwanted access (or at least make it blindingly obvious someone else has access), if there wasn’t a problem they won’t hurt anything, and I remember the last time something like this happened and Linode’s response then kept it from being more than an annoyance for most people.

  62. Did you first log in with your current password? You should be prompted to reset right there, no email required. If that doesn’t work, you should email support@linode.com with the issue, as well as the account you are trying to reset.

  63. The app uses an API key after you first log in. We don’t have any reason to believe that you need to reset any API keys.

  64. Correct! Here’s a link to the documentation: http://krl.io/190c9

  65. On this blog post it says it has 64 responses / comments… but at the moment i only see 11.

    Is this a bug or problem caused by the recent ddos attacks? Or is Linode censuring and/or deleting responses / comments?

  66. It is a bug in the blog, and we’re working on figuring out what’s causing it.

    EDIT: Here’s a link to all the comments: https://blog.linode.com/2016/01/05/security-notification-and-linode-manager-password-reset/comment-page-1/

  67. We’d been our Atlanta box quite smacked some days ago.
    Now it works fine.
    Keep it up

  68. >> Bob January 6th, 2016 at 4:30 pm
    >> Most of these comments are hilariously stupid. Go neckbeards!

    Agree.

    Comments about how awful Linode are for not offering 2 factor auth. Ridiculous. Please tell me these people aren’t running public-facing servers, if they can’t even find a basic account option like 2FA in Linode Manager?

  69. Also I’d like to add that, as someone who does run a web server like I’m sure many people reading this do, if you have no understanding or sympathy for the fact that Linode have (a) suffered such attacks and (b) done the CORRECT thing to disclose this, then … you have no soul.

    Someone in the organisation urged a full disclosure, which is a tough thing to do for some organisations and a natural one for others. Either way it means the management is listening to the employees, or the organisation is driven by good ethical values.

    And either way, I’m happy to continue to do business with Linode… it’s not what happens, it’s how you deal with it.

  70. @Alex Fornuto

    > The app uses an API key after you first log in. We don’t have any reason to believe that you need to reset any API keys.

    If someone has my password they can get an API key and my resetting the password will not do anything to the API key they got. I need to go and revoke all API keys and create new ones.

    Or are you saying that although someone might have my password you are 100% sure that they have not used it (to get an API token or for anything else). How can you be sure?

  71. Manager.linode.com doesn’t seem to load for me. I just keep getting “The connection was reset”.

  72. @Les Aker: We did not expire API keys, and we don’t have any reason to believe you need to reset them.

    @Rodrigo: We discovered two Linode.com user credentials on an external machine but have no indication that anyone obtained or used them to access our database. No vulnerability was identified, but we chose to expire all customer passwords out of an abundance of caution.

    @Wolfram: The suspicious activity occurred within the past few weeks. When discovered, we immediately launched an investigation. We will share a detailed report in the near future once our investigation is complete.

    @Ryush00: While the hashes are salted, we advise you to reset your credentials anywhere you have used them. Moving forward you will want to ensure that you are using security best practices, such as not reusing passwords on various services.

    @Denny: At this time our investigation shows that only the users table may have been accessed. The users table does not contain credit card data.

    @TC: We began sending out emails within a few hours of our public announcement. The emails went out in batches so our customer support representatives would not be overwhelmed with questions all at once. All Linode customers have now been notified by email. A link to the announcement appeared on the password reset page, so anyone prompted to reset their password before receiving email notification would have seen the notice there.

    @Jorge: As soon as we concluded that a password reset was warranted, we immediately invalidated all passwords, posted the public notice and began sending the notification emails. We made the decision not to include a link to reset the password in the email so our customers would not mistake the notice for a phishing scam.

    @Kyhwana: Our information security policies do not permit us to disclose this information.

    @Nate: We would be happy to provide you with the login entries for your account. Please submit a support ticket.

    @Alexander: This is because the app uses an API key. We did not expire the API keys.

  73. @mkorsak: The replies are nice, but your answer to Kyhwana’s question about password hashing, that “Our information security policies do not permit us to disclose this information,” does not sound good. It’s the answer PR hacks regurgitate to cover up the egregious incompetence of faceless mega-corporations; a tech company can do better. I’m sure someone at Linode has heard of bcrypt and scrypt, but it’s difficult to give you the benefit of the doubt and put total faith in the decision-making of a company with an unbroken string of annual security incidents. I hope your “information security policies” will be reconsidered before your full write-up is published. And i hope your password storage is good, and that, if it wasn’t before, you took the golden opportunity this reset offered to fix it.

  74. I still can’t get to linode via FF28, both here and at the office. I was able to via safari. Did you change the website?

  75. @Matt What algorithms are you using to secure your information? That’s pretty much what you’re asking Linode to reveal. mkorsak answer, IMO, is 100% acceptable from a responsible company. We do have to trust them at some level that they know what they are doing. Everyone is in a position to get hacked, everyone.

    @Brian Tkatch, FF28 is old, very old, try the latest version. Current version is 43.0.4 Ironic that in a thread about security, you’re using something so very outdated.

  76. @Waldo you will find a lot of people use FF28 or a fork thereof. Linode worked beautifully in FF28 until very recently, which s what is so surprising.

  77. How soon is soon? I want to know what really happen here.

    I’ve test testing my server on Linode for few month, and I’m satisfied, but after another security breach I’ll wait for few month before migrate all of my domain/website to Linode.

  78. @Alexander:

    If they can get into your account they can issue a new API key but they can’t retrieve existing ones. You’d want to kill any API keys you see that you don’t recognize but you don’t need to regenerate the ones you do recognize as ones you created. If you don’t see any API keys you don’t recognize, you don’t see any other suspicious changes on your account and your server logs aren’t showing any unauthorized access to your servers, there’s no reason to be more paranoid than normal at this point.

  79. my appreciations for keeping the security of the site good and healthy 🙂

  80. The status page states that Linode will publish an in depth review of the DOS attack and consequent mitigations. It’s been a while. Is that ever going to happen, and if so, when?

  81. I agree with @Jorge’s concern, and it’s not fully addressed by @mkorsak. An attacker that had stolen the hashes and cracked my password could log in and reset the password (and perhaps also change the e-mail address). It doesn’t matter much for this attack at this point, but for future password resets, I would suggest a reset link by e-mail. @mkorsak mentions concerns about looking like phishing, but this reset link should not arrive unsolicited, rather as a response to the first login after the site-wide expiry, with an explanation in the UI.

  82. These things happen, even to the best of the best. I’d wager most companies out there would cover up something like this, but Linode has done the right thing. I have not lost trust in Linode, the opposite in fact.

    Thank-you for this disclosure.

  83. Nice information!!!!!!! Thank you for sharing this.

Leave a Reply