Linode Manager Security Features

May 8, 2009 9:45 am

We’ve had requests for additional security and notification options for the Linode Manager, and recently we’ve noticed an upward trend in Linode Manager brute force attempts, so we decided to tackle a couple related features on our long and ever-growing feature request list (keep them coming).

Login IP Whitelisting and Notification Feature
You’ll receive a notification via email if someone attempts to log into the Linode Manager as your username from an IP not on your whitelist. The email contains instructions for adding that IP to your whitelist. IPs not on your whitelist are denied access.

Passwords are required to be more complex
Passwords must contain characters from three out of four sets: lower case, upper case, numbers, and punctuation, in addition to the old requirement of being a minimum of 6 characters long.

DNS Manager zone AXFR control
You can now specify whether a zone can be AXFRed from our nameservers. Right now it’s an on/off thing, but we’ll soon be adding support for specifying ranges and/or specific IPs that can transfer the zone.

We’ve also laid the groundwork for passwords with an expiration date (we’re now keeping track of when a password was set). Some other ideas are still on the table, like only allowing a few failed log in attempts within a short time span, to reduce the possibility of brute forcing accounts that may not have the IP whitelisting feature enabled…

Security can be a pain, but it’s a necessary evil. It’s a compromise between security and convenience, and we hope we’ve struck a fair balance.

46 Responses

  1. Effective immediately?

  2. Great additions, but one minor request:

    in order to avoid having people on dynamic ip connections continuously approve new IPs you should provide some sort of management of the ip whitelist — perhaps allowing whole subnets.

  3. Thanks @cosmix — we’re working on that right now.

  4. This sounds great.

    Outside of the whole whitelist feature, I would be happy with an email alert every time there is a Linode Manager login under my account, with the details on the connection (username, IP). If I could set the email address for the alerts, it would be a form of logging.

    Lastly along these security improvements, there should be some sort of activity logging. [timestamp] User nnnnn deleted linodeNNNNN, [timestamp] User nnnnn created disk image “Ubuntu 8.04” on linodeNNNNN. That sort of thing πŸ™‚

  5. If I could add to the login whitelist suggestion – possibly to the point of useless, but whatever.

    In addition to a stock IP whitelist, add a DNS whitelist, both reverse DNS mapping and standard A address resolution.

    For example, I’d be able to add *.ut.comcast.net to the reverse DNS mapping to only allow logins from Utah on Comcast. This solves the “localized dynamic IP address changing” problem.

    Or, a forward A address lookup. A lot of people have dynamic DNS entries for their own computers, but are not always in control of their reverse DNS (in fact, rarely so).

    I could whitelist *.ut.comcast.net in the reverse DNS, and then whitelist laptop.averageurl.com in the forward DNS, allowing me access from anywhere in my home state with the option of using my laptop while mobile.

    Of course, you could probably add a few hundred “fun little security tricks” to the manager; the question is if the time is warranted πŸ™‚

  6. Perhaps it would be a nice idea to be able to edit the whitelist online via the profile section?

    In other words, a textarea box that lists the IP addresses and/or ranges that may be whitelisted.

    Perhaps in the future, a user may want to remove a particular IP address from the whitelist for whatever reason??

    Implementing a blacklist may be a good idea too!

    All in all, it’s good to see that the Linode team is concerned about security. Keep up the good work boys!! πŸ™‚

  7. A couple of suggestions:

    1. Securid tokens (name.com enabled these through verisign)
    2. Notifications for nodes additions/deletions/etc., period.
    3. Click password authentication, exmaple: ingdirect.com or hsbdirect.com where you have to click your password in to fool key loggers.
    4. Secondary passwords to delete/add/change nodes (some domain companies offer this moniker has maxlock to change domains and fabulous offers this)
    5. Allow only certain ips to do axfr, dnsmadeeasy.com offers this.

  8. Hello,
    freezing the account for 60 minutes after every, say 20 failed login attempts is 100% prohibitive for brute-force attempts, in my opinion and also adds virtually zero inconvenience. I would personally welcome that feature very much.

    On the other hand, please don’t force us to periodically change passwords.
    Once the password is compromised, every attacker ensures that his access is not interrupted if the victim changes his password. And given the usual period of several months this renders password expiration completely useless and only adds frustration to the user.
    Thanks,
    Marc

  9. Maybe add SSL Personal Certificates to your todo list of features. This may be a better option for those that don’t have a static IP. A certificate gets installed into your browser and the server will look for that when you try to sign on. if you don’t have it you either get denied access or required to provide more info. depending on how its implemented.

    Another option would be a 2 factor authentication scheme.
    2 methods come to mind.
    1) send an authentication key to an email address or cell phone text message
    2) use the annoying little RSA fobs.

  10. Personal certificates would be a wonderful addition. I truly hope it’s being considered. IP Whitelisting really doesn’t feel comfortable for me. Think about it, I either should allow all internet subscribers from China, or keep on allowing my new ip every single day.

    Useless.

  11. So when a botnet tries to brute my password, I’ll get 100,000 emails asking if they should whilelist those IP’s?
    No thanks.

  12. If it’s hard to improve security, then reduce the insecurity.

    Preventing a Brute Force or Dictionary Attack: How to Keep the Brutes Away from Your Loot: http://www.wwwcoder.com/parentid/148/tabid/68/type/art/site/6488/default.aspx

    Obviously, I first thing I did do was to disable the Account Security.

  13. @Marc Freezing the account after several login failures is unappealing–anyone could lock you out of your own account.

    Perhaps the IP address causing the login failures could be blocked for 60 minutes, after 3 failed attempts.

    I just went through my first whitelisting procedure. Since my IP is dynamic, it makes me concerned that any delay in email would keep me from getting into the account when really I need to–for example, if the Linode hosting that email account is down.

  14. I second Paul’s latest comment, freezing accounts due to failed logins is definitely a bad idea.

  15. Locking out an account after repeated login failures allows someone without credentials to lock you out of your own account. This strategy also doesn’t do much to mitigate “slow & patient” style botnet attacks, a 60-minute lock doesn’t bother them in the least, they’re fine with devoting months to the attempt.

    Thanks for the ability to disable AXFR! πŸ™‚

  16. Better yet, let them keep trying their spam logins with false negatives if they’re blocked.

    The reverse-dns whitelist seems like a bad thing if there could be any level of dns poisoning though.

  17. How about allowing whitelist ips to be added via a lish/screen command predicated on the fact that I logged into lish with an ssh key.

    In other news, I love you guys, but the speed at which this change occurred and lack of seeming transparency is very unlike you. I wish this wasn’t the case.

  18. “Better yet, let them keep trying their spam logins with false negatives if they’re blocked.”

    That’s even worse, I won’t know why my logins are failing.

    If there has to be something like this, don’t block IPs that have successfully logged in with that account recently, and let people unblock IPs via their account’s e-mail (encourage this to be hosted by another provider), and let people unblock IPs via lish.

  19. Considering a lot of people have ssh public key authorization already in place for linodes login, maybe it would be possible to reuse the same mechanism for authorization? Say, ssh to lish to get one-time password needed for crucial operations…

  20. Well, it is a good attempt. But I don’t like your implementation, I just don’t see the point of associating email as the only valid identifier. What if my email get hacked or forgot my password to my email? Or what if I have to travel all the time and thus getting different IP addresses? that means I will have to enable whitelist all these IP addresses every time I would like to do the panel, which makes the user panel less effective. I think a better approach is to use this for certain features like system reboot, install a new OS etc, it is your call to make this executive decisions or let the users to decide which features they want to set ACL on. By conclusion, it is a great feature and potential to be a useful feature, but I rate it as a failing feature at this point.

  21. Have you considered using http://www.cipherdyne.org/fwknop ? fwknop originally started as a port knocking implementation but has now been extended so that you can use a public/private key pair to unlock access to a service on a port. The private key is encrypted and sent to the server which listens in promiscuous mode. The client, holding the private key can authenticate from any ip address.

    The result is that potential attackers won’t even be able to see the management interface listening on a port because it is invisible until you authenticate. Upon authenticate fwknop inserts a rule in the iptables access list to allow the client access. After a specified timeout the rule is deleted from the access list.

    I’ve used this quite sucessfully to limit access to the ssh port on hosts that I manage remotely.

  22. I really think that you’re asking the passwords to be TOO complex. What’s wrong with a mix of letters and numbers? That can still be fairly complex, yet easy to remember.

    Requiring us to mix cases and punctuation is only going to make it harder to remember a password, let alone type it in. Please rethink this.

  23. So I just tried to login and was told an email was being sent so I could verify my IP. Twenty minutes later and I have no email. How does one go about proving they own the account and should be let in, if no email gets sent?

  24. I’m also at a dynamic ip ISP and I haven’t received any mails..

  25. The whitelisting feature is kind of annoying, but its kind of cool to check my email and see different ips that are not mine trying to log into my account.

  26. I also wish to express my general annoyance at character set requirements on passwords. A similar degree of security may be achieved by increasing the required password length. To illustrate:

    6 character numeric [0-9[ password ==> ~20 bits of entropy
    6 character alphabetic [a-z] password ==> ~28 bits of entropy
    6 character mixed-case alpha-numeric [A-Za-z0-9] password ==> ~36 bits of entropy
    8 character numeric [0-9] password ==> ~27 bits of entropy
    8 character alphabetic [a-z] password ==> ~38 bits of entropy

    So, by upping the character minimum limit to 8 characters and disallowing only entirely numeric passwords, you achieve a greater degree of entropy than by requiring 3 out of the 4 major character groups in a 6 character password.

    However, I’ll admit that there’s one major flaw with all my pretty numbers — they assume perfectly random passwords. And it’s true that requiring representation from several character sets *does* force people away from using their kid’s name.

    Still, I’ve always found things like Diceware passphrases and >=14-digit random all-lowercase alphabetic passwords easier to remember than ]aTKxK`( or 8E5S2+’c. Each to their own, I suppose.

    I also support the burst rate limiting from individual IPs, but it’s important to remember that this won’t protect poor passwords from botnets. To illustrate:

    Firstly, for humour, let’s suppose that my password may be found in a dictionary. For me, wc -l /usr/share/dict/words yields just under 100000 results. If we assume an acceptable risk of password compromise as 1 in 1000 every 4 months (you really aught to change your password this often) this means a brute force rate of 25/month by any particular attacker would be unacceptable.

    Now, in seriousness let’s say your adversary has a solidly-sized botnet of 40000 bots. If, due to IP-specific rate-limiting, each bot can try 3 passwords per hour, then in four months they can collectively try 3*24*30*4*40000 = 345,600,000 passwords. This is equivalent to *every* possible permutation of 6-character alpha-only passwords. It is also 7 orders of magnitude greater than the “acceptable” limit for a dictionary word password (in fact, my dictionary password could be cracked with 100% probability within the first hour by such an adversary.)

    What’s this mean? Well, we can hardly employ rate-limiting as a complete solution. Without IP-specificity it would be cause more of a problem than it would solve as malicious parties could lock individuals out of their accounts at will. With IP-specificity, it is only a part of a solution as botnets could still mount a formidable assault.

    Clearly this is where the white-list becomes quite valuable, but the white-list is a major spanner in the usability works for anyone with a dynamic IP (and thus should be able to be disabled and/or modified via SSH as suggested by tjfontaine above). Also, the email-verification method has problems — 40000 emails courtesy of Ms Mallory Botnet is no-one’s idea of fun and it the emails are rate-limited, then we’re no better of than with plain rate-limiting as legitimate requests could easily be lost.

    I think Brandon’s idea of personal SSL certificates is friggn’ excellent, but still won’t be practical for everyone. And then there’s the issue of how to verify a CSR (be it a user’s first request or because they lost their old one) is coming from the real user. Email? Most email sites use password protection. So we’re back to square one (although I’d still encourage Linode to support PKCS certificates!)

    So if we have to rely on passwords, how complex should they be?

    Well, let’s again assume a password turnover frequency of 4 months and an acceptable probability of password compromise of 1 in 1000 (for any given 4month cycle). Our adversary still has 40000 bots. Rate-limiting has been implemented at the slightly more lenient rate of 5 tries per hour per IP address. white-listing is off. This is comparable to GMail, so far as I can tell.

    To achieve an acceptable password strength, we now need a password with 5*24*30*4*40000*1000 permutations, ie ~ 39 bits of entropy. Let’s round it up to 40 bits. How long is such a password? Well:

    For a mixed-case alpha-numeric [A-Za-z0-9] password, we’d need ln(2^40)/ln(62) = 7 characters

    For an alpha-numeric [a-z0-9] password, we’d need ln(2^40)/ln(36) = 8 characters

    For an alpha-only [a-z] password, we’d need ln(2^40)/ln(26) = 9 characters

    Of course, this assumes “perfectly” random passwords, as in from a password generator such as pwgen or apg (with appropriate switches set). It also assumes 40000 bots could attack a single site for 4 months without the network admins at Linode noticing and stepping in manually. It is *also* based on guestimates of acceptable rate-limits. In the absence of formal rate-limiting, I don’t know what practical rate limit would be imposed by the network, but if each bot could try 100 password/sec, the above minimum characters would need to go up to 10, 11 & 12 respectively. But this is 40000000 (40 million) request/sec. That sounds more like a DDoS than a distributed brute-force attack ^^.

    So what have we learnt from all this? I’m not entirely sure. Maybe I should summarise…

    Dictionary words == Awful passwords. Never use them. Ever. Maybe do a dictionary test against all prospective new passwords and reject any that match.

    Compulsory character set passwords == bad. Password strength should be based on simplified potential entropy calculations.

    >35 bits of entropy == Acceptable password on a white-list-protected or rate-limited-by-IP account. (This is a 6-char mixed-case alpha-numeric p/w or an 8-char alpha-only p/w or equivalent.)

    >50 bits of entropy == Good password on an otherwise unprotected account. (This is a 10-char mixed-case alpha-numeric p/w or a 12-char alpha-only p/w or equivalent. This might be over-kill.)

    Rate-limiting == Okay if only acting per IP-address. 5 tries/hour probably a good balance. Not the whole solution.

    Whitelist == Awesomely powerful solution, but raises usability issues. Should be able to be manually modified and disabled entirely at the user’s request.

    PKCS support == Super sexy, super convenient & secure, but its security ultimately depends on the strength of the user-verification mechanism for CSR signing or public key uploading. ie depends on existing password/rate-limiting/white-list infrastructure.

  27. You could offer a paid two-factor authentication upgrade for those who are really paranoid about security. Maybe incorporate something like the Yubikey (http://www.yubico.com/products/yubikey/) or a SMS-based system. Something in the range of $5-$10 a month for this service would be fairly reasonable.

  28. On a related note, I notice that the Linode SSL certificate uses MD5 as its signing hash. That’s really not a great idea, and I’d encourage its renewal prior to its expiration date next year. Surely Equifax will grant you a new certificate gratis if you complain loudly enough — the attacks are quite well documented & very well publicised. (To the point of being alarmist in many cases, but the underlying problem remains.)

  29. Please _don’t_ add password expiration that forces me to change my password every X days. Just make it so that people are forced to select complex passwords.

  30. you should allow longer passwords and extended characters in your passwords.

  31. Don’t add support for expiring passwords (unless it can be disabled) and allow for ip ranges for dyanmic ip stuff..

  32. Re. security:

    Look at the following line from the ssh log on my computer ‘unicom’ (running Linux, but not a linode customer):

    May 30 19:11:28 unicom sshd[27875]: Failed password for root from [redacted] port 39197 ssh2
    May 30 19:11:29 unicom sshd[27877]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[redacted].members.linode.com user=root

    This has been going on for quite a while now. What do you do against this abuse to other people by your customers?

    hjt

  33. @Hendrik-Jan Thomassen: We open an abuse ticket as soon as it is brought to our attention. I have done so with this customer.

  34. Not a bad feature to have, but I would appreciate it if in the future you would send email notifications about any significant changes you make to your service.

  35. By the way, can you start supporting openid for login?

  36. I logged into the platform manager tonight and it asks me to renew my password. I’m thinking – no, this isn’t right – and bail. Instead I reset my linode via LISH instead.

    I like the private certificate thing. I want to set this up on my own server. Where do I find out more?

  37. Please let me add subnets to the whitelist. The way it is now is a total waste of time and adds virtually no additional security.

    You could also turn around the email confirmation. What is that, a 3-minute cron job?

  38. @Daniel You can already add subnets in the whitelist editor (off My Profile).

  39. Hh, okay, thanks. I expected it under “Users & Permissions”, (where the password can also be set).

  40. Is there a good reason for your server to wait more than a couple of minutes to send the confirmation email? Why not send it immediately?

    I’m connecting to the net via a DSL service that reassigns the IP address every few days, often from an entirely different block.

  41. The email is sent out immediately. If there’s a delay, it’s not from our end.

  42. Can you double check that? I tried to log in at 3:48:15 PM CST, but your internal email header shows that it was generated at 3:54:01 PM.

    This delay is consistent with my experience ever since I started using it.

  43. Now it’s immediate. Thanks!

  44. Nope, it’s still delayed. (I guess I was just lucky last time.) Triggered at 00:45 CST, you didn’t send the email until 01:53 CST.

  45. Nope, it’s still delayed. (I guess I was just lucky last time.) Triggered at 00:45 CST, you didn’t send the email until 00:53 CST.

  46. This is an old thread but I’ve noticed a delay too. I just tried to sign in 3 times over the course of a couple of minutes and I started getting nervous when no email arrived. So I went in via SSH and tailed my mail logs. All of a sudden all 3 whitelist emails from Linode came through at exactly the same time as I watched realtime in the logs.

    Not a big deal – I’m not sure where the delay is, but give it a few minutes for the messages to arrive.

Leave a Reply