Lassie, the shutdown watchdog

October 26, 2007 11:18 am

[b]Lassie – Linode Autonomous System Shutdown Intelligent rEbooter[/b]

Lassie is a shutdown watchdog. It will automatically reboot your Linode if it powers off unexpectedly. It works by detecting that your Linode has powered off without a shutdown job being responsible for the power off condition. To prevent a loop, it will give up after 5 boots have occurred within a 15 minute period.

It’s enabled by default, but the setting can be changed on the [url=https://www.linode.com/members/utilities/watchdog.cfm]Lassie page[/url], located off of the new [url=https://www.linode.com/members/utilities/]Utilities sub-tab[/url].

A side effect of this is that now “reboot” from inside your Linode will function as expected, but a “shutdown” from inside your Linode will result in Lassie kicking in and issuing a boot. Previously both of those commands resulted in a shutdown. I can’t win :). A work-around for this is to issue a shutdown job via Lish, by creating an alias for “shutdown” to do: “ssh LinodeUsername@hostXX.linode.com shutdown”.

Notifications via email are planned.

Enjoy!
-Chris

10 Responses

  1. [quote:f17891f4fb=”caker”]
    A side effect of this is that now "reboot" from inside your Linode will function as expected, but a "shutdown" from inside your Linode will result in Lassie kicking in and issuing a boot. Previously both of those commands resulted in a shutdown. I can’t win :). [/quote]

    For what it’s worth, I think this a better default. I might reboot from inside the linode to make sure new libraries are in use, or just invoke the nostalgia of my WIndows day, but the only reason I shutdown is to re-arrange disks or somesuch, which I’m going to be doing from the Web UI anyway.

  2. [quote:df0090ef37=”caker”][b]Lassie – Linode Autonomous System Shutdown Intelligent rEbooter[/b]

    Lassie is a shutdown watchdog.
    [/quote]

    Since it’s [i]almost[/i] Halloween, please rename it to:

    [b]L[/b]inode [b]A[/b]utonomous [b]S[/b]ystem [b]S[/b]hutdown [b]I[/b]ntelligent r[b]E[/b]booter ag[b]E[/b]nt [b]A[/b]nd [b]T[/b]im[b]ER[/b] (LASSIE EATER)

    James

  3. I’m a little curious — have shutdowns been a problem?

    I’ve been on Linode since ’04, and mine has never gone down on its own.

    I’m not trying to diss a new and valuable feature — but is it solving a problem that people actually have?

  4. [quote:c80aeb5696=”astrashe3″]I’m a little curious — have shutdowns been a problem?[/quote]
    Definitely not widespread. But, Linux has bugs from time to time, and so does UML. It’s just a nice thing to have, rather than having your Linode go down 5 minutes after you go to bed, and not knowing about it until you get up the next morning. This also solves the "It’s been 2 years since I’ve logged into linode.com, lost my password, the email address you have for me no longer works, and I typed reboot in my Linode and it didn’t come back up" issues (because I hack UML to shutdown on "reboot" so it’ll pick up any config changes between boots).

    -Chris

  5. I got a pretty major problem when I went to resize my main drive with the new disk space we’ve just been given. Lassie started booting before it finished and pretty nastily barfed the drive.

    Lassie should check that another job isn’t already running before forcing a boot!

  6. Lassie doesn’t boot your Linode — it just queues up a boot job. Jobs run one after another. And, the resize jobs *should* still be checking that your volume isn’t in use before doing anything… I’ll have a look at the logs.

    -Chris

  7. OK, I found, and squashed a bug. It was a race. We use the pid file that UML writes out in determining if a Linode is running or not. It turns out that UML doesn’t immediately write this file. If another job ran just after the Linode booted and then checked to see if the Linode was active, it would not find the pid file and assume it was ok to do its thing.

    Now boot jobs wait a few seconds, while checking to see if UML wrote out its pid file, before completing.

    This bug has been in here for a LONG time (way before Lassie), so I’m grateful to you for reporting this, but sorry it ended up in your filesystem being munched on a bit.

    Regards,
    -Chris

  8. [quote:2a883d7a6d=”caker”]
    This bug has been in here for a LONG time (way before Lassie), so I’m grateful to you for reporting this, but sorry it ended up in your filesystem being munched on a bit.
    [/quote]

    That’s cool. It seems to have e2fscked without any real problems. My only issue now is that the Linode dashboard reports the fs as 20 gigs but inside the linode it’s still 16 gigs. I suspect it might need some manual command-line hackery to get it back in shape, but not sure where to start.

    Feel free to bring my linode down if you think you can resize it to its proper size. Otherwise, any ideas on how to do it?

  9. Try shrinking it to 16G (or 15G) and then re-enlarging? Also, what does fdisk show for the partition size? It’s possible that the partition got enlarged, but not the file
    system. If that’s the case, then you should be able to enlarge the file system from within the linode.

  10. [quote:2a7ae073e2=”SteveG”]Try shrinking it to 16G (or 15G) and then re-enlarging?[/quote]

    That did the trick. Thanks!

Leave a Reply