Mark Smith's Journal

Work related musings of a geek.

A day in the life... (Reply)

[staff profile] mark

We had a server lock up today (this is fairly normal and our infrastructure is resilient against it, so no downtime happened!). Anyway, there was an email thread that is pretty typical of how things happen behind the scenes here, so I figured I'd reproduce it here. (With permission!)

[personal profile] alierak: Memory usage suddenly went through the roof on lb01, which is really just a web/mogile server and doesn't have perlbal or varnish set up. I can't get into it to fix anything, I think we'd have to have ServerBeach powercycle it, but I'm not going to be around this evening.


[staff profile] mark: I'll take this one -- enjoy your evening! :)

I agree re: the powercycle, although I am going to give it 10 minutes and see if I can possibly SSH in. If it's swapping, it might eventually let me in and I can see what happened...


[staff profile] mark: It's back. I never got in via SSH and rebooted it. Took ~25 minutes to come back up -- fsck it looks like -- but it's back and serving requests now.


[staff profile] denise: maybe it just wanted a nap.


[staff profile] mark: We don't allow our servers to nap. It's not in their contracts. They have to be up 24/7/365.

They're not even allowed religious observances.

It's really kind of a bum deal.


[staff profile] denise: let's hope they don't decide to unionize!

...now i'm trying to remember if there's ever been a rabbinical ruling about tech companies owned by jews who keep shabbat strictly, and whether the servers can stay up during. on the one hand, you're not supposed to cause any fires to be kindled (which modern observance takes to mean no turning-on of things; electricity = fire), which means that using a computer is forbidden. on the other hand, servers are on all the time, and the restriction is generally held to be only on turning-on: things that are on all the time are all right. (if you remember abusefest '05: desh turned on the living room and bathroom lights ahead of time and taped an index card over the switches saying "please do not turn these off": that was okay, because the light was turned on before the shabbat started.) so a server that's always-on wouldn't be "melakahah", or "causing work to be done". so it would probably be acceptable even for people who kept strict observance. but i'm really curious if there's ever been a ruling, now!

--D, who is not going to get sucked into researching that...


[staff profile] mark: hahahahahaha -- can I post that to [staff profile] mark? "a look into the life of DW at work". :)


[staff profile] denise: lol. feel free, someone might know the answer <3


So there you have it: a day in the life of Dreamwidth behind the scenes!

This account has disabled anonymous posting.
(will be screened if not validated)
If you don't have an account you can create one now.
HTML doesn't work in the subject.
Notice: This account is set to log the IP addresses of everyone who comments.
Links will be displayed as unclickable URLs to help prevent spam.