Evergreens

Mark Smith's Journal

Work related musings of a geek.

A day in the life...

[staff profile] mark

We had a server lock up today (this is fairly normal and our infrastructure is resilient against it, so no downtime happened!). Anyway, there was an email thread that is pretty typical of how things happen behind the scenes here, so I figured I'd reproduce it here. (With permission!)

[personal profile] alierak: Memory usage suddenly went through the roof on lb01, which is really just a web/mogile server and doesn't have perlbal or varnish set up. I can't get into it to fix anything, I think we'd have to have ServerBeach powercycle it, but I'm not going to be around this evening.

...

[staff profile] mark: I'll take this one -- enjoy your evening! :)

I agree re: the powercycle, although I am going to give it 10 minutes and see if I can possibly SSH in. If it's swapping, it might eventually let me in and I can see what happened...

...

[staff profile] mark: It's back. I never got in via SSH and rebooted it. Took ~25 minutes to come back up -- fsck it looks like -- but it's back and serving requests now.

...

[staff profile] denise: maybe it just wanted a nap.

...

[staff profile] mark: We don't allow our servers to nap. It's not in their contracts. They have to be up 24/7/365.

They're not even allowed religious observances.

It's really kind of a bum deal.

...

[staff profile] denise: let's hope they don't decide to unionize!

...now i'm trying to remember if there's ever been a rabbinical ruling about tech companies owned by jews who keep shabbat strictly, and whether the servers can stay up during. on the one hand, you're not supposed to cause any fires to be kindled (which modern observance takes to mean no turning-on of things; electricity = fire), which means that using a computer is forbidden. on the other hand, servers are on all the time, and the restriction is generally held to be only on turning-on: things that are on all the time are all right. (if you remember abusefest '05: desh turned on the living room and bathroom lights ahead of time and taped an index card over the switches saying "please do not turn these off": that was okay, because the light was turned on before the shabbat started.) so a server that's always-on wouldn't be "melakahah", or "causing work to be done". so it would probably be acceptable even for people who kept strict observance. but i'm really curious if there's ever been a ruling, now!

--D, who is not going to get sucked into researching that...

...

[staff profile] mark: hahahahahaha -- can I post that to [staff profile] mark? "a look into the life of DW at work". :)

...

[staff profile] denise: lol. feel free, someone might know the answer <3

...

So there you have it: a day in the life of Dreamwidth behind the scenes!

20.02.2013 01:53 am (UTC)

(no subject)

dmsj: (working)
Posted by [personal profile] dmsj
If you're ever looking to expand the staff, you should totally include this in the job description. I don't even know how to do anything you guys do, but this thread makes me want to work for you anyway! :)
20.02.2013 02:06 am (UTC)

(no subject)

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Posted by [staff profile] denise
We definitely have fun. :)
20.02.2013 02:40 am (UTC)

(no subject)

misskat: Picture of Kat with black plastic frames (Default)
Posted by [personal profile] misskat
My job description is "keep being AWESOME." <-- not even kidding