A day in the life...

[mark]

We had a server lock up today (this is fairly normal and our infrastructure is resilient against it, so no downtime happened!). Anyway, there was an email thread that is pretty typical of how things happen behind the scenes here, so I figured I'd reproduce it here. (With permission!)

alierak: Memory usage suddenly went through the roof on lb01, which is really just a web/mogile server and doesn't have perlbal or varnish set up. I can't get into it to fix anything, I think we'd have to have ServerBeach powercycle it, but I'm not going to be around this evening.

...

mark: I'll take this one -- enjoy your evening! :)

I agree re: the powercycle, although I am going to give it 10 minutes and see if I can possibly SSH in. If it's swapping, it might eventually let me in and I can see what happened...

...

mark: It's back. I never got in via SSH and rebooted it. Took ~25 minutes to come back up -- fsck it looks like -- but it's back and serving requests now.

...

denise: maybe it just wanted a nap.

...

mark: We don't allow our servers to nap. It's not in their contracts. They have to be up 24/7/365.

They're not even allowed religious observances.

It's really kind of a bum deal.

...

denise: let's hope they don't decide to unionize!

...now i'm trying to remember if there's ever been a rabbinical ruling about tech companies owned by jews who keep shabbat strictly, and whether the servers can stay up during. on the one hand, you're not supposed to cause any fires to be kindled (which modern observance takes to mean no turning-on of things; electricity = fire), which means that using a computer is forbidden. on the other hand, servers are on all the time, and the restriction is generally held to be only on turning-on: things that are on all the time are all right. (if you remember abusefest '05: desh turned on the living room and bathroom lights ahead of time and taped an index card over the switches saying "please do not turn these off": that was okay, because the light was turned on before the shabbat started.) so a server that's always-on wouldn't be "melakahah", or "causing work to be done". so it would probably be acceptable even for people who kept strict observance. but i'm really curious if there's ever been a ruling, now!

--D, who is not going to get sucked into researching that...

...

mark: hahahahahaha -- can I post that to mark? "a look into the life of DW at work". :)

...

denise: lol. feel free, someone might know the answer <3

...

So there you have it: a day in the life of Dreamwidth behind the scenes!

Comments

[jecook]

Heh- Occasionally, one of our servers decides to take an un-authorized coffee break- Generally, this results in a number of applications going comatose, as said server is one of the nodes in our VMware cluster and there's usually several virtual machines on each node at any given point.

[dmsj]

If you're ever looking to expand the staff, you should totally include this in the job description. I don't even know how to do anything you guys do, but this thread makes me want to work for you anyway! :)

[denise]

We definitely have fun. :)

[misskat]

My job description is "keep being AWESOME." <-- not even kidding

[desh]

Gotta say, kinda flattered to show up in an official DW email thread!

(...that was seriously EIGHT years ago?)

[desh]

Oh, and denise, I'm pretty sure that running servers on Shabbat is rather universally accepted, unless you venture into the fringes of the Jewish world that would call computers and/or the internet forbidden in general, Shabbat or no. Where it could get complicated (among folks who make and follow these rulings) is having non-Jewish employees be responsible for emergency server maintenance on Shabbat. But even in that case, I bet many would say it's fine as long as the employees are not Jewish, and are responsible for emergency maintenance throughout the week, whenever it's needed, not specifically just on Shabbat.

Somewhat relatedly, I recently read a ruling addressing the idea of owning stock for a business that does work on Shabbat (i.e. most businesses). Because owning stock is like owning the business, right?

themoreyouknow.jpg

[musyc]

This was the suggestion/idea I had. That there be a dedicated non-Jewish employee to flip the switch, as I believe it's permissible to do so in order to prevent great financial loss or illness. Servers on = no losing money and/or no giving Rah/Mark/Fu/Kat ulcers. XD

[mark]

Thank you for the elucidation! And yeah, I suppose I could see an argument that owning stock is like owning the business... but only if you stretch it to the breaking point. Actually, even then, stock is more like owning interest in a business. Ah!

[azurelunatic]

Disclaimer: not Jewish here, but I have been on the fringes of a good helping of discussions of this sort of thing. And the answer is probably going to vary depending on who you ask, on the grounds that debate is a cultural sport. :D

Where you'd get into trouble, I think, would be stuff like cronjobs where you know what's going to happen at what time, because that's directly causing work to be done on the Sabbath. (One does not ask/tell the Shabbos goy to do a thing, one hints that it might be nice to have a cup of tea or whatever, and waits for them to pick up on it.) Same with automation -- if you set up something to automatically reboot the server or pull some stats in the middle of the Sabbath, even if you're performing the setup on another day, the work's still being done and thus not on. However, there's a particular subset of tech specializing in introducing an unpredictable delay into electrical switches, such that pushing the button won't immediately cause work to be done.