hey ~teammates,
there has been a bit of discussion on irc about this, but the main limiting factor that we've been running into here on ~team is the disk space.
this machine has two 250gb ssd disks that were autoconfigured into raid 1 by the default installer image. i originally missed this step during setup (the option to reconfigure disks with the hetzner-specific settings).
since all user data is already backed up hourly, i'm fairly comfortable switching off the redundancy of raid 1. this would involve copying / to a temporary machine that i plan on renting short-term as a staging ground, reinstalling with the hetzner installer, then copying / back onto the same machine, but with the disks reconfigured as raid 0 - up to 500 gb!
since this will inevitably incur some downtime, i wanted to reach out and make sure that this is something that we want to see happen and also check if there are any times that this should *not* be performed.
holler here on the list or on irc if you have any thoughts, concerns, or other time constraints that i should be aware of.
thanks! ~ben
holler here on the list or on irc if you have any thoughts, concerns, or other time constraints that i should be aware of.
It does sound like raid1 is a bit redundant in this case. go for it!
On Thu, 20 Dec 2018 at 14:25, Ben Harris ben@tilde.team wrote:
hey ~teammates,
there has been a bit of discussion on irc about this, but the main limiting factor that we've been running into here on ~team is the disk space.
this machine has two 250gb ssd disks that were autoconfigured into raid 1 by the default installer image. i originally missed this step during setup (the option to reconfigure disks with the hetzner-specific settings).
since all user data is already backed up hourly, i'm fairly comfortable switching off the redundancy of raid 1. this would involve copying / to a temporary machine that i plan on renting short-term as a staging ground, reinstalling with the hetzner installer, then copying / back onto the same machine, but with the disks reconfigured as raid 0 - up to 500 gb!
since this will inevitably incur some downtime, i wanted to reach out and make sure that this is something that we want to see happen and also check if there are any times that this should *not* be performed.
I'm really glad tilde.team has raid 1 redundancy. Hourly backups are great, but if the disk fails (which it will, in 3-5 years), then there will be unscheduled down time and the need to get hetzner to install a new disk and copy from the latest fully-intact hourly backup.
My suggestion would be to have a tilde.team referendum (a program registered on "tilde" for voting should suffice), and set a minimum participation level and percentage of votes required for the issue to pass. This will also set the precedent for how governance issues should be handled in the future.
holler here on the list or on irc if you have any thoughts, concerns, or other time constraints that i should be aware of.
thanks! ~ben _______________________________________________ tildeteam mailing list -- tildeteam@lists.tildeverse.org https://lists.tildeverse.org/hyperkitty/list/tildeteam@lists.tildeverse.org/... to unsubscribe send an email to tildeteam-leave@lists.tildeverse.org
On Thu, Dec 20, 2018 at 05:30:57PM +1100, login wrote:
I'm really glad tilde.team has raid 1 redundancy. Hourly backups are great, but if the disk fails (which it will, in 3-5 years), then there will be unscheduled
this box has ssds, not spinning disks. they have a far longer expected lifespan. that's not something that i'm overly concerned about.
down time and the need to get hetzner to install a new disk and copy from the latest fully-intact hourly backup.
My suggestion would be to have a tilde.team referendum (a program registered on "tilde" for voting should suffice), and set a minimum participation level and percentage of votes required for the issue to pass. This will also set the precedent for how governance issues should be handled in the future.
sounds good to me. could even just be something distributed here on the mailing list.
also, you don't need to quote the list footer or the entire message. just keep the necessary parts for whatever you're replying to. additionally, please wrap your messages to 80 chars or less.
thanks for the input!
On Thu, 20 Dec 2018, Ben Harris wrote:
On Thu, Dec 20, 2018 at 05:30:57PM +1100, login wrote:
I'm really glad tilde.team has raid 1 redundancy. Hourly backups are great, but if the disk fails (which it will, in 3-5 years), then there will be unscheduled
this box has ssds, not spinning disks. they have a far longer expected lifespan. that's not something that i'm overly concerned about.
I'm not overly concered with downtime of ~team (Or any ~tilde) to be honest.
It should be considered best practice to have accounts on at least two tilde systems, with content mostly mirrored, or to have your configs and data in some sort of source control anyways.
Basically, ensure you can live online like a migrant worker.
down time and the need to get hetzner to install a new disk and copy from the latest fully-intact hourly backup.
I never had an expectation of data being backed up, although, that is an appreciated service.
My suggestion would be to have a tilde.team referendum (a program registered on "tilde" for voting should suffice), and set a minimum participation level and percentage of votes required for the issue to pass. This will also set the precedent for how governance issues should be handled in the future.
sounds good to me. could even just be something distributed here on the mailing list.
I believe many FOSS projects use a disparate mailing list for votes. Any non-vote email is just rejected from the list during voting periods. Just an idea.
However, for voting, if that is such a thing, should be done via a cli too. This ensures easy automatic tallying, anonymous voting, and a good way to ensure one-vote, one person. It's hard to forge an account here.
thanks for the input! _______________________________________________
On Wed, 19 Dec 2018, Ben Harris wrote:
since this will inevitably incur some downtime, i wanted to reach out and make sure that this is something that we want to see happen and also check if there are any times that this should *not* be performed.
holler here on the list or on irc if you have any thoughts, concerns, or other time constraints that i should be aware of.
I guess this really depends on how critical is the stuff that the other tilders have hosted here. It should not be that critical anyway, as this is a "volunteer effort" (with no warranties) so to speak.
For me, go for it, if there's the need of more space. I just want to say that it can give you more than one headache in the future (and maybe while you migrate it too).
If the possibility exists, adding hard disks to the existing machine and moving /home there is a better and safer idea (of course more expensive).
About the voting, here on list is just fine. But it has 2 drawbacks: a) vote counting (which will have to be manual) b) participation. I know from experience that mailing lists are not engaging on non-pure-tech environments. Maybe another method notified here and on IRC would increase participation.
Cheers,
holler here on the list or on irc if you have any thoughts, concerns, or other time constraints that i should be aware of.
Next week may be the best (or worst) time to do this since most cultures across the globe are celebrating of some sort next, and even more so the following week for New Years. This means we will either have a steep rise or fall in users depending on if they use that free time to log in or log out of the server. In any case, if you're able to do it then it will probably be the least impactful time to do so.
I'd just be sure to have tilde.team temporarily redirect to some banner saying "temporary downtime" of some sort with an expected ETA for all not in the mailing list.
~aewens
I'd just be sure to have tilde.team temporarily redirect to some banner saying "temporary downtime" of some sort with an expected ETA for all not in the mailing list.
yes please. ben, didn't you have a backup server set up or something? what happened to that?
On Thu, Dec 20, 2018 at 09:28:38PM -0500, fosslinux@tilde.team wrote:
yes please. ben, didn't you have a backup server set up or something? what happened to that?
yeah, there's a static github pages site at http://status.tilde.team that just recommends that people check the freenode channel #tilde.team for updates. that's the first place i'll go if something happens.
another follow up: i have provisioned a short-term auction server from hetzner. when i get some down-time over the next week, i'll be doing the reinstall. i'll send another notice the day before.
Hey all, I'm planning on doing this tonight.
Any objections?
The downtime will be a couple hours.
No objections from me
i'll send another notice the day before.
Hey all, I'm planning on doing this tonight.
Any objections?
The downtime will be a couple hours.
No objections from me either.
Would the mail server be down, and if so would messages sent while it's down be received after the server comes back up?
On 1/5/19 3:43 PM, Anton McClure wrote:
Would the mail server be down, and if so would messages sent while it's down be received after the server comes back up?
the whole server will be down, so yes.
incoming mails are will be held on the originating server and retried until the server comes back up.
outgoing will fail since the server is down.
aaaaand we're back!
had a bit of a hairy time getting mysql back up and running, but everything should be back and where you left it.
holler if anything seems amiss :)
tildeteam@lists.tildeverse.org
-
aewens＠tilde.team -
ahriman -
Anton McClure -
Ben Harris -
fosslinux＠tilde.team -
login -
Paco Esteban -
Robert Miles -
ubergeek