Distributed Proofreaders Outage Update: Monday, March 3, 2014

Unfortunately we have not been able to bring the server (prod6) back online remotely. After evaluation, we have decided to prepare a new server and migrate DP to that machine earlier than originally planned.

As noted previously, this is one of two identical machines graciously donated by Tom Kowal.

Prod6 has (literally) served DP well since 2006 and was slated for replacement in summer 2014 but, given the current situation, making that transition now should minimize the total disruption of service.

The new machine (prod7) has been in preparation since early this morning, already has the OS upgrade on it, and is being prepared/configured with the necessary software. In addition to being more recent hardware, prod7 also gives us two additional drive bays, a battery-backup unit on the drive controller, twice the RAM, and twice as many cpu cores as we currently have with prod6. Those factors combined with a move to the ext4 filesystem should make the server a bit more responsive during heavy load.

Brief stats: 2x Xeon @2.00GHz, 4 cores each. 8G RAM. 8x 300G drives (2 as RAID mirror for the OS, 4 as RAID-5 for the data, and 2 hot spares).

Once the build-out is complete, the new hardware will be transported on-site and we will begin transferring the databases, project files, etc. over. This may take an extended time, as we do not yet know exactly what conditions we will be working under.

Transport and on-site work is tentatively scheduled to take place tomorrow (Tuesday, March 4).

Once again, thanks for your patience. We’re working hard to bring you an improved experience.

David (donovan)

15 Responses to Distributed Proofreaders Outage Update: Monday, March 3, 2014

  1. Charlie Howard says:

    Thank you very much, David, and everyone else who is helping you with this. It’s a difficult situation and we are lucky to have such capable people available to resolve these issues. In the long run, the new server will more than compensate for this unforeseeable outage.

  2. Peter says:

    Sounds highly technical. Wonderful to have such knowledgeable and dedicated volunteers. Looking forward to an even more productive system in the coming days (or weeks). Good work David D and all who participate in this upgrade.

  3. Bel says:

    Thanks, David. Good luck with it all!

  4. Don C. (DACSoft) says:

    David, thanks for keeping us all up-to-date. Good luck with the migration! DP is indeed fortunate to have volunteers like you and the tech team. Your expertise and dedication during such an unforseen situation is exemplary.

  5. Meena Bhende says:

    It is great knowing that we have such a dedicated team working on this. Looking forward to having everything on line again.

  6. Matthias Grammel says:

    Thank you to you David and the team for getting the weird UNIX stuff running again!
    Good luck for installing the necessary software and transfering all the data to the new storage.
    I cross my fingers for you all and your great work!

  7. CJ says:

    Thank you for the further update and all the hard work you guys are doing to get us reliably up and running again as soon as possible.

  8. Ann Jury says:

    RIP prod6; long live prod7! And so many thanks to the team making it all happen.

  9. harfanger says:

    Thanks for getting it done so quickly. That’s awesome. –harfang/Simone

  10. yralih says:

    Thanks, donovan, et al. Be sure to stay rested, hydrated & fed. Hungry tigers will find double-stuf Oreos in my pocketses.

    ~ Susan

    • Miller says:

      But… but… but… can we at least argue about ellipses here? And .. and blank lines?! What will the blank lines do without us in the interim?! Withdrawal…. I’ll need a lie down….

      Best wishes to the tiger and all those helping. We are grateful… if not patient.

  11. Old_Prof_TX says:

    We do have console access to the machine and have verified that all data is[**are] present.

  12. JeniM says:

    Well done chaps. Don’t understand a word of what you are doing but grateful that you do!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: