DP Site Maintenance – 2015 Saturday 12

September 11, 2015

www.pgdp.net will be down for scheduled maintenance to upgrade the forum software starting Saturday September 12th at 2pm US Eastern. The site should be up again by 8pm US Eastern. During this time the site will be completely unavailable. Please note that although the scheduled outage period is six hours, the expectation is closer to four hours.

We’ll keep this blog post updated with progress during the outage. You can also find us in the pgdp Jabber conference room (pgdp@conference.jabber.org).


Update 2:03pm: Maintenance has started.

Update 3:00pm: Forum conversion progressing as expected. The system is working through 830k forum posts at the moment.

Update 4:00pm: 80% done with the forum posts then the system starts on the 300k private messages — we’re a chatty bunch!

Update 5:00pm: The forum posts are behind us and we’re half way through the private messages.

Update 5:59pm: Database conversion is finished and we’re now busy doing final forum configuration changes and tweaks.

Update 7:02pm: We’re almost done and are putting the finishing touches on it now. We expect to have it completed and the site up well within the next hour.

Update 7:46pm: Site is back up and operational, thank you for your patience!

DP on Twitter

November 7, 2014


Distributed Proofreaders is on Twitter! Now you can keep up with DP’s milestones, new blog posts, and other news. Just follow @DProofreaders to stay in the know.

Distributed Proofreaders is back!

March 8, 2014

Our OS Upgrade/Server Migration is now complete.

Because we have moved to new hardware and a newer version of the operating system, there may be some minor issues left to resolve. If you encounter anything unusual, please let us know in this thread.

If you’re having trouble reaching the site, please try clearing your cache, removing cookies for pgdp.net, and/or restarting your browser.

Thanks to all the squirrels for their assistance in making this major transition happen, and thanks again to our volunteers for waiting patiently while we accomplished this task.

(background music: One Week)

Please check on the DP forums over the next few days for a post with additional details and information once I’ve returned from being on-site.

David (donovan)

Friday Update on our DP Server Status

March 7, 2014

As of 7pm EST Thursday, the new server is in place at the hosting site and squirrels other than donovan have been able to access the system and begin the last stages of configuration, tweaking, validation, etc.

So far, the majority of the site is functioning as expected, with the exception of the wiki, possibly the mailing lists, and some mail functions. Our current focus is to ensure that all the back-end infrastructure is in place and correctly configured.

We’ll continue to provide updates on our progress and expect DP to be back in business fairly soon.

Thanks again for your patience! We’re definitely getting closer!


Brief Update on our Progress

March 5, 2014

We’re making progress!

Prod6 has stopped sulking and is once again communicating with us. As a precautionary measure, we’ve taken full backups of it, and we’re now copying the production data over to the new machine which will become our new production server.

We’ll continue to update you on our progress.

Thanks for your patience and good will!


Distributed Proofreaders Outage Update: Monday, March 3, 2014

March 3, 2014

Unfortunately we have not been able to bring the server (prod6) back online remotely. After evaluation, we have decided to prepare a new server and migrate DP to that machine earlier than originally planned.

As noted previously, this is one of two identical machines graciously donated by Tom Kowal.

Prod6 has (literally) served DP well since 2006 and was slated for replacement in summer 2014 but, given the current situation, making that transition now should minimize the total disruption of service.

The new machine (prod7) has been in preparation since early this morning, already has the OS upgrade on it, and is being prepared/configured with the necessary software. In addition to being more recent hardware, prod7 also gives us two additional drive bays, a battery-backup unit on the drive controller, twice the RAM, and twice as many cpu cores as we currently have with prod6. Those factors combined with a move to the ext4 filesystem should make the server a bit more responsive during heavy load.

Brief stats: 2x Xeon @2.00GHz, 4 cores each. 8G RAM. 8x 300G drives (2 as RAID mirror for the OS, 4 as RAID-5 for the data, and 2 hot spares).

Once the build-out is complete, the new hardware will be transported on-site and we will begin transferring the databases, project files, etc. over. This may take an extended time, as we do not yet know exactly what conditions we will be working under.

Transport and on-site work is tentatively scheduled to take place tomorrow (Tuesday, March 4).

Once again, thanks for your patience. We’re working hard to bring you an improved experience.

David (donovan)

Distributed Proofreaders – Extended Outage

March 1, 2014

Yes, Distributed Proofreaders is experiencing an extended downtime.

The March 1, 2014, OS upgrade took place as scheduled at 10am and went smoothly, completing in just under one hour. However, upon reboot we encountered a boot configuration issue which is preventing the server from getting past the bootloader and back online. Since then, we have tried several approaches to correct this problem remotely (and have come tantalizingly close) but have not yet succeeded. I’ll skip the technical details, but note that it has been incredibly frustrating to be able to see the drives on the controller and the filesystem contents, and yet be unable to fully boot into it.

We do have console access to the machine and have verified that all data is present.

We will continue to work at least part of tonight and tomorrow (Sunday March 2) to resolve the issue remotely, but we are close to exhausting those options.

If we cannot resolve the issue remotely, then I will have to travel to the server, or the server will have to be shipped to me. In either case, we are probably looking at being down for approximately a week.

It’s worth noting that the shipping option does open up the possibility for us to migrate to one of the two servers which were donated to DP by Tom Kowal. They are underutilized in their current role providing an additional layer of off-site backups.

Please check back here for further updates on our progress. I apologize for the inconvenience and thank everyone for their patience during this outage. Please wish us luck!

Thanks to Casey and Brian for their assistance today, and special thanks to the staff at Interserver for their hands-on help.

David (donovan)

%d bloggers like this: