Yes, Distributed Proofreaders is experiencing an extended downtime.
The March 1, 2014, OS upgrade took place as scheduled at 10am and went smoothly, completing in just under one hour. However, upon reboot we encountered a boot configuration issue which is preventing the server from getting past the bootloader and back online. Since then, we have tried several approaches to correct this problem remotely (and have come tantalizingly close) but have not yet succeeded. I’ll skip the technical details, but note that it has been incredibly frustrating to be able to see the drives on the controller and the filesystem contents, and yet be unable to fully boot into it.
We do have console access to the machine and have verified that all data is present.
We will continue to work at least part of tonight and tomorrow (Sunday March 2) to resolve the issue remotely, but we are close to exhausting those options.
If we cannot resolve the issue remotely, then I will have to travel to the server, or the server will have to be shipped to me. In either case, we are probably looking at being down for approximately a week.
It’s worth noting that the shipping option does open up the possibility for us to migrate to one of the two servers which were donated to DP by Tom Kowal. They are underutilized in their current role providing an additional layer of off-site backups.
Please check back here for further updates on our progress. I apologize for the inconvenience and thank everyone for their patience during this outage. Please wish us luck!
Thanks to Casey and Brian for their assistance today, and special thanks to the staff at Interserver for their hands-on help.