← Previous Change
Next Change →
side by side
lines around each change
Show the changes in full context
White space changes
Sep 25, 2010, 8:51:07 PM (
= About the recent downtime (2010-09-24 and 25) =
You certainly have noticed that trac-hacks.org was down for more than 24 hours. I apologize for any inconvenience caused by this incident. Here is what happened:
trac-hacks.org is hosted as a virtual server (based on [http://linux-vserver.org linux-vserver]) on the physical server of the [http://madwifi-project.org MadWifi project]. That server suffered from an issue that caused the host server to more or less hang. Our monitoring notified me of the problems by SMS. At that time I was on a holiday trip without internet access, so I asked my "remote hands" to reboot the server. That indeed helped to fix the original issue and the monitoring reported the server to be up and running again. Unfortunately the trac-hacks.org vserver encountered an issue during startup, preventing it from coming up normally, which remained unnoticed by the monitoring - and me. I only noticed it when I came online this evening after returning home.
On the positive side, this incident pointed out how to improve the monitoring and the startup process of the hosted vservers to prevent these issues in the future. These things will be taken care of asap.