Server4: "Read-only file system" error [Fixed]

[Ronnie]

Unexpectedly, server4 has apparently changed its server's file system to "read only" mode... And every operation on the server (http, email, etc.) is causing "read-only file system" error messages, and are down. Although I personally have never seen this error in over 5+ years, we are working through it by completing a FSCK (File system check) and a reboot. Hopefully this will bring things back to normal.

If there is other corruption or damage to the main hard drive, we will promptly bring a new server online and immediately begin copying our offsite backups from our backup file server to the new server and begin restoring sites, to get things back to normal that way. We are hoping the FSCK and a system reboot is enough, as that is the preferred method. If we need to restore from the offsite backup, it will be slightly older data.

We are extremely sorry for any trouble this has or is causing any of our clients, and we assure you we are working as dilligently as possible to get everything fixed. We sincerely apologize for any trouble this has caused you, our loyal customers, and hope to have things 100% back to normal asap.

I will post an update in this thread as I know more and have an ETA.

[Ronnie]

06/29/2006: 3:05:46 PM,

Data Center tech is in the process of rebooting the server and will perform a manual file system check (FSCK). This may take 10 minutes, or as last as an hour or more. We will post an update on the progress when it has been completed.

[Ronnie]

File system check (FSCK) has been completed on server4, and has repaired a few damaged blocks. The server is now online again, and all services appear to be running normally! Please submit a ticket if you are still having any trouble.