E-NDMS

Stability Problems Explained

We've had several failures in the last few days. They include:

• Friday: One of the key servers in our 3 server setup of E-NDMS froze and had to be restarted. The outage was extended because it did not start without manual intervention.
• Monday and Tuesday: Our mail servers appeared to be under some kind of DoS attack both days which likely made logins so slow as to be intolerable.
• Tuesday: The authentication database of the BETA system was corrupted.

We are actively taking steps to address these issues.

• The server that has frozen almost monthly for the past quarter is in the process of being replaced.
• Mail servers will be aggressively filtered if we see unusual traffic activity. Due to the asynchronous nature of e-mail, it's not always obvious when something is going wrong, but we're putting tools in place to keep a better eye on performance.
• As to the authentication database issue, this is one that has plagued E-NDMS as traffic has continued to grow. We are actively working on a change to the login procedure that will eliminate that database completely from the authentication procedure. It will take us a few weeks to make some of that work available so please bear with us as we try to keep everything up and running in the meantime.

Thanks for your patience as we deal with these issues.

Posted at 02:35PM Sep 18, 2007 by Jason Koeninger in Maintenance  |  Comments[2]