E-NDMS

E-NDMS On the Move

What is Changing?

E-NDMS will soon move from its existing hosting infrastructure to the Amazon Elastic Compute Cloud. In the process, E-NDMS will be reduced to one site from the two that we have now and will run what you know as the BETA site for everyone.

Why is it Changing?

Today, E-NDMS runs on 3 servers co-located in a data center in Tulsa. 2 of those servers are nearing the end of their useful life, especially as E-NDMS continues to grow. E-NDMS is quickly closing in on 1,000,000 Model Services with Follow-up Contacts trailing close behind in the 850,000 range. To process that much data efficiently given the concurrency patterns of its users, E-NDMS is going to need more power.

E-NDMS also suffers from what I would term seasonal traffic patterns. While we encourage specialists to enter data weekly or even daily, we all know that doesn't always happen. As a result, there are a couple of times each year where E-NDMS traffic spikes dramatically. For the last 2 years, the most critical time of year has been May, but there is another longer-running uptick in traffic during the first couple of months of a new school year. As previous blog posts have revealed, the system has wilted under the pressure of the May traffic spike more than once.

When E-NDMS launched 6 years ago, the only viable option was to purchase hardware and co-locate it in a data center. Leasing servers or outsourcing the hosting was simply too expensive at the time. Since then, a technology called virtual private servers (VPS) has come onto the scene and drastically decreased hosting fees. Many of the new sites we host run on VPS systems.

VPS systems offer us tremendous flexibility. If we need more power, we can upgrade with just a flick of a switch. If hardware fails, the VPS provider migrates our virtual server to another physical server without any effort on our part. The limiting factor for E-NDMS has been the amount of power available and the performance of the storage subsystems in hosting a reasonably large database.

The Amazon Elastic Compute Cloud will give us the flexibility, convenience, and reliability of a typical VPS hosting arrangement, but it also allows us to right-size the VPS performance to current traffic patterns. So if there is a bottleneck in the application server during May, we can upgrade its resources or load balance across two systems until the traffic spikes subside. We can even schedule the availability of additional resources to handle the daily lunch time traffic spike.

How does this Affect You?

It's possible your network administrator had to open access to E-NDMS through a firewall. If that was the case, you will likely need to have the administrator change the IP address. As soon as that new address is available, I will post it on the site so you can work with your local network administrator to verify your access.

Other than that, I don't think John will mind me including a quote from his recent e-mail:

Re: the Test site - wow - that is fast - I know I'm the only one on there, but I ran some large reports and they were generated very quickly. I ran all sorts of reports and large downloads, and no problems.

The goal is to create a system that matches current E-NDMS performance under ideal conditions, but it appears that we may exceed that goal in our initial launch on Amazon EC2.

Please feel free to contact me in the comments or at endms@jag.org if you have any questions about the upcoming move.

Posted at 10:22AM Oct 29, 2008 by Jason Koeninger in System Administration  |  Comments[11]