E-NDMS
A Few Problems Our First Day
We've run into a couple of minor and one major problem on our first day at Amazon.
- Some data integrity code didn't work correctly on the upgraded JAG database and was causing problems with any function that attempted to modify a roster. That was corrected by mid-day.
- The management reports were not running due to a problem with security permissions. That has been corrected, and they should now work.
- Most importantly, we've had some stability problems. If you get a System Down message while entering data, please wait at least 2-3 minutes without attempting to change pages, and then click Reload on your browser. In most cases, this should allow you to save your data without losing any work.
Continue reading if you want the technical details.
During the day today, I noticed high memory consumption from our Transaction server. After further investigation, it appears that the upgraded copy of the ODBC driver for the JAG database has a serious memory leak. This caused one of the outages today. There also appear to be some other bugs in the ODBC driver that are affecting stability as well.
The ODBC driver has caused us problems throughout the life of E-NDMS, and I'm sure all of you are as ready to see those problems go away as I am. So, I'm going to rewrite the E-NDMS Transaction layer in Java/JDBC using components I know can be trusted to be stable and reliable.
Here's the catch. This involves porting just shy of 30,000 lines of C++ code to Java. In other words, it will take some time. As soon as I get an idea how long each module will take to port, I'll make a new entry here to provide a target date.
Posted at 12:14AM Dec 02, 2008 by Jason Koeninger in Maintenance | Comments[2]
Just as an FYI, I was entering some Model Services data today. The system was very zippy, much improved! I had no problems signing on, and when I went to enter the second and third days' data, the roster I had selected for the first day was still selected. This was a real time saver and convenience.
As an old IT person, I don't envy your task of rewriting 30000 lines of C++. But take your time and do it right. Quality is worth waiting for.
Posted by jim hanak on December 02, 2008 at 09:28 AM CST #
Posted by Jason on December 02, 2008 at 11:11 AM CST #