ITA Cross-System Team
We received a lot of rain this last winter. Some of us even had our roof leak (a terrible experience). Did you know that IT applications can also spring a leak?
This month the ITA Spotlight shines on an ITA cross-system team that had to diagnose and fix a serious memory leak that affected over 3,500 users (Documentum, FMS and LATAX Support Team, CGI, and OpenText). The ITA Documentum Team led an intensive effort spanning many nights and weekends to troubleshoot, find the root cause and resolution actions necessary to fix the critical issues preventing attachments in FMS.
As part of normal IT operations, software versions in Documentum and web apps servers (Tomcat, DA, and WebTop) were upgraded to maintain currency and security. Despite careful planning, testing and monitoring of the software prior to the upgrade in production, new features introduced in Java V.8 lead to Java memory leaks; when users tried to upload/download invoice documents in FMS. A "memory leak" is a condition caused by a program that does not free up the extra memory it allocates.
When memory is allocated, but not released or deallocated, a memory leak occurs (the memory has leaked out of the computer). If too many memory leaks occur, they can use up all the available memory in the computer, and bring the system to a halt - which is what happened to FMS and its 3,500 users. This memory leak was particularly hard to troubleshoot since all tests prior to the move to production had performed perfectly, and environments, such as Java, usually allocate and release memory automatically.
To troubleshoot and pinpoint the root cause of the memory allocation and release issue, the Documentum Support Team led our vendor partners (CGI and OpenText technical support engineers) through dozens of hours of problem diagnosis and determination efforts involving system traces, collecting and analyzing various log files and error messages, recompiling the FMS-Documentum interface code, isolating and running FMS-Documentum interface on a standalone server, and tweaking the server memory settings. Through the team's hard work, the cause was finally pinpointed and the steps to stop the memory leak discovered.
The ITA Executive team would like to recognize and thank the following team members and vendors for their hard work, persistence, and dedication in resolving a very complicated, elusive hardware and software issue that affected thousands of City employees
ITA Documentum Support Team:
Sohn Nguyen, Nima Asgari, Kevin Cheng
ITA FMS and LATAX Support Team:
Jose Alvarez, Bill Aguilar, Sam Vardarajan
CGI Technical Support Team:
David Schulman, Tyler LeBeau, Brian Berg
OpenText Documentum Technical Support Team:
Gary Bluhm, David Prado, Tony Qiu, Tom Steele