Author Topic: Memory leaks in JavaScript  (Read 3096 times)

ea0522

  • EA User
  • **
  • Posts: 134
  • Karma: +5/-0
    • View Profile
Memory leaks in JavaScript
« on: November 11, 2024, 11:56:30 pm »
At the moment, I'm writing a JavaScript to read attributes from a CSV file to add them to table objects.
Running EA v16.1 on a local pc using a local QEA file for development.
Importing the first hundreds is no problem, it takes about 23 sec for the first 500.
The next 500 take already 50 secs while 3500 - 4000 take 14 mins.
Looking at the memory consumption of the EA process shows an increasing use of memory accumulating from 500 MB to 2-3 GB together with a high energy consumption level.

I've already tried some 'tricks' to minimize / optimize the memory usage:
  • Release the objects used (e.g. curElement = null;)
  • Implement SQL result parsing using string manipulation instead of using an XML COM Object from EAScriptLib
  • Generate string for logging instead of using object references
  • Release the objects used by the CSV import module from EAScriptLib
  • Only show progress for every 500 imports instead of every import row

These 'tricks' have already improved the import of 8000 lines from 4.5 hours to 1 hour but I still think this is too much.
One last option I have is to split the CSV files into smaller chunks, but this still doesn't really solve the issue as it will increase the manual processing load.

Any suggestions on how to tackle this issue?
Maybe a tool to analyse the memory usage of the JavaScript?

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13257
  • Karma: +554/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Memory leaks in JavaScript
« Reply #1 on: November 12, 2024, 03:59:16 pm »
If you share the code we might give you some useful hints.
Without seeing the code it's hard to provide guidance.

Geert

ea0522

  • EA User
  • **
  • Posts: 134
  • Karma: +5/-0
    • View Profile
Re: Memory leaks in JavaScript
« Reply #2 on: November 12, 2024, 08:58:26 pm »
Hai Geert,
Thanks for the quick reply.
My first thought was there might be some debugging tools giving insight into this issue.
But if you want to have a look at my code, I made a smaller test script with the same functionality, its included at the end of this post.
The results it delivered are:

Quote
Started TestImportPerformanceCSV at 2024-11-12 09:15:25!
2024-11-12 09:15:31 Processed row[ 1 ]
2024-11-12 09:16:00 Processed row[ 500 ]
2024-11-12 09:16:44 Processed row[ 1000 ]
2024-11-12 09:17:39 Processed row[ 1500 ]
2024-11-12 09:18:49 Processed row[ 2000 ]
2024-11-12 09:20:31 Processed row[ 2500 ]
2024-11-12 09:22:48 Processed row[ 3000 ]
2024-11-12 09:25:49 Processed row[ 3500 ]
2024-11-12 09:30:56 Processed row[ 4000 ]
2024-11-12 09:37:37 Processed row[ 4500 ]
2024-11-12 09:44:01 Processed row[ 5000 ]
2024-11-12 09:46:34 Processed row[ 5500 ]
2024-11-12 09:50:12 Processed row[ 6000 ]
2024-11-12 09:56:18 Processed row[ 6500 ]
2024-11-12 10:03:33 Processed row[ 7000 ]
2024-11-12 10:12:28 Processed row[ 7500 ]
2024-11-12 10:24:41 Processed row[ 8000 ]
2024-11-12 10:39:43 Processed row[ 8500 ]
Finished TestImportPerformanceCSV at 2024-11-12 10:51:39!

Code: [Select]
//[group=BaatDiagramScripts]
!INC Local Scripts.EAConstants-JavaScript
!INC EAScriptLib.JavaScript-Dialog
!INC EAScriptLib.JavaScript-CSV
!INC EAScriptLib.JavaScript-Logging

/*
 * This code has been included from the default Diagram Script template.
 * If you wish to modify this template, it is located in the Config\Script Templates
 * directory of your EA install path.
 *
 * Script Name: TestImportPerformanceCSV
 * Author: J. de Baat
 * Purpose: Import the Attribute information from a CSV file into the repository
 * Date: 12-11-2024
 *
 */

const IDBTaggedValueIDBKey       = "IDBKey"; // Definition of the TV Property used as IDB Key in DataBricks

const ImExColumnName             = "Name"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnNotes            = "Notes"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnClassGUID        = "CLASSGUID"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnClassType        = "CLASSTYPE"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnStereotype       = "Stereotype"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnConnectorGUID    = "CONNECTORGUID"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnConnectorID      = "Connector_ID"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnConnectorType    = "Connector_Type"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnStartObjectID    = "Start_Object_ID"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnEndObjectID      = "End_Object_ID"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnCSOClassGUID     = "CSO_CLASSGUID"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnCTOClassGUID     = "CTO_CLASSGUID"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnAttributeName    = "AttributeName"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnAttributeType    = "AttributeType"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnIsCollection     = "IsCollection"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnIsOrdered        = "IsOrdered"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnIsStatic         = "IsStatic"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnLength           = "Length"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnPrimaryKey       = "PrimaryKey"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnForeignKey       = "ForeignKey"; // Definition of the Excel Column Name used as key for the ImEx Modules
const ImExColumnUnique           = "Unique"; // Definition of the Excel Column Name used as key for the ImEx Modules

const strTaggedValuesPrefix      = "TAG_";
const IDBKeyPrefix               = "IDB_";
const IDBKeyColumnName           = "Object_ID"; // Definition of the DB Keys to get the IDB key from a TaggedValue
const IDBKeyTable                = "t_objectproperties"; // Definition of the DB Keys to get the IDB key from a TaggedValue

const CSVFilterString            = "CSV Files (*.csv;*.txt)|*.csv;*.txt|All Files (*.*)|*.*||";

// The level to log at
var   LOGLEVEL = LOGLEVEL_ERROR; // Choose from: LOGLEVEL_ERROR, LOGLEVEL_WARNING, LOGLEVEL_INFO, LOGLEVEL_DEBUG, LOGLEVEL_TRACE
let   intImportRowNr = 0;
let   intShowDelta   = 500;
let   intShowRowNr   = 0;
let   strMessage     = "";

/*
 * Handle the CSVImport for importing Attributes
 */
function IDBImportAttributes( )
{

try {

// Get the CSV fileName for this Import session, true for readonly.
let curCSVFileName = DLGOpenFile( CSVFilterString, 1 );
if ( ( curCSVFileName == null ) || ( curCSVFileName == "" ) ) {
LOGError( "DLGOpenFile could NOT Get curCSVFileName!" );
return " IDBImportAttributes DLGOpenFile could NOT Get curCSVFileName!!!";
}

// Start the CSVImport for this session
LOGTrace( " Started CSV Import for curCSVFileName " + curCSVFileName + "!!!" );
CSVIImportFile( curCSVFileName, true );

// Return "" as successful result
return "";

} catch (catch_err) {
LOGError( " catched error " + catch_err.message + "!" );
return "IDBImportAttributes catched error " + catch_err.message + "!";
}

}

/*
 * Process the Attributes found in the CSV Import file
 */
function OnRowImported( theRow )
{

var curElement as EA.Element;

let importRow = 0;

try {

importRow = importCurrentRow;
intImportRowNr++;
strMessage = " importCurrentRow(" + importRow + ").length = " + importCurrentRow.length + ", Action = " + importCurrentRow[0] + "!!!";
LOGTrace( strMessage );

// Find and process the elements available in this importRow
curElement = IDBFindCSVElementByTVIDBKey( IDBTaggedValueIDBKey );
if ( curElement != null ) {
// Process element found
strMessage = "Processing curElement.Name = " + curElement.Name + ", ElementID = " + curElement.ElementID + ", ParentID = " + curElement.ParentID + ", PackageID = " + curElement.PackageID + "!!!";
LOGTrace( strMessage );

// Process the Attributes for curElement found
let curResult = IDBProcessElementAttribute( curElement );
// Session.Output( "Processed row[ " + importRow + " ] for " + curElement.Name + " at " + _LOGGetDisplayDate() + ", curResult = "  + curResult + "!" );
if ( intImportRowNr >= intShowRowNr ) {
strMessage = _LOGGetDisplayDate() + " Processed row[ " + intImportRowNr + " ] for " + curElement.Name + ", curResult = "  + curResult + "!";
Session.Output( strMessage );
intShowRowNr += intShowDelta;
}
} else {
strMessage = _LOGGetDisplayDate() + "Processed row[ " + intImportRowNr + " ] for null curElement!";
Session.Output( strMessage );
}

} catch (catch_err) {
LOGError( " catched error " + catch_err.message + "!" );
}

// Free up memory
curElement        = null;
currentLine       = null;
currentLineTokens = null;
}

/*
 * Process the Attributes for curElement found with information as provided
 */
function IDBProcessElementAttribute( theElement )
{

// Cast theElement to EA.Element so we get intellisense
var curElement     as EA.Element;
var curAttributes  as EA.Collection;
var curAttribute   as EA.Attribute;

// try {

// Get and check the ImExCSVColumnAttributeName parameter
let newAttributeName = CSVIGetColumnValueByName( ImExColumnAttributeName );
if ( ( newAttributeName == null ) || ( newAttributeName == "" ) ) {
// Return null because not found
return "IDBProcessElementAttribute: newAttributeName not available so Attribute NOT created!!!";
}

// Check all Attributes in theTypePackage whether the requested theAttributeName already exists
curElement    = theElement;
curAttributes = curElement.Attributes;
curAttribute  = IDBGetCollectionObjectByName( curAttributes, newAttributeName );

// Create new Attribute when it does not exist yet
if ( curAttribute == null ) {

// Get and check the ImExCSVColumnAttributeType parameter
let newAttributeType = CSVIGetColumnValueByName( ImExColumnAttributeType );
if ( ( newAttributeType == null ) || ( newAttributeType == "" ) ) {
// Use default value because not found
newAttributeType = "string";
LOGTrace("Use default value string for newAttributeType because not found!!!" );
}

// Create the new curAttribute
curAttribute = curAttributes.AddNew( newAttributeName, newAttributeType );
curAttribute.Update();
curAttributes.Refresh();
}

strMessage = "Started curAttribute[ " + curAttribute.Name + " ] !";
LOGTrace( strMessage );

// Update the parameters for curAttribute
curAttribute = IDBUpdateAttributeProperties( curAttribute );
curAttributes.Refresh();

let curResult = "IDBProcessElementAttribute: Processed " + curAttribute.Name + " for Type " + curAttribute.Type + " !!!";

// Free up memory
curElement    = null;
curAttribute  = null;
curAttributes = null;

return curResult;
//
// } catch (catch_err) {
// LOGError( " catched error " + catch_err.message + "!" );
// return "ERROR IDBProcessElementAttribute: catched error " + catch_err.message + "!";
// }

}

/*
 * Update theAttribute with information as provided
 */
function IDBUpdateAttributeProperties( theAttribute )
{

// Cast theAttribute to EA.Attribute so we get intellisense
var curAttribute as EA.Attribute;

// try {
curAttribute = theAttribute;

// Check information found
let newAttributeNotes = CSVIGetColumnValueByName( ImExColumnNotes );
if ( ( newAttributeNotes == undefined ) || ( newAttributeNotes == "" ) ) {
// Define newAttributeNotes because not found
newAttributeNotes = curAttribute.Name + " created by the TestImportPerformanceCSV script for Type " + curAttribute.Type + " on " + _LOGGetDisplayDate() + ".";
}

// IsCollection indicates if the current feature is a collection or not. If the attribute represents a database column this, when set, represents a ForeignKey.
let newAttributeIsCollection = CSVIGetColumnValueByName( ImExColumnIsCollection );
if ( ( newAttributeIsCollection == undefined ) || ( newAttributeIsCollection == "" ) ) {
// Get newAttributeIsCollection from ImExCSVColumnForeignKey because ImExCSVColumnIsCollection not found
newAttributeIsCollection = CSVIGetColumnValueByName( ImExColumnForeignKey );
}
if ( ( newAttributeIsCollection != undefined ) && ( newAttributeIsCollection != "" ) ) {
curAttribute.IsCollection = newAttributeIsCollection;
}

// IsOrdered indicates if a collection is ordered or not. If the attribute represents a database column this, when set, represents a PrimaryKey.
let newAttributeIsOrdered = CSVIGetColumnValueByName( ImExColumnIsOrdered );
if ( ( newAttributeIsOrdered == undefined ) || ( newAttributeIsOrdered == "" ) ) {
// Get newAttributeIsOrdered from ImExCSVColumnPrimaryKey because ImExCSVColumnIsOrdered not found
newAttributeIsOrdered = CSVIGetColumnValueByName( ImExColumnPrimaryKey );
}
if ( ( newAttributeIsOrdered != undefined ) && ( newAttributeIsOrdered != "" ) ) {
curAttribute.IsOrdered = newAttributeIsOrdered;
}

// IsStatic indicates if the current attribute is a static feature or not. If the attribute represents a database column this, when set, represents the 'Unique' option.
let newAttributeIsStatic  = CSVIGetColumnValueByName( ImExColumnIsStatic );
if ( ( newAttributeIsStatic == undefined ) || ( newAttributeIsStatic == "" ) ) {
// Get newAttributeIsStatic from ImExCSVColumnUnique because ImExCSVColumnIsStatic not found
newAttributeIsStatic = CSVIGetColumnValueByName( ImExColumnUnique );
}
if ( ( newAttributeIsStatic != undefined ) && ( newAttributeIsStatic != "" ) ) {
curAttribute.IsStatic = newAttributeIsStatic;
}

// Length indicates the attribute length, where applicable.
let newAttributeLength = CSVIGetColumnValueByName( ImExColumnLength );
if ( ( newAttributeLength != undefined ) && ( newAttributeLength != "" ) ) {
curAttribute.Length = newAttributeLength;
}

strMessage = "Started newAttribute[ " + curAttribute.Name + " ] for IsOrdered " + curAttribute.IsOrdered + ", newAttributeIsOrdered " + newAttributeIsOrdered + " !";
LOGTrace( strMessage );

// Update the parameters for curAttribute
curAttribute.Notes = newAttributeNotes;
curAttribute.Update();

strMessage = "Updated newAttribute[ " + curAttribute.Name + " ] for Type " + curAttribute.Type + ", IsCollection " + curAttribute.IsCollection + " !";
LOGTrace( strMessage );
//
// } catch (catch_err) {
// LOGError( " catched error " + catch_err.message + "!" );
// }

return curAttribute;
}

/*
 * Get the requested CollectionObject from theCollectionObjects using theObjectName as parameter
 */
function IDBGetCollectionObjectByName( theCollectionObjects, theObjectName )
{
// Cast theCollectionObjects to EA.Collection so we get intellisense
var curCollectionObjects as EA.Collection;
var curCollectionObject  as EA.Element;

// Check all Elements in theCollectionObjects whether the requested curElement is already defined
curCollectionObjects = theCollectionObjects;

// Loop over curCollectionObjects to find theObjectName
let curCollectionObjectsCount = curCollectionObjects.Count;
for ( var i = 0 ; i < curCollectionObjectsCount ; i++ )
{
curCollectionObject = curCollectionObjects.GetAt( i );
if ( curCollectionObject.Name === theObjectName ) {
strMessage = "Found CollectionObject ( " + curCollectionObject.Name + " ) as part of " + curCollectionObjectsCount + " CollectionObjects!!!";
LOGTrace( strMessage );

// Clean up memory
curCollectionObjects = null;

return curCollectionObject;
}
// strMessage = " TESTED CollectionObject ( " + curCollectionObject.Name + " ) against theObjectName " + theObjectName + " !!!";
// LOGTrace( strMessage );
}

// Clean up memory
curCollectionObjects = null;
curCollectionObject  = null;

// theObjectName not found as part of curCollectionObjects
strMessage = "DID NOT FIND theObjectName ( " + theObjectName + " ) as part of " + curCollectionObjectsCount + " CollectionObjects!!!";
LOGTrace( strMessage );
return null;

}

/*
 * Find curElement using the TaggedValue in theTVIDBKey Property
 */
function IDBFindCSVElementByTVIDBKey( theTVIDBKey )
{

var curElement         as EA.Element;
let theCSVColumnIDBKey  = "";
let curElementIDBKey    = "";
let curElementObject_ID = "";

// Find the Element identified by theTVIDBKey
theCSVColumnIDBKey  = strTaggedValuesPrefix + theTVIDBKey;
curElementIDBKey    = CSVIGetColumnValueByName( theCSVColumnIDBKey );
curElementObject_ID = IDBGetObjectIDFromTVProperty( theTVIDBKey, curElementIDBKey );
curElement          = GetElementByID( curElementObject_ID );
if ( curElement != null ) {

// Return curElement found
strMessage = "( " + curElementIDBKey + " ) found curElement.Name = " + curElement.Name + " and curElementObject_ID " + curElementObject_ID + "!!!";
LOGTrace( strMessage );
return curElement;
}

//  Nothing found so return null
strMessage = "( " + theTVIDBKey + " ) did not find curElement for curElementIDBKey = " + curElementIDBKey + " and curElementObject_ID " + curElementObject_ID + "!!!";
LOGTrace( strMessage );
return null;

}

/*
 * Get the GUID related to the theTVProperty with value theTVValue from the t_objectproperties table
 */
function IDBGetObjectIDFromTVProperty( theTVProperty, theTVValue )
{

// Get the Object_ID registered for theTVProperty with value theTVValue
let IDBKeyWhereClause   = IDBKeyTable + ".Property = '" + IDBTaggedValueIDBKey + "'"
+ " and " + IDBKeyTable + ".Value = '" + theTVValue + "'";
let curElementObject_ID = IDBGetFieldValueString( IDBKeyColumnName, IDBKeyTable, IDBKeyWhereClause );
if ( curElementObject_ID.length > 0 ) {

// Return the curElementObject_ID for curElement found
strMessage = "( " + theTVValue + " ) found curElementObject_ID = " + curElementObject_ID + "!!!";
LOGTrace( strMessage );
return curElementObject_ID;
}

//  Nothing found so return null
strMessage = "( " + theTVValue + " ) did not find any curElementObject_ID for theTVProperty " + theTVProperty + "!!!";
LOGTrace( strMessage );
return null;

}

/**
 * Queries the repository database for the first field value whose corresponding row matches the
 * specified WHERE clause.
 *
 * @param[in] columnName (string) The column name whose field value will be queried for
 * @param[in] table (string) The table that the column resides in
 * @param[in] whereClause (string) The SQL where clause that the query will use to select the
 * appropriate row
 *
 * @return A String representing the requested field value
 */
function IDBGetFieldValueString( columnName /* : String */, table /* : String */, whereClause /* : String */ ) /* : String */
{
var stringValue = "";

// Construct and execute the querySQL and parse the queryResult
let querySQL               = "SELECT " + columnName + " FROM " + table + " WHERE " + whereClause;
let queryResult            = Repository.SQLQuery( querySQL );
let queryResultParts       = queryResult.split( columnName );
let queryResultPartsLength = queryResultParts.length;

// Check the queryResult for the stringValue found
if ( queryResultPartsLength == 3 )
{
if ( queryResultParts[1].startsWith(">") ) {
stringValue = queryResultParts[1].substring( 1, ( queryResultParts[1].length - 2 ) );
}
strMessage = "Repository found queryResultParts[" + 1 + "] = " + queryResultParts[1] + ", stringValue = " + stringValue + "!!!";
LOGTrace( strMessage );
}

// Free up memory
queryResult            = null;
queryResultParts       = null;
queryResultPartsLength = null;

return stringValue;
}

/*
 * Diagram Script main function
 */
function TestImportPerformanceCSV()
{
// Show the script output window
Repository.EnsureOutputVisible( "Script" );

Session.Output( "======================================= Started TestImportPerformanceCSV at " + _LOGGetDisplayDate() + "!" );

try
{

// Get the Attributes from import file
let curResult = IDBImportAttributes();
if ( curResult.length > 0 ) {
LOGError( curResult );
} else {
LOGInfo( "Finished processing!" );
}
}
catch(catch_err)
{
LOGError(" found Error: " + catch_err + "!!!" );
}

Session.Output( "======================================= Finished TestImportPerformanceCSV at " + _LOGGetDisplayDate() + "!" );
}

TestImportPerformanceCSV();

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13257
  • Karma: +554/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Memory leaks in JavaScript
« Reply #3 on: November 12, 2024, 09:25:23 pm »
Hi,

I had a quick look at your code, and I do have some recommendations

- for each row you are processing, you are getting the object again using a database query on the tagged values.
In your case I would process the whole CSV, get all the needed object keys, and do a single query to the database. Then make a dictionary with key, and object ID.
Don't get the EA object (getElementByID before you need it)

- for each attribute, you use the element.Attributes collection to figure out if it exists already. Don't do that. Use a database query (using the objectID you got earlier) to figure out if the attribute exists. For even better performance you might want to try to minimize the number of queries you need to do by making a single query that returns all existing attributes that are referenced in your csv in one go.

Remember you only need an element object, and the Element.Attributes collection if you are adding or deleting an attribute.

Database queries are quick (especially if you can use a single query for a whole set).
EA API objects (and especially EA.Collections) are slow.

I think using string manipulation instead of XML Dom processing is trying to optimize the part of the code that is not the bottleneck.
I did experiment once with direct database queries (instead of using Repository.SQLQuery) That was actually also major improvement over my existing code, but has a serious downside of having to know the database type and connection string.
It is not that important as avoiding the API objects and collections though.

Geert

ea0522

  • EA User
  • **
  • Posts: 134
  • Karma: +5/-0
    • View Profile
Re: Memory leaks in JavaScript
« Reply #4 on: November 12, 2024, 10:40:32 pm »
Hai Geert,
Thanks for these recommendations to improve the performance of my script.

However, I cannot see how it explains the exponential increase in processing times needed as the numbers grow.
That's why I asked for suggestions to detect a memory leak as it looks like the process is increasing the amount of memory needed for every line in the csv.

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13257
  • Karma: +554/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Memory leaks in JavaScript
« Reply #5 on: November 12, 2024, 10:51:19 pm »
one explanation might be, that as you are adding attributes, you also need more time to check if the attribute already exists.

If there are 0 attributes, then that is instant.
If you have to iterate over 2000 attributes to check if an attribute exists already, that is going to take more time.

That is why I would try to do those check beforehand.
Load everything you need into dictionaries that you can quickly access.
And don't initiate any object you don't really need.

Geert