At the moment, I'm writing a JavaScript to read attributes from a CSV file to add them to table objects.
Running EA v16.1 on a local pc using a local QEA file for development.
Importing the first hundreds is no problem, it takes about 23 sec for the first 500.
The next 500 take already 50 secs while 3500 - 4000 take 14 mins.
Looking at the memory consumption of the EA process shows an increasing use of memory accumulating from 500 MB to 2-3 GB together with a high energy consumption level.
I've already tried some 'tricks' to minimize / optimize the memory usage:
- Release the objects used (e.g. curElement = null;)
- Implement SQL result parsing using string manipulation instead of using an XML COM Object from EAScriptLib
- Generate string for logging instead of using object references
- Release the objects used by the CSV import module from EAScriptLib
- Only show progress for every 500 imports instead of every import row
These 'tricks' have already improved the import of 8000 lines from 4.5 hours to 1 hour but I still think this is too much.
One last option I have is to split the CSV files into smaller chunks, but this still doesn't really solve the issue as it will increase the manual processing load.
Any suggestions on how to tackle this issue?
Maybe a tool to analyse the memory usage of the JavaScript?