Import from EA Native XML File

[PawelGarbacz]

All,

I need to import a 1GB native XML file to an existing EA model.

When I start the import, I got an 'Out of memory' error and then EA becomes unresponsive.
The error is thrown so early in the process that there is no log file nor info in the import window.

The initial model is of reasonable size - it has around several hundred classes in around two thousand diagrams.
It is stored as an MS SQL Server database.

The import involves only UML classes and is NOT executed as a single transaction.

Is there some setting in EA I can change to run it through?
Or at least some log when I can find more informative message?
(I've already checked ...\AppData\Roaming\Sparx Systems\EA\DBError.txt, but there is nothing of use there.)

My sort of worst dream is that the error is due to the fact that EA is a 32bit app and that the XML file takes up more than 2GB when expanded in memory. In this case I suspect I cannot do anything about it except for splitting the file into smaller chunks.

I am running EA on a Windows Server 2016 with 40GB of RAM, so it does not seem that the physical memory is exhausted by the process. Btw., I am using EA 15.2.1554 Corporate Edition.

Thanks in advance for your help,

Pawel Garbacz

[qwerty]

IIRC there's a "one transaction" check mark which you should probably uncheck. Just a gues, though.

q.

[PawelGarbacz]

Thanks, qwerty, but I've already done that.


Pawel

[Geert Bellekens]

1 gb is extremely large for a model with only a few hundred classes and 2000 diagrams.
Importing large xml files is always problematic.
I would try to see what causes this giant size and then see if you can somehow reduce it.

Also maybe try the other xml formats such as xmi 1.1 or xmi 2.1

Geert

[qwerty]

Where did you get the XML from? Any chance to export it in chunks (packages)?

q.

[Eve]

There is also a Native XML complete project export/import option that makes sure the file sizes can be handled effectively.

[Paolo F Cantoni]

There is also a Native XML complete project export/import option that makes sure the file sizes can be handled effectively.

Which, as of b1554, seems to be working correctly.     I use it often - in preference, wherever possible.

Paolo

[PawelGarbacz]

There is also a Native XML complete project export/import option that makes sure the file sizes can be handled effectively.

Thanks for this.

But importing the complete model with my 1GB file resulted in this:

DAO.Database [0x00000bdb]

System resource exceeded.

Context:SQL: Select Diagram_ID,Package_ID,Name,ParentID, Diagram_Type, TPos, ea_guid, Locked from t_diagram Where  Package_ID = 0 AND  ParentID <> 0 Order By Package_ID, ParentID

[PawelGarbacz]

I would probably split the file into chunks and use the package level native XML import.

[PawelGarbacz]

@Geert Bellekens
As for:

1 gb is extremely large for a model with only a few hundred classes and 2000 diagrams.

I might have misspoken - we need to load 1GB worth of XML data into an existing model that before the import has several hundred classes in around two thousand diagrams.

As for:

Also maybe try the other xml formats such as xmi 1.1 or xmi 2.1

We chose not to use XMI imports because we don't have the python tools to create them - compared to XMI creating EA native XML file is easy.
Also https://sparxsystems.com/enterprise_architect_user_guide/15.1/model_publishing/native_xml_file.html recommends using the native format over XMI mainly because of performance.

[Geert Bellekens]

So it's handcrafted "native" xml?
Not sure if that would have been my choice.

But since it's handcrafted I guess you can cut it up in smaller chunks to import.

I usually solve this type of thing differently. In your case I would probably write my own little import script based on whatever format you have the data in (xml, csv, excel,...)
That is also much more flexible if you ever need to synchronize the model with the external data.

Geert

[PawelGarbacz]

@Geert Bellekens
Yes, this is the current plan - to split the data into chunks.

Actually, the final dataset will be bigger - it is the model of OS Open Names > https://www.ordnancesurvey.co.uk/business-government/products/open-map-names.

I am not sure what you have in mind when you mention the script.
Do you mean writing directly to the underlying database using SQL queries?

(Obviously, we tried EA Interop, but it is ages too slow for this size.)

[Geert Bellekens]

I was thinking about a script in EA that uses the API.

I'm not sure if that will be much slower than importing via native xml.
You can get reasonable speeds when importing elements, but you have to be careful and avoid the pitfalls.
I had some pretty good results importing stuff from excel using VBA

Geert