Slow import of large codebase

[mikaele]

Hi,
(Edit: This is using the trial version of 13.5)
We have a very large codebase (> 100K Java classes) which I am trying to import into EA with the code loading.

Even when I only take a subset (around 8000 classes) and import it takes a long time. The subset takes 5-6 hours to get imported. I am importing to a local eap file (on a SSD) with Jet3. The base import and and package generation does take some time but its not that bad, the real slowup happens during the "---relationships for <class>" part, then many classes takes several seconds to be handled.

I have tried to change the settings in the "Import directory" dialogue but have not gotten any significant changes, not generating diagrams and not importing private members gives some speedup in the initial phase but does not seem to change anything for the relationship generation.

Am I missing something in some configuration or is this the normal behaviour in EA when trying to load this many classes?

Regards,
/Mikael

[qwerty]

100k classes sounds like a monster. Alas, rather than using a Mickeymouse (aka MS Access) database for a monster you should use a real RDBMS instead.

q.

[Geert Bellekens]

100k classes sounds like a monster. Alas, rather than using a Mickeymouse (aka MS Access) database for a monster you should use a real RDBMS instead.

q.

While I agree with qwerty I don't think it will help all too much with the performance though.
My best bet would be to use a local instance of MySQL for optimal performance.

Geert

[Paolo F Cantoni]

100k classes sounds like a monster. Alas, rather than using a Mickeymouse (aka MS Access) database for a monster you should use a real RDBMS instead.

q.

While I agree with qwerty I don't think it will help all too much with the performance though.
My best bet would be to use a local instance of MySQL for optimal performance.

Geert

And if you can't do that, upgrade your EA to use Jet4 (See [Ctrl+F9] General Page) or SQL Server Express.

Paolo

[mikaele]

Thanks for the comments,

Yes it is a monster  , its the result of code migration of an existing COBOL/COOLGEN system.

I have not checked out the speed with jet4, since I could not find any mention of speed difference between them but I will do so now. If that then I guess I will have to go for a local DB. My thought there was that the communication times would take away the speed increase but it seems that it will be worth a try.

[Geert Bellekens]

Thanks for the comments,

Yes it is a monster  , its the result of code migration of an existing COBOL/COOLGEN system.

I have not checked out the speed with jet4, since I could not find any mention of speed difference between them but I will do so now. If that then I guess I will have to go for a local DB. My thought there was that the communication times would take away the speed increase but it seems that it will be worth a try.

If you would log all activity on the EA database you'll see that it executes a whole bunch of very small SQL queries in a very short time.
When doing something like an import that could go up to a few thousand per second.

So the question is: which database will perform best under these circumstances.

 - Jet 3
 - Jet 4
 - SQL Server
 - MySQL
 - Other?

I think in all cases hosting the database on another machine would definitely create a bottleneck as each query has to be sent over the network, so I would definitely go for a local database.
I've witnessed some amazing performance differences between a local SQL Server and a local MySQL when doing large imports (5 minutes for MySQL against 30 minutes for SQL Server), so that is why I would bet on MySQL.

Geert

[MaXyM]

What type of engine did you use for mysql? If MyISAM (which is non-logged/non-transactional) then it would explain the boost comparing to MS SQL. If InnoDB then it would be real benchmark.

Speaking about databases I can suggest to try postgresql

[Geert Bellekens]

What type of engine did you use for mysql? If MyISAM (which is non-logged/non-transactional) then it would explain the boost comparing to MS SQL. If InnoDB then it would be real benchmark.

I don't know. It was a colleague's setup, not mine.

Geert