Author Topic: My Add-In is on a go slow  (Read 8294 times)

ZuluTen

  • EA User
  • **
  • Posts: 56
  • Karma: +0/-0
    • View Profile
My Add-In is on a go slow
« on: August 12, 2014, 06:30:02 am »
I wonder whether I'm expecting too much...?

I've got EA version 11 running directly on a reasonable spec PC (3.2GHz i5  12Gb RAM running Windows 7/64).

I have written an Add In that parses an xml source file in order to generate a number of packages, objects and diagrams.

The xml file is approx 5Mb, and contains roughly500 nodes, which I have to burst in order to generate the commensurate number of packages and diagrams.

Each object also gets populated with a number of tagged values derived from the parsed file.

There are no connections (yet).

The completed .eap file is a little under 4Mb in size, because I waste much of the source data.

My Add In is, I think, fairly efficient in its structure – it's written in vb using Visual Studio 2010 and compiles to dll without any apparent problems.

The sticking point is that the process takes about 10 minutes to run!

It doesn't seem linear though: The process kicks off, processes the first 20 nodes, updates the Project Browser, and then apparently stalls for literally minutes with the EA window showing 'Not Responding'. Eventually it recovers, then does a variable amount (between 20 and 50) and then apparently stalls again. This repeats until the whole xml has been parsed, and the correct desired outcome is reached

The stalling always occurs at the same xml nodes, and although these don't seem to differ from any other nodes, it suggests to me a programmatic shortcoming in the seat/keyboard interface.

So: Are there any obvious stupid mistakes which I, as a beginner, might have made that could result in such an appallingly slow process?

I wondered about the use of update or refresh and whether the underlying database could be getting out of sync with the Add In, and eventually recovering, but that seems a bit unlikely.

I used EA's built-in vbscript to develop some of the modules and frankly that didn't seem much faster, so I'm bewildered.

I wondered whether I was using/abusing objects badly in my code, or possibly failing to declare variables  correctly, but a revisit doesn't seem to have improved things.

I shan't be offended no matter how basic the suggestions of what to check...
..or suggestions of how to use any diagnosis tools.

Thanks in advance.

qwerty

  • EA Guru
  • *****
  • Posts: 13584
  • Karma: +396/-301
  • I'm no guru at all
    • View Profile
Re: My Add-In is on a go slow
« Reply #1 on: August 12, 2014, 08:14:14 am »
XML might become a monster eating your resources. It simply happens under the hood. If you use queries for the XML then this could well be the reason for the slowness. Hard to give any advice without seeing the code (and even then it'd be tough). Just spit in some debug prints with time stamps to trace down the reason.

q.

Aaron B

  • EA Administrator
  • EA User
  • *****
  • Posts: 941
  • Karma: +18/-0
    • View Profile
Re: My Add-In is on a go slow
« Reply #2 on: August 12, 2014, 10:19:07 am »
If you are only adding new items to your repository (not deleting), then you should not need to call Refresh() on your collections at any point that I know of.  Calling Refresh() will re-query the database and re-build your collection. Could be causing a performance hit if you are calling it frequently.  Try commenting out your Refresh() calls and see if it makes a difference.

Setting Repository.BatchAppend = true and Repository.EnableUIUpdates = false at the beginning of your import could help improve performance depending upon exactly what you are doing.

Apart from that, like qwerty said - it's hard to know where the problem may be without seeing your code.

ZuluTen

  • EA User
  • **
  • Posts: 56
  • Karma: +0/-0
    • View Profile
Re: My Add-In is on a go slow
« Reply #3 on: August 12, 2014, 02:02:26 pm »
Thanks gentlemen,
I've made the suggsted changes, but no significant improvement.
Further reading suggests that the use of DOM and X Path is extremely wasteful, when all I'm trying to do could be readily achieved by XMLReader instead.

Time for a re-write...

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13387
  • Karma: +566/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: My Add-In is on a go slow
« Reply #4 on: August 12, 2014, 03:59:39 pm »
First of all I would try to determine the root cause.
Is it because of EA or because of your XML handling.

I'm pretty sure it's EA and not the XML libraries you are using.

You could be surprised how slow EA can be when dealing with (at a first glance) simple function.

One of those pitfalls is the EA.Collection. You would think that once you have loaded say all attributes from an element into memory that you can iterate them at hardly no cost at all right?
No! Try to turn on a database profiler when you are testing your add-in. You'll notice that EA will query the database each time you iterate to the next attribute.

So you should try avoid having to iterate EA.Collections at all times, and if you have no choice make sure you iterate each collection only once.
What could be helpful is
- Create an in-memory List of the objects you need to iterate repeatedly
- Use repository.SQLQuery() and Repository.Get<XXX>ByID() to instantiate only those object you really need.

Avoid creating objects you don't really need.

Example:
Suppose you want to create a class in a package, but you want to make sure there isn't already a class with the same name in the package.
The naive way of doing this is to iterate package.Elements and check the name of each of those elements. Depending on the number of elements in your package this can easily take seconds or even minutes.

The better way of doing this is to use SQLQuery to get the ID of the element with the same name and the same packageID.
If no such element exists you haven't instantiated a single object. If it does you can use that ID to call GetElementByID() and instantiate only the object you need instead of creating an object for each element in the package.

I hope this explanation was a bit clear :-/

Geert

martin1

  • EA User
  • **
  • Posts: 57
  • Karma: +0/-0
    • View Profile
Re: My Add-In is on a go slow
« Reply #5 on: August 12, 2014, 09:12:25 pm »
Quote
My Add In is, I think, fairly efficient in its structure – it's written in vb using Visual Studio 2010 and compiles to dll without any apparent problems.
Do you compile it for "Debug" or "Release"? Release enables compiler optimizations. I can't give you any concrete numbers, but for calculation-intense applications speedups around factor 3-10 are possible.
« Last Edit: August 12, 2014, 09:13:24 pm by martin1 »

ZuluTen

  • EA User
  • **
  • Posts: 56
  • Karma: +0/-0
    • View Profile
Re: My Add-In is on a go slow
« Reply #6 on: August 15, 2014, 07:04:02 pm »
Thank you all for your hints and tips:

qwerty: I've speeded up UML handling by only processing one node of the 500 nodes in the source file at a time.

This has confirmed Geert's theory that it's the calls to the database that are the impediment.

I therefore reassessed my code and removed some of the test that were done in the 'for each something in a collection' style and re-written these as SQL queries straight to the database.

Sadly, Geert's other suggestions don't really apply as the code never repeats its use of any references .

I followed Aaron's suggestion and removed superfluous refresh commands.

I have checked Martin's suggestion for ensuring that Release versions of DLLs are generated.

All of these have made some difference – most notably the use of SQL instead of through repository queries.

All good so far, yet still the code, when triggered by the Add In, will run for approx 6 seconds adding packages and empty diagrams as it goes, but then it stops (the top blue bar shows “Not Responding”) for a further 7 seconds before continuing. This cycle repeats, interestingly with gradually longer 'paused' delay periods, but conversely faster processing times until the task is finished and all 500 packages have been created with two digrams in each.

This makes me wonder if EA is being constrained in some way by the operating system?

Finally:  

Has anyone else noticed the really iritating disparity between the treatment of parentID in the t_package table which uses “Parent_ID”, whereas the t_diagram table uses “ParentID”? That took some finding I can tell you...

« Last Edit: August 15, 2014, 07:04:31 pm by ZuluTen »

qwerty

  • EA Guru
  • *****
  • Posts: 13584
  • Karma: +396/-301
  • I'm no guru at all
    • View Profile
Re: My Add-In is on a go slow
« Reply #7 on: August 15, 2014, 08:22:14 pm »
Quote
Has anyone else noticed the really iritating disparity between the treatment of parentID in the t_package table which uses “Parent_ID”, whereas the t_diagram table uses “ParentID”? That took some finding I can tell you...

 
EA is consistent in those inconsistencies. E.g. like pointed out in another recent thread the notes in t_object are named Note. Using the plural in the correct way in database design was obviously also something Mr. G. Sparks missed in the university lessons. :P Even worse none of the following architects dared to correct all those obvious mistakes. :(

q.

P.S. I also faintly remember such "muddy" code behavior but I never got a real grip on it. Usually it was only with code rarely used for conversion or so and then gone to tray. In other cases the tray was also the solution as to start with a different approach. A hard way to go, I know...
« Last Edit: August 15, 2014, 08:26:08 pm by qwerty »

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13387
  • Karma: +566/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: My Add-In is on a go slow
« Reply #8 on: August 21, 2014, 10:30:25 pm »
I'm guessing EA is doing something in the background while when you see the "Not Responding". Possibly it noticed that something has changed, and started rebuilding its in-memory representation of the model.
The only way to be sure is to put some kind of profiling on so you can see what EA is doing.

If not then the bottleneck must be somewhere else.

Geert

martin1

  • EA User
  • **
  • Posts: 57
  • Karma: +0/-0
    • View Profile
Re: My Add-In is on a go slow
« Reply #9 on: August 22, 2014, 05:41:30 pm »
Not to forget that the database format which is carried by the EAP file is fairly old. Another option would be to upsize EA to some other database type like e.g. MySQL.