Author Topic: Automation stops  (Read 8787 times)

nrocha

  • EA Novice
  • *
  • Posts: 11
  • Karma: +0/-0
  • I love YaBB 1G - SP1!
    • View Profile
Automation stops
« on: December 19, 2007, 07:47:11 am »
Hi,

I'm using EA automation to generate diagram images.
In my EA project I have an enormous number of requirements (2300) and each requirement has its own diagram.
The project is in a SQL Server repository.

I'm calling PutDiagramImageToFile so that I can have a file in disk with each diagram.

But this process stops after the generation of 445 diagrams...(yes, it's always after 445 diagrams and I've already deleted some of the first ones to see if there was some sort of problem with them, but this number keeps unchanged). The PutDiagramImageToFile of the 446th call returns true but there is no file in disk...

I tried to generate diagrams also from a local .eap file and the reporting mechanism also stopped with some errors ("The resource was unavailable" and something related with the clipboard) and EA crashed.

Is there any kind of limitation in terms of the memory used by the EA.exe process or, more strangely, in terms of the number of diagrams that is generated (it is a little bit strange that, using automation, the number of correctly generated files is always the same...). Could it be that there are appearing the same small dialog windows with the errors that appeared when I tried to generate a report?

Regards

«Midnight»

  • EA Guru
  • *****
  • Posts: 5651
  • Karma: +0/-0
  • That nice Mister Grey
    • View Profile
Re: Automation stops
« Reply #1 on: December 19, 2007, 08:45:07 am »
Your automation call stores the diagrams in the Windows clipboard. You are likely running into a limitation there.

From the EA side the only setting I can think of is Image Memory Limit, which you can find on the Tools | Options | Diagram dialog.

On the Windows side this is likely an OS setting someplace in the registry; there might be some other way to get at it as well. You'll have to look this up on your own.

You can always call the Windows API (someplace in Win32) to clear the clipboard. Note that doing so might affect other applications that are using the clipboard at the same time.

HTH, David
No, you can't have it!

nrocha

  • EA Novice
  • *
  • Posts: 11
  • Karma: +0/-0
  • I love YaBB 1G - SP1!
    • View Profile
Re: Automation stops
« Reply #2 on: December 19, 2007, 08:53:19 am »
The PutDiagramImageToFile method also uses the clipboard? I thought only PutDiagramImageOnClipboard used the clipboard...

And if it used the clipboard, wouldn't the next call to the method simply replace the existing clipboard content?


«Midnight»

  • EA Guru
  • *****
  • Posts: 5651
  • Karma: +0/-0
  • That nice Mister Grey
    • View Profile
Re: Automation stops
« Reply #3 on: December 19, 2007, 11:57:21 am »
Oops.

My bad, I misread your post. You were very clear. I suspect that since I've so often used the clipboard method it was what I expected, so it became what I 'saw.'

So, we're back to square one. Still, it might be worth tweaking the image limit in EA to see if that changes anything. Perhaps there's some internal mechanism that's getting overloaded.

Some long shots here. What happens if you reload the project from time to time? Or if you completely disconnect, flush EA (if you are in .Net), and then reconnect?

David
No, you can't have it!

nrocha

  • EA Novice
  • *
  • Posts: 11
  • Karma: +0/-0
  • I love YaBB 1G - SP1!
    • View Profile
Re: Automation stops
« Reply #4 on: December 20, 2007, 03:28:55 am »
Hi,

so, one of the long shots worked :)

I tried tweaking the image limit in EA and didn't work. Same thing if I reloaded the project from time to time. The process keeped stopping after 445 diagrams.

The last tip worked: I'm disconnecting from EA (exiting the process) and then reconnecting every 400 diagrams and I've managed to get to the end.

What does this mean? That there is some kind of memory issue or something related with the resources when EA is under heavy usage? Is this a known bug?

I've also tried to generate these diagrams from the EA reporting engine and never got to the end. It crashed somewhere in the middle of the process. I've also noted that it crashed faster if I had the EA window visible. If I minimized it, then the process would last longer (but never got to the end of it, though).

Thank you very much for the "long shots"  ;)

Regards.

«Midnight»

  • EA Guru
  • *****
  • Posts: 5651
  • Karma: +0/-0
  • That nice Mister Grey
    • View Profile
Re: Automation stops
« Reply #5 on: December 20, 2007, 04:51:20 am »
The most important thing is that you are back up and running.

As to the rest, I suspect this is a bug. In the case of the reporting engine this is definitely the case.

So, what you need to do now is submit a bug report to Sparx. They take these pretty seriously, particularly in cases where the product could crash.

We have a best practice for making such reports, while keeping everybody in the loop. Please search my recent posts - use the phrase "bug report" - for how we do this. [The forum thread I mention will be this thread.] If you don't find the directions write back and I'll write them again.

David
No, you can't have it!

tanja

  • EA User
  • **
  • Posts: 71
  • Karma: +0/-0
    • View Profile
Re: Automation stops
« Reply #6 on: May 05, 2012, 05:32:05 am »
Happened to me as well with PutDiagramImageToFile: you have to call CloseDiagram(diagramId) in a finally clause. Note that this id is the EA local ID (int), not the UUID (String) needed to obtain the diagram - sic!

I have been using PutDiagramImageToFile to get .png images (from Java API) but that is terribly slow. I have:
1) some 50% performance deterioration as compared when I don't export any diagram, and,
2) with Windows 7, while I read the model and walk through it, including exporting diagrams, my application "steals" the mouse focus and I literally cannot do anything with my computer.

Does anybody know why this happens and whether there is a workaround, at least for point 2? (BRW, why writing this, I've tried with the PutDiagramImageOnClipboard(guid, 1) to get bitmaps, but I get the same focus stealing as in 2 above).

I haven't tried with a background thread: although my only usage of the project object, obtained from repository, is for getting the images, and in theory it should work, but I didn't dare to waste time for something that might not work by design ! Somebody on the forum confirmed that repository is not thread safe, even if read-only.

I'm getting desperate with the API bad performance and now try to go through the clipboard - but this is not easy with Java...

Any hints here?

tanja

  • EA User
  • **
  • Posts: 71
  • Karma: +0/-0
    • View Profile
Re: Automation stops - PutDiagram methods performa
« Reply #7 on: May 05, 2012, 07:34:54 pm »
Here the update.

On the forum, somebody wrote that apparently EA goes through the clipboard internally even when exporting to file, so I thought clipboard only may be faster and implemented the possiblity to use EA's export of a .bmp file to clipboard then saving that .bmp for pasting into Word/link for XML. However, performance-wise, it's a bit slower than the EA's export to .png file. For one diagram export to file, it takes between 300-500ms. For ~100 diagrams, it's ~40 seconds. When going through clipboard, for the same ~100 diagrams it takes 3 seconds more (~ +8% time).

I'm running EA 932 with Java API (Java 6, Windows 7, ThinkPad T410), and saving files to a directory that is excluded from antivirus scan-on-access. Here the code snippet:
<code>
            // ...
            try {
                  File pic = (removeAfterExit) ? Util.createTempImageFile(dirPath, fileName,
                              retainedFormat, removeAfterExit) : new File(dirPath, fileName);
                  if (throughClipboard) {
                        _logger.debug("Saving image from clipboard to '" + pic.getAbsolutePath() + ".");
                        _eaProj.PutDiagramImageOnClipboard(diagramUuid, 1);
                        Util.saveImageFromClipboard(pic);
                  } else {
                        _logger.debug("Saving exported image to '" + pic.getAbsolutePath() + ".");
                        _eaProj.PutDiagramImageToFile(diagramUuid, pic.getAbsolutePath(), 1);
                  }
                  return pic;
            } finally {
                  _eaRep.CloseDiagram(diagramId);
                  _logger.debug(String.format("... saved in %s ms", System.currentTimeMillis() - start));
            }
</code>

Are there any experiences with running this kind of things in a background thread? I don't want to waste time implementing if somebody can confirm it wouldn't work.

My problem with focus "stealing" is still actual - this was not happening with Windows XP, only with Windows 7; I've also reported the issue as a registered user...

Otherwise, is there any other way one could speed up diagram export?

ps Geert: SQL doesn't help here :-), but I'll spend my weekend trying it out - with help of qwerty's e-book - for model browsing because the API performance is more than sad.

qwerty

  • EA Guru
  • *****
  • Posts: 13584
  • Karma: +396/-301
  • I'm no guru at all
    • View Profile
Re: Automation stops
« Reply #8 on: May 07, 2012, 02:19:35 am »
Hi Tanja,
the problem with PutDiagramImageToFile is simple: EA creates the rendering on-the-fly. This is per se very time consuming. Usually you don't notice that when working by hand. But when doing that via automation you're really pushing machine performance to its limits. EA has to retrieve all elements on a diagram (how many do you have per diagram?). Same for all relations. Now the image must be rendered internally so you get a PNG or something alike. Closing the diagram after rendering is a must since memory will blast your EA quite soon.

The only solution: have a VERY FAT machine. And a responsive SQL server. You might work with multiple machines in parallel if you can divide the diagrams.

You also might try to do the rendering yourself by going through the relevant tables. But I doubt it will be faster when implemented in Java.

Not much help...

q.
« Last Edit: May 07, 2012, 02:26:10 am by qwerty »

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13287
  • Karma: +557/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Automation stops
« Reply #9 on: May 07, 2012, 06:11:40 pm »
My thoughts exactly.
There is simply no way around it when you want the diagram images, you'll have to let EA render them.
And that process indeed takes both CPU as a lot of SQL calls.
It might help if you could run in on the same machine as the database.
I've even often noticed performance improvements when running against a local .eap file compared to running against an SQL Server on the network.

Geert

PS. 40 seconds for 100 diagrams doesn't seem bad at all :-/

tanja

  • EA User
  • **
  • Posts: 71
  • Karma: +0/-0
    • View Profile
Re: Automation stops
« Reply #10 on: May 07, 2012, 06:43:14 pm »
Thanks to both of you for reply.

I'm already running everything on my laptop, with .eap file (I am just allergic to RDBMS ...)

Q1) You mention that EA does rendering of diagrams on the fly: is it possible that that is the reason that EA steals the focus on my T410, which does not have a separate graphics card (but uses the intel cpu?)

Q2) Any experience with using EA project interface in a background thread, if I can guaranty that nothing related to EA repository will be used from the main thread?

Despite my reluctance to SQL, I did some progress in learning and applying it and I can report significant speed improvement. With a small test model that has some ~700 attributes, I've provided an SQL+XML based implementation (including attribute's constraints and tagged values) - everything else is still through EA API and collections. For pure reading, after opening file, I could go from 26 s down to 18 s: 30% speed improvement. I'm moving forward with the rest...

It is really sad that it is at all possible to access EA tables this way: that fact makes the Sparx impossible to refactor their DB design, which is - well - very exotic. If they were providing a decently fast and intuitive API (think: tagged values) - as we all are actually doing for our apps - they would have been free to change the DB implementation at their will. Now they are stuck because that very implementation is public...

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13287
  • Karma: +557/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Automation stops
« Reply #11 on: May 07, 2012, 06:49:09 pm »
Q1: I know that EA needs to actually open a diagram in the application before it can export its image. I have no idea if that has anything to do with stealing focus.

Q2) No, not really.

In fact the Sparxians always tell us that the reason the database is not documented, is because then they have the liberty to change it. (so using the database directly is at our own risk).
But in reality the database structure has hardly changed in +5 years.

Geert

PS. "Exotic" is very nicely worded ;D

qwerty

  • EA Guru
  • *****
  • Posts: 13584
  • Karma: +396/-301
  • I'm no guru at all
    • View Profile
Re: Automation stops
« Reply #12 on: May 07, 2012, 09:41:07 pm »
Q1) I run document generation in the background right the same way: open diagram, export, close. I have no problem with any focus change. EA stays nicely in background. But the instance is of course blocked for interaction.

Q2) also no experience.

Regarding the database structure: It's not really open - just semi-open. My e-book is not promoted on Sparx site for that reason. However, Sparx started shoveling their own grave as they added the SQL-search feature and linking table names in their documentation. As Geert says: except for one single major upgrade (auditing at around 6.0/7.0) the database is still that of (I guess) 0.9, but definitely that of 3.5 when I started working with EA.

q.
« Last Edit: May 07, 2012, 09:43:28 pm by qwerty »

tanja

  • EA User
  • **
  • Posts: 71
  • Karma: +0/-0
    • View Profile
Re: Automation stops - t_taggedvalues
« Reply #13 on: May 28, 2012, 09:48:02 pm »
I found the significance of some non very obvious tables:

- t_taggedvalues holds tagged values for at least association ends (referred to as ASSOCIATION_SOURCE and ASSOCIATION_TARGET, not any id/guid !), by referring to the GUID of the connector; I don't care for tagged values of other connectors' ends.

- t_xref holds full set of stereotypes for most of UML items by their GUID in "Client" column - except for diagrams and connector ends; I still didn't start with operations and their parameters, so some of these might also hold their own stereotypes (like diagrams and connector ends)

I tend to think that those who designed this database must have been on crack  :o I now also understand why the EA API is not very uniform and how much it reflects the database design.

Will post timing comparisons with EA API vs. SQL+XML when done. I hope that this huge pain will pay off with speed ...

qwerty

  • EA Guru
  • *****
  • Posts: 13584
  • Karma: +396/-301
  • I'm no guru at all
    • View Profile
Re: Automation stops
« Reply #14 on: May 29, 2012, 08:34:41 am »
Hi Tanja,
thanks for the feedback. I'll incorporate the info about t_taggedvalues. But regarding t_xref: as I wrote this is stuff for a complete book. I do have some pointers but it looks like a haystack. Nothing you'd like to sort.

q.