Performance Issues with EA+SVN+LDAP configuration

[tim-southerland]

(This seperates and continues a thought from a previous thread: http://www.sparxsystems.com/cgi-bin/yabb/YaBB.cgi?num=1282572108/0)

Hi Geert,

I looked through SVN logs and found that, during checkout, each .xml file goes through 3 steps that occur in a cumulative 45 second window. Steps are "status," "update", "lock."    This means that if I checkout all 77 .xml files, this could take up to an hour (77files*45sec= ~57min)

I also looked in apache access logs.   I do see that every time a file is checked out, a checkout request begins w/ a 401 error on a "options" request  ("OPTIONS /svn/<my repo name>/<my repo dir> 401 401").  Immediately after this auth failure, the same request is made and is successful  ("OPTIONS /svn/<my repo name>/<my repo dir> 200 200").  It's not clear how much time passess between these two events, but is definitely less than 15 seconds.

In summary-- checking out each file takes approx 45 sec / file.  Apache & SVN events (Authentication, status, updates, and lock)compose this 45 sec window.

Is this typical?  How can it be reduced?

Thanks!



History: --------------------------------------------

SVN + EA too slow check operations
Aug 23rd, 2010, 2:01pm    Hi, I'm using the trial version of EA 8 integrated with the lastest version of visual svn.

First we guess that only need to do the basic configuration of SVN client with EA but every package have to be configured in order to use the version control.

And when any of the team testing group try to checkout or checkin the operation is really slow (5m - 10m) over a LAN of 100Mpbs.

The testing is in order to select an enterprise architect tool to make the basis of enterprise continuum.

We are testing too Visual Paradigm and Telelogic.

Please give us a response about how to avoid the slow operations and excesive configuration in every package.

Thanks.

--------------------------------------------
Re: SVN + EA too slow check operations
Reply #11 - Yesterday at 8:39pm      Hi Geert,

I'm running into the same issue.  I have EA v8 configured to work w/ SVN 1.6. (via https) and authenticate to a Windows 2k3 AD server via LDAP (non-encrypted).  The EA client application communicates w/ the SVN server over 100/1000mbps connection and the two machines are physically colocated.  I am able to check models in and out, but the performance is excruciatingly slow (10-20 min).  This is especially slow when I try to add a model to version control for the first time.

I'm not sure if I'm running into a configuration issue, or a limitation of EA.  I suspect the former, but I need a concrete test that will help me verify.   The Network, memory and CPU are not showing any signs of stress.. it seems like the application just stalls or works at a very slow pace.

Given this configuration, can you describe the sequence of events that occur when I click on "add branch to version control" ?   Where would you expect bottlenecks to lie?  How can I verify which issue to attribute this performance problem to?

Thanks!
Tim

--------------------------------------------
Re: SVN + EA too slow check operations
Reply #12 - Today at 5:57am

Tim,

EA does version control using xmi files of the controlled packages. This means that everytime you checkout a file EA will get the latest version of the xmi file stored in VC and perform an import of that xmi file.
Same story when checking in, EA exports the package to xmi and checks that file in into version control.

The more packages you check-in/-out , the more time it takes to import/export the xmi file.

The total execution time of a a checkin operation should be the sum of
- time to export each package to xmi
- time to checkin each created xmi file to VC.

If this doesn't add up then there might be a configuration issue between EA and SVN. You might be able to get more detailed information on what is actually happening (and which operations take the most time) if you inspect the SVN logs on the SVN server side.

Geert

--------------------------------------------
Re: SVN + EA too slow check operations
Reply #14 - Today at 1:32pm      

Thanks Geert,

I'm checking in 77 .xml files from about 22 folders.  The largest one is about 1MB.

how much time would you estimate is involved with exporting one 1MB .xmi file? importing?

I'll check svn logs next.

Tim

[Geert Bellekens]

Importing or exporting a ~1MB xmi file takes about 20 seconds on my model.
This is on EA 8.0.862 with an SQL 2008 database and a model of about 50.000 elements.

This is time that you can't shave anything off of (unless you create even smaller packages).
The other 25 seconds should then be accounted to the checkin-out operation on SVN, which seems a bit long to me.
How long does a manual check-in take using the SVN tools?

Of course you don't need to check out/in all 77 packages at the same time, the idea is to only check out what you need.

Geert

[tim-southerland]

I observed today that between selecting "add to version control" and a progress dialogue that says "Exporting to Version Control" .. there is a period of time when EA is busy or appears to hang (doesn't repaint). This lasted 4min 15 seconds for an 85MB model.

This appears to be in addition to the steps you mentioned before:

The total execution time of a a checkin operation should be the sum of
- time to export each package to xmi
- time to checkin each created xmi file to VC.

What is happening during this period of time?  

[tim-southerland]

I'm still experiencing crippling performance issues. As our models have increased in size, so has the time required to check in and check out.  It started off being "slow" but its becoming "unusable".

Currently our models contain about 4960 elements,5550 connections 96 packages, and 40 diagrams.   EA Ultimate v8 software is running on windows XP sp2, with an intel core 2 duo, 2.66Ghz w/ 4GB ram.  We're running on an unhindered 100mbps network.  We have a Collabnet Subversion on win2k3 servers on the back end.  The server stats present no noticeable performance bottlenecks.

The size of the exported data is about 460MB.   It takes btw 40 minutes and 1.5 hours to completely check out all data using the "get all latest" function or "check out branch..".   Using Tortoise SVN, it takes 40 seconds.  Both clients interact w/ the SVN server over SSL, authenticating w/ the same cached certificate.

The import process causes the EA application to consume about 50% of the CPU time & 400MB RAM.

Help! How do I improve performance so my architects can model collaboratively?

Tim

[Geert Bellekens]

Tim,

I think the only solution is to move to a shared DBMS model repository.
You can choose between a number of different databases to store your model on.
This will not reduce the time to check-in/out, (it may even increase it), but it will reduce the need to check-in/out packages, as you will only need to check-out/in what you actually want to change, no more need to do getlatest and stuff as the model will always reflect the latest changes.
We are working on a model with ~50.000 elements, 5500 packages and ~5000 diagrams on SQL2008, with part of the model version controlled in TFS, and we are experiencing no real performance issues.

Geert

[tim-southerland]

Great. I was thinking the same thing and this was my next step.  

My concern with this strategy is that it limits us to performing updates at a single location, rather than the multi-site capability that our customer needs.  In the long run, we want to support multiple contributors around the world.  Somehow I need to configure the system to be a hybrid of the "Distributed Team using local Model" & "multiple site locations" scenarios that are described in the version control whitepaper (http://www.sparxsystems.com/WhitePapers/Version_Control.pdf)

I'll try deploying EA w/ SQL and let everyone know what I what I find.

I'm also very happy to hear that there is a lot of room for growth and that we're not reaching model limitations w/ the number of elements we have. Thanks for your reponse !

[Geert Bellekens]

Tim,

Even with a central DBMS you should still be able to maintain multiple local copies of the model using version control.
I have this scenario with a model that is shared between two different SQL repositories, but I don't see why there should be a difference if I created a local EAP file as well.

Geert

[tim-southerland]

Hi Geert,
 
In your central DBMS scenario..

How long does it take for you to log into and open your DBMS managed projects?  Also, what kind of authentication are you using to connect to your DB?   Is your DB on a central server or deployed onyour desktop?

Thanks !
Tim

[tim-southerland]

I have configured our repository to work with SQL 2005.

The average check-out time per .xml file is 47 seconds ( time that elapses between svn status,update,lock requests).  In one of our repositorires we have 45 .xml files ..  The largest xml file in that repository is 12MB, average is approx 150Kb. It takes about 35 minutes for a check-in.  

How long does it take to other people to check in or check out a single XML file?  (based on subversion log)?

How can we reduce the amount of time it takes to check in/out?
Tim

[Geert Bellekens]

Tim,

We have a central server with SQL2008.
Opening a model with EA (v 7.5) takes anything between 10 and 20 seconds.
We use windows authentication to connect to the database, and EA "security" on top of that to login into the model.
Once openened the performance of EA is really acceptable. The longest we ever have to wait is about 10 seconds, but that is when opening a class with a few hundred operations (I know  , was created before my time here)

I see you are still going on about the checkin of the whole model, but the whole point of having a centralised DBMS model is that you now only need to check-in and out separate packages, and not the whole model anymore.

Geert

[tim-southerland]

Thanks Geert,

I realize that its more efficient to check out portions of the model rather than the whole thing.  In most cases you are right, we will only be checking out particular packages in the model.  However, for changes that are more sweeping we will have to touch and check-out more packages.   We will have to develop some business process to minimize the number of packages that we are checking in and out daily.

Nonetheless, after deeper investigation a single package (xmi2.1 file) requires almost exactly 47 seconds to check out regardless of how many diagrams or elements are in it. This metric is consistent regardless of whether we are using SQL or the embeded DB as the EA backend.  In fact, we are now counting packages and able to predict exactly how long checkout will take.  The main problem that I have with this is that during the majority of this time, the application is waiting, no network activity, no cpu activity, no disk activity.  It spawns svn.exe processes occasionally, but these open and close within about 8 seconds each (max).  

This EA behavior is problematic for us, especially since we can check packages in and out, and then import much quicker manually. This is a bug.

Tim

Sparx EA 8.0.858 (w/ BPMN, Visio, UPDM, TOGAF, SysML add ins)
Collabnet SVN 1.6.x
Win XP sp 3 (clients)  , 4Gb, Dual Core 2.5Ghz
Win2k3 Active Directory
100mbps network

[Geert Bellekens]

47 seconds? If it where 42 seconds then OK, but 47 makes no sense at all

No serious, I'd ask Sparx for help and/or experiment with other VC to figure out if this is the same for all VC, or just with the combination you're using.

I know EA is pretty picky in the choice of dll/drivers and stuff like that, so you might just be using the wrong SVN client or something like that (or the wrong version)

Geert

[tim-southerland]

Thanks Geert

Yes. 47 seconds   Odd but true.  I would much prefer your 20 seconds

It seems pretty clear to me that its an issue on the Sparx EA side with how the app handles spawning svn clients and importing data. Drivers could be at the root, but I would need Sparx to tell me what config they recommend. I'm now in touch with Allan Butt from Sparx and hope to be working with him to resolve this issue.  

  Really appreciate your help in narrowing the issue down.

[tim-southerland]

I was able to work around this isssue.  It turns out it was not a problem with Sparx EA.  Instead it was a problem with the Apache configuration.  

I was able to reduce check out/check in time to btw ~2-5 sec /package.  This was accomplished by disabling SSL connections to Subversion.  This is not an ideal configuration since SVN transfers passwords in the clear.

I am hoping that future server-side adjustments to the SSL module (in Apache) will improve performance to the extent that it is usable.