Version Control: Practical limit in size?

[Svend Erik Nygaard]

Is there a limit for how big EA repositories should be put under version control?
Our repository has 22.000+ elements. I don't know the actual size of it in the SQL server, but in XMI exports it is 500 Mb.
Now we plan to put it all under version control. Are there any size issues in that?

[qwerty]

No known ones. However, you will encounter timing problems when checking out/in. A checkout will delete the entire SQL representation of the package and the import the VC version. A check in will export the XMI of the package and then replace it in VC.

Do you have any good reason for using VC? From my POV it does not really make sense (in the sense of "versions").

q.

[Svend Erik Nygaard]

Thanks qwerty.

I think, we will apply VC to many packages, thus having many 'discrete' XMI files. It is my understanding that that will make it possible to make relatively quick export/imports (checkins/outs).
I do have two concerns in that picture:
1) will there be any loss of connections between such 'discrete' XMIs (the documentation's word 'discrete' worries me here)?
2) Will that make a mess out of moving packages around later?

As for our reasons for VC: It is mostly for managing multiple concurrent projects on the same model (the model is an enterprise wide model). Things like 'undo-facility', audit changes, change management, package revision history, user access control.  Most important is probably the 'undo'. As it is right now, someone may unintentionally delete a portion of the model without anybody finding out for a long time. That makes me nerveous. (i guess the audit feature can help us somewhat with that even without VC)

[qwerty]

I'll try to answer shortly:

1- Yes and no. If you work in regular environment EA will keep everything in synch. If you use VC you should move away from a central repository. This will come in the way if users start Get All Packages (which they definitely will do). Sometimes I read issues with VC (and there is definitely one; see below) but in general it seems to work.
2- Yes. You can move packages but it is difficult as you have to remove them from VC, move them and put them under VC again. If you have EA name the VC packages they do not correspond to the name but the GUID wich makes it difficult to identify them later. Choosing your own naming convention is not easy either.

Caveat: If you have sequence diagrams you MUST use objects/lifelines and NOT classes inside (which is anyway meaningful but unfortunately not restrictively checked by EA). If you don't stick to that you will mess up your model with VC. This is explained somewhere in Sparx docu.

Regarding the "benefits" of VC: undo is not really an advantage. What does it mean if someone undoes his changes? Modeling by try and error? For that you should better trust in backups. Unintentional deletion? Well, of course this happened to me too. But if, it happened after some hours of modeling. And a VC undo meant to loose all my modifications. So a restore from a backup was much more meaningful. The audit is more like an airbag. You don't need it if you have your belt on (backup). But it makes you feel more save :-)

You might also read here.

q.

[Svend Erik Nygaard]

Thanks again qwerty
for a very elaborate answer  .

Although a little disencouraging  :-/

It's my impression that you have got quite some experience with this topic.

Well it looks like, IF we go ahead with it, we need to perform quite some tests/experimenting on the chosen solution.

1- Get All Packages: 'will come in the way' - I suppose that means: too heavy.  I hope that any VC will recognize if a package (XMI) has not been changed since the client's existing package (?) in that way we MIGHT be able to control it.
2- moving packages - yes, that worries me. I think, especially on this one, we'll need to do some experimenting on different scenarios.

Sequence diagram - interesting, who would have thought that!.

Benefits:
- Undo: But (assuming private models) I could export an XMI with the messed up private model, then undo, then compare with my XMI (possibly creating an extra, local repository from that XMI), fix everything   - and then check-in.  In that way, other users would remain unaffected by the incident.
- audit: that COULD though, in many cases make me aware of somebody's deletion perhaps months earlier than otherwise.
- backup: yes/no for us. The more concurrent users/projects, the worse a restore will be. (even with a continuous backup - although less so)

But of course, we will have to assess whether the VC will give more work than benefit! - and I guess, none of the challenges you mention depend on which VC we choose.

Again, thank you very much for your answers - they certainly have made me more concerned  

[qwerty]

I could write a further novel :-) Of course you should do a lot of testing ahead.

The issue wit Get All Latest is that eventually the repository gets messed up if one issues this command (and you can not disallow it). It is meant to be used in a stand-alone EAP. Anyway VC makes more sense with single distributed EAPs. As said: to much to go here. I might once write another book covering this...

q.

[Geert Bellekens]

My two cents:

Yes VC is possible if you keep the size of the packages (and thus xmi files) small.
If you have a big model I would anyway use a shared central database. That avoids the need to to GetAllLatest operations as the database is always the latest version.

Undo is possible, but is probably still dangerous anyway. Undo in one package might not be enough as there may be dependencies in other packages that will be broken etc...

We use two central dmbs models with only VC on the shared parts of the model. (our two models combined have about 110.000 elements and 35 users)

On some rare occasions we had problems with accidental deletion or similar.

On these occasions I ask for a backup of the model to be restored on another database, and I export the part I need to restore to xmi. Then I import that part in the main model.

That is usually enough.

So my conclusion is: Yes VC is nice to have, but you can get by without it as well.

Geert

[Svend Erik Nygaard]

Thanks Geert.

So what are actually the scenarios/actions where you have had use for your VC?

I get the impression that in your 35 users are software analysts/designers/programmers mostly ?

At my site we are modeling business elements (strategy,processes,concepts, objects - and infrastructure - and more and more we get business people themselves active in modeling
But we want to be able to relate elements accross aspects and areas. So from your and qwerty's response I think, I may drop the plan about VC. I'm considering a solution with a single shared model where we checkin but never checkout and never get - but if needed I can checkout a revision of some packag(s) into a temporary secondary repository to resolve any issues. But I guess, I cannot block the checkout/get functions in the single shared repository's EA clients ?

[Geert Bellekens]

No, about 1/3 of our users are pure business analysts, so not only technical people involved.

I don't get your idea of "only checkin but never check-out". What do you mean by that?

Our only scenario for using VC is the "shared" part of our two models.

We have two main applications, and each application has its own EA database.
But both applications use the same framework and reference data.
We have put the shared part into VC, and this part is present in both models.

So if we need to change something in the shared model we can checkout the shared model in either EA model and do our modifications.
After the modifications we check-in again and we do a "getlatest" on the other model.

Geert

[Svend Erik Nygaard]

'1/3 of our users are pure business analysts' - wow, then 110.000 elements is a really big repository - I thought ours was big because the elements are mostly handcrafted business elements.

"Check-in only":
I.e. we will actually only use the VC as a revision archive
  -  UNTIL we have an issue (e.g. unintended delete) and need to go back through the earlier revisions , find the relevant one - 'restore' it (check-out) into a new, temporary model - then manually fix the issue in the main single shared model.

Actually, a second  use could be to use the VC to build the Published model. Of course then "Check-in" would also mean "Publish" (and another 'Published-EA-Model' should "get all packages then").

Do you have any means of discriminating between a work-editing and a published-edition in your models?

How big is  your portion of shared packages I guess it is small compared to the 110.000 elements?

[Geert Bellekens]

In order to be able to check-in, you first need to checkout (EA uses exclusive locks only) so that means you won't be able to work with two users on the same package.
So it will be a bit inconvenient, but in theory that should work.

The question is only whether it is worth all the trouble.
For issues such as unintentional delete's you can also use a backup to fix things, and it is less hassle.

Yes our shared model is much-much smaller then the main model(s)

And not all of our elements are hand crafted. We also regularly reverse  engineer the database into EA, so that would count for a couple thousand elements as well.

Geert

[Svend Erik Nygaard]

Oh, - the  'Published-EA-Model' will only 'get all packages' from the VC (not check-out).  The  'Published-EA-Model' is not for modifying - only read. People can read and search the model via EA lite - or via an HTML site extracting its info from the  'Published-EA-Model'.

BACKUP: But I would need to go to my DBA and ask for a restore (into a tmp db) - and repeat doing so until I found the correct 'revision'.
If I don't know how far to go back - I think/hope it is easier to use VC to retrieve the needed revision. Especially if we have a number of smaller XMI-packages. I may need to retriev several different packages - but I only need to find one package to determine how far back I need to go.
 - maybe I'm too optimistic about that (?)

Ok, but 111.000 is still a lot (how big repositories do you know of?)
 - It's encouraging for me to see that it is ok with big EA repositories.

And in our 22.000, actually a few thousand elements are reversed engineered as well (DB and XML schemas). We have <impl>realizations showing where our <BusInfo> elements are implemented in DBs and in our canonical model.

(I have scripts breaking up the columns (attributes) and XML elements (often attributes) into separate fully valid model elements - making it possible to create the <impl> realizations - that accounts for a lot of elements)

[qwerty]

BACKUP: But I would need to go to my DBA and ask for a restore (into a tmp db) - and repeat doing so until I found the correct 'revision'....

You could backup yourself using XMI export of the whole model during night time. I did so in the past. Also I used VC to check in this export.

q.

[Geert Bellekens]

Hi,

Just to be clear, the 110.000 is the total of two repositories, one with 85.000 elements, and the other with 25.000 elements.

But Yes, if you have a good DBA and database infrastructure the size of the model isn't really an issue.

The only thing that really tends to slow things down is having a lot (as in hundreds) of features (operations) on the same class. EA tends to load all of them in memory when you try to access the class in any way. That can sometimes take up to 10 seconds. (and don't ruin my good mood by asking why we have classes with hundreds of operations, just don't )

Yes, indeed, if you have VC on the whole model that could be easier to find out when something wrong happened, but...

1. If your users are all working on that same central repository there is little need to check-in anything. (because the other users can already see and use the other stuff already). If you don't have frequent check-ins then you don't really have a granular enough changes, and reverting to a previous version may be too much of a change to be useful. (we had one user that worked on his own corner in the model, and that hadn't checked in for months. Then because he made a mistake he decided to do an undo-checkout. Luckily we had backups  )

2. When using backups you always go back to a consistent model. When reverting only one (or a few) package from VC you might end up with an inconsistent model because you haven't reverted the whole model back to a certain point in time.

So yes, if you can get all of your users to

A) keep the size of the controlled packages small enough
B) check-in regularly (at least once a day)

Then VC can be useful. If not then you better forget VC.

Geert

[Svend Erik Nygaard]

Yes I'm going with the single shared model.

(Geert, you made me laugh with your "don't ruin my good mood by asking ..."  )

So Yes, I see that for backup/rollback purpose I will probably rely on the DBMS restore (as Geert mentions) and encourage users to make focused XMI exports immediately before big updates on focused parts of the model.

And then I will look to the user securituy policies to manage team/user concurrence. I guess either of the two security policy modes ("User/Group" Locking and "Require User Lock to edit Mode") can will work without VC (?)