XMI Import Data Corruption/Loss in EA 6.5 (802)

[darren.sampson]

Hi Everyone,

After the import fixes in Build 802 (thanks to the Sparxians involved) we re-ran our checks to identify if there are any remaining data corruption/data loss issues.

So far, we have identified 5 issues and have noticed 1 fix:

Issues:
1) Notelink connectors have their direction reversed. We believe note links should always be from note to element/connector, however, this issue relates to arbitrary direction changes in the DB.
2) Dependencies with direction Destination -> Source when imported have source and target elements swapped. Direction is not swapped.  This results in a change in the rendering and semantics in the diagram
3) Some diagrams have visible realisations which were invisible before export/import.
4) There are less diagram links in the DB than on the diagram.
5) Some attributes lose their classifiers on import. The ones we identified had a classifier name in the XMI which appears more than once in the t_object table, however the classifier GUID appears in the type field of the ownedAttribute in the XMI.

Fix:
1) Notelinks that should have been in diagram links (801) now are (802)

I am reporting this as a bug, and a speedy resolution (as per the 801->802 fixes) is needed.

[VK]

Hello Darren,

Thanks for the update. I can confirm that at least the first two of the issues you mentioned are already resolved for an upcoming release. We will probably need some more detail about the remaining issues to reproduce them. I will contact you via support to discuss this further.

Note: Regarding the first issue, Notelink is actually undirected and may be created either from the class to the note or vice versa. So the actual direction for notelinks is unimportant.

Cheers.

[Paolo F Cantoni]

Note: Regarding the first issue, Notelink is actually undirected and may be created either from the class to the note or vice versa. So the actual direction for notelinks is unimportant.

No Vimal, the actual "direction" is important.  In a separate topic, we shall be discussing "Directionality" more completely; since we are unsatisfied with the current EA implementation (and our observations of the problems introduced by the current implementation bear this out).

However, in summary, how our proposal relates to the Notelink (whether it is a Note connected to a vertex or to an edge), is that regardless of how the user draws the link, it should be stored as the note being the origin and the other element being the destination.  This is because the Notelink is (as you say) undirected.  As a consequence, it makes sense where a Note is attached to make it the origin since that way queries are simpler (and consistent where multiple destinations are involved).

Thoughts? Votes?
Paolo & Darren

[darren.sampson]

In a separate topic, we shall be discussing "Directionality" more completely; since we are unsatisfied with the current EA implementation

The topic is now posted here.

[Paolo F Cantoni]

Just a clarification on Darren's original posting in this topic.

The differences we found are generated by taking a pre-801 created repository, exporting it under 802 and importing into a new 802 repository.

We are trying to find a process to upgrade and consolidate our current dispersedly developed packages and since we haven't been able to remove corruption or data loss, we still have pre-801 (built) repositories with our designers.

We have run tests round-tripping a fully 802 created repository and there are still issues with that, although not so extensive.  We will provide further analysis when completed.

From what we have observed so far, the issues principally occur on import rather than on export.  We are therefore continuing to develop under 802 using pre-801 repositories until we can sort out the round-tripping.

HTH,
Paolo

[Paolo F Cantoni]

Just a clarification on Darren's original posting in this topic.
[size=13][SNIP][/size]
We have run tests round-tripping a fully 802 created repository and there are still issues with that, although not so extensive.  We will provide further analysis when completed.
[size=13][SNIP][/size]

We've now analysed the single operation that was dropped on the pure 802 round trip and observed that the return type is actually defined in a different controlled package.

HTH,
Paolo

[«Midnight»]

Paolo,

I'm not quite sure what you mean by "return type is actually defined in a different controlled package."

I'm assuming that the "return type" refers to the operation's return type.

What do you mean when you say it is defined elsewhere?

David

[Paolo F Cantoni]

Paolo,

I'm not quite sure what you mean by "return type is actually defined in a different controlled package."

I'm assuming that the "return type" refers to the operation's return type.

Yes...

What do you mean when you say it is defined elsewhere?

David

The class that is the return type is defined in a different controlled package.  Thus, the reference may disappear if the other package is replaced (during the import process).

Essentially, as (I think) I explained elsewhere, the deletion of a package for teh purposes of import should be different from the deleteion of a package for normal purposes.  EA seems to treat these two situations as the smae which is the primary source of most of the prolems we've discovered in the export/import cycle.  It creates dangling references accross the controlled packages.

HTH,
PAolo

[darren.sampson]

Status update for 6.5 (803)

Issues:
1) Notelink connectors have their direction reversed.
This is corrected in 803

2) Dependencies with direction Destination -> Source when imported have source and target elements swapped.
Partly corrected in 803.  Dependencies not shown on any diagram may still be reversed.

3) Some diagrams have visible realisations which were invisible before export/import.
4) There are less diagram links in the DB than on the diagram.
We have tracked down these issues and discovered that issue 3 is caused by issue 4. This is not corrected in 803.

5) Some attributes lose their classifiers on import.
This is corrected in 803

Issues 2, 3 and 4 only occur when the XMI is imported for the second time.  We encounter this regularly due to dependency issues between controlled packages (see here)

Darren.

[«Midnight»]

With regard to 2):

Is it possible that the DiagramLink object is being corrected but the underlying Connector object is not? This situation might display the observed symptom.

With regard to 4):

About a year ago I was manipulating large models in a variety of ways. I discovered that some diagram links are apparently drawn on the fly by EA, and not stored in the DB at all. Unless I forced an entry into the DB the links were drawn to some internal EA default pattern, including labels, etc.

The specific thing that caused me problems was that on diagrams created entirely through the API, EA often rendered links on its own, but did not store these in the DB. The default visibility setting was to show these links. Thus, even if I removed them, rendering the diagram again, or sometimes even reloading it, would cause the links to be displayed. Not good on complex diagrams.

I don't have the code handy, but I think I had to hack into the DB and store a record from outside of EA (i.e. through a programmed interface into the DB, not through EA's UI or API). Although this produced the expected results, and the resulting diagram links retained visibility (and other) settings I inserted into the DB, this is a nasty hack.

Seems your problem is a case of this coming back to bite us.

[Essnet]

Please,

The issue 4 refers to this topic: http://www.sparxsystems.com.au/cgi-bin/yabb/YaBB.pl?board=general;action=display;num=1163383934 ??

This is a critical issue. I hope that will be fixed in next build...  :-/

[darren.sampson]

Hi Everyone,

We have worked closely with Sparx to test these issues and believe all the issues raised in this post have been corrected in 804.

For the model we tested, it is now the case that if you export a whole EAP in "pieces" and then re-import (twice) into a new EAP, you will now get an EAP that is functionally identical to the original.

There are still other issues - if you make changes to connectors that have endpoints in two different XMI but don't re-export both the XMI, you may have connector information in the two XMIs that conflicts (eg. one XMI says connector c connects elements x and y, and the other XMI says connector c connects elements w and y).

In this case, the result of a re-import depends on the order you load the XMI files.

In my opinion, the fix for this problem is here and I hope Sparx seriously considers implementing it.

Darren