Author Topic: After Synchronise with DBMS a number of tables are not longer in Browser  (Read 21132 times)

jandtr

  • EA Novice
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Hi All,

This stumper of an issue is on v15.2 and v14.1.1428 ultimate versions

We're doing some development of micro services for encapsulating/obfuscating DBMS access. So we need the physical DBMS schema in the repository in order to use these in analysis and architecture diagrams describing the various micro services.

The DBMS schema evolves slightly and thus we do a 'show differences' in the DB Builder, this runs, show some minor differences, we import these and done. The generated diagram is somewhat clumsy, a half an hour later that is nicely formatted, the world looks great.

However when wanting to drag a table from the browser on a service architecture diagram I discover that the table is not present in the Project Browser, yet is clearly present in the diagram generated by the DB Builder. On closer inspection about 40% of the tables are not visible in the Project Browser.

When we search in the model these 'invisible' tables all show up in the result, we can ctrl+U and the diagrams are given correctly, the 'Find in Project Browser' however returns nothing.

We export CSV the Tables package and all the tables are present in the package csv export, even those 'invisible' ot visible in the Project Browser.

I write a quick script to dump the Package.Elements collection on the screen/file and again all the tables are present in the dump, also those not visible.

Last check via a SQL query on the repository (SQL Server) again all tables present, including the invisible ones.

Did an Integrity Check, some minor items, but nothing on the tables or the Database Package.
One peculiar thing was that when I did an XMI export of the Tables Package only the visible tables where exported.

Conclusion:
- all tables are correctly stored in the repository in the correct package;
- the generated diagram is complete and correct, the Project Browser is not having all tables.

A stumper if one wants to drag one of those ínvisible' tables onto a diagram, one cannot drag from diagram to diagram.

I've checked types, Stereotypes, FQStereotypes, Profile Types, PackageID, ...
Compared the diagram objects with the Package Elements and all that seems to be correct.
I've transferred repository to new server with new DBMS instance, went from DBMS based to eapx based, nothing helped.

Question is now: how to render those invisible tables again visible in the Project Browser.

Has anybody experienced this? And found a way out whitout reloading the DBMS again.
As a number of those tables are already in architecture diagrams an thus have a bunch of links and relations  to other elements in our repository, a clean reload anew from scratch would undo all that work, something I would like to avoid.

Has anyone an idea how Sparx populates the Project Browser? Can we script-activate this? And why some tables remain invisible, while apparently present in the Tables Package?

All help would be greatly appreciated.

Jan
 

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13070
  • Karma: +544/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #1 on: January 07, 2021, 06:02:59 pm »
Hi Jan,

I've never heard of something like this that couldn't be fixed by the project integrity check.

Definitely worth sending to Sparx support.

What I would do is query t_object for a table that is visible, and a table that isn't visible, and then do a field by field comparison.
That should provide you with the clue.

The thing I would look most closely at is the parentID, as that (and the packageID) determines where in the project browser an element should be shown.

But in my experience, even if you set the parentID to an element that isn't shown in the project browser (e.g. a boundary) EA still shows the element under the package.
(at least in version 15.2, haven't tested this in earlier versions)

Geert

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8561
  • Karma: +254/-129
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #2 on: January 07, 2021, 07:34:56 pm »
Hi Jan,

You mentioned some issues in the Integrity Check.  Did you repair the repository after that or are the errors still there?  If the errors are still there you need to repair them.  Until you do, "all bets are off".

Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

jandtr

  • EA Novice
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #3 on: January 07, 2021, 09:29:43 pm »
Hi Geert, Paolo,

@ Paolo yes the correction were executed the project reloaded same result some are invisible.

@Geert the Integrity checks dealt with connector (duplicate and wrong type) and one duplicate GUID on an ArchiMate_ApplicationComponent, I suspect an import or CSV upload. The correction was done, project reloaded, even a close and re-open, same result some are invisible.

What strikes me is that the visible tables all have t_object.Object_Type = 'Class' whereas the invisible ones have t_object.Object_Type = 'Table'.



GRB_ADP is invisible but GNBS_VlA2005 is visible.

Now, as I understand it the t_object.Object_Type is the same as EA.Element.Type, or am I missing something?

So when I do an csv export these two table both report CLass for Element.Type.?
What is happening here?

GNBS_VLA2005   Class{AD908E5F-3BB2-425d-A700-9BFD4BCB8C9F}   table   Implemented   EAUML::Table   EAUML::table   SQL Server 2012   
GRB_ADP   Class   {675834A8-2B9C-407f-ABD7-06F0BE70B2D2}   table   Implemented   EAUML::Table   EAUML::table   SQL Server 2012   

I can try now the obvious and with an SQL update set all the t_object.Object_Type to Class.
Especially since the t_object.Object_type on a different demo  DB I just uploaded all report Çlass for t_object.Object_Type.

Question then remains, how did itchange the t_object.Object_Type to Table?

Before I continue with the SQL, any thoughts?

Cheers,

Jan

« Last Edit: January 07, 2021, 09:38:56 pm by jandtr »

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13070
  • Karma: +544/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #4 on: January 07, 2021, 10:10:44 pm »
It should indeed be Object_Type = "Class" and Stereotype = "Table"

Object_Type = "Table" is definitely wrong.

Geert

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8561
  • Karma: +254/-129
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #5 on: January 07, 2021, 10:55:03 pm »
It should indeed be Object_Type = "Class" and Stereotype = "Table"

Object_Type = "Table" is definitely wrong.

Geert
Wot 'e sed!

Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

qwerty

  • EA Guru
  • *****
  • Posts: 13545
  • Karma: +395/-300
  • I'm no guru at all
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #6 on: January 08, 2021, 01:16:09 am »
EA's query results are mingled and not a direct representation of the table contents. Esp. object type is changed towards the metatype defined for the element.

q.

Uffe

  • EA Practitioner
  • ***
  • Posts: 1859
  • Karma: +133/-14
  • Flutes: 1; Clarinets: 1; Saxes: 5 and counting
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #7 on: January 08, 2021, 02:24:25 am »
Hey guys,


I did a bit of a dive on this. Find something to hold on to, and make sure you've got some paracetamol to hand.

Object_Type = "Table" is definitely wrong.
Well... For a given value of "definitely", perhaps. But this. Is. EA!!! (cue Gerard Butler)

It is, in fact, the Class/«table» elements that are wrong. The correct representation is Table/«table».

You can test this very easily (this is on 1557).
If you create a table in a Data Modeling diagram, the element gets Object_Type "Table" and stereotype «table».
If you import a schema from a DBMS, again you get Table/«table» (and View/«view»).
Whether this is something that's been changed recently I can't say. I did a quick check of the histories and there's nothing that indicates that it has, but it is possible.

Now, as I understand it the t_object.Object_Type is the same as EA.Element.Type, or am I missing something?
You are. The profile-defined metatype.

Element.Type is sort of the UML base type. The API documentation lists the valid contents, and you'll note Table isn't in there but Class is.
t_object.Object_Type, on the other hand, holds the metatype if there is one and the base type only if there isn't. In this case, the metatype is Table, as you can see in the corresponding Element.Metatype attribute.

Now I say that Element.Type is a "sort of" base type, and unfortunately that's because this is pretty sloppily defined. Again, if you refer to the API documentation you'll see that it's not just UML base types in there (Activity, Class, Component...) but also some of EA's core extensions such as GUIElement, Issue and Requirement.

EA's representation of a database table, however, is not a separate element type in the core extensions, but a profile-defined metatype.

There are essentially four categories of element type in EA:
  • Types defined in the UML standard;
  • Types defined as their own element types in EA (Requirement);
  • Types defined with their own metatypes in Sparx-supplied profiles (Table), which get special treatment in the GUI and various functions (more below);
  • Types defined with their own metatypes in third-party profiles, which don't get special treatment.
I don't think that EA actually differentiates between categories 1 and 2. They're just what we might call built-in types, and what's listed under the Type attribute in the Element API documentation is the sum total.
Telling categories 1/2 and 3 apart can be very difficult, and again, I don't know of any documentation that clarifies it other than that list.

The in-project representation gets a little tricky because there is no "Metatype" column in t_object. There's .Object_Type and .Stereotype, and that's it. (I'm choosing to ignore .NType, as far as I've been able to ascertain it doesn't enter into this.)

t_object.Object_Type will hold the (cat 3/4) metatype if one is defined, and only if the element has no profile-defined metatype will .Object_Type hold the (cat 1/2) base element type. (Whether a stereotype has a profile-defined metatype is a choice made by the profile designer.)
In both cases, t_object.Stereotype will hold the stereotype's simple name. The fully-qualified stereotype is stored in t_xref.

When going through the API, EA does it differently because there are separate Element attributes for .Type, .Metatype, .Stereotype and .FQStereotype.

So a correctly created database table element should appear as follows:
  • Database
    • t_object.Object_Type = 'Table'
    • t_object.Stereotype  = 'table'
    • t_xref.Description   = '@STEREO;Name=table;FQName=EAUML::table;@ENDSTEREO;'
  • API
    • Element.Type         = 'Class'
    • Element.Metatype     = 'Table'
    • Element.Stereotype   = 'table'
    • Element.FQStereotype = 'EAUML::table'
As I was writing this, Qwerty responded.
EA's query results are mingled and not a direct representation of the table contents. Esp. object type is changed towards the metatype defined for the element.
Normally I defer to Q on the details of the database, but are you actually sure about this? I would have assumed that EA stores the metatype as I describe above, not that it fudges the query results.

In either case, the CSV import/export also differentiates between the base element type and the metatype, and I suspect this is what's causing the problem. In a CSV specification, the metatype is referred to as "Profile Metatype" and it must be included for a CSV import to work when dealing with metatyped elements.

A proper export of a package containing a correctly defined database table should look something like
    GUID;Type;Profile Metatype;Stereotype;Fully Qualified Stereotype;Name
    {E91B88AA-AB9F-40ae-B970-34D6CA0E9C8B};Class;EAUML::Table;table;EAUML::table;Tabellen Ellen

(I've included the simple stereotype name for completeness. It should be omitted in production.)

As you can see, the CSV import/export also includes the profile name in the metatype name. The API does not, and it isn't stored that way in t_object.

So.

Quote
So when I do an csv export these two table both report CLass for Element.Type.?
What is happening here?
You are exporting only the base type, not the metatype.
This should be obvious when you import the data into another project, but unfortunately EA obscures this error by being "helpful".

If your CSV data contains Type and Stereotype, but not Profile Metatype, then on import EA won't know that you want to create Table/«table» elements. It will find only Class/«table» elements in there, and will create exactly that.
(I am assuming that you are working with simple stereotypes too. I believe fully-qualified stereotypes might also have worked, but I haven't tested this.)

But here's where it gets murky. In a fresh project created from scratch (that is, from the default base project) EA includes a bunch of in-project stereotypes which do not belong to any profiles at all. Crucially, there is a non-profile «table» stereotype in amongst them.

So it is quite possible to create a Class in a regular class diagram, then set its stereotype to the profile-less «table».
You can also set the stereotype of a Class to «table» from the EAUML profile. Of course you can -- the EAUML::table stereotype applies to Class, after all.

If you set a class stereotype to EAUML::table, EA changes its t_object.Object_Type from 'Class' to 'Table'.
If you set a class stereotype to the profile-less «table», EA sets its t_object.Object_Type to 'Class'. So EA does a lot of footwork when you change stereotypes, and importantly, you can change a properly-defined Table/«table» element to an ill-defined Class/«table» this way.

This "helpfulness" isn't limited to the ill-defined profile-less «table» stereotype. EA also treats a Class/«table» as if it were a proper Table/«table»: it shows it as a database table in diagrams, and when prompted to open the properties dialog it opens the table-specific dialog instead of the generic class one, allowing you to edit columns instead of attributes, etc. But the project browser does show the difference between a Table/«table» and a Class/«table» in that only a Table/«table» has the table-style icon.

Finally, you mention that
Quote
I've checked types, Stereotypes, FQStereotypes, Profile Types, PackageID, ...
This omits both metatypes and GUIDs. I've talked (at length) about metatypes but if you've somehow managed to get duplicate GUIDs in t_object your project is seriously broken, although I would expect the integrity check to pick up on that.

Going forward, your CSV specifications should always include metatypes and only ever use fully-qualified stereotypes, and all new projects should immediately be cleared of all "helpful" stereotypes and tagged value types. It is worth setting up your own base project for this.

I hope this helps. I really do. :)


/Uffe
My theories are always correct, just apply them to the right reality.

qwerty

  • EA Guru
  • *****
  • Posts: 13545
  • Karma: +395/-300
  • I'm no guru at all
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #8 on: January 08, 2021, 03:22:18 am »
Normally I defer to Q on the details of the database, but are you actually sure about this? I would have assumed that EA stores the metatype as I describe above, not that it fudges the query results.
Hmm, well, EA ... damn. I usually don't fiddle with databases and now tried to create a table. Though it offered me the Databse Engineering MDG it is not shown in the toolboxes. Nor does it offer any diagrams from there. I'm sure in former EA versions I could use ist. But probably they have under the hood altered the licensing. Why is it shown and why does it allow me to enable it? Ok. Next time I just throw it out of the window.

Alas. Will see if I find another example, but I pretty well remember that I'm right. Of course with age comes bad memory.

q.

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13070
  • Karma: +544/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #9 on: January 08, 2021, 03:26:59 am »
Uffe,

If you use a query tool other than EA you'll notice that you can't trust EA when it comes to query results.
I really does manipulate the Object_Type field to show the metatype.

Even in EA you can see that if you use a query like this in the SQL Scratch pad

Code: [Select]
Select o.Object_Type as MyType,  o.Name from t_object o where o.ea_guid = '{7AB08A1E-E5A3-4ee9-85CE-3D45994E336A}'This gets me "Class" as type

This query however
Code: [Select]
Select * from t_object o where o.ea_guid = '{7AB08A1E-E5A3-4ee9-85CE-3D45994E336A}'Returns "Table" in the field as Object_Type

Geert

qwerty

  • EA Guru
  • *****
  • Posts: 13545
  • Karma: +395/-300
  • I'm no guru at all
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #10 on: January 08, 2021, 03:30:56 am »
I added a BusinessProcess from BPMN. The query shows "BusinessProcess" for object_type and element.type contains Activity.

q.

jandtr

  • EA Novice
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #11 on: January 08, 2021, 03:47:24 am »
Thanks Uffe, you made it all very clear.  :)
I followed your advice made the corrections and now all the tables are back in the Browser.
Too bad much of what you wrote is absent from the Sparx Documentation, especially the Profile 'influence'.

Thx

Jan

Uffe

  • EA Practitioner
  • ***
  • Posts: 1859
  • Karma: +133/-14
  • Flutes: 1; Clarinets: 1; Saxes: 5 and counting
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #12 on: January 08, 2021, 06:50:51 am »
Oh man you guys, you're right. Except it's even worse. The Object_Type returned in a query changes with... Hell, I don't know. Barometric pressure?

But try this.
  • Create a new project.
  • Create a Data Modeling diagram, and create a table in it.
  • Exit EA.
  • Start EA again and open the project.
  • Without opening the package with the table in it, run select * from t_object
    Note that you get Object_Type = 'Class'.
  • Open the diagram, or reload the package containing it. Rerun the query.
    Ta-daa! You now get Object_Type = 'Table'.
Reloading the project resets the reported Object_Type to 'Class'. Yay - consistency!

I think, and this is the purest speculation, that MDG technologies only get loaded when needed. So EA only loads the technology containing the EAUML profile once you expand a package, or open a diagram, containing a table. And the smarts that translate, or shall we say obfuscate, the Object_Type are part of the technology.
So while the technology isnt loaded, tables are reported as 'Class'. Once the technology is loaded, they instead become 'Table'.


Buuut at the end of the day, the point is that CSV exports must include the Profile Metatype (or possibly the Fully Qualified Stereotype is sufficient) or elements with profile-defined metatypes won't import right.

Too bad much of what you wrote is absent from the Sparx Documentation, especially the Profile 'influence'.
Yeah, that kinda sucks. It looks like they gave up on supplying user documentation years ago and just started copy-pasting the marketing blurbs into the manual. But that's why there's a forum. :)


/Uffe
My theories are always correct, just apply them to the right reality.

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8561
  • Karma: +254/-129
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #13 on: January 08, 2021, 11:39:47 am »
Oh man you guys, you're right. Except it's even worse. The Object_Type returned in a query changes with... Hell, I don't know. Barometric pressure?

Quote
[SNIP]
Hi Uffe,
Not Barometric pressure, but close!!!

It is most likely related to "Browser Pressure".  ;) ::)

It is highly probable that when the metatype is returned, you have applied pressure to the browser and "opened" the item in the browser and made it visible.  Otherwise, it will return the base type.

This can be seen in the Project Search output where metatype will be displayed (seemingly randomly) for the Type column.  When I enquired about this inconsistency I was told that was the reason why.  Go figure how labyrinthine the code has to be for that to be true!  ::)  How can there be a rationale for such functionality?  Vertex Type + Profile Stereotype = Profile Metatype.  Can't get simpler than that!  If you're going to put the metatype on the Type column in the Project Search, it's a straight mapping, no?

Anyway, that may help.

Also, as to this (problematic) idea that all MDG Technologies shall be available but only loaded when required.  In the testing I've been doing regarding some missing links on package import, I used the QuickLinker to select a <MyProfile>::ControlFlow (caption: flows to) and I didn't notice that the was another arc, (caption: Flows to) which appears "higher up the list" and was the first I saw.  EA delivered me a SysML 1.5::ItemFlow (and InformationFlow) rather than what I was expecting.  SysML 1.5 IS NOT a SELECTED Technology, so what is it doing in the QuickLinker?

Exasperated,
Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

Uffe

  • EA Practitioner
  • ***
  • Posts: 1859
  • Karma: +133/-14
  • Flutes: 1; Clarinets: 1; Saxes: 5 and counting
    • View Profile
Re: After Synchronise with DBMS a number of tables are not longer in Browser
« Reply #14 on: January 08, 2021, 09:39:04 pm »
Vertex Type + Profile Stereotype = Profile Metatype.  Can't get simpler than that!  If you're going to put the metatype on the Type column in the Project Search, it's a straight mapping, no?

Well, it's pretty clear that EA doesn't use an internal metamodel. Or at least if it is, it's not the UML metamodel.
In fairness, UML itself has been changing a fair bit over the years and the MOF didn't even exist when EA was first designed cobbled together.

Quote
Also, as to this (problematic) idea that all MDG Technologies shall be available but only loaded when required.
[CLIP]

Yeah. It's all sorts of weird. Again, it's pretty clear EA isn't so much engineered as hacked.
All these variations on the theme of which technologies are used and which not is a prime example of what I was on about the other day where the same functionality is implemented multiple times that don't work together (and quite possibly not even each on their own either).

Unfortunately, while it's fun to do a deep dive every once in a while and try to work out how stuff actually works doesn't work, it's an exercise in futility because there is no way of telling how long the results will be valid. Which, also on a topic I was recently on, makes it less rewarding for the community to actually pitch in and help make the whole EA experience better.


/U
My theories are always correct, just apply them to the right reality.