Author Topic: Neo4j and Sparx EA  (Read 1558 times)

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8505
  • Karma: +250/-129
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Neo4j and Sparx EA
« on: February 22, 2023, 09:52:45 pm »
We're looking at having Sparx EA and Neo4j interact.  Our "Proof of Concept" involves exporting the EA reverse-engineered structures for a very complex SQL Server database and examining paths between the various tables using Foreign Key Constraints.

Is anybody engaged in making EA and Neo4j interact?  If so, would you be prpeared to shareany knowledge or wisdom you have acquired in the process?

TIA,
Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

Sunshine

  • EA Practitioner
  • ***
  • Posts: 1268
  • Karma: +114/-10
  • Its the results that count
    • View Profile
Re: Neo4j and Sparx EA
« Reply #1 on: March 06, 2023, 08:59:02 pm »
Think that sounds like you are breaking new ground. Not heard of any of my contacts using sparx EA doing  that.
Happy to help
:)

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8505
  • Karma: +250/-129
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: Neo4j and Sparx EA
« Reply #2 on: March 08, 2023, 09:24:42 am »
I think that sounds like you are breaking new ground. Not heard of any of my contacts using Sparx EA doing that.
Hi Sunshine,
It also sounds like it in the Neo4j world (from what I can gather).

Fortunately, one of my colleagues has a little experience with Neo4j, so he got me "kick-started".

After some testing, we've developed an architecture to allow my concept of "same semantics, different syntax" to be instantiated in both products.  We're currently moving metadata between the two products via CSV files, but my colleague will provide some sample code to start using the Neo4j API - giving us a more direct connection.

Having gotten used to Cypher's (the Neo4j Query Language) syntax, we're now able to query a model (in this case of a reverse-engineered database) and enrich the model with the inferences we've found.

If there's continuing interest, I'll keep everyone posted.

Paolo

Fun Fact: the first DBMS I created (over 4 decades ago in my Grad. Dip. EDP project) was an in-memory, dynamic-schema graph database (written n Algol-68R). So using Neo4j is like a "blast from the past"!
« Last Edit: March 08, 2023, 09:28:00 am by Paolo F Cantoni »
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

duncan.greenyer

  • EA User
  • **
  • Posts: 27
  • Karma: +0/-0
    • View Profile
Re: Neo4j and Sparx EA
« Reply #3 on: April 05, 2023, 06:12:32 pm »
Why move the data at all? Why not create a cypher query engine in javascript that can use the ea object model to traverse the links then you could use graph querying directly in the tool.

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8505
  • Karma: +250/-129
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: Neo4j and Sparx EA
« Reply #4 on: April 06, 2023, 08:09:08 pm »
Why move the data at all? Why not create a cypher query engine in javascript that can use the ea object model to traverse the links? Then you could use graph querying directly in the tool.
Hi Duncan,
As I hinted in the original post, our knowledge of Neo4j and cypher was scant (and, in my case, non-existent)!  I subsequently discovered my colleague's knowledge was limited to setting up the DB and running simple queries on an already created DB.

Consequently, we were up for a very steep learning curve.  Having now had about a month's experience I can say that the idea of creating a cypher query engine is the equivalent of asking a Sparx EA newbie to create a sophisticated C# add-in on day one.

Notwithstanding that, the Sparx EA database design is "less than optimal".  We can create much more useful properties in Neo4j that will make it much easier to traverse the nodes and relationships in the Neo4j structure than in the Sparx EA structure.  This exemplifies my concept of "same semantics, different syntax".

We've also realised that using Neo4j is like using C++ - there seems to be a lot of "it's your foot" stuff going on.  It is really easy to make errors in DB loading since it has very few constraints.  "The good news is that Neo4j is extremely flexible; the bad news is that Neo4j is extremely flexible!"

We have been working on creating metamodels for storage in the DB that allows us to run validation queries against the loaded data - analogous to referential integrity and other data constraints in RDBMs.  We can then be confident in the validity of our data.

Recalling this was a proof of concept to help us understand if graph databases would be useful for our main data stores.  Trying to do what we've done over the last month is probably the most conceptually difficult thing we'd have to do in graph databases for a while, so the experience has been most valuable!

Finally, we found that creating the metamodel in the Neo4j DB helped us to realise that the relationships we had in the Sparx EA repository were conceptually incorrect!  This was an unexpected but very useful benefit of the process!

Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

duncan.greenyer

  • EA User
  • **
  • Posts: 27
  • Karma: +0/-0
    • View Profile
Re: Neo4j and Sparx EA
« Reply #5 on: April 07, 2023, 08:23:30 pm »
I was thinking that you could look beyond the database and see that regardless of how EA stores its data conceptually EA, as a tool, IS a graph and therefore could be queried as such. But sounds like you have a solution, I just bought the cypher book from packt so might have a go at seeing if I can write a cypher query parser in javascript on EA.