Model Navigation From CTF

[Chris Brandt]

I'm doing a bunch of code generation work right now. I'm finding that the CTF is a limited in terms of scalability and model navigation.

My main suggestion is to add the ability to work with a linked element's properties (attributes, and tagged values mainly). Here is my scenario:

I'm code generating a data access layer in which each table has a Value Object and a Data Access Object. The value objects are easy to generate with the CTF. The data access objects are more challenging because they need to be able to set property values in their associated value object.  So by model navigation, I mean the ability to access the operations and attributes (and their tagged values) from a related class.

I get the distinct feeling that I'm pushing the limits of what the CTF was designed for.  I've been looking at CodeSmith and I really like that approach, except that it requires a separate tool. I would rather model and generate from one tool.

[Eve]

Yes, you're definately pushing beyond what the templates were designed for.

The only suggestion that I can think of is making you can call an addin to do the navigation the templates are incapable of.  An example of this would be to get a tagged value named "foo" from the class at the other end of a connector.

%EXEC_ADD_IN("MyGeneratorHelper","GetClassTag", connectorDestElemGUID,"foo")%

Obviously this isn't really a trivial solution, and I know it would get very complex very quickly.  But it's a possible solution.

Happy code engineering.

Simon.

PS. I'm curious about what you actually trying to do.

[Paolo F Cantoni]

I get the distinct feeling that I'm pushing the limits of what the CTF was designed for.  I've been looking at CodeSmith and I really like that approach, except that it requires a separate tool. I would rather model and generate from one tool.

Hi,

I took the CodeSmith route...  I decided to decouple the process of modelling from the process of Code Generation (NOTE: NOT Code Round-Trip-Engineering).

So far, it works a "Ripper" as we say here in  OZ

I've run some tests and the principle seems to work.  I emit the model data and metadata via CTF as XML according to my specified XSD (via a new language) and then use CodeSmith to generate multiple artifacts from the same source.  This can be narrative, code, tests, data DB Designs.  The cost of CodeSmith is more than repaid by the savings.

I'm currently refactoring the pre-production prototypes for more serious work.

I thoroughly recommend CodeSmith.

Paolo

[Chris Brandt]

That's the direction I've started to head. It appears to me that there are three choices for integration between EA and CodeSmith:

1.  As you have done, emit an Xml representation of the model and generate code from that. How did you create the XML schema? Is it domain specific? I'd be interested in hearing more about that.

2. Setup CodeSmith templates and functions to navigate the EA model database. I'm envisioning a top level template with a package name as a property. Key in the package name and hit Run. Alas, CodeSmith seems to only work with SQL databases and we are using an EAP file (Access dB). So I'm not sure if I'll get to try this route.

3. Create a set of CodeSmith templates with properties constrained to model elements accessible to the CTF. Then write a CodeSmith API interface that will be called from an EA file template. I'm kind of working in this direction right now, but I'm realizing that it's probably the weakest of the choices.

My chief concern is keeping the development process model driven. Most code generation tools, especially those geared toward data access layers seem to work on the belief that we should be generating code from an existing database implementation. I think that's bass ackwards. If I can create a database model from which my database entities are created, then surely I can create a model of the associated data access objects; probably automatically. Then generate the whole thing at once (tables, stored procedures, C# classes...).

Anyway, I'd love to hear more about your experiences with CodeSmith and model integration. There doesn't seem to be much chatter about code generation in the EA forums.  

[Paolo F Cantoni]

That's the direction I've started to head. It appears to me that there are three choices for integration between EA and CodeSmith:

1.  As you have done, emit an XML representation of the model and generate code from that. How did you create the XML schema? Is it domain specific? I'd be interested in hearing more about that.

I created the XML schema from first principles.  I'm using XMLSpy to manage the XSD creation at present.  
It isn't domain specific in the normal sense of the word, but I'm customising it to my modelling domain.  That is to say, I'm principally concerned with Conceptual Modelling.  I create a Conceptual Model and then forward engineer using a set of defined transforms to a physical representation - be that narrative, code, DB Scripts etc.  
So the XSD concentrates on  what conceptual data and metadata do I need to emit to tell the downstream generators what to do.  I also "enhance" UML to achieve this end.

Thus, if you intend to essentially create "physical" or "logical" models and forward engineer those, your XSD may be different from mine.

I create separate XSDs to handle the physical manifestation of the concept.  Thus in the conceptual model I may have a UML Datatype that actually corresponds to a conceptual domain (bruce will like this ) such that for this attribute the allowable range of values is 0-24.  
I emit this information and the downstream physical generator can decide whether (for a particular technology) to use an unsigned int, number(2), integer or whatever - AND what constraints to generate!

I have other XSDs that describe how the analysis and design patterns are to be instantiated (for this run)

I have another XSD to decide what part of the model needs to be assembled into the artifact I'm creating and what policies and parameter bindings need to be applied for this run.

All of these XSDs are used by CodeSmith to decide what to emit as an artifact (based upon the contents of the particular instance of the XML which conforms to the XSD).

That's about as far as I've got.

I've done enough to prove the concept (to me at least) and I'm now refactoring as a result of the proof of concept experience.

I'm building a model narrator first to allow me to see what the model is "saying" and thereby fix the model when it speaks rubbish.  I've used this technique in the past with both my own Conceptual Modelling Tool and Rational Rose to great effect.

Then once I know the model correctly represents what's in my (or others) head, then I'll work on how do I convert that to code.  When I (actually) write code, I'm always abstracting out to how this looks in UML and so since I've been doing that for a while, I know how (in principle at least) I can turn the model in my head to code manually.  Since the model in my head is now the model in EA...

By using CodeSmith's region substitution capability, I'm able to progressively refactor existing code to the model.

My first coding project will be to create an add-in for EA to emit the WHOLE model to  the XML - since as you will quickly discover, CTF is primarily geared toward the explicit requirements of existing coding languages.  So, for example, you can't get access to ALL the Class's data and metadata; let alone other classifiers like Use Cases etc.  

So, since I know nothing about EA Add-ins, I'll write a tiny one first, manually.  Then I'll bring it into the model and make sure I can regenerate the bits I've got modelled.  Thereafter, I'll progressively expand the model -> code linkage till I've got as much as I can in the model.

I'd appreciate feedback on whether this is a viable approach.  I'd REALLY appreciate the provision, by some generous soul, of any working Add-In code that traverses the model and generates any textual output - so I can reduce the learning curve regarding EA internals.

Subject to Commercial IP constraints - which I'm following up with my employers.  I have sufficient "prior art" with these concepts that I feel I can at least discuss the principles involved.  If people are interested.

Hope this all makes sense since it was a bit of "stream of consciousness" stuff.

Cheerz,
Paolo

[Chris Brandt]

Wow, you're way ahead of us and by the looks of it most of the industry.

I'm curious about your definition of Conceptual Modelling. Can you give me an example of a conceptual model? Can you generate code from a conceptual model (I suppose it depends upon the model)?

I'm also curious about the process you used to define your XSDs. My instinct would have me working from artifact back to model. So take a sample of the artifact that I want to generate, create a first cut template for it. Then use the template parameters to start the definition of the intermediate language. Then try to emit the intermediate language from the model.

How difficult was it to generate the intermediate XML from the model through the CTF? Do you do it directly in the templates or do you call an external component?

I'll understand if you can't answer some of these questions due to IP concerns.

Chris.

[Paolo F Cantoni]

Wow, you're way ahead of us and by the looks of it most of the industry.

Thank you, Chris.  I suspect I am a bit ahead - but who knows?  The big problem for me is getting people to understand what I'm doing and why...
One of my managerial colleagues recently said: "You're so close to the Holy Grail, you can smell it!"  Which is pretty true for me...  That's why I'm so excited!
Also, why should I be ahead... You're working on, potentially, similar lines.  The stuff I'm doing is, essentially, from first principles...

I'm curious about your definition of Conceptual Modelling. Can you give me an example of a conceptual model? Can you generate code from a conceptual model (I suppose it depends upon the model)?

I don't model systems directly.  I create models of the world and then implement them in systems.  The reasoning for this is: In a battle between the real world and the system, who's going to win?  So make sure your system is consistent (that word again) with the world in which it needs to operate...
For example, in a previous life, I was involved in medical fraud detection.  I built a (principally) data model of what a health claim looked like (not the form, but it's conceptual content), added doctors, patients, insurance providers, addresses, diagnoses, treatments etc.  I use a mechanism that allows me to output the model content as narrative - the model "speaks" back to you it's business rules (as best I can model them).  The narrative rules are directly intelligible to a domain expert who knows nothing about modelling.
Thus these models are directly conceptual.  They model the business.  One of the problems with CASE tools is that they are limited by the fact that they are targeted at code generation.  Therefore they have only enough syntax and semantics to "cut code".  To conceptually model, you need more than that.  However, with UML2, I can pretty-much accomplish everything I need with UML2 and a few "tweaks".
Does that explain things better?

I'm also curious about the process you used to define your XSDs. My instinct would have me working from artifact back to model. So take a sample of the artifact that I want to generate, create a first cut template for it. Then use the template parameters to start the definition of the intermediate language. Then try to emit the intermediate language from the model.

Yes, I've been through that process, except now I'm targeting multiple artifacts.  I've had the fortune of having led a varied professional life, I've coded, designed, architected, tested, business analysed, consulted etc - pretty much done everything.  In each area, I've thought about what information (metadata) would I need to ensure I did the right job and produced the needed outcome?
So the approach I took was:  What do I need from the model?
Next I created a conceptual model of the world I needed to work in - the model.  I created this model in XMLSpy.  The XSD was the form (and the output), but I was modelling!  In my model, A Model, consists of Namespaces with Elements (Sparxians are you listening?  Everything is an element!).  The Elements are subtyped into Classifiers (Shapes), Relations (Connectors), Containers (Packages, Diagrams etc).  From there into the particular etc...  
At present, I'm using the CTF.  So the model is most complete in that area.  However, to make sure I can expand into the full model later after my bootstrap, I've modelled a lot of the other stuff as well (to a greater or lesser degree).

How difficult was it to generate the intermediate XML from the model through the CTF? Do you do it directly in the templates or do you call an external component?

Surprisingly easy!  Once you get your head around the way the CTF works!  I can basically emit almost all I need (in the Class diagram area) with the CTF.  I did have to make one compromise.  In the CTF, you can only get those relations outbound from you (as far as I can see) so I've split the relations into bi-directional links which are joined by the GUID.  Each Classifier Element contains a list of outbound links.  Within CodeSmith, I will reassemble the links into n-ary relations, as required.

I'll understand if you can't answer some of these questions due to IP concerns.

Chris.

I've had some clarification and I can keep talking, for the present.
Does this help?

Cheerz,
Paolo

[Chris Brandt]

Also, why should I be ahead... You're working on, potentially, similar lines.  The stuff I'm doing is, essentially, from first principles...

I'm just cutting my teeth on this stuff. Trying to get a sense of how I'm best to proceed. And unfortunately, this is only a side-bar to what I should be working on.

I don't model systems directly.  I create models of the world and then implement them in systems. ... Does that explain things better?

Yes. And I'm inspired! I can see that [glb]holy grail[/glb], can't smell it yet, but I see it.  Being able to generate multiple views (code, narrative business rules...) from one model would be a huge boon for us.

So the approach I took was:  What do I need from the model?
Next I created a conceptual model of the world I needed to work in - the model.  I created this model in XMLSpy.  The XSD was the form (and the output), but I was modelling!  In my model, A Model, consists of Namespaces with Elements (Sparxians are you listening?  Everything is an element!).  The Elements are subtyped into Classifiers (Shapes), Relations (Connectors), Containers (Packages, Diagrams etc).  From there into the particular etc...  

As I mentioned earlier I've started down this path with the intention of code generating a Data Access Layer.  Assume that the DAL exists, thus it can be modelled. If I understand your methodology correctly, you would look at that model as the real world, and decide what you need to get from it. Then you would model the information you need to get from it, in an intermediate language. Finally build the infrastructure to emit and consume documents. The emitter possibly being a CTF template, the consumer possibly being a CodeSmith template. Sound about right?

Thanks.
Chris.

[Paolo F Cantoni]

I'm just cutting my teeth on this stuff. Trying to get a sense of how I'm best to proceed. And unfortunately, this is only a side-bar to what I should be working on.

Chris,
If this is a side-bar then I'd recommend going to the CodeSmith forums and looking at some of the seriously funky existing templates they have for this sort of stuff...  They even have a forum dedicated purely to CSLA and another to .NET Tiers.  The reasons for this I'll expound below...  But principally, why re-invent the wheel?  (Well actually to build a better wheel... - An Aussie inventor has recently done this - but I digress).

Yes. And I'm inspired! I can see that [glb]holy grail[/glb], can't smell it yet, but I see it.  Being able to generate multiple views (code, narrative business rules...) from one model would be a huge boon for us.

So why go direct to CodeSmith?  Well, do you need the single view right now?  No? Then defer (for the present).  Have you extensive experience in DAL creation and multi-layer communication?  No, have a look at how some experts have done it...  Are you a CodeSmith guru?  No, passim... See below...

As I mentioned earlier I've started down this path with the intention of code generating a Data Access Layer.  Assume that the DAL exists, thus it can be modelled. If I understand your methodology correctly, you would look at that model as the real world, and decide what you need to get from it. Then you would model the information you need to get from it, in an intermediate language. Finally build the infrastructure to emit and consume documents. The emitter possibly being a CTF template, the consumer possibly being a CodeSmith template. Sound about right?

Thanks.
Chris.

Yes, in principle, the approach is fine.  It is the slight? modifications that can make a big difference.  Firstly, I would view the world as the conceptual problem of: "How do I synchronise changes in data in an arbitrary number of application with a common persistent repository?"  In other words, I start as abstract as that.  
Having formulated the abstract problem, especially in a new area to me, I'd go and see how the real world works...  In this case, read up on this stuff - preferably in the abstract.  For example, I read the book on CSLA.  Then I'd go and see how others have approached the problem (CodeSmith site) and observe the issues they had to deal with and their solutions.  Now, if I had to actually deliver something quickly, I'd formulate a very-high-level model and then compare the various solutions against that.  Use the existing stuff to get me going and to provide a breathing space to allow me to think clearly without rushing.
Now comes the part where you consider what you need to store in the model to drive the solution you've decided upon.  Now you can consider what other information you need to store/emit to create the documentation, the testing etc.
Then define you interface content and structure and away you go...  Not bound by the fact that you need to deliver immediately - you've already got a solution going.
NOTE: with the CTF, you can only (as far as I can tell) deal only with Class diagrams...

I'm working one level more abstractly:  "How can I arbitrarily model anything and emit that information to a downstream process.  Once emitted, how can I take that information and generate an artifact in an arbitrary formalism?"

Wish me luck!

Paolo