Author Topic: Searching Linked Documents  (Read 6219 times)

shimon

  • EA User
  • **
  • Posts: 167
  • Karma: +5/-0
    • View Profile
Searching Linked Documents
« on: July 26, 2024, 01:32:26 am »
Hi,
We use linked documents to write the Use Cases. The normal Find of EA does not look through these documents. Has anyone created a scriplet or addin to search through these RTF's?

Thanks in advance,
Shimon

qwerty

  • EA Guru
  • *****
  • Posts: 13584
  • Karma: +396/-301
  • I'm no guru at all
    • View Profile
Re: Searching Linked Documents
« Reply #1 on: July 26, 2024, 06:32:02 am »
Just search t_document for elements linked via ElementId with ElementType = ModelDocument

q.

Takeshi K

  • EA User
  • **
  • Posts: 593
  • Karma: +39/-1
    • View Profile
Re: Searching Linked Documents
« Reply #2 on: July 26, 2024, 09:12:27 am »
The linked documents are stored in the t_document table, as Thomas wrote, but they are binary and compressed. So we have to export all the data as RTF files, decompress them and then search for them. This must be a very heavy process, so I guess EA does not (cannot) offer a feature to search in the linked documents.
« Last Edit: July 26, 2024, 09:27:55 am by Takeshi K »
--
t-kouno

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8607
  • Karma: +257/-129
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: Searching Linked Documents
« Reply #3 on: July 26, 2024, 11:19:28 am »
The linked documents are stored in the t_document table, as Thomas wrote, but they are binary and compressed. So we have to export all the data as RTF files, decompress them and then search for them. This must be a very heavy process, so I guess EA does not (cannot) offer a feature to search in the linked documents.
Hi Takeshi-san,
So, technically they are embedded documents not linked documents, yes?  ;)

Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

Takeshi K

  • EA User
  • **
  • Posts: 593
  • Karma: +39/-1
    • View Profile
Re: Searching Linked Documents
« Reply #4 on: July 26, 2024, 11:33:41 am »
Hi Paolo-san,

I think that 'Linked' means the RTF document is linked to an element, not mean a document is stored as a separated file.
There is similar property in the 'Files' tab of the Properies dialog.

The Linked Document (right-click an element in a diagram | Linked Document)
-> the data (as RTF document) is stored in the t_document as zipped binary

The file path / URL for an element (double-click an element in a diagram | the Files group)
-> the path information (as text) is stored in the t_objectfiles as simple string

We might feel the latter is 'linked'  :)
--
t-kouno

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 13404
  • Karma: +567/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Searching Linked Documents
« Reply #5 on: July 26, 2024, 07:14:18 pm »
I have a script search that can be used to search in Linked Documents (regardless whether that is a semantically correct description of the concept)

Code: [Select]
'[path=\Projects\Project A\Search Scripts]
'[group=Search Scripts]
option explicit

!INC Local Scripts.EAConstants-VBScript
!INC Wrappers.Include

'
' This code has been included from the default Search Script template.
' If you wish to modify this template, it is located in the Config\Script Templates
' directory of your EA install path.   
'
' Script Name: In Linked Documents
' Author: Geert Bellekens
' Purpose: Search for a certain string in the linked documents
' Date: 2021-06-25
'

' TODO 1: Define your search specification:
' The columns that will apear in the Model Search window
dim SEARCH_SPECIFICATION
SEARCH_SPECIFICATION = "<ReportViewData>" &_
       "<Fields>" &_
        "<Field name=""CLASSGUID""/>" &_
        "<Field name=""CLASSTYPE"" />" &_
        "<Field name=""Name"" />" &_
        "<Field name=""Type"" />" &_
        "<Field name=""Stereotype"" />" &_
        "<Field name=""Description"" />" &_
        "<Field name=""Path"" />" &_
       "</Fields>" &_
       "<Rows/>" &_
      "</ReportViewData>"

'
' Search Script main function
'
sub OnSearchScript()
 'get the search term
 dim searchTerm
 searchTerm = InputBox( "Please enter term to search for", "search term" )
 'ask if we should look only under selected branch
 dim response
 response = MsgBox("Search only in selected package branch?", vbYesNo+vbQuestion, "Search scope")
 dim package as EA.Package
 if response = vbYes then
  set package = Repository.GetTreeSelectedPackage
 else
  set package = nothing
 end if
 'get the linked documents
 dim allLinkedDocuments
 set allLinkedDocuments = getAllLinkedDocuments(package)
 
 
 ' Create a DOM object to represent the search tree
 dim xmlDOM
 set xmlDOM = CreateObject( "MSXML2.DOMDocument.6.0" )
 xmlDOM.validateOnParse = false
 xmlDOM.async = false
 
 ' Load the search template
 if xmlDOM.loadXML( SEARCH_SPECIFICATION ) = true then
 
  dim rowsNode
  set rowsNode = xmlDOM.selectSingleNode( "//ReportViewData//Rows" )
 
  ' TODO 2: Gather the required data from the repository
  dim element as EA.Element
  for each element in allLinkedDocuments.Keys
   dim ldText
   ldText = allLinkedDocuments(element)
   if instr(1,ldText, searchTerm,  1) > 0 then
    AddRow xmlDOM, rowsNode, element
   end if
  next
'  'debug
'  dim debugFile
'  set debugFile = new TextFile
'  debugFile.Contents = xmlDOM.xml
'  'save the debug file
'  debugFile.FullPath = "I:\temp\scriptDebug.xml"
'  debugFile.Save
'  'end debug
  ' Fill the Model Search window with the results
  Repository.RunModelSearch "", "", "", xmlDOM.xml
 
 else
  Session.Prompt "Failed to load search xml", promptOK
 end if
end sub

'
' TODO 3: Modify this function signature to include all information required for the search
' results. Entire objects (such as elements, attributes, operations etc) may be passed in.
'
' Adds an entry to the xml row node 'rowsNode'
'
sub AddRow( xmlDOM, rowsNode, element)
 
 ' Create a Row node
 dim row
 set row = xmlDOM.createElement( "Row" )
 
 ' Add the Model Search row data to the DOM
 AddField xmlDOM, row, "CLASSGUID", element.elementGUID
 AddField xmlDOM, row, "CLASSTYPE", element.Type
 AddField xmlDOM, row, "Name", element.Name
 AddField xmlDOM, row, "Type", element.Type
 AddField xmlDOM, row, "Stereotype", element.Stereotype
 AddField xmlDOM, row, "Description", element.Notes
 AddField xmlDOM, row, "Path", element.FQName
 
 ' Append the newly created row node to the rows node
 rowsNode.appendChild( row )

end sub

'
' Adds an Element to the DOM called Field which makes up the Row data for the Model Search window.
' <Field name "" value ""/>
'
sub AddField( xmlDOM, row, name, value )

 dim fieldNode
 set fieldNode = xmlDOM.createElement( "Field" )
 
 ' Create first attribute for the name
 dim nameAttribute
 set nameAttribute = xmlDOM.createAttribute( "name" )
 nameAttribute.value = name
 fieldNode.attributes.setNamedItem( nameAttribute )
 
 if len(value) > 0 then
  ' Create second attribute for the value
  dim valueAttribute
  set valueAttribute = xmlDOM.createAttribute( "value" )
  valueAttribute.value = value
  fieldNode.attributes.setNamedItem( valueAttribute )
 end if
 ' Append the fieldNode
 row.appendChild( fieldNode )

end sub

'returns a dictionary of all elements that have a linked document as key and the text of the linked document as value.
function getAllLinkedDocuments(package)
 dim queryString
 queryString = "select o.object_ID from t_document d              " & vbNewLine & _
     " inner join t_object o on d.ElementID = o.ea_guid " & vbNewLine & _
     " where d.ElementType = 'ModelDocument'            "
 if not package is nothing then
  'get package tree id string
  dim packageTreeIDString
  packageTreeIDString = getPackageTreeIDString(package)
  queryString = queryString & vbNewLine & _
      " and o.package_ID in (" & packageTreeIDString & ")"
 end if
 dim elementsWithLinkedDocument
 set elementsWithLinkedDocument = getElementsFromQuery(queryString)
 dim linkedDocumentsDictionary
 set linkedDocumentsDictionary = CreateObject("Scripting.Dictionary")
 dim element as EA.Element
 'loop the elements and add element and its linked document to the dictionary
 for each element in elementsWithLinkedDocument
  dim linkedDocumentText
  linkedDocumentText = getLinkedDocumentContent(element, "TXT")
  linkedDocumentsDictionary.Add element, linkedDocumentText
 next
 set getAllLinkedDocuments = linkedDocumentsDictionary
end function


OnSearchScript()
Geert

shimon

  • EA User
  • **
  • Posts: 167
  • Karma: +5/-0
    • View Profile
Re: Searching Linked Documents
« Reply #6 on: July 30, 2024, 12:50:03 am »
Hi Geert,
Thanks. I'll try to test this soon.
Sincerely,
Shimon

shimon

  • EA User
  • **
  • Posts: 167
  • Karma: +5/-0
    • View Profile
Re: Searching Linked Documents
« Reply #7 on: July 23, 2025, 08:35:39 pm »
Hi Geert,
The need arose again, so I got around to trying this. The script fails on the following two lines.

BrsrGRP.In_Linked_Documents error: Variable is undefined: 'getPackageTreeIDString', Line:155
BrsrGRP.In_Linked_Documents error: Variable is undefined: 'getElementsFromQuery', Line:160

I understood that I needed the scripts and functions that the Wrappers.Include refer to.

It took me a while to import all the necessary folders of scripts to my EA environment, but I got it working at the end.

Since I want to use this as a Model Add-in, I  copied the contents of the Utils\Util script into the In Linked Documents script, and it worked fine without the Imports.

Now I have to find an easy way to port this to Javascript.
I tried Perplexity and I didn't manage yet ( about 15 tries, and didn't seem to be getting there).

Claude did a pretty good job after 10 tries, but asked me to go pro if I wanted to continue.

I put down the 20 USD  (for the month) and got a working solution.

It's not perfect, and not yet in a Model Add-in but I hope to get there soon.


Thanks alot,
Shimon
« Last Edit: July 24, 2025, 02:19:10 am by shimon »

shimon

  • EA User
  • **
  • Posts: 167
  • Karma: +5/-0
    • View Profile
Re: Searching Linked Documents
« Reply #8 on: July 27, 2025, 07:51:15 pm »
Hi all,
I got it working as part of an easily importable Model Addin.
This can be found here.
https://github.com/shimonj/ModelAddinStarter.

It is still in work (for me, as I want to add the ability to search RTL characters), so the code is slightly bloated.
If any of you give it a go, let me know how it behaves.
I would have liked to pack all the helper functions in separate classes, but then it wouldn't be (so)easily portable, as I found the the Import of the Utility classes, is not imported automatically.
I will add a credit to Geert, before my next version.
Sincerely,
Shimon


shimon

  • EA User
  • **
  • Posts: 167
  • Karma: +5/-0
    • View Profile
Re: Searching Linked Documents
« Reply #9 on: July 28, 2025, 04:28:53 pm »
'[path=\Projects\Project A\Search Scripts]
'[group=Search Scripts]
'GB_SearchInLinkedVB.vbs Util functions of Geert,  added by SJ

option explicit

!INC Local Scripts.EAConstants-VBScript
' !INC Wrappers.Include

'
' This code has been included from the default Search Script template.
' If you wish to modify this template, it is located in the Config\Script Templates
' directory of your EA install path.   
'
' Script Name: In Linked Documents
' Author: Geert Bellekens
' Purpose: Search for a certain string in the linked documents
' Date: 2021-06-25
'

' TODO 1: Define your search specification:
' The columns that will apear in the Model Search window
dim SEARCH_SPECIFICATION
SEARCH_SPECIFICATION = "<ReportViewData>" &_
       "<Fields>" &_
        "<Field name=""CLASSGUID""/>" &_
        "<Field name=""CLASSTYPE"" />" &_
        "<Field name=""Name"" />" &_
        "<Field name=""Type"" />" &_
        "<Field name=""Stereotype"" />" &_
        "<Field name=""Description"" />" &_
        "<Field name=""Path"" />" &_
       "</Fields>" &_
       "<Rows/>" &_
      "</ReportViewData>"

'
' Search Script main function
'
sub OnSearchScript()
 'get the search term
 dim searchTerm
 searchTerm = InputBox( "Please enter term to search for", "search term" )
 'ask if we should look only under selected branch
 dim response
 response = MsgBox("Search only in selected package branch?", vbYesNo+vbQuestion, "Search scope")
 dim package as EA.Package
 if response = vbYes then
  set package = Repository.GetTreeSelectedPackage
 else
  set package = nothing
 end if
 'get the linked documents
 dim allLinkedDocuments
 set allLinkedDocuments = getAllLinkedDocuments(package)
 
 
 ' Create a DOM object to represent the search tree
 dim xmlDOM
 set xmlDOM = CreateObject( "MSXML2.DOMDocument.6.0" )
 xmlDOM.validateOnParse = false
 xmlDOM.async = false
 
 ' Load the search template
 if xmlDOM.loadXML( SEARCH_SPECIFICATION ) = true then
 
  dim rowsNode
  set rowsNode = xmlDOM.selectSingleNode( "//ReportViewData//Rows" )
 
  ' TODO 2: Gather the required data from the repository
  dim element as EA.Element
  for each element in allLinkedDocuments.Keys
   dim ldText
   ldText = allLinkedDocuments(element)
   if instr(1,ldText, searchTerm,  1) > 0 then
    AddRow xmlDOM, rowsNode, element
   end if
  next
'  'debug
'  dim debugFile
'  set debugFile = new TextFile
'  debugFile.Contents = xmlDOM.xml
'  'save the debug file
'  debugFile.FullPath = "I:\temp\scriptDebug.xml"
'  debugFile.Save
'  'end debug
  ' Fill the Model Search window with the results
  Repository.RunModelSearch "", "", "", xmlDOM.xml
 
 else
  Session.Prompt "Failed to load search xml", promptOK
 end if
end sub

'
' TODO 3: Modify this function signature to include all information required for the search
' results. Entire objects (such as elements, attributes, operations etc) may be passed in.
'
' Adds an entry to the xml row node 'rowsNode'
'
sub AddRow( xmlDOM, rowsNode, element)
 
 ' Create a Row node
 dim row
 set row = xmlDOM.createElement( "Row" )
 
 ' Add the Model Search row data to the DOM
 AddField xmlDOM, row, "CLASSGUID", element.elementGUID
 AddField xmlDOM, row, "CLASSTYPE", element.Type
 AddField xmlDOM, row, "Name", element.Name
 AddField xmlDOM, row, "Type", element.Type
 AddField xmlDOM, row, "Stereotype", element.Stereotype
 AddField xmlDOM, row, "Description", element.Notes
 AddField xmlDOM, row, "Path", element.FQName
 
 ' Append the newly created row node to the rows node
 rowsNode.appendChild( row )

end sub

'
' Adds an Element to the DOM called Field which makes up the Row data for the Model Search window.
' <Field name "" value ""/>
'
sub AddField( xmlDOM, row, name, value )

 dim fieldNode
 set fieldNode = xmlDOM.createElement( "Field" )
 
 ' Create first attribute for the name
 dim nameAttribute
 set nameAttribute = xmlDOM.createAttribute( "name" )
 nameAttribute.value = name
 fieldNode.attributes.setNamedItem( nameAttribute )
 
 if len(value) > 0 then
  ' Create second attribute for the value
  dim valueAttribute
  set valueAttribute = xmlDOM.createAttribute( "value" )
  valueAttribute.value = value
  fieldNode.attributes.setNamedItem( valueAttribute )
 end if
 ' Append the fieldNode
 row.appendChild( fieldNode )

end sub

'returns a dictionary of all elements that have a linked document as key and the text of the linked document as value.
function getAllLinkedDocuments(package)
 dim queryString
 queryString = "select o.object_ID from t_document d              " & vbNewLine & _
     " inner join t_object o on d.ElementID = o.ea_guid " & vbNewLine & _
     " where d.ElementType = 'ModelDocument'            "
 if not package is nothing then
  'get package tree id string
  dim packageTreeIDString
  packageTreeIDString = getPackageTreeIDString(package)
  queryString = queryString & vbNewLine & _
      " and o.package_ID in (" & packageTreeIDString & ")"
 end if
 dim elementsWithLinkedDocument
 set elementsWithLinkedDocument = getElementsFromQuery(queryString)
 dim linkedDocumentsDictionary
 set linkedDocumentsDictionary = CreateObject("Scripting.Dictionary")
 dim element as EA.Element
 'loop the elements and add element and its linked document to the dictionary
 for each element in elementsWithLinkedDocument
  dim linkedDocumentText
  linkedDocumentText = getLinkedDocumentContent(element, "TXT")
  linkedDocumentsDictionary.Add element, linkedDocumentText
 next
 set getAllLinkedDocuments = linkedDocumentsDictionary
end function


'--

'get the package id string of the given package tree
function getPackageTreeIDString(package)
   dim allPackageTreeIDs
   set allPackageTreeIDs = CreateObject("System.Collections.ArrayList")
   dim parentPackageIDs
   set parentPackageIDs = CreateObject("System.Collections.ArrayList")
   if not package is nothing then
      parentPackageIDs.Add package.PackageID
   end if
   'get the actual package ids
   getPackageTreeIDsFast allPackageTreeIDs, parentPackageIDs
   'return
   getPackageTreeIDString = Join(allPackageTreeIDs.ToArray,",")
end function


function getPackageTreeIDsFast(allPackageTreeIDs, parentPackageIDs)
   if parentPackageIDs.Count = 0 then
      if allPackageTreeIDs.Count = 0 then
         'make sure there is at least a 0 in the allPackageTreeIDs
         allPackageTreeIDs.Add "0"
      end if
      'then exit
      exit function
   end if
   'add the parent package ids
   allPackageTreeIDs.AddRange(parentPackageIDs)
   'get the child package IDs
   dim sqlGetPackageIDs
   sqlGetPackageIDs = "select p.Package_ID from t_package p where p.Parent_ID in (" & Join(parentPackageIDs.ToArray, ",") & ")"
   dim queryResult
   set queryResult = getVerticalArrayListFromQuery(sqlGetPackageIDs)
   if queryResult.Count > 0 then
      dim childPackageIDs
      set childPackageIDs = queryResult(0)
      'call recursive function with child package id's
      getPackageTreeIDsFast allPackageTreeIDs, childPackageIDs
   end if
end function

function getVerticalArrayListFromQuery(sqlQuery)
   dim xmlResult
   xmlResult = Repository.SQLQuery(sqlQuery)
   set getVerticalArrayListFromQuery = convertQueryResultToVerticalArrayList(xmlResult)
end function

function getSingleValueFromQuery(sqlQuery)
   dim singleValue
   singleValue = ""
   dim result
   set result = getArrayListFromQuery(sqlQuery)
   dim row
   for each row in result
      dim column
      for each column in row
         singleValue = column
         exit for
      next
      exit for
   next
   getSingleValueFromQuery = singleValue
end function


Function convertQueryResultToVerticalArrayList(xmlQueryResult)
    Dim result
   set result = CreateObject("System.Collections.ArrayList")
    Dim xDoc
    Set xDoc = CreateObject( "MSXML2.DOMDocument" )
    'load the resultset in the xml document
    If xDoc.LoadXML(xmlQueryResult) Then       
      'select the rows
      Dim rowList
      Set rowList = xDoc.SelectNodes("//Row")
      Dim rowNode
      Dim fieldNode
      dim firstRow
      firstRow = true
      'loop rows and find fields
      For Each rowNode In rowList
         if firstRow then
            For Each fieldNode In rowNode.ChildNodes
               'add an arraylist for each column
               result.Add CreateObject("System.Collections.ArrayList")
            next
         end if
         'loop the field nodes
         dim i
         i = 0
         For Each fieldNode In rowNode.ChildNodes
            'add the contents to the correct column arraylist
            result(i).Add fieldNode.Text
            i = i + 1
         Next
      Next
   end if
    set convertQueryResultToVerticalArrayList = result
end function


'returns an ArrayList with the elements accordin tot he ObjectID's in the given query
function getElementsFromQuery(sqlQuery)
   dim elements
   set elements = Repository.GetElementSet(sqlQuery,2)
   dim result
   set result = CreateObject("System.Collections.ArrayList")
   dim element
   for each element in elements
      result.Add Element
   next
   set getElementsFromQuery = result
end function

'gets the content of the linked document in the given format (TXT, RTF or EA)
function getLinkedDocumentContent(element, format)
   dim linkedDocumentRTF
   dim linkedDocumentEA
   dim linkedDocumentPlainText
   linkedDocumentRTF = element.GetLinkedDocument()
   if format = "RTF" then
      getLinkedDocumentContent = linkedDocumentRTF
   else
      linkedDocumentEA = Repository.GetFieldFromFormat("RTF",linkedDocumentRTF)
      if format = "EA" then
         getLinkedDocumentContent = linkedDocumentEA
      else
         linkedDocumentPlainText = Repository.GetFormatFromField("TXT",linkedDocumentEA)
         getLinkedDocumentContent = linkedDocumentPlainText
      end if
   end if
end function


'--


OnSearchScript()

shimon

  • EA User
  • **
  • Posts: 167
  • Karma: +5/-0
    • View Profile
Re: Searching Linked Documents
« Reply #10 on: July 28, 2025, 04:30:57 pm »
Hi,
This is the original script of Geert. I just added the Util functions used, so that this script is usable as is, without installing the framework.
Thanks alot to Geert, who has always been helpful and shares so much of his work with all of us.
Sincerely,
Shimon

shimon

  • EA User
  • **
  • Posts: 167
  • Karma: +5/-0
    • View Profile
Re: Searching Linked Documents
« Reply #11 on: August 10, 2025, 05:12:29 pm »
Hi,
I now saw that the quirk of the dependency not being registered was fixed in version Build 1712.

Notes for Enterprise Architect v17.1 (Build 1712)
27th of June 2025
General
Corrected loading of Model Add-ins with dependencies.


So maybe I should at least make one with the proper division of responsibilities, instead of packing all the functions into one Class.
Sincerely,
Shimon