Author Topic: connection via ODBC to Databricks deltalake  (Read 9809 times)

soad1981

  • EA Novice
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
connection via ODBC to Databricks deltalake
« on: March 26, 2024, 08:37:51 pm »
Hi,
our source database is Databricks datalake that supports ODBC connection and it works fine (we can connect from outside of databricks). Now we have EA v 15/16 and we would like to connect to databricks also from EA and to read data some metadata from Databricks tables. Is it supported by scripting in EA? Can you provide a simple snippet that opens connection via ODBC and reads content of specific table?
Thanks
« Last Edit: March 26, 2024, 11:09:42 pm by soad1981 »

ea0522

  • EA User
  • **
  • Posts: 134
  • Karma: +5/-0
    • View Profile
Re: connection via ODBC to Databricks deltalake
« Reply #1 on: April 30, 2024, 08:07:07 pm »
Hai,
Is there an answer to this question?
We would also like to connect to Databricks from our EA instance running in the cloud.

We are particularly interested in modelling the Databricks warehouse and define relations to data objects as used by other applications.
That way we think we can use the modelling capabilities of EA together with the DWH capabilities of Databricks.
The Database Builder seems to be a fine option to use for this.

Another way to connect to DataBricks is using the Delta Sharing option supported by Databricks. Is this option also supported by EA?

Thanks.

Eve

  • EA Administrator
  • EA Guru
  • *****
  • Posts: 8061
  • Karma: +118/-20
    • View Profile
Re: connection via ODBC to Databricks deltalake
« Reply #2 on: May 01, 2024, 08:18:01 am »
Enterprise Architect doesn't know how Databricks stores its metadata, so will not be able to import it into your model.

MattAtRamsay

  • EA Novice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: connection via ODBC to Databricks deltalake
« Reply #3 on: January 30, 2025, 01:08:24 am »
Hello

I also came to the forum to seek an answer to the question "can I reverse engineer Databricks data schema into EA?". That was the question raised here, right? Except the questions seemed to suggest the data itself rather than the schema. Just for clarity, not for pickiness.

Anyhow, if no-one has yet cracked this, but still would like to, I'd like to get the conversation on the move. Might need to think outside the box.

Cheers, Matt

Eve

  • EA Administrator
  • EA Guru
  • *****
  • Posts: 8061
  • Karma: +118/-20
    • View Profile
Re: connection via ODBC to Databricks deltalake
« Reply #4 on: January 30, 2025, 07:46:39 am »
When I talk about metadata for data definition language I am talking about the schema, although it could include additional information beyond the schema.

Given EA does not have the capability to import that for you, you would need to query Databricks yourself to get the information and then use our API to add elements.

MattAtRamsay

  • EA Novice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: connection via ODBC to Databricks deltalake
« Reply #5 on: January 30, 2025, 08:08:28 pm »
Thank you Eve

Yes, although I was wondering if there might be an XSLT that could transform Databricks DDL to SQL DDL and then leverage the existing import routines? Early day thinking to kick ideas around.

All the best, Matt