Skip to main content

A Relational Data Integration Engine

Teiid has been an open source project for few months now and the response from community has been fairly positive. New users are taking it for test drive and we have busily been executing on our roadmap. Still I sometimes feel, we may have not explained what Teiid software really does?

Since I am java six pack, I am going to dumb it down to level that I understand, and present it to you.

If you look around Teiid site depending, you will find a multitude of terms and definitions, here are some examples

  • It is a EII (Enterprise Information Integration) tool

  • It is a Virtual Database

  • It is Data Integration Engine

  • Data Mash up Engine

  • Data Virtualization Engine

All the above are accurate. But, I want to describe Teiid's base processor more precisely.

"Teiid is a Relational Data Integration Engine"

Using Teiid and its supplied tools, you build a brand new logical data model for your data in multiple heterogeneous data sources using SQL as glue language. Now, the schema from this logical model is what you can expose to your end applications as domain model.

So, in effect you took multiple existing data sources, and defined relationships among the data you have, to create a data model you need without much effort, i.e. you got model driven data integration.

Once this logical schema is deployed into a Teiid runtime engine, Teiid behaves like a relational database. It provides a secure, preferment and transactionally aware system that can be accessed by your end user applications. Note, that the physical location of your data still be in the same place as before, however the logical model results are built at runtime for you by Teiid.

Well, all is good so far but I said, "Teiid is relational data integration engine" does this mean Teiid can only integrate data from multiple relational databases? absolutely not! Teiid can integrate data from any RDBMS like Oracle, SQL Server, DB2 etc. as well from Text files, Web Services, Excel sheets, XML, Main Frame or any of your custom data. If you are wondering how that is possible? The answer lies in Teiid Connectors.

Teiid Connector: A connector is data access mechanism to a "source" system like DBMS, file or a web service. Also, another very important function of the Connector is that it maps the underlying source's data model into relational entities (like tables and procedures etc). So, naturally when you are dealing with RDBMS sources they represent the sources underlaying data model. However, if a connector represents a CSV file, it exposes the metadata information about csv data in "relational" terms to the out bound systems, thus a CSV file looks and behaves like a RDBMS source to a calling system. Using this technique can can virtually transform any source and represent it in relataional terms. Thus you can even call Teiid as "Relational Data Abstraction" engine. So, if you have non standard source of data you can map that into relational terms and use it as RDBMS. Now, as you can imagine, magically all the sources under Teiid system look and behave like relational sources. Now on to integration of these sources.


Query Engine: Now let's assume, a Teiid system is setup with two separate connectors. When a end user application submits a SQL based query, the Query Engine uses the metadata/schema of the logical model to parse, validate, optimize and split the query into multiple source specific queries. The query engine will parallelize the source access if possible and process the data into the final result. The query engine has numerous optimizations. The most important one is that it pushes the maximum amount of work to the source connector, thus avoiding any expensive in-memory operations.

End user applications can access the Teiid mainly using JDBC API. That means you could use Hibernate easily and never see any difference. With some additional tooling you can even access your data as Web Service. Teiid project provides tools for every aspect of the development and deployment of model driven data integration.


Before I end this article, there is one very important concept I want leave with and that is Teiid is not a ETL or Data Mart tool, where the data is duplicated from the original source then served to target systems. Teiid works with data from their original locations as it is, there is no unnecessary duplication/coping of data.


Now that hopefully you know "What Teiid really is", you can see how it can solve your company's data integration needs easily. Let us know how we can help.


Ramesh..

Comments

  1. Would be interested to hear how well Teiid supports REST.

    For example, could I use Teiid to extract data from an RDBMS and make it available via REST that would be presented to a client in a mashup that merges data from several sources?

    ReplyDelete
  2. Teiid can consume a REST service through the XML connector - either as a document or modeled as tables. We'll be documenting our preferred approach to exposing integrated views as REST under https://jira.jboss.org/jira/browse/TEIID-677.

    ReplyDelete
  3. Hi
    I am impressed with this. I would like to try out this. Where can i find the documentation around this? One question i have now is, We use Simple .Net application to get the data from Oracle and SQL Server and we join this data using ADO.Net objects. So the code is tightly coupled. Now we want to this with data federation tools like Teiid. So do you have any reference document around this?
    Thanks -- Vijaya Kadiyala
    www.DotNetVJ.com

    ReplyDelete
  4. Hi Vijaya,

    You should have a look at the main site http://www.jboss.org/teiid for more documentation. There isn't a specific example covering Oracle and SQL Server, but they are both relational sources and easy to integrate with Designer http://www.jboss.org/teiiddesigner. If you need any help with your efforts, just post to the users forum.

    ReplyDelete
  5. Hi For some reason your site http://www.jboss.org/teiid is not working. I am getting all those Dev Tag issues. So can't download any documentation on this.

    Thanks - Vijaya Kadiyala

    ReplyDelete
  6. Hi Vijaya,

    Sorry for the inconvenience, Teiid site seem to have issues in supporting IE6 browser, if you are using IE6 can you use later versions of IE like 7 or 8? You can also use Firefox otherwise.

    Also, you can use the Teiid project mailing list at http://www.jboss.org/index.html?module=bb&op=viewforum&f=298 for questions you may have.

    Thank you.

    ReplyDelete

Post a Comment

Popular posts from this blog

Teiid 8.11 Beta1 and 8.10.1 Released

Teiid 8.11 Beta1 is now available from the  downloads  and maven.  Feature highlights since Alpha2 include: TEIID-3434 More caching control over ttls in the result set cache down to the schema/table level. TEIID-3412 MS Access support via the UCanAccess driver. The UCanAccess support is necessary for those running on Java 1.8 as the JDBC ODBC bridge has been removed from the JRE. The waiting continues on EAP 6.4 Alpha1 - it still should be available shortly and should be the platform target for Teiid 8.11 Beta2. Of course, let us know if you find any issues with these early releases.  There's still plenty of time to get fixes into the final release. Teiid 8.10.1 is also available.  It addresses 5 important issues discovered since 8.10 was released: [ TEIID-3409 ] - PostgreSQLExecutionFactory TranslatorProperty annotation in wrong place [ TEIID-3437 ] - Inconsistencies with row count handling [ TEIID-3438 ] - Null value returned from BlobImpl

Tech Tip: Teiid SQL Language MAKEDEP Hint Explained

In this article I will explain what a MAKEDEP hint is, how and when, why it should be used in Teiid. What: MAKEDEP is query hint.  When a query hint is defined in the SQL query it influences the Teiid query planner to optimize the query in a way that is driven by the user. MAKEDEP means "make this as a dependent join". What is a Dependent Join? For example if we have query like: SELECT * FROM X INNER JOIN Y ON X.PK = Y.FK Where the data for X, and Y are coming from two different sources like Oracle and WebService in Teiid, so in relational algebra you can represent above query as Here the result tuples from node X and node Y are being simultaneously fetched by Teiid query engine, then it joins the both the results inside Teiid engine based on the specified X.PK = Y.PK condition and returns the filtered resulted to the user. simple.. Now, what if, if X table has 5 rows and Y table has 100K rows? In order to do the JOIN naively Teiid need sto read all the 5

Teiid Spring Boot 1.7.0 Released

Teiid Spring Boot version 1.7.0 to support Teiid 16.0 has been released. This release is mainly to support the Teiid's latest version.  In this release, the support for OpenAPI code generation based on VDB has been removed as there is no community interest and moreover it was at OpenAPI 2.0, and the industry has moved to 3.0 and beyond. There are no plans to further pursue this feature. VDB maven plugin was also removed, which was intended to be a replacement for the VDB importing feature was to be used when working on OpenShift, however, since it requires the Maven repository and does not completely represent the feature as defined on the WildFly based deployments this is also removed. You can still use the VDB import feature with Teiid Spring Boot, simply define the VDB with your "IMPORT DATABASE" statements and provide the additional files along with the main VDB file. During the start of the application, Teiid Spring Boot will load all the necessary DDL files for the