Skip to main content

Data Integration in 5 minutes

Data integration between multiple disparate sources is now as simple as writing SQL code in your application as if all your schema (tables, views and procedures) from different databases are co-located in a single data source.

No, I am not talking horizontal partitioning of your data that have same schema defined in all the databases that some other tools support. You can have segments of data to your liking or the way they were given to you. That not all, you can even add text files to this mix and use them as another source that is participating in data integration. All this is doable with Teiid.

All you have to do is to define the "data source(s)" you would like to integrate in a Teiid configuration file and configure Teiid as your JDBC data source in your application and start writing your application, which may be a simple JDBC or Hibernate application.

Sounds simple huh? Thanks to a new feature called "Connector Supplied Metadata", which is being introduced in the Teiid 6.2 release. This feature is so powerful that using it you can write a single EJB that reads or writes data to/from multiple sources, or you can write Hibernate application with single "Session Factory" that has domain objects from multiple different sources, where you can write Hibernate HQL query that joins two tables from different sources.

So What? Teiid had this capability all along!

Yes, you are right. There is nothing new in terms of additional capabilities of Teiid, it always supported data integration as its core capability. What is new is how it does it, and how flexible it is, and how fast you get from problem to the solution. Prior to Teiid 6.2 release in order to integrate two or more sources in Teiid the following steps must have to be taken.




Step 1: Using Teiid Designer, you need to import and create Source models

Step 2: Using the Teiid Designer, you need to create a VDB and add model files to VDB and configure the "Connectors" for the sources.

Step 3: Deploy into Teiid runtime.

Now, contrast the above scenario to below

as you can see the complete "modeling" step and building a VDB is eliminated (Step 1&2 from above) and metadata that is required for data integration is imported dynamically at runtime and working models are built automatically using "Connector Supplied Metadata" feature. However, user still needs to supply a configuration file where they can specify information about their sources! This configuration file called "Dynamic VDB". And you are required to deploy this configuration file into Teiid runtime engine.

This model does have its pros and cons, and may not suit all the data integration needs.

Pros
  • Metadata about your source imported dynamically at runtime, you have an option to cache after the first import.
  • No static model of your metadata, so it can keep changing as you change the source structure.
  • No Teiid Designer tool to use, cutting down on time and learning curve with Designer.
  • No VDB, i.e no custom artifacts to define your metadata, just define a simple XML file with your favourite editor.
  • Possible to write dynamic discovery of sources and integrate them on the fly.

Cons:
  • You can not define virtual tables, procedures composing of multiple sources.
  • You can not define a single view of your data.
  • Many enterprise features may not be available.
  • Currently only JBDC and Text based sources are supported, but more coming soon.
Teiid still supports both the Designer based VDB and Dynamic VDB modes. If you need a simple and quick data integration use the "Dynamic VDB", otherwise use the Teiid Designer to create a virtual database as before, after all it is all about flexibility.

So, next time you think about combining data from your Excel sheet with your Oracle data, do not custom code the integration layer, use Teiid instead. Give it a try! and let us know what you think. Thank you for reading.

You can check out a quick example here.

Thanks.

Ramesh..

Comments

Popular posts from this blog

Teiid 8.11 Beta1 and 8.10.1 Released

Teiid 8.11 Beta1 is now available from the  downloads  and maven.  Feature highlights since Alpha2 include: TEIID-3434 More caching control over ttls in the result set cache down to the schema/table level. TEIID-3412 MS Access support via the UCanAccess driver. The UCanAccess support is necessary for those running on Java 1.8 as the JDBC ODBC bridge has been removed from the JRE. The waiting continues on EAP 6.4 Alpha1 - it still should be available shortly and should be the platform target for Teiid 8.11 Beta2. Of course, let us know if you find any issues with these early releases.  There's still plenty of time to get fixes into the final release. Teiid 8.10.1 is also available.  It addresses 5 important issues discovered since 8.10 was released: [ TEIID-3409 ] - PostgreSQLExecutionFactory TranslatorProperty annotation in wrong place [ TEIID-3437 ] - Inconsistencies with row count handling [ TEIID-3438 ] - Null value returned from BlobImpl

Tech Tip: Teiid SQL Language MAKEDEP Hint Explained

In this article I will explain what a MAKEDEP hint is, how and when, why it should be used in Teiid. What: MAKEDEP is query hint.  When a query hint is defined in the SQL query it influences the Teiid query planner to optimize the query in a way that is driven by the user. MAKEDEP means "make this as a dependent join". What is a Dependent Join? For example if we have query like: SELECT * FROM X INNER JOIN Y ON X.PK = Y.FK Where the data for X, and Y are coming from two different sources like Oracle and WebService in Teiid, so in relational algebra you can represent above query as Here the result tuples from node X and node Y are being simultaneously fetched by Teiid query engine, then it joins the both the results inside Teiid engine based on the specified X.PK = Y.PK condition and returns the filtered resulted to the user. simple.. Now, what if, if X table has 5 rows and Y table has 100K rows? In order to do the JOIN naively Teiid need sto read all the 5

Teiid Spring Boot 1.7.0 Released

Teiid Spring Boot version 1.7.0 to support Teiid 16.0 has been released. This release is mainly to support the Teiid's latest version.  In this release, the support for OpenAPI code generation based on VDB has been removed as there is no community interest and moreover it was at OpenAPI 2.0, and the industry has moved to 3.0 and beyond. There are no plans to further pursue this feature. VDB maven plugin was also removed, which was intended to be a replacement for the VDB importing feature was to be used when working on OpenShift, however, since it requires the Maven repository and does not completely represent the feature as defined on the WildFly based deployments this is also removed. You can still use the VDB import feature with Teiid Spring Boot, simply define the VDB with your "IMPORT DATABASE" statements and provide the additional files along with the main VDB file. During the start of the application, Teiid Spring Boot will load all the necessary DDL files for the