Monday, June 24, 2013

Cisco enters Data Virtualization Space

In case you haven't heard, networking  giant CISCO systems on Thursday June 20th, 2013 announced that it is buying Composite Software http://www.zdnet.com/cisco-to-acquire-composite-software-for-180m-7000017085/ for $180 Million

This acquisition highlights the growing interest in the Data  Virtualization market by customers that vendors are trying to fill.

The phrase Enterprise Information Integration ( EII ) was first introduced by MetaMatrix (which is predecessor for Teiid http://teiid.org) back in 2000 and was first entrant into the market. Composite Software soon joined ranks of EII vendors and quickly established itself as a fierce competitor. MetaMatrix was acquired by Red Hat in 2007.  Red Hat initiated the process to open source the MetaMatrix software as it does with all acquired code that's not yet open. The Teiid community project was born out of these effort in January 2009.  On the path to open source, we took the opportunity to step back, re-evaluate and revamp Teiid architecture and base it on JBoss AS to best meet the needs of our customers. Though based on the enterprise-proven MetaMatrix technology, Teiid debuted as a relatively unknown software project.  The Teiid project has grown into a vibrant community today, and has established itself as de-facto open source software in this space. Now our users and customers are driving the direction of the software.

In 2011, Red Hat introduced JBoss Enterprise Data Services based Teiid (and ModeShape, another community project to emerge from MetaMatrix) in the Data Virtualization space. This put Red Hat's Enterprise Data Services Platform in the market (again) and back as a competitor with Composite Software.

So, if you are looking for Data Virtualization software, or already using Composite Software and would like to evaluate other options, try Teiid - it is FREE. If you prefer a vendor-supported offering, then contact Red Hat!

Disclaimer: I do work for Red Hat, opinions mentioned here are my own not of my employer.

Ramesh..

Saturday, June 22, 2013

Access Secure Web Service from Teiid

Do need to access a web-service from Teiid, that implements WS-Security? Please check out this blog for an example by David Hauser


Ramesh..

Wednesday, June 19, 2013

Planning to Migrate Your Relational database to MongoDB?

Are you planning on migrating your relational database data in Oracle, MS-SQL, MySQL, DB2, PostgreSQL etc., into MongoDB for scalability and performance reasons?

Or you want to use MongoDB for your new project, but not do not want to spend too much time learning MongoDB APIs? However, you are a seasoned SQL developer?

Or you have data in databases, CSV files and other applications that you need to integrate together and store in MongoDB and do further analysis?

If "Yes", please keep reading...

If you have previously attempted at migration of data yourself, I am sure you may have come across these below links before.

http://docs.mongodb.org/manual/reference/sql-comparison/
http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/
http://docs.mongodb.org/manual/tutorial/model-embedded-one-to-one-relationships-between-documents/
http://docs.mongodb.org/manual/tutorial/model-referenced-one-to-many-relationships-between-documents/
http://docs.mongodb.org/manual/tutorial/model-embedded-one-to-many-relationships-between-documents/

MongoDB lays out various concepts on how one can convert their data into MongoDB and then explains how to access the same using their API. MongoDB is not a relational database, it is a document database. MongoDB stores the data BSON format. MongoDB does not support JOINS like relational databases for joining the related data between multiple documents, however MongoDB document model supports "embedded" documents, where using above techniques one can convert their relational data by carefully mapping 1-to-1, 1-to-many, many-to-many relations into MongoDB based documents. You can effectively view these mapping operations as de-normalization of data in relational terms. De-normalization brings its own rewards and risks, so we will not go that debate here why it is good or bad. For our desire to use MongoDB and access it using SQL, we have to find an acceptable medium that we can work with.

All good so far, however it takes lot of time and effort to do the above mapping of relational schema into MongoDB document structure and learn to use the MongoDB querying APIs. This is where Teiid steps in.

Starting with Teiid 8.4 release, Teiid is introducing MongoDB translator and MongoDB JCA adapter. Essentially implementation of this translator does exactly what is been prescribed in the above links by MongoDB. Out of the box, it gives you a simple way to design MongoDB document schema based on your existing relational schema. You can export DDL from your database and use it with Teiid with some minor modifications to add some extended metadata in aid to design the MongoDB document structures. Once you defined your MongoDB database, you can migrate data from your relational store to this newly created MongoDB store using Teiid by simply issuing statements like

"SELECT * INTO MongoDB.Table FROM Oracle.Table"

At the same time, Teiid gives you full SQL based access to the data residing MongoDB, with out you using any MongoDB specific APIs. If you already have end user applications written for the old database, you can essentially switch it over to use MongoDB without affecting them using Teiid.

Not trying to convert relational database? Your data is in CSV files, or web services, saleaforce? No matter where the data currently resides, using Teiid you can bring that data in and store in the MongoDB. Depending upon where the currently exists, you may have to write a connector and a translator. You must be thinking there must be some gotchs..

Sure, for full disclosure there are

  • giving up on the "schema-less" or dynamic schema model that MongoDB supports
  • Some automatic duplication of data to achieve de-normalization
  • Not all tables can be designed for joins, where the relationships do not exist. Here Teiid query engine steps into picture and provides JOINS on non-key based columns or on un-related tables, but those queries will be executing in the Teiid query engine, not in MongoDB
For more information on MongoDB translator read about it here. There few more enhancements we have planned, but your input in driving this is most appreciated. So, take it for test drive, come to Teiid forums and discuss about any questions and comments or enhancement requests.

In the next blog, I will walk through an example in taking a sample database and show you how to migrate step by step. Till then...

Thanks

Ramesh..

Wednesday, June 12, 2013

Teiid 8.4.Final Released, MongoDB support added

In keeping with our 3 month release window we are proud to announce our next time boxed release - Teiid 8.4.Final.

Here are top three (3) features:
  1. The deployment platform is upgraded to EAP 6.1 Alpha. You can also use EAP 6.1 Final, but you need a developer subscription from Redhat. For the difference between community and subscription version see http://www.redhat.com/products/jbossenterprisemiddleware/community-enterprise/. If you have concerns about switching from JBoss AS 7.1.1 to EAP please read http://www.jboss.org/jbossas/faq
  2. Pluggable row and column based security. Read more about it:  https://docs.jboss.org/author/display/teiid84final/Data+Roles
  3. Support for MongoDB. Use Teiid and its SQL language to access the data from MongoDB. This release includes a new MongoDB translator and resource adapter. Read more about it: https://docs.jboss.org/author/display/teiid84final/MongoDB+Translator.  More blogs coming on this soon.
Here is complete run down of features in this release:
  • TEIID-2444 EAP Platform - the deployment platform for Teiid has been changed to EAP 6.1.Alpha1
  • TEIID-2429 Improved sort performance - improved sorts for large data sets especially under heavy load.
  • TEIID-2449 VDB Zips - added the ability to deploy non-Designer based vdb zip artifacts with the ability to place DDL outside of the VDB.xml via the DDL-FILE metadata repository.
  • TEIID-2423 TEIID-2470 Additional Embedded deployments - xml and zip vdb deployment options are available for Teiid Embedded.
  • TEIID-1092 Session Variables - added the teiid_session_set and teiid_session_get methods to maintain session variables.
  • TEIID-2326 Shared Mat Views - materialized views from imported vdbs will be reused rather than recreated in each importing vdb.
  • TEIID-2344 Automatic DB Version Detection - the JDBC translator database version property in most circumstances will not need to be set manually - see the compatibility note below.
  • TEIID-2453 STRING_AGG Added the string_agg aggregate function for concatenating binary and character strings.
  • TEIID-2504 Improved socket results processing large results processing over a socket transport have been significantly improved for forward only results. Scrollable results will have only a marginal improvement.
  • TEIID-2427 TEIID-2311 Column Masking added column masking and refined row-based security. Also allow for permissions to be granted via the MetadataFactory methods addPermission and addColumnPermisison to enable adding roles via a MetadataRepository.
  • TEIID-2507 Session scoped mat views - mat views can now be scoped to sessions via the cache hint, rather than to just the global scope.
  • TEIID-2329 MongoDB Support - added a mongodb translator/connector with a wide feature set including pushdown support for lobs, composite keys, and embedded joins. See the Reference for more.
We would like to thank our growing community for their participation in forums, JIRA and mailing list. Keep them coming. If you have a write up or a blog, let us know and we will link from the Teiid website.

If you have feature request let us know, we are actively developing Teiid 8.5 now, so speak up!

Thank you.

Ramesh & Steve.

Tuesday, June 11, 2013

Expose Excel Data as OData feed using Teiid

For that matter you can expose any data source that is supported by Teiid through its translators as OData service. You can convert
  • Relational databases
  • XML, JSON, CSV files
  • Web Services 
  • Sales Force
  • Excel documents
sources, or write your on top of your specific source.

Teiid added support for exposing its virtual database (VDB) as OData service without any additional work. Create a virtual database with your required sources and create any additional views and deploy in the the JBoss EAP 6.1 server, that has Teiid 8.3 or greater installed.  See https://docs.jboss.org/author/display/teiid84final/OData+Support for more information.

So, lets build a simple example of exposing the Excel Sheet. Teiid accesses Excel sheets via JDBC-ODBC bridge, and Windows OS provides a ODBC driver for Excel documents. 

Edit the standalone-teiid.xml, and create Connection Factory
 

    
        jdbc:odbc:Driver={Microsoft Excel Driver (*.xls)};Dbq=c:\ODBC\ExcelData.xls
        odbc
        TRANSACTION_NONE
        
            false
            false
            FailingConnectionOnly
        
        
            {user}
            {password}
        
         
    
        
         sun.jdbc.odbc.JdbcOdbcDriver
     
     



Now create VDB, using the excel as the source
 


    
    
        
    



save the above as "odbc-vdb.xml" and deploy into Teiid Server. See https://docs.jboss.org/author/display/TEIID/Deploying+VDBs for deployment instructions.  Make sure your VDB deployed in "ACTIVE" state.

Now, you can access your Excel data using OData API. Either using the your Web Browser or favorite OData client, go to 

 
     http://:/odata/odbc/excel.Sheet1
For more complete information about OData API see https://docs.jboss.org/author/display/TEIID/OData+Support

Thanks
Ramesh..