Tuesday, December 17, 2013

Teiid 8.6.Final Released

In time for holidays, Teiid team is very happy to announce the final version Teiid 8.6 is out and available for download. 8.6 has been mainly a bug fix release - with over 80 issues addressed.

Main features in this release include:
  • TEIID-2424 OSGI Bundles - Teiid jars are now built also as OSGI bundles. A starter Karaf osgi container features.xml file is available in maven repository with the build
  • TEIID-1311 Cassandra Connectivity - is now offered through a community contribution by Radek Koubský of a resource adapter and translator pair. See the reference and admin guides for more.
  • TEIID-2309 Conformed tables - a feature to improve planning when identical tables exist in multiple sources.
  • TEIID-2685 SAP Netweaver - enhanced the OData translator for use with Netweaver.
  • Direct Query Procedure Support
    • TEIID-2740 - MongoDB
    • TEIID-2734 - Cassandra
As always, lots of great community feedback went into this release from the forums and with issues from Mark Addleman and team, Ivan Chan who has been instrumental in testing sources like MongoDB/Cassendra, Andy Yip, and many more community users. We sincerely thank all your community participation.

We have already started work on Teiid 8.7, which is going to be a feature packed version with support for Amazon SimpleDB, Apache SOLR, Apache Accumulo and more. If you have experience in these sources any other [Big] Data sources, please help us in any form to develop these translators. We always looking for suggestions.

If have been using Teiid and would like fully supported version, Redhat is releasing Data Virtualization V6 (DV6) very soon. If you are Redhat customer, grab a beta copy of this software from Redhat Customer Portal. BTW, Redhat also provides services/training on DV6, so take advantage of it, call your Redhat Sales rep.

Happy Holidays 2013.

Thanks and Happy Teiiding,
Steve and Ramesh

Friday, December 6, 2013

Teiid Designer 8.3 Final Released

Teiid Designe 8.3 Final is now available. This release reflects a lot of work bug fixing and maturing Teiid Designer's integration with JBoss Tools AS server framework as well as continuing to add functionality that supports more of Teiid translator capabilities and VDB features.

In particular...
  • New LDAP importer that allows creating a connection profile to connect to your LDAP service and import a relational representation of your data.
  • Model extensions to allow setting relational properties specific to MongoDB entities.
  • Added ability to set custom properties for importing relational metadata through your deployed data sources.
  • Added option for Flat File importer to specify a remote URL in addition to importing local flat files.
Install it now and exercise it's new functionality.
Teiid Designer Project Team

Thursday, December 5, 2013

Teiid 8.6 CR1 Posted

8.6 CR1 to has been posted to the downloads.

Remote JDG 6 support has been added via a new resource adapter.  There have been 75 issues addressed in total. 

Any feedback on your issues or general testing is appreciated and if done soon would still be in time for the final release.

8.7 development has also started and we are excited to pick up another community contribution.  Radim Hopp has contributed an Amazon SimpleDB translator/resource adapter that furthers our integration abilities - more on this in the 8.7 posts.

Thanks,
Steve

Thursday, November 21, 2013

Teiid 8.6 Beta2 Available

8.6 Beta2 to has been posted to the downloads.  Feature highlights include:
  • TEIID-2309 Added initial conformed table feature to improve planning when identical tables exist in multiple sources.
Mongo support has been greatly improved thanks to Ramesh and community user Ivan Chan.  There have been over 60 other issues addressed in total.

Any early feedback you can provide should be in time for the final release, so just let us know if you encounter anything.

Any new feature work beyond what has already been started will not get picked up until 8.7.  Looking forward during the 8.7 cycle there will be more thought and requirements gathering toward 9.0 so that work can begin in parallel with 8.8.

Thanks,
Steve

Friday, October 18, 2013

Teiid 8.6 Alpha2 Posted

We're well underway with 8.6 development and have posted 8.6 Alpha2 to the downloads.  Feature highlights include:
  • TEIID-2424 Teiid jars are now built also as OSGI bundles. A starter Karaf osgi container features.xml file is available in maven repository with the build
  • TEIID-1311 Cassandra connectivity is now offered through a community contribution by Radek Koubský of a resource adapter and translator pair. See the reference and admin guides for more.
In addition to also adding support for a generic VALUES statement, there have been over 30 other issues addressed. We will likely continue the theme of being very targeted about feature work - while addressing as many other issues as possible.

There's still plenty of time to get fixes and minor enhancements into the release so feel free to engage the community forums, issue tracker, etc.

Thanks,
Steve

Thursday, October 17, 2013

Teiid Designer 8.3 Beta2 available

Our latest milestone, 8.3 Beta2, is available for downloading and testing. 

This milestone incorporates more fixes and includes a couple of new features:
  • Added support for MongoDB-specific metadata through new extension properties definitions
  • LDAP  to relational model importer.

Check it out on our What's New in 8.3 page.

Teiid Designer Team

Wednesday, October 9, 2013

JBoss Data Virtualization 6 Beta

A quick downstream plug - the next major productization of Teiid by Red Hat has reached beta in JBoss Data Virtualization Platform 6

For those new to Teiid Designer or Data Virtualization in general check out the video links.

Steve

Monday, October 7, 2013

Bring HTML Pages Into Relational World Using Web Scraping Teiid Translator

Huge thanks to our community member Andriy Rokhmanov for writing and sharing a Teiid translator with whole community.


Are you working with Teiid? Share your work in blog, code contribution, forum help etc and get recognized as Teiid ninja.

Thanks Andriy.

Ramesh..


Thursday, September 26, 2013

Teiid Designer 8.3 Beta 1 milestone available

Teiid Designer 8.3, targeted for release late next month, is focused on bug fixing and improving the integration of the JBoss server and Teiid runtime frameworks.

We've released our first milestone, 8.3 Beta1 and it's available from our downloads page via zip archive or update site.

Highlights include...

Addressed problems re-deploying non-JDBC data sources
Improved Teiid Configuration tree behavior
  • TEIIDDES-1730 - Add Create Teiid VDB Data Source action to VDB selection
  • TEIIDDES-1835 - Add preview artifact management actions
  • TEIIDDES-1847 - Filter preview artifacts in tree if not tied to current runtime user     
  • TEIIDDES-1814 - Remove Teiid-specific server creation dialog
Teiid Designer Team

Monday, September 16, 2013

Teiid 8.5 Released

We are pleased to announce the next iteration of Teiid is available - Teiid 8.5 Final.

Feature highlights include:
  • TEIID-2528 Encrypted Buffer Files - the buffer manager now supports an option to encrypt Teiid temp data files using 128-bit AES.
  • TEIID-2249 Dependent Join Enhancements -
    • TEIID-2559 greater control over full depdendent join pushdown has been added including make dep hint options max/join to control the extent of depdendent join pushdown.
    • TEIID-2555 the join hint enables the pushdown the entire dependent join when supported by the source.
    • TEIID-2558 JDBC translator support to dependent join pushdown for a broad set of sources including Oracle, PostgreSQL, MySQL, SQLServer, Sybase, etc.
    • use the enableDependentJoins translator property.
    See the Reference for more.
  • TEIID-2527 Configurable Runtime Multisource - the server will now allow multisource models to have sources added and removed at runtime. See the admin addSource and removeSource methods.
  • TEIID-2571 Broader Must Pushdown - must pushdown functions tied to a particular model that supports selects without from can now be called without at any time.
  • TEIID-2572 Function native-query - source functions for most translators now support the native-query extension metadata property to specify alternative syntax.
  • TEIID-2067 Global temporary tables - may now be declared that are backed by Teiid session specific instance tables. See the Reference for more.
  • TEIID-2443 Simple aggregate rollup - added support for a single ROLLUP grouping element to produce multiple levels of aggregation. See the Reference for more.
  • TEIID-2614 Anonymous procedure blocks - anonymous procedure blocks may be used as statements. See the Reference for more.
  • TEIID-247 Expanded array support - adding parsing and JDBC and other support for arrays. See the Reference for more. Note that this support should be considered a tech preview and will be expanded/refined over subsequent releases.
  • TEIID-2584 Materialization management - added a tech preview of management routines for external materialized views including scheduled refresh via extension metadata properties.
In total 108 issues were resolved - including several important Teiid procedure and OData production issues.

Lots of great community feedback went into this release from the forums and with issues from Mark Addleman and team, Red Hatter Graeme Gillies, long time user Luca Gioppo, and others including several new faces.

Housekeeping as of late includes an on going overhaul of the examples/quickstarts and we've created new JIRA projects to track:
This will help keep the Teiid JIRAs as focused as possible - so don't hesitate to log issues as you find them with Teiid or any related component.

Also if you haven't seen it yet, we've pulled together a single Security Guide from documentation that was spread among the other guides.

Thanks and Happy Teiiding,
Steve and Ramesh

Thursday, August 29, 2013

Teiid 8.5 CR1

Teiid 8.5 CR1 for EAP 6.1.Alpha is now available in our downloads.  As we are closing in on the release there is only one new feature to highlight since Beta2:
  • TEIID-2584 Materialization management - added management routines for external materialized views including scheduled refresh via extension metadata properties.
In total now there have been nearly 100 issues addressed for the release.  Anything that you can help spot before the final release will be greatly appriciated. 

We'll be starting work shortly on the 8.6 release - again targeting a three month release window.  Please vote for and log issues that you believe should be included in the next release.

Thanks,
Steve

Monday, August 19, 2013

Teiid Designer 8.2 Final available

Release 8.2.Final is now available!

This release includes a number of bug fixes as well as enhancements to our integration with JBoss AS server framework.  8.2 now supports multiple versions of the Teiid runtime including 7.7.x, 8.3.x and 8.4.x.

Check out our What's New page for more details and some screen shots.

You can install via an Eclipse-based zip archive or install it via our release update site.

Teiid Designer Project Team


Wednesday, August 14, 2013

Teiid 8.5 Beta2

Teiid 8.5 Beta2 for EAP 6.1.Alpha is now available in our downloads.  Feature highlights since Beta1 include:
  • TEIID-2614 Anonymous procedure blocks - anonymous procedure blocks may be used as statements. See the Reference for more.  
  • TEIID-247 Expanded array support - added parsing, JDBC and other support for arrays. See the Reference for more. Note that this support should be considered a tech preview and will be expanded/refined over subsequent releases.
Given that array support is an evolving feature if you have any interest in its please provide as much early feedback as possible.

Also issues are starting to be moved into the 8.6 release target.  If you have anything pressing, please log/vote for it so that it can be prioritized appropriately.

Steve

Thursday, August 1, 2013

Teiid 8.5 Beta1 Posted

Teiid 8.5 Beta1 for EAP 6.1.Alpha is now available in our downloads.  Feature highlights since Alpha1 include:
  • TEIID-2571 Broader Must Pushdown - must pushdown functions tied to a particular model that supports selects without from can now be called without at any time.
  • TEIID-2572 Function native-query - source functions for most translators now support the native-query extension metadata property to specify alternative syntax.
  • TEIID-2067 Global temporary tables - may now be declared that are backed by Teiid session specific instance tables. See the Reference for more.
  • TEIID-2443 Simple aggregate rollup - added support for a single ROLLUP grouping element to produce multiple levels of aggregation. See the Reference for more. 
We're still on track for the September release so any remaining feature work will be narrow in scope - and any early feedback will help ensure a quality release.  On that note thanks to community member Marco Ardito ODBC access has also been improved with a number of clients and pg driver versions.
 
Steve

Unfortunately there is regression with the Salesforce access in this Beta that has already been addressed on master.  If you are held up by this prior to the next Beta release just let us know and we'll provide a patch.



Tuesday, July 2, 2013

Teiid 8.5 Alpha1 Available

Teiid 8.5 Alpha1 for EAP 6.1.Alpha (which is basically the same as AS 7.2) is now available in our downloads.  The feature highlights are:
  • TEIID-2528 Encrypted Buffer Files - the buffer manager now supports an option to encrypt Teiid temp data files using 128-bit AES.
  • TEIID-2249 TEIID-2559 TEIID-2555 Dependent Join Enhancements - greater control over full depdendent join pushdown has been added including make dep hint options max and min to control the extent of depdendent join pushdown. The join hint is also available to pushdown the entire dependent join when supported by the source.
  • TEIID-2527 Configurable Runtime Multisource - the server will now allow multisource models to have sources added and removed at runtime. See the admin addSource and removeSource methods.
It is still early in the release cycle, so there will be more refinements, documentation, and enhancements for dependent joins coming.   

And of course there is still plenty of time to log enhancement requests for and issues against 8.5.

Thanks,
Steve

Monday, June 24, 2013

Cisco enters Data Virtualization Space

In case you haven't heard, networking  giant CISCO systems on Thursday June 20th, 2013 announced that it is buying Composite Software http://www.zdnet.com/cisco-to-acquire-composite-software-for-180m-7000017085/ for $180 Million

This acquisition highlights the growing interest in the Data  Virtualization market by customers that vendors are trying to fill.

The phrase Enterprise Information Integration ( EII ) was first introduced by MetaMatrix (which is predecessor for Teiid http://teiid.org) back in 2000 and was first entrant into the market. Composite Software soon joined ranks of EII vendors and quickly established itself as a fierce competitor. MetaMatrix was acquired by Red Hat in 2007.  Red Hat initiated the process to open source the MetaMatrix software as it does with all acquired code that's not yet open. The Teiid community project was born out of these effort in January 2009.  On the path to open source, we took the opportunity to step back, re-evaluate and revamp Teiid architecture and base it on JBoss AS to best meet the needs of our customers. Though based on the enterprise-proven MetaMatrix technology, Teiid debuted as a relatively unknown software project.  The Teiid project has grown into a vibrant community today, and has established itself as de-facto open source software in this space. Now our users and customers are driving the direction of the software.

In 2011, Red Hat introduced JBoss Enterprise Data Services based Teiid (and ModeShape, another community project to emerge from MetaMatrix) in the Data Virtualization space. This put Red Hat's Enterprise Data Services Platform in the market (again) and back as a competitor with Composite Software.

So, if you are looking for Data Virtualization software, or already using Composite Software and would like to evaluate other options, try Teiid - it is FREE. If you prefer a vendor-supported offering, then contact Red Hat!

Disclaimer: I do work for Red Hat, opinions mentioned here are my own not of my employer.

Ramesh..

Saturday, June 22, 2013

Access Secure Web Service from Teiid

Do need to access a web-service from Teiid, that implements WS-Security? Please check out this blog for an example by David Hauser


Ramesh..

Wednesday, June 19, 2013

Planning to Migrate Your Relational database to MongoDB?

Are you planning on migrating your relational database data in Oracle, MS-SQL, MySQL, DB2, PostgreSQL etc., into MongoDB for scalability and performance reasons?

Or you want to use MongoDB for your new project, but not do not want to spend too much time learning MongoDB APIs? However, you are a seasoned SQL developer?

Or you have data in databases, CSV files and other applications that you need to integrate together and store in MongoDB and do further analysis?

If "Yes", please keep reading...

If you have previously attempted at migration of data yourself, I am sure you may have come across these below links before.

http://docs.mongodb.org/manual/reference/sql-comparison/
http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/
http://docs.mongodb.org/manual/tutorial/model-embedded-one-to-one-relationships-between-documents/
http://docs.mongodb.org/manual/tutorial/model-referenced-one-to-many-relationships-between-documents/
http://docs.mongodb.org/manual/tutorial/model-embedded-one-to-many-relationships-between-documents/

MongoDB lays out various concepts on how one can convert their data into MongoDB and then explains how to access the same using their API. MongoDB is not a relational database, it is a document database. MongoDB stores the data BSON format. MongoDB does not support JOINS like relational databases for joining the related data between multiple documents, however MongoDB document model supports "embedded" documents, where using above techniques one can convert their relational data by carefully mapping 1-to-1, 1-to-many, many-to-many relations into MongoDB based documents. You can effectively view these mapping operations as de-normalization of data in relational terms. De-normalization brings its own rewards and risks, so we will not go that debate here why it is good or bad. For our desire to use MongoDB and access it using SQL, we have to find an acceptable medium that we can work with.

All good so far, however it takes lot of time and effort to do the above mapping of relational schema into MongoDB document structure and learn to use the MongoDB querying APIs. This is where Teiid steps in.

Starting with Teiid 8.4 release, Teiid is introducing MongoDB translator and MongoDB JCA adapter. Essentially implementation of this translator does exactly what is been prescribed in the above links by MongoDB. Out of the box, it gives you a simple way to design MongoDB document schema based on your existing relational schema. You can export DDL from your database and use it with Teiid with some minor modifications to add some extended metadata in aid to design the MongoDB document structures. Once you defined your MongoDB database, you can migrate data from your relational store to this newly created MongoDB store using Teiid by simply issuing statements like

"SELECT * INTO MongoDB.Table FROM Oracle.Table"

At the same time, Teiid gives you full SQL based access to the data residing MongoDB, with out you using any MongoDB specific APIs. If you already have end user applications written for the old database, you can essentially switch it over to use MongoDB without affecting them using Teiid.

Not trying to convert relational database? Your data is in CSV files, or web services, saleaforce? No matter where the data currently resides, using Teiid you can bring that data in and store in the MongoDB. Depending upon where the currently exists, you may have to write a connector and a translator. You must be thinking there must be some gotchs..

Sure, for full disclosure there are

  • giving up on the "schema-less" or dynamic schema model that MongoDB supports
  • Some automatic duplication of data to achieve de-normalization
  • Not all tables can be designed for joins, where the relationships do not exist. Here Teiid query engine steps into picture and provides JOINS on non-key based columns or on un-related tables, but those queries will be executing in the Teiid query engine, not in MongoDB
For more information on MongoDB translator read about it here. There few more enhancements we have planned, but your input in driving this is most appreciated. So, take it for test drive, come to Teiid forums and discuss about any questions and comments or enhancement requests.

In the next blog, I will walk through an example in taking a sample database and show you how to migrate step by step. Till then...

Thanks

Ramesh..

Wednesday, June 12, 2013

Teiid 8.4.Final Released, MongoDB support added

In keeping with our 3 month release window we are proud to announce our next time boxed release - Teiid 8.4.Final.

Here are top three (3) features:
  1. The deployment platform is upgraded to EAP 6.1 Alpha. You can also use EAP 6.1 Final, but you need a developer subscription from Redhat. For the difference between community and subscription version see http://www.redhat.com/products/jbossenterprisemiddleware/community-enterprise/. If you have concerns about switching from JBoss AS 7.1.1 to EAP please read http://www.jboss.org/jbossas/faq
  2. Pluggable row and column based security. Read more about it:  https://docs.jboss.org/author/display/teiid84final/Data+Roles
  3. Support for MongoDB. Use Teiid and its SQL language to access the data from MongoDB. This release includes a new MongoDB translator and resource adapter. Read more about it: https://docs.jboss.org/author/display/teiid84final/MongoDB+Translator.  More blogs coming on this soon.
Here is complete run down of features in this release:
  • TEIID-2444 EAP Platform - the deployment platform for Teiid has been changed to EAP 6.1.Alpha1
  • TEIID-2429 Improved sort performance - improved sorts for large data sets especially under heavy load.
  • TEIID-2449 VDB Zips - added the ability to deploy non-Designer based vdb zip artifacts with the ability to place DDL outside of the VDB.xml via the DDL-FILE metadata repository.
  • TEIID-2423 TEIID-2470 Additional Embedded deployments - xml and zip vdb deployment options are available for Teiid Embedded.
  • TEIID-1092 Session Variables - added the teiid_session_set and teiid_session_get methods to maintain session variables.
  • TEIID-2326 Shared Mat Views - materialized views from imported vdbs will be reused rather than recreated in each importing vdb.
  • TEIID-2344 Automatic DB Version Detection - the JDBC translator database version property in most circumstances will not need to be set manually - see the compatibility note below.
  • TEIID-2453 STRING_AGG Added the string_agg aggregate function for concatenating binary and character strings.
  • TEIID-2504 Improved socket results processing large results processing over a socket transport have been significantly improved for forward only results. Scrollable results will have only a marginal improvement.
  • TEIID-2427 TEIID-2311 Column Masking added column masking and refined row-based security. Also allow for permissions to be granted via the MetadataFactory methods addPermission and addColumnPermisison to enable adding roles via a MetadataRepository.
  • TEIID-2507 Session scoped mat views - mat views can now be scoped to sessions via the cache hint, rather than to just the global scope.
  • TEIID-2329 MongoDB Support - added a mongodb translator/connector with a wide feature set including pushdown support for lobs, composite keys, and embedded joins. See the Reference for more.
We would like to thank our growing community for their participation in forums, JIRA and mailing list. Keep them coming. If you have a write up or a blog, let us know and we will link from the Teiid website.

If you have feature request let us know, we are actively developing Teiid 8.5 now, so speak up!

Thank you.

Ramesh & Steve.

Tuesday, June 11, 2013

Expose Excel Data as OData feed using Teiid

For that matter you can expose any data source that is supported by Teiid through its translators as OData service. You can convert
  • Relational databases
  • XML, JSON, CSV files
  • Web Services 
  • Sales Force
  • Excel documents
sources, or write your on top of your specific source.

Teiid added support for exposing its virtual database (VDB) as OData service without any additional work. Create a virtual database with your required sources and create any additional views and deploy in the the JBoss EAP 6.1 server, that has Teiid 8.3 or greater installed.  See https://docs.jboss.org/author/display/teiid84final/OData+Support for more information.

So, lets build a simple example of exposing the Excel Sheet. Teiid accesses Excel sheets via JDBC-ODBC bridge, and Windows OS provides a ODBC driver for Excel documents. 

Edit the standalone-teiid.xml, and create Connection Factory
 

    
        jdbc:odbc:Driver={Microsoft Excel Driver (*.xls)};Dbq=c:\ODBC\ExcelData.xls
        odbc
        TRANSACTION_NONE
        
            false
            false
            FailingConnectionOnly
        
        
            {user}
            {password}
        
         
    
        
         sun.jdbc.odbc.JdbcOdbcDriver
     
     



Now create VDB, using the excel as the source
 


    
    
        
    



save the above as "odbc-vdb.xml" and deploy into Teiid Server. See https://docs.jboss.org/author/display/TEIID/Deploying+VDBs for deployment instructions.  Make sure your VDB deployed in "ACTIVE" state.

Now, you can access your Excel data using OData API. Either using the your Web Browser or favorite OData client, go to 

 
     http://:/odata/odbc/excel.Sheet1
For more complete information about OData API see https://docs.jboss.org/author/display/TEIID/OData+Support

Thanks
Ramesh..

Friday, May 24, 2013

Teiid 8.4 CR1 Available

Teiid 8.4 CR1 for EAP 6.1.Alpha* is now available in our downloads.  The new features Beta2:
  • TEIID-2504 Improved socket results processing large results processing over a socket transport have been significantly improved for forward only results. Scrollable results will have only a marginal improvement.
  • TEIID-2427 TEIID-2311 Column Masking added column masking and refined row-based security. Also allow for permissions to be granted via the MetadataFactory methods addPermission and addColumnPermisison to enable adding roles via a MetadataRepository.

Additional web service security features and mongo connectivity have made it into 8.4 and will be fully doc'ed and in the release notes by the next CR/Final.

If needed we will produce a CR2 in approximately 10 days and will generally be selective about what issues get released before the Final. 

Beyond the 8.4 Final we now have the 8.4.1 and 8.5 release version targets active to take additional issues.

*And now more on the switch to EAP 6.1...

If you have any concerns about switching from AS to EAP first read: http://www.jboss.org/jbossas/faq

The most important take away is that EAP 6.1 Alpha is roughly the same as what AS 7.2 Final would have been - including licensing.  So don't be scared off by the name, version, or alpha designation.

Q: How does all this tie into the Teiid release schedule?

A: Teiid 8.4 and subsequent 8.x releases will be targeting the EAP line.  It is not clear however if there will be another public release of EAP off of the AS 7.x line. 

Q: Will subsequent Teiid 8.x releases remain compatible with EAP 6.1 Alpha (or whatever the latest public release is)?

A: Yes, until Wildfly 8 is released and a Teiid 9.0 is available targeting it at which point the community effort will track only the latest.  More than likely this implies both a Teiid 8.5 and 8.6.

Q: When will Teiid 9.0 be available?

A: That's not definitive.  Wildfly has its first alpha already available - http://www.wildfly.org/download/ and https://issues.jboss.org/browse/WFLY targets the release in November.  That would put Teiid 9.0 tentatively at the beginning of 2014.
   

Thursday, May 9, 2013

Teiid 8.4 Beta2 Posted

Teiid 8.4 Beta2 for EAP 6.1.Alpha is now available in our downloads.  The new features in this beta (since Teiid 8.3 Final):
  • TEIID-2444 EAP Platform - the deployment platform for Teiid has been changed to EAP 6.1.Alpha1
  • TEIID-2429 Improved sort performance - improved sorts for large data sets especially under heavy load.
  • TEIID-2449 VDB Zips - added the ability to deploy non-Designer based vdb zip artifacts with the ability to place DDL outside of the VDB.xml via the DDL-FILE metadata repository.
  • TEIID-2423 TEIID-2470 Additional Embedded deployments - xml and zip vdb deployment options are available for Teiid Embedded.
  • TEIID-1092 Session Variables - added the teiid_session_set and teiid_session_get methods to maintain session variables.
  • TEIID-2326 Shared Mat Views - materialized views from imported vdbs will be reused rather than recreated in each importing vdb.
  • TEIID-2344 Automatic DB Version Detection - the JDBC translator database version property in most circumstances will not need to be set manually (non-available JDBC sources in partial results mode or source with connection factories that require an ExecutionContext to obtain a connection will require manual setting of the database version metadata property.  The affected sources are: db2, derby, oracle, postgresql, sqlserver, sybase, teiid)
  • TEIID-2453 STRING_AGG - added the string_agg aggregate function for concatenating binary and character strings.
As you can see we have been quite busy.  There have been nearly 60 issues in total addressed as well as an initial commit of the column masking feature (more to come on this before final).  If all goes well additional ws* security features and mongo connectivity will make it into 8.4 also.

From here a CR can be expected in approximately 2 weeks.  Please take the time to try this release as there is plenty of time to get fixes in.  Beyond the 8.4 Final we now have the 8.4.1 and 8.5 release version targets active to take additional issues.

Thanks,
Steve

Monday, April 29, 2013

Teiid Designer 8.1 Final Released


(download/install)


Key feature for this release is providing the ability to develop and test your VDBs against both Teiid 7.7 and Teiid 8.3 runtimes (JBoss AS 7.1 and AS 5.1 platforms).  This feature allows managing your older VDB artifacts with latest tooling and provides a mechanism to migrate and test your data services on the newer server version.

Other features include:
  • New relational object wizards (Table, View, Procedure, Index)
  • Improved synchronization of workspace models with VDB artifacts
  • Multi-source binding support in VDBs (TEIIDDES-472)
  • New model importer using deployed data source through Teiid runtime (TEIIDDES-1575)

For more details, check out What's New !

Thursday, April 4, 2013

(Teiid 8.4 + EAP 6.1) Alpha

Teiid has been using JBoss AS 7.1.1 as deployment server for last few versions. We wanted to move to later versions of the JBoss AS, but no JBoss AS community versions released after 7.1.1. Then JBoss EAP came along...

Teiid 8.4 Alpha1 is our first attempt to make Teiid deploy using EAP 6.1 Alpha. Even though EAP 6.1 Alpha, has a tag of "Alpha" this is much more stable version of the JBoss AS (7.2.Final).

If you are concerned about licensing issues with EAP 6.1 Alpha, please see this thread.

The installation is same as before, download the EAP 6.1 Alpha and then Teiid 8.4.Alpha1 and unzip the contents of Teiid archive over the EAP's directory.

One notable change you will see is, how Teiid deploys the resource adapters in this new version. Instead of deploying them as RAR files, they are now deployed as the JBoss Modules.  You can still use the RAR file deployment as before,as there is no changes there.

Take it for spin and let us know if you find any issues. Also, we are still considering features for 8.4 version, so bring them to our forums.

Thanks.

Ramesh & Steve.

Thursday, March 14, 2013

Teiid 8.3.0.Final Released

It took us little more than 12 weeks from 8.2 Final, however Teiid 8.3.0.Final is released with many many exciting features.

The features we really excited about are
  1. OData Support - Teiid can now either consume from a OData source or produce results using OData protocol. See related blogs on this topic.
  2. JSON Support - Similar to how user can use SQLXML and define results in XML, Teiid is introducing JSON  language constructs, such that you can produce results in JSON. Now you can generate that Rest web service that returns JSON!!!
  3. Source Temp Tables - create a temporary tables on sources, and generate plans that are efficient in pushing down more work towards the sources.
Many more.. See below for full list

Teiid 8.3 is still based on JBoss AS 7.1.1  The JBoss folks have decided not to release any community releases off of AS 7.x branches, however they recently announced  that EAP 6.1-alpha1 (a productised version of JBoss AS that RedHat offers subscription) is released to community developers.  We will release Teiid 8.4 based on EAP 6.1.x.

Highlighted Features:
  • TEIID-2253 Multi-source improvements - multi-source planning was significantly improved, which will result in much better plans when using multi-source models. The user may also now configure the multi-source column name, auto-populate the multi-column in the actual metadata, and use non-literal multi-source column values.
  • TEIID-1557 Generated Key Retrieval - JDBC now supports generated key retrieval, see the Client and the Developers Guides for more.
  • TEIID-2311 Permission Conditions - data roles can now specify SQL conditions that will enforce row-based security and physical table insert/update checked constraints. See the Reference Guide for more.
  • TEIID-2341 Join Planning Improvements - ANSI join structures can now be preserved with the preserve hint, i.e. FROM /*+ PRESERVE */ (tbl1 INNER JOIN tbl2 ON ...), which is similar to the Oracle ORDERED hint. Also sources restricted to only outer joins (such as Salesforce) can have INNER or effectively INNER joins pushed to them.
  • TEIID-2312 Source security domains - Ldap, Salesforce, google-spreadsheet and ws (basic auth) translators can now define "security-domain" in their resource-adaptor configuration for security. When defined along with caller-identity the logged in user's credentials are passed to source for authentication.
  • TEIID-2182 Events - Capture the VDB loading and unloading events now from registering a listener on EventDistributor
  • TEIID-2241 CLI script for Standalone - Install Teiid into any AS7 standalone configuration using the provided CLI script.
  • TEIID-2152 Basic MERGE support - support for a simplistic MERGE statement was added. See the Reference for more.
  • TEIID-1757 OData Translator - A translator for OData based sources has been added. 
  • TEIID-1854 OData Transport - Access any deployed VDB into Teiid using OData protocol, essentially Teiid is now a OData server.
  • TEIID-2363 Concurrent Join Processing Enhancements - both sides of a join will processed more when possible. See also the org.teiid.joinPrefetchBatches system property in the Admin Guide.
  • TEIID-196 Source Temporary Tables - temporary (locally scoped) tables may be defined at runtime against physical sources with a CREATE FOREIGN TEMPORARY TABLE statement. See the Reference for more.
  • TEIID-2159 JSON Production - added the JSONARRAY, JSONARRAY_AGG, JSONOBJECT, and JSONPARSE constructs for producing JSON. See the Reference for more.
  • TEIID-2248 Bulk Salesforce Inserts - added the ability for the Salesforce translator to perform bulk inserts.

This work was due in large part to the great participation from Teiid community users, a BIG thanks to all who participated in providing valuable input through their JIRAs and community suggestions.

Get Involved with Teiid:

Got a idea to better way plan a query or some cool feature that Teiid can implement, come and open a forum discussion, let's get it captured and put it on roadmap.

Wrote a blog? did any performance testing on Teiid? Tell us what worked and what didn't? Help us fix the issues to make Teiid successful.

Thank you.

Ramesh & Steve

Wednesday, February 27, 2013

Teiid 8.3 CR1 Available

Teiid 8.3 CR1 has been posted to the downloads.  The new features since beta2 include:
  • TEIID-196 Source Temporary Tables - temporary (locally scoped) tables may be defined at runtime against physical sources with a CREATE FOREIGN TEMPORARY TABLE statement. See the Reference for more.
  • TEIID-2159 JSON Production - added the JSONARRAY, JSONARRAY_AGG, JSONOBJECT, and JSONPARSE constructs for producing JSON. See the Reference for more.
  • TEIID-2248 Bulk Salesforce Inserts - added the ability for the Salesforce translator to perform bulk inserts.
We've been getting great feedback on the beta releases, which has led to several important fixes - so keep up the good work with the CR!  Be sure to contribute any documentation or Javadoc updates that you see are needed as well. 

8.3 Final is on track for 2 weeks from now so now is a great time to help shape the 8.4 release.

Steve

Monday, February 25, 2013

Integrate OData Source Using Teiid

Teiid 8.3.x release has a new translator "odata", that supports consuming of OData sources. Using this translator a user can integrate data from a OData source with other sources in Teiid.

If you like to know what OData is and Teiid support for it, please read this previous post http://teiid.blogspot.com/2013/02/odata-support-in-teiid.html

In this blog, we will write a Teiid VDB, that shows how to consume a OData source, and how Teiid uses OData's CSDL metadata document to expose it in relational form such that it can be integrated with other sources.

So, let's begin..

In a Teiid system, OData translator is a extension of the "ws" web services translator, since all OData queries are REST based web service calls, and results are published in "atom/pub" or "json" format, it uses the same resource adapter "teiid-connector-ws.rar". For this example, we will use a example OData service at http://services.odata.org/Northwind/Northwind.svc/

The below is a CLI Script to create connection to OData source
 
/subsystem=resource-adapters/resource-adapter=northwindDS:add(module=org.jboss.teiid.resource-adapter.webservice)
/subsystem=resource-adapters/resource-adapter=northwindDS/connection-definitions=northwindDS:add(jndi-name=java:/northwindDS, class-name=org.teiid.resource.adapter.ws.WSManagedConnectionFactory, enabled=true, use-java-context=true)
/subsystem=resource-adapters/resource-adapter=northwindDS/connection-definitions=northwindDS/config-properties=EndPoint:add(value=http://services.odata.org/Northwind/Northwind.svc)
/subsystem=resource-adapters/resource-adapter=northwindDS:activate

Once the connection is created, let's create a Dynamic VDB (northwind-vdb.xml) that uses the connection created above and defines virtual database

<vdb name="northwind-rw" version="1">
    <model name="nw">
        <property name="importer.importKeys" value="true"/>
        <property name="importer.importProcedures" value="true">
         <source connection-jndi-name="java:/northwindDS" name="northwind-connector" translator-name="odata"/>
    </model>
</vdb>

When you deploy the above VDB in Teiid server, it automatically imports the metadata of this web service from http://services.odata.org/Northwind/Northwind.svc/$metadata and creates relational structure based on it.

For example, from the metadata call above one of the entity type in the CSDL document looks like below
<EntityType Name="Customer">
  <Key><PropertyRef Name="CustomerID"/></Key>
  <Property Name="CustomerID" Type="Edm.String" Nullable="false" MaxLength="5" Unicode="true" FixedLength="true"/>
  <Property Name="CompanyName" Type="Edm.String" Nullable="false" MaxLength="40" Unicode="true" FixedLength="false"/>
  <Property Name="ContactName" Type="Edm.String" Nullable="true" MaxLength="30" Unicode="true" FixedLength="false"/>
  <Property Name="ContactTitle" Type="Edm.String" Nullable="true" MaxLength="30" Unicode="true" FixedLength="false"/>
  <Property Name="Address" Type="Edm.String" Nullable="true" MaxLength="60" Unicode="true" FixedLength="false"/>
  <Property Name="City" Type="Edm.String" Nullable="true" MaxLength="15" Unicode="true" FixedLength="false"/>
  <Property Name="Region" Type="Edm.String" Nullable="true" MaxLength="15" Unicode="true" FixedLength="false"/>
  <Property Name="PostalCode" Type="Edm.String" Nullable="true" MaxLength="10" Unicode="true" FixedLength="false"/>
  <Property Name="Country" Type="Edm.String" Nullable="true" MaxLength="15" Unicode="true" FixedLength="false"/>
  <Property Name="Phone" Type="Edm.String" Nullable="true" MaxLength="24" Unicode="true" FixedLength="false"/>
  <Property Name="Fax" Type="Edm.String" Nullable="true" MaxLength="24" Unicode="true" FixedLength="false"/>
  <NavigationProperty Name="Orders" Relationship="NorthwindModel.FK_Orders_Customers" FromRole="Customers" ToRole="Orders"/>
  <NavigationProperty Name="CustomerDemographics" Relationship="NorthwindModel.CustomerCustomerDemo" FromRole="Customers" ToRole="CustomerDemographics"/>
</EntityType>

It will be converted to a relational table like below in the Teiid's metadata
CREATE FOREIGN TABLE Customers (
    CustomerID string(5) NOT NULL OPTIONS (FIXED_LENGTH TRUE),
    CompanyName string(40) NOT NULL,
    ContactName string(30),
    ContactTitle string(30),
    Address string(60),
    City string(15),
    Region string(15),
    PostalCode string(10),
    Country string(15),
    Phone string(24),
    Fax string(24),
    PRIMARY KEY(CustomerID)
) OPTIONS (UPDATABLE TRUE, EntityAlias 'Customer', EntityType 'NorthwindModel.Customer');

All "NavigationProperty" elements will be converted into FOREIGN KEYS, based on the "Association" and "AssociationSet" elements defined in the CSDL document.

 * If there is One-to-One relation, the FK are created at both ends of the table.
* If there is One-to-Many relation, at Many end of the table, a FK is created to single end of the table.
* If there are Many-to-Many relation, a virtual mapping table is created with PK's of the both tables.

The mapping table is created so that a VDB user can issue a SQL query that joins these both tables together in their user queries. One limitation is user can not select the columns from this mapping table, it can be strictly used only for the table join purposes. When a user queries their deployed VDB using JDBC/ODBC using SQL, those queries will be automatically converted into OData queries and results will be gathered and presented to user in relation form. For example you can issue a query like

select CustomerID, CompanyName from Customers

CustomerIDCompanyName
ALFKIAlfreds Futterkiste
ANATRAna Trujillo Emparedados y helados
ANTONAntonio Moreno Taquería
AROUTAround the Horn

I hope the above gave glimpse of how OData translator works in Teiid. So, take it for test drive and let us know any issues or comments you may have on this subject.

 Ramesh..

Monday, February 18, 2013

Access Teiid from node.js

Are you writing a "node.js" application and would like to access Teiid VDB from it?

If "yes", this is currently possible using NPM package "pg".  Since, Teiid supports the PG transport, you can use this PostgreSQL client for "node.js" for accessing the Teiid.

For example if you have VDB called "northwind" deployed on your Teiid server, and it has table called "customers" and you are using default configuration such as

user = 'user'
password = 'user'
host = 127.0.0.1
port = 35432

then you can use following to write simple access program to Teiid
    
    var pg = require('pg');
    var connectionString = "pg://user:user@localhost:35432/northwind"
    pg.connect(connectionString, function(err, client) {
        client.query('SELECT CustomerID, ContactName, ContactTitle FROM Customers', function(err, result) {
        console.log(result.rows)
    });
    });
If you want access one row at a time, you can also use event mechanism and write
    var pg = require('pg');
    var connectionString = "pg://user:user@localhost:35432/northwind"
    pg.connect(connectionString, function(err, client) {
        var query = client.query('SELECT CustomerID, ContactName, ContactTitle FROM Customers');
        query.on('row', function(row) {
            console.log(row);
        });
        query.on('error', function(error) {
            console.log("failed to query");
        });
        query.on('end', function(error) {
            console.log("closing client");
            client.end();
        });
    });
For more information please take look at

https://npmjs.org/package/pg
https://github.com/brianc/node-postgres

If you do write an application using these please do share your experiences with us.

Tuesday, February 12, 2013

OData Support In Teiid

What Is OData?

"The Open Data Protocol (OData) is a Web protocol for querying and updating data that provides a way to unlock your data and free it from silos that exist in applications today. OData does this by applying and building upon Web technologies such as HTTP, Atom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores. The protocol emerged from experiences implementing AtomPub clients and servers in a variety of products over the past several years.  OData is being used to expose and access information from a variety of sources including, but not limited to, relational databases, file systems, content management systems and traditional Web sites. " - taken directly from http://www.odata.org

 OData is specification from OASIS committee, and this specification is backed by Microsoft, SAP AG, IBM, Citrix, Progress Software and WSO2.

Why you need OData?

In today's service oriented enterprise architectures more and more companies are exposing their enterprise data and business services using the REST based services. Even though REST defines easy ways to access your services, it does not define a unified calling semantics to access your services nor provides details data model. Thus everybody inventing their own REST API access mechanisms, defining specific ways define their data model to the end user application.  OData fills this gap, and provides specification that defines standard ways define CRUD operations on your data source along with standard ways to define your data/schema model.

How OData helps?

If an enterprise implements OData specification to expose their data and services, this layer self describes the data model and will define standard ways access and query in very powerful ways. Thus the client applications that are trying to use these services can write their applications using the rich metadata provided by OData and at the same time avoid vendor lock-in as it is a standard.

For example, Netflix a on-line movie streaming service, exposes details about their movie catalogue using OData. Using this service any developer who is writing a enterprise application or mobile application, can show case movies and details about them using OData. Since the OData also exposes the data/schema model about the services, they are self explanatory.

Another example is SAP Netweaver Gateway. SAP also uses OData to expose their services to end customers. The result, customers know exactly how to access SAP services and integrate them in their applications.

See the growing ecosystem of OData implementations.

What this got to do with Teiid?

Well, good question. A LOT.

Teiid is a data integration and virtualization engine. Teiid provides seamless integrations with many different varieties of sources like RDBMS, Flat Files, Web Services, packaged applications etc.  So, OData was a natural fit to provide features around data integration.

The support for OData in Teiid is two (2) fold.

OData Translator - Using this translator, you can bring in any OData sources as part of virtual database you create in Teiid and integrate this source's data with any other sources. Given SQL based user query, Teiid is capable of converting that request into a REST based OData call and fetch the results automatically and provide them in a relational perspective.

OData Server - If you have created a virtual database using Teiid, then with out any additional work, you can expose your VDB through OData. Previously Teiid only supported access through JDBC, ODBC and custom SOAP and REST  protocols. Now Teiid supports ability to access your VDB using OData.

Teiid uses couple different frameworks to make this possible. Primarily, the OData support is provided through OData4J project. Then Resteasy used for rest engine inside JEE container. Since Teiid server is already installed inside a JBoss Application Server, there is no more additional software you need to install. Both the translator and server are available when you install Teiid on JBoss AS.

How can I use them? any examples?

The support for both features is available in Teiid 8.3 Beta2  or later. I am in process of writing two more blogs in next couple weeks, one on each feature as show case, how to use them. So, stay tuned. If you are eager, there is documentation available.



The translator usage is no different to others, so, take a test drive and let us know if you have any questions or see any issues.

Thanks

Ramesh..


Monday, February 11, 2013

Teiid 8.3 Beta2 Posted

Teiid 8.3 Beta2 is now available in our downloads.  The new features in this this beta:
  • TEIID-2363 Concurrent Join Processing Enhancements - both sides of a join will processed more when possible. See also the org.teiid.joinPrefetchBatches system property in the Admin Guide.
  • TEIID-196 Source Temporary Tables - temporary (locally scoped) tables may be defined at runtime against physical sources with a CREATE FOREIGN TEMPORARY TABLE statement. See the Reference for more.
  • TEIID-2159 JSON Production - added the JSONARRAY, JSONARRAY_AGG, JSONOBJECT, and JSONPARSE constructs for producing JSON. See the Reference for more.
We are on track for the first candidate release in two weeks.  Looking ahead we'll be creating an 8.4 release target and should begin work on a 9.0 release by the end of that cycle.  Of course any feedback on the 8.3 release before final is greatly appreciated.

In addition to the AS Teiid Web Console management option, an initial plugin binary for RHQ 4.* has also been posted.  There should be another blog post on this along with subsequent pre-releases so stay tuned.

Steve

Monday, January 28, 2013

Teiid 8.3 Beta1 Available

The march toward 8.3 continues with Beta1 now available in our downloads.  Over the last alpha this beta features:
  • TEIID-2182 Events - Capture the VDB loading and unloading events now from registering a listener on EventDistributor
  • TEIID-2241 CLI script for Standalone - Install Teiid into any AS7 standalone configuration using the provided CLI script.
  • TEIID-2152 Basic MERGE support - support for a simplistic MERGE statement was added. See the Reference for more.
  • TEIID-1757 OData Translator - A translator for OData based sources has been added.
TEIID-1854 and OData support in general is progressing nicely - there will be more on this later in the release.  There has also been lots of important fixes so this should be a good pre-release to take for a test drive and provide early feedback.

The 8.3 development cycle will be extended slightly beyond the normal quarterly window due to the slowdown over the holidays and to accommodate getting a few more features complete.  The release is currently expected by mid-March.

Steve

Friday, January 11, 2013

Planning started on Teiid Designer Komodo release

With the changes in Teiid 8.x coming on-line and the use of Dynamic VDB's in Openshift becoming more visible, it's become apparent that Teiid's primary tooling, Teiid Designer, should re-visit it's architecture and embrace these concepts.

A few discussions have already occurred amongst community members and a release code-name Komodo (a type of lizard akin to Teiid, the whip-tail lizard) was adopted to identify the next-generation Teiid Designer for the time being.

Options for incorporating elements of Teiid's new direction could be accomplished through drastically modifying/removing Designer's current underlying EMF framework and developing anew from a fresh user perspective. In addition there may be benefits to creating a tighter coupling between Komodo and Teiid runtime, as well as incorporating other JBoss technologies/projects like ModeShape and SRAMP work being done in Overlord.

A summary of the discussions that have taken place and links to applicable JIRAs are available via wiki at: Teiid Designer Komodo Release Development Plan

Check it out...feel free to chime in, comment on or add your ideas!

Barry LaFond
Teiid Designer Project

Wednesday, January 9, 2013

Hello 2013 - Teiid 8.3 Alpha2 is Available

8.3 Alpha2 is now available in our downloads.  This alpha features:
  • TEIID-2311 Permission Conditions - data roles can now specify SQL conditions that will enforce row-based security and physical table insert/update checked constraints. See the Reference Guide for more.
  • TEIID-2341 Join Planning Improvements - ANSI join structures can now be preserved with the preserve hint, i.e. FROM /*+ PRESERVE */ (tbl1 INNER JOIN tbl2 ON ...), which is similar to the Oracle ORDERED hint. Also sources restricted to only outer joins can have INNER or effectively INNER joins pushed to them.
  • TEIID-2312 Source security domains - Ldap, Salesforce, google-spreadsheet and ws (basic auth) translators can now define "security-domain" in their resource-adaptor configuration for security. When defined along with caller-identity the logged in user's credentials are passed to source for authentication.
TEIID-1854 also had it's initial commit, which will provide Teiid with the ability to automatically be an OData server.  There will be more on this later in the release.

Please provide any early feedback as we'll shortly process to the beta stage and begin pushing feature work to later releases.

Steve

Teiid Powers Jaspersoft Virtualization

We're excited to pass along that Jaspersoft 5 is providing data virtualization through the Teiid community project.  This is a natural fit for us as we provide a great underpinning to reporting/analysis tooling.  It will be interesting to see what Jaspersoft integrator and end users will bring to the Teiid community.

Steve