Skip to main content

Teiid User Defined Aggregate Support

Building upon our support for pushdown and non-pushdown user defined scalar functions, Teiid with 8.0 CR1 also offers the ability to specify user defined aggregate functions (UDAF).

UDAFs are useful to extend Teiid's already extensive aggregate function library.  They also allow Teiid to implement the final processing over pushdown aggregation. For partitioned datasets there is no additional work for Teiid to do, but the UDAF may be needed to express custom aggregation to the participating data sources. For non-partitioned datasets single argument UDAFs may be marked as decomposable to indicate that not only can the initial aggregation be pushed down, but that Teiid will perform the final integration/aggregation of results itself. This provides map-reduce like behavior where processing can be performed in parallel on a large number of source systems.

UDAFs are defined similarly to UDFs and share most of the same metadata properties.  Building on the prior post using Dynamic VDB DDL metadata we can add a user defined aggregate to mimic the commonly supported GROUP_CONCAT aggregate function:

dynamic-portfolio-vdb.xml

 <vdb name="DynamicPortfolio" version= "1">  
   <model name="MarketData">  
     <source name="text-connector" translator-name="file" connection-jndi-name="java:/marketdata-file"/>  
   </model>  
   <model visible = "true" type = "VIRTUAL" name = "portfolio">  
      <metadata type = "DDL"><![CDATA[  
        CREATE VIEW stock (  
         symbol varchar,  
         price decimal  
         ) AS   
          select stock.* from (call MarketData.getTextFiles('*.txt')) f,   
          TEXTTABLE(f.file COLUMNS symbol string, price bigdecimal HEADER) stock;

        CREATE VIRTUAL FUNCTION GROUP_CONCAT(val STRING, sep CHAR) RETURNS STRING 
          OPTIONS (AGGREGATE 'TRUE', "NULL-ON-NULL" 'TRUE', JAVA_CLASS 'example.GroupConcat',
            "ALLOWS-ORDERBY" 'TRUE', JAVA_METHOD 'addInput');
      ]]>  
      </metadata>  
   </model>  
 </vdb> 

The Java class shown below to back the non-pushdown function is straight-forward. The class, designated by the JAVA_CLASS option, must extend org.teiid.UserDefinedAggregate and provide a method, designated by the JAVA_METHOD option, with a signature that accepts the expected inputs.


public static class GroupConcat implements UserDefinedAggregate {
  
 private boolean first = true;
 private boolean isNull = true;
 private StringBuffer buffer = new StringBuffer();
  
 public void addInput(String val, char separator) {
  if (!first) {
   buffer.append(separator);
  }
  buffer.append(val);
  first = false;
  isNull = false;
 }
  
 @Override
 public String getResult(org.teiid.CommandContext commandContext) {
  if (isNull) {
   return null;
  }
  return buffer.toString();
 }

 @Override
 public void reset() {
  first = true;
  isNull = true;
  buffer = new StringBuffer();
 } 
}

You can follow the instructions in the Developer's Guide for more on coding and deploying user defined functions. With the VDB and the UDAF module deployed you should now be able to use the aggregate in your queries.

For example:
SELECT GROUP_CONCAT(SYMBOL, ',' ORDER BY PRICE) AS STOCK_LIST FROM STOCK

For now though this feature is for leading edge users as Teiid Designer support is still unfolding.  Designer 8.0 will overhaul their usage of the Function model which has traditionally been used for UDFs.  So there will be more to come from both the Teiid and Teiid Designer sides shortly.    

Steve

Comments

Popular posts from this blog

Teiid 8.11 Beta1 and 8.10.1 Released

Teiid 8.11 Beta1 is now available from the  downloads  and maven.  Feature highlights since Alpha2 include: TEIID-3434 More caching control over ttls in the result set cache down to the schema/table level. TEIID-3412 MS Access support via the UCanAccess driver. The UCanAccess support is necessary for those running on Java 1.8 as the JDBC ODBC bridge has been removed from the JRE. The waiting continues on EAP 6.4 Alpha1 - it still should be available shortly and should be the platform target for Teiid 8.11 Beta2. Of course, let us know if you find any issues with these early releases.  There's still plenty of time to get fixes into the final release. Teiid 8.10.1 is also available.  It addresses 5 important issues discovered since 8.10 was released: [ TEIID-3409 ] - PostgreSQLExecutionFactory TranslatorProperty annotation in wrong place [ TEIID-3437 ] - Inconsistencies with row count handling [ TEIID-3438 ] - Null value returned from BlobImpl

Tech Tip: Teiid SQL Language MAKEDEP Hint Explained

In this article I will explain what a MAKEDEP hint is, how and when, why it should be used in Teiid. What: MAKEDEP is query hint.  When a query hint is defined in the SQL query it influences the Teiid query planner to optimize the query in a way that is driven by the user. MAKEDEP means "make this as a dependent join". What is a Dependent Join? For example if we have query like: SELECT * FROM X INNER JOIN Y ON X.PK = Y.FK Where the data for X, and Y are coming from two different sources like Oracle and WebService in Teiid, so in relational algebra you can represent above query as Here the result tuples from node X and node Y are being simultaneously fetched by Teiid query engine, then it joins the both the results inside Teiid engine based on the specified X.PK = Y.PK condition and returns the filtered resulted to the user. simple.. Now, what if, if X table has 5 rows and Y table has 100K rows? In order to do the JOIN naively Teiid need sto read all the 5

Teiid Spring Boot 1.7.0 Released

Teiid Spring Boot version 1.7.0 to support Teiid 16.0 has been released. This release is mainly to support the Teiid's latest version.  In this release, the support for OpenAPI code generation based on VDB has been removed as there is no community interest and moreover it was at OpenAPI 2.0, and the industry has moved to 3.0 and beyond. There are no plans to further pursue this feature. VDB maven plugin was also removed, which was intended to be a replacement for the VDB importing feature was to be used when working on OpenShift, however, since it requires the Maven repository and does not completely represent the feature as defined on the WildFly based deployments this is also removed. You can still use the VDB import feature with Teiid Spring Boot, simply define the VDB with your "IMPORT DATABASE" statements and provide the additional files along with the main VDB file. During the start of the application, Teiid Spring Boot will load all the necessary DDL files for the