com.toolsverse.etl.core.engine

Class Extractor

Object

com.toolsverse.etl.core.engine.Extractor

public class Extractor
extends Object

Extracts data from the multiple sources. Extracts can run in parallel.

Since:

1.0

Version:

2.0

Constructor Summary

Constructors

Constructor and Description
Extractor(TransactionMonitor transactionMonitor) Instantiates a new extractor.

Method Summary

Modifier and Type	Method and Description
void	extract(EtlConfig config, Scenario scenario) Extracts data from all sources.
void	extract(EtlConfig config, Scenario scenario, Loader loader, int index) Extracts data from all sources.
void	extract(EtlConfig config, Scenario scenario, Source source, Destination destination, AddRecordCallback addRecordCallback) Extracts data from the source.
void	extractMandatory(EtlConfig config, Scenario scenario) Extracts mandatory data sets.
boolean	hasMixOfStreamAndNotStream(EtlConfig config, Scenario scenario) Checks if the scenario contains mix of stream and not stream transformations.
static Object	lookup(EtlConfig config, Scenario scenario, String connectionName, String name, String fieldName, String sql, TypedKeyValue<String,Object>... args) Extract data set, execute provided SQL statement and return the value of the provided field.This method is thread safe.
static DataSet	lookup(EtlConfig config, Scenario scenario, String connectionName, String name, String sql, TypedKeyValue<String,Object>... args) Extract data set, and optionally execute provided SQL statement.
static Object	lookupNoCache(EtlConfig config, Scenario scenario, String connectionName, String name, String fieldName, String sql, TypedKeyValue<String,Object>... args) Extract data set, execute provided SQL statement and return the value of the provided field.
static DataSet	lookupNoCache(EtlConfig config, Scenario scenario, String connectionName, String name, String sql, TypedKeyValue<String,Object>... args) Extract data set, and optionally execute provided SQL statement.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Extractor

public Extractor(TransactionMonitor transactionMonitor)

Instantiates a new extractor.

Parameters:

transactionMonitor - the transaction monitor

Method Detail

extract

public void extract(EtlConfig config,
                    Scenario scenario,
                    Loader loader,
                    int index)
             throws Exception

Extracts data from all sources.

Parameters:

config - the etl config

scenario - the scenario

loader - the loader

index - the index

Throws:

Exception - in case of any error

extract

public void extract(EtlConfig config,
                    Scenario scenario)
             throws Exception

Extracts data from all sources.

Parameters:

config - the etl config

scenario - the scenario

Throws:

Exception - in case of any error

hasMixOfStreamAndNotStream

public boolean hasMixOfStreamAndNotStream(EtlConfig config,
                                          Scenario scenario)

Checks if the scenario contains mix of stream and not stream transformations.

Parameters:

config - the config

scenario - the scenario

Returns:

true, if scenario has mix of stream and not stream sources

lookup

public static DataSet lookup(EtlConfig config,
                             Scenario scenario,
                             String connectionName,
                             String name,
                             String sql,
                             TypedKeyValue<String,Object>... args)
                      throws Exception

Extract data set, and optionally execute provided SQL statement. This method is thread safe. Cache the dataset using the following key: "stored_dataset_" + scenario.getName() + "_" + connectionName + "_" + name

Parameters:

config - the config

scenario - the scenario

connectionName - the connection name

name - the data set name, also can be used as a file name for file based connections

sql - the sql

args - the arguments for SQL query in the format key/value.

Returns:

the data set

Throws:

Exception - in case of any error

lookupNoCache

public static DataSet lookupNoCache(EtlConfig config,
                                    Scenario scenario,
                                    String connectionName,
                                    String name,
                                    String sql,
                                    TypedKeyValue<String,Object>... args)
                             throws Exception

Extract data set, and optionally execute provided SQL statement. This method is thread safe.

Parameters:

config - the config

scenario - the scenario

connectionName - the connection name

name - the data set name, also can be used as a file name for file based connections

sql - the sql

args - the arguments for SQL query in the format key/value.

Returns:

the data set

Throws:

Exception - in case of any error

lookup

public static Object lookup(EtlConfig config,
                            Scenario scenario,
                            String connectionName,
                            String name,
                            String fieldName,
                            String sql,
                            TypedKeyValue<String,Object>... args)
                     throws Exception

Extract data set, execute provided SQL statement and return the value of the provided field.This method is thread safe.

Parameters:

config - the config

scenario - the scenario

connectionName - the connection name

name - the data set name, also can be used as a file name for file based connections

fieldName - the field name

sql - the sql

args - the arguments for SQL query in the format key/value.

Returns:

the data set

Throws:

Exception - in case of any error

lookupNoCache

public static Object lookupNoCache(EtlConfig config,
                                   Scenario scenario,
                                   String connectionName,
                                   String name,
                                   String fieldName,
                                   String sql,
                                   TypedKeyValue<String,Object>... args)
                            throws Exception

Extract data set, execute provided SQL statement and return the value of the provided field. Cache the dataset using the following key: "stored_dataset_" + scenario.getName() + "_" + connectionName + "_" + name This method is thread safe.

Parameters:

config - the config

scenario - the scenario

connectionName - the connection name

name - the data set name, also can be used as a file name for file based connections

fieldName - the field name

sql - the sql

args - the arguments for SQL query in the format key/value.

Returns:

the data set

Throws:

Exception - in case of any error

extract

public void extract(EtlConfig config,
                    Scenario scenario,
                    Source source,
                    Destination destination,
                    AddRecordCallback addRecordCallback)
             throws Exception

Extracts data from the source. Runs extract in its own thread if requested.

Parameters:

config - the etl config

scenario - the scenario

source - the source

destination - the destination

addRecordCallback - the add record callback

Throws:

Exception - in case of any error

extractMandatory

public void extractMandatory(EtlConfig config,
                             Scenario scenario)
                      throws Exception

Extracts mandatory data sets. The mandatory data sets cannot be combined with the load.

Parameters:

config - the etl config

scenario - the scenario

Throws:

Exception - in case of any error