com.toolsverse.etl.core.engine

Class MetadataTracker

Object

com.toolsverse.etl.core.engine.MetadataTracker

public class MetadataTracker
extends Object

The collection of the utility methods to check and update metadata in the source and in the destination.

Since:

5.9

Version:

1.0

Constructor Summary

Constructors

Constructor and Description
MetadataTracker()

Method Summary

Modifier and Type	Method and Description
void	createTableAndIndexesOrChangeFieldDataTypeIfNeeded(Scenario scenario, EtlConfig config, Destination destination, DataSet dataSet, Alias alias, CodeGen codeGen) Creates the table and indexes or change field data type if needed.
String	getLoadKey(EtlConfig config, Destination destination, String destTableName, List<String> fields, DataSetFields dataSetFields, Driver driver) Gets the load key.
String	getSourceKeys(Destination destination, EtlConfig config, DataSetFields dataSetFields) Gets the source keys.
boolean	needToUpdateMetadataInTarget(Destination destination, DataSet dataSet) Checks if ETL engine needs to update metadata for the destination based on the existing data.
boolean	recreateDestTableIfNeededWhenHWMIsEnabled(Scenario scenario, Source source, Destination destination, EtlConfig config) Recreate destination table if needed when the high watermark replication is enabled.
void	updateMetadataInTarget(EtlConfig config, Destination destination, DataSet dataSet, Scenario scenario) Update metadata in target.
void	updateOrderOfFields(Destination destination, DataSet dataSet) Update order of fields.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MetadataTracker

public MetadataTracker()

Method Detail

getSourceKeys

public String getSourceKeys(Destination destination,
                            EtlConfig config,
                            DataSetFields dataSetFields)

Gets the source keys.

Parameters:

destination - the destination

config - the config

dataSetFields - the data set fields

Returns:

the source keys

getLoadKey

public String getLoadKey(EtlConfig config,
                         Destination destination,
                         String destTableName,
                         List<String> fields,
                         DataSetFields dataSetFields,
                         Driver driver)

Gets the load key.

Parameters:

source - the source

destination - the destination

sourceTableName - the source table name

destTableName - the destination table name

Returns:

the load key

needToUpdateMetadataInTarget

public boolean needToUpdateMetadataInTarget(Destination destination,
                                            DataSet dataSet)

Checks if ETL engine needs to update metadata for the destination based on the existing data.

Parameters:

destination - the destination

dataSet - the data set

Returns:

true, if successful

updateMetadataInTarget

public void updateMetadataInTarget(EtlConfig config,
                                   Destination destination,
                                   DataSet dataSet,
                                   Scenario scenario)

Update metadata in target. Used by Snowflake and Redshift flows

Parameters:

config - the config

destination - the destination

dataSet - the data set

scenario - the scenario

updateOrderOfFields

public void updateOrderOfFields(Destination destination,
                                DataSet dataSet)

Update order of fields.

Parameters:

destination - the destination

dataSet - the data set

createTableAndIndexesOrChangeFieldDataTypeIfNeeded

public void createTableAndIndexesOrChangeFieldDataTypeIfNeeded(Scenario scenario,
                                                               EtlConfig config,
                                                               Destination destination,
                                                               DataSet dataSet,
                                                               Alias alias,
                                                               CodeGen codeGen)
                                                        throws Exception

Creates the table and indexes or change field data type if needed.

Parameters:

scenario - the scenario

config - the config

destination - the destination

dataSet - the data set

alias - the alias

codeGen - the code gen

Throws:

Exception - the exception

recreateDestTableIfNeededWhenHWMIsEnabled

public boolean recreateDestTableIfNeededWhenHWMIsEnabled(Scenario scenario,
                                                         Source source,
                                                         Destination destination,
                                                         EtlConfig config)
                                                  throws Exception

Recreate destination table if needed when the high watermark replication is enabled.

Parameters:

scenario - the scenario

source - the source

destination - the destination

config - the etl config

Throws:

Exception - the exception