com.toolsverse.etl.core.engine

Class EtlMetricsTracker

Object

com.toolsverse.etl.core.engine.EtlMetricsTracker

All Implemented Interfaces:

Serializable

public class EtlMetricsTracker
extends Object
implements Serializable

This class tracks basic ETL metrics, such as records extracted, records loaded, timestamps, exceptions, etc.

Since:

5.0

Version:

1.0

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
EtlMetricsTracker()

Method Summary

Modifier and Type	Method and Description
void	finishTrackingExtract(String name, DataSet dataSet, String exception) Finish tracking extract.
void	finishTrackingLoad(String name, Destination destination, String exception) Finish tracking load.
Long	getCurrentNumberOfExtractedRecords(String name) Gets the current number of extracted records.
Long	getCurrentNumberOfLoadedRecords(String name) Gets the current number of loaded records.
Map<String,FileMetrics>	getFileMetrics() Gets the file metrics.
String	getFileMetricsFileName() Gets the file metrics file name.
String	getFileName() Gets the file name.
String	getFormat() Gets the format.
Object	getHighWatermark(EtlConfig config, Scenario scenario, Source source, Destination destination, Boolean... onLoad) Gets the high watermark value.
Object	getHighWatermark(String name, Boolean... onLoad) Gets the high watermark value.
Map<String,EtlMetrics>	getMetrics() Gets the ETL metrics.
TypedKeyValue<String,String>	prepareTrackingExtract(Scenario scenario, Source source, Destination destination, String sql, EtlConfig config) Prepare tracking extract.
void	read() Read metrics from the data folder using given format.
void	readMetricsFromCsv() Read metrics from csv.
void	readMetricsFromJson() Read metrics from json.
boolean	setCurrentNumberOfExtractedRecords(String name, Long value) Sets the current number of extracted records.
boolean	setCurrentNumberOfLoadedRecords(String name, Long value) Sets the current number of loaded records.
void	setFileName(String fileName) Sets the file name.
void	setFormat(String format) Sets the format.
void	setLastCreatedFile(String name, String fileName) Sets the last created file.
void	setLastReadFile(String name, String fileName) Sets the last created file.
void	startTrackingExtract(String name, Source source, DataSet dataSet, Destination destination, String flow) Start tracking extract.
String	startTrackingFile(FileMetrics fileMetric) Start tracking file.
void	startTrackingLoad(String name, Destination destination, String flow) Start tracking load.
void	stopTrackingFile(FileMetrics fileMetric, String exception, String action) Stop tracking file.
void	updateEtlMetrics(Map<String,EtlMetrics> from) Update etl metrics.
void	write(boolean merge) Write metrics to the data folder.
void	writeFileMetrics() Write file metrics to the data folder.
void	writeMetricsToCsv(Map<String,BaseMetrics> metrics, String metricFileName) Write metrics to csv file.
void	writeMetricsToJson(Map<String,BaseMetrics> metric, String metricFileName) Write metrics to csv file.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EtlMetricsTracker

public EtlMetricsTracker()

Method Detail

finishTrackingExtract

public void finishTrackingExtract(String name,
                                  DataSet dataSet,
                                  String exception)

Finish tracking extract.

Parameters:

name - the name

dataSet - the data set

exception - the exception

finishTrackingLoad

public void finishTrackingLoad(String name,
                               Destination destination,
                               String exception)

Finish tracking load.

Parameters:

name - the name

destination - the destination

exception - the exception

getCurrentNumberOfExtractedRecords

public Long getCurrentNumberOfExtractedRecords(String name)

Gets the current number of extracted records.

Parameters:

name - the name

Returns:

the current number of extracted records

getCurrentNumberOfLoadedRecords

public Long getCurrentNumberOfLoadedRecords(String name)

Gets the current number of loaded records.

Parameters:

name - the name

Returns:

the current number of loaded records

getFileMetrics

public Map<String,FileMetrics> getFileMetrics()

Gets the file metrics.

Returns:

the file metrics

getFileMetricsFileName

public String getFileMetricsFileName()

Gets the file metrics file name.

Returns:

the fileName

getFileName

public String getFileName()

Gets the file name.

Returns:

the fileName

getFormat

public String getFormat()

Gets the format.

Returns:

the format

getMetrics

public Map<String,EtlMetrics> getMetrics()

Gets the ETL metrics.

Returns:

the metrics

updateEtlMetrics

public void updateEtlMetrics(Map<String,EtlMetrics> from)

Update etl metrics.

Parameters:

from - the map of etl metrics to update from

getHighWatermark

public Object getHighWatermark(String name,
                               Boolean... onLoad)

Gets the high watermark value.

Parameters:

name - the name of the etl block

onLoad - if true or null - get high watermark from the onload event, otherwise - from the onextract.

Returns:

the high watermark value

getHighWatermark

public Object getHighWatermark(EtlConfig config,
                               Scenario scenario,
                               Source source,
                               Destination destination,
                               Boolean... onLoad)
                        throws Exception

Gets the high watermark value.

Parameters:

config - the config

scenario - the scenario

source - the source

destination - the destination

onLoad - the on load

Returns:

the high watermark value

Throws:

Exception - the exception

read

public void read()
          throws Exception

Read metrics from the data folder using given format.

Throws:

Exception - the exception

readMetricsFromCsv

public void readMetricsFromCsv()
                        throws Exception

Read metrics from csv.

Throws:

Exception - the exception

readMetricsFromJson

public void readMetricsFromJson()
                         throws Exception

Read metrics from json.

Throws:

Exception - the exception

setCurrentNumberOfExtractedRecords

public boolean setCurrentNumberOfExtractedRecords(String name,
                                                  Long value)

Sets the current number of extracted records.

Parameters:

name - the name

value - the value

Returns:

true, if successful

setCurrentNumberOfLoadedRecords

public boolean setCurrentNumberOfLoadedRecords(String name,
                                               Long value)

Sets the current number of loaded records.

Parameters:

name - the name

value - the value

Returns:

true, if successful

setFileName

public void setFileName(String fileName)

Sets the file name.

Parameters:

fileName - the fileName to set

setFormat

public void setFormat(String format)

Sets the format.

Parameters:

format - the new format

setLastCreatedFile

public void setLastCreatedFile(String name,
                               String fileName)

Sets the last created file.

Parameters:

name - the name

fileName - the file name

setLastReadFile

public void setLastReadFile(String name,
                            String fileName)

Sets the last created file.

Parameters:

name - the name

fileName - the file name

startTrackingExtract

public void startTrackingExtract(String name,
                                 Source source,
                                 DataSet dataSet,
                                 Destination destination,
                                 String flow)

Start tracking extract.

Parameters:

name - the dataset name

source - the source

dataSet - the data set

destination - the optional destination

flow - the flow name

prepareTrackingExtract

public TypedKeyValue<String,String> prepareTrackingExtract(Scenario scenario,
                                                           Source source,
                                                           Destination destination,
                                                           String sql,
                                                           EtlConfig config)
                                                    throws Exception

Prepare tracking extract.

Parameters:

scenario - the scenario

source - the source

destination - the destination

sql - the sql

config - the config

Returns:

the updated sql

Throws:

Exception - the exception

startTrackingFile

public String startTrackingFile(FileMetrics fileMetric)

Start tracking file.

Parameters:

fileMetric - the file metric

Returns:

the uuid

startTrackingLoad

public void startTrackingLoad(String name,
                              Destination destination,
                              String flow)

Start tracking load.

Parameters:

name - the name

destination - the destination

flow - the flow name

stopTrackingFile

public void stopTrackingFile(FileMetrics fileMetric,
                             String exception,
                             String action)

Stop tracking file.

Parameters:

fileMetric - the file metric

exception - the exception

action - the action

write

public void write(boolean merge)
           throws Exception

Write metrics to the data folder.

Parameters:

merge - if true system will merge metrics with existing (granted file does exist)

Throws:

Exception - the exception

writeFileMetrics

public void writeFileMetrics()
                      throws Exception

Write file metrics to the data folder.

Throws:

Exception - the exception

writeMetricsToCsv

public void writeMetricsToCsv(Map<String,BaseMetrics> metrics,
                              String metricFileName)
                       throws Exception

Write metrics to csv file.

Parameters:

metrics - the metrics

metricFileName - the metric file name

Throws:

Exception - the exception

writeMetricsToJson

public void writeMetricsToJson(Map<String,BaseMetrics> metric,
                               String metricFileName)
                        throws Exception

Write metrics to csv file.

Parameters:

metric - the metric

metricFileName - the metric file name

Throws:

Exception - the exception