com.toolsverse.etl.connector.text

Class TextConnector

Object

com.toolsverse.ext.BaseExtension

com.toolsverse.etl.connector.BaseDataSetConnector

com.toolsverse.etl.connector.text.TextConnector

All Implemented Interfaces:

DataSetConnector<TextConnectorParams,ConnectorResult>, ExtensionModule, StreamingConnector<TextStreamingPackage,TextProducerPackage>, Serializable, Comparable<ExtensionModule>

public class TextConnector
extends BaseDataSetConnector
implements DataSetConnector<TextConnectorParams,ConnectorResult>, StreamingConnector<TextStreamingPackage,TextProducerPackage>

Reads and writes delimited and fixed-length text files. Supports data streaming.

Since:

2.0

Version:

2.0

See Also:

Serialized Form

Field Summary

Fields inherited from class com.toolsverse.etl.connector.BaseDataSetConnector

DEFAULT_IMPORT

Fields inherited from interface com.toolsverse.ext.ExtensionModule

APP, COMMAND, ETL_DRIVER, FORMATTER, IDE_NODE, NO_RANK, WIDGET

Constructor Summary

Constructors

Constructor and Description
TextConnector()

Method Summary

Modifier and Type	Method and Description
void	beforeStream(TextProducerPackage producerPackage) Executed before record has been streamed.
boolean	canRead() Checks if connector can read data.
boolean	canWrite() Checks if connector can write data.
void	cleanUp(TextConnectorParams params, DataSet dataSet, Driver driver) Cleans up after DataSetConnector populated or persisted DataSet.
void	closeStream(TextStreamingPackage streamingPackage) Close stream.
boolean	consume(TextStreamingPackage streamingPackage, Object message, AtomicInteger row) Consume the next record.
void	endStream(TextProducerPackage producerPackage) Executed after record has been streamed.
TextConnectorParams	getDataSetConnectorParams() Gets the DataSetConnectorParams.
String	getDisplayName() Gets the display name.
static int[]	getLengthArray(TextConnectorParams params, int fieldCount) Gets the array of lengths for each field in the fixed-length file.
String	getName() Gets the name.
String	getStringToPersist(String value, boolean stripNotPrintable, boolean removeEol) Gets the string to persist.
String	getVendor() Gets the vendor.
String	getVersion() Gets the version of the extension module.
String	getXmlConfigFileName() Gets the path to the xml configuration file for the module.
String	initQueryLabel() The init query label.
void	inlinePersist(TextConnectorParams params, DataSet dataSet, Driver driver, DataSetRecord record, int row, int records) Persists data set row.
boolean	isFileConnector() Checks if this is a connector to the file based data source.
boolean	isTextFormat() Checks if this is a connector for the text-based data format, such as JOSN, XML, etc.
String[]	parseLineWithCharSeparators(String line, String separator, String charSeparator, Character quotePrefix, org.apache.commons.csv.CSVFormat format) Parses the line with char separators.
ConnectorResult	persist(TextConnectorParams params, DataSet dataSet, Driver driver) Persists data set.
ConnectorResult	populate(TextConnectorParams params, DataSet dataSet, Driver driver) Populates data set.
void	postPersist(TextConnectorParams params, DataSet dataSet, Driver driver) Called when DataSetConnector finished persisting data set.
void	prePersist(TextConnectorParams params, DataSet dataSet, Driver driver) Called when DataSetConnector starts persisting data set.
void	startStream(TextProducerPackage producerPackage) Start the stream.
boolean	supportsNested() Supports nested.
boolean	supportsReadStreaming() Supports read streaming.
boolean	supportsReordering(TextConnectorParams params) Supports reordering of the fields.
boolean	supportsWriteStreaming() Supports write streaming.
ConnectorResult	writeMetaData(TextConnectorParams params, DataSet dataSet, Driver driver) Writes meta data.

Methods inherited from class com.toolsverse.etl.connector.BaseDataSetConnector

addValue, basePostPersist, getCommitPolicy, getExt, getInputStream, getIoProcessor, getLicensePropertyName, getReader, getType, postPersistWhenEmpty, setCommitPolicy, stream, supportsWorksheets, testConnection

Methods inherited from class com.toolsverse.ext.BaseExtension

compareTo, getConfigFileName, getExtensionRank, getIconPath, getLocalUnitClassPath

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.toolsverse.etl.connector.DataSetConnector

getExt, supportsWorksheets, testConnection

Methods inherited from interface com.toolsverse.io.StreamingConnector

getCommitPolicy, setCommitPolicy

Constructor Detail

TextConnector

public TextConnector()

Method Detail

canRead

public boolean canRead()

Description copied from interface: DataSetConnector

Checks if connector can read data.

Specified by:

canRead in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

true, if connector can read data

canWrite

public boolean canWrite()

Description copied from interface: DataSetConnector

Checks if connector can write data.

Specified by:

canWrite in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

true, if connector can write data

cleanUp

public void cleanUp(TextConnectorParams params,
                    DataSet dataSet,
                    Driver driver)
             throws Exception

Description copied from interface: DataSetConnector

Cleans up after DataSetConnector populated or persisted DataSet. Called regardless of exception. Usually used to free allocated resources such as streams, handlers, etc.

Specified by:

cleanUp in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the DataSetConnectorParams

dataSet - the data set

driver - the driver

Throws:

Exception - in case of any error

getDataSetConnectorParams

public TextConnectorParams getDataSetConnectorParams()

Description copied from interface: DataSetConnector

Gets the DataSetConnectorParams.

Specified by:

getDataSetConnectorParams in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

the DataSetConnectorParams

getDisplayName

public String getDisplayName()

Description copied from interface: ExtensionModule

Gets the display name. This name will be used to display module in the various UI controls.

Specified by:

getDisplayName in interface ExtensionModule

Returns:

the display name

getLengthArray

public static int[] getLengthArray(TextConnectorParams params,
                                   int fieldCount)

Gets the array of lengths for each field in the fixed-length file.

Parameters:

params - the TextConnectorParams

fieldCount - the field count

Returns:

the length array

getName

public String getName()

Description copied from interface: DataSetConnector

Gets the name.

Specified by:

getName in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

the name

getVendor

public String getVendor()

Description copied from interface: ExtensionModule

Gets the vendor.

Specified by:

getVendor in interface ExtensionModule

Returns:

the vendor

getVersion

public String getVersion()

Description copied from interface: ExtensionModule

Gets the version of the extension module.

Specified by:

getVersion in interface ExtensionModule

Returns:

the version

getXmlConfigFileName

public String getXmlConfigFileName()

Description copied from interface: ExtensionModule

Gets the path to the xml configuration file for the module. The xml configuration file usually contains module specific UI elements, resides inside package and deployed with the code in the jar file.

Specified by:

getXmlConfigFileName in interface ExtensionModule

Overrides:

getXmlConfigFileName in class BaseExtension

Returns:

the path to the configuration name

inlinePersist

public void inlinePersist(TextConnectorParams params,
                          DataSet dataSet,
                          Driver driver,
                          DataSetRecord record,
                          int row,
                          int records)
                   throws Exception

Description copied from interface: DataSetConnector

Persists data set row. For example adds line to the text file.

Specified by:

inlinePersist in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the DataSetConnectorParams

dataSet - the data set

driver - the driver

record - the record

row - the row

records - the total number of records

Throws:

Exception - in case of any error

getStringToPersist

public String getStringToPersist(String value,
                                 boolean stripNotPrintable,
                                 boolean removeEol)

Gets the string to persist.

Parameters:

value - the value

Returns:

the string to persist

isFileConnector

public boolean isFileConnector()

Description copied from interface: DataSetConnector

Checks if this is a connector to the file based data source.

Specified by:

isFileConnector in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

true, if this is a connector to the file based data source.

isTextFormat

public boolean isTextFormat()

Description copied from interface: DataSetConnector

Checks if this is a connector for the text-based data format, such as JOSN, XML, etc.

Specified by:

isTextFormat in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

true, if this is a connector for the text-based data format.

parseLineWithCharSeparators

public String[] parseLineWithCharSeparators(String line,
                                            String separator,
                                            String charSeparator,
                                            Character quotePrefix,
                                            org.apache.commons.csv.CSVFormat format)
                                     throws Exception

Parses the line with char separators.

Parameters:

line - the line

separator - the separator

charSeparator - the char separator

Returns:

the string[]

Throws:

Exception - the exception

persist

public ConnectorResult persist(TextConnectorParams params,
                               DataSet dataSet,
                               Driver driver)
                        throws Exception

Description copied from interface: DataSetConnector

Persists data set.

Specified by:

persist in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the DataSetConnectorParams

dataSet - the data set

driver - the driver

Returns:

the ConnectorResult

Throws:

Exception - in case of any error

consume

public boolean consume(TextStreamingPackage streamingPackage,
                       Object message,
                       AtomicInteger row)
                throws Exception

Description copied from interface: StreamingConnector

Consume the next record.

Specified by:

consume in interface StreamingConnector<TextStreamingPackage,TextProducerPackage>

Parameters:

streamingPackage - the streaming package

message - the message to consume

row - the current row number

Returns:

true, if successful

Throws:

Exception - in case of any error

beforeStream

public void beforeStream(TextProducerPackage producerPackage)
                  throws Exception

Description copied from interface: StreamingConnector

Executed before record has been streamed.

Specified by:

beforeStream in interface StreamingConnector<TextStreamingPackage,TextProducerPackage>

Parameters:

producerPackage - the producer package

Throws:

Exception - in case of any error

startStream

public void startStream(TextProducerPackage producerPackage)
                 throws Exception

Description copied from interface: StreamingConnector

Start the stream.

Specified by:

startStream in interface StreamingConnector<TextStreamingPackage,TextProducerPackage>

Parameters:

producerPackage - the producer package

Throws:

Exception - the exception

endStream

public void endStream(TextProducerPackage producerPackage)
               throws Exception

Description copied from interface: StreamingConnector

Executed after record has been streamed.

Specified by:

endStream in interface StreamingConnector<TextStreamingPackage,TextProducerPackage>

Parameters:

producerPackage - the producer package

Throws:

Exception - in case of any error

closeStream

public void closeStream(TextStreamingPackage streamingPackage)

Description copied from interface: StreamingConnector

Close stream.

Specified by:

closeStream in interface StreamingConnector<TextStreamingPackage,TextProducerPackage>

Parameters:

streamingPackage - the streaming package

populate

public ConnectorResult populate(TextConnectorParams params,
                                DataSet dataSet,
                                Driver driver)
                         throws Exception

Description copied from interface: DataSetConnector

Populates data set.

Specified by:

populate in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the DataSetConnectorParams

dataSet - the data set

driver - the driver

Returns:

the ConnectorResult

Throws:

Exception - in case of any error

postPersist

public void postPersist(TextConnectorParams params,
                        DataSet dataSet,
                        Driver driver)
                 throws Exception

Description copied from interface: DataSetConnector

Called when DataSetConnector finished persisting data set. Never called if there was an exception.

Specified by:

postPersist in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the DataSetConnectorParams

dataSet - the data set

driver - the driver

Throws:

Exception - in case of any error

prePersist

public void prePersist(TextConnectorParams params,
                       DataSet dataSet,
                       Driver driver)
                throws Exception

Description copied from interface: DataSetConnector

Called when DataSetConnector starts persisting data set.

Specified by:

prePersist in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the DataSetConnectorParams

dataSet - the data set

driver - the driver

Throws:

Exception - in case of any error

writeMetaData

public ConnectorResult writeMetaData(TextConnectorParams params,
                                     DataSet dataSet,
                                     Driver driver)
                              throws Exception

Description copied from interface: DataSetConnector

Writes meta data. For example along with delimited text file the DataSetConnector can create a xml file with the same name which contains field definitions.

Specified by:

writeMetaData in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the DataSetConnectorParams

dataSet - the data set

driver - the driver

Returns:

the ConnectorResult

Throws:

Exception - in case of any error

initQueryLabel

public String initQueryLabel()

Description copied from interface: DataSetConnector

The init query label. If not empty the init query, for example XML XQuery, will be enabled and label in editor will be set to the value of this field

Specified by:

initQueryLabel in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

the string

supportsReadStreaming

public boolean supportsReadStreaming()

Description copied from interface: DataSetConnector

Supports read streaming.

Specified by:

supportsReadStreaming in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

true, if successful

supportsWriteStreaming

public boolean supportsWriteStreaming()

Description copied from interface: DataSetConnector

Supports write streaming.

Specified by:

supportsWriteStreaming in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

true, if successful

supportsNested

public boolean supportsNested()

Description copied from interface: DataSetConnector

Supports nested.

Specified by:

supportsNested in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Returns:

true, if connector supports nested data sets

supportsReordering

public boolean supportsReordering(TextConnectorParams params)

Description copied from interface: DataSetConnector

Supports reordering of the fields.

Specified by:

supportsReordering in interface DataSetConnector<TextConnectorParams,ConnectorResult>

Parameters:

params - the params

Returns:

true, if connector supports reordering of the fields