Running ETL Scenarios

Embedding ETL scenario into existing Java application

There are three easy steps:

1. Instantiate EtlConfig
2. Instantiate EtlProcess
3. Execute scenario

public static void main(String[] args)
{
        LoadScenarioAndConfigurationFromFile engine = new LoadScenarioAndConfigurationFromFile();
 
        try
        {
            // instantiates ETL configuration
            EtlConfig etlConfig = new EtlConfig();
 
            // set log level to INFO which increases verbosity of the etl engine
            Logger.setLevel(EtlLogger.class, Logger.INFO);
 
            // creates embedded ETL process
            EtlProcess etlProcess = new EtlProcess(EtlProcess.EtlMode.EMBEDDED);
 
            // print out framework version
            System.out.println(SystemConfig.instance().getTitle(
                    EtlConfig.DEFAULT_TITLE)
                    + " "
                    + SystemConfig.instance().getSystemProperty(
                            SystemConfig.VERSION));
 
            // load configuration which contains source and destination
            // connections and ETL scenario name. Load and execute ETL scenario.
            // If no full path
            // provided the configuration file test_etl_config.xml is expected
            // to be under app_home/config
            EtlResponse response = engine.loadConfigAndExecute(etlConfig,
                    "test_etl_config.xml", etlProcess);
 
            // print out formatted output from the ETL response
            System.out.println(engine.getMessage(response,
                    "Exampes/Engine/db2file.xml"));
 
        }
        catch (Exception ex)
        {
            System.out.println(Utils.getStackTraceAsString(ex));
        }
 
        System.exit(0);
}

Look at other examples of embedding ETL engine into Java application.

Running ETL scenario using standalone executable

1. Open APP_HOME/config/etl_config.xml file in your favorite text editor.
2. Add connections for the particular ETL scenario
3. Specify connections to use and scenarios to run
4. Save
5. Run ETL executable. For example c:\etl\etl.exe on Windows
6. When it is finished check the etl.log file located under APP_HOME/logs

Example of the etl_config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <connections>
      <connection alias="test javadb">
         <driver>org.apache.derby.jdbc.EmbeddedDriver</driver>
         <url>jdbc:derby:{app.root.data}/demo/javadb</url>
      </connection>
      <connection alias="test oracle">
         <driver>oracle.jdbc.driver.OracleDriver </driver>
         <url>jdbc:oracle:thin:@localhost:1521:orcl1</url>
         <userid>user</userid>
         <password>password</password>   
         <params/>
      </connection>
   </connections>
 
   <active.connections>
      <sourses>
         <source alias="test javadb" />
      </sourses>
      <destination alias="test oracle"/> 
   </active.connections>
   <execute>
       <scenario name="move.xml" action="extract_load" />
   </execute>
</config>

In this example move.xml ETL scenario located under the {app.data}/scenario folder will be executed using extract_load action. Alias test javadb will be used for the source connection and alias test oracle for the destination.

Creating and running ETL scenario using Data Explorer

The best way to create, run and schedule ETL, data integration and data migration scenario is using Data Explorer - an integrated ETL IDE.