Writing data integration and data migration scenarios - Revision history

Maxlmus at 18:38, 29 October 2014

2014-10-29T18:38:28Z

Maxlmus at 23:27, 21 August 2014

2014-08-21T23:27:22Z

Maxlmus at 20:29, 4 August 2014

2014-08-04T20:29:17Z

Maxlmus at 20:27, 4 August 2014

2014-08-04T20:27:58Z

Maxlmus at 20:25, 4 August 2014

2014-08-04T20:25:46Z

Maxlmus at 20:23, 4 August 2014

2014-08-04T20:23:40Z

Maxlmus at 21:09, 3 August 2014

2014-08-03T21:09:18Z

Maxlmus at 19:06, 3 August 2014

2014-08-03T19:06:09Z

Maxlmus: /* Using database specific features */

2014-07-27T22:50:33Z

‎Using database specific features

Maxlmus at 22:47, 27 July 2014

2014-07-27T22:47:53Z

@@ Line 374: / Line 374: @@
 == Things to do ==
-Typically a full blown data integration process includes at least extract and load. When everything said and done data from the source are moved to the destination. In some cases it makes sense to split process in steps which can performed in the different time frames. For example you can:
+Typically a full blown data integration process includes at least extract and load. When everything said and done data from the source are moved to the destination. In some cases it makes sense to split process in steps which can be performed in the different time frames. For example you can:
 # Extract data and store them in some intermediate format, for example XML.
 # Load data sometime later.

@@ Line 1: / Line 1: @@
 == Basic stuff ==
-Any data integration or data migration process can be described as an extract-transform-load (ETL) or extract-load-transform (ELT). With that is mind writing scenario becomes a
+Any data integration or data migration process can be described as an extract-transform-load (ETL) or extract-load-transform (ELT). With that in mind, writing scenarios becomes a process of splitting task into extracts, loads and transformations. A scenario glues them all together and adds purpose and logic.
-process of splitting task on extracts, loads and transformations. Scenario glues them all together and adds purpose and logic.
-[[http://www.toolsverse.com/products/etl-framework/ ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML is a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said, you can chose to stream data so extract and load are combined in one logical operation and run things in parallel where actual order of extracts and loads is not guarantee.
+[[http://www.toolsverse.com/products/etl-framework/ The ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML is a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said, you can chose to stream data, so extract and load are combined in one logical operation or run them in parallel so the actual order of extracts and loads is not guaranteed.
-ETL engine makes it possible to concentrate on task in hands without reinventing the wheel. It hides complexity of data integration so in most cases the same techniques can be applied when working with any SQL and Non SQL data sources. It is however possible to use a full power of the target database and do things like direct data loads, parallel extracts, etc.
+The ETL engine makes it possible to concentrate on a task at hand without reinventing the wheel. It hides the complexity of data integration so in most cases the same techniques can be applied when working with any SQL and Non-SQL data sources. It is possible to use a full power of the target database and do things like direct data loads, parallel extracts, etc.
 Please take a look at the [http://www.toolsverse.com/products/etl-framework/examples ETL scenario examples].

@@ Line 5: / Line 5: @@
 [[http://www.toolsverse.com/products/etl-framework/ ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML is a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said, you can chose to stream data so extract and load are combined in one logical operation and run things in parallel where actual order of extracts and loads is not guarantee.
-ETL engine makes it possible to concentrate on task in hands without reinventing the wheel. It hides complexity of data integration so in most cases the same techniques and language artifacts can be applied when working with any SQL and Non SQL data sources. It is however possible to use a full power of the target database and do things like direct data loads, parallel extracts, etc.
+ETL engine makes it possible to concentrate on task in hands without reinventing the wheel. It hides complexity of data integration so in most cases the same techniques can be applied when working with any SQL and Non SQL data sources. It is however possible to use a full power of the target database and do things like direct data loads, parallel extracts, etc.
 Please take a look at the [http://www.toolsverse.com/products/etl-framework/examples ETL scenario examples].

@@ Line 5: / Line 5: @@
 [[http://www.toolsverse.com/products/etl-framework/ ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML is a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said, you can chose to stream data so extract and load are combined in one logical operation and run things in parallel where actual order of extracts and loads is not guarantee.
-ETL engine makes it possible to concentrate on task in hands without reinventing the wheel. It hides complexity of data integration so in most cases the same techniques and language artifacts are used to work with any SQL and Non SQL data sources. It is however possible to use a full power of the target database and do things like direct data loads, parallel extracts, etc.
+ETL engine makes it possible to concentrate on task in hands without reinventing the wheel. It hides complexity of data integration so in most cases the same techniques and language artifacts can be applied when working with any SQL and Non SQL data sources. It is however possible to use a full power of the target database and do things like direct data loads, parallel extracts, etc.
 Please take a look at the [http://www.toolsverse.com/products/etl-framework/examples ETL scenario examples].

@@ Line 3: / Line 3: @@
 process of splitting task on extracts, loads and transformations. Scenario glues them all together and adds purpose and logic.
-[[http://www.toolsverse.com/products/etl-framework/ ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML is a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said you can chose to stream data so extract and load are combined in one logical operation and run things in parallel where actual order of extracts and loads is not guarantee.
+[[http://www.toolsverse.com/products/etl-framework/ ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML is a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said, you can chose to stream data so extract and load are combined in one logical operation and run things in parallel where actual order of extracts and loads is not guarantee.
 ETL engine makes it possible to concentrate on task in hands without reinventing the wheel. It hides complexity of data integration so in most cases the same techniques and language artifacts are used to work with any SQL and Non SQL data sources. It is however possible to use a full power of the target database and do things like direct data loads, parallel extracts, etc.

@@ Line 3: / Line 3: @@
 process of splitting task on extracts, loads and transformations. Scenario glues them all together and adds purpose and logic.
-[[http://www.toolsverse.com/products/etl-framework/ ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML was a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said you can chose to stream data so extract and load combined in one logical operation and run things in parallel where actual order of extracts and loads is not guarantee.
+[[http://www.toolsverse.com/products/etl-framework/ ETL engine]] which powers Toolsverse products uses XML-based language to create scenarios. Before reading this page please take a look at [[ETL scenario language]] specification. XML was a natural choice because it enforces a structure: loads follow extracts and transformations follow extracts and loads. That said you can chose to stream data so extract and load are combined in one logical operation and run things in parallel where actual order of extracts and loads is not guarantee.
 ETL engine makes it possible to concentrate on task in hands without reinventing the wheel. It hides complexity of data integration so in most cases the same techniques and language artifacts are used to work with any SQL and Non SQL data sources. It is however possible to use the full power of the target database and do things like direct data load, anonymous SQL blocks, etc.
 Please take a look at the [http://www.toolsverse.com/products/etl-framework/examples ETL scenario examples].
 == Simple data migration scenario ==

@@ Line 292: / Line 292: @@
 == Using database specific features ==
 <syntaxhighlight lang="xml">
 <?xml version="1.0" encoding="UTF-8"?>
 <scenario>
-      <name>Oracle Extract using slplus and Load using sql*loader</name>
+      <name>Oracle Extract using sql*plus and Load using sql*loader</name>
       <script>oracle_sql_loader</script>
       <driver>com.toolsverse.etl.driver.oracle.OracleDriver</driver>

@@ Line 286: / Line 286: @@
                           filename="{app.root.data}\demo\matrix-output.xlsx" sheetname="output"/>
                 </load>
            </destination>
       </destinations>
@@ Line 323: / Line 404: @@
                      "test_etl_config.xml", etlProcess);
 </syntaxhighlight>