Difference between revisions of "Installing and Configuring ETL Framework"

From Toolsverse Knowledge Base
Jump to: navigation, search
 
(18 intermediate revisions by the same user not shown)
Line 19: Line 19:
 
ETL Framework comes pre-configured to provide highest levels of performance and compatibility.
 
ETL Framework comes pre-configured to provide highest levels of performance and compatibility.
  
ETL framework requires a valid not expired license. Licenses are located under  ETL_FRAMEWORK_HOME\license folder (*.lic files) or under ETL_FRAMEWORK_HOME\lib (*.jar files).
+
Framework requires a valid not expired license. Licenses are located under  ETL_FRAMEWORK_HOME\license folder (*.lic files) or under ETL_FRAMEWORK_HOME\lib (*.jar files).
  
 
'''Note''': ETL Framework comes with a 20-days evaluation license. It will expire after 20 days since the first use. You can a buy a commercial license which will never expire. License can be delivered as a lic or jar file. Using jars for licenses decreases dependency on external folders.   
 
'''Note''': ETL Framework comes with a 20-days evaluation license. It will expire after 20 days since the first use. You can a buy a commercial license which will never expire. License can be delivered as a lic or jar file. Using jars for licenses decreases dependency on external folders.   
Line 27: Line 27:
 
=== Changing configuration properties ===
 
=== Changing configuration properties ===
  
ETL framework uses etl.properties file located under ETL_FRAMEWORK_HOME/config folder to load configuration properties from. If file does not exist framework uses default values.
+
ETL framework uses etl.properties file located under ETL_FRAMEWORK_HOME/config folder to load configuration properties from. You can change default name of the configuration file by setting property config.file.name. Example: <pre>-Dconfig.file.name=c:/etl/config/test.properties.</pre>
  
The only two properties you can safely change are:
+
If file does not exist framework will use default values.
* network.appupdateurl - the URL used to call uuto-update Web service
+
 
 +
You can add any property to the file using format property_name=value. You can also completely ignore configuration file and set properties programmatically using System.setProperty(name, value) or by passing -D props to the Java program.
 +
 
 +
Some of the properties:
 +
* network.appupdateurl - the URL used to call auto-update Web service
 
* network.appserverurl - the URL used to call ETL Web service
 
* network.appserverurl - the URL used to call ETL Web service
  
Line 38: Line 42:
 
network.appserverurl=http://localhost:8080/dataexplorer/ide
 
network.appserverurl=http://localhost:8080/dataexplorer/ide
 
network.appupdateurl=http://www.toolsverse.com/api/services/CheckForUpdates
 
network.appupdateurl=http://www.toolsverse.com/api/services/CheckForUpdates
 +
# never change properties below
 
app.update.key=etl
 
app.update.key=etl
 
app.name=etlprocess
 
app.name=etlprocess
 
app.title=ETL Framework
 
app.title=ETL Framework
 +
</pre>
 +
 +
You can pass XML configuration file name (or any other -D property) as a command line argument:
 +
<pre>-Detl.config.name=import_data_config.xml</pre>
 +
 +
=== Configuring HTTP Proxy ===
 +
 +
Use your properties file (the default is ETL_FRAMEWORK_HOME/config/etl.properties) to configure HTTP proxy:
 +
 +
<pre>
 +
network.httpproxyhost=
 +
network.httpproxyport=
 +
network.httpproxyuser=
 +
network.httpproxypassword=
 +
# none, http, https, socks
 +
network.httpproxytype=
 +
# comma separated list of hosts to exclude, for example localhost,198.168.1.1. Can leave it blank
 +
network.httpproxyexclude=
 +
</pre>
 +
 +
=== Configuring EMail Notifications ===
 +
 +
If you are going to use email notifications you need to configure mailer. Use your properties file (the default is ETL_FRAMEWORK_HOME/config/etl.properties):
 +
 +
<pre>
 +
# example: smtp.mail.host=smtp.gmail.com                   
 +
smtp.mail.host=
 +
# example: smtp.mail.port=587
 +
smtp.mail.port=
 +
# example: mailer@gmail.com
 +
smtp.mail.user=
 +
# example: passwd
 +
smtp.mail.password=
 +
# example: ops@gmail.com
 +
smtp.mail.to=
 +
# example: robot@gmail.com
 +
smtp.mail.from=
 
</pre>
 
</pre>
  
Line 74: Line 116:
 
Alternatively, since JDBC drivers are just jar files you can simply put them in the ETL_FRAMEWORK_HOME\lib folder and they will be automatically loaded and added to the classpath.
 
Alternatively, since JDBC drivers are just jar files you can simply put them in the ETL_FRAMEWORK_HOME\lib folder and they will be automatically loaded and added to the classpath.
  
== Redistribution ==
+
== Redistribution and managing dependencies ==
 
+
Configure ETL framework just the way you like it and exclude all unneeded components. You can find redistribution list and dependencies [[Redistribution and Dependencies|here]].
+
  
Most jars under '''lib''' are required unless you don't need a particular functionality, for example reading and writing Excel files. All jars under '''lib''' folder are automatically added to the classpath on the start-up. Jars under '''plugin''' folder are optional and loaded based on rules and available licenses.
+
Configure ETL framework the way you like, exclude all unneeded components. You can find redistribution list and recommendations how to manage dependencies [[Redistribution and managing dependencies|here]].
  
 
== Checking for updates ==
 
== Checking for updates ==
Line 110: Line 150:
  
 
# Download update
 
# Download update
# Replace exiting files on the files from update
+
# Replace exiting files on files from the update

Latest revision as of 13:40, 9 February 2015

Downloading

Look for clearly marked ETL Framework downloads. On Windows and OS X ETL Framework can be downloaded as an installer. Use archive downloads for other operating systems.

Installing

There are two ways to install ETL Framework: using installer or extracting files from the archive.

When using installer just execute it and follow instructions on the screen.

OS X installer is offered in a form of self-extracting DMG file. When prompted drag and drop ETL Framework folder to the Applications.

When installing from archive you will need to manually extract files using utility provided by operating system. All files are contained in the enclosing folder named etl-version.

Installing from archive will not add any entries to the Start menu, add desktop launchers or register the software in the registry.

Configuring

ETL Framework comes pre-configured to provide highest levels of performance and compatibility.

Framework requires a valid not expired license. Licenses are located under ETL_FRAMEWORK_HOME\license folder (*.lic files) or under ETL_FRAMEWORK_HOME\lib (*.jar files).

Note: ETL Framework comes with a 20-days evaluation license. It will expire after 20 days since the first use. You can a buy a commercial license which will never expire. License can be delivered as a lic or jar file. Using jars for licenses decreases dependency on external folders.

You can read more about licensing here.

Changing configuration properties

ETL framework uses etl.properties file located under ETL_FRAMEWORK_HOME/config folder to load configuration properties from. You can change default name of the configuration file by setting property config.file.name. Example:
-Dconfig.file.name=c:/etl/config/test.properties.

If file does not exist framework will use default values.

You can add any property to the file using format property_name=value. You can also completely ignore configuration file and set properties programmatically using System.setProperty(name, value) or by passing -D props to the Java program.

Some of the properties:

  • network.appupdateurl - the URL used to call auto-update Web service
  • network.appserverurl - the URL used to call ETL Web service

Example:

network.appserverurl=http://localhost:8080/dataexplorer/ide
network.appupdateurl=http://www.toolsverse.com/api/services/CheckForUpdates
# never change properties below
app.update.key=etl
app.name=etlprocess
app.title=ETL Framework

You can pass XML configuration file name (or any other -D property) as a command line argument:

-Detl.config.name=import_data_config.xml

Configuring HTTP Proxy

Use your properties file (the default is ETL_FRAMEWORK_HOME/config/etl.properties) to configure HTTP proxy:

network.httpproxyhost=
network.httpproxyport=
network.httpproxyuser=
network.httpproxypassword=
# none, http, https, socks
network.httpproxytype=
# comma separated list of hosts to exclude, for example localhost,198.168.1.1. Can leave it blank 
network.httpproxyexclude=

Configuring EMail Notifications

If you are going to use email notifications you need to configure mailer. Use your properties file (the default is ETL_FRAMEWORK_HOME/config/etl.properties):

# example: smtp.mail.host=smtp.gmail.com                    
smtp.mail.host=
# example: smtp.mail.port=587 
smtp.mail.port=
# example: mailer@gmail.com
smtp.mail.user=
# example: passwd
smtp.mail.password=
# example: ops@gmail.com
smtp.mail.to=
# example: robot@gmail.com
smtp.mail.from=

Changing JVM Parameters

  1. Open ETL_FRAMEWORK_HOME\etlappstart.properties in your favorite text editor
  2. Change line containing app.vm.options
  3. Save

Changing Logging Parameters

  1. Open ETL_FRAMEWORK_HOME\config\log4j.properties in your favorite text editor
  2. Make a change
  3. Save

Adding JDBC drivers

Each copy of the ETL Framework comes with the following JDBC drivers:

  • DB2
  • Informix
  • Microsoft SQL Server
  • Sybase Advanced Server
  • MySQL
  • Oracle
  • PostgreSQL
  • Java DB (Derby)

They are located in the ETL_FRAMEWORK_HOME\jdbc folder. ETL Framework automatically recognizes (on start-up) and dynamically loads (when needed) jdbc drivers under this folder.

To install new jdbc driver just add a sub folder under ETL_FRAMEWORK_HOME\jdbc and put all required files there. Restart Data Explorer.

Alternatively, since JDBC drivers are just jar files you can simply put them in the ETL_FRAMEWORK_HOME\lib folder and they will be automatically loaded and added to the classpath.

Redistribution and managing dependencies

Configure ETL framework the way you like, exclude all unneeded components. You can find redistribution list and recommendations how to manage dependencies here.

Checking for updates

ETL Framework can check for new version and update itself if new version exists.

To check for new new version run ETL framework executable with -v parameter.

Example:

C:\projects\toolsverse\java\dist\etl>etl.exe -v
Toolsverse Etl Framework 4.0-19862. Use -? for help.

Checking for updates...
Update status: new version available

To auto update ETL framework run executable with -u parameter.

Example:

C:\projects\toolsverse\java\dist\etl>etl.exe -u
Toolsverse Etl Framework 4.0-19862. Use -? for help.

Manually updating

You can update framework to the new version manually:

  1. Download update
  2. Replace exiting files on files from the update