Difference between revisions of "Redistribution and managing dependencies"

From Toolsverse Knowledge Base
Jump to: navigation, search
 
(38 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
== ETL Framework Redistribution List ==
+
=== Redistribution list ===
 +
 
 +
__FORCETOC__
  
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
! Folder !! File !! Required !! Needed for
+
! Folder !! File !! Required !! Needed for || Depends on
 
|-
 
|-
−
| '''home''' || || '''Yes''' ||  
+
| '''home''' || || '''Yes''' || Root folder ||  
  
 
|-
 
|-
−
| || etl.exe || No || Windows executable  
+
| || etl.exe || No || Windows executable ||
 
|-
 
|-
−
| || etl.app || No || Mac application bundle  
+
| || etl.app || No || Mac application bundle ||
 
|-
 
|-
−
| || etl.sh || No || Unix/Linux shell script
+
| || etl.sh || No || Unix/Linux shell script ||
 
|-
 
|-
−
| || etlappstart.properties || No || JVM properties
+
| || etlappstart.properties || No || JVM properties ||
 
|-
 
|-
−
| || etl.jar || '''Yes''' || Main executable jar
+
| || etl.jar || '''Yes''' || Main executable jar ||
  
 
|-
 
|-
−
| '''home\config''' || || No ||  
+
| '''home\config''' || || No || Default configuration folder ||
 +
 
 +
|-
 +
| || etl.properties || No || Configuration properties ||
 +
|-
 +
| || etl_config.xml || No || Default [[Configuration_file|ETL configuration file]]  ||
 
|-
 
|-
−
| '''home\data''' || || No ||  
+
| || log4j.properties || No || Log configuration  ||
  
 +
|-
 +
| '''home\license''' || || No || Folder for licenses ||
 +
|-
 +
| || *.lic || No || Licenses (ETL framework will also look at license.jar under home\lib folder) ||
  
  
 
|-
 
|-
−
| || etl.properties || No || Configuration properties
+
| '''home\data''' || || No || Default folder for data ||
 +
 
 
|-
 
|-
−
| || etl_config.xml || No || Default [[Configuration_file|ETL configuration file]]  
+
| '''home\data\scenario''' || || No || Default folder for ETL scenarios ||
 +
|-
 +
| '''home\data\schema''' || || No || Default folder for XLT schema files ||
 +
|-
 +
| || dataset2webrowset.xsl || No || XSL schema file for Toolseverse XML dataset to [http://docs.oracle.com/javase/7/docs/api/javax/sql/rowset/WebRowSet.html WebRowSet] XML transformation || 
 +
|-
 +
| || webrowset2dataset.xsl || No || XSL schema file for [http://docs.oracle.com/javase/7/docs/api/javax/sql/rowset/WebRowSet.html WebRowSet] XML to Toolseverse XML dataset transformation || 
 +
|-
 +
| || xmldataset.xsd || No || Schema file for Toolsverse XML dataset file format || 
 +
 
 +
|-
 +
| '''home\lib''' || || No || Folder for core and third party jar files. Files from this folder automatically added to the classpath ||
 +
 
 +
|-
 +
| || antlr-3.0.1.jar || '''Yes'''|| Core third party file ||
 +
 
 +
|-
 +
| || antlr.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-beanutils-1.9.1.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-codec-1.6.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-collections4-4.0.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-dbcp-1.4.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-logging-1.1.3.jar  || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-math3-3.0.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-net-2.0.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || commons-pool-1.5.4.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || dom4j-1.6.1.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || httpclient-4.3.3.jar  || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || httpclient-cache-4.3.3.jar  || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || httpcore-4.3.2.jar  || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || httpmime-4.3.3.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || javassist.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || json-io.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || log4j-1.2.16.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || rsyntaxtextarea.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || RSyntaxTextArea.License.txt || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || saxon9he.jar || '''Yes''' || Core third party file ||
 +
 
 +
|-
 +
| || toolsverse-core.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-etl-common.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-etl-connector-sql.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-etl-connector.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-etl-core.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-etl-driver.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-etl-metadata.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-io.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-license.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-mvc.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-security.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-service.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-storage.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
|-
 +
| || toolsverse-updater.jar || '''Yes''' || Toolsverse core file ||
 +
 
 +
 
 +
|-
 +
| || derby.jar || No || Derby DB JDBC driver ||
 +
 
 +
|-
 +
| || iText-4.2.0-com.itextpdf.jar || No || Read and write PDF files ||
 +
 
 +
|-
 +
| || javax.json-1.0.4.jar || No || Read and write JSON files ||
 +
 
 +
|-
 +
| || javax.mail.jar  || No || Send emails ||
 +
 
 +
|-
 +
| || jsch-0.1.43.jar  || No || SFTP client ||
 +
 
 +
|-
 +
| || poi-3.8-20120326.jar || No || Read and write Excel files ||
 +
 
 +
|-
 +
| || poi-ooxml-3.8-20120326.jar || No || Read and write Excel files ||
 +
 
 +
|-
 +
| || poi-ooxml-schemas-3.8-20120326.jar || No || Read and write Excel files ||
 +
 
 +
|-
 +
| || servlet-api.jar || No || Required if you are going to call Web services  ||
 +
 
 +
|-
 +
| || socks.jar || No || Socks proxy used together with FTP  ||
 +
 
 +
|-
 +
| '''home\plugin''' || || No || Folder for plugins. In other to use files in this folder you must have a valid not expired license ||
 +
 
 +
|-
 +
| || toolsverse-etl-connector-excel.jar || No || Required if you are going to read and write Excel files || toolsverse-etl-connector-json.jar
 +
 
 +
|-
 +
| || toolsverse-etl-connector-html.jar || No || Required if you are going to read and write HTML files ||
 +
 
 +
|-
 +
| || toolsverse-etl-connector-json.jar || No || Required if you are going to read and write JSON files ||
 +
 
 +
|-
 +
| || toolsverse-etl-connector-pdf.jar || No || Required if you are going to read and write PDF files ||
 +
 
 +
|-
 +
| || toolsverse-etl-connector-text.jar || No || Required if you are going to read and write CVS and fixed-length files || toolsverse-etl-connector-json.jar, toolsverse-etl-connector-xml.jar
 +
 
 +
|-
 +
| || toolsverse-etl-connector-xml.jar || No || Required if you are going to read and write XML files || toolsverse-etl-connector-json.jar
 +
 
 +
|-
 +
| || toolsverse-etl-db2.jar || No || Required if you are going to work with DB2 databases ||
 +
 
 +
|-
 +
| || toolsverse-etl-informix.jar || No || Required if you are going to work with Informix databases ||
 +
 
 +
|-
 +
| || toolsverse-etl-oracle.jar || No || Required if you are going to work with Oracle databases ||
 +
 
 +
|-
 +
| || toolsverse-etl-mysql.jar || No || Required if you are going to work with MySQL databases ||
 +
 
 +
|-
 +
| || toolsverse-etl-postgres.jar || No || Required if you are going to work with Postgres databases ||
 +
 
 +
|-
 +
| || toolsverse-etl-sybase.jar || No || Required if you are going to work with Sybase ASE databases ||
 +
 
 +
|-
 +
| '''home\jdbc''' || || No || Default folder for [[Installing_and_Configuring_ETL_Framework|JDBC drivers]] ||
 +
 
 +
 
 
|}
 
|}
  
−
Example:
+
=== Example ===
  
 
<pre>
 
<pre>
Line 99: Line 302:
 
       toolsverse-storage.jar             
 
       toolsverse-storage.jar             
 
       toolsverse-updater.jar             
 
       toolsverse-updater.jar             
−
      xercesImpl.jar                   
 
 
       xmlbeans-2.3.0.jar                 
 
       xmlbeans-2.3.0.jar                 
 
   plugin
 
   plugin
Line 115: Line 317:
 
       toolsverse-etl-sqlserver.jar
 
       toolsverse-etl-sqlserver.jar
 
</pre>
 
</pre>
 +
 +
=== Managing dependencies, files and folders ===
 +
 +
Files and folders marked as '''required''' must exist in order for framework to work. Others are optional.
 +
 +
Framework is tested with the particular versions of the third party libraries (see redistribution list). It is likely framework will work with more recent versions than with older versions.
 +
 +
Jars under '''lib''' and '''plugin''' folders are automatically added to the classpath on start-up if there is an available license. Only commercial components of the ETL framework require license.
 +
 +
You can reduce number of folders ETL framework depends on:
 +
* Move all files from the '''plugin''' folder to '''lib'''.
 +
* Package existing licenses (*.lic files) in the '''license''' folder into jar file and copy jar into '''lib''' folder. Name of the jar file doesn't matter but lic files must be located in the '''com/toolsverse/license''' folder inside jar.
 +
* Files in the '''config''' folder and '''config''' folder itself are not required. You can pass properties to the framework using -Dproperty_name=value.
 +
 +
Framework uses '''HOME/data/''' folder for temporary files, scripts and data files. If folder does not exist it is automatically created. 
 +
 +
By default framework is looking for ETL scenarios in the '''HOME/data/scenario folder'''. If you want to change base folder for scenarios use -D property '''scenario.path'''.
 +
 +
Look at the src/com/toolsverse/config/SystemConfig.java how ETL frameworks uses system folders.
 +
 +
Look at the src/com/toolsverse/etl/core/config/EtlConfig.java how ETL frameworks uses ETL specific folders.

Latest revision as of 23:47, 8 February 2015

Redistribution list

Folder File Required Needed for Depends on
home Yes Root folder
etl.exe No Windows executable
etl.app No Mac application bundle
etl.sh No Unix/Linux shell script
etlappstart.properties No JVM properties
etl.jar Yes Main executable jar
home\config No Default configuration folder
etl.properties No Configuration properties
etl_config.xml No Default ETL configuration file
log4j.properties No Log configuration
home\license No Folder for licenses
*.lic No Licenses (ETL framework will also look at license.jar under home\lib folder)


home\data No Default folder for data
home\data\scenario No Default folder for ETL scenarios
home\data\schema No Default folder for XLT schema files
dataset2webrowset.xsl No XSL schema file for Toolseverse XML dataset to WebRowSet XML transformation
webrowset2dataset.xsl No XSL schema file for WebRowSet XML to Toolseverse XML dataset transformation
xmldataset.xsd No Schema file for Toolsverse XML dataset file format
home\lib No Folder for core and third party jar files. Files from this folder automatically added to the classpath
antlr-3.0.1.jar Yes Core third party file
antlr.jar Yes Core third party file
commons-beanutils-1.9.1.jar Yes Core third party file
commons-codec-1.6.jar Yes Core third party file
commons-collections4-4.0.jar Yes Core third party file
commons-dbcp-1.4.jar Yes Core third party file
commons-logging-1.1.3.jar Yes Core third party file
commons-math3-3.0.jar Yes Core third party file
commons-net-2.0.jar Yes Core third party file
commons-pool-1.5.4.jar Yes Core third party file
dom4j-1.6.1.jar Yes Core third party file
httpclient-4.3.3.jar Yes Core third party file
httpclient-cache-4.3.3.jar Yes Core third party file
httpcore-4.3.2.jar Yes Core third party file
httpmime-4.3.3.jar Yes Core third party file
javassist.jar Yes Core third party file
json-io.jar Yes Core third party file
log4j-1.2.16.jar Yes Core third party file
rsyntaxtextarea.jar Yes Core third party file
RSyntaxTextArea.License.txt Yes Core third party file
saxon9he.jar Yes Core third party file
toolsverse-core.jar Yes Toolsverse core file
toolsverse-etl-common.jar Yes Toolsverse core file
toolsverse-etl-connector-sql.jar Yes Toolsverse core file
toolsverse-etl-connector.jar Yes Toolsverse core file
toolsverse-etl-core.jar Yes Toolsverse core file
toolsverse-etl-driver.jar Yes Toolsverse core file
toolsverse-etl-metadata.jar Yes Toolsverse core file
toolsverse-io.jar Yes Toolsverse core file
toolsverse-license.jar Yes Toolsverse core file
toolsverse-mvc.jar Yes Toolsverse core file
toolsverse-security.jar Yes Toolsverse core file
toolsverse-service.jar Yes Toolsverse core file
toolsverse-storage.jar Yes Toolsverse core file
toolsverse-updater.jar Yes Toolsverse core file


derby.jar No Derby DB JDBC driver
iText-4.2.0-com.itextpdf.jar No Read and write PDF files
javax.json-1.0.4.jar No Read and write JSON files
javax.mail.jar No Send emails
jsch-0.1.43.jar No SFTP client
poi-3.8-20120326.jar No Read and write Excel files
poi-ooxml-3.8-20120326.jar No Read and write Excel files
poi-ooxml-schemas-3.8-20120326.jar No Read and write Excel files
servlet-api.jar No Required if you are going to call Web services
socks.jar No Socks proxy used together with FTP
home\plugin No Folder for plugins. In other to use files in this folder you must have a valid not expired license
toolsverse-etl-connector-excel.jar No Required if you are going to read and write Excel files toolsverse-etl-connector-json.jar
toolsverse-etl-connector-html.jar No Required if you are going to read and write HTML files
toolsverse-etl-connector-json.jar No Required if you are going to read and write JSON files
toolsverse-etl-connector-pdf.jar No Required if you are going to read and write PDF files
toolsverse-etl-connector-text.jar No Required if you are going to read and write CVS and fixed-length files toolsverse-etl-connector-json.jar, toolsverse-etl-connector-xml.jar
toolsverse-etl-connector-xml.jar No Required if you are going to read and write XML files toolsverse-etl-connector-json.jar
toolsverse-etl-db2.jar No Required if you are going to work with DB2 databases
toolsverse-etl-informix.jar No Required if you are going to work with Informix databases
toolsverse-etl-oracle.jar No Required if you are going to work with Oracle databases
toolsverse-etl-mysql.jar No Required if you are going to work with MySQL databases
toolsverse-etl-postgres.jar No Required if you are going to work with Postgres databases
toolsverse-etl-sybase.jar No Required if you are going to work with Sybase ASE databases
home\jdbc No Default folder for JDBC drivers


Example

home
   etl.exe
   etl.app
   etl.jar
   etl.sh
   etlappstart.properties 
   config
      etl.properties
      etl_config.xml
      log4j.properties
   license
      *.lic
   data
      scenario
      schema
         dataset2webrowset.xsl
         webrowset2dataset.xsl
         xmldataset.xsd
   lib	
      antlr-3.0.1.jar                   
      antlr.jar                         
      commons-beanutils-1.9.1.jar          
      commons-codec-1.6.jar                
      commons-collections4-4.0.jar         
      commons-dbcp-1.4.jar                 
      commons-logging-1.1.3.jar            
      commons-math3-3.0.jar                
      commons-net-2.0.jar                  	
      commons-pool-1.5.4.jar            
      derby.jar                         
      dom4j-1.6.1.jar                   
      httpclient-4.3.3.jar              
      httpclient-cache-4.3.3.jar        
      httpcore-4.3.2.jar                
      httpmime-4.3.3.jar                
      iText-4.2.0-com.itextpdf.jar      
      javassist.jar                     
      javax.json-1.0.4.jar              
      javax.mail.jar                    
      jsch-0.1.43.jar                   
      json-io.jar                       
      log4j-1.2.16.jar                  
      poi-3.8-20120326.jar              
      poi-ooxml-3.8-20120326.jar        
      poi-ooxml-schemas-3.8-20120326.jar
      rsyntaxtextarea.jar               
      RSyntaxTextArea.License.txt       
      saxon9he.jar                      
      servlet-api.jar                   
      socks.jar                         
      toolsverse-core.jar               
      toolsverse-etl-aliasreader.jar    
      toolsverse-etl-common.jar         
      toolsverse-etl-connector-sql.jar  
      toolsverse-etl-connector.jar      
      toolsverse-etl-core.jar           
      toolsverse-etl-driver.jar         
      toolsverse-etl-metadata.jar       
      toolsverse-io.jar                 
      toolsverse-license.jar            
      toolsverse-mvc.jar                
      toolsverse-security.jar           
      toolsverse-service.jar            
      toolsverse-storage.jar            
      toolsverse-updater.jar            
      xmlbeans-2.3.0.jar                
   plugin
      toolsverse-etl-connector-excel.jar
      toolsverse-etl-connector-html.jar 
      toolsverse-etl-connector-json.jar 
      toolsverse-etl-connector-pdf.jar  
      toolsverse-etl-connector-text.jar 
      toolsverse-etl-connector-xml.jar  
      toolsverse-etl-db2.jar            
      toolsverse-etl-informix.jar       
      toolsverse-etl-mysql.jar          
      toolsverse-etl-oracle.jar         
      toolsverse-etl-postgres.jar       
      toolsverse-etl-sqlserver.jar

Managing dependencies, files and folders

Files and folders marked as required must exist in order for framework to work. Others are optional.

Framework is tested with the particular versions of the third party libraries (see redistribution list). It is likely framework will work with more recent versions than with older versions.

Jars under lib and plugin folders are automatically added to the classpath on start-up if there is an available license. Only commercial components of the ETL framework require license.

You can reduce number of folders ETL framework depends on:

  • Move all files from the plugin folder to lib.
  • Package existing licenses (*.lic files) in the license folder into jar file and copy jar into lib folder. Name of the jar file doesn't matter but lic files must be located in the com/toolsverse/license folder inside jar.
  • Files in the config folder and config folder itself are not required. You can pass properties to the framework using -Dproperty_name=value.

Framework uses HOME/data/ folder for temporary files, scripts and data files. If folder does not exist it is automatically created.

By default framework is looking for ETL scenarios in the HOME/data/scenario folder. If you want to change base folder for scenarios use -D property scenario.path.

Look at the src/com/toolsverse/config/SystemConfig.java how ETL frameworks uses system folders.

Look at the src/com/toolsverse/etl/core/config/EtlConfig.java how ETL frameworks uses ETL specific folders.

Retrieved from "https://www.toolsverse.com/docs/index.php?title=Redistribution_and_managing_dependencies&oldid=306"