Showing posts with label Other. Show all posts
Showing posts with label Other. Show all posts

Monday, June 24, 2013

Datastage scheduler Vs Third party scheduler

Datastage scheduler:

1)By using Datastage Director we scan schedule only the Datastage Jobs

2)The main drawback of Datastage scheduler is not possible to schedule other components like unix scripts,sql scripts,Informatica Jobs etc.In any  project dependency exists in all these components.In such scenarios you require an explicit scheduling tool

Third party scheduler:

1)TWS, Autosys, Control-M are etc popular schedulers available in market


2)By using Third party schedular we can schedule unix scripts,sql scripts,Informatica Jobs etc all the components.Assume a scenario when your project executes multiple type of components on different machines including UNIX machine, Windows Machine etc.Each schedular has  own features like e-mail notification which is not present in Datastage schedular

Wednesday, April 17, 2013

Difference between server jobs and parallel jobs?


Server jobs:-
1)In server jobs it handles less volume of data.
2)It is having less number of components.
3)Data processing will be slow.
4)Executed by Datastage serve engine
5)compiled into Basic
6)No parallel Capability  is one of the  drawback of Server jobs

Parallel jobs:-
1)It handles high volume of data.
2)Executed by Datastage parallel engine
3)It is having more number of components compared to server jobs.
4)Supports  pipeline and partition parallelsim
5)Compiled into OSH

Similarity Between Server jobs and Parallel jobs:

Runtime monitoring in DataStage Director

Friday, April 12, 2013

When to use which stage in Datastage?

Copy STAGE-To drop a Particular column

Sort STAGE-sorting,generating Key change and similar to order by clause in oracle

Filter STAGE-Similar to where clause in oracle but we can not perform Join operation

Lookup,Join,Merge-To perform Join operation

Pivot Enterprise STAGE-Rows to columns and columns to Rows

External Filter STAGE-Filter the records by using Unix filter commands like Grep etc

MODIFY STAGE-Metadata conversion,Null Handling and similar to conversion functions in oracle

FUNNEL STAGE -Combining the multiple input data into a single output.Metadata should be same for all the inputs

REMOVE DUPLICATES STAGE-To  remove  duplicate  values  from  a single sorted  input.

ENCODE / DECODE STAGES:To encode/compress a data set using UNIX encoding  commands like gzip etc

TRANSFORMER STAGE:
a)Filtering the Data(constraints)
b)Metadata conversion(Using Functions)
c)Rows to columns and columns to Rows(Using Stage variables)
d)Looping
e)Creating a counter(Using macros)-Counter using Transformer

SURROGATE KEY GENERATOR STAGE-To generate SURROGATE KEYs similar to oracle Database sequence

Aggregator Stage:To perform Group by Operations like max,min etc similar to Group by clause in oracle

ROW GENERATOR STAGE:To generate a set of mock data fitting the specified metadata when no real data is available

XML OUTPUT STAGE -To convert tabular data such as tables and sequential files to XML hierarchical structures.

SWITCH STAGE-  It performs an operation similar to  the  switch  statement  in  C and to filter the data

CHANGE CAPTURE STAGE-To identify Delta changes(inserts,updates,deletes etc) between two sources

oracle connector-To connect to the oracle Database.