Infopackages

These are used to fetch data from ECC to BW. There are different update modes which can be used when fetching data from the source system.

Above is the picture of the various tabs of an infopackage.

• The data selection tab is the place where we can add filters when extracting data from the source. The filters needed to be maintained in the source system to be seen here.
• The extraction tab gives the details of type of the datasource (*needs more appropriate explanation)
• Processing tab gives you the detail till what level the data will be processed, like till psa, data targets only.
• Data Targets gives the detail of all the objects for which data needs to be loaded.
• Update tab is the most important tab as it has details on how the datasource should work. More details are given below.
• Schedule is for triggering the data extraction

Update Tab
This is the important tab in the infopackage. here, we mention the type of update.

Full Update

When you run the using Full Upload, what ever data is there in source everything will be pulled to BW. If you use this, you can't run the Delta as there is no pointer set to find delta records. Only after initializing the datasource, delta records can be fetched.

A full update requests all data that corresponds to the selection criteria you determined in the scheduler. In a certain period of time specified in the scheduler, 100,000 data records are accumulated. With a full update, all 100 000 data records are requested.

When running the full loads, if we have more down time, we can fill setup tables and do a ful l load, else, we can do an Intialization without data transfer which doesn’t fetch any data but it places a pointer for delta and the next time the data is extracted, it fetches the full load records and delta records.

Initialize Delta Process

Only after doing the initialization with any of the below options, the datasource is made delta capable and can be seen in RSA7. The option of Delta is Infopackage is also available only after this process.

• Initialization with Data Transfer
• When executed, this will create an entry in the RSA7, which means the pointer is set for delta records. It also fetches all records from ECC and after its completed , it places the pointer. It is Full Load + Initialization.
• Initialization without Data Transfer
• When executed, this will create an entry in the RSA7, which means the pointer is set for delta records. It fetches a single record to BW which is a header record.
• Early Delta Initialization
• With early delta initialization, you have the option of writing the data into the delta queue or into the delta tables for the application during the initialization request in the source system. This means that you are able to execute the initialization of the delta process (the init request), without having to stop the updating of data in the source system.
• You can only execute an early delta initialization if the DataSource extractor called in the source system with this data request supports this.

Delta

A delta update only requests data which has appeared since the last delta. Before you can request a delta update, you must first initialize the delta process.  A delta update is only possible for loading from SAP source systems.  If a delta fails (status ’red’ in the monitor) or the overall    status of the delta request has been set to red manually, the next data request will be performed in repeat mode.

• F : Flat file provides the delta
• E : Extractor determines the delta, Ex: LIS, COPA
• D: Application determines the delta, Ex: LO, FI-AR/AP
• A: Use ALE change log delta

Full Repair  Request

This is present in the ‘Scheduler’ menu -> Repair Full Request, on the top when you open the infopackage.  These are used to fill in the gaps where delta extractions did not extract all delta records , when delta extraction process failed and cannot be extracted/restarted.

The request gets data from setup tables without delta being disturbed, wherein the setup tables are filled only according to the criteria.

These are also useful when you are initially extracting large volumes of data, whereby you execute an Init w/o data transfer and execute multiple Infopackage (Only Full Update) that are full repair request with specific selection criteria.

Finding the Infopackage Job

Once the infopackage is triggered for running, if you want to see the job details,

Environment -> Process Overview -> In the Source System

Else,

Copy the request number , go to the source system SM37 Add 'BI<request number>' and user as '*'

Data transfer using Infopackages

Data transfer using infopackes is done using IDocs and TRfc. Check LINK for more info
Infopackages can fail due to a number of reason.
Some of the common issues can be found : LINK

LO - Adding new fields to the existing data flow

LO Extraction

Activating new datasource in LBWE

• Go to LBWE and look for the datasource
• The datasource can be one of the following states

• Click on Inactive . This would make the datasource active. System might ask for a customizing transport. It gives a pop up with MCEX notification. continue and the datasource will be activated.
• The above step will activate the datasource in development client. If development and testing are done in different client, use SCC1 tcode to copy the changes between the clients.

Enhancing existing datasources

Adding field in LBWE
Check if the field you are looking for is present in the datasource structure. Go to LBWE and look at maintenance . If you have the field there, we can directly add the field there by moving it from right to left.

Adding field as an enhancement
If the field is not present in the extract structure, we need to add the field to the append strucuture of the datasource.

There is also a way to add the field to enhance the MC structure for the new field, which I never tried...!!

Adding fields in LBWE

Now that we already have the field in LBWE, all we need is to move the field from right side to left
side. We can do that following below steps.

Step 1 
check and Clear the setup tables.

Step 2
Clear delta Q from LBWQ
If there are entries in LBWQ , run the V3 job - this should move entries to RSA7.
You can also delete the queues in LBWQ but you will be missing those entries.

Step 3
Go to LBWE, Click on 'Maintainance'.
click on the fields needed and move them left.Once the field appears in the pool, it can be moved from right side to the left side and they are ready to use.

There will be couple of pop-ups with information, read( if you want) and click okay.  Once the field is added, you can see that the datasource becomes read and inactive.

Now click on the datasource and this will activate the datasource and take you to the maintainance screen. The new fields will be hidden by default. You need to remove the check for 'Hide Fields' and field available only in customer exit.

This step will make the datasource to change from Red to Yellow.
Click on Job control ( Inactive ) and this will make it Active.

The field is successfully added to the datasource and is all active.
Note : If you have separate development and dev.testing clients,
you need to copy the customizing changes to test client. ( It is not test system, this is only if you have test client ).
Use TCode SCC1 in the target system and run it for the customizing transport. check 'Including Request Subtasks' also.
Since customizing transports are at client level, they need to be copied to different clients this way.

Possible Errors

If you get the below error , that means that LBWQ is not cleared.

" Entries for application 13 still exist in the extraction strcuture"
Sol :
Run the V3 job and clear the entries in LBWG

Struct appl 13 cannot be changed due to setup table
Sol :
Delete the setup tables in LBWG.

Running Setup Tables
When the tcode for setup tables are run , you might below error.
"DataSource 2LIS_02_*** contains data still to be transferred"
That means that all the data from ECC queues has not been transferred to BW.
So make sure that LBWQ is empty for the extractor ( most of the cases, the V3 jobs runs regularly every 15/30 minutes, so these queues will be cleared). Next, clear the delta queues in RSA7 by pulling them to BW. Sometimes it you might need to run the infopackages multiple times, if there is activity happening in the source system).
Once LBWQ and RSA7 is empty, you can run the t-code and it will go through fine. 

Adding fields in structure

Step 1
Go to RSA6 -> Click on the datasource and go to the MC* Structure.
If you want to create a new append structure , click on Append Strucutre - New
Else, go to any of the existing append structures and ad your fields there.

Step 2
Open the datasource again in RSA6 and you can see that all fields are available in Hidden. Click in edit and unhide the fields.

Step 3
Next code for extracting these fields should be added in CMOD or BADI implementation whichever the company is using.

ECC Tables

ECC Operational Tables

Tables	Description	Tables	Description
CDHDR	Change document header ( Any changes to the document will be documented over here )	RODELTAM	Gives Delta properties for different deltas
CDPOS	Change document items	ROIDOCPRMS	Control parameters for data transfer from the source system - IDOC Configuration
BDCP	Change pointer Table	ROOSSHORTN	DataSource Short Name
BDCP2	Aggregated Change Pointers (BDCP, BDCPS) - Master data delta changes	ROOSPRMSC	Control Parameter Per DataSource Channel - Delta Initialisations
SEOCOMPO	Gives the list of all BADI implementations(Classes) and corresponding methods	TRFCQOUT	tRFC Queue Description (Outbound Queue) - SMQ1
ROOSGENDLM	Generic Delta Management for DataSources ( Gives details of when the last delta was run)	ARFCSSTATE	Description of ARFC Call Status (Send) - SAP Note 378903
TMCEXACT	LO Data Extraction: Activate Data Sources/Update ( When activating new LIS datasource check for inconsistencies.)	CLBW_SOURCES	Data Sources for Classification Data
TCDOB/T	Objects for change document creation
TRFCQOUT	tRFC Queue Description (Outbound Queue)
BWOM_SETTINGS	BW CO-OM: Control Data : This table has all FI related control parameters https://blogs.sap.com/2013/04/02/bwomsettings-for-fi-loads-in-sap-bi/
BWOM2_TIMEST	BW CO-OM: Timestamp Table for Delta Extraction
BWFIAA_AEDAT_AS	FIAA-BW: New and Modified Master Records for Delta Upload
.
.
.
.
.
.
.
.
.

Functional Tables

Tables	Description	Tables	Description
BKPF
BSEG
FAGLFLEXA	General Ledger: Actual Line Items
KNA1	Customer General Data	KNB1	Customer Master u2013 Co. Code Data (payment method, reconciliation acct)
KNB4	Customer Payment History	KNB5	Customer Master u2013 Dunning info
KNBK	Customer Master Bank Data	KNKA	Customer Master Credit Mgmt.
KNKK	Customer Master Credit Control Area Data (credit limits)	KNVV	Sales Area Data (terms, order probability)
KNVI	Customer Master Tax Indicator	KNVP	Customer Partner Function key
KNVD	Output type	KNVS	Customer Master Ship Data
KLPA	Customer/Vendor Link	MARA	Material Master: General
MAKT	Material Master: Short description	MARM	MM Conversion factors
MVKE	Sales <Sales Org, Distr Ch>	MLAN	MM Sales <Country>
MAEX	MM Export Licenses	MARC	Material Plant
MBEW	MM Valuation	MLGN	MM WM Inventory
MLGT	WM Inventory type	MVER	MM Consumption Plant
DVER	MM Consumption MRP Area	MAPR	MM Forecast
MARD	MM Storage Location	MCH1	MM X Plant Batches
MCHA	MM Batches	MCHB	MM Batch Stock
MARCH	MM C Segment : History	MARDH	MM Storage Location Segment History
MBEWH	Material Valuation History	MCHBH	Batch Stocks : History
MKOLH	Special Stocks from Vendor : History	MSCAH	Sales Order Stock at Vendor: History
MSKAH	Sales Order Stock: History	MSKUH	Special Stocks at Customer: History
MSLBH	Special Stocks at Vendor: History	MSPRH	Project Stock: History
MSSAH	Total Sales Order Stocks: History	MSSQH	Total Project Stocks: History
AUFK	Order master data	VBPA	Sales Partner Functions
ADR6	E-Mail Addresses (Business Address Services) Email	VBPA2	Sales document: Partner (used several times)
FPLTC	Payment cards: Transaction data - SD Sales	AEOI	ECH: Object Management Records for Change Master - Change Records - Revisions
AENR	Change Master	NACH	Detailed output data / output Conditions TCode : VV31/32/33
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

Master Data

Master Data Tables

Basic Notes

Click HERE to get basic information on Master data tables and Architecture

Deleting Master Data

As time goes on some junk/obselete values gets accumulated in the master data tables. The master data which is not being used in any of the infoproviders can be deleted directly from the table with out any issues.

Default : 

If the deletion of master data is selected, system automatically deletes the entries which are no longer used. When this is executed, the entry gets deleted from Ptable(/BI0/PCUSTOMER) of the infoobject , but is still visible in SID table (/BI0/SCUSTOMER) , the following way.

Here , the CHCKFL, DATAFL, INCFL are all blank, because the specific entry is deleted from Ptable, but the SIDs are not deleted.

The reason for not deleting SID value is that, if in future the same customer number arrives as correct data, there is no need for the system spend time generating a new SID value, instead the existing SID value is used.

Howver, if we know the data is incorrect, it can deleted with SIDs.

We can select how we want to delete the master data from below options.

Delete SIDS

When deleting the entries, the system also deletes the corresponding SID values.

Delete Texts

It deletes the texts

Simulation Mode

This simulate the deletion without deleting the actual data in the system

Store Master data Where used list

<need to check>

Search Mode

In the output, it gives information about the data being selected.

When 'O' - if we are deleting a customer, it will give one of many places the specific customer number is being used.

When 'P' - Gives one value with the corresponding each and every infoprovider

When 'E' - One usage

When 'A' - All the places where the customer number is being used.

Program to delete master data
RSDMDD_DELETE_BATCH

The advantage of using this program, is that we can run the deletion of individual entry in the master data using the 'Change Filter' option. Once the program is run, log can be seen in Tcode 'SLG1' ( see below for details).

'Check also NLS' should be selected for running the deletion to happen. ( NLS - Near Line Storage)

 

Running the program with out selecting any option would delete the master data entries which are present in the infoobject but are not being used anywhere with out deleting the SIDs.


Tcode :
Logs after master data deletion : SLG1
Enter below values to run the tcode.
give Object : RSDMD
SubObject : MD_DEL

Usage of SIDs

FM :  RSDDCVER_USAGE_MDATA_BY_SID

Points To Remember

- Making Display Attribute as Navigational Attribute

• Making an attribute of InfoObject Display to Navigational does not effect other object where it is being used. No other activation are required. Sometimes, the infoobject might not get actived in the first try giving an error : characteristic the attributes sid table(s) could not be filled .
• Activate the object again and it goes through ( not sure why )

Extended Star Schema

Extended Star Schema

Extended Star Schema has a fact table in the middle surrounded by Dimension tables. The Dimension tables have dimension ids and SID ids. The SID table is the table which connects Dimension table and Actual data (Master Data) Table.

In the below diagram, if we want to know the revenue from a material by a specific customer, Each and every customer number has a SID generated and in the same way, each and every Material has an SID generated.

For each Multi-Dimension Data model, we have A FACT table, which holds all the key figure values. For each Dimension defined, there is a Dimension Table which has Dimension ID and SID ID. For every InfoObject value, there will a SID table generated which connects the InfoObject values to the Dimension values.

When we are defining attributes, there are 3 ways of doing it.

• We have it as a  characteristic in the attributes
• Add as an Navigational/Display attribute(more notes later)  where it directly does not reside in the cube but can be called by drilling down
• As a hierarchy.

*Customer *	Product *	*Color***	*Sales *	*Fact *
C1	P1	Black	S1	30

Customer, Sales is individual dimensions and Product and Color belong to the same dimension.

When this record comes to BW, There will be SID-tables created for each one of them. Below you can see there are 4 different SID Tables (Highlighted in Yellow) and each of the value has a SID value created (boxed in red)

Once the SID are created, there will be DIM tables. Since Sales and Customer are different Dimensions, there will be separated DIM tables. Product and Color belong to the same Dimension. Hence, there is only one Dim-table.  The values of the DIM tables are filled as shown in the figure below.

        

 Hence the entry is completed written. Now there comes another entry. 

The values are populated as follows.

*Customer *	Product *	*Color***	*Sales *	*Fact *
C1	P1	Red	S1	40

  

Advantages:

• The Master Data is kept outside the infocube, which helps in accessing it from other InfoCubes/InfoProviders.
• Alpha numeric values are converted to SID values (Surrogate IDs) which are numbers , increasing the processing speed.