Applied dimensionality » etl

New recruit to my ETL toolbox

ykud — Sat, 17 Dec 2011 11:29:54 +0000

I’ve recently completed my first real DataStage project and took a chance to get certified while all the stuff is still “fresh”. Certification itself is quite complex and I didn’t use most of the tricks depicted in questions up until the moment when one of the jobs had to process a quarter billion of rows in a reasonable timeframe. From that point on I learned quite a lot about partitioning, balancing, debugging and choosing right stages to do the job (who would’ve thought that RemoveDuplicates is waaaaay slower than Sort (with Remove Duplicates option) — why put RD stage in at all?) Anyhow, now I’m also an IBM Certified Solution Developer — Infoshere DataStage v8.5 )

So my current ETL tools breakdown goes smth like (not counting PoC and likes):

Oracle Data Integrator — 3 projects
Pentaho Data Integrator — 2 projects
IBM InfoSphere DataStage — 1 project

And that’s my current preference list as well. I love ODI’s flexibility (it’s actually very simple once you get it how it works and it’s extremely configurable), ELT approach (I’d rather be tuning my DBMS than DBMS and a separate ETL engine). PDI is very open and quite user-friendly (compared to DS, for example) and it’s easier to understand & debug than ODI. PDI community edition is enough for most small data sized integration projects and enterprise version is very affordable. Datastage is terrifically well-suited for big data volume tasks and parallel processing, but is quite an overkill in small projects.

It’s interesting that although I did quite a bit of DWH model design I written have just a few posts on this topic. But every time I think about writing out some advice — I think that the best advice is to just go read the books. And if you still have questions — reread them ) I’m reread Kimball’s books a few times already and every time gives you an “ah, that’s what they meant” moment based on your recent experience.

Anyhow, my last couple major DWH projects were for government agencies and I packed a number of simple but effective modeling tips exactly for them. Hopefully I’ll write them out in nearest future. Just need a free weekend or two.

Using session variables in Cognos BI

ykud — Mon, 22 Aug 2011 15:52:14 +0000

Just a quick Cognos BI hint: you can use session variables to store project-level constant values.

I’m a big fan of ‘feature-rich’ ETL reports showing not only what dimension element mismatch between systems, but also allowing seamless editing of element mapping. This usually means drilling down from report into external application for dimension mapping. Parameters are usually passed via URL (easiest possible way).

After server name and, therefore, URL changed for a second time in current project, I’ve set up a project level ‘severname’ constant to avoid XML find\replace for each report.

It’s really easy:

1) Add a session variable to your Framework project, write your required constant value there. Like ‘servername’ = ‘awesomebi’

2) Use it in Report Studio, just type

#sq($variable_name)#

in expression editor. sq encloses string in single quotes.

ETL Testing

ykud — Fri, 24 Sep 2010 11:44:37 +0000

I’m a fan of testing in each DWH project, because it allows to:

- be sure that at least tested parts work

- change logic w\o retesting all results

Receiving “All OK” in the morning means that there’s time for a cup of coffee )

A list of approaches I use for testing ETL-procedures:

‘Water in a Sieve’
‘Excluded Middle’
‘Do your maths’
‘Like in good old days’

‘Water in a Sieve’

Checking whether we could carry all required data into DWH without ‘spilling’ it out.

Common things to check:

- row counts in source\DWH

- grand totals in source\DWH

Common mistakes found:

- precision errors

- missing dimension mappings

‘Excluded Middle’

Checking dimension mapping for a selected dimension. This extends previous test by including dimension totals for checking.

So if we want to check whether ‘products’ were mapped correctly — we compare totals(sums) by time, store, etc, listing all dimensions except products.

If counts(sums) differ in source\DWH — we know that products were mapped incorrectly. Moreover we know the specific data subset containing error, which helps a lot as well.

‘Do your maths’

Checking DWH calculations. If we’re doing some data transformation\calculating something in DWH — it should be tested as well.

2 approaches to testing:

- overall logic testing. For example, if we’re allocating HQ expenses to get regional reports it logical to expect the overall sum of expenses to stay the same )

- testing a specific data subset. We can select a single account and verify logic on it.

These two approaches should be combined.

‘Like in good old days’

Checking some heuristic expectations, based on previously loaded data.

1) There’s no way daily sales can jump 70% compared to quarter average

2) We’re usually getting about this number of rows from this source

3) If we’re reloading 3 months of data daily — we expect modest amount of changes in past days.

If you’re using partitioned loads ‘last_load’ partition can be used for such testing.

It’s best when tests are written not by procedure developer himself. You can always apply cross-checking (you write tests for my procedures, I write for yours).

Pay attention to Chapter 4 of ETL Toolkit — this post is just a simple list of typical tests, the methodology is described there.

Freebase GridWorks — a data-driven approach to ETL

ykud — Tue, 11 May 2010 14:11:32 +0000

Take 5 minutes to watch screencast for FreeBase Gridworks — an interesting new approach to data transformation. Instead of ‘transformation-based’ approach of every tool on the market, this tool uses ‘data-based’ approach, which looks rather intriguing. Especially ‘Undo-Redo’ function )

I keep thinking when to use this tool. User driven data load in DWH? This tool lacks MDM capabilities in such case. Manual cleaning of csv files?…