Applied dimensionality » BI&CPM

Oracle Essbase and Oracle BI 11

ykud — Thu, 23 Jun 2011 07:07:57 +0000

Just in case you missed it — read this two posts by Mark Rittman (OBIEE 11g for Hyperion Users – Are We There Yet?, Incremental Essbase Metadata Imports Now Possible with OBIEE 11g) on the oh-so painful topic of Essbase-BI integration. I participated in some projects with this combination a couple of years ago (when BI 10g was the only option) and this experience left me very confused. Reading Mark’s cover-up for 11g — it seems a lot has changed, but still not enough (

Both Oracle BI and Essbase are totally great tools, but this 3 (or even 4) year long integration story gives us a couple interesting points to think about.

Efficient MDX-generation is hard. 50+ MDX queries for a single BI 11g report are a real showstopper, since you can kill even such a mammoth as Essbsase with such load. Moreover, such query patterns (1 query for a couple of cells) negate Essbase caching and calculation efficiency. So almost all Oracle BI reports on Essbase I’ve encountered are handwritten MDX, tied to BI report by using Evaluate functions. And that’s a thing everyone should avoid.
Managing OLAP metadata in relational repository is hard. When you’re trying to incorporate both conventional table structures and flatten OLAP dimensions in the same wire-model, you get cut by torn strings. Incremental OLAP model updates are very hard, since you have to preserve user changes and yet to ”find out what’s new”.

So everyone using Oracle BI + Essbase I’ve talked to end up using scheme like this:

Use some tool for rapid OLAP data exploration (Excel Add-In, Visual Explorer)
After some query layout is formally accepted — molten it in concrete by adding handwritten MDX for Oracle BI report

Given that in this OLAP + RDBMS BI scheme there’s also a problem of security settings synchronization (no built-in integration for this as well), introducing OLAP in BI system is always a big decision.

I may be overly pessimistic, but given the rate of RDBMS adapting for analytic purposes (aggregate awareness, partitioning, MPP-architecture, compression, columnar storage) and BI tools complexities when working with OLAP, we might witness slow decrease of OLAP engines usage for reporting (whereas in areas like planning OLAP is still unbeatable).

To cover both sides of the story I’m aware of — I’ll write about the way Cognos works with OLAP engines in next post.

BI&DWH Booklist

ykud — Fri, 11 Feb 2011 01:36:27 +0000

More time I spend doing BI\DWH projects, I become more aware that such projects a totally a ‘craft‘, as my professor once said. And this ‘craft’ has many different aspects in it, each having a number of of very good texts. telling you all you need to know to avoid simple mistakes and deliver predictable results. But I encounter a very small number people who read even half of these books. Maybe it’s the ease of ‘bicycle invention’, maybe it’s boredom of reading about others mistakes, instead of earning your own scars. So it’s another ‘book-list’ of texts that make up a competent BI\DWH professional.

I’m trying to split it up into section of interest.

DWH Classic texts
Data Modeling.
Achieving DWH performance
Data Quality
Data Visualization & Report Design
Product documentation

DWH Classic texts

There is no way you can do DWH modeling or ETL design, or even reports developing without reading at least some of Kimball Classics (Dimensional Modeling, ETL Toolkit, Lifecycle Toolkit). And if you’re into this for more than one project — you should start memorizing Lifecycle by heart.

You should read Bill Inmon as well to be aware of 2 different approaches to modeling. Also see Data Modeling section just below.

Data Modeling.

These books are not DWH-specific and most modeling tricks don’t apply directly, but you should be aware of typical problems and solutions, since this allows fast source system model examination and profiling — you can recognize modeling techniques used in developing and potential caveats in data.

There are numerous data modeling books out there, I’d recommend reading Hoberman and Witt&Simsion.

And I insist on reading Data Model Resource Books. It has 3 volumes and I suggest starting with 3rd one and then picking your business area of interest (HealthCare, Telco, Insurance — there’s a lot to choose from) and looking the model in first two volumes.

Achieving DWH performance

If you’re interested in getting it all faster, take a look at Mastering DataWarehouse Aggregates. And there’s a deep insight out there in Relational Database Index Design and the Optimizers — read this one only if you’re really into performance )

Data Quality

It’s a very important, yet most forgotten topic. After reading ETL Toolkit and Lifecycle parts on data quality, I suggest following with one of works by Larry English. If you’re very serious about separate DQ project, pick up Executing Data Quality Projects.

Data Visualization & Report Design

Data visualization is a most expected BI-revolution that won’t come in a way vendors think about it (another feature pack or product). It will come by evolution, just not of BI products, but of us, BI professionals and data analysts. As we will get more acquainted with analysis techniques and get more experience with well designed visual aids (as opposite to Flash-based eye candy), we’ll shift our demands towards more analytical oriented tools and vendors will have to provide them )

I’d recommend anyone to skim through Head’s First Data Analysis, since it’s a good introductory book on how gain some insights from data.

Anyone designing any dashboard should read Stephen Few’s books. It’s an absolute must. Stop Pie-Charts invasion!

If you’re up to thinking about some new ways of representing information or gaining a deeper understanding of how things work, read Tufte’s works. And then follow up with Collin Ware deep texts on how our minds really work while processing visual information.

Product documentation

And, of course, you should read all the documentation provided by vendors of your tools and follow up most prominent blogs (there are ones on almos every tool out there) and forums. It’s a shame to see people inventing built-in functionality or not knowing that’s a screwdriver they’re are trying to slam a nail with. I find myself doing this with TM1 recently, so I’m back to reading manuals

I’ll update this post when I’ll encounter a good text on MDM or more good books on above mentioned subjects.

Any suggestions?

Gartner BI Quadrant for 2010 is out

ykud — Tue, 01 Feb 2011 06:52:21 +0000

It’s an interesting reading indeed.
I read vendor sections in these reports with interest, because after some years it’s possible to wipe all vendor-based marketing claims and assumptions made by people (analysts) who aren’t using the products daily and see something resembling “true state”.
For example, if you take a statement “QlikTech offers limited metadata management” and read it with cynic-hat on, you’ll read out that there’s no single metadata repository for an installation. Each so-called application packs it’s own metadata (+ data loading scripts), so you can easily have ‘Net Sales’ sourced from different tables in applications developed for different users. “Single source of truth”
With this hat on we get:
- Oracle 11 was too long in development labs and didn’t get much out of it. But all those new stuff will be added some time later.
- SAP still haven’t decided what they bought BO for — that’s hysterically funny
- Microsoft BI is still a DIY kit, but now with more shiny new tools (PowerPivot) inside. But once again packed without assembly instruction )
- Cognos is hard to implement and slow, ain’t it?
- Tableau is terribly nice, useful and furry. Just small for enterprise-scale
- etc
Seems like at Gartner guys like SPSS acquisition by IBM much. And consider Cognos 10 a break-through in performance. Need to check it myself. After SP2 )

Anaplan

ykud — Wed, 19 Jan 2011 02:04:55 +0000

In case you haven’t heard about it yet — “Adaytum team” is back to business. With their new planning solution ) — Anaplan, or “In-Memory Analyst in the clouds”.
Take a look at demos and site, especially History page — reminds you anything?
My takes:

Technology has all needed buzzwords, although I’d prefer some more common, not ‘home-made’ parts, like HFS for file-system. Not so many details out there, nothing to comment more on.

I’m really fascinated by implementation cycle proposed. I’ve been doing it almost ten years up to now and met only one 3-4 weeks project on the way. Doing requirements gathering off-site, from Excel templates — implementation team must be totaly great.

Take a look at prices. It’s not just cheap, it’s just for free. In this case, to earn money, Anaplan must attract literally thousands of clients. SalesForce for planning?

Data integration. This is an interesting part, since a planning system is just a fake if it sits in the air (cloud ) without strong roots to the ground (actual data). In a complete CPM solution — DW srores your actuals and loads them into planning on demand. And loads back accepted plan figures for analysis. How this integration is to be done, when a part of the system is out on the cloud — is not a magic, but some tricky task anyway. Integrating with other cloud-based services (i.e. Saleforce) should be easy.

People. I’d really love to chat with those guys one day

Overall: I haven’t heard of a great success of any cloud-based BI services (and there was and is much hype on topic). Planning is bit like BI, because there are a few\none industry-standard solutions, it’s mostly an open LEGO constructor for you to build a castle.
And if you don’t have a repeated\reusable model, you don’t earn that much money scaling up number of customers, since building from scratch is required in every case. SalesForce provides a standard CRM with some customizations for client, but the required functionality is typical for all client base. This allows ‘reusing’.
Can this be done with Enterprise Planning?

It looks very interesting, although cloud-based thingie won’t go in Russia right now, I’ll be watching. I really liked idea behind Analyst.
And as I’ve already said, I’d rather tweak APL and J technology for 64bit and multi-core….

Microsoft SQL Server reporting database configuration

ykud — Thu, 23 Sep 2010 12:18:07 +0000

A check-list for using Microsoft SQL Server as a reporting database. This is a ‘be aware’ list for myself in first place, since I forget some details as time passes.

Table of contents

Overall database settings

Table configuration

ETL

Query tuning

Books to read

Overall database settings

Use simple recovery mode

Set up simple recovery mode for database to minimize performance issues with transaction log records. Usually you can use daily backups with DWH, since data is loaded daily. In more complex situations you can use ETL-based ‘checkpoints’, but in general, daily backups are more than enough.

Separate log and data io channels

Use decent I\O systems and separate data and log I\O channels. This is obvious, just reiterating.

RAID 1+0 seems to be preferable choice of disk configuration for DWH loads. See this entry, this one or test it yourself )

2008 r2

Use SQL Server 2008r2 Enterprise if you have a choice. 2008 adds table\index compression (which is great) and get’s partitioning to work way better in queries. And I’ve 20% performance speed-up against 2005 on the same query set (with same execution plans and that surprised me).

Auto shrink Off

Set Auto shrink off, it kicks unpredictable I\O strain. Shrink database daily, just after backup in the same maintenance task.

Table configuration

identity columns as primary keys

Use identity columns as table primary keys. And using this column as clustered index will help I\O in most cases.

Set up foreign keys

Again, obvious, but I’ve seen this violated more than once. Use foreign keys for fact-dimension table reference. This will help optimizer a lot.

You can disable these keys to speed up inserts, but in 90% cases it’s not needed.

Table partitioning

Always use partitioned tables (even fake, with 1 partition) to:

- achieve partition elimination during query execution. Especially in Ms SQL 2008

- use partition switching for data load (check this article as well)

Table compression in 2008

SQL Server 2008 introduces table compression which helps to reduce I\O (but increases CPU load), which is very good for reporting tasks. Use it for fact tables.

ETL

partition switching as default load strategy

More benefits of partition switching for loading:

- while you’re loading data in ‘new_data’ partition, base table is accessible and consistent

- after you’ve prepared ‘new_data’ partition you can check it’s data for quality (planning to write about later)

You can implement switching by creating another temporary partition, which I like to call ‘last load’. So you switch:

- current -> last_load

- new_data -> current

in one transaction

After that you have ‘last_load’ to:

- switch back in emergency

- check data from current\previous load for some more quality checks

Filtered indexes for calculations

Another 2008-only feature is called filtered indexes. These indexes can be essentially smaller than common ones if you know for sure what region of data you’ll be accessing. And this turns out to be very efficient for SQL code you’re using for different calculations. Since I mostly work with Oracle Data Integrator as an ETL tool and it’s an SQL-generating machine: I get up to10 times calculation speed up by building small, calculation-specific filtered indexes.

Tablock hint

All inserts should be done with TABLOCK hint, which speeds up insert by acquiring a full table lock during the whole transaction. Since in DWH main inserts are performed by ETL there should be no conflicts. TABLOCK also speeds up logging during bcp.

Query tuning

Indexes

Really a vast topic, but to sum it up ) :

- use covering indexes. It seems better to have a ‘wide index’ than a number of indexes on all dimensions and to apply index intersection.

- columns in indexes should be ordered by selectivity

Be very-very careful, ‘your mileage may vary’:

- creating a huge number of large indexes may actually slow system down due to I\O overload

- you should try to create indexes that can help a number of reports\query instead of tuning a single request

See Books session below, skip through at least one of them before starting it out wild.

Filtered indexes

As with ETL calculations — filtered indexes may be used for tuning specific queries.

Index compression

Indexes can be compressed in the same way as tables (which is obvious given that a table is a clustered index) and this speeds queries by reducing IO

Books to read

I’d recommend following

Professional SQL Server 2005 Performance Tuning

SQL Server Execution Plans

SQL Server 2008 Query Performance Tuning Distilled (Expert’s Voice in SQL Server)

And if you have time and will

Relational Database Index Design and the Optimizers

Replacing strings in Oracle BI Publisher reports

ykud — Wed, 08 Sep 2010 16:09:07 +0000

Oracle BI publisher works awfully bad with Answers type of datasource in at least two points:

- column names are created by answers formulas instead of column names, so if you have long formulae, it’ll be shortened to some 128 chars. And if first 128 chars of formulas are identical, you’ll get only one of them in publisher
- if columns return null in answers – they will not be available at all

We ended up using logical SQL generated in answers as a source for publisher reports, which works like charm up until you start changing repository or changing underlying data you’re using for filters.

- repository changes can be handled by adding aliases to columns being renamed

- data changes are way more difficult to tackle

For example when you change city name from ‘NY’ to ‘New York’ in geography dimension (this should be solved by BI 11 id\description feature) you have to change all formulas and filters using this value.

So I wrote a simple Python3 script, doing exactly that — replacing all needed strings in publisher xdo files. If there’s any need for that — mail me, I’ll send it over.

Pivot Tables and temp files in Oracle BI EE

ykud — Fri, 27 Aug 2010 13:32:39 +0000

Pivot tables

Pivot tables are by far most used data view in Oracle BI Answers, since they provide ‘multidimensional’ analysis capabilities, aka dimension drag&drop, filtering and some basic calculations.

From server-side view — this component provides worst performance, since all future ‘cube’ data has to loaded into BI server, aggregations and formulas calculated and transformed to internal format.

So what happens when user clicks a pivot for 200,000 cells cube?

1) SQL-query is fired into database

2) All resulting rows are loaded into temporary directory of BI server

3) If there are calculations to be performed — initial temp file is read, calculated and new temp written out

There are two important instanceconfig.xml parameters for Pivot tables:

- CubeMaxPopulatedCells – The maximum number of populated cells in the Pivot Table. If this maximum is exceeded, the user receives an error when rendering the pivot table.

and

- CubeMaxRecords – The maximum number of records returned by a query for the Pivot Table engine to process. This roughly governs the maximum number of cells that can be populated in a pivot table (unpopulated cells in a sparse pivot table do not count). If this maximum is exceeded, the user receives an error when rendering the pivot table.

All instanceconfig.xml options can be found here.

Temp files.

Temp files can be rather huge, I’m currently seeing files about 1gb size for medium sized reports. That, surely, because we’re using MS SQL server and BI Server cannot “push” some calculations in database, but it’s nice to keep in mind the fact that temp files can occupy some 20-30 Gb.

Temp files are cleaned out when session logs out, so some of them stay there for whole day.

Location of temporary files is set up by WORK_DIRECTORY_PATHS variable in NQSConfig.ini and by SATEMP enviroment variable. See this post for instructions on how to move tmp folder.

Speeding things up.

Rather huge files are written to temporary directory, so the IO performance becomes an obvious bottleneck for process.

What can be done:

1) Use multiple tmp\ directories on different disks to balance the load

2) Use Ramdisks for tmp\ — I’ve tried this way and got a some fantastic 5x speedup on most IO bound reports. But it’s pretty hard to find free 30 Gb of RAM, so we’re using multiple disks for now

3) Use SSD for tmp — that’s what I plan to try next. I bought an SSD for my laptop about a year ago and my life has never been the same, so stay tuned )

BI system metadata impact analysis and what BI system APIs are for

ykud — Wed, 19 May 2010 10:52:23 +0000

“Okay, this column in our BI metadata was supposed to be named Sales, not Sals.

But when we change it, what will happen to the reports that were created, using this name? And how many reports use this column anyway? Maybe we shouldn’t touch anything to keep things working…”

Ever been in such situation?

This question can be answered by opening each report and then looking at it’s contents ) A horrible task indeed.

Actually, this is exactly what BI system API are made for. Using them you should be able to automagically look into each report and find whether this column is used in it.

And as usual in this ‘BI system Needs and Means’ series, let’s look how it’s done in IBM Cognos BI and Oracle BI.

Cognos BI

Cognos BI has had an SDK (Software Development Kit) since ReportNet (for more than 5 years already) and this ‘long-time in market’ allowed to develop a whole ecosystem of tools to breed on top of it.

Concerning the question at the start of the post — there’s:

Cognos Dynamic Report Updater — a tool to replace change column in every report.

http://www-01.ibm.com/support/docview.wss?uid=swg24021248

Moreover, look at whole bunch of tools, based on SDK:

http://www-01.ibm.com/software/data/support/cognos_diagnostictools.html

and tools, developed by third-party vendors:

http://www.brightstarpartners.com/bspsoftware.php

Oracle BI

In Oracle BI, finding such changes is somehow easier, since all reports are stored as xml files in presentation catalog folder. You just write a simple script that iterates on files, finds a string in file and reports back it’s findings (or replaces this string with a new one). Sounds easy, but after spending a couple days at it, I’m not so sure anymore )

There are a few major point against this way

1) It’s unsupported, there are no declared report modification API’s, AFAIK, so you’re scripting at your own sake. That would not be a big problem, if not:

2) Oracle BI presentation service adds a hash tag to report (and folder) names if they exceed 55 characters in length (see more on this topic here: http://obiee101.blogspot.com/2010/02/obiee-report-name-length.html).

2.1) If report’s names are not in ASCII, they are converted to UTF-8 (but starting x symbol is replaced by #), so each non-ASCII symbol takes 5. So if you find file, containing desired string, you can only decode back first 9 letters of it’s name — report names are usually much longer than that. Therefore your searching capabilities are seriously limited.

2.2) Folder name hashing poses another problem if you’re using Windows. It impossible to iterate through catalogcontaining hashes an # in it’s name — OS listing methods stop working. Hope that situation is better on Linux. An easy but entirely manual workaround is to rename catalog folders to numbers (1,2,3,4) before starting the script ant to rename it back after.

I totally wish that this situation will change in nearest future and there will be a simple Presentation Server API call that will bring report name from file name and vica versa.

As usual, if anyone is interested in scripts I wrote for the task (python, as usual + vbs) — ping me.

Multidimensional reporting functions in Cognos and Oracle

ykud — Sun, 25 Apr 2010 17:47:32 +0000

As many of you already know I have a mixed attitude to Cognos BI DMRs (dimensionally modeled relational source) models:

performance is allways a consideration, full-table scans are unpredictable
there’s always ‘local processing’ vs ‘database only and some things not working’ dilemma
using multidimensional functions in reports is oh-so-hard to debug when they don’t work in complex situations.

Even after 4 years working with Cognos BI I still have to start multidimensional reports with simple cases to prove that functions work as I expect them to (sigh).

BUT — at least Cognos has those functions!

We’ve just spent more than a week trying to replicate members() function in Oracle BI project. Function is as simple as can be, it just returns all dimension members from all levels in a single list (see documentation on IBM site). And if you add measure to report — correct totals will be calculated. Oracle BI has the dimension with levels and aggregations concept, but no such function, so you have to turn over your head to achieve this. Replicating dimension table in logical model as many times as there are levels seems the easiest solution.

I’m really frustrated by this.

Definitely not the problem I’d like to spend time trying to solve :(

Connecting BI system to Active Directory. Oracle BI vs IBM Cognos BI. Scripts to extract Active Directory information.

ykud — Tue, 13 Apr 2010 06:00:31 +0000

I consider Single Sign-On setup through Active Directory(AD) a key point in every system’s life.

Up to this point — it’s just a PoC, a prototype or a department (10-20 users) level application.

AD integration step is usually prosponed, because it’s documented somewhere and there are much more important things to do, like showing that this system is of any value at all.

But adding an AD connection mean that system is rooted (even if lightly) in company’s IT ecosystem.

There’s always a dilemma on groups and rights storage:

1) You can use AD groups for BI level user rights. AD admins are usually against this, it means more work for them.

2) Just use AD as a user catalog and add all groups and rights information in BI system. This means introducing more work to BI administrators and adding potential security problems. Imagine if BI admin won’t be notified of a person transfer from department to department or from sales to marketing. And why should BI administrator even care about that )

Pro’s and cons of each way are easy to deduct and choice is always organization-dependent, so I won’t stop on that and go right to complains section.

Using Active Directory in Cognos and Oracle BI

Cognos

About a year ago we had a PoC on Cognos, where the requirement was to not just common “users and groups”, but also we had to use custom created AD properties for data filtering.

It’s easy to do in Cognos 8 BI: you just add Framework Manager macro that returns this property. Piece of cake really. And it turned out to be hard for our competitors.

Description on IBM portal, historical number — 1027162

Oracle

Recently I’ve had to do the same with Oracle BI and it turned out that Oracle BI’s current integration with Active Directory is pretty basic. The only way to communicate with AD is authentificate users through it, there’s no way to access User groups or custom properties (there is a way, but it requieres changing ActiveDirectory, which isn’t good at all). See Metalink note on this subject: 544828.1

So there’s no built-in way in OraBI to get all those delicious groups and other properties. But there’s always a way out called scripting )

Extracting Active Directory information to csv

You can write a script that will extract needed information from Active Directory to a csv file and then schedule it for nightly extracts, load it into a table and use this table in Oracle BI security variables. This is a reliable solution, but it introduces a time lag between AD modifications and your extracts, which can be important from security point of view. Same example –user is transfered and can still access BI data he’s no longer authorized to see until Active Directory information is extracted again. Setting small extract lags can bring unwanted load on Active Directory, so it’s once again a balanced choice to make.

Links for extracting Active Directory information to csv:

Step by Step Active Directory Export Guide
http://www.rlmueller.net/ADOSearchTips.htm
http://www.computerperformance.co.uk/Logon/LDAP_attributes_active_directory.htm
JXplorer — a graphical tool to browse any LDAP server