Cognos BI

Oracle BI

Using Active Directory in Cognos and Oracle BI

Cognos

Oracle

Extracting Active Directory information to csv

Links for extracting Active Directory information to csv:

Step by Step Active Directory Export Guide
http://www.rlmueller.net/ADOSearchTips.htm
http://www.computerperformance.co.uk/Logon/LDAP_attributes_active_directory.htm
JXplorer -- a graphical tool to browse any LDAP server

You can use macro functions in Report Studio to do some manipulations with prompt values before they are used for actual processing in filters, report functions, etc.
As a general (and very common) example, you can use macro functions to do datetime format transformations, to allow one various filters to be based on a single prompt (selecting 'Version X, year 2009' in prompt, which you can split up to two filters, for year and version) and do some other string conversion stuff.
Usually such need arises, when you have drill-through reports with different source-target dimension member formats, such as OLAP (Transformer, TM1, Ms AS, Essbase) -> Relational or vice versa. For example (a very simple and not close to real-life), if you have a detailed transaction-level report based on relational database and a MOLAP-aggregated data on top of it.

Fictional Example

If, for example, your date MUNs in OLAP are like '2010-02-10' and relational database expects something like '2010.02.10', you can change string value of prompt by changing filter in detail report from

date = ?date_prompt?

to

date = #subtitute('-','.',prompt(date_prompt,'string','1'))#

prompt macro function returns prompt value and substitute subtitutes strings )

Real-life example

One of real-life examples, where I had to use macro functions extensivly was OLAP(Essbase) to Relational drillthrough setup.
When using Essbase it's better to have unique member names for dimension elements since this speeds query time a bit. So usually some unique identifiers are added to common conformed dimensions. On this project we've constructed a cube with financial transactions, with a number of participants for each operation. The same company can be a client in some operation and a supplier in another. So unique postfixes were added to each of dimensions in Essbase cube. Relational model for Oracle, however, had a single "Party" dimension table, which contained all possible participants.
This problem is easily solved with tecnique given above, I've just removed unique identifier.
The other problem in the same case is that Essbase dimensions usually have hierarchies ) Since operation participants can be aggregated by industry or region I can receive industry (or region) label in dimension prompt (building special prompts for each level is just impossible).
So I had to detect the level of member I've received (which was easier by having those unique level identifiers) and then to filter accoding column in relational table. So I can receive values like 'Pharmaceuticals SUPPLIER INDUSTRY' or '10241235 SUPPLIER ID' from Essbase. Creating such a filter turns into filter like:

when (#sq(sb(csv (grep(': SUPPLIER_INDUSTRY';array(prompt('SUPPLIER_ID';'string';'1')))) ))#,'[]') THEN
([Relational Souce].[Supplier].[SUPPLIER_INDUSTRY] = #substitute(': SUPPLIER SUPPLIER_INDUSTRY';'';prompt('SUPPLIER_ID';'string';'1'))#)
when (#sq(sb(csv (grep(': SUPPLIER ID';array(prompt('SUPPLIER_ID';'string';'1')))) ))# ,'[]') THEN
([Relational Souce].[Supplier].[ID]= #substitute(': SUPPLIER ID';'';prompt('SUPPLIER_ID';'string';'1'))#)
else (1=1)

This really looks awfull, but it's the only way I could find out to do 'search': that is to form an array from prompt values and do a 'grep' over it and compare it with empty string. But it does the trick nicely.

Oracle BI EE Pro's:

1) Aggregate navigation

-- ability to set up aggregate tables in BI EE itself, which gives it 'aggregate awareness' while generating SQL. So if you have monthly sales aggregate you just set up BI metadata accordingly and all reports will target this table for monthly data.
There's no such feature in Cognos metadata setting.
But using this feature arises some questions:
a) whether aggregate definition should be BI-tool specific. A lot of practioners, including Kimball, insist that aggregate navigator should reside at database level so that every tool querying datawarehouse will benefit from aggregate avalaibility. This seems rather reasonable, since every datawarehouse is usually queried by more than one analytical application (BI, datamining, more BI) so you either direct all applications to BI EE, or agree to performance degradation.
b) aggregate tables desynch. When you just update the base fact table, aggregate tables become "out of synch", providing incorrect query results untill they're recalculated. This means that you either can guarantee that nobody will access reports in this period, or you can have incorrect data. Since more&more datawarehouses squezee load windows to reach real-time this problem gains priority.
c) choosing which agregate to use for answering questions. That's what statistics is all about in dbms, knowing number of rows and value distribution (lol). Oracle BI EE has the "table row count" feature (although I haven't seen it affect SQL generation yet, need more examples). But there's no value distribution analysis in there, so it's just one side of the coin (not talking about I\O device speed and other characteristics).

In general, it's recommended to use database-specific aggregate table functionality (Oracle Materialized Views or DB2 MQT) since it solves all the 3 questions given above and usually simplifies ETL process (and sometimes even speeds it up, since databases use their own transaction logs to detect what data has changed and what aggregates should be rebuilt). Too bad indexed views in Ms SQL do not work with aggregate dimensions (there's no way to define dimension hierachies there).

2) Cache management

-- OraBI has really profound cache management facility. You can "cache" any database query to OraBI specific storage structure (cache file), wich will allow subsequent queries to the same table\query to run without actual database request being made. This can greatly speed things up. I especially like the Event Polling Table feature: you add a table, which records when dwh table was last updated. OraBI then reads at given intervals and automatically invalidates old cache entries, based on this table records.
Moreover, if you have OraBI cluster this cache can be shared among servers.

There isn't anything even close in Cognos.

Although I greatly like this feature, I just want to warn about overusing it. It's easy to imagine BI developers boosting performance by adding more&more cache untill OraBI becomes a fully blown aggregate system. And sometimes it's just about 1 aggregate table at the dwh level ) Or about introducing OLAP server in the enviroment )))

3) SQL generation.

It's tricky subject, but for now I like OraBI generated SQL more. But it's "apples to oranges" for sure, since I usually use DMR's in Cognos which encumbers SQL greatly and there's nothing identical in OraBI EE.

Oracle BI EE Con's:

1) Multidimensionality

Cognos has Analysis studio and the ability to navigate hierarchies in both directions (you won't believe it, but in OraBI there's no 'Drill-Up', only'Drill-Down'). And DMR's and analytical functions (but their usage is a bit annoying, as it seems now, hope to write about it later). Anyway, Cognos is way much more 'multidimensionally-ready' than OraBI.

OraBI + Essbase is a work in progress and has a huge number of caveats (some fixed by patches, some not, some introduced). And the only way to use all Essbase functionality is to write direct MDX via Evaluate functions. That's a big problem, since it's hard for us to suggest OraBI on top of Essbase for now (till 11g once again). The only alternative is Visual Explorer, which is a very good tool, but it for top-analysts only (thick client, costly).

2) Metadata model development

Instead of OraBI's 'only-star schema', '3 layers of model' Cognos FM Manager doesn't impose any design principles, which allows more mistakes, but it makes some things way more simple. One of the first things I wanted to do in OraBI was a report using just a single table.  Well that's a really funny exercise (see posts over here) if a star schema is a must.

But the main problem is the lack of API for metadata changing and browsing. There's udml, but it's not supported officially. Therefore all current integration scripts (like adding users, merging repositories and working with hierarchy depth changes) are out of the law. Which doesn't stop anyone, but is pretty annoying.

3) Pixel-perfect reports

Oracle BI Publisher is a specific tool, aimed at generating a huge number of formatted reports, based on XML format definition files. It wasn't a part of Siebel BI, so 'integration stitches' still stand out. It's a nice tool, but it certainly lacks web-interface ) Therefore in Oracle BI there's a deep distinction between a simple formatted report (with lots of possible logic in it) and making this report 'printer-friendly' since for the latter you basically have to start from scratch by opening Ms Word )  This will change in 11g as they say )

Having said all that, I really wait for Oracle BI 11g edition to start using it with Essbase and I kinda like the product as it is for "relational-only" reporting.

I surely wanted to write a simple bullet point list at first )

Those who aren't struck by occasional CAM-AAA-0071 “An internal error occurred” which stops the whole server can skip this post.
This error was discussed at:

http://www.ibm.com/developerworks/forums/thread.jspa?threadID=244357

http://www.cognoise.com/community/index.php?topic=4971.0

The story goes like this:
- java.exe (that powers BI), grows up to the point where it cannot obtain contigious memory from server and crashes
- Cognos BI doesn't recognize this error, so it throws out a general CAM-AAA-0071
- Cognos BI is unaccessible until restart

This error occurs more often in heavy-usage enviroments. In one of our projects, with lots of Event triggered admin-links and reports, it took only a coulple of hours before it striked.

Solution is not complete, but as I see it now:
1) Put the latest SP on BI (or a special hotfix, if one is avalaible)
2) Lower the avalaible memory for Cognos BI service (to 768 mbs)
3) Modify the HeapDecommitFreeBlockThreshold registry property

It's interesting if 8.4 is prone to such error.

PS: I'm a certified Cognos Technical Specialist now, as well. Need to get first-line support partnership, you know )

Quote from Architecture and Deployment Guide.

Depending on the size of reports and the amount of available memory, IBM Cognos 8 may access a physical disk when processing reports. To improve performance, you can ensure that report processing uses available memory rather than disk space. Using memory instead of disk space is particularly beneficial in cases where temporary files are
created on IBM Cognos 8 servers, causing information transfer from memory to disk. You can monitor the occurrence of temporary files using the c8_location\temp directory.Monitor this folder during report processing periods to determine whether temporary files are created as cclvpage*.tmp. To ensure that IBM Cognos 8 uses memory instead of disk space, in the rsvpproperties.xml file, edit the VirtualMemoryDiagnostics property to use unlimited memory (value = 2) rather than limited memory (value = 0):
<property>VirtualMemoryDiagnostics</property>
<value type="long">2</value>
Note: Remove the comment to enable the VirtualMemoryDiagnostics property.
For information about using the rsvpproperties.xml file, see "Advanced Report Processing Configuration Settings" (p. 96).

But it's strangely slow (tried on a couple of computers) and all in all looks kinda "alpha" and plugged in at last moment. Installation is not that easy too, it requires war building and deployement, procedures that not every BI consultant is aware off.

So I think we should wait for another couple of service packs and there'll be a new standard tool for portal page development. XCelcious should be afraid )

The problem, described shortly is:

1. you have 2 databases in MS SQL residing on the same physical server

2. you create a join “table A from db1 with table B from db2”

3. Cognos creates two select queries (select * from db1.A) and (select * from db2.B) and then does join locally on server. That’s terribly slow. Cognos should be generating one select query, performing the join on the db server and that’d be really fast.

The answer is simple: there’s no way Cognos can find out that these two databases are on the same physical server. But we can help old fella a bit, setting a datasource property (Content Manager Datasource) the same for both datasources. Therefore, the same datasource prefix will be generated for both tables and they’ll be regarded as residing on the same server. Query speed will increase dramatically.

Morale:  always set the datasource property for MS SQL databases located on the same server. 

Gave me some trouble recently, when language changes led to unappropriate publish folder name. Moreover, we were not able to publish anything at all )

Changing this file helped. Found via FileMonitor from sysinternals, as usual.

Reasons vary, but mostly it's that I've been, you know, busy-busy. As I now start to reflect on it, it's always a point of view thing and a question of self-control and ability to say no )

But enough philosphy, brief recap on what happend while I was out "there" in real world. I'll divide this into two parts (by vendors ))

Big Blue

Made some POCs in autumn, nothing that special (some neat integration, but not cognos-related). Carried on a couple of projects with  EP+BI typicalities. All in all, nothing that new.

BUT I've finally completed the other certification path there is at Cognos\IBM, notably, the "BI Professional" road. It's a deal longer, as of now, you have to pass 3 "role-based" exams, 2 mandatory (Report Author and Metadata Model Developer) and one of you choice to get a chance of passing BI Pro exam. It's used to coast 10k, which could be cut down to reasonable 200$ if you attended a Cognos Workshop (which costed 2,5k), but thanks to IBM that's gone and now it's 200$ from the start. Just opened certifications link and I'm, frankly, quite surprised with new IBM names for old Cognos certifications. According to this I'm:

• IBM Certified Designer -- Cognos 8 BI Reports
• IBM Certified Developer -- Cognos 8 BI Metadata Models
• IBM Certified Developer -- Cognos 8 BI OLAP Models
• IBM Certified Solution Expert -- Cognos 8 BI

and it turns out I was an

• IBM Certified Solution Expert -- Cognos 8 Enterprise Planning

instead of Cognos EP Professional.

Wording is something you cannot easily take from IBM ) Will put that list in about page, just to get more mail asking for exam questions.

Hint: In comparison with the old Modeler exam, current one is easier.

And in the first 2 months of this year we did a "nightmare-style" POC with EP + TM1 + BI, where I could finally get an idea of what TM1 really is and fully realize some ideas like incremental administration links and, therefore, "real-time" EP -> TM1  data propagation. I'm planning to write about this in separate posts, though. And, boy, it was hard, and we flanked the final presentation due to me being ill and bla-bla, and I'm still recovering breath from this.

But enough about IBM, let's turn to

Oracle

We've made a really nice billion-facts cube with our colleagues from "SportMaster" and talked about it in exceeding detail on Russian Oracle Technical Forum 2008 and Oracle BI Forum 2009. Essbase turned out to be a good engine for this kind of task and the moment when it finally kicked-in on 100gb dataset will retain as one of best in 2008. The fact that one of dimensions was a million rows parent-child didn't raise that much of a problem.

Moreover on another site we duly built dimension outline with around 30 mln members. Although duly is not the right word, "kicking and screaming" would be a better wording ) And the whole area of trying to tune something in underlying Berkeley DB is still open. But I must admit, an 8 gb outline file is really impressive.

So I've spent pretty much time toying around with essbase (except when I wasn't with TM1)). And I'm also a Hyperion Essbase 9.3 Developer Certified Expert (I won't enter Prometric for a while, I hope).

So it was a rather crowded time and I hope that nothing will stop me from blogging some details and future encounters. At least, I'd try my best to. Cheers to your patience, folks )