There are a few things that can be duplicated in PAD, so I’ll describe each situation accompanied with code I wrote to check for duplicated items. Fortunately, export dialogue shows what it’s checking at failure point, so you’ll know where to look.

Macros

All you need to know about macros is store in P_MACRO* tables. P_MACRO holds general macro info, including it’s name that is potential duplicate. EP developers must’ve overlooked the fact that macro name is case-sensitive from Administration Console point of view and case-insensitive from export deployment wizard’s point of view. So if you have two macros: copy_sales and copy_Sales, you’ll find out about this duplication only when you’ll try to export deployment.

Code checking for such situations is very straight-forward.

SELECT * FROM p_macro WHERE
 LOWER(rtrim(ltrim(macroname))) IN
 (
 SELECT LOWER(rtrim(ltrim(macroname))) FROM P_MACRO
 GROUP BY LOWER(rtrim(ltrim(macroname)))
 HAVING COUNT (*) >1
 )
 ORDER BY LOWER(rtrim(ltrim(macroname)))

Found some duplicated macros this way, but the problem persisted.

Macros are also recorded in PAD xml (see below), but table approach is way easier.

Applications

Applications pose a more interesting problem.

ADMINOPTION table

Most of the application details are stored in ADMINOPTION table of each applications scheme (Oracle) or database (Ms SQL Server). I wrote some time ago about updating JVM options for Contibutor client via this table. In my search for duplicates I was interested whether there are duplicate application ids or display names. So it made sense to query all adminoption tables at once.

Here’s a sample Oracle SQL script for such queries. It’s easy to replicate it for MS SQL server (seems that didn’t publish such examples as of yet, mail me if you need them).

CREATE TABLE temp
(USR_NAME VARCHAR(50) NULL,
OPTIONID VARCHAR(250) NULL,
DESCRIPTION VARCHAR(512) NULL,
OPTIONVALUE VARCHAR(512) NULL);
commit;
BEGIN
  FOR usr IN (SELECT owner FROM all_tables
  WHERE TABLE_NAME = 'ADMINOPTION')
  LOOP
  EXECUTE IMMEDIATE 'INSERT INTO temp (usr_name, optionid, description, optionvalue)
  select '''|| (usr.owner) ||''' USR_NAME, OPTIONID, DESCRIPTION, OPTIONVALUE from ' || (usr.owner) ||'.ADMINOPTION' ;
  END LOOP;
END;
 
-- check for duplicate application_display_name
SELECT optionvalue, COUNT(*) FROM temp
WHERE optionid = 'APP_DISPLAY_NAME'
GROUP BY optionvalue
HAVING COUNT (*) > 1

Strangely, such exercise showed me some duplicated names, but detaching those apps from PAD didn’t help, Export wizard was still complaining about duplicated apps.
So I started SQL profiling to see what was really happening under the hood and was impressed by finding

PAD xml

Turns out, all application details are stored in an XML file located in P_PAD table. This file is actually huge and messy, but it contains some fields that are not available in ADMINOPTION table or Administration console output. Deep inside this file, applications are described like:

<applications>
<application id="{94BF5F99-E6AF-4D9C-BA63-E2303055296B}" name="go_capex_contributor" 
objidref="obj_APPLICATION"><properties><property propidref="pr_DATASTORE_CON_DOC">
<![CDATA[<connection>   
...
  <applicationname>    go_capex_contributor   </applicationname>  
</connection>  ]]>
</property><property propidref="pr_DISPLAY_NAME"><![CDATA[go_capex_contributor]]></property>
<property propidref="pr_UNIQUE_ALIAS"><![CDATA[test]]></property>
...
</applications>

And it turns out that the ‘name’ property held duplicates in my case. Since manually extracting properties is tad too tedious for me, I wrote a simple python script that does exactly that. It’s not a complete solution, more an approach sample. I’m using pyparsing as usual.

# -*- coding: utf-8 -*-
#Function: Parses Cognos Enterprise Planning PAD contents file xml and extract all aplication unique names
#Usage: run python ep_pad_analyzer.py -i file_with_pad_contents -o output_csv_file
#Author: Yuri Kudryavcev email mail@ykud.com
#
import codecs
import csv
import os
import sys
import re
import string
import time
import getopt
import datetime
from pyparsing import Word, Literal, OneOrMore, nums, Suppress, SkipTo ,makeXMLTags, alphas
 
def writeToLog(message):
 print ('%s: %s')%(str(datetime.datetime.now()), message)
#
def SafeFileOpen(filename,mode,encoding):
 try:
 fl=codecs.open(filename,mode,encoding,'ignore')
 except IOError:
 print "Cannot open file "+filename+"\n"
 return (-1)
 return fl 
#
def main(argv):
 """input format: python ep_pad_analyze.py -i c:\pad.xml -o result.csv -l log_file.txt
 
 Arguments:
 -i or --input file, grabbed from P_PAD table of Contributor Database
 -o or --output -- file for pad info csv format
 -l or --log log file
 -h or --help This message
 """
 WorkLogFile=""
 LogDirectory=""
 inFile = ""
 OutDirectory=""
 out_put_file = "pad_applications.csv"
 # getting command line arguments
 try:
 opts, args = getopt.getopt(argv, "hi:o:l:", ["help","input=","output=","log="])
 except getopt.GetoptError:
 print main.__doc__
 sys.exit(2)
if len(argv) <= 1:
 print main.__doc__
 sys.exit(2)
 for opt,arg in opts:
 if opt in ("-h","--help"):
 print main.__doc__
 sys.exit(2)
 elif opt in ("-i","--input"):
 inFile = arg
 elif opt in ("-o","--output"):
 out_put_file = arg
 elif opt in ("-l","--log"):
 WorkLogFile = arg
 
 if WorkLogFile=="":WorkLogFile="pad_analyze_log.txt"
 #setting up logging
 WorkLogFilefl = SafeFileOpen(WorkLogFile,"w",'utf-8')
 out_put_fl = SafeFileOpen(out_put_file,"wb",'utf-8-sig')
 #backup_stdout = sys.stdout
 #backup_stderr = sys.stderr
 #sys.stdout = WorkLogFilefl
 #sys.stderr = WorkLogFilefl
 english_chars = u''.join(unichr(x) for x in xrange(0x041, 0x05B)) + u''.join(unichr(x) for x in xrange(0x061, 0x07B))
 application_unique_alias = Suppress('property propidref="pr_DISPLAY_NAME"><![CDATA[') + Word(english_chars + nums +' ()' + '_-') + Suppress(']]></property>')
 application_id = Suppress('<application id="{') + Word (english_chars + nums + '-').setResultsName("application_id") + Suppress('}" name="') + Word(english_chars+ '_').setResultsName("application_name")# + SkipTo('<applicationname>') + '<applicationname>' + Word(alphas).setResultsName('datastore_app_name') + '</applicationname>'
 writeToLog("Starting scanning.")
 writeToLog("Input file is: %s .Log file is: %s"%(inFile, WorkLogFile))
 fl = SafeFileOpen(inFile, 'r','utf-8')
 contents = fl.read().encode('utf-8')
 for x in application_id.searchString(contents):
 print x
 fl.close()
 #closing up logging
 out_put_fl.close()
 WorkLogFilefl.close()
 #sys.stdout = backup_stdout
 #sys.stderr = backup_stderr
# 
if __name__ == "__main__":
 #main(sys.argv[1:])
 main(['-i','pad.xml'])

Administration Links

There’s a P_INTERAPPLINK table in PAD, but it doesn’t contain link names, so it’s worthless for duplicate search. But you can use it for some reporting and finding connected applications.

But xml file mentioned above contains all you need, there’s a link description part that goes like this:

<links>
<link_instance id="{B4FC0A77-CD3F-4239-859C-FC4208539B77}" name="application_link_test"
 objidref="obj_LINK_INSTANCE"><properties><property propidref="pr_DISPLAY_NAME"><![CDATA[application_link_test]]></property></properties>
....
</links>

So using the same approach as for applications works and you can find duplicated links without going through all of them in the PAD.

Nothing revolutionary, performance improvements, fighting with Java-based Rich Client memory footprint and performance. Publishing enhancements seem interesting.

Since BI upgrade is required as well, I’d wait for first couple of BI service packs before trying this one out.

• application databases become inconsistent during administration tasks so having a backup of database in a middle of GTP crash doesn’t give you anything, you’ll have to roll back to GTP start time. So backing up before and after administrative task is better.
• publish databases are created via bulk load, which gets rather slow with full logging as log files grow very fast.

In ideal situations, application databases should be in Full Recovery Mode not to lose contributor input and publish databases should be in Simple Recovery mode. But in most cases Simple Recovery Mode and daily backups suffice.

Because otherwise — and it’s oh so common situations — I arrive at the sites with planning databases occupying hundreds of Gb (no, really, I’ve seen another such case just yesterday).

Here’s a simple script that turns all planning databases (selected by cognos_ep prefix) into Simple Recovery Mode and then shrinks them.

USE master;
 
DECLARE 
    DBNames CURSOR
FOR
    SELECT
	  NAME 
    FROM sysdatabases
where name like 'cognos_ep%'
 
OPEN DBNames
 
DECLARE @Name varchar(50)
 
FETCH NEXT FROM DBNames
INTO @Name
 
WHILE (@@FETCH_STATUS <> -1)
BEGIN
	print @Name
	exec ('ALTER DATABASE '+ @Name + ' SET AUTO_SHRINK ON WITH NO_WAIT')
 
	exec('ALTER DATABASE ' +  @Name + ' SET RECOVERY SIMPLE WITH NO_WAIT')
	exec('ALTER DATABASE ' + @Name + ' SET AUTO_SHRINK ON')
	exec('ALTER DATABASE ' + @Name +' SET RECOVERY SIMPLE') 
 
    DBCC SHRINKDATABASE( @Name , 0)
 
    FETCH NEXT FROM DBNames
    INTO @Name
 
END
 
CLOSE DBNames
DEALLOCATE DBNames

And a bottom line: Worry about backups! Hire a DBA (setting up backups is a one day job — pay part-time)

Reasons vary, but mostly it’s that I’ve been, you know, busy-busy. As I now start to reflect on it, it’s always a point of view thing and a question of self-control and ability to say no )

But enough philosphy, brief recap on what happend while I was out “there” in real world. I’ll divide this into two parts (by vendors ))

Big Blue

Made some POCs in autumn, nothing that special (some neat integration, but not cognos-related). Carried on a couple of projects with  EP+BI typicalities. All in all, nothing that new.

BUT I’ve finally completed the other certification path there is at Cognos\IBM, notably, the “BI Professional” road. It’s a deal longer, as of now, you have to pass 3 “role-based” exams, 2 mandatory (Report Author and Metadata Model Developer) and one of you choice to get a chance of passing BI Pro exam. It’s used to coast 10k, which could be cut down to reasonable 200$ if you attended a Cognos Workshop (which costed 2,5k), but thanks to IBM that’s gone and now it’s 200$ from the start. Just opened certifications link and I’m, frankly, quite surprised with new IBM names for old Cognos certifications. According to this I’m:

• IBM Certified Designer — Cognos 8 BI Reports
• IBM Certified Developer — Cognos 8 BI Metadata Models
• IBM Certified Developer — Cognos 8 BI OLAP Models
• IBM Certified Solution Expert — Cognos 8 BI

and it turns out I was an

• IBM Certified Solution Expert — Cognos 8 Enterprise Planning

instead of Cognos EP Professional.

Wording is something you cannot easily take from IBM ) Will put that list in about page, just to get more mail asking for exam questions.

Hint: In comparison with the old Modeler exam, current one is easier.

And in the first 2 months of this year we did a “nightmare-style” POC with EP + TM1 + BI, where I could finally get an idea of what TM1 really is and fully realize some ideas like incremental administration links and, therefore, “real-time” EP -> TM1  data propagation. I’m planning to write about this in separate posts, though. And, boy, it was hard, and we flanked the final presentation due to me being ill and bla-bla, and I’m still recovering breath from this.

But enough about IBM, let’s turn to

Oracle

We’ve made a really nice billion-facts cube with our colleagues from “SportMaster” and talked about it in exceeding detail on Russian Oracle Technical Forum 2008 and Oracle BI Forum 2009. Essbase turned out to be a good engine for this kind of task and the moment when it finally kicked-in on 100gb dataset will retain as one of best in 2008. The fact that one of dimensions was a million rows parent-child didn’t raise that much of a problem.

Moreover on another site we duly built dimension outline with around 30 mln members. Although duly is not the right word, “kicking and screaming” would be a better wording ) And the whole area of trying to tune something in underlying Berkeley DB is still open. But I must admit, an 8 gb outline file is really impressive.

So I’ve spent pretty much time toying around with essbase (except when I wasn’t with TM1)). And I’m also a Hyperion Essbase 9.3 Developer Certified Expert (I won’t enter Prometric for a while, I hope).

So it was a rather crowded time and I hope that nothing will stop me from blogging some details and future encounters. At least, I’d try my best to. Cheers to your patience, folks )

Well, sometime it helped and we now have allocation tables in administration links. Although I’m sure my post didn’t have any impact, some deem hope remains

I’m going to repost this into Communities and into Insight. And, maybe, we can create some noise by voting for these enchancements in Insight aka registering those enchancements again (they get piled up together, as I’ve found out).

Some those two echs are:

1) Easy as: “Add the ability to import Calculation Options and Weighting while importing a dlist”

If you’ve ever done some external dimension management, you know that:

- you have to use unique names to keep dlist IID the same when names (but not unique codes) of items change

- you can import formulas as calculation text, just put item names in curly brackets )

- if you use Update\Remove obsolete it always deletes all Calc Options and Weighting from dlist. That means that you cannot update any cacluation dimensions, such as Accounts. 

So you manage all simlple dimensions by updating them (with Remove Obsolete selected) and unique names, and you manage all dimensions with weighting by manually changing them. 

Shouldn’t be like that.

2) Even simpler: Add a Contributor macro for importing translations

It seems weird to have a single table with translations for Cognos BI and then to manually (or semi-automatically by text import) copy it into Contirbutor application. Shouldn’t be hard, ability to import translation text is already there, just need a macro step

3) And another one (maybe no one will notice): Import deployment packages as a macro step too  

Automating dev-prod. backup-restore — all those things that should be automagical

So, if these things annoy you as well — drop me a line, and I’ll send you Cognos Insight ench numbers to vote for.

Well, with a huge lot of bugs inside, it’s time, gentlemen, to power up virtual machines and start testing it all over. We need to log on maximum of bugs before sp1.

New enterprise planning goes under Cognos 8.4 (BI version was out a month earlier), and this minor number increase doesn’t reflect the amount of changes. They’d better start it of with 9.0 it seems.

Today I just want to add some of points:

1 If you’re up to publishing some serious amount of data, variant with adding a dummy dlist with 1 item and using it as dimension for publish seems a very bad idea.  You’re terribly slowing proccessing and wasting tablespace. So you have to choose a dimension for publish to reduce table size and speed things up

2 Dimension for publish must be:

a Stable. It shouldn’t change or change very rarely, since these changes will cause Framework Manager models \ ETL models changes.
b Have sane (~3-33) number of elements. If less than 3, there’s no win in performance. If more than 30, ensure it won’t change, because big dimensions change more often than small ones.

3 There are dlists that are naturally stable aka measures, like {Quantity;Price;Sales}

4 In other cases timescale seems a really good choice. For example, if months dimension doesn’t have year signs in name (like Jan, Feb, Mar), columns in publish won’t change ever.  To work with timescale published data you can use sql unpivot queries, that will virtually turn it into view publish, but it’ll way more effective than publishing it by 1elem dimension in first place. I’ve settled for this variant in my current project. Moreover, if you wrap this unpivot sql into ETL procedure, you can treat all published data uniformly while loading into datamarts (publish all needed cubes on timescale, use the same etl procedure).