Really, PowerPlay cubes building via scripts is a so 1990s, 21st century is for enterprise scale, failover, monitoring and you-know-the-pitch.

If we make Transformer cube build a Report/Job in Cognos Connection, we get:

• load-balanced cluster builds (cube building task will start on any avalaible server, we need to ensure server will be able to run the build script)
• monitoring (the same Cognos System \ Schedule pane in Administration) and build history
• event-management (builds will be recorded in audit tables and we can do notifications, restarts and scheduling in Event Studio)

The question is how to make cube build a Job / Report run with minimum of hassle. I was originally thinking of a simple ASP page, running a cube build batch script, used as a web-service datasource with cubename as parameter, but (that’s the a-ha moment) you can use a simple Java application, running in the same application server as Cognos itself. That’ll give platform independance and nice url (ibmcognos/buildcube.jsp?cubename=…).

Moreover, you can package cube build scripts into the application itself, make them configurable (model location, backup location, number of copies to retain, etc), make the application detect Cognos version (to know how to swap cubes) and detect operating system (what scripts to use) and you’ll have all the cube build logic packed into a neat single file container that you just copy accros all servers in cluster. That’ll take away all script maintenance burden and give all the benefits of centralized enterprise way. And it’s not a big development effort, I’d estimate it a couple of weeks spent with coding hat on.

Strangely that neither of big “all-things Cognos” shops (BSP or Motio) didn’t do something like that, there’s still a huge lot of PowerPlay shops around that’d love something like this. And some of them are going from 8 to 10 right now and that requires rewriting parts of cube-building scripts

And any post is better with a cat picture in it, don’t you agree? This one is about a casual TM1 developer embrasing the ocean of MDX possibilities.

But MDX-dialect of TM1 isn’t well-documented (again, being very polite), so learning it’s intricacies is mostly trial and error and an open field for experiments. And whilst you can write MDX-driven subset definitions (wonderfully described by Phillip Bichard and Martin Findon) in Architect, there’s no actual tool where you can write “select years on rows, measures on columns from sales” and see you table (all the Cognos studios do that, but in background and with quite inefficient way of writing you own). You can go wild on the ODBO page of a TI process as well, but that’s a bit hard from error handling and results viewing.

So at the end of a day I wrote a small Java based script that allows you to write MDX queries to your TM1 server and get result back as a csv file (there’s an Excel workbook doing the same thing, but I’m a Java guy and I needed the querying code to conquer the universe anyway).

How does it work?

1) You fill in a query you want to fire in an xml file with some additional options (admin server, login/password, file to write query results to, will you need headers and all that)

2) Run java tm1_mdx your_query_file.xml and get either an incorrect MDX syntax error (bad, I get this often) or results in a CSV file (good)

As usual, there’s source code in the package, so you can adjust it \ use it as a base for something bigger.

How to install it?

1. You will need  Java runtime engine (jre). This the only requirement, there’s no need to have TM1 installed or anything.
2. Download the tool (this package is for 9.5.2, ping me if you want it for some newer\older version, it’s just a matter of recompiling), unpack to some directory (c:\tm1mdx, for example)
3. Cryptic step of adding security certificate to your Java.
1. Find jre\bin\keytool.exe utility on your computer
2. Open the command line and navigate to jre\bin
3. Run command line keytool -keystore ..\lib\security\cacerts -alias Company -import -file c:\tm1mdx\applixca.der
4. Enter ‘changeit’ as password
5. Type yes
4. Fill in the mdx_query.xml
5. Open command line, navigate to c:\tm1mdx, run java tm1_mdx mdx_query.xml

PS: By the way, I glimpsed into TM1 10.1 API docs and all the APIs were greatly enhanced in this version (mostly for Performance Modeler, it seems), so you finally have a per line grained access to rule files and such. It’s obvious that Performance Modeler uses Java API (it’s using Cognos Rich Client Platform, it’s inevitable),  so it’s extensively used and developed and might even get documented at some point )

PSPS: It’s interesting whether there’ll be any IBM-made XML/A driver for TM1 at some stage. It seems possible to do a  straw olap4j driver for TM1 even now and that’d be a funny way of opening it up a bit further.

Built-in Cognos Security administration is quite awkward (compared to so long gone Access Manager, sigh), so questions like above are so frequent. There are 3 main solutions:

1) Buy an externally developed tool from BSP or Motio. I had a chance to look at BSP MetaManager recently and it does a whole lot of stuff to make your administration life easy. Bit of an overkill to just ask “who’s there”, but totally worth it if you’re going big scale \ long term.

2) Buy an SDK license and use the sample script from IBM as a base and extend it just for this task and then use SDK left and right to do anything you want in Cognos. If you have an SDK license through some global license deal — the sample script maybe just what you need to get the job done.

3) Write SQL queries to Content Store database to return just this info. It’s an unsupported (I’ll even stress this) unsupported way of doing it, but if you just want to have a quick look or are happy to loose it at some upgrade point (although it works on both 8.4 and 10.1, CS database schema isn’t changing that much), here are the scripts for both MS SQL Server and Oracle.

I used the script from this SQL.RU topic as a base and added group nesting unwrapping with Connect By in Oracle and CTE’s in SQL Server. You can turn this SQL into FM query subject and have a nifty report.

Oracle

SELECT DISTINCT parent_group_id,
                  group_info.name group_name,
                  user_info.ldap_id,
                  user_info.last_name,
                  user_info.first_name,
                  user_info.email,
                  user_info.user_id
                  FROM
    ( --unwrapping groups and roles hierarchy, by recursively connecting all child roles or groups to parent
    SELECT rel.cmid parent_group_id, grp.cmid child_group_id  FROM
    cmreford1 rel, cmobjects grp
    WHERE
    rel.refcmid = grp.cmid
   AND grp.classid IN (26, 54) -- groups and roles
    CONNECT BY PRIOR  rel.cmid = grp.cmid
    -- adding groups that just have users directly without any parent groups
    UNION ALL
    SELECT rel.cmid parent_group_id, rel.cmid child_group_id  FROM
    cmreford1 rel, cmobjects grp
    WHERE
    rel.refcmid = grp.cmid
    -- users
   AND grp.classid IN (5)
   -- no parents
   AND rel.cmid NOT IN
   (
    SELECT rel.refcmid child_group_id  FROM
    cmreford1 rel
   )
    ) grp_unwrapped,
    cmreford1 user_group_relation,
    ( -- user information
    SELECT c33.cmid user_id, c33.NAME AS ldap_id,
                     c1.surname last_name, c1.givenname first_name,
                     c1.email email, c1.objid
               FROM cmobjprops1 c1
               LEFT JOIN cmobjprops33 c33 ON c33.cmid = c1.cmid
               UNION
               SELECT cmobjprops1.cmid user_id, cmobjprops33.NAME AS ldap_id,
                      cmobjprops1.surname last_name,
                      cmobjprops1.givenname first_name, cmobjprops1.email,
                      cmobjprops1.objid
                 FROM cmobjprops1, cmobjprops33
                 WHERE cmobjprops1.cmid = cmobjprops33.cmid
             ) user_info,
    ( -- group_information
      SELECT cmobjnames.cmid cmid, cmobjnames.NAME AS NAME,
                    cmobjects.created created, cmobjects.modified modified,
                    cmobjects.disabled disabled,
                    cmobjects.classid
               FROM cmobjects, cmobjnames
               WHERE cmobjects.cmid = cmobjnames.cmid
                    AND cmobjects.classid IN (26, 54)
                     --search in english groupnames only, remove this filter to search any language
                     AND  cmobjnames.mapdlocaleid = 24
    ) group_info
  WHERE (1=1)
  AND user_group_relation.cmid = grp_unwrapped.child_group_id
  AND user_group_relation.refcmid = user_info.user_id
  AND group_info.cmid = grp_unwrapped.parent_group_id
 
-- add your filters here
-- search by group name like
 
AND group_info.name LIKE '%Planning%Users%'
 
-- or search by user name like
 -- and user_info.ldap_id like '%Jack%Smith%'

MS SQL Server

WITH groups_unwrapped (parent_group_id, child_group_id, lvl)
AS
(
SELECT rel.cmid parent_group_id, grp.cmid child_group_id, 0 lvl FROM
cmreford1 rel, cmobjects grp
WHERE
rel.refcmid = grp.cmid
AND grp.classid IN (26, 54)
-- highest ranking groups, without any parents for anchor expression
AND rel.cmid NOT IN
(SELECT grp.cmid child_group_id FROM
cmreford1 rel, cmobjects grp
WHERE
rel.refcmid = grp.cmid
AND grp.classid IN (26, 54)
)
UNION ALL
SELECT rel.cmid parent_group_id, grp.cmid child_group_id, lvl + 1 FROM
cmreford1 rel, cmobjects grp, groups_unwrapped unwp
WHERE
rel.refcmid = grp.cmid
AND grp.classid IN (26, 54)
AND rel.CMID = unwp.child_group_id
)
SELECT DISTINCT parent_group_id,
group_info.name group_name,
user_info.ldap_id,
user_info.last_name,
user_info.first_name,
user_info.email,
user_info.user_id
FROM
(
SELECT * FROM groups_unwrapped
-- adding just groups with users, without parents
UNION ALL
SELECT rel.cmid parent_group_id, rel.cmid child_group_id, 0 lvl FROM
 cmreford1 rel, cmobjects grp
 WHERE
 rel.refcmid = grp.cmid
 -- users
 AND grp.classid IN (5)
 -- without any parent groups
 AND rel.cmid NOT IN
 (SELECT rel.refcmid child_group_id FROM
 cmreford1 rel
 )
)
grp_unwrapped,
cmreford1 user_group_relation,
( -- user information
SELECT c33.cmid user_id, c33.NAME AS ldap_id,
c1.surname last_name, c1.givenname first_name,
c1.email email, c1.objid
FROM cmobjprops1 c1
LEFT JOIN cmobjprops33 c33 ON c33.cmid = c1.cmid
UNION
SELECT cmobjprops1.cmid user_id, cmobjprops33.NAME AS ldap_id,
cmobjprops1.surname last_name,
cmobjprops1.givenname first_name, cmobjprops1.email,
cmobjprops1.objid
FROM cmobjprops1, cmobjprops33
WHERE cmobjprops1.cmid = cmobjprops33.cmid
) user_info,
( -- group_information
SELECT cmobjnames.cmid cmid, cmobjnames.NAME AS NAME,
cmobjects.created created, cmobjects.modified modified,
cmobjects.disabled disabled,
cmobjects.classid
FROM cmobjects, cmobjnames
WHERE cmobjects.cmid = cmobjnames.cmid
AND cmobjects.classid IN (26, 54)
--search in english groupnames only, remove this filter to search any language
AND cmobjnames.mapdlocaleid = 24
) group_info
WHERE (1=1)
AND user_group_relation.cmid = grp_unwrapped.child_group_id
AND user_group_relation.refcmid = user_info.user_id
AND group_info.cmid = grp_unwrapped.parent_group_id
 
-- add your filters here
-- search by group name like
 
AND group_info.name LIKE '%Consumer%'
 
-- or search by user name like
-- and user_info.ldap_id like '%Jack%'

‘Open download and use but paid support’ policy advocated by Oracle (for example) can actually attract more people to work with the products and to buy them eventually, even when they are already at a different company. It really worked well on russian DBMS market, where Oracle is the only ‘big enterprise’ choice.

On the other hand, both BI and Express are quite complex products, so mere effort of trying to make them work in non-obvious tasks can lead to utmost frustration. But both of them are packed with demos and samples, so that should make life a bit easier.

All in all, I’m really glad this is happening and am rooting for time-unlimited trials. You won’t build a serious system without fix packs anyway )

How to do it:

1) go to server, open command line, type systeminfo (processor info will be in first ten lines). Or if you don’t like it cool, open Control Panel, CPUs are described on second pane )

2) check whether your CPU’s are in this list

3) ask you server admin, or manually reboot server, look into BIOS and turn HyperThreading off

4) Test your TM1 processes, they may speed up considerably

What’s behind the scenes.

Hyper-threading, in a nutshell, is a technique showing each physical CPU core as 2 logical ones. Since during normal work a lot of time is spent in thread switching and related registry load/unload and yada yada, physical cores actually are underloaded in common multithreaded system. Adding a duplicate registry set and a ‘virtual’ core allows to utilise physical core up to 20-40% more, earning throughput benefits.

Key words here are “threading” and “throughput”, meaning that HT benefit multithreaded applications where lots of small request are processed at the same time. OLTP databases are a prominent example.

But most of the systems I work with, almost all OLAP engines and even DWH-tuned DBMSes, actually suffer in this scenario.

For example, TM1 calculations are executed in a single thread, so if this thread is assigned half an CPU core, speed drops significantly. The same logic goes for Essbase metadata update process, for example, or for Cognos Enterprise Planning Job processing.

I’ve seen significant performance degradation (30%) in Cognos EP job processing due to HT context switching and have advised turning HT off on all Cognos EP Job servers.

Encountering the same issue with TM1 recently left me puzzled, since there was plenty of empty CPUs on server and there shouldn’t be any thread switching. But we’ve got 2+ time speedup after flickering the switch anyhow )

I promised a bonus and here it goes:

When buying a TM1 server with Intel CPUs, buy way more cores than you’ll license. Physical cores, not logical.

This will turn on Intel Turbo-Boost and it might give you another 30-40% speed boost. I haven’t tried that yet and there’s a bit of scepticism of current boost-detection approach in Wikipedia, but it’s definitely worth testing in a lab before you buy hardware. Cores are cheap compared to licence costs, so you may save yourself a ton of money by having a faster system. As in wikipedia example, for Core i7-2920XM you can boost up to 3,5 Ghz per core with 2,5 Ghz base speed. That’s a hefty speed up for a good old one-threaded TM1.

Well, the biggest change was trivial: I’ve forgot everything

But that turned out to be good news, since version 8.3 to version 10 changes are huge. I remember reading SDK Developer Guide overnight for 8.3, but now it’s 3,500+ pages long! As it turns out, you can still read all the needed stuff in a short time period, but understanding what to read is a challenge.

So it’s best to start with this proven practice article, providing some kind of helicopter view of what can be achieved by what parts of SDK.

From there, I suggest you dive into sdk\samples folder and look through the code provided there. However complicated it may look, it’s actually way more integrated than any Knowledge Base\Technote code sample you can download, since extending those turns into rewriting the built-in samples functionality. Trust me on this, I’ve spent half a day doing exactly that before giving up and building upon existing samples.

Where to look when you’re stuck trying to understand a code piece or function? I used javadoc generated documentation instead of Developer Guide, because it’s Java-specific. I used Developer Guide just like an overall Class&Methods reference.

Some more SDK sample links:

Cognoise SDK for Cognos 8 forum

Cognoise SDK for Cognos 10 forum

IBM SDK-related articles and Technotes

PS: It’s funny, I’ve wrote both TM1 Java API code and BI 10 SDK in last three months ) Can’t think about any combined methods that’d be useful. Any ideas?

Just a quick Cognos BI hint: you can use session variables to store project-level constant values.

1) Add a session variable to your Framework project, write your required constant value there. Like ‘servername’ = ‘awesomebi’

2) Use it in Report Studio, just type

#sq($variable_name)#

in expression editor. sq encloses string in single quotes.

Overall approach excites coder part of me: modular development, agile, best practice source code — all this is actually very helpful. Having so much Turbo Integrator ‘how-to’s solves a lot of ‘re-inventing the bicycle’ cases.

As far as I understand bulk of this project’s code is developed by CubeWise. Hats of to them for being so open and willing to push all TM1 community further.

It’s interesting to see whether this site will gain momentum — whether there’ll be non-CubeWise commiters, how will adoption go and etc.

I would be proud to contribute some code there someday, if I’ll be doing TM1 related stuff. Maybe they will accept my small utilities for start….

It seems that the easiest way to gain full article list is to filter by cognos_business_intelligence tag. This reminds of one of my former bosses who inquired Oracle, whether they had special training on how to find content on their site.

I ended throwing RSS-feed for this search in my reader, so I hope not to miss any updates.