With mostly very cryptical error messages or without any.
One of the problems might be just that DataManager service (which does the transfer) cannot allocate enough memory for operation. This is what we’ve encountered just today.

Good news are that this parameter is easily configurable, you can just edit cognos_install_path\bin\dataimportserviceconfig.xml file and set JMX (Java maximum allocated memory) to something more appropriate than default 128M (1024 is a nice number, for example).

Try this if your links fail

PS: In this ‘wonderfull’ Java world, setting JVMX is a common boost-up technique. And there’s a lot of places to apply it. Think about Cognos Rich Client Platform (Cognos 8.4 Contributor) for start…

Some basic definitions:
Cognos Enterprise Planning (aka Adaytum EP) is a very scalable system, all computing is client-side, nice workflow organization, nice GUI (version 8.4 Eclipse one especially) with huge limitations when it comes to the moment of “show me the whole budget in one place” or to serious what-if modeling. But scaling is trivial, each server acts as many clients simultaneously (number of cores\CPUs) and adding more servers scales the system linearly. So to build really scalable systems, we made Contributor only working places, with all analysis done in Cognos 8 BI and Analyst was turned into a modeling tool. In adaytum and cognos 7 times a lot of things(consolidating, what-if) could be done (and to be honest, still can be) in Analyst, but just up the point you reached 40mln cells cube limit in Analyst, then you started dividing cubes and the system became nightmare. Transforming the same consolidation task into series of Contributor applications connected by admin links gave huge performance enhancements, but the “show me the whole budget” limitation was a major problem.
Cognos TM1 (aka Applix TM1) is an in-memory OLAP engine, ultra-speed calculations, server side computing, excel based gui (with web publication), limited workflow capabilities (compared with current EP), scalability limitations (imagine putting servers across whole Russia to ensure response time)

So when it comes to a planning system with large user base – it’s EP, but if some serious modeling and calculations are requied as well – it’s time to fire up TM1 as well.

Overall interaction scheme seems pretty simple:

1) Most of users work in EP inputting budgets, submitting and doing all other workflow related activities
2) After some user actions (saving budget, submitting, accepting it) data is transferred into TM1.
a. Detecting the desired action is easy – we’ve got Event Studio for that (see here and here).
b. Incremental publish is fired and changed elist is republished – that’s pretty fast
c. Then there’s a problem to extract only changed data from publication. That’s the same as with incremental admin links, you just write a simple database procedure that logs last data transfer time and create views showing only recent data (as usual, you want code samples – drop a line)
d. Changed data is bcp’ed\sqlloaded to TM1 server
e. TM1 TurboIntegrator process is fired, loading the extracted data into TM1
3) Analysts work with “live” data in TM1, doing what-if analysis (they dynamically add scenario versions for this purpose), having access to all enterprise budget and actual data.
4) Then a version could be used for top-down propagation and exported into EP (that’s pretty simple, just some writing into EP applications import tables)

The whole system seems pretty healthy, data transfers happens with 1-2 minutes latency (we used Save as trigger action). Doing any kind what-if analysis on a budget version only 2 minutes “stale” with ultra-speed seems very impressive to me. For instance, in that particular PoC the goal was to speed up overall company budget consolidation, including complex allocation and elimination rules. It took over 8 hours in their current system. And only 4 minutes in Tm1 — imagine all the benefits you can get from that kind of speed.

Since there are more and more rumours about coming TM1+EP bundle, I just hope that step 2 will be more automated. For example, they can track delta xml’s sent by contributor client, containing all data updates, instead of publishing and bcping. But I can see no way they can get rid of neccesity of creating 2 models, one in Analyst for Contributor applications and another in TM1. Although dlists\dimensions can be synchronized, calculations are utterly different (you can target specific cube cell in TM1 formula and only the cube slice in EP).

Today I just want to add some of points:

1 If you’re up to publishing some serious amount of data, variant with adding a dummy dlist with 1 item and using it as dimension for publish seems a very bad idea.  You’re terribly slowing proccessing and wasting tablespace. So you have to choose a dimension for publish to reduce table size and speed things up

2 Dimension for publish must be:

a Stable. It shouldn’t change or change very rarely, since these changes will cause Framework Manager models \ ETL models changes.
b Have sane (~3-33) number of elements. If less than 3, there’s no win in performance. If more than 30, ensure it won’t change, because big dimensions change more often than small ones.

3 There are dlists that are naturally stable aka measures, like {Quantity;Price;Sales}

4 In other cases timescale seems a really good choice. For example, if months dimension doesn’t have year signs in name (like Jan, Feb, Mar), columns in publish won’t change ever.  To work with timescale published data you can use sql unpivot queries, that will virtually turn it into view publish, but it’ll way more effective than publishing it by 1elem dimension in first place. I’ve settled for this variant in my current project. Moreover, if you wrap this unpivot sql into ETL procedure, you can treat all published data uniformly while loading into datamarts (publish all needed cubes on timescale, use the same etl procedure).

I really expect and encourage comments on this naming proposal. Maybe together we will create something like Sun Java Code Conventions )

Whoa, it’s a really huge document, maybe I’ll it finish someday…

[1]: http://www.amazon.com/OLAP-Solutions-Building-Multidimensional-Information/dp/0471400300

Problem statement

We have 2 cubes, one containing product groups sales (for example) and the other containing detailed product data. Users should be able to correct overall group totals and correct detailed product distribution.
If group A contains Product 1 and Product 2 and if Total(A) is 100, it’s initially split 50\50.
User should be able to modify Product 1 to 75, yielding result of 75\25.
Normal break back won’t work this way, because it’ll see 75\50 as profile for break back and will produce 60\40 as a result.

So you’d need to create an own break back. That’s easy )

Verbal Description of the algorithm

It’s a copy-paste from communities topic, don’t bother re-reading.

1 For example we have a 2 dim cube
Total 100
A 70
B 30

I want A to become 800

2 Calculate the amount I need to break-back everywhere else to get total of x + 800 = 100
x = 100 – 800 = -700

3 Copy all other initial numbers, except A (set A=0) — that’s potential trouble, you’ll have to insure no-zero data input
Total 70
A 0
B 70

4 Break-back -700 across it
Total -700
A 0
B -700

5 Sum it up with A amount of 800
Total 100
A 800
B -700

Implementation

A working Analyst library is attached so I just want to stress some details
1 inputValue and value as subtotal. That’s the only way to allow input into break-backed cell.
2 transactional logic is implemented in very same way I described it.
3 no manual items allocation in dlinks on dimensions other that transactionalLogic_1, only match descriptions and self-updating allocation table. This allows easy updates.
4 3 elements are redundant in attached library (see comments). They are left for clarity of implementation.
5 dlists are named according to my own convention, I’ll write about it a bit later

Thanks Robert, it was an interesting task.

One of the applications was discussed in this communities topic. By the way, I’m spending much time on communities these days (it’s a very slow site, so no surprise) and I recommend it as a valuable resource for finding cognos-related answers. A steady pace of 10 topics a day gives a rather solid knowledge base now, after 4 months of uptime. If only it was faster, sigh…

So how to catch the fact that user changed this specific cell?

Simple, you just have to remember about virtual dimensions and calculation rules to build a “trigger”.

1 You add a simple dlist {value, old_value, new_value_flag}, with new_value_flag = (value?=old_value). new_value_flag is a dlist-formatted element with (yes\no) elements.
2 After user inputs new data into cell, this flag will change to yes (by IF formula) and you can use this data by writing an accumulation dlink using only new data (select yes value from new_value_flag virtual dimension).
3 When you’ve finished processing freshly inputed data, you just run another link that copies value to old_value, and that sets the flag back.

Ps: It’s too bad we didn’t follow that topic on communities to a logical end.

Using @Test(Restart) allowed doing a lot of interesting things, like calculating optimal production plans (repeat until reach given condition), even a sample calculation of admixtures for zink was made once

But most common use of @Test(Restart) was building your own Slice update macro. A detailed instruction follows.

Most common reason for DIY SliceUpdate is that you don’t want to put C>A links in UpdateList of cubes. Sincerely it’s not a reason at all, because if you follow exp\imp guidelines, you’ll have fully separate cubes for exporting from contributor and will be able to anything with their updatelists. But if you didn’t and the project is nearly at the production stage and you just can’t do C>A links due to volume limits:
1) Create an allocation cube with your elist and {standard, current step,what to transfer} dlist.
2) Fill standard column with increasing numbers 1..number_of_elist_items (they’ll be elists)
3) Make ‘what to transfer’ dlist-formated (by elist) and containing the formula ‘if current step = 1 then standard’
4) Put this cube in C>A links using what to transfer column
5) In macro, first copy standard to current step then iterate the following with @TEST(current step > 0;@Restart)
5.1) Run link — only 1 elist will be transfered
5.2) Decrease current step column by 1 (subtract dlink) — move the ruler down one step
5.3) Loop
Salt, pepper and serve with wine of preference.

PS By placing 1 in couple of rows at start you transfer 2 elists at a time, in three rows — 3 elists and so on.
PSPS Publish is faster )

Here are some ideas of how to speed admin links up, using the same principle as in incremental publish: transfer only changed data. So when only 2 CFO’s submitted data, all 200 don’t have to be moved and recalculated once again.

Let’s look at the scheme of incremental publish:
- every time inc publish is run, a timestamp is set to point the run time
- during run inc publish selects only elists with change date more than inc timestamp, this meaning that these elists data was changed since last inc publish

So we need 2 tools:
- One to set & store & query the run time. Let’s call it “Egg-Clock”
- Other to run the task selecting only limited number of elists. Lets call it “Select Elists, Run Job” or “SERJ” )

Egg-Clock is pretty simple in all cases, you just record the timestamp somewhere (text file, table etc) and return it back.
SERJ was almost simple for publish — you’ve had to recreate publish macro, specifying needed elist guids.

How can we do this with administration links? I’m pretty sure some people in Cognos are writing built-in utility for inc admin links right now, but sometimes you have to get things done without waiting for another major release.

1 Use query subjects and package import
How to:
* You set up incremental publish on application to make it’s data always ready for transfer.
* Create a query subject on that published data that you need to transfer
* Set up a filter on that query subject using Egg Clock to select only elists that have changed since last incremental admin link run time
* Create an administration link from filtered query subject to your target application.
* Create a macro that will run this link and update Egg Clock and there you go. You can add an incremental publish in this macro to make it completely independent

It’s a rather straight-forward solution and quite easy to set up.

2 Use duplicate applications and No Data Settings
We’ll try to use following rule: when administration links are run No-Data items are not transfered. It’s rather cumbersome solution, but maybe someone will like it )

* Create a copy of your source application (yep, the same application, just another copy)
* Add an imported access table in that application, opening selected elists on read or write. Set default level to No-Data. It should apply to all cubes in application, delete other ATs if necessary.
* Use Egg Clock to generate this access table, select only elists changed since last run time.
* Create link, copying your source application to this duplicated application.
* Create all your desired links from duplicated application rather than from source one.
And in macro do following:
1 Create access table based on Egg Clock — only changed elists
2 Import this AT into duplicate application
3 GTP duplicate — this will cut-down all other elists, except changed ones
4 Run links from source to duplicate — since No-Data is applied only changed will transfer
5 Run links from duplicate to target — again only No-Data elements

3 Try to modify .cal export files and import them back
Just to mention — don’t do it.

Multi-Line is only way to go when you have a potentially huge dimension only a tiny bit of which should be available to end-user at a time. Like employee planning, whole dimension of 10k people, strictly less than a 100 in department. So you create a fake dimension of 1..100 and add an Employee name column, decreasing cube volume by 100. Access tables + cut-down might help, but it’s sometimes better to allow people pick up any employee,client, product given up that there won’t be more than fixed number of rows.

Main problem with this modeling technique arises when it’s time to import data in such cube. That data usually doesn’t have an 1..100 running count attached, so it’s your task to add it.

In this post I’ll sum up the ideas of how to calculate running count split by elist (that’s usual, ain’t it?).

As a example, I’ll use this simple table

And you have departments as an elist, so you have to number rows so that numbering will restart for each dept. Numbering requires an order so let’s alphabetical order of employee names.

So this is what we want to get:

Source code for table:

WITH data_t AS
(
SELECT 'Finance' DEPT,'Pete' EMP, 100 SALARY FROM sys.dual
UNION ALL
SELECT 'Finance','Ann', 200  FROM sys.dual
UNION ALL
SELECT 'Finance','Jo', 300  FROM sys.dual
UNION ALL
SELECT 'HR','Nick', 100  FROM sys.dual
UNION ALL
SELECT 'HR','Sam', 200  FROM sys.dual
)
SELECT * FROM data_t

So what are our choices?

0 Use a framework package as source and a running-count calculation in query subject

Only since Cognos 8.2, slower than some variants, but definitely most cognos-style.

Create a query subject and add an expression of running-count like

running-count ( [SALARY] for [DEPT] )

1 Add sub query to count this rows position within elist slice.

That’s “works everywhere, but terribly slow” solution. Easy to do, use it if you’re about to import a 100-1000 rows table and really don’t care about time it takes.

Sample code:

SELECT DEPT,EMP,SALARY,
(
SELECT COUNT(*)
FROM data_t T2
WHERE T2.DEPT = T.DEPT
AND T2.EMP <= T.EMP
) AS running_count
 FROM data_t T
 ORDER BY dept,running_count

2 Use SQL’99 standard rownum() function

Way much better optimized (in Oracle ) ) than subquerying variant above. Problem is: works only on Oracle, Ms SQL 2005+ and DB2. Ms SQL 2000 import — sorry, no SQL’99 there. See original syntax

Sample code:

SELECT DEPT,EMP,SALARY, ROW_NUMBER()
OVER (partition BY dept ORDER BY emp) FROM data_t

3 Use cursors to calculate rownumber

Brute-force solution, non portable, as fast as possible. Only way to do serious lifting in MS SQL 2000.

I’ll place sample code for Ms SQL 2000 if somebody needs it

PS You’re on Oracle 10g+? Lucky you, you can also try to do running_counts it via oracle model by clause.

The task relatively simple for 1 application (yet required view publish for version prior to 8.2, now there’s a special table with users). You just join nodestate with users and states tables and enjoy nice pie chart of submitted,in progress and not started. Even a direct drill-through in application, if you’d like.
But it becomes messy as more and more applications are added.
I’ve seen variants in Cognos ReportNet, PHP, IIS log parsing and liked none.

Yet all applications are listed in pad datastore and you just have to iterate the “1 application solution”.
So I just dream of audit package for EP that would automatically add all applications and give admins some rest of programming and framework changing,

Did anyone out there do any of audit reporting on EP? What were your problems & solutions?