Look how bad it is with all dense cubes, like (copy-paste these values)
R2;spring
R2;autumn
R1;spring
R1;autumn

and how good with more or less sparse ones

R2;spring;store1
R2;autumn;store2
R1;spring;store1
R1;autumn;store3

Task is quite simple&straightforward: calculate number of periods in which current stock will be sold in a store. Like: today we have sales enough for 3 weeks. That's a very common sales demand analysis parameter (Months of Sales (MoS), Weeks of Sales (WoS), DoS and etc).

Formula is quite simple: Current Stock / Average Sales over some period

But there were some difficulties in my case.

I've got only partial data, some periods were completely missing, and the requirement was to treat missing sales data as zeros. And I also had to get the last known stock as current stock, because stock data could be missing too.
Last stock is pretty easy to get using last_value oracle function, but getting average sales is hard. You cannot use fixed window width in partition by, since data can be missing. Moreover period varied other the years, so I've ended up adding up a field in time dimension called periods_ago, that contained time_key reference from which to sum sales.

So I've started with simple subqueries

 
SELECT
        time_key,
        product,
        company,
        sales_qty,
        (SELECT SUM(sales_qty)
           FROM dwh.sales sales2
         WHERE
            sales2.product = sales.product,
            sales2.company = sales.company,
            sales2.time_key BETWEEN weeks.periods_ago AND sales.time_key
        ) AS running_sales_sum
FROM
        dwh.time_dim weeks, dwh.sales sales
        WHERE
           weeks.key = sales.time_key
 

and that is terribly slow on 10^6+ rows. ~20 minutes.

But with model by it'll be like:

SELECT
        TIME,
        product,
        company,
        sales_qty,
        last_stock_qty
FROM
        dwh.time_dim weeks, dwh.sales sales
        WHERE
           weeks.key = sales.time_key
      model
        IGNORE NAV
          PARTITION BY (product,company)
            dimension BY (time_key)
            measures
            (
                periods_ago,
                sales_qty,
                0 eight_weeks_sales_sum_qty,
                NVL(LAST_VALUE(STOCK IGNORE NULLS)
                OVER (PARTITION BY COMPANY, PRODUCT, SEGMENT, DISTRIB_LEVEL, SYSTEM
                ORDER BY TIME ASC),0) AS last_stock_qty
            rules   UPDATE
            (
                running_sales_sum[ANY] = SUM(sales_qty)[TIME BETWEEN periods_ago[cv()] AND cv(time_key)]
                            END
			)
 
            )
 

around 3 minutes ) Notice the last stock calculation.

Here's a couple links to good model by examples:
A wonderful tutorial at SQLSnippets
Official documentation restriction section -- read that carefully

Go the links from Tom Kyte's blog.

Disclaimer: I'm Oracle & ODI n00b -- there are (?) better ways of accomplishing these tasks and if you can point in right direction, i'd be eternally grateful.

So the task is to drop all indexes on a table before loading data (loading gets significantly faster, as they write in 1st grade textbooks these days). And not to specify every index to drop manually, since index are built by separate package and their names are not passed to loading part.

It can be done rather easy:
1 Get all table indexes from ALL_INDEX table
2 Form a drop command for every index
3 Execute command

Couldn't figure a nice&clear way of doing it via sql procedure, but jython solution is easy. Table_Name is a procedure option telling what table's indexes to drop.

You can put such a step in Knowledge Module and get the option (Drop_Indexes) and use a substitution method getJDBCConnection(Target) instead of specifically writing out connection string to database. And beware of tables with same names -- it's not checked here.

Jython Procedure Code:


import java
# Set your connection. If including this as a step in IKM, you'd better use JDBCConnection substition method instead of 2 next steps
driver, url, user, passwd = ('oracle.jdbc.driver.OracleDriver','jdbc:oracle:thin:@server:port:instance','user','password')
# Register Driver
java.lang.Class.forName(driver)
# Create a Connection Object
myCon = java.sql.DriverManager.getConnection(url, user, passwd)
# Create a Statement
selectStmt = myCon.createStatement ()
# Run a Select Query and get a Result Set
selectRs = selectStmt.executeQuery ("select INDEX_NAME from ALL_INDEXES WHERE table_name = '<%=odiRef.getOption( "Table" )%>'")
# Loop over the Result Set
dropStmt = myCon.createStatement ()
while (selectRs. next ()):
dropStmt.executeQuery ( "DROP INDEX " + "%s" %(selectRs. getString ("INDEX_NAME")))
myCon.close()


Suppose we need to rename the file test.txt to test_currdate.txt.

On my laptop "echo %date%" gives "05.12.2006".
Be careful, output depends on regional settings (I've got Russian), so you'll need to modify this script.
"test_05.12.2006.txt" doesn't look that good, I'd prefer "test_05122006.txt", so we'll need to calculate 05122006 from 05.12.2006.
This can be done via :~ operator, allowing to cut substrings. Syntax is str:~start_position,length.

So the following line will calculate the variable currdate, holding "05122006".

 
set curdate=%date:~6,4%%date:~3,2%%date:~0,2%
 

rename command allows to change name of file, therefore you just run

 
set curdate=%date:~6,4%%date:~3,2%%date:~0,2%
rename test.txt test_%currdate%.txt
 

Script for grabbing specified files from servers below.

This bat file accepts servernames as arguments and iterates through them to rename&gather all PlanningErrorLog.csv logs in one place (namely, C:\Logs).

 
@echo off
rem Gathering the Logs from command line arguments=servernames
rem directory to write
set destination="C:Logs\"
set user=Admin
set pwd=~~~~~~~~
set log_directory="WindowsTemp\"
set log_name=PlanningErrorLog
echo Monitoring %date% %time%>>c:logsmonitor_log.txt
:LOOP
IF "%1" == "" GOTO END
net use \\%1\c$ %pwd% /USER:%1\%user%set curtime=%date:~6,4%%date:~3,2%%date:~0,2%_%time:~0,2%%time:~3,2%%time:~6,2%
if EXIST \\%1\c$%log_directory%%log_name%.csv rename \\%1\c$\%log_directory%%log_name%.csv %1_%log_name%%curtime%.csv
& xcopy \\%1\c$\%log_directory%%1_%log_name%%curtime%.csv  %destination% /DY
net use /delete \\%1\c$ /Y
SHIFT
GOTO LOOP
:END
 

That was a rather pathetic preambula. Let's get straight to business.

Problem.
On recent project I was doing some technical support (dllhost problems, if you recall, and more). It is a rather mature installation (about 8 months of production) with quite a number of servers, so logs contained around 100 Mb of data. And common question was -- ok, this error, did it appear before? When? Accompanied by what errors? What's the overall trend? Common patterns? Charts?
All that analytical questions posed up on EP error logs.

As we all know, Cognos Contributor errors are recorded into PlanningErrorLog.csv files. And those log files contain rather detailed information, including ep version, module, time and, of course, error description.

I've tried to find some complete solution, but as usual on Microsoft platform, there are only paid tools, yet not solving the task completely.

Solution.
Having a whole Cognos 8 BI at finger-tips, I thought it would be nice to have all that errors in PowerPlay cube. I'm an OLAP-guy, after all.

So, technically the task was divided into 3 parts:

1. Gathering the logs
2. Forming a datasource for Transformer
3. Creating a Transformer model and cube

Step 1 is solved by .bat file -- I'll post it in separate as "Backing up with timestamping"
Step 3 is quite straight-forward, if you have a single file, containing errors from multiple ones, except for time dimension as usual.
More details on forming a datasource. Got no time? Skip to this scheme for overall picture.
Merging the logs.
At first I thought it was a rather simple task since the source files are csv (comma separated) and they just need to merged n to 1 with some additional transformations (adding top level error categories based on error desc, for example).
It's never easy, I must admit.
Well, csv is comma-separated for all but Cognos, PlanningErrorLogs are tab-separated. That's not a problem, let's get n tab files to csv.
Planning logs contain some wonderful pack of unprintable chars(meant for Excel easy opening I hope, because there is no reason otherwise). In those chars there are some kind "EOF" chars, for example (I can see only their hex codes anyway), so VBScript cannot parse those files line by line correctly. There is a variant to open the file in Excel and save it to "normal csv", but that's impossible with 60 Mb log file I've got here.
For sake of my nerves and Internet space I won't describe all other problems like parsing returned sql statements in error descriptions (those contain tabs and ";" in the same line).

Since I like Python much the final script is .py. It takes a directorу containing logs, merges them into 1 file, adds timesort values and error categories.

So that's the final scheme.

We add client servers to this sending net and plan to use 8 BI Event Notifier as technical support catalyst.

-----
I'm eager to give out the scripts for the same reasons as I do this blog (also a dim hope that overall Cognos support will get better), so if you're interested - mail bark-bark ykud.com. A rather good error categorization can be created with some joined effort, imho.

When toying with gentoo, I've found a post, consisting of mostly "I'll always use logrotate" "I'll always use logrotate" "I'll always use logrotate".
Can't find it now, so tribute unknown.
"I’ will always backup first" "I’ will always backup first"
"I'will always backup first" "I'will always backup first"
"I'will always backup first" "I'will always backup first"
"I'will always backup first" "I'will always backup first"
"I'will always backup first" "I'will always backup first"

Restart is very fast (less than a second) and rather reliable. So here are COM+ restarting scripts.

A .vbs script restarting desired COM component. Set desired component name.
restart_com.vbs. Press more for code.

And some bat file to do basic logging. restart_com.bat@ echo off

rem restart specific COM+ component

cmd /c "CSCRIPT.EXE restart_com.vbs /B /nologo">>com_restart_log.txt

]]>
			http://ykud.com/blog/bicpm/com-restarting/feed
		1
		
		
		
		http://ykud.com/blog/bicpm/server-side-asp-sleep
		http://ykud.com/blog/bicpm/server-side-asp-sleep#comments
		Fri, 08 Sep 2006 13:45:34 +0000
		ykud
				
		
		
		

		http://ykud.com/blog/2006/09/08/server-side-asp-sleep/
		
			While implementing an idea of web-based execution of Contributor Macros (see previous post), I've encountered the problem of some kind of progress-bar drawing in asp page.

When user hit's asp page calling a Contributor Macro (which can take 15 min), he ought to see some action, telling him that "work is still going". Mine idea was to draw sequential line of blue squares.
So the ASP page code will be like that:
set WScript = Server.CreateObject("WScript.Shell")

set oExec = WScript.exec("c:\macro.bat")

do while oExec.Status = 0

response.write("<img src='square.jpg'>")

response.flush()

loop
That's ok -- but it draws a square every computer "tick"   Can fill up a page in 2 secs.
A Sleep(5 secs) is needed in cycle body, just before response.flush()
But there's no sleep instruction in ASP. Workarounds are drastic.
* Create  an inner cycle to slow up execution. 100% CPU load -- very nice.
* Install a WaitFor dll. What, on every client site to use?
* Use an SQL to call Ms SQL Timeout function. Fantastic.
So we had to find another workaround for sleep. It quite fits in above mentioned list.
1 Create a sleep.vbs file with: "Wscript.Sleep 200000" (a very big number of your choice) line. A script to sleep forever.

2 From ASP page run "cscript.exe /T:your_time_to_sleep sleep.vbs" . So it just stops sleep.vbs  execution on timeout of   your_time_to_sleep.
Final code looks like:
set WScript = Server.CreateObject("WScript.Shell")

set oExec = WScript.exec("c:\macro.bat")

do while oExec.Status = 0 response.write("<img src='square.jpg'>")

Set WShell = Server.CreateObject("WScript.Shell")

Wshell.run("cscript.exe /T:your_time_to_sleep sleep.vbs") response.flush()

loop
]]>
			http://ykud.com/blog/bicpm/server-side-asp-sleep/feed
		2