Archive for the ‘open source’ Category

One of the main things I carried out of my alma-mater is passion for data. When you've got data, so many interesting things can be done . Various analysis, comparison, stats - just givе the data.

That was a rather pathetic preambula. Let's get straight to business.

Problem.
On recent project I was doing some technical support (dllhost problems, if you recall, and more). It is a rather mature installation (about 8 months of production) with quite a number of servers, so logs contained around 100 Mb of data. And common question was -- ok, this error, did it appear before? When? Accompanied by what errors? What's the overall trend? Common patterns? Charts?
All that analytical questions posed up on EP error logs.

As we all know, Cognos Contributor errors are recorded into PlanningErrorLog.csv files. And those log files contain rather detailed information, including ep version, module, time and, of course, error description.

I've tried to find some complete solution, but as usual on Microsoft platform, there are only paid tools, yet not solving the task completely.

Solution.
Having a whole Cognos 8 BI at finger-tips, I thought it would be nice to have all that errors in PowerPlay cube. I'm an OLAP-guy, after all.

So, technically the task was divided into 3 parts:

1. Gathering the logs
2. Forming a datasource for Transformer
3. Creating a Transformer model and cube

Step 1 is solved by .bat file -- I'll post it in separate as "Backing up with timestamping"
Step 3 is quite straight-forward, if you have a single file, containing errors from multiple ones, except for time dimension as usual.
More details on forming a datasource. Got no time? Skip to this scheme for overall picture.
Merging the logs.
At first I thought it was a rather simple task since the source files are csv (comma separated) and they just need to merged n to 1 with some additional transformations (adding top level error categories based on error desc, for example).
It's never easy, I must admit.
Well, csv is comma-separated for all but Cognos, PlanningErrorLogs are tab-separated. That's not a problem, let's get n tab files to csv.
Planning logs contain some wonderful pack of unprintable chars(meant for Excel easy opening I hope, because there is no reason otherwise). In those chars there are some kind "EOF" chars, for example (I can see only their hex codes anyway), so VBScript cannot parse those files line by line correctly. There is a variant to open the file in Excel and save it to "normal csv", but that's impossible with 60 Mb log file I've got here.
For sake of my nerves and Internet space I won't describe all other problems like parsing returned sql statements in error descriptions (those contain tabs and ";" in the same line).

Since I like Python much the final script is .py. It takes a directorу containing logs, merges them into 1 file, adds timesort values and error categories.

So that's the final scheme.

We add client servers to this sending net and plan to use 8 BI Event Notifier as technical support catalyst.

-----
I'm eager to give out the scripts for the same reasons as I do this blog (also a dim hope that overall Cognos support will get better), so if you're interested - mail bark-bark ykud.com. A rather good error categorization can be created with some joined effort, imho.