Thanks for pointing at BOMs, will remeber that for future.

Yet my other problem was that some files were indeed csv delimited (somebody opened them in Excel and saved) and had another wide pack of excel-only characters. Python decode worked extremely well in given time&effort constraints.

Yep, we’ve got around 2gbs of 3 year log data, searchable on cognos connection, but it mostly shows us the things we already known — database crushes and that stuff. But it sure looks nice )

http://en.wikipedia.org/wiki/Byte_Order_Mark

Use Python’s “codecs” module to open them as “utf-16″:

f= codecs.open(logFilePath,”r+t”,”utf-16″)

Yes, they are tab-separated as commas appear very often in the text. The file extension is just to abuse the file association in Windows for Excel. If there are tabs in the content, it should be replaced with a “~”.

Cool to see good work being done on the logs.