What's the best tool to parse log files?


I use grep to parse through my trading apps logs, but it's limited in the sense that I need to visually trawl through the output to see what happened etc.

I'm wondering if Perl is a better option? Any good resources to learn log and string parsing with Perl?

I'd also believe that Python would be good for this. Perl vs Python vs 'grep on linux'?

1/4/2010 12:35:49 AM

In the end, it really depends on how much semantics you want to identify, whether your logs fit common patterns, and what you want to do with the parsed data.

If you can use regular expressions to find what you need, you have tons of options. Perl is a popular language and has very convenient native RE facilities. I personally feel a lot more comfortable with Python and find that the little added hassle for doing REs is not significant.

If you want to do something smarter than RE matching, or want to have a lot of logic, you may be more comfortable with Python or even with Java/C++/etc. For instance, it is easy to read line-by-line in Python and then apply various predicate functions and reactions to matches, which is great if you have a ruleset you would like to apply.

1/3/2010 8:58:48 AM

Licensed under: CC-BY-SA with attribution
Not affiliated with: Stack Overflow