How do you look at log files?

Tim_Astaro · #1 (**permalink**) 05-05-2009, 05:48 PM

Hi Everbody,

I have used several techniques for looking at log information. I am wondering which is the most common. If you use a specific app, or technique through command line, post it here! It may help a fellow Astaro User in the future.

My personal favorite (at the moment) is commandline with cat/tail/less/grep/awk commands in all sorts of permutations. I have become close to awk lately.

If I use a GUI tool, I like notepad++. Not only can it parse for a word or phrase in all documents that are currently open, but you can view two log entries side by side, etc...

BAlfson · #2 (**permalink**) 05-05-2009, 07:19 PM

Tim, because of the structure of individual files and directories, it's easier to do searches on the box itself. Since I'm not that adept with linux, I just use the WebAdmin search capability.

I do wish there were an easy way to import them into Excel - is there a name for the file format that might allow one to find a converter?

Cheers - Bob

thtran · #3 (**permalink**) 05-06-2009, 06:44 AM

WebAdmin is too slow.
I have a Windows-PC with KIWI Syslog Deamon installed. I'm sending the most important logs via syslog to that PC. Then I use scripts to filter the logs. Well, I only have a few very basic scripts so far. I'm still working on it.

cyberknutte · #4 (**permalink**) 05-06-2009, 09:03 AM

Splunk!

http://www.splunk.com/

It's a syslog server and reporting tool in one. The web interface is intuitive and easy to use, and at the same time the search syntax is extremely powerfull. You will never need any commandline tools (grep,awk) again.

The free version can index 500 MB of log data per day.

// Andreas

BAlfson · #5 (**permalink**) 05-06-2009, 01:58 PM

I tried splunk and decided against it. 500MB is too small, and I'd rather not have any of my clients' information uploaded to a third-party service. Plus, the blasted thing decided it was allowed to load every time I rebooted, so I removed it.

I know there are some smart people who really like it; it's just not my "color" I guess.

Cheers - Bob

BarryG · #6 (**permalink**) 05-07-2009, 02:32 AM

Bob,

re Splunk: I don't know of any completely free alternatives, but I do know that loading logs into databases isn't _too_ hard; it depends what you want to do with the data.

Some of the free NMS systems like ZenOSS, Zabbix, OpenNMS, etc. have some syslog support, but I'm not sure that they're good for storing LOTs of log data.
see Comparison of network monitoring systems - Wikipedia, the free encyclopedia for a list.

re Excel: logs are usually space-delimited; open in Excel, and if they're all in one column, click that column, click on Data, Text-to-Columns, ...

A lot can be done with Perl, or sed/awk, ...

grep --color is nice, btw.

Barry

PatrickMc · #7 (**permalink**) 06-30-2009, 05:19 PM

We often use biterscripting for customized processing of log files. This is in addition to other 'packages' we use. biterscirping is just a scripting language which makes parsing, extracting and composing rather easy. Being a system admin, I like that ability.

There is a good script posted at http://www.biterscripting.com/SS_WebLogParser.html . It is a sample script written in biterscripting. You can start with it. To try,

1. Download biterscripting. It is free.
2. Start biterscripting. Enter the following command (this installs all the sample scripts they provide).

script "http://www.biterscripting.com/Download/SS_AllSamples.txt"

3. Call the WebLogParser script with the following command.

script SS_WebLogParser.txt logfile("C:\somfolder\somelogfile")

One good thing about biterscripting is that is is fairly easy to learn. I have also seen tutorials some place.

Patrick