Log Management Has Come a Long Way, but Still Hangs on Normalization, Search


Logs -- for some administrators, the word sends a chill down their spine. When I was a systems administrator managing over 50 UNIX servers, I had more logs than I knew what to do with, and frequently I did nothing with them. What I found most difficult then was how to collect log data from over 50 servers and have it available in one place to report on. What I did, like every other administrator would do, was develop a set of scripts to strip out what I wanted, FTP it to another server, and load it into a database. I remember it taking quite a while to work out what I needed to collect and pulling it from the logs. Also, I frequently had problems with FTP to get my data where it needed to be. It was the network, of course.


The SANS Institute recently published a report, the SANS Annual 2009 Log Management Survey, that surveyed organizations on how they collect, report and use log data. According to the report, the top four most challenging aspects of the log management lifecycle were:


Using log data to enhance IT operations

Other (we don't know what this means)

Normalization of data

Searching log data


The organizations were asked why they collected log data. The top three responses were:


To track suspicious behavior and user activity

Forensics analysis and correlation

Day-to-day IT operations


I think the security benefits are obvious. However, we need this data in real time, or as close to real time as possible. If not, then we are performing a forensics investigation because it already happened. Indeed, according to the survey, management's number-one response on how they would benefit most from log data was event detection.


I don't think that there are any doubts that log data is an extremely valuable tool. I am glad to see that the technology has matured to the point where the basic collection of the data is not the biggest challenge anymore. However, I am surprised that normalization and searching are a problem, given all of the database and reporting tools available today. Are you collecting log data? If so, what problems are you running into?