Logging down the drain

We all did it, most of us still do it. First we create nice log files, log everything we think is useful and save it to a file in a logical place.

Three months later the file is growing out of bounds, its missing the important informations but is full of useless crap that is not detailed enough to tell anything about whats going on and the place where it is doesn’t make any sense anymore. Why is this happening? Why do we do this again and again, even truncating the log files without stopping the whole process?

Yeah, logging is good. The more data we log the better we can solve problems. Problems we get known of through the complaints from our clients because we log the error but we do not alert anyone. Why is it always the client that has to tell that something is going wrong? Shouldn’t it be the other way round? Shouldn’t we as Service Provider or whatever know whenever one of our services fails and tell our clients about it? I guess that would make the relationship to the clients massively better.

So what would be a better way to all of this? I think its pretty simple. And I hope I solved it for now for my current project.

We already had a Log-Class that we were using in existing projects, I just extended that to make life easier. Now we have a configurable log level, for both logging to a file and logging to a DB. For example we log warnings and above to the file and errors and above to the DB. We add a unique identifier (mainly for the file if there can be multiple accesses and the log messages would be mixed), a message and a log level. The identifier can contain the IP of the server and the path to the script as well to be able to identify the exact location of the problem.

On another server there is running a script as cronjob every 5 minutes. This script checks the DB with the log entries sends an summarized email if needed, marks the existing log entries as done and deletes all data that is older than 24 hours. So this is a very easy way to really know whats going on when it is going on.

Now I wonder why I not always did it like that. Maybe there was this irrational thought in my brain that logging is only needed while there is active development, but of course this is not the case, errors come always.

I hope this will remind me of keeping it like this and not falling back to the old behaviour, one day…

Logging down the drain

3 thoughts on “Logging down the drain

  1. Briddaa (not so IT) says:

    Ok, I can’t really understand or follow up what you are talking about but it sounds like a good idea. Tracing back problems and identifying clients before they do, sounds like a pretty good plan to me 🙂
    Well done and keep up the good work

  2. Olli says:

    I don’t see a problem with that. Every five minutes is close to no load at all, its a flat table with very few entries (just for 24 hours).
    Right now I use a MySQL Cluster table for that, I guess I will change that to a simple MyISAM once. But I wanted to put some stuff into the cluster…. 😉

    I know what you wanna say with that, I hope not everyone has such a bad experience with Netconcepts as you do. 😮

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s