Smartlog slow to return results

[TrevorB]

How fast should i expect to see results on searches? I always thought it was touted that I could look through billions of logs quickly.

Currently, our SmartLog server shows 1,028,744,775 logs over a 7 day period of time (October 3 - October 10). That does seem to be inline with the fact that we've set it to only index 7 days worth of logs. However, when I do a query for something and it has to search through all 7 days worth of logs to find (or not) what I was looking for it takes several minutes. We're talking over 20+ minutes to get through all logs. Obviously, if it finds something more current like in today's logs those results come back first but if it requires finding something that may have happened several days ago then the search takes forever. I guess that it is faster than me opening all these individual logs files and doing searches via SmartTracker, but still.

Does anyone else have a massive amount of logs and this is the case? Should I just live with it and hope that I only really need to do searches for current date and time?

[EricAnderson]

I think the first question you'll get from most is about the hardware specs. Yes, SmartLog can be very fast to return results, even with your numbers. However, running on under-powered gear can easily make for a less-than-stellar (bad) experience.

So, before we look further, what can you tell us about CPU, RAM, disk speed, etc. (of the log server)?


-E

[cciesec2006]

I think the first question you'll get from most is about the hardware specs. Yes, SmartLog can be very fast to return results, even with your numbers. However, running on under-powered gear can easily make for a less-than-stellar (bad) experience.

So, before we look further, what can you tell us about CPU, RAM, disk speed, etc. (of the log server)?


-E

You should definitely look into kibana: https://www.elastic.co/products/kibana. It can use LEA to collect checkpoint log

I can search through 5 billions records in less than 45 seconds.