[prelude-user] Massive Prelude-LML performance improvement

[Yoann Vandoorselaere]

Hi folks,

For some time, we had report about LML performance problem on heavy
loaded server. I started looking at it some days ago, and the
performance result were pretty bad (around 500 EPS (Events [line of log]
processed Per second) on a 2.4Ghz machine).

As a result I've made a complete overhaul of the PCRE plugin in LML,
discovering some bugs in the pattern matching algorithm (some alert were
reported several time, and too many regex match attempt were done
compared to what was really needed). I also made it more easy to use the
"chained" rule attribute, which was originally added to help with
performance, but was not easy to use from an end user perspective.

The end result is a raw ~850% performance improvement (consistently
repeatable on low end as well as high end machine). To give a number, my
machine now top at > 5000 EPS per seconds. 

With some more enhancement, I also was able to make it top at 12000 EPS,
but that will require some more work since this particular change bring
a risk of false negative which is not acceptable.

The improvement will be available in the next Prelude-LML release
candidate.

Regards,

-- 
Yoann Vandoorselaere <yoann.v at prelude-ids.com>