Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

That tool looks great :) but since we're already seeing <1s search times and the tool is only used by internal support employees, I'm mostly going with "never touch a running system" these days.

While for a database like ES you'd put all of the data into one big pile and then filter by keywords, e.g. host=ftp service=ftp query=IP, for logfiles you usually search on a much smaller set. They are rotated by day and logs are broken down by host and service by rsyslog, so instead of filtering the full 150TB - which is what ES has to do - my grep only needs to look at the 1-2 GB of data inside the file where host, service, and date match.



Do you understand something about such thing as ES indices?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: