Logtop

INTRODUCTION

Tired to write 'tail -n 500 some.log | cut -d' ' -f1 | sort | uniq -c | sort -gr' ? You're on the right page !

tl;dr for Debian users :

aptitude install logtop; tail -f access.log | cut -d' ' -f7 | logtop

Logtop is a very basic yet powerfull live log (or any text) analyzer.

HOW DOES IT LOOKS ?

$ tail -f access.log | cut -d' ' -f1 | logtop 

WHAT DOES LOGTOP DO ?

• Display live results like top.
• Display line rank, frequency, count, and the actual line.
• Exposes an API for C and Python if you want to run it headless.

WHAT LOGTOP DOES NOT DO ?

• Eat all your CPU and RAM : logtop is designed to run in production server without impacting them.
• Understand or parse your log format, use cut or awk for this.
• Open your log files, use tail -f (or -F) for this.

INSTALLATION ON DISTRO USING DPKG

$ apt-get install logtop

INSTALLATION ON OTHER DISTRO

$ git clone https://github.com/JulienPalard/logtop.git
          $ cd logtop
          $ // Install dependencies : libncurses5-dev uthash-dev
          $ make && make install # not war
          

USAGE

You have to tail and parse your log lines, logtop will count what you're giving to him and compute their frequency.

For the sake of readability I will use cut in examples. Beware that cut buffers, so if your source is slow, logtop will get the data by bursts. You may prefer awk to explicitly flush each line like this :

tail -f access.log | awk '{print $9; fflush();}' | logtop

Let start with some easy examples, in the case you want to test logtop with NCSA log format (used by apache, nginx, varnish, ...) :

Get IP's requesting your server:

tail -f access.log | cut -d' ' -f1 | logtop 

Get URL's requested on your server:

tail -f access.log | cut -d' ' -f7 | logtop 

Show status codes your server is replying:

tail -f access.log | cut -d' ' -f9 | logtop 

Show user agents requesting your server:

tail -f access.log | cut -d' ' -f12- | logtop 

C API

Logtop can be used by your C programs, you may want to compile with logtop's sources (src/{avl.c,history.c,logtop.c,libavl/avl.c}) or with liblogtop, obtained using 'make liblogtop`.

C API is exposed in src/logtop.h. In short, you may need :

• struct logtop *new_logtop(size_t history_size)
• void delete_logtop(struct logtop *this)
• void logtop_feed(struct logtop *this, char *line)
• struct logtop_state *logtop_get(struct logtop *this, size_t qte)
• double logtop_timespan(struct logtop *this)
• unsigned int logtop_qte_of_elements(struct logtop *this)

You can find an example the C API in examples/example1.c

PYTHON API

You should read the pydoc of the python module, but in short logtop module exposes a logtop class containing :

• logtop.__init__(history_size) to build a new logtop keeping at most history_size lines.
• logtop.feed(line) to feed a new line in logtop.
• logtop.get(qte_of_elements) to get the top qte_of_elements lines.
• logtop.qte_of_elements() to get the current total number of lines.
• logtop.timespan() to get the duration from the oldest line to now.

timespan may be less than the runtime as logtop drop old lines to keep at most history_size lines, given in the constructor of the logtop class.

An example can be found in examples/example1.py