In most Linux distros, system administrators would keep an eye on log files from time to time in production environments, in order to get a glimpse at the health of the system, the running state of applications, potential memory issues, events with high priority…This will help them improve the overall system performance and to proactively avoid future problems which might affect the users and their applications. Viewing and analyzing the log files is no easy task if done without using the appropriate tools and utilities.
In this article, we will be looking at some of the best log file monitoring and management applications that are in use today.
Logcheck is a widely used open source log management. It can identify unknown issues automatically as well as security breaches related events in log files. It sends the reported findings by email periodically after filtering out unnecessary entries. An example email is shown below :
It runs by default every hour as a cronjob and after every bootup. It has three different logfile filtering levels:
Paranoid: is meant for high-security systems.
Server: is the default filtering level
Workstation: is meant for sheltered systems. This includes rules specified under server and paranoid levels.
Logcheck can sort reported messages into three categories: Security events, system events and system attack alerts and has the following features :
- Has predefined templates for reports
- Easy log filtering mechanism with regular expressions
- Instant email notifications
- Has cloud based dispatch handling system
- Instant security issues alerts
find out more about Logcheck here.
Logstash is an open source data processing and logging system. This log file viewer can process, collect and forward events as well as system log messages.It has the ability to quickly read and load unstructured data of all sizes. Logstash has several predefined filters which helps in data indexation and transformation. Logstash is able to unify and ingest data from different log sources, such as web applications,metrics, various AWS services and data stores, allowing system administrators to compare, cleanse , analyze and visualize their log data.
Logstash has the following features :
- Has a flexible plugin architecture
- Can handle any data source independently of its shape and size
- Integration with Elasticsearch, Kibana and Beats
- Easy orchestration and management of pipelines
- Seamless integration with the built-in security features
- Derive structure out of unstructured data
Visit Logstash website here.
Splunk is a fully integrated, robust and powerful enterprise log management software. It allows to collect, search , diagnoze, report and store any machine generated log data regardless of its shape, i.e. structured or unstructured, and complexity.
Furthermore, splunk supports various log management use cases such as ;
- Log consolidation and retention
- IT operations troubleshooting,
- Application troubleshooting
- Compliance reporting.
Splunk has the following features :
- Machine data indexation
- Data monitoring and alerting
- Enables to freely pivot across data
- Flexible access to relational databases, CSV files, enterprise data stores (NoSQL)
- Supports local and remote data sources
- Data searching and correlation
- Fully integrated and easily scalable
Visit Splunk website here.
Graylog is a powerful open source fully integrated log management framework. It helps system administrators analyze, aggregate and extract disorganized, systematic and framed data from server logs which are gathered with Syslog. It also enables users to visualize and search the logs using an intuitive diagrams rich interface.
Graylog uses MongoDB in order to store the configuration data and not the log data. Only metadata such as stream configurations or user information, is stored. Here some of feature of Graylog :
- Seamless enterprise level production environments Integration
- Log processing in real time
- Accessible, speedy and secure
- Contains predefined templates for data display
- Provides an operational data hub
- Highly customizable and extensible
- Unstructured data parsing enabling
- Can ingest any structured data from any log source
- Integrated dashboard and alerting system
- Various alert notifications
Visit Graylog website here.
Xlogmaster is a log viewer that allows systems administrators to monitor everything that’s occurring in your system regardless of the number of log files and devices. It has a simple graphical interface that allows users to easily configure almost any action and helps keep track of everything you can think of in the log files realm.It can be used to raise, lower and hide lines in the log file as well as to launch scripts via triggers.
It has the following features:
- Runs status gathering programs
- It enables data translation and display with filters
- Highly configurable interface
- Keyboard accelerators
- Logfile rotation catching
- Supports a system wide entry database
You can visit xLogmaster here.
Lnav is an advanced log file viewer aimed at the small-scale Linux environments. It helps systems administrators watch and their log files from within the terminal. It is very easy to setup since it requires no server and no complex pre-configuration.
Lnav has the following features :
- Watch and analyze log files from a terminal.
- Seamless Log file directory scanning
- Automatic detection of file formats
- Unpacking compressed files on the fly
- Help highlight the important log parts
- Filter out the noise in the log
You can find out more about Lnav here.
Logwatch is a multipurpose, easily customizable and powerful log analyzer. System administrators can receive a single report with summarized logs from different systems. It also allows to create custom scripts , insert additional plugins and provides a periodical report which is specified by user criteria. It has the ability to scan the log files and outputs the data in a human readable form.
- It offers a full report that contains all the actions that occurred on a server
- Easy to use for regular users and system administrators
- Instant alert messaging
- Customizable dashboard
- Powerful search capabilities
- Smart filtering system
- Security breach and threat detection and analysis
Visit the homepage of Logwatch.
Nagios is a powerful centralized log management and monitoring tool. It is able to monitor system logs, application logs, event logs and syslog data. When a potential threats arise, it sends alerts to notify all involved parties who can quickly query the corresponding log data in order to promptly analyze the issue .
Nagios allows system administrators to view log data in real-time so that they can quickly solve and analyze problems as they occur.
Here are some features of Nagios :
- Increased security
- Increased services, application and server availability
- Fast detection of protocol failures and network outages
- Historical archiving of all events
- Fast detection of failed processes, cron jobs, services, and batch jobs
- Audit compliance
- Regulatory compliance
Visit Nagios website here.
GoAccess is an interactive open source log analyzer that runs in a terminal or in a browser, i.e. web based . It provides real time quick analysis and overview of web server statistics. it it is able to generate a self-contained, complete real-time HTML report (great for monitoring, analytics and data visualization).
Its main features are :
- Offers systems administrators HTTP statistics report on the fly
- Real-time, fast, millisecond updates
- Most web log formats are supported (Nginx, Apache, Elastic Load Balancing, Amazon S3 ,CloudFront, etc)
- Beautiful bootstrap and terminal dashboards
Visit GoAccess here.
journalctl is a command line utility that is used to view log messages in the system journal. When used without any parameters or switches, it will show the whole content of the system journal displayed in a pager (less is used by default ). Options and filters can be used to modify the output of journalctl. Options for instance can control the number of lines displayed, enables ‘follow’ mode, alter the displayed field, indicate a time range, etc. Filters however control the display of services and units information, etc.
You can find out more about journalctl in our detailed article here.
LOGalyze is a centralized open source network monitoring and log management software. LOGalyze is the right choice when it comes to managing all of your log data in one place. It supports network devices, Linux/Unix servers, Windows hosts. It offers a real-time event detection as well as advanced search capabilities.
It allows system administrators to easily collect, analyze and normalize log data from any device as well as to define alerts and events by correlating any log data.
LOGalyze has the ability to identify the collected logs, classifies them by source host, severity and type and splits them into different fields before finally storing them for future analysis.
LOGalyze has the following features :
- Offers real-time correlated event detection and multi-dimensional statistics
- Straightforward incident review capabilities and management
- Provides plug-in style Alert modules which notifies other parties when an event is triggered
- Provides a web-based customizable user interface
You can visit LOGalyze website here.
KSystemLog displays all log files content of your system, grouped by General services ( Authentication, Default system log, Kernel, X.org…), and optional Services ( Cups, Apache etc, …).
It has many features that help to read your log files :
- Log lines marking severity wise
- Simultaneous display of several logs using tabbed views
- Real time log display
- Provides detailed log lines information
You can find out more about KSystemLog here.
You have seen some of the most used open source log management applications. Although we have not covered all available software in the market and whether you are a simple user or a system administrator, you can use any of the ones we have outlined above for your log handling purposes. We will be updating this article with other log management applications in the future. If you know other similar applications, we would appreciate if you could leave a comment so that we can update this article accordingly.