Updated – Ubuntu logs system events into the log files in order to help administrators maintain, analyze and diagnose system related issues and applications problems.
Log files contain messages related to kernel, services and applications events that are kept on a centralized repository of log files under /var/log directory.
There are various log files in Linux Ubuntu for different informations. For instance, there is a default log file dedicated to the system, another for security messages another for cron tasks. These log files are actually plain text (ASCII) written in a standardized log file format. Many of these log files are generated by the so called system log daemon, syslogd while some other applications generate their proper logs by writing to files under the directory /var/log.
The log files generated can therefore be classified into four categories: event Logs, application Logs , system Logs and service Logs.
In this article, we will provide an overview of Ubuntu log files, and show how to configure and use syslogd. You will also learn how log rotation works and how to view and read the log files. A brief introduction to systemd will be provided at the end .
System Logs
The system logs are rather related to the way Ubuntu system is functioning without including the additional user programs or applications.Some examples consist of system daemons, authorization mechanisms, system messages and of course syslog, the all-encompassing system log per se. In the sections below we will introduce the most important logs on Ubuntu and in similar Linux distros.
Daemon Log
A daemon, is a process or program that runs in the background without any intervention from users. You might perhaps have noticed that most daemons’ names end with the letter “d”. For instance sshd which takes care of SSH remote access connectivity , rsyslogd (see below), httpd which is the daemon that manages the Apache server, mysqld which is MySQL database daemon, the Bluetooth HCI daemon hcid, … Daemons are often started at boot time in Linux or Unix.The Daemon Log is stored under the directory /var/log/daemon.log and provides information about the application daemons and the running system.
You may be interested to read our article about Linux directories.
System Log
Most of the system information is stored in the system log /var/log/syslog by default. Information contained within this log may not be found in other logs but might be very useful in certain situations. Events from some applications which do not have their own log file, might be stored in the system log file which contains (for Ubuntu, Debian and similar distros) also all logs that used to be written in the file /var/log/messages.
Authorization Log
Authorization related events are tracked in the authorization log. This includes user password prompt mechanisms like the Pluggable Authentication Module system (PAM), sshd remote logins, the sudo command…The authentication log file, which is useful for user logins investigation as well as the sudo command usage, can be found under the directory /var/log/auth.log.
For instance, in order to see sshd logins related information from the authorization Log file , you could use the grep based command below:
grep sshd /var/log/auth.log | less
Debug Log
The debug log is stored under the directory /var/log/debug. It contains detailed debug related messages from the system ( Ubuntu or Debian or similar distro) and also from the applications which log their corresponding events/messages to syslogd at the DEBUG level.
Kernel Log
Ubuntu Linux kernel sends its log events or messages to the kernel log file /var/log/kern.log. This can be used by system administrators to troubleshoot kernel related issues.
Kernel Ring Buffer
The kernel ring buffer is an area in the running kernel and not a log file in its right. It can be used to make queries related to kernel boot-up messages using the dmesg tool.
In order to display the messages, run the command below:
dmesg | less
Or to look up lines that mention ssh for instance, use the command as follows:
dmesg | grep ssh | less
All bootup messages, by default, are sent by the system initialization script, i.e. /etc/init.d/bootmisc.sh, to the /var/log/dmesg file also.
Read: Best Linux log file management and monitoring tools
Application Logs
Many application logs are also stored under /var/log directory. If you display of /var/log, you will find many common application names such as /var/log/samba which is the log file for the Samber server, or /var/log/apache2 which contains the logs for the Apache 2 web server.
In the following section, some example logs will be provided along with the information they contain.
Apache HTTP Server Logs
As was briefly mentioned above, Apache2 default installation on Ubuntu, creates a subdirectory dedicated to storing its log events/messages, i.e. /var/log/apache2. This contains two log files each of which has a distinct purpose:
- /var/log/apache2/access.log – stores every page that was served and every file that was loaded by the web server.
- /var/log/apache2/error.log – Ubuntu error log messages triggered by the HTTP server.
By default, every time a file or page is accessed by Apache, many parameters will be recorded on the log file, such as the IP address, date, time, HTTP result code, browser identification string as well as the text of the actual query.
Samba SMB Server Logs
The Samba (Server Message Block Protocol (SMB) server ), is commonly used for file sharing between an Ubuntu machine and other machines which support the SMB protocol. Three different types of logs are kept by Samba in the subdirectory /var/log/samba:
- log.nmbd – Samba’s NETBIOS related messages over IP functionality ( network related)
- log.smbd – Samba’s SMB/CIFS functionality related messages (print and file sharing related)
- log.[IP_ADDRESS] – Messages dealing with service requests that are coming from the IP address mentioned in the log file name.
For more on how to install Samba on Ubuntu, you may want to read our article on how to install Samba on Ubuntu.
Non-Human-Readable Logs
The /var/log subdirectory contains some log files which are designed to be parsed by applications and not necessarily by humans. You will find below some examples of such log files :
- The login failures log, /var/log/faillog, can be parsed by the faillog command. For instance, in order to display the recent login failures, run the command:f
aillog
- The last logins log , /var/log/lastlog is an example of a log that should not be examined by humans, but should be used instead with the lastlog command. For instance, to display the logins list using the lastlog command, run the following (one pager):
lastlog | less
- Login Records Log, /var/log/wtmp contains login record. It is not used to display recent logins, unlike /var/log/lastlog, but rather, it is used by other tools such as the ‘who’ command in order to show currently logged-in users.
System logging daemon (syslogd)
Also known as sysklogd, the Linux system logging daemon syslogd, receives logging messages from various sources and redirects them to the appropriate file destination (or on network).
Messages logged to this daemon usually consist of common elements such as time-stamps and system hostnames along with the specific log information.
rsyslogd daemon
The daemon called rsyslogd is a system tool that helps in logging messages both locally and remotely. rsyslogd is an enhanced alternative if not a replacement for sysklogd which enables extended filtering, provides input and output modules, guarantees secure message relay and allows support for TCP or UDP transportation.
Configuration of syslogd
The configuration file of syslogd daemon is the file /etc/syslog.conf. Each entry or record in this file contains two fields which are the selector and the action. The selector field indicates a facility that would need to logged, like for instance the auth facility which is related to authorizations and also a priority level (such as info or warning) to log such information.
The action field on the other hand, contains a target for the log message, like for example the hostname of a distant machine to which the log information is to be sent, or a standard log file (i.e. /var/log/syslog). In the next section, we outline some common Linux syslog fields.
Here is for instance an Ubuntu log message with the default format. It originates from the sshd daemon, which handles system remote logins. This example log message indicates a failed login attempt:
Jun 1 10:11:10 net2_server sshd[41458] : Failed password for root from 10.1.1.2 port 22 ssh2
Syslog fields
you will find below the most commonly used fields in syslog that might be useful when analyzing or troubleshooting issues (some of these fields do not appear in the standard format).
- The timestamp: It indicates the date and time the message was triggered on the original system.
- The hostname: (“net2_server” in the example shown above) specifies the host name or the system name that sent the message.
- The app-name: indicates the name of the application that triggered the message.
- The priority: or pri for short, specifies the urgency or severity of the event. It’s a concatenation of two fields: facility and severity. The facility indicates the type of process that triggered the event ( kernel messages or local applications) and it ranges from 0 to 23. The severity can be an emergency or a debug event or an error…it ranges from 0–7 where 7 is for a debug event.
Echoing Messages to syslogd With Logger
If you want to enter messages into the Ubuntu system Log (i.e. /var/log/syslog) for the username your_username you could run a command that looks like the following in your terminal:
logger your_message
A new line will be inserted in the file /var/log/syslog like shown below:
where the command was logger “message from net2”. At the end of this article we will outline the way to see the logs via GUI.
Log Rotation
When displaying listings of directories in /var/log or any of its subfolders, you may find log files having names like daemon.log.1, daemon.log.2.gz. These are known to be the rotated log files. Indeed, Logrotate, which is a default system utility, handles automatically this rotation as well as the compression of log files.
It is possible to manually alter the log rotation frequency by editing the file /etc/logrotate.conf.
Logrotate is invoked from the /etc/cron.daily/logrotate system wide cron script and its configuration file is located at /etc/logrotate.conf. Single configuration files can be inserted into /etc/logrotate.d.
if there was no such rotation and compression mechanism, log files would eat up all disk space.
You may be interested to read our articles about compression tools on Linux such as Tar and Gzip.
Manipulating system log files using the terminal
Displaying log files
For those acquainted with the nano editor, you can run the command below in order to edit a log file, .e.g. /var/log/logfile.log :
nano /var/log/logfile.log
Or you can change to the /var/log directory via the command ls and then simply nano the file directly without the full path.
In order to exit, just press Ctrl+X. You will be prompted to Save in case you made changes, but in most cases you do not change the log file.
You can find a complete guide about nano in our article on how to use nano.
Viewing Files
To look at the content of a file, you can use the less command, which displays one screen at a time:
less logfile.log
Press q to quit.
Viewing the Beginning of Files
In order to see the first 10 lines of a log file, you could use the head command as follows:
head logfile.log
for instance to view the first ten lines of the log file boot.log, issue the command below(if you have sudo privileges) :
head boot.log
You can add the -n option to see some other number of lines , for instance, to see the first four lines of the file boot.log, run the command :
head -n 4 boot.log
Viewing the End of Files
Now to view the last 10 lines of the log file, use the command tail as shown below:
tail logfile.log
Using the -n switch :
tail -n 4 logfile.log
Live log changes display
Using the -f (“follow”) option, will enable you to see the latest additions to the file content as they get inserted. This feature might be useful for system administrators for monitoring applications’ output in real time. To see this in action, run the command below using your specific log file :
tail -f logfile.log
To stop or quit, press Ctrl+C .
Searching through log files
Since log files can grow to large sizes, the grep command can help you strip out the content of interest. For instance, in order to find all the log lines in a file which contain the word “process”,you can execute command below:
grep “process” logfile.log
The snapshot above shows the lines in the file kern.log which contain the word “process”.
In case the returned result spans many pages, you can pipe it out to the command less as follows :
grep “process” logfile.log | less
Viewing system log files using the GUI
In order to display the content of log files using a simple, easy to use, graphical tool, open up the Log File Viewer utility from your Dash :
Once you click on the Log file Viewer shortcut, you would need to press the install button if it is not installed in your computer.
Once this is done, hit the Launch button and you are all set.
By default, the Log File Viewer, as shown above in the snapshot, shows a number of logs, including your package manager log (file dpkg.log), system log (file syslog), graphical server log (file Xorg.0.log) and authentication log (file auth.log).
You will be able to display the logs in a separate window in case a new log event is inserted, it will show up automatically in this window.
If you want to view other log files, for instance, a specific application log file, you can just click on the the File menu :
Then select Open, to open the log file from its location. It will show up alongside the standard log files and will be automatically updated and monitored much like the other logs.
As mentioned earlier in this article, you can display a message manually in the log file syslog by issuing the command :
logger “your_message”
systemd
systemd, which is a service and process manager, is shipped with many Linux distributions. systemd has its own logging utility called journald that can complement or replace syslog. Journald logging is completely different when compared to systemd.
To learn more about systemd and journald , you can refer to our two articles :
– How to use systemd to troubleshoot Linux problems
– How to analyze Linux systemd logs using journalctl advanced filtering options
Conclusion
In this article you have seen the different log files that exist on Ubuntu and similar Linux distributions. Some example application logs like Apache and Samba were also provided.
You have also learned about syslog fields and log rotation which consists in compressing and renaming log files. Using the terminal as well as the GUI, you were able to view the content and write to log files .
At first glance, all these log files are overwhelming. Fortunately, there are different tools designed specifically for log management. One example is Papertrail’s solution, which centralizes logs in a single location, allows you to manage what type of logs each user can access, and establishes infrastructure-wide policies, among others.
If you like the content, we would appreciate your support by buying us a coffee. Thank you so much for your visit and support.
