Linux System Logs: How Do They Work

What is the Linux System Log?

The various parts of the built in Linux logging system records almost all actions (especially errors and security alerts) that take place on your system and provides administrators with a great resource in analyzing system activities. Log files provide evidence of all system activity, including unauthorized access, so these files can provide evidence of hacker intrusions so you need to understand how to these files can be manipulated in order to obfuscate intrusions. The Linux logging system records information based on the series of configurable rules and can be tampered with by intruders so it is important to understand how the system functions.

How Linux Logging Works

Developed in the mid-1980s, syslog created by Eric Allman, became a standard for logging system and application events in Linux and its big brother, Unix. What made the syslog protocol popular is that it defined a standard message format specifying the details of an event, useable by the operating system, applications, and error generating devices. This format includes:

• Timestamp
• Type
  • The type of event is referred to as a facility value and defines what is generating the event message.  You can read more about the the message formatting in the in the rsyslog rule formatting portion of the article.
• Severity
• Details of an event.

There are main linux logging applications Rsyslog: Wit the “r” in the name standing for “rocket fast”, speed is the name of the game for rsyslog and the rsyslog application is the standard logging package for many Linux distributions. Systemd-journald: As you can guess by its name, this is part of the Systemd application for system startup and initialization. While many Linux distributions use this for logging, it obviously uses a completely different way of reporting and storing system and application events compared to Rsyslog because no one ever wants to make life easy for Linux system users. We are going to spend sometime taking a look at both logging applications as you are likely to encounter them in the wild, either as a system defender or when you attempt to break into the banking system of some emerging country.

The rsyslog Logging Daemon

Linux uses a daemon (A daemon is a computer program that runs as a background process, rather than being under the direct control of an interactive user. The process names of a daemon end with the letter d, for clarification that the process is in fact a daemon, and for differentiation between a daemon and a normal computer program) called syslogd to log events on your system. It is important to note that there are several variations of syslog depending on the Linux distribution you are using that have some minor variations in function and design, so what we cover here (using a Debian based system) may not apply on your system, including rsyslog and syslog-ng, are used on different distributions of Linux. One of the first things to know about rsyslog is that it is pervasive across your Linux system, so you need to figure out which one to work with, so use the locate function to find the “rsyslog.conf” file as seen in the screenshot below.

How Do You Configure Linux Logs?

The rsyslog Configuration File

The rsyslog system uses the rsyslogd program to monitor events and, like most applications in Linux, is managed by a plaintext configuration file named “rsyslog.conf” located in the /etc directory. We will open that file in the Nano editor with the following command:

nano /etc/rsyslog.conf

After the command runs, the configuration file will be open and you should see something that looks like this:

The rsyslog.conf file is well documented and is self explanatory.  The main area of focus for us is going to be the Rules section.

The rsyslog Logging Rules

The logging rules in the rsyslog.conf file determine:

• The information is logged;
• Which pro­grams have their messages logged; and,
• Where logs are stored. 

As system administrator, be aware that any decent hacker, will look to target these rules to modify what the system is logging when they are attacking your system.   Scrolling down the rsyslog.conf file to the rules section, you will see the following:

Each line of the rsyslog.conf file is a separate logging rule that determines which and where messages are logged.

rsyslog Rule Formatting

The basic rule format is:

facility.priority                  action

The term “facility” indicates the Linux program, (e.g. mail, authentication, kernel) whose messages we want logged.  The term “priority” determines what kind of messages to log for that program (programs generate different level of messages to indicate the severity of an error). The “action” keyword tell you where the log is stored. 

Understanding the Facility Portion

The facility term refers to the application which is generating a particular log file and may refer to any of the following applications:

Code	Keyword	Description
0	kern	Messages generated by the system kernel.
1	user	Messages generated by user events
2	mail	Messages from a mail application
3	daemon	Messages from system applications running in background
4	auth	Security or authentication messages
5	syslog	Messages generated by the logging application itself
6	lpr	Printer messages
7	news	Messages from the news application
8	uucp	Messages from the Unix-to-Unix copy program
9	cron	Messages generated from the cron job scheduler
10	authpriv	Security or authentication messages
11	ftp	File Transfer Protocol application messages
12	ntp	Network Time Protocol application messages
13	security	Log audit messages
14	console	Log alert messages
15	solaris-cron	Another scheduling daemon message type
16-23	local0-local7	Locally defined messages

In some rsyslog.conf files, the rule section may have an asterisk wildcard (*) in place of a word refers to all facilities.   Additionally, you can select more than one facility by listing them separated by a comma.

Understanding the Priority Portion

Priority tells the Linux logging system which levels of messages to record. Messages have a range of priority, ranging from “debug“, which has the lowest priority, to “panic” which has the highest priority, while if the priority of a facility is “*“, all messages get logged.  When spec­ifying a priority,  you Linux system logs all messages of that priority and higher (if you specify a priority code of “err“, the system logs messages classified as “err“, “criticism“, “alert“, “emerge“, and “panic“).  The full list of valid codes for priority, in ascending order are:

Code	Keyword	Description
0	emerg	An event that causes the system to be unusable.
1	alert	An event that requires immediate attention.
2	crit	An event that is critical but doesn’t require immediate attention.
3	err	An error condition that allows the system or application to continue.
4	warning	A non-normal warning condition in the system or application.
5	notice	A normal but significant condition message.
6	info	An informational message from the system.
7	debug	Debugging messages for developers.

• warn
• error
• err
• emerg (depends where you read)
• panic

Understanding the Action Portion

The action is usually a filename /location where the logs are recorded, usually being in the /var/log directory with a filename that describes the facility that generated them, such as mail and would be sent to /var/log.mail.log.

Log Rule Examples

mail.* /var/log/mail This rule logs all mail events, regardless of priority to /var/log/mail. kern.crit /var/log/kernel This example will log kernel events of critical (crit) priority or higher to /var/log/kernel.  *.emerg * This logs all events of the emergency (emerg) priority to all logged­ on users. By modifying these rules, a system intruder can determine where the log files are located, change the priorities, or even disable specific logging rules.

How to Manage and Monitor Linux system Logs

logrotate: Managing Logs

If left unattended, log files constantly fill up, taking up an ever increasing amount of space, eventually the entire hard drive.  Alternatively, deleting log files too frequently means logs may be unable to be used in investigations at some future point in time.  The Linux command “logrotate” can help deal with this situation;  log rotation is the process of moving logs to a new location for the purpose of archiving them and a fresh log file. That archived loca­tion gets cleaned up after a specified period of time. Most Linux systems regularly rotate their log files based on a cron job that triggers the execution of the logrotate utility.

Viewing the logroate.conf File

The logrotate.conf file chooses the frequency with which your system rotates its logs. You can configure the logrotate utility to choose the regu­ larity of your log rotation with the /etc/logrotate.conf text file. Let’s open it with a text editor and take a look:

In the image above, you should attempt to understand that:

• Log files are rotated weekly, this is the default setting;
• 4 weeks worth of backlogs are kept.  You can extend this to 52 if you want to keep them for a year;
• Linux creates new (empty) log files after rotating old ones.  and,
• You can also choose to compress your rotated log filesx; and
• When rotated, log files are renamed and pushed toward the end of the chain of logs as a new log file is created, replacing the current log file. For instance, /var/log.auth will become /var/log.auth.1, then /var/log.auth.2, and so on. If you rotate logs every four weeks and keep four set of backups, you will have /var/log.auth.4, but no /var/log.auth.5, meaningthat /var/log.auth.4 will be deleted rather than being pushed to /var/log/ auth.5.

Discovering Intruders

Once a hacker compromises a Linux system,  the often disable logging in order remove any evidence of intrusion in the log files thereby reduces the chances of detection. This can be done a number of ways.

Removing Evidence

Hackers may  want to remove any logs of their activity. They could simply open the log files and precisely remove, line by line, remove any logs detailing  activity. However, this could be time­consuming and leave time gaps in the log files, which would look suspicious. Also, deleted files can generally be recovered by a skilled forensic investigator.

Shred’ing Files

A more secure solution to get rid of log files is to shred them.  Since it is possible to recover delete files, you need a way increase the difficulty of file recovery by overwriting the deleted files it several times, making it much harder to recover as the more times the file is overwritten, the harder it is to recover.  The “shred” command does just this. In its most basic form, the shred’s syntax is :

shred

The “shred” command deletes the file and overwrite it 4 times by default; keep in mind that each overwrite takes time, so for very large files, shredding may become time­ consuming.

As you can see from the screen shot above, the command has a number of flags:

• “-f ” option: Which changes the file permissions on the files to allow them to be overwritten; and,
• The “–n” option: which lets you choose how many times to overwrite the files. As an example, we’ll shred the log files in /var/log/auth.log 7 times, and
• The path of the file we want to shred, we include the wildcard asterisk so we’re shredding not just the auth.log file, but also any logs that have been created with logrotate, such as auth.log.1, auth.log.2, and so on.

As you can see, when you open the folder, it is gibberish.  Now if the security engineer or forensic investigator examines the log files, they will find nothing of use because none of it is recoverable.

Disabling Logging

Another option used covering  tracks is to simply disable logging. When a hacker takes control of a system, they could immediately disable logging to prevent the system from keeping track of their activities. This, of course, requires root privileges.  To disable all logging, the hacker could simply stop the rsyslog daemon. Stopping any service in Linux uses the same syntax, shown here:

service rsyslog stop

Now Linux will not generate log files until the service is restarted, enabling a hacker to operate without leaving behind any evidence in the log files.