Logging Reference

Logging in Riak KV is handled by a Basho-produced logging framework for Erlang called lager.

lager provides a number of configuration options that you can use to fine-tune your Riak cluster’s logging output. A compact listing of parameters can be found in our configuration files documentation. A more thorough explanation of these options can be found in this document.

Log Directory

Riak’s log files are stored in a /log directory on each node. The location of that directory differs from platform to platform. The table below shows you where log files are stored on all supported operating systems.

OS	Directory
Ubuntu, Debian, CentOS, RHEL	`/var/log/riak`
Solaris, OpenSolaris	`/opt/riak/log`
Source install and Mac OS X	`./log` (where the `.` represents the root installation directory)

Log Files

Below is a list of files that can be found in each node’s /log directory:

File	Significance
`console.log`	Console log output
`crash.log`	Crash logs
`erlang.log`	Logs emitted by the Erlang VM on which Riak runs.
`error.log`	Common errors emitted by Riak.
`run_erl.log`	The log file for an Erlang process called `run_erl`. This file can typically be ignored.

Log Syntax

Riak logs tend to be structured like this:

<date> <time> [<level>] <PID> <prefix>: <message>

The date segment is structured YYYY-MM-DD, time is structured hh:mm:ss.sss, level depends on which log levels are available in the file you are looking at (consult the sections below), the PID is the Erlang process identifier for the process in which the event occurred, and the message prefix will often identify the Riak subsystem involved, e.g. riak_ensemble_peer or alarm_handler (amongst many other possibilities).

Warning: Log messages may contain newline characters

As of Riak KV 3.0.11 a few of the log messages may contain newline characters, preventing reliable identification of the end of each log when attempting log files ingestion by external tools.

A known workaround is ingesting not the logs enabled by the log.console configurable parameter but rather the logs as enabled by the log.syslog configurable parameter and processed by syslog, e.g. exploiting the no-multi-line option (e.g. see this StackExchange topic answer) - or equivalent - of syslog implementations.

The exception to this syntax is in crash logs (stored in crash.log files). For crash logs, the syntax tends to be along the following lines:

<date> <time> =<report title>====
<message>

Here is an example crash report:

2014-10-17 15:56:38 =ERROR REPORT====
Error in process <0.4330.323> on node 'dev1@127.0.0.1' with exit value: ...

Log Files

In each node’s /log directory, you will see at least one of each of the following:

File	Contents
`console.log`	General messages from all Riak subsystems
`crash.log`	Catastrophic events, such as node failures, running out of disk space, etc.
`erlang.log`	Events from the Erlang VM on which Riak runs
`run_erl.log`	The command-line arguments used when starting Riak

Log File Rotation

Riak maintains multiple separate files for console.log, crash.log, erlang.log, and error.log, which are rotated as each file reaches its maximum capacity of 100 KB. In each node’s /log directory, you may see, for example, files name console.log, console.log.0, console.log.1, and so on. Riak’s log rotation is somewhat non traditional, as it does not always log to *.1 (e.g. erlang.log.1) but rather to the oldest log file.

After, say, erlang.log.1 is filled up, the logging system will begin writing to erlang.log.2, then erlang.log.3, and so on. When erlang.log.5 is filled up, it will loop back to erlang.log.1.

SASL

SASL (System Architecture Support Libraries) is Erlang’s built-in error logger. You can enable it and disable it using the sasl parameter (which can be set to on or off). It is disabled by default. The following would enable it:

sasl = on

Error Messages

By default, Riak stores error messages in ./log/error.log by default. You can change this using the log.error.file parameter. Here is an example, which uses the default:

log.error.file = ./log/error.log

By default, error messages are redirected into lager, i.e. the log.error.redirect parameter is set to on. The following would disable the redirect:

log.error.redirect = off

You can also throttle the number of error messages that are handled per second. The default is 100.

log.error.messages_per_second = 100

Crash Logs

Riak crash logs are stored in ./log/crash.log by default. You can change this using the log.crash.file parameter. This example uses the default:

log.crash.file = ./log/crash.log

While crash logs are kept by default, i.e. the log.crash parameter is set to on, you can disable crash logs like this:

log.crash = off

Crash Log Rotation

Like other Riak logs, crash logs are rotated. You can set the crash logs to be rotated either when a certain size threshold is reached and/or at designated times.

You can set the rotation time using the log.crash.rotation parameter. The default is $D0, which rotates the logs every day at midnight. You can also set the rotation to occur weekly, on specific days of the month, etc. Complete documentation of the syntax can be found here. Below are some examples:

$D0 - Every night at midnight
$D23 - Every day at 23:00 (11 pm)
$W0D20 - Every week on Sunday at 20:00 (8 pm)
$M1D0 - On the first day of every month at midnight
$M5D6 - On the fifth day of the month at 6:00 (6 am)

To set the maximum size of the crash log before it is rotated, use the log.crash.size parameter. You can specify the size in KB, MB, etc. The default is 10MB.

Other Crash Log Settings

The maximum size of individual crash log messages can be set using the log.crash.maximum_message_size, using any size denomination you wish, e.g. KB or MB The default is 64 KB. The following would set that maximum message size to 1 MB:

log.crash.maximum_message_size = 1MB

Syslog

Riak log output does not go to syslog by default, i.e. the log.syslog setting is set to off by default. To enable syslog output:

log.syslog = on

If syslog output is enabled, you can choose a prefix to be appended to each syslog message. The prefix is riak by default.

log.syslog.ident = riak

Syslog Level and Facility Level

If syslog is enabled, i.e. if log.syslog is set to on, you can select the log level of syslog output from amongst the available levels, which are listed in the table below. The default is info.

alert
critical
debug
emergency
error
info
none
notice
warning

In addition to a log level, you must also select a facility level for syslog messages amongst the available levels, which are listed in the table below. The default is daemon.

auth
authpriv
clock
cron
daemon
ftp
kern
lpr
mail
news
syslog
user
uucp

In addition to these options, you may also choose one of local0 through local7.

Console Logs

Riak console logs can be emitted to one of three places: to a log file (you can choose the name and location of that file), to standard output, or to neither. This is determined by the value that you give to the log.console parameter, which gives you one of four options:

file - Console logs will be emitted to a file. This is Riak’s default behavior. The location of that file is determined by the log.console.file parameter. The default location is ./log/console.log on an installation from source, but will differ on platform-specific installation, e.g. /var/log/riak on Ubuntu, Debian, CentOS, and RHEL or /opt/riak/log on Solaris-based platforms.
console - Console logs will be emitted to standard output, which can be viewed by running the riak attach-direct command
both - Console logs will be emitted both to a file and to standard output
off - Console log messages will be disabled

In addition to the the placement of console logs, you can also choose the severity of those messages using the log.console.level parameter. The following four options are available:

info (the default)
debug
warning
error

Enabling and Disabling Debug Logging

Checkout Cluster Operations: Enabling and Disabling Debug Logging