Inspecting a Node

When inspection of a Riak node to gather metrics on performance or potential issues is desired, a number of tools are available to help, and are either included with Riak itself or made available through the Riak community.

This guide provides starting points and details on some of the available tools for inspecting a Riak node.

riak-admin status

riak-admin status is a subcommand of the riak-admin command that is included with every installation of Riak. The status subcommand provides data related to the current operating status for a node. The output of riak-admin status is categorized and detailed below.

Please note, for some counters, such as node_get_fsm_objsize, a minimum of 5 transactions is required for statistics to be generated.

Performance

We recommended checking stats every 90-120 seconds for best performance.

Repeated runs of the riak-admin status command should not have a negative performance impact as the statistics are cached internally in Riak.

Active Stats

Active Stats represent current activity on the node.

Average Stats

Average Stats represent an average calculated as (total occurrences / number of samples) since this node was started. In the below stats the sample time is 1s, giving us a per-second average. Currently, the only Average Stats are reported by Sidejob - an Erlang library that implements a parallel, capacity-limited request pool.

One-Minute Stats

One-Minute Stats represent the number of times a particular activity has occurred within the last minute on this node.

General One-Minute Stats

FSM Time

FSM Time Stats represent the amount of time in microseconds required to traverse the GET or PUT Finite State Machine code, offering a picture of general node health. From your application’s perspective, FSM Time effectively represents experienced latency. Mean, Median, and 95th-, 99th-, and 100th-percentile (Max) counters are displayed. These are one-minute stats.

GET FSM Siblings

GET FSM Sibling Stats offer a count of the number of siblings encountered by this node on the occasion of a GET request. These are one-minute stats.

GET FSM Objsize

GET FSM Objsize Stats represent a view of the sizes of objects flowing through this node’s GET FSMs. The size of an object is obtained by summing the length of the bucket name, key, serialized vector clock, value, and serialized metadata of each sibling. GET FSM Objsize and GET FSM Siblings are inextricably linked. These are one-minute stats.

Total Stats

Total Stats represent the total number of times a particular activity has occurred since this node was started.

Timestamps

Some of the Erlang applications that Riak is comprised of contribute statistics to riak-admin status. The below timestamps record, in Epoch time, the last time statistics for that application were generated.

Ring

General ring information is reported in riak-admin status.

CPU and Memory

CPU statistics are taken directly from Erlang’s cpu_sup module. Documentation for which can be found at ErlDocs: cpu_sup.

Memory statistics are taken directly from the Erlang virtual machine. Documentation for which can be found at ErlDocs: Memory.

Erlang VM

The below statistics describe properties of the Erlang VM.

Miscellaneous Information

Miscellaneous Information provide additional details particular to this node.

Pipeline Metrics

The following metrics from from riak_pipe are generated during MapReduce operations.

Application and Subsystem Versions

The specific version of each Erlang application and subsystem which makes up a Riak node is present in the riak-admin status output. Each application is linked below next to it’s version identifier.

Riak Search Statistics

The following statistics related to Riak Search message queues are available.

Note that under ideal operation and with the exception of riak_search_vnodes_running these statistics should contain low values (e.g., 0-10). Presence of higher values could be indicative of an issue.

riak-debug

The riak-debug command is used to identify and diagnose common problems with your Riak KV nodes.

riak-debug also runs riak-admin diag, which runs a small suite of diagnostic checks against a Riak KV node to discover common problems. It often offers recommendations about how to resolve those problems as well.

Strong Consistency Stats

Riak tabulates a variety of stats related to Riak’s optional strong consistency feature. The table below lists those stats.

GET-related stats

PUT-related stats

riak-admin diag

Running riak-admin diag by itself will perform a check of all of the data partitions in your cluster. It will return a listing of partitions that have been checked, each of which looks something like this:

{1392993748081016843912887106182707253109560705024, % the partition checked
 'dev-rel@127.0.0.1'},                              % that partition's nodename

At the end of that (potentially very long) listing of checked partitions, it will print notices, warnings, and other pieces of information about issues that it has found, including date/time, message type, and a detailed description. Here’s an example:

15:34:52.736 [warning] Riak crashed at Wed, 07 Dec 2011 21:47:50 GMT, leaving crash dump in /srv/riak/log/erl_crash.dump. Please inspect or remove the file.
15:34:52.736 [notice] Data directory /srv/riak/data/bitcask is not mounted with 'noatime'. Please remount its disk with the 'noatime' flag to improve performance.

Messages bear the following types (derived from syslog security levels):

• debug
• info
• notice
• warning
• error
• critical
• alert
• emergency

Command flags

Attaching the --help flag will return a list of flags and commands that can be used with Riaknostic:

Usage: riak-admin diag [-d <level>] [-l] [-h] [--export] [check_name ...]

-h, --help            Display help/usage dialogue
-d, --level           Minimum message severity level (default: notice)
-l, --list            Describe available diagnostic tasks
--export              Package system info in '/export.zip'
check_name            A specific check to run

Running riak-admin diag with the --list flag will return a list of available diagnostic checks. The following checks are available:

The --level flag enables you to specify the log level and thus to filter messages based on type. You can pass in any of the message types listed above (debug, info, etc.).

The --level flag can be used when running riak-admin diag with or without specifying a diagnostic check.

Contributing

Do you have an idea that would help us improve Riaknostic? If so, fork the GitHub repository and send us a pull request with your changes. The code is documented with edoc, so give the API Docs a read before you contribute.

If you want to run the Riaknostic script while developing and you don’t have it hooked up to your local Riak installation, you can invoke it directly like so:

./riaknostic --etc ~/code/riak/rel/riak/etc --base ~/code/riak/rel/riak --user `whoami` [other options]

Those extra options are usually assigned by the riak-admin script for you, but here’s how to set them:

• --etc — The location of your Riak configuration directory (usually /etc). In the example above, configuration is in the generated directory of a source checkout of Riak.
• --base — The “base” directory of Riak, usually the root of the generated directory or /usr/lib/riak on Linux. Scan the riak-admin script for how the RUNNER_BASE_DIR variable is assigned on your platform.
• --user — The user/UID as which the Riak node runs. In a source checkout, it’s the current user; on most systems, it’s riak.

Related Resources

• The riak-admin configuration management tool
• Riaknostic
• HTTP API Status