V3 Multi-Datacenter
Replication Operations

This document explains how to manage replication with the riak-repl command. Some of these commands can be set or behavior altered by setting appropriate configuration values.

All commands need to be run only once on a single node of a cluster for the changes to propagate to all other nodes. All changes will persist across node restarts and will automatically take effect when nodes are added to the cluster.

Cluster Connectivity

clustername

Set the clustername for all nodes in a Riak cluster.

  • Without a parameter, returns the current name of the cluster
  • With a parameter, names the current cluster

To set the clustername:

  • Syntax: riak-repl clustername <clustername>
  • Example: riak-repl clustername Boston

To get the clustername:

  • Syntax: riak-repl clustername
  • Example: riak-repl clustername

connect

The connect command establishes communications from a source cluster to a sink cluster of the same ring size. The host:port of the sink cluster is used for this. The IP and port to connect to can be found in the advanced.config of the remote cluster, under riak_core and cluster_mgr.

The host can be either an IP address

  • Syntax: riak-repl connect <ip>:<port>
  • Example: riak-repl connect 192.168.2.1:9080

…or a hostname that will resolve to an IP address.

  • Syntax: riak-repl connect <host>:<port>
  • Example: riak-repl connect Austin:9080

disconnect

Disconnecting a source cluster from a sink cluster.

You may define a host:port combination

  • Syntax: riak-repl disconnect <host>:<port>
  • Example: riak-repl disconnect 192.168.2.1:9080

…or use the name of the cluster.

  • Syntax: riak-repl disconnect <sink_clustername>
  • Example: riak-repl disconnect Austin

connections

Display a list of connections between source and sink clusters.

  • Syntax: riak-repl connections
  • Example: riak-repl connections

clusterstats

Displays current cluster stats using an optional ip:port as well as an optional protocol-id.

protocol-id can be one of the following:

  • cluster_mgr
  • rt_repl
  • fs_repl

The clusterstats command in use:

  • Syntax: riak-repl clusterstats <host>:<port> <protocol-id>
  • Example: riak-repl clusterstats 192.168.2.1:9080
  • Example: riak-repl clusterstats 192.168.2.1:9080 fs_repl

Realtime Replication Commands

realtime enable

Enable realtime replication from a source cluster to sink clusters.

This will start queuing updates for replication. The cluster will still require an invocation of realtime start for replication to occur.

  • Syntax: riak-repl realtime enable <sink_clustername>
  • Example: riak-repl realtime enable Austin

realtime disable

Disable realtime replication from a source cluster to sink clusters.

  • Syntax: riak-repl realtime disable <sink_clustername>
  • Example: riak-repl realtime disable Austin

realtime start

Start realtime replication connections from a source cluster to sink clusters. See also realtime enable (above).

  • Syntax: riak-repl realtime start <sink_clustername>
  • Example: riak-repl realtime start Austin

realtime stop

Stop realtime replication from a source cluster to sink clusters.

  • Syntax riak-repl realtime stop <sink_clustername>
  • Example riak-repl realtime stop Austin

Fullsync Replication Commands

These behaviors can be altered by using the advanced.config fullsync_on_connect parameter. See the Configuration Guide for more information.

fullsync enable

Enable fullsync replication from a source cluster to sink clusters. By default, a fullsync will begin as soon as a connection to the remote cluster is established.

  • Syntax: riak-repl fullsync enable <sink_clustername>
  • Example: riak-repl fullsync enable Austin

fullsync disable

Disables fullsync for a cluster.

  • Syntax: riak-repl fullsync disable <sink_clustername>
  • Example: riak-repl fullsync disable Austin

fullsync start

Starts a fullsync. If the application configuration fullsync_on_connect is set to false, a fullsync needs to be started manually. This is also used to trigger a periodic fullsync using a cron job. While a fullsync is in progress, a start command is ignored and a message is logged.

  • Syntax: riak-repl fullsync start <sink_clustername>
  • Example: riak-repl fullsync start Austin

fullsync stop

Stops a fullsync.

  • Syntax: riak-repl fullsync stop <sink_clustername>
  • Example: riak-repl fullsync stop Austin

Cascading Realtime Writes

realtime cascades

Shows the current cascading realtime setting.

  • Syntax: realtime cascades
  • Example: riak-repl realtime cascades

realtime cascades always

Enable realtime cascading writes.

  • Syntax: realtime cascades always
  • Example: riak-repl realtime cascades always

realtime cascades never

Disable realtime cascading writes.

  • Syntax: realtime cascades never
  • Example: riak-repl realtime cascades never

NAT

Note: See the V3 Multi Data Center Replication With NAT for more information.

nat-map show

Show the current NAT mapping table.

  • Syntax: nat-map show
  • Example: riak-repl nat-map show

nat-map add

Adds a NAT map from the external IP, with an optional port, to an internal IP.

  • Syntax: nat-map add <externalip>[:port] <internalip>
  • Example: riak-repl nat-map add 128.205.106.1:5555 192.168.1.2

nat-map del

Deletes a specific NAT map entry.

  • Syntax: nat-map del <externalip>[:port] <internalip>
  • Example: riak-repl nat-map del 128.205.106.1:5555 192.168.1.2

NAT changes will be applied once fullsync and/or realtime replication has been stopped and started.

Riak CS MDC Gets

proxy-get enable

Enable Riak CS proxy_get requests from a sink cluster (if proxy_get has been enabled in advanced.config).

  • Syntax: proxy-get enable <sink_clustername>
  • Example: riak-repl proxy-get enable newyorkbackup

proxy-get disable

Disable Riak CS proxy_get requests from a sink cluster (if proxy_get has been enabled in advanced.config).

  • Syntax: proxy-get disable <sink_clustername>
  • Example: riak-repl proxy-get disable newyorkbackup

add-block-provider-redirect

Provide a redirection to the <to-cluster-id> for proxy_get if the <from-cluster> is going to be decommissioned.

  • Syntax: riak-repl add-block-provider-redirect <from-cluster> <to-cluster>
  • Example: riak-repl add-block-provider-redirect "{'dev1@127.0.0.1',{1391,544501,519016}}" "{'dev3@127.0.0.1',{1299,512501,511032}}"

show-block-provider-redirect

Show the mapping for a given cluster-id redirect.

  • Syntax: riak-repl show-block-provider-redirect <from-cluster>
  • Example: riak-repl show-block-provider-redirect "{'dev1@127.0.0.1',{1391,544501,519016}}"

delete-block-provider-redirect

Delete a existing redirect such that proxy_gets go again to the original provider cluster id.

  • Syntax:* riak-repl delete-block-provider-redirect <from-cluster>
  • Example:* riak-repl delete-block-provider-redirect "{'dev1@127.0.0.1', {1391,544501,519016}}"

show-local-cluster-id

Display this cluster’s cluster-id tuple, for use with the *-block-provider-redirect commands.

Note: A cluster-id is surrounded by double quotes, which need to be included when passed to *-block-provider-redirect.

  • Syntax: riak-repl show-local-cluster-id

  • Example:

    riak-repl show-local-cluster-id

    Possible output:

    local cluster id: "{'dev1@127.0.0.1',{1391,544501,519016}}"

riak-repl Status Output

Details about the riak-repl status command can be found under Statistics.

Tuning

These tuning values may also be set via the node’s advanced.config file. See the Configuration Guide for more information.

fullsync max_fssource_node

This limits the number of fullsync workers that will be running on each individual node in a source cluster. This is a hard limit for all fullsyncs that are enabled. Additional fullsync configurations will not increase the number of fullsync workers allowed to run on any node. This only affects nodes on the source cluster on which this parameter is defined via the configuration file or command line.

  • Syntax: riak-repl fullsync max_fssource_node <value>
  • Default: 1
  • Example: riak-repl fullsync max_fssource_node 2

fullsync max_fssource_cluster

This is the hard limit of fullsync workers that will be running on the source side of a cluster across all nodes on that cluster for a fullsync to a sink cluster. This means if one has configured fullsync for two different clusters, both with a max_fssource_cluster of 5, 10 fullsync workers can be in progress. Only affects nodes on the source cluster on which this parameter is defined via the configuration file or the command line.

  • Syntax: riak-repl fullsync max_fssource_cluster <value>
  • Default: 5
  • Example: riak-repl fullsync max_fssource_cluster 5

fullsync max_fssink_node

This limits the number of fullsync workers allowed to run on each individual node in a sink cluster. This is a hard limit for each fullsync source node interacting with a sink node. Thus, multiple simultaneous source connections to a sink node will have to share the sink node’s number of maximum connections. Only affects nodes on the sink cluster on which this parameter is defined via the configuration file or command line.

  • Syntax: riak-repl fullsync max_fssink_node <value>
  • Default: 1
  • Example: riak-repl fullsync max_fssink_node 5

Mixing Version 2 Replication with Version 3 Replication

Riak Version 2 Replication and Version 3 Replication can be safely used at the same time. If you choose to move to Version 3 Replication completely, we recommend disabling Version 2 realtime replication bucket hooks with the riak-repl modes command.

riak-repl modes

modelist is one or both of mode_repl12 (Version 2) or mode_repl13 (Version 3) separated by spaces (without commas).

  • Syntax: riak-repl modes <modelist>

  • Example:

    riak-repl modes mode_repl12 mode_repl13

    Possible output:

    Current replication modes: [mode_repl12,mode_repl13]

To check the current replication modes:

  • Syntax: riak-repl modes

  • Example:

    riak-repl modes

    Possible output:

    Current replication modes: [mode_repl12,mode_repl13]

Configurations and Metadata in Replication

Fullsync and Realtime replication replicates data from source clusters to sink clusters, but some configurations and metadata (such as bucket properties) will not be replicated.

Non-replication of certain configurations and metadata supports heterogenous cluster configurations in Replication, but there operational things you can do when you want homogeneous cluster configurations.

Buckets and Bucket Types in Replication

Buckets and Bucket Type properties on the source cluster will not be replicated from source clusters to sink clusters.

If you want the properties for Buckets or Bucket Types present on the source cluster to be propagated to sink clusters you should update this data for each cluster at the same time you would change the source cluster.