Spark-Riak Connector Add-on (Riak TS) Working With Spark Dataframes

Spark Dataframes With TS Table

To enable DataFrames functionality the first steps are:

val sc = new SparkContext()
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
ts_table_name = "test_table"

sc = pyspark.SparkContext(conf=conf)
sqlContext = pyspark.SQLContext(sc)
ts_table_name = "test_table"

To read data from the existing TS table test-table standard SQLContext means can be used by providing a org.apache.spark.sql.riak data format and using a Riak TS range query:

val df = sqlContext.read
  .option("spark.riak.connection.hosts","riak_host_ip:10017")
    .format("org.apache.spark.sql.riak")
    .load(ts_table_name)
  .select(“time”, “col1”, “col2”)
    .filter(s"time >= CAST($from AS TIMESTAMP) AND time <= CAST($to AS TIMESTAMP) AND  col1= $value1")

df = sqlContext.read \
  .option("spark.riak.connection.hosts","riak_host_ip:10017") \
    .format("org.apache.spark.sql.riak") \
    .load(ts_table_name) \
  .select(“time”, “col1”, “col2”) \
    .filter(s"time >= CAST($from AS TIMESTAMP) AND time <= CAST($to AS TIMESTAMP) AND  col1= $value1")

Schema may be provided using the .schema() method. If not provided, it will be inferred. Any of the connector options can be provided in .option() or .options().

Alternatively, org.apache.spark.sql.riak.RiakSQLContext can be created and then queried with range query using sql() method:

val riakSqlContext = new RiakSQLContext(sc, ts_table_name)
val alternativeDf = riakSqlContext.sql(s"SELECT time, col1 from $ts_table_name WHERE time >= CAST($from AS TIMESTAMP) AND time <= CAST($to AS TIMESTAMP) AND  col1= $value1")

A DataFrame, inputDF, that has the same schema as an existing TS table (column order and types) can be saved to Riak TS as follows:

inputDF.write
   .option("spark.riak.connection.hosts","riak_host_ip:10017")
   .format("org.apache.spark.sql.riak")
   .mode(SaveMode.Append)
   .save(ts_table_name)

inputDF.write \
   .option("spark.riak.connection.hosts","riak_host_ip:10017") \
   .format("org.apache.spark.sql.riak") \
   .mode(SaveMode.Append) \
   .save(ts_table_name)

SaveMode.Append is the only mode available. Any of the connector options can be provided in .option() or .options().

In

Spark Dataframes With TS Table