How to Connect to Cloudant Database from SparkR kernel
Below i will show how to do it from Bluemix but it will apply Jupyter Notebook running on any environment.
Connecting to Cloudant from IBM Bluemix - Juypter Notebooks on Spark
Below i will show how to do it from Bluemix but it will apply Jupyter Notebook running on any environment.
Connecting to Cloudant from IBM Bluemix - Juypter Notebooks on Spark
- Create an account in bluemix(ibm offers 30 days free trial) - https://console.ng.bluemix.net/registration/
- Create a spark service (https://www.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index.html)
- Now create notebook with sparkR as language.
- spark context needs to know which driver to use to connect to Cloudant database. In bluemix spark service enivornment the driver is loaded by default.
- https://github.com/cloudant-labs/spark-cloudant
- Also if you are in different environment, you use binary
- https://github.com/cloudant-labs/spark-cloudant/releases/download/v1.6.3/cloudant-spark-v1.6.3-125.jar
- For ex. use %Addjar -f https://github.com/cloudant-labs/spark-cloudant/releases/download/v1.6.3/cloudant-spark-v1.6.3-125.jar to add it to your spark.
- Once you have spark-cloudant connector in your spark.
- You are going to need to have 3 configuration parameters set for you spark context
cloudant.host","ACCOUNT.cloudant.com"
"cloudant.username", "USERNAME" "cloudant.password","PASSWORD"
So in sparkR, you would need to use one of the sparkEnv variableto pass your environment variables to all the executors.sc <- sparkR.init(sparkEnv = list("cloudant.host"="c8dca934-d2a4-4dcc-9123-2189ce9f5812-bluemix.cloudant.com","cloudant.username"="c8dca934-d2a4-4dcc-9123-2189ce9f5812-bluemix","cloudant.password"="XXXXXXXXXXXXXXXXXXXX"))Once you execute above. Your sparkcontext is ready to use cloudant-connector.All you need to do is specify that you are reading using com.cloudant.sparkpeople <- read.df(sqlContext, database, header='true', source = "com.cloudant.spark",inferSchema='true')I have complete Notebook published on this github repo. Feel Free to use it.
