You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/sparkr.md
+6-4Lines changed: 6 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,9 +27,9 @@ All of the examples on this page use sample data included in R or the Spark dist
27
27
<divdata-lang="r"markdown="1">
28
28
The entry point into SparkR is the `SparkContext` which connects your R program to a Spark cluster.
29
29
You can create a `SparkContext` using `sparkR.init` and pass in options such as the application name
30
-
etc. Further, to work with DataFrames we will need a `SQLContext`, which can be created from the
31
-
SparkContext. If you are working from the SparkR shell, the`SQLContext` and `SparkContext` should
32
-
already be created for you.
30
+
, any spark packages depended on, etc. Further, to work with DataFrames we will need a `SQLContext`,
31
+
which can be created from the SparkContext. If you are working from the SparkR shell, the
32
+
`SQLContext` and `SparkContext` should already be created for you.
33
33
34
34
{% highlight r %}
35
35
sc <- sparkR.init()
@@ -62,7 +62,9 @@ head(df)
62
62
63
63
SparkR supports operating on a variety of data sources through the `DataFrame` interface. This section describes the general methods for loading and saving data using Data Sources. You can check the Spark SQL programming guide for more [specific options](sql-programming-guide.html#manually-specifying-options) that are available for the built-in data sources.
64
64
65
-
The general method for creating DataFrames from data sources is `read.df`. This method takes in the `SQLContext`, the path for the file to load and the type of data source. SparkR supports reading JSON and Parquet files natively and through [Spark Packages](http://spark-packages.org/) you can find data source connectors for popular file formats like [CSV](http://spark-packages.org/package/databricks/spark-csv) and [Avro](http://spark-packages.org/package/databricks/spark-avro).
65
+
The general method for creating DataFrames from data sources is `read.df`. This method takes in the `SQLContext`, the path for the file to load and the type of data source. SparkR supports reading JSON and Parquet files natively and through [Spark Packages](http://spark-packages.org/) you can find data source connectors for popular file formats like [CSV](http://spark-packages.org/package/databricks/spark-csv) and [Avro](http://spark-packages.org/package/databricks/spark-avro). These packages can either be added by
66
+
specifying `--packages` with `sparm-submit` or `sparkR` commands, or if creating context through `init`
67
+
you can specify the packages with the `packages` argument.
66
68
67
69
We can see how to use data sources using an example JSON input file. Note that the file that is used here is _not_ a typical JSON file. Each line in the file must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.
0 commit comments