Donsez: Created page with " http://spark.apache.org/docs/latest/sql-programming-guide.html val sc: SparkContext // An existing SparkContext. val sqlContext = new org.apache.spark...."

2016-05-25T19:25:10Z

Created page with " http://spark.apache.org/docs/latest/sql-programming-guide.html <source lang="scala"> val sc: SparkContext // An existing SparkContext. val sqlContext = new org.apache.spark...."

New page

http://spark.apache.org/docs/latest/sql-programming-guide.html

<source lang="scala">
val sc: SparkContext // An existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)

// Create the DataFrame
val df = sqlContext.read.json("examples/src/main/resources/people.json")

// Show the content of the DataFrame
df.show()
// age name
// null Michael
// 30 Andy
// 19 Justin

// Print the schema in a tree format
df.printSchema()
// root
// |-- age: long (nullable = true)
// |-- name: string (nullable = true)

// Select only the "name" column
df.select("name").show()
// name
// Michael
// Andy
// Justin

// Select everybody, but increment the age by 1
df.select(df("name"), df("age") + 1).show()
// name (age + 1)
// Michael null
// Andy 31
// Justin 20

// Select people older than 21
df.filter(df("age") > 21).show()
// age name
// 30 Andy

// Count people by age
df.groupBy("age").count().show()
// age count
// null 1
// 19 1
// 30 1

</source>

Spark/DataFrame - Revision history

Donsez: Created page with " http://spark.apache.org/docs/latest/sql-programming-guide.html val sc: SparkContext // An existing SparkContext. val sqlContext = new org.apache.spark...."