spark 예제파일 submit 하기

https://console.bluemix.net/docs/services/AnalyticsforApacheSpark/spark_app_example.html

아래와 같이 폴더를 생성하고 spark에서 제공하는 example 코드를 복사해온다.

$ mkdir spark-example
$ mkdir spark-example/project
$ mkdir spark-example/src
$ mkdir spark-example/src/main
$ mkdir spark-example/src/main/scala

$ cp /usr/local/cluster/spark/spark-2.3.1-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkKMeans.scala .

spark-example 폴더 아래에 build.sbt 파일 작성

name := "sparkExam"

version := "1.0"

scalaVersion := "2.11.12"

val sparkVersion = "2.3.1"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.1"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.2"

resolvers += "MavenRepository" at "https://mvnrepository.com/"
autoScalaLibrary := false

spark-submit으로 보내준다.

spark-submit --class SimpleSkewedGroupByTest --master spark://203.2.xxx.xxx:xxxx target/scala-2.11/sparkexam_2.11-1.0.jar

spark-submit --class org.apache.spark.examples.SparkKMeans --master spark://203.xxx.xx.xx:xxxx target/scala-2.11/sparkexam_2.11-1.0.jar

*패키지명도 같이 써줘야 한다.