Developing Spark jobs in Scala
In this short article, I would like to cover the entire development cycle of Spark jobs. I will explain following topics: setup of sbt multi-module project how to write modular code how to write testable code how to write unit tests how to write integration tests Source code is located here: https://github.com/JurajBurian/spark-example . The sbt project. In this sample project, we have defined a single Spark job as a module alongside a common module to demonstrate modular development. This approach is useful when developing several Spark jobs, that share models or functionality. ...