Posts

In this short article, I would like to cover the entire development cycle of Spark jobs. I will explain following topics: setup of sbt multi-module project how to write modular code how to write testable code how to write unit tests how to write integration tests Source code is located here: https://github.com/JurajBurian/spark-example . The sbt project. In this sample project, we have defined a single Spark job as a module alongside a common module to demonstrate modular development. This approach is useful when developing several Spark jobs, that share models or functionality. ...

I like physics. The reason behind it is that physics has a beautiful and consistent type system. Let’s start with very simple scalar model, that is used in Newton mechanics. What we need here are several types like Time, Acceleration, Velocity, Displacement for kinematics. If we also deal with dynamic, we need Mass, Force, Momentum and Energy. We are also introducing Constant, that is just a number and can be used solely as multiplier (or divider, but not implemented). ...

In this article we combine opaque type alias hierarchies with implicit resolutions to do calculations. Scala 3 brings several language enhancements that give programmers even better control over types . One of them is opaque keyword. For a deeper look what opaque is please visit opaques page at dotty.epfl.ch. Let’s continue with an example and define some types: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 package types { // base type aliases opaque type StringBased = String opaque type IntBased = Int // instantiable type aliases opaque type TString1 <: StringBased = String opaque type TString2 <: StringBased = String opaque type TInt <: IntBased = Int // constructor functions def TString1(p: String): TString1 = p def TString2(p: String): TString2 = p def TInt(p: Int): TInt = p } Let’s explain these things a bit. We have declared several new type aliases in types scope (in our case scope is package types). Inside of the types scope all of these aliases works as ordinary type aliases, so we can easily create “constructor” functions. Also we declared “base aliases” named StringBased and IntBased. So expression: TString1 <: StringBased we understand that TString1 is subtype of StringBased. Let’s imagine we need do a calculation with our new types, let us call it Transformer . ...