Lei Gong

Apache Spark is arguably the most fast-growing open source project in the big data world for the past few years. It provides, as quoted from its website, “a fast and general engine for large-scale data processing”. With the recent release of Spark 2.0, the project becomes even better for building big data applications with consolidated APIs and improved performance. To help people get started or transition from 1.x, I am starting this series of posts to talk about some of Spark 2.0’s important c ...

The concept of abstract class should be familiar to anyone with some background in Java or object-oriented programming (OOP) in general. It essentially 1) defines methods that enables inheriting subclass to use, 2) defines absract methods that requires implementation in subclass and 3) provides a common interface that allows interchanges among subclasses. Scala borrows many things from Java, including abstract class, and provides some of its own, often times more powerful, variations, e.g. trait ...

Pattern matching is one of the most used features in Scala. Imagine yourself in a situation where you need to write a series of if-else-if statements to analyze the arguments of a certain function. It sometimes just a bit annoying to have those extra space/parentheses and you may hope to have something more elegent. Well, pattern matching is here for the rescue (some smarty pants may think of hash map, more on this later). It is like those switch statement in C/C++/Java, but better and more powe ...

Almost every programming languages have certain mechanism to deal with the null values, i.e. the absence of some values, in order to work safely in the messy real world. Many languages choose to have a separate object for these absent values, e.g. the null in Java. And defensive programming teaches us to handle this scenario separately, given whether it is a value that is returned or it is the null object. So, you might find the this description familiar: the null checkers are placed all over th ...

One of the functional programming tricks in Scala that I recently learned and enjoyed is folding, namely the fold, foldLeft and foldRight functions. As implied by their names, the three methods share many concepts in common, but there are also subtle differences in their implementations. As I am a firm believer of learning by examples, I put together some code snippets (many thanks to this post) that hopefully could help you better understand the nitty-gritty of fold, foldLeft and foldRight. Co ...

I have been procrastinating on building a personal blog for quite a while. It is one of those things that you know it is important and would do great good to you, but you just can’t seem to find time to finally work on it (and, yes, almost always, it is just your another excuse). Things have changed after I am half way through this awesome book (Soft Skills: The software developer’s life manual) by John Sonmez. The book again emphasizes the importance of a well-maintained blog to your personal b ...