It's almost another month. In the early August, I attended Strata + Hadoop World Beijing, the first ever Strata in China, and shared about Apache Gearpump next-gen streaming engine. Following that, Apache Gearpump (incubating) made its first Apache release. Meanwhile, Apache Spark, Apache Storm and Akka also released new versions.

Release

• Databricks has introduced Apache Spark 2.0. Most eye-catching new features are

  • expanded SQL Support (including all 99 TPC-DS query)
  • unified DataFrames and Datasets API in Scala/Java
  • A single entry point SparkSession subsuming both SQLContext and HiveContext
  • whole-stage code generation that makes Spark 10X faster
  • Structured Streaming (Learn more about the feature at Structured Streaming in Apache Spark and Continuous Applications: Evolving Streaming in Apache Spark 2.0)

• Apache Storm 1.0.2 has released fixing bugs, improving performance, stability and fault tolerance. One thing to note is Backpressure can cause spout to stop emitting and stall topology and is disabled by default.

• Akka 2.4.9 is available with improved performance on Akka HTTP and Akka Streams. Akka HTTP performance is on-par or better than Spray.

Streaming

• Yelp has a series covering its real-time streaming data infrastructure.
  • Billions of Messages a Day - Yelp's Real-time Data Pipeline
  • Streaming MySQL tables in real-time to Kafka
  • More Than Just a Schema Store
• LINE is applying Kafka Streams for internal message delivery pipeline. If you are looking for a thin streaming layer (without execution framework, resource scheduler, etc) that works best with Kafka, Kafka Streams could be the right choice.
• Writing SQL on Streaming Data with Amazon Kinesis Analytics – Part 1 provides an overview of streaming analytics on AWS.

Scala

• Martin Odersky outlined significant plans for Scala after a quiet 2015

  • The Scala Center
  • Scala 2.12
  • Plans to rethink the Scala libraries
  • Development of new target platforms
  • DOT and Dotty

• A related news is Samsung SAMI team enabled live IoT data pipeling with Akka and reactive architecture

• Speaking of reactive, Akka Streams is an implementation of Reactive Streams where back pressure is the key. Simply explained: Akka Streams Backpressure walks through an example to illustrate what back pressure is.

• If you are building applications on Akka or with Typesafe Config, Effective Typesafe Config is a handy guide.

Java

• Oracle Java Architect, Brian Goetz looks at lessons of Java evolution and where Java platform is headed.

• For Java applications to work, dependencies must be present on the classpath. We usually build fat jars with all dependencies using maven-assembly-plugin or maven-shade-plugin. Deploying a fat every time could be slow and inefficient while a better way is to upload dependencies to a distributed file system for once individually.

That's it for last month. Wish I could return to my weekly timeline.