Home - noonecare/spark GitHub Wiki

1 scala

Design Motto: Do More With Less Code.

1. 1 scala 和 Java

1.1.1 scala 引用 java 中包

1.1.2 jvm 虚拟机

1.2 Function

1.2.1 byname parameter

1.2.2 curried parameter ?

1.2.3 loop in a functional way

1.2.4 partial function

1.2.5 tail recursive

1.2.7 implicit parameter

1.2.8 implicit conversion

1.2.9 special function call forms

1.2.10 high order functions

1.2.11 function literal

1.3 Control Abstraction

1.3.1 if

1.3.2 for loop

1.3.3 try catch

1.3.4 match

1.3.5 writing new control structures

1.4 Class and Object

1.4.1 access control(private, protected, companion object)

1.4.2 Singleton

1.4.3 Factory method

1.4.5 trait

1.4.6 extending classes (final class 是不能被 extends 的 class, sealed class 是只能在当前文件中被 extends 的class)

1.4.7 overriding methods and fields

1.4.8 invoking superclass method

1.4.9 final members

1.4.10 multiple inherit

1.4.11 imports

1.4.12 Implicit imports

1.4.13 override operator

1.4.14 fields

1.4.15 overload

1.5 type parametric

1.6 Concurrency

1.6.1 actor

1.6.1.1 Actor class and Actor trait

1.6.1.2 send and recieve message

1.6.2 akka

1.7 Basic Types and variable

1.7.1 Collections API

1.7.2 Hierarchy

1.7.3 handy type and method

1.7.3.1 App trait

1.7.3.2 foreach method

1.8 Case Pattern

1.8.1 value pattern

1.8.2 type pattern

1.8.3 guide pattern

1.8.4 List pattern

1.8.5 Tuple pattern

1.8.6 wildcard _

1.8.7 variable bind @

1.8.8 Extractor

1.9 Implicit Conversions and Parameters

1.9.1 Rules for implicit

1.9.1.1 Marking Rule: Only definitions marked implicit are available.

1.9.1.2 Scope Rule: An inserted implicit conversion must be in scope as a single identifier, or be associated with the source or target type of the conversion.

1.9.1.3 One-at-a-time Rule: Only one implicit is tried.

1.9.1.4 Explicits-First Rule: Whenever code type checks as it is written, no implicits are attempted.

1.9.2 View Bounds

1.9.2.1 T <% Ordered[T] means T which can be treated as and Ordered[T]

1.9.3 When multiple conversions apply

1.9.3.1 multiple implicit conversions can apply will cause a compile error

1.9.3.2 multiple implicit parameter can apply will use the first.

1.10 Annotations

2 spark

2.1 RDD

2.2 DataFrame

2.3 Join Key/value Pairs

2.4 Broadcast and Increment

2.5 Spark Streaming

2.6 Develop and Deploy

2.7 MLlib

2.8 implementation

2.8.1 netty: 任何分布式的系统都会用到 RPC 或者 RESTFUl API， Spark 用 netty 实现 RPC。最近我学习了一下 netty。

Use Cases