Config Parameters - ignacio-alorre/Spark GitHub Wiki

When you have an application which may change its configuration parameter, for example different cluster url for DEV or UAT environment, it is a good idea to store this parameters in an external file which can be updated without changing the code. There are several approaches to achieve this:

Option 1: Using yalm files

First thing it is required to create a case class with the attributes which are expected in the configuration file. For example:

import scala.beans.BeanProperty

class ConfigParams {
  @BeanProperty var inputPath: String = null
  @BeanProperty var outputPath: String = null
  @BeanProperty var fileType: String = null
  @BeanProperty var clusterURL: String = null
} 

Then you can read the yalm file and generate a ConfigParams Object with this information:

import org.yaml.snakeyaml.Yaml
import org.yaml.snakeyaml.constructor.Constructor

val yaml = new Yaml(new Constructor(classOf[ConfigParams]))
val ios = new FileInputStream(new File(args(0)))
val configParams = yaml.load(ios).asInstanceOf[ConfigParams]

Finally it is possible to access each parameter independently when required:

val type : String = configParams.fileType