UnsafeCrailSerializer - animeshtrivedi/notes GitHub Wiki

Here we document our concerns with the UnsafeCrailSerializer.

Enable 'UnsafeCrailSerializer`

The default SQL serializer is UnsafeRowSerializer. This can be changed now with :

spark.sql.serializer			UnsafeCrailSerializer 

Enable buffering vs direct mode

It has two variants buffered and buffer-less. Thse can be activated for serialization and deserialization separately by :

spark.sql.serializer.buffer	        true
spark.sql.deserializer.buffer	        true

Setting size of the buffer

Next important parameter is the buffer size - which will be the same for both serialization and de-serialization. Set using

spark.sql.serializer.buffer.size 1048576

To use crail serializer or not. For now it does not do much apart from the right file type casting.

spark.sql.serializer.buffer.size