Cloudera Hadoop - youdar/How-to GitHub Wiki

Hue Workflow

When trying to use the Hue Workflow editor SSH process,
there are issues if adding any parameters to the command executed via the SSH
To Overcome this

Create a script on the local file system of the machine you want to SSH to, such as:

$ nano ls_l_script
#!/bin/bash   
ls -l > /home/user/tmp/hue_workflow/ssh_log_file

Then change permissions of the file, to make it executable
$ chmod u+x ls_l_script
Then in Hue -> Workflow -> Editors -> Workflow
$ userd@host_address /home/user/tmp/hue_workflow/ls_l_script

Issues and solutions:

Error when trying to use pyspark on hdfs files

Error produced in the PySpark consul by:

>>> fn_txt = 'hdfs://nameservice1/user/some_text_file.txt'   
>>> l = sc.textFile(fn_txt)  
>>> l.count()

Problem:

the Hadoop cluster node we worked on is not a Spark gateway.
The value of this command "hdfs getconf -confKey fs.defaultFS" when ran from a command line and checking spark.eventLog.dir "hdfs://namenode:8021/directory" under /etc/spark/conf/spark-default.conf were not same.

Solution: Administrator need to make the Hadoop node we are working on both Yarm and Spark gateway

Example for adding a gateway for spark to provide to the admin team: You can go to Spark service from Cloudera Manager Then: Spark-> Instances-> Click on Add Role Instances >Select the node you want to be a gateway> Continue->Done. Once a gateway is Installed for spark service. Go to Spark->Actions->Deploy Client Config.

Using PySpark in IDE Even if you set up SPARK_HOME environmental variable on the cluster node you're working on, when you try to run PySpark code it might try to run on a cluster node that does not have this variable. To avoid this issue, add the following code to your function:

os.environ['SPARK_HOME'] = '/opt/cloudera/parcels/CDH/lib/spark'

When using modules in your code, to make sure those module are available when you use PySpark follow the example below:

try:
    # regular import statment
    from source.general_functions.date_time.back_datetime import get_back_date_time
except ImportError:
    # the local file system location of the function, on the cluster node.
    sys.path.append('/home/youval.dar/youval/dev-hadoop-pyspark/source/general_functions/date_time')
    from back_datetime import get_back_date_time

Impyla

http://blog.cloudera.com/blog/2014/04/a-new-python-client-for-impala/

Example:

from impala.dbapi import connect
impyla_conn = connect(host='<host>.ringcentral.com', auth_mechanism='GSSAPI')
impyla_curr = impyla_conn.cursor()
impyla_curr.execute('show databases')
for x in impyla_curr.fetchall():
print x
impyla_curr.close()

Impala SQL

When creating a table using CREATE TABLE AS, if you need to create a partitioned parquet table
use the following command:

CREATE TABLE db_name.table_name PARTITIONED BY (field_name) STORED AS PARQUET AS
SELECT
...

General notes

Hive JDBC driver : org.apache.hive.jdbc.HiveDriver