Spark 文档

https://spark.apache.org/docs/2.2.0/ml-clustering.html#k-means http://spark.apachecn.org/docs/cn/2.2.0/index.html http://cwiki.apachecn.org/pages/viewpage.action?pageId=9406765

环境配置

关掉防火墙

sudo systemctl disable firewalld

Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
systemctl stop firewalld

==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: hadoop
Password:
polkit-agent-helper-1: pam_authenticate failed: Authentication failure
==== AUTHENTICATION FAILED ===
Failed to stop firewalld.service: Access denied
See system logs and 'systemctl status firewalld.service' for details.
-bash-4.2$ systemctl stop firewalld
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: hadoop
Password:
==== AUTHENTICATION COMPLETE ===5

sudo systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset:
enabled)
Active: inactive (dead) since Tue 2018-10-09 11:00:59 PDT; 15s ago
Docs: man:firewalld(1)
Main PID: 886 (code=exited, status=0/SUCCESS)

重启Hadoop

top-all.sh
jps
kill -9 <hadoop process>
jps
start-all.sh

进入 http::50070

注：不要进入50075，先解决50070 name node，可从50070进入data node

第一步：安装软件开发工具，及 git.

注册root用户

-bash-4.2$ sudo -i
[sudo] password for hadoop:
[root@master ~]#

[root@master ~]# yum clean all
Loaded plugins: fastestmirror, langpacks
Cleaning repos: base epel extras google-chrome updates
Cleaning up everything
也许你需要: rm -rf /var/cache/yum, to also free up space taken by orphaned
data from disabled or removed repos
Cleaning up list of fastest mirrors8
[root@master ~]# yum groupinstall "development tools"
...

Installed:
autoconf.noarch 0:2.69-11.el7 automake.noarch 0:1.13.4-3.el7 bison.x86_64 0:3.0.4-1.el7 byacc.x86_64
0:1.9.20130304-3.el7
cscope.x86_64 0:15.8-10.el7 ctags.x86_64 0:5.8-13.el7 diffstat.x86_64 0:1.57-4.el7 doxygen.x86_64 1:1.8.5-
3.el7
flex.x86_64 0:2.5.37-3.el7 gcc.x86_64 0:4.8.5-28.el7_5.1 gcc-c++.x86_64 0:4.8.5-28.el7_5.1 gcc-gfortran.x86_64
0:4.8.5-28.el7_5.1
git.x86_64 0:1.8.3.1-14.el7_5 indent.x86_64 0:2.2.11-13.el7 intltool.noarch 0:0.50.2-7.el7 libtool.x86_64 0:2.4.2-
22.el7_3
patchutils.x86_64 0:0.3.3-4.el7 rcs.x86_64 0:5.9.0-5.el7 redhat-rpm-config.noarch 0:9.1.0-80.el7.centos rpm-build.x86_64
0:4.11.3-32.el7
rpm-sign.x86_64 0:4.11.3-32.el7 subversion.x86_64 0:1.7.14-14.el7 swig.x86_64 0:2.0.10-5.el7 systemtap.x86_64
0:3.2-8.el7_5
Dependency Installed:
cpp.x86_64 0:4.8.5-28.el7_5.1 dwz.x86_64 0:0.11-3.el7 efivar-libs.x86_64 0:31-4.el7 gettext-commondevel.noarch 0:0.19.8.1-2.el7
gettext-devel.x86_64 0:0.19.8.1-2.el7 kernel-debug-devel.x86_64 0:3.10.0-862.11.6.el7 libgfortran.x86_64 0:4.8.5-28.el7_5.1
libmpc.x86_64 0:1.0.1-3.el7
libquadmath.x86_64 0:4.8.5-28.el7_5.1 libquadmath-devel.x86_64 0:4.8.5-28.el7_5.1 libstdc++-devel.x86_64 0:4.8.5-28.el7_5.1
mokutil.x86_64 0:12-1.el7.centos
perl-Error.noarch 1:0.17020-2.el7 perl-Git.noarch 0:1.8.3.1-14.el7_5 perl-TermReadKey.x86_64 0:2.30-20.el7 perl-srpmmacros.noarch 0:1-8.el7
subversion-libs.x86_64 0:1.7.14-14.el7 systemtap-client.x86_64 0:3.2-8.el7_5 systemtap-devel.x86_64 0:3.2-8.el7_5
Dependency Updated:
libgcc.x86_64 0:4.8.5-28.el7_5.1 libgomp.x86_64 0:4.8.5-28.el7_5.1 libstdc++.x86_64 0:4.8.5-28.el7_5.1 rpm.x86_64 0:4.11.3-32.el7
rpm-build-libs.x86_64 0:4.11.3-32.el7
rpm-libs.x86_64 0:4.11.3-32.el7 rpm-python.x86_64 0:4.11.3-32.el7 systemtap-runtime.x86_64 0:3.2-8.el7_5
Complete!

安装docker和docker-compose

安装Docker

首先要更新package数据库： sudo yum check-update 下载 Docker的最新版本，安装： curl -fsSL https://get.docker.com/ | sh 完成，启动Docker daemon： sudo systemctl start docker 查看运行： sudo systemctl status docker

在没有 Sudo 的情况下执行docker命令 (可选) 默认情况下，运行docker 命令需要 root 权限—即，必须在命令前面加上前缀 sudo。其也可以由docker组中的用户运行，在安装docker时会自动创建。如果运行 docker命令，而不将其与 sudo 加前缀或不在docker组中，将有如下输出： docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. See 'docker run --help'. 如果想在运行docker命令时避免键入sudo，将你的用户名添加到docker组即可：

sudo usermod -aG docker $(whoami)

退出，再登录。如果要加如新用户：

sudo usermod -aG docker username

新用户也要退出，再登录。查看有没有DOCKER集装箱在运行：

-bash-4.2$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
-bash-4.2$11

现在你已经安装Docker，继续安装 Docker Compose

首要条件是安装python-pip：

sudo yum install epel-release

sudo yum install -y python-pip

接着安装Docker Compose：

sudo pip install docker-compose

升级python：

sudo yum upgrade python*

用Docker Compose运行集装箱：

公用Docker注册表Docker Hub包含一个简单的Hello World图像。现在我们已经安装了Docker Compose，接下来用这个非常简单的例子来测试它。首先为YAML文件创建一个目录：

mkdir hello-world

变成目录

cd hello-world

现在创建 YAML file。用你最常用的文本编辑器（老师用的nano）：

nano docker-compose.yml

将以下内容放入文件，保存文件，然后退出文本编辑器：

docker-compose.yml
my-test:
image: hello-world
-bash-4.2$ docker-compose up
Pulling my-test (hello-world:)...
latest: Pulling from library/hello-world
d1725b59e92d: Pull complete
Digest: sha256:0add3ace90ecb4adbf7777e9aacf18357296e799f81cabc9fde470971e499788
Status: Downloaded newer image for hello-world:latest
Creating hello-world_my-test_1 ... done
Attaching to hello-world_my-test_1
my-test_1 |
my-test_1 | Hello from Docker!
my-test_1 | This message shows that your installation appears to be working correctly.
my-test_1 |
my-test_1 | To generate this message, Docker took the following steps:
my-test_1 | 1. The Docker client contacted the Docker daemon.
my-test_1 | 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
my-test_1 | (amd64)
my-test_1 | 3. The Docker daemon created a new container from that image which runs the
my-test_1 | executable that produces the output you are currently reading.
my-test_1 | 4. The Docker daemon streamed that output to the Docker client, which sent it
my-test_1 | to your terminal.
my-test_1 |
my-test_1 | To try something more ambitious, you can run an Ubuntu container with:
my-test_1 | $ docker run -it ubuntu bash
my-test_1 |
my-test_1 | Share images, automate workflows, and more with a free Docker ID:
my-test_1 | https://hub.docker.com/
my-test_1 |
my-test_1 | For more examples and ideas, visit:
my-test_1 | https://docs.docker.com/get-started/
my-test_1 |
hello-world_my-test_1 exited with code 0
-bash-4.2$

使用Hadoop的Docker

用 docker-compose 运行docker 集装箱检查docker container运行状态：

docker ps 停止docker container： docker stop 启动，在home目录中创建文件夹： mkdir ~/docker cd ~/docker

Docker – Hadoop

在github下载Hadoop

git clone https://github.com/big-data-europe/docker-hadoop.git

以上命令用于建一个叫docker-hadoop的文件夹第一步要建docker container

cd docker-hadoop docker-compose up

由于docker图像要从 internet 下载，所以会花些时间，但是启动是很快的16 第二步：运行docker-compose， (请注意，你需要在有 docker-compose.yml file目录中运行 )，可以按以下命令查看docker comtainer：然后运行docker-compose

[gjen@localhost docker-hadoop]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
35d0c2cbaa60 bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /bin/…" 3 days ago
Up About a minute 0.0.0.0:10000->10000/tcp, 10002/tcp docker-hive_hive-server_1
f6968b39d719 bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3
days ago Up About a minute (healthy) 0.0.0.0:50070->50070/tcp dockerhive_namenode_1
17878fa8d061 shawnzhu/prestodb:0.181 "./bin/launcher run" 3 days ago Up
About a minute 0.0.0.0:8080->8080/tcp docker-hive_presto-coordinator_1
63bf46860496 bde2020/hive-metastore-postgresql:2.3.0 "/docker-entrypoint.…" 3 days ago
Up About a minute 5432/tcp docker-hive_hive-metastore-postgresql_1
97c73779ee9d bde2020/hadoop-datanode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3 days
ago Up About a minute (healthy) 0.0.0.0:50075->50075/tcp docker-hive_datanode_1
c9ba85362e3a bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /opt/…" 3 days ago
Up About a minute 10000/tcp, 0.0.0.0:9083->9083/tcp, 10002/tcp docker-hive_hive-metastore_1
[gjen@localhost docker-hadoop]$

浏览Hadoop nodes

Namenode: http://<dockerhadoop_IP_address>:50070/dfshealth.html#tab-overview
History server: http://<dockerhadoop_IP_address>:8188/applicationhistory
Datanode: http://<dockerhadoop_IP_address>:50075/
Nodemanager: http://<dockerhadoop_IP_address>:8042/node
Resource manager: http://<dockerhadoop_IP_address>:8088/

举个例子

(10.0.0.38 是运行docker的服务器）： http://10.0.0.38:50070/dfshealth.html#tab-overview

Data node: http://10.0.0.38:50075/

如何进入 docker container?

列出docker container：

docker ps

[gjen@localhost docker-hadoop]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
35d0c2cbaa60 bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /bin/…" 3 days ago Up 36 minutes 0.0.0.0:10000-
>10000/tcp, 10002/tcp docker-hive_hive-server_1
f6968b39d719 bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3 days ago Up 36 minutes (healthy)
0.0.0.0:50070->50070/tcp docker-hive_namenode_1
17878fa8d061 shawnzhu/prestodb:0.181 "./bin/launcher run" 3 days ago Up 36 minutes 0.0.0.0:8080-
>8080/tcp docker-hive_presto-coordinator_1
63bf46860496 bde2020/hive-metastore-postgresql:2.3.0 "/docker-entrypoint.…" 3 days ago Up 36 minutes 5432/tcp
docker-hive_hive-metastore-postgresql_1
97c73779ee9d bde2020/hadoop-datanode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3 days ago Up 36 minutes (healthy)
0.0.0.0:50075->50075/tcp docker-hive_datanode_1
c9ba85362e3a bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /opt/…" 3 days ago Up 36 minutes 10000/tcp,
0.0.0.0:9083->9083/tcp, 10002/tcp docker-hive_hive-metastore_1
[gjen@localhost docker-hadoop]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
35d0c2cbaa60 bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /bin/…" 3 days ago Up 42 minutes 0.0.0.0:10000-
>10000/tcp, 10002/tcp docker-hive_hive-server_1
f6968b39d719 bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3 days ago Up 42 minutes (healthy)
0.0.0.0:50070->50070/tcp docker-hive_namenode_1
17878fa8d061 shawnzhu/prestodb:0.181 "./bin/launcher run" 3 days ago Up 42 minutes 0.0.0.0:8080-
>8080/tcp docker-hive_presto-coordinator_1
63bf46860496 bde2020/hive-metastore-postgresql:2.3.0 "/docker-entrypoint.…" 3 days ago Up 42 minutes 5432/tcp
docker-hive_hive-metastore-postgresql_1
97c73779ee9d bde2020/hadoop-datanode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3 days ago Up 42 minutes (healthy)
0.0.0.0:50075->50075/tcp docker-hive_datanode_1
c9ba85362e3a bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /opt/…" 3 days ago Up 42 minutes 10000/tcp,
0.0.0.0:9083->9083/tcp, 10002/tcp docker-hive_hive-metastore_1

docker exec -it f6968b39d719 bash

创建HDFS文件夹 (data node上的Hadoop文件系统)，命名 /my_storage

root@f6968b39d719:/# hdfs dfs -mkdir /my_storage
将文件从本地文件系统复制到HDFS /my_storage
root@f6968b39d719:/hadoop/dfs# echo "hello" > /tmp/x
root@f6968b39d719:/hadoop/dfs# cat /tmp/x
hello
root@f6968b39d719:/hadoop/dfs# hdfs dfs -put /tmp/x /my_storage
root@f6968b39d719:/hadoop/dfs# hdfs dfs -ls /my_storage
Found 1 items
-rw-r--r-- 3 root supergroup 6 2018-09-05 22:35 /my_storage/x

停止docker containers

Example: [gjen@localhost ~]$ for i in $(docker ps | grep -v "CONTAINER ID" | awk '{print $1}'); do echo $i; docker stop $i; done 26e137126e86 26e137126e86 33647766d692 33647766d692 347967329c71 347967329c71 35d0c2cbaa60 35d0c2cbaa60 f6968b39d719 f6968b39d719 17878fa8d061 17878fa8d061 63bf46860496 63bf46860496 97c73779ee9d 97c73779ee9d c9ba85362e3a c9ba85362e3a 337d1ceadea9 337d1ceadea9 11b4ed699a01 11b4ed699a01 0e6173a49bbe 0e6173a49bbe ad7df1951ed7 ad7df1951ed7 efef75fd1b11 efef75fd1b11 8296299f8d84 8296299f8d84 147f4e01a01e 147f4e01a01e [gjen@localhost ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Docker – HIVE container（还包括 Hadoop，意味着你不需要启动另一个 "hadoop"，除非其他 hadoop 集群上有数据）在github下载docker-hive cd ~/docker git clone https://github.com/big-data-europe/docker-hive.git

在 ~/docker/docker-hive中创建目录

cd docker-hive 启动 docker containers （因为需要先从网上下载，运行需要点时间）

[gjen@localhost docker-hive]$ docker-compose up -d
Starting docker-hive_presto-coordinator_1 ... done
Starting docker-hive_hive-server_1 ... done
Starting docker-hive_hive-metastore-postgresql_1 ... done
Starting docker-hive_hive-metastore_1 ... done
Starting docker-hive_namenode_1 ... done
Starting docker-hive_datanode_1 ... Done

[gjen@localhost docker-hive]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
35d0c2cbaa60 bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /bin/…" 3 days ago Up 4
seconds 0.0.0.0:10000->10000/tcp, 10002/tcp docker-hive_hive-server_1
f6968b39d719 bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3 days ago
Up 3 seconds (health: starting) 0.0.0.0:50070->50070/tcp docker-hive_namenode_1
17878fa8d061 shawnzhu/prestodb:0.181 "./bin/launcher run" 3 days ago Up 4 seconds
0.0.0.0:8080->8080/tcp docker-hive_presto-coordinator_1
63bf46860496 bde2020/hive-metastore-postgresql:2.3.0 "/docker-entrypoint.…" 3 days ago Up 4
seconds 5432/tcp docker-hive_hive-metastore-postgresql_1
97c73779ee9d bde2020/hadoop-datanode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 3 days ago
Up 3 seconds (health: starting) 0.0.0.0:50075->50075/tcp docker-hive_datanode_1
c9ba85362e3a bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /opt/…" 3 days ago Up 3
seconds 10000/tcp, 0.0.0.0:9083->9083/tcp, 10002/tcp docker-hive_hive-metastore_1

将数据加载到Hive

进入docker container：

[gjen@localhost docker-hive]$ docker-compose exec hive-server bash
root@35d0c2cbaa60:/opt# hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-
2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4jlog4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in file:/opt/hive/conf/hive-log4j2.properties
Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions.
Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
> CREATE TABLE pokes (foo INT, bar STRING);
> LOAD DATA LOCAL INPATH '/opt/hive/examples/files/kv1.txt' OVERWRITE INTO
TABLE pokes;
quit;

Docker – Spark

要让Hive和Hadoop运行，在 docker-hive直接启动他们就ok 使用 docker-compose创建简单的独立集群： docker-compose up

[gjen@localhost docker-hive]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
33647766d692 gettyimages/spark "bin/spark-class org…" 4 days ago Up 3
hours 7012-7015/tcp, 8881/tcp, 0.0.0.0:8081->8081/tcp
docker-spark_worker_1
347967329c71 gettyimages/spark "bin/spark-class org…" 4 days ago Up 3 hours
0.0.0.0:4040->4040/tcp, 0.0.0.0:6066->6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:8181->8181/tcp, 7001-
7005/tcp docker-spark_master_1
35d0c2cbaa60 bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /bin/…" 5 days ago Up
5 seconds 0.0.0.0:10000->10000/tcp, 10002/tcp
docker-hive_hive-server_1
f6968b39d719 bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 5 days ago
Up 5 seconds (health: starting) 0.0.0.0:50070->50070/tcp
docker-hive_namenode_1
17878fa8d061 shawnzhu/prestodb:0.181 "./bin/launcher run" 5 days ago Up 5
seconds 0.0.0.0:8080->8080/tcp dockerhive_presto-coordinator_1
63bf46860496 bde2020/hive-metastore-postgresql:2.3.0 "/docker-entrypoint.…" 5 days ago
Up 5 seconds 5432/tcp dockerhive_hive-metastore-postgresql_1
97c73779ee9d bde2020/hadoop-datanode:2.0.0-hadoop2.7.4-java8 "/entrypoint.sh /run…" 5 days ago
Up 5 seconds (health: starting) 0.0.0.0:50075->50075/tcp
docker-hive_datanode_1
c9ba85362e3a bde2020/hive:2.3.2-postgresql-metastore "entrypoint.sh /opt/…" 5 days ago Up
5 seconds 10000/tcp, 0.0.0.0:9083->9083/tcp, 10002/tcp
docker-hive_hive-metastore_1
[gjen@localhost docker-hive]$

案例研究1房地产：

由于行业，房地产活动概述为与收购，出售，所有者经营或租赁物业相关的任何经济交易。此外，还包括创收住宅物业，如公寓，建筑物和单间房租。

房地产服务并未包含在该行业内。此外，房地产服务样本包括经纪人，物业管理，评估师，投资物业分析师和不同的顾问。

所有使用大数据的分析师都在使用Hive或其他工具来查询数据集并轻松获得结果。虽然存在其他查询语言，但与传统方法相比， Hive为我们提供了各种新功能。

因此，根据房地产领域加速消费者的需求，过滤了一系列信息并将其移交给数据分析师团队。

给定要执行分析的数据集，请回答以下来自买家的最受好评的问题。30 使用scp将文件从PC或Mac复制到Virtualbox上运行的CentOS VM 若你是Mac，即已有scp命令若你是Windows，可以从中下载winscp

https://winscp.net/eng/index.php

复制real_estate.csv（已上传至qq群文件），使用scp复制到CentOS VM上的/ tmp， Gerge老师将在课堂上演示。然后在CentOS上，将real_estate.csv复制到运行Hive服务器的docker容器中的/ tmp 复制文件到docker或docker之外的语法是

docker cp --help

要将real_estate.csv复制到docker container id 45fab2fdc91e到其文件夹/ tmp，请执行以下操作：

docker cp /opt/hadoop/docker/use_cases/realestate/real_estate.csv 45fab2fdc91e:/ tmp

在Hive中创建一个表：

hive> create table realEstate (Street string, City string, Zip int, State string, Beds int,
Baths int, sq_feet int, flat_type string, Price int) row format delimited Fields terminated
BY ',' stored AS textfile;
OK

将数据导入表realEstate：

hive> load data local inpath '/tmp/real_estate.csv' into table realEstate;
Loading data to table default.realestate
OK
Time taken: 1.316 seconds32
查询表realEstate：
hive> select * from default.realestate limit 10;
OK
Time taken: 2.038 seconds

street city NULL state NULL NULL NULL type NULL
3526 HIGH ST SACRAMENTO 95838 CA 2 1 836 Residential NULL
51 OMAHA CT SACRAMENTO 95823 CA 3 1 1167 Residential NULL
2796 BRANCH ST SACRAMENTO 95815 CA 2 1 796 Residential NULL
2805 JANETTE WAY SACRAMENTO 95815 CA 2 1 852 Residential NULL
6001 MCMAHON DR SACRAMENTO 95824 CA 2 1 797 Residential NULL
5828 PEPPERMILL CT SACRAMENTO 95841 CA 3 1 1122 Condo NULL
6048 OGDEN NASH WAY SACRAMENTO 95842 CA 3 2 1104 Residential NULL
2561 19TH AVE SACRAMENTO 95820 CA 3 1 1177 Residential NULL
11150 TRINITY RIVER DR Unit 114 RANCHO CORDOVA 95670 CA 2 2 941 Condo NULL
Time taken: 1.427 seconds, Fetched: 10 row(s)

修改如下：

hive> alter table realEstate set tblproperties ("skip.header.line.count"="1");
OK
Time taken: 0.073 second

hive> select * from default.realestate limit 10;
OK
3526 HIGH ST SACRAMENTO 95838 CA 2 1 836 Residential NULL
51 OMAHA CT SACRAMENTO 95823 CA 3 1 1167 Residential NULL
2796 BRANCH ST SACRAMENTO 95815 CA 2 1 796 Residential NULL
2805 JANETTE WAY SACRAMENTO 95815 CA 2 1 852 Residential NULL
6001 MCMAHON DR SACRAMENTO 95824 CA 2 1 797 Residential NULL
5828 PEPPERMILL CT SACRAMENTO 95841 CA 3 1 1122 Condo NULL
6048 OGDEN NASH WAY SACRAMENTO 95842 CA 3 2 1104 Residential NULL
2561 19TH AVE SACRAMENTO 95820 CA 3 1 1177 Residential NULL
11150 TRINITY RIVER DR Unit 114 RANCHO CORDOVA 95670 CA 2 2 941 Condo
NULL
7325 10TH ST RIO LINDA 95673 CA 3 2 1146 Residential NULL
Time taken: 0.143 seconds, Fetched: 10 row(s)

如要查看表中的列，执行下面的命令：

hive> describe default.realestate
> ;
OK
street string
city string
zip int
state string
beds int
baths int
sq_feet int
flat_type string
price int
Time taken: 0.043 seconds, Fetched: 9 row(s)

在输出显示列标题：

hive> set hive.cli.print.header=true
> ;
hive> select * from default.realestate limit 10;
OK
realestate.street realestate.city realestate.zip realestate.state realestate.beds realestate.baths
realestate.sq_feetrealestate.flat_type realestate.price
3526 HIGH ST SACRAMENTO 95838 CA 2 1 836 Residential NULL
51 OMAHA CT SACRAMENTO 95823 CA 3 1 1167 Residential NULL
2796 BRANCH ST SACRAMENTO 95815 CA 2 1 796 Residential NULL
2805 JANETTE WAY SACRAMENTO 95815 CA 2 1 852 Residential NULL
6001 MCMAHON DR SACRAMENTO 95824 CA 2 1 797 Residential NULL
5828 PEPPERMILL CT SACRAMENTO 95841 CA 3 1 1122 Condo NULL
6048 OGDEN NASH WAY SACRAMENTO 95842 CA 3 2 1104 Residential NULL
2561 19TH AVE SACRAMENTO 95820 CA 3 1 1177 Residential NULL
11150 TRINITY RIVER DR Unit 114 RANCHO CORDOVA 95670 CA 2 2 941 Condo
NULL
7325 10TH ST RIO LINDA 95673 CA 3 2 1146 Residential NULL
Time taken: 0.106 seconds, Fetched: 10 row(s)

docker hadoop hive spark - denlly/ai-tutorial GitHub Wiki

Spark 文档

环境配置

关掉防火墙

重启Hadoop

进入 http::50070

安装docker和docker-compose

安装Docker

用Docker Compose运行集装箱：

使用Hadoop的Docker

Docker – Hadoop

浏览Hadoop nodes

举个例子

如何进入 docker container?

将数据加载到Hive

Docker – Spark

案例研究1房地产：

在Hive中创建一个表：

⚠️ GitHub.com Fallback ⚠️

docker hadoop hive spark - denlly/ai-tutorial GitHub Wiki

Spark 文档

环境配置

关掉防火墙

重启Hadoop

进入 http::50070

安装docker和docker-compose

安装Docker

用Docker Compose运行集装箱：

使用Hadoop的Docker

Docker – Hadoop

浏览Hadoop nodes

举个例子

如何进入 docker container?

将数据加载到Hive

Docker – Spark

案例研究1房地产：

在Hive中创建一个表：

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️