Ambari自动部署--Hadoop
Ambari自动部署Hadoop
目标
-
能够了解Ambari应用场景
-
能够了解Ambari功能
-
能够安装并配置Ambari Server
-
能够安装并配置Ambari Agent
-
能够通过Ambari实现Hadoop集群自动部署
-
能够掌握Ambari自动部署的Hadoop集群使用测试
1)Ambari介绍
Apache Ambari项目旨在通过开发用于配置，管理和监控Apache Hadoop集群的软件来简化Hadoop管理。Ambari提供了一个由RESTful API支持的直观，易用的Hadoop管理Web UI。
Ambari使系统管理员能够：
-
提供Hadoop集群
-
Ambari提供了跨任意数量的主机安装Hadoop服务的分步向导。
-
Ambari处理群集的Hadoop服务配置。
-
-
管理Hadoop集群
Ambari提供集中管理，用于在整个集群中启动，停止和重新配置Hadoop服务。
-
监控Hadoop集群
-
Ambari提供了一个仪表板，用于监控Hadoop集群的运行状况和状态。
-
Ambari利用进行指标收集。
-
Ambari利用进行系统警报，并在需要您注意时通知您（例如，节点出现故障，剩余磁盘空间不足等）。
-
Ambari使应用程序开发人员和系统集成商能够：
-
使用轻松将Hadoop配置，管理和监控功能集成到自己的应用。
2）Ambari部署
架构
| 主机名 | IP | 角色 |
|---|---|---|
| ambariserver.a.com | 192.168.139.30 | ambari-server |
| hd1.a.com | 192.168.139.31 | NameNode;ambari-agent |
| hd2.a.com | 192.168.139.32 | NameNode;ambari-agent |
| hd3.a.com | 192.168.139.33 | ResourceManager;ambari-agent |
| hd4.a.com | 192.168.139.34 | DataNode;NodeManager;Zookeeper;Journalnode;ambari-agent |
| hd5.a.com | 192.168.139.35 | DataNode;NodeManager;Zookeeper;Journalnode;ambari-agent |
| hd6.a.com | 192.168.139.36 | DataNode;NodeManager;Zookeeper;Journalnode;ambari-agent |
目标
Ambari本身也是一个分布式架构软件，主要由两部分组成：Ambari Server和Ambari Agent。用户通过AmbariServer通知Ambari Agent安装对应的软件，Agent会定时发送各个机器每个软件模块的状态给Server,最终这些状态信息会呈现给Ambari的GUI,方便用户了解到集群中各组件状态，做出相应的维护策略。
2.1）环境准备
-
Centos7.9 7台
-
1台做Ambari Server
-
6台做Ambari Agent[即可部署hadoop集群]
-
-
静态IP地址配置及集群主机之间多机互信
7个主机都要改我只写一个
[root@hd1 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.139.30 ambariserver.a.com ambariserver
192.168.139.31 hd1.a.com hd1
192.168.139.32 hd2.a.com hd2
192.168.139.33 hd3.a.com hd3
192.168.139.34 hd4.a.com hd4
192.168.139.35 hd5.a.com hd5
192.168.139.36 hd6.a.com hd6
-
时间同步、firewalld、 selinux状态
7个主机都要改我只写一个
[root@hd1 ~]# ntpdate time.windows.com
[root@hd1 ~]# systemctl stop firewalld
[root@hd1 ~]# systemctl disable firewalld
[root@hd1 ~]# setenforce 0
[root@hd1 ~]# sed -i '7s#enforcing#disabled#g' /etc/selinux/config
-
SSH免密登录
[root@ambariserver ~]# ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ''
[root@ambariserver .ssh]# cd /root/.ssh/
[root@ambariserver .ssh]# rm -f known_hosts
[root@ambariserver .ssh]# cp id_rsa.pub authorized_keys
[root@ambariserver .ssh]# for i in hd{1..6} ;do scp -r /root/.ssh $i:/root/ ; done
-
安装jdk
[root@ambariserver ~]# cd /usr/local/src/
[root@ambariserver src]# tar xf jdk-8u221-linux-x64.tar.gz
[root@ambariserver src]# mv jdk1.8.0_221 /usr/local/jdk
[root@ambariserver src]# for i in hd{1..6} ;do scp -r /usr/local/jdk $i:/usr/local ; done
[root@ambariserver src]# vim /etc/profile
.......
export JAVA_HOME=/usr/local/jdk
export PATH=${JAVA_HOME}/bin:$PATH
[root@ambariserver src]# source /etc/profile
[root@ambariserver src]# java -version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
[root@ambariserver src]# for i in hd{1..6} ;do scp /etc/profile $i:/etc/profile ; done
`分别source剩下主机`
source /etc/profile
-
安装数据库
-
版本
-
mysq|5.6
-
mariadb 5.5
[root@ambariserver ~]# yum -y install mariadb mariadb-server
[root@ambariserver ~]# systemctl enable --now mariadb.service
[root@ambariserver ~]# mysqladmin -uroot password "123456"
[root@ambariserver ~]# mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 5.5.68-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
-
ambariserver数据库创建
-
agent把数据汇报server ,存在数据库里面
-
hadoop集群状态
-
监控数据
-
管理使用的web UI框架
[root@ambariserver ~]# mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 5.5.68-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> create database ambari character set utf8;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> grant all on ambari.* to 'ambari'@'%' identified by 'abc123';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> grant all on ambari.* to 'ambari'@'ambariserver.a.com' identified by 'abc123';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> grant all on ambari.* to 'ambari'@'ambariserver' identified by 'abc123';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]>
`验证`
[root@ambariserver ~]# mysql -h ambariserver -uambari -pabc123
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 5.5.68-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
-
安装数据库连接工具
[root@ambariserver ~]# yum -y install mysql-connector-java
-
安装httpd
[root@ambariserver ~]# yum -y install httpd
[root@ambariserver ~]# systemctl enable --now httpd.service
-
解决已获取软件资源至/var/www/html
资源可用私信我获取
[root@ambariserver html]# unzip HDP、Ambari.zip
[root@ambariserver ~]# cd /var/www/html/
[root@ambariserver html]# ll
total 12
drwxr-xr-x 3 root root 4096 Oct 10 2020 ambari
drwxr-xr-x 40 root root 4096 Oct 10 2020 HDP
drwxr-xr-x 21 root root 4096 Oct 10 2020 HDP-UTIL
-
配置本地YUM源
[root@ambariserver ~]# yum install -y createrepo
[root@ambariserver ~]# createrepo /var/www/html/HDP
[root@ambariserver ~]# createrepo /var/www/html/HDP-UTIL/
[root@ambariserver ~]# createrepo /var/www/html/ambari/
[root@ambariserver ~]# cat /etc/yum.repos.d/hdp.repo
[HDP-2.6]
name=HDP Version - HDP-2.6.4.0
baseurl=http://ambariserver/HDP/
gpgcheck=1
gpgkey=http://ambariserver/HDP/
gpgkey=http://ambariserver/HDP/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://ambariserver/HDP-UTIL/
gpgcheck=1
gpgkey=http://ambariserver/HDP-UTIL/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[root@ambariserver ~]# cat /etc/yum.repos.d/ambari.repo
[ambari-2.5.1]
name=ambari
baseurl=http://ambariserver/ambari/
gpgcheck=0
enabled=1
验证
[root@ambariserver ~]# yum repolist
-
拷贝yum源到hd1~6
[root@ambariserver ~]# for i in hd{1..6} ;do scp /etc/yum.repos.d/ambari.repo $i:/etc/yum.repos.d/ ;done
[root@ambariserver ~]# for i in hd{1..6} ;do scp /etc/yum.repos.d/hdp.repo $i:/etc/yum.repos.d/ ;done
2.2）部署Ambari Server
[root@ambariserver ~]# yum -y install ambari-server
2.3）初始化Ambari Server
2.3.1）初始化前导入数据库
[root@ambariserver ~]# ll /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
-rwxr-xr-x 1 root root 80064 May 27 2017 /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
[root@ambariserver ~]# mysql -h ambariserver -uambari -pabc123
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 5.5.68-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use ambari;
Database changed
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql ;
2.3.2）Ambari Server初始化
[root@ambariserver ~]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y # y
Enter user account for ambari-server daemon (root):root #root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3 # 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk # /usr/local/jdk
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y # y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3 # 3
Hostname (localhost): ambariserver.a.com # ambariserver.a.com
Port (3306): 3306
Database name (ambari): ambari # mbari
Username (ambari): ambari # ambari
Enter Database Password (bigdata):
Re-enter password: # abc123
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y #y
Extracting system views...
ambari-admin-2.5.1.0.159.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
2.4）启动Ambari Server
`启动服务`
[root@ambariserver ~]# ambari-server start
`开机自启`
[root@ambariserver ~]# chkconfig ambari-server on
2.5）访问Ambari Server服务器
http://192.168.139.30:8080/ 登录用户名和密码均为：admin
2.6）Ambari Agent部署
部署（hd1~hd6）Ambari Agent,我演示一个
[root@hd1 ~]# yum -y install ambari-agent libtirpc-devel
禁用https验证方式(所有的agent主机),我演示一个
[root@hd1 ~]# vim /etc/python/cert-verification.cfg
[https]
verify=disable
ambari-agent配置文件中添加https验证的方式
[root@hd1 ~]# vim /etc/ambari-agent/conf/ambari-agent.ini
........
[security]
........
force_https_protocol=PROTOCOL_TLSv1_2 #添加一行
启动所有ambari-agent，我这里演示一个
[root@hd1 ~]# systemctl start ambari-agent
`开机自启`
[root@hd1 ~]# chkconfig ambari-agent on
2.7）Ambari Server Web端操作
`私钥`
`因为6个主机的私钥都是一样的`
[root@hd6 ~]# cat .ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAzQ6UnBxiG7wWiRHlKZdIp+dr7dLzV+thRM/hcBFjWCXOcwVV
Mt15bTvHoa0zfqFWUSMIJxeeQJYB/bfT1VbYMpLvbtWumjkMf6hci4LXHMgZQ87E
arxVapC0zBn2noAl2x+6bLYs0cLzr561DNIvQBFCZ4kQwaXG8JFOU5wwCDqv2MQT
WQkGWG1s6GZV9COy3qBt97SBiAdh836VpLIiA6w6wRHGmonMtyMCS3I73uG2Ot6H
dO0fla7BWO7iVQsb2dgpoeLKx+8Xwi49eADiYOnIOMGNxe+DssCVNnwf46i4yT1e
MQQBmKdHPuiosLctjJSlVxZjvlr/roOpkYloXQIDAQABAoIBAEZj5+3C7Ia6xT2e
T/07+mvwzCLilDgtC8HUXn3a6jBbIghn3zxLvKAbska0rwWcGoZVsTnCKMfg9qQC
04vbePG7cMeqQHqedMTppYMakf9GTS3yr4g4SFYBgJ7Dlt4H/Xa/6H3i93W2J3K0
10+kYyXqBtKFwaPH5Ye1SZ738aEQ7lJj/+gLzXJy0/oXQ+c0OCuHPx3vERSUBoU8
T4Gz60tP85D4lQVI82MBME2yamXPTZhU3QAkrD7gtgXAoubL57X4szmSdl5eTsF7
A4r4oGVU6ZuhBLtaGwnp7eIXdkxMXJbzQbYXYCxspDpChiOBP19Y6+w9wIK656Ux
appWUp0CgYEA+vEF8r2ZC4R/yEBwsSitWxRJJrnMTdW6XbV/+NmYVbt5i2ffaDp7
b9eMoO0PWV/4Rk7Jv6XweztQW/gP/PHT9T9EU/pHbPqKBzMSIvZs1gBwgf/oei5T
Q13PYcKO5V+0/GPPRtrEEEBSoiRDRgMiPgVt16FKaXU3/y1zywZPQAsCgYEA0TDF
drDm7u8CBMclvnvfxJFk6isj90DzUud9uN2tpjFhNHPyoiEJs3V94JFYkAnvpUuV
FvOTNVpkWThUhDEr5vvMKOzgUHHosAT9kpePqcOsxxYfYpVEoFPLEj9xAfxfJXgV
G+J50JkDPc3Q3Hhm5XwQlo2Hdzx6bMEXs1nRsjcCgYEAw2y8syPQclXHpc35dR42
kz2jElOTtfICZ0QX21ap+eo8Z8KHC4av5pO44wJ696Fzml11ZJj4RSAWHul6mgRC
izWwIq5AvbAHmj+9mhjK/YXuGG+RPsWg9k+yV3HLEvqL2hpoq8dm9TkCoV8YPQfn
vOFBf7IOXz54U3O0P6I0Sa0CgYAMKu4RbVDBbR2M8Zh8b/AgGuTnygKZiymcFOL3
Mc278PtadbQTQiCyU48y1MF1/131G2RcpWZWoB9eISypU8Ld+J9zcfPKOhXyxxeI
w1AT69HM8rh4n34SZU1WjmIgX3GZWFbB/HZhb8UEf0g6yrYRexnr5+eNQ/NLCCA8
ymisGQKBgA3CW2FqHTdZoxJ8t5Bg59iaycyk0yz0HP08c+eHAjIX+9NuSMEZ8SaS
A4wr2/w6ovlDQV0YbmT2VAazjlF8RROfpP6EqN3TrrxQr46PJCMmjFvPBYGVC4fg
FGmDZtGUujcT9XcA3PlVfI5Hu73jetntWDivUyL0zTujIVgdkuOr
-----END RSA PRIVATE KEY-----
选主
选从
服务配置
展示
3）验证Hadoop集群可用性
[root@hd1 ~]# su - hdfs
`创建目录`
[hdfs@hd1 ~]# hdfs dfs -mkdir /input
`准备文件`
[hdfs@hd1 ~]# vim test.txt
nihap
12333
123332
12333
192.168.139.33
192.168.139.33
192.168.139.31
nihap
`上传文件`
[hdfs@hd1 ~]# hdfs dfs -put /root/test.txt /input
`运行实例`
[root@hd1 ~]# hadop jar /usr/hdp/2.6.1.0-129/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /input /output/00
`查看输出结果`
[hdfs@hd1 ~]# hdfs dfs -ls /output/00
Found 2 items
-rw-r--r-- 3 root supergroup 0 2023-12-03 20:35 /output/00/_SUCCESS
-rw-r--r-- 3 root supergroup 59 2023-12-03 20:35 /output/00/part-r-00000
[hdfs@hd1 ~]# hdfs dfs -cat /output/00/part-r-00000
12333 2
123332 1
192.168.139.31 1
192.168.139.33 2
nihap 2