devbox
IT小白
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

虚拟机环境下用docker在centos环境中搭建Ambari HDP大数据平台

作者：IT小白 | 2024-06-12 02:08:27

赞

踩

ambari编译docker

利用虚拟机安装docker，在docker环境中搭建HDP大数据平台

前言 : 在一台虚拟机上搭建3个docker容器 作为安装HDP的环境，虚拟机环境centos7.6 docker环境为centos7.6

因为想要对ambari进行二次开发，所以本次选择手动编译ambari。

https://hub.docker.com/_/centos 在此网站查询想要的版本号

一、在虚拟机上准备工作

1.1安装docker

yum -y install docker

 

1.2下载镜像

  1. docker search centos      　　　　   //搜索镜像
  2. docker pull centos:7.6.1810        //从官网拉取镜像docker images            　　　　　　//查看本地镜像docker ps -a              　　　　  //查看容器

 

1.3启动容器

因为是在1台虚拟机搭建3个docker的缘故，所以这次选择的docker 网卡类型为bridge模式 网段为 172.17.0.0/16 网关为172.17.0.1

三台容器的主机名和ip分别为

  1. server     172.17.0.2
  2. agent1     172.17.0.3
  3. agent2     172.17.0.4

※注意点：bridge模式容器ip无法自己指定

开始启动3台容器，如下图

docker run -it -d --name server --network bridge --hostname server docker.io/centos:7.6.1810 /bin/bash

1.4查看容器并进入

docker ps

进入容器

docker exec -it server /bin/bash

二、搭建编译环境

2.1获取Ambari2.7.5源码

下载地址：http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz

  1. wget http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz
  2. tar zxvf apache-ambari-2.7.5-src.tar.gz

2.2搭建编译环境

  1. 安装包已放到百度云 maven jdk nodejs 
  2. 链接：https://pan.baidu.com/s/1SVf7n5d2M_byquSpwN5yXg 
  3. 提取码：i4e6

2.2.1安装网络工具

yum install net-tools.x86_64

 2.2.2 安装jdk

rpm -ivh jdk-8u74-linux-x64.rpm

2.2.3 安装maven，nodejs

tar -zxvf apache-maven-3.3.9-bin.tar.gztar -zxvf node-v4.5.0-linux-x64.tar.gzyum install epel-release

2.2.3.1 升级nodejs

  1. npm install n -g
  2. n stable

2.2.4 配置maven，nodejs的环境变量

vim /etc/profile

export MAVEN_HOME=/opt/apache-maven-3.3.9export PATH=$MAVEN_HOME/bin:$PATH

export NODE_HOME=/opt/node-v4.5.0-linux-x64

export PATH=$PATH:$NODE_HOME/bin

export NODE_PATH=$NODE_HOME/lib/node_modules

source /etc/profile

2.2.5 python 2.6或2.7(系统自带)

2.2.6安装rpmbuild

yum install rpm-build

2.2.7安装g++

yum install gcc-c++

2.2.8安装python-devel

yum install python-devel

2.2.9安装bower、gulp

  1. npm install -g bower
  2. npm install -g gulp

2.2.10安装git

yum install git

2.2.11安装yarn

yum install yarn

三、提前下载并设置

 下载源码,设置版本信息

  1. wget https://www-eu.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz 
  2. tar -xfvz apache-ambari-2.7.5-src.tar.gz
  3. cd apache-ambari-2.7.5-src
  4. mvn versions:set -DnewVersion=2.7.5.0.0
  5.  
  6. pushd ambari-metrics
  7. mvn versions:set -DnewVersion=2.7.5.0.0
  8. popd


有些包比较大，或者编译时下载时间较长，可以提前下载到本地目录，再修改pom.xml文件指定到本地目录(pom文件位置:/opt/apache-ambari-2.7.5-src/ambari-metrics/pom.xml)

3.1 hbase下载

  1. mkdir -p /tmp/hbase
  2.  
  3. wget -O /tmp/hbase/hbase-2.0.2.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz

3.2 grafana 包下载

  1. mkdir -p /tmp/grafana
  2.  
  3. wget -O /tmp/grafana/grafana-6.4.2.linux-amd64.tar.gz  https://dl.grafana.com/oss/release/grafana-6.4.2.linux-amd64.tar.gz

3.3 hadoop包下载

mkdir -p /tmp/hadoop

wget -O /tmp/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz

3.4phoenix包下载

mkdir -p /tmp/phoenix

wget -O /tmp/hadoop/phoenix-5.0.0.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/phoenix/phoenix-5.0.0.3.1.4.0-315.tar.gz

3.5 开始编译 

mvn -B clean install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -Preplaceurl

最后

编译好后的rpm包在

  1. cd ambari-server/target/rpm/ambari-server/RPMS/x86_64/
  2.  
  3. ambari-server-2.7.4.0-0.x86_64.rpm

编译遇到的问题

问题1：

  1.  [INFO] ------------------------------------------------------------------------
  2.  [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
  3.  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed.
  4.  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
  5.  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
  6.  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
  7.  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
  8.  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
  9.  at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
  10.  at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
  11.  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
  12.  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
  13.  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
  14.  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
  15.  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
  16.  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
  17.  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  18.  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  19.  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  20.  at java.lang.reflect.Method.invoke(Method.java:498)
  21.  at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
  22.  at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
  23.  at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
  24.  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
  25.  Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed.
  26.  at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:362)
  27.  at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
  28.  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
  29.  ... 20 more
  30.  Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
  31.  at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:377)
  32.  at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
  33.  at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:610)
  34.  at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:352)
  35.  ... 22 more
  36.  [ERROR] 
  37.  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  38.  [ERROR] 
  39.  [ERROR] For more information about the errors and possible solutions, please read the following articles:
  40.  [ERROR] [Help 1] [http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException] [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :ambari-admin解决方法：修改 /XXXX/ambari-admin/src/main/resources/ui/admin-web/package.json 的bower 版本将1.3.8 改为 1.8.8 问题解决


 

  1. /ambari-admin/src/main/resources/ui/admin-web/package.json
  2. @@ -3,7 +3,7 @@
  3.    "version": "0.0.0",
  4.    "dependencies": {},
  5.    "devDependencies": {
  6. -    "bower": "1.3.8",
  7. +    "bower": "1.8.8",
  8.      "gulp": "^3.8.8",
  9.      "gulp-add-src": "^0.2.0",
  10.      "gulp-autoprefixer": "0.0.7",

问题2：Too many files with unapproved license


解决办法：

由于许可license没有造成的，需要再mvn命令中增加如下内容：

-Drat.skip=true

问题3：Unexpected character 0x0 in identifier编译ambari-utility，报错信息：

 

说明：

出错的Java文件编码和CheckStyle设置的编码不同。CheckStyle里设置的编码是UTF-8

解决办法：

将“._xxx.java”文件直接删除即可。

(注：其实有很多类似这样的文件报错，将类似这样的文件删除掉即可)

问题4： Download HBase超时

 

解决办法：

将./ambari-metrics/pom.xml的

<hbase.tar>http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>

替换为本地路径

<hbase.tar>file:///tmp/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>

问题5：Unable to build the RPM

Unable to build the RPM: Error while executing process. Cannot run program “rpmbuild”

解决办法：

  1. rpm -qa | grep rpm-build #检测是否已安装rpmbuild
  2. yum install rpm-build #如果没安装则手动安装


问题6. buildNumber : unbound variable

  1. vim /usr/sbin/ambari-server
  2. 将${buildNumber}这行换成 HASH="${VERSION}"

该文件源码位置位于：./ambari-server/sbin/ambari-server，修改后，在进行编译，可以一劳永逸。

 

问题7：

[INFO] Ambari Admin View .................................. FAILURE

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed.

单独编译报错：

ECMDERR Failed to execute "git ls-remote --tags --heads https://github.com/twbs/bootstrap.git", exit code of #128

Additional error details:fatal: unable to access 'https://github.com/twbs/bootstrap.git/': Peer reports incompatible or unsupported protocol version.

解决：个人将ambari-admin(包括子目录ambari-admin/src/main/resources/ui/admin-web)里面生成的node、node_modules、package-lock.json删除，保持干净的源码环境，再单独编译ambari-admin

#cd ambari-admin/src/main/resources/ui/admin-web目录下，编辑 .bowerrc ，修改后的内容如下：

  1. {
  2.     "directory": "app/bower_components",
  3.     "allow_root": true
  4. }

在上述目录下，安装npm依赖包，全局安装gulp、bower

  1. npm install -g bower
  2.  
  3. npm install -g gulp

安装bower的依赖包(个人在ambari-admin/src/main/resources/ui/admin-web/下执行)

bower install

(上述错误原因在于bower install命令失败，个人将之前成功编译的bower_components目录(ambari-admin/src/main/resources/ui/admin-web/app目录下)拷贝到ambari-release-2.6.1/ambari-admin/src/main/resources/ui/admin-web/app目录下，再次运行成功)

安装 gulp-webserver

npm install gulp-webserver --save-dev

运行命令mvn install -Drat.skip=true -Preplaceurl -X重新单独编译ambari-admin模块(此命令在ambari-admin目录下运行)

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/IT小白/article/detail/705652
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号