首页 > 网络 > 云计算 >

虚拟机ubuntu+离线安装hadoop

2016-09-30

虚拟机ubuntu+离线安装hadoop。

1.修改hosts

192.168.1.10 m1 master #说明m1 主机名字master映射的名字

192.168.1.11 s1 slave01

2.修改hostname

m1

3.关闭防火墙

ufw disable

4.安装JDK7,CDH5要求至少是Oracle JDK7

(过程中会弹出个框,选择YES,因为要下载二进制包,所以可能会慢一些)

root@m1:~# add-apt-repository ppa:webupd8team/java

root@m1:~# apt-get update

root@m1:~# sudo apt-get upgrade

root@m1:~# apt-get install oracle-java7-installer

将Oracle 7 设置成默认版本

root@m1:~# apt-get installoracle-java7-set-default

查看当前Java版本

root@m1:~# java -version

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

5 安装MySql最新版,CM的数据库我们用My Sql管理

root@m1:~# apt-get install mysql-server

修改MySql配置,方便CM使用

root@m1:~# cp /etc/mysql/my.cnf /etc/mysql/my.cnf.bak root@m1:~# vi /etc/mysql/my.cnf #bind-address = 127.0.0.1 bind-address =0.0.0.0

6 创建后面在CM中会使用的数据库

root@m1:~# service mysql restart

root@m1:~# netstat -tulpn | grep :3306

tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2207/mysqld

root@m1:~# apt-get install libmysql-java

Reading package lists... Done

Building dependency tree

Reading state information... Done

Suggested packages:

liblog4j1.2-java libcommons-logging-java

The following NEW packages will be installed:

libmysql-java

0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.

Need to get 894 kB of archives.

After this operation, 1,060 kB of additional disk space will be used.

Get:1 http://mirrors.sohu.com/ubuntu/ trusty/universe libmysql-java all 5.1.28-1 [894 kB]

Fetched 894 kB in 1s (718 kB/s)

Selecting previously unselected package libmysql-java.

(Reading database ... 96338 files and directories currently installed.)

Preparing to unpack .../libmysql-java_5.1.28-1_all.deb ...

Unpacking libmysql-java (5.1.28-1) ...

Setting up libmysql-java (5.1.28-1) ...

root@m1:~#

7 查看 MySql3306端口,并安装MySql对Java的支持

root@m1:~# netstat -tulpn | grep :3306

tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2207/mysqld

root@m1:~# apt-get install libmysql-java

8 将CDH5的软件源添加到Apt中

root@m1:~# curl "http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/cloudera.list" -o /etc/apt/sources.list.d/cloudera_precise.list

root@m1:~# curl -s http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -

root@m1:~# apt-get update

9安装Cloudier Manager

root@m1:~# apt-get install cloudera-manager-daemons cloudera-manager-server

10 修改Cloudera-Manager-Server的配置文件db.properties,设置MySql连接

修改Cloudera-Manager-Server的配置文件db.properties,设置刚才我们在(4.7)Mysql中创建的cmf数据库、用户名、密码(4台机器都要执行

root@m1:~# vi /etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql

com.cloudera.cmf.db.host=m1.idoall.org

com.cloudera.cmf.db.name=cmf

com.cloudera.cmf.db.user=cmf

com.cloudera.cmf.db.password=xyz

11 安装ntp ,CM agents需要用到"ntpdc"

root@m1:~# apt-get install ntp

12 重启Cloudera Manager Server

root@m1:~# service cloudera-scm-server start

Starting cloudera-scm-server: * cloudera-scm-server started

root@m1:~#

查看Cloudera manager的启动日志,如果能够看到下面的类似信息,说明启动正确,tailf -100 /var/log/cloudera-scm-server/cloudera-scm-server.log或者/var/log/cloudera-scm-server/cloudera-scm-server.out,如果看不到类似下面的信息,那么也可以看到哪里出了问题,大多数都是和数据库的配置有关系。

2014-08-26 23:07:52,643 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1182ms: GC pool 'Copy' had collection(s): count=1 time=1641ms

2014-08-26 23:07:52,644 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1634ms: GC pool 'Copy' had collection(s): count=1 time=1641ms

2014-08-26 23:07:52,696 INFO [WebServerImpl:servlet.DispatcherServlet@339] FrameworkServlet 'Spring MVC Dispatcher Servlet': initialization completed in 18068 ms

2014-08-26 23:07:52,793 INFO [WebServerImpl:cmon.JobDetailGatekeeper@127] ActivityMonitor configured to allow job details for all jobs.

2014-08-26 23:07:53,407 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@399] Initializing SearchTemplateManager:2014-08-27T06:07:53.407Z

2014-08-26 23:07:53,730 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@403] Generating entities:2014-08-27T06:07:53.730Z

2014-08-26 23:07:53,821 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@407] Num entities:112

2014-08-26 23:07:53,822 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@409] Generating documents:2014-08-27T06:07:53.822Z

2014-08-26 23:07:53,891 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@411] Num docs:124

2014-08-26 23:07:53,892 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@352] Constructing repo:2014-08-27T06:07:53.892Z

2014-08-26 23:07:53,979 INFO [WebServerImpl:mortbay.log@67] jetty-6.1.26.cloudera.2

2014-08-26 23:07:54,008 INFO [WebServerImpl:mortbay.log@67] Started SelectChannelConnector@0.0.0.0:7180

2014-08-26 23:07:54,009 INFO [WebServerImpl:cmf.WebServerImpl@292] Started Jetty server.

2014-08-26 23:07:55,276 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@354] Finished constructing repo:2014-08-27T06:07:55.276Z

13 通过Web方式继续安装

创建update-alternatives工具在/usr/sbin/下创建软链

root@m1:~# sudo ln -s /usr/bin/update-alternatives /usr/sbin/update-alternatives

14 打开浏览器,输入http://m1:7180/开始安装

使用浏览器打开能够看到登录页面,那么恭喜你成功了。http://m1:7180/(帐号和密码都是admin)

这里忽略 界面 设计 步骤

15 离线方式安装CDH的Parcel

使用迅雷到http://archive.cloudera.com/cdh5/parcels/5.1.2/目录,下载CDH-5.1.2-1.cdh5.1.2.p0.3-precise.parcel和mainfest.json文件

上传到/opt/cloudera/parcel-repo目录中,同时创建一个CDH-5.1.2-1.cdh5.1.2.p0.3-precise.parcel.sha文件,内容为"a492e4b6dece2850f0a37f2bf613ecb2980dfd37",这个值可以在下载目录中的manifest.json文件中找到。然后看相对应的json->parcelName同一级的hash值。

设置文件的使用权限

root@m1:/opt/cloudera/parcel-repo# chmod 777 -R .

root@m1:/opt/cloudera/parcel-repo# chown cloudera-scm:cloudera-scm ./*

错误处理

DNS反向解析PTR localhost:

描述:

DNS反向解析错误,不能正确解析Cloudera Manager Server主机名

日志:

Detecting Cloudera Manager Server...

Detecting Cloudera Manager Server...

BEGIN host -t PTR 192.168.1.198

198.1.168.192.in-addr.arpa domain name pointer localhost.

END (0)

using localhost as scm server hostname

BEGIN which python

/usr/bin/python

END (0)

BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' localhost 7182

Traceback (most recent call last):

File "", line 1, in

File "", line 1, in connect

socket.error: [Errno 111] Connection refused

END (1)

could not contact scm server at localhost:7182, giving up

waiting for rollback request

解决方案:

将连不上的机器 /usr/bin/host 文件删掉,执行下面命令:

sudo mv /usr/bin/host /usr/bin/host.bak

复制代码

错误:

Reading package lists...

Building dependency tree...

Reading state information...

Some packages could not be installed. This may mean that you have

requested an impossible situation or if you are using the unstable

distribution that some required packages have not yet been created

or been moved out of Incoming.

The following information may help to resolve the situation:

The following packages have unmet dependencies:

cloudera-manager-agent : Depends: libssl-dev but it is not going to be installed

E: Unable to correct problems, you have held broken packages.

解决方案 执行 apt install libssl-dev

错误:

libssl-dev : Depends: libssl1.0.0 (= 1.0.1-4ubuntu5) but 1.0.1-4ubuntu5.3 is to be installed

Recommends: libssl-doc but it is not going to be installed

E: Unable to correct problems, you have held broken packages.

原因:

已安装的libssl1.0.0版本太高, 无法支持

解决方案

root@jack-slave:~# sudo apt-get install aptitude

sudo aptitude install libssl-dev

下列动作将解决这些依赖关系:

保持 下列软件包于其当前版本:

1) libssl-dev [未安装的]

是否接受该解决方案?[Y/n/q/?] n

下列动作将解决这些依赖关系:

降级 下列软件包:

1) libssl1.0.0 [1.0.1f-1ubuntu2.4 (now) -> 1.0.1f-1ubuntu2 (trusty)]

是否接受该解决方案?[Y/n/q/?] y

下列软件包将被“降级”:

libssl1.0.0

下列“新”软件包将被安装。

libssl-dev libssl-doc{a} zlib1g-dev{a}

0 个软件包被升级,新安装 3 个, 1 个被降级, 0 个将被删除, 同时 2 个将不升级。

需要获取 3,040 kB 的存档。 解包后将要使用 8,253 kB。

您要继续吗?[Y/n/?] y

读取: 1 http://cn.archive.ubuntu.com/ubuntu/ trusty/main libssl1.0.0 amd64 1.0.1f-1ubuntu2 [825 kB]

读取: 2 http://cn.archive.ubuntu.com/ubuntu/ trusty/main zlib1g-dev amd64 1:1.2.8.dfsg-1ubuntu1 [183 kB]

读取: 3 http://cn.archive.ubuntu.com/ubuntu/ trusty/main libssl-dev amd64 1.0.1f-1ubuntu2 [1,066 kB]

读取: 4 http://cn.archive.ubuntu.com/ubuntu/ trusty/main libssl-doc all 1.0.1f-1ubuntu2 [965 kB]

已下载 3,040 kB,耗时 2秒 (1,128 kB/s)

正在预设定软件包 ...

dpkg:警告:即将把 libssl1.0.0:amd64 从 1.0.1f-1ubuntu2.4 降级到 1.0.1f-1ubuntu2

(正在读取数据库 ... 系统当前共安装有 178253 个文件和目录。)

正准备解包 .../libssl1.0.0_1.0.1f-1ubuntu2_amd64.deb ...

正在将 libssl1.0.0:amd64 (1.0.1f-1ubuntu2) 解包到 (1.0.1f-1ubuntu2.4) 上 ...

正在选中未选择的软件包 zlib1g-dev:amd64。

正准备解包 .../zlib1g-dev_1%3a1.2.8.dfsg-1ubuntu1_amd64.deb ...

正在解包 zlib1g-dev:amd64 (1:1.2.8.dfsg-1ubuntu1) ...

正在选中未选择的软件包 libssl-dev:amd64。

正准备解包 .../libssl-dev_1.0.1f-1ubuntu2_amd64.deb ...

正在解包 libssl-dev:amd64 (1.0.1f-1ubuntu2) ...

正在选中未选择的软件包 libssl-doc。

正准备解包 .../libssl-doc_1.0.1f-1ubuntu2_all.deb ...

正在解包 libssl-doc (1.0.1f-1ubuntu2) ...

正在处理用于 man-db (2.6.7.1-1) 的触发器 ...

正在设置 libssl1.0.0:amd64 (1.0.1f-1ubuntu2) ...

正在设置 zlib1g-dev:amd64 (1:1.2.8.dfsg-1ubuntu1) ...

正在设置 libssl-dev:amd64 (1.0.1f-1ubuntu2) ...

正在设置 libssl-doc (1.0.1f-1ubuntu2) ...

正在处理用于 libc-bin (2.19-0ubuntu6) 的触发器 ...

root@jack-slave:~# dpkg -l *libssl*

配置远程 Parcel 存储库 URL

安装 sudo apt-get install apache2

在 /var/www/lpz 目录 将CDH 和。sha放入目录中

相关文章
最新文章
热点推荐