Setup CDH cluster on AWS

Select instance type

Login to

$ ssh -i us-west-2.pem ec2-user@ec2-54-183-11-203.us-west-1.compute.amazonaws.com

__| __|_ )

_| ( / Amazon Linux AMI

___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/

8 package(s) needed for security, out of 13 available

Run "sudo yum update" to apply all updates.

[ec2-user@ip-172-31-9-213 ~]$

[ec2-user@ip-172-31-9-213 ~]$ wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

--2018-04-11 12:20:09-- http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

Resolving archive.cloudera.com (archive.cloudera.com)... 151.101.188.167

Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.188.167|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 519594 (507K) [application/octet-stream]

Saving to: ‘cloudera-manager-installer.bin’

cloudera-manager-installer.bin 100%[==================================================================================>] 507.42K --.-KB/s in 0.1s

2018-04-11 12:20:09 (5.03 MB/s) - ‘cloudera-manager-installer.bin’ saved [519594/519594]

[ec2-user@ip-172-31-9-213 ~]$ ls -l

total 508

-rw-rw-r-- 1 ec2-user ec2-user 519594 Feb 21 14:30 cloudera-manager-installer.bin

[ec2-user@ip-172-31-9-213 ~]$ chmod +x cloudera-manager-installer.bin

[ec2-user@ip-172-31-9-213 ~]$ sudo ./cloudera-manager-installer.bin

[ec2-user@ip-172-31-9-213 ~]$ sudo df -h

Filesystem Size Used Avail Use% Mounted on

devtmpfs 3.8G 44K 3.8G 1% /dev

tmpfs 3.8G 0 3.8G 0% /dev/shm

/dev/nvme0n1p1 20G 1.1G 19G 6% /

[ec2-user@ip-172-31-9-213 ~]$ sudo lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

nvme0n1 259:0 0 20G 0 disk

├─nvme0n1p1 259:3 0 20G 0 part /

└─nvme0n1p128 259:4 0 1M 0 part

[ec2-user@ip-172-31-9-213 ~]$

[ec2-user@ip-172-31-9-213 ~]$ wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

--2018-04-11 12:20:09-- http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

Resolving archive.cloudera.com (archive.cloudera.com)... 151.101.188.167

Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.188.167|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 519594 (507K) [application/octet-stream]

Saving to: ‘cloudera-manager-installer.bin’

cloudera-manager-installer.bin 100%[==================================================================================>] 507.42K --.-KB/s in 0.1s

2018-04-11 12:20:09 (5.03 MB/s) - ‘cloudera-manager-installer.bin’ saved [519594/519594]

[ec2-user@ip-172-31-9-213 ~]$ ls -l

total 508

-rw-rw-r-- 1 ec2-user ec2-user 519594 Feb 21 14:30 cloudera-manager-installer.bin

[ec2-user@ip-172-31-9-213 ~]$ chmod +x cloudera-manager-installer.bin

[ec2-user@ip-172-31-9-213 ~]$ sudo ./cloudera-manager-installer.bin

[ec2-user@ip-172-31-9-213 ~]$

[Downloads]$ chmod 400 cdh-us-west-1.pem

[Downloads]$ ssh -i cdh-us-west-1.pem centos@ec2-54-215-204-150.us-west-1.compute.amazonaws.com

[centos@ip-172-31-15-110 ~]$ sudo su

[root@ip-172-31-15-110 centos]#

[root@ip-172-31-15-110 centos]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/nvme0n1p1 20G 843M 20G 5% /

devtmpfs 3.7G 0 3.7G 0% /dev

tmpfs 3.7G 0 3.7G 0% /dev/shm

tmpfs 3.7G 17M 3.7G 1% /run

tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup

tmpfs 753M 0 753M 0% /run/user/1000

[root@ip-172-31-15-110 centos]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

nvme0n1 259:0 0 20G 0 disk

└─nvme0n1p1 259:1 0 20G 0 part /

Disable SELinux

[root@ip-172-31-15-110 centos]# vi /etc/selinux/config

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - No SELinux policy is loaded.

SELINUX=disabled

# SELINUXTYPE= can take one of three two values:

# targeted - Targeted processes are protected,

# minimum - Modification of targeted policy. Only selected processes are protected.

# mls - Multi Level Security protection.

SELINUXTYPE=targeted

Reboot the system

[root@ip-172-31-15-110 centos]# init 6

Login and check SElinux status

[centos@ip-172-31-15-110 ~]$ sudo su

[root@ip-172-31-15-110 centos]# getenforce

Disabled

[root@ip-172-31-15-110 centos]# cd /etc/yum.repos.d/

[root@ip-172-31-15-110 yum.repos.d]# wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo

Update repo with latest link

[root@ip-172-31-15-110 yum.repos.d]# cat cloudera-manager.repo

[cloudera-manager]

name=Cloudera Manager

#baseurl=http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/

baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.14.2/

gpgkey = http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera

gpgcheck=1

[root@ip-172-31-15-110 yum.repos.d]# yum install cloudera-manager-daemons cloudera-manager-server

[root@ip-172-31-15-110 yum.repos.d]# rpm -qa 'cloudera-manager-*'

cloudera-manager-server-5.14.2-1.cm5142.p0.8.el7.x86_64

cloudera-manager-daemons-5.14.2-1.cm5142.p0.8.el7.x86_64

[root@ip-172-31-15-110 yum.repos.d]#

Install https://dev.mysql.com/doc/mysql-repo-excerpt/5.6/en/linux-installation-yum-repo.html