无法在Cloudera Quickstart VM (5.10)中安装spark 2.2

Question

我已经关注了这里的博客(下面提到的)，并根据需要下载了包裹并放入。请让我知道是否有人已经安装和步骤。

(https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html)

/opt/cloudera/csd/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658-el5.parcel

但是服务cloudera-scm-server restart没有执行。要使用Cloudera Express (免费)，请运行：

sudo /home/cloudera/cloudera-manager --express

这至少需要8 GB的RAM和至少2个虚拟CPU。

Answer 1

SPARK 2.2 Installation Setup on Cloudera VM

Step 1: Download a quickstart_vm from the link:
Prefer a vmware platform as it is easy to use, anyways all the options are viable.
Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids. 


Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.


Step 3: Please open the terminal and switch to root user as:
         su root
         password: cloudera

Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
(a). Downloading Java:
wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

(b). Switch to /usr/java/ directory with “cd /usr/java/” command.

(c). cp the java download tar file to the /usr/java/ directory.

(d). Untar the directory with “tar –zxvf jdk-8u31-linux-x64.tar.gz”

(e). Open the profile file with the command “vi ~/.bash_profile” 

(f). export JAVA_HOME to the new java directory.
       “export JAVA_HOME=/usr/java/jdk1.8.0_131”

       Save and Exit.


(g). In order to reflect the above change, following command needs to be executed on the shell:
       source ~/.bash_profile

Step 5:  The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.

(a). Switch to /opt/  directory with the command:
“cd /opt/”

(b). Download spark with the command:
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz

(c). Untar the spark tar with the following command:
tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz

(d). We need to define some environment variables as default settings:
Please open a file with the following command:
vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
Paste the following configurations in the file:
SPARK_MASTER_IP=192.168.50.1
SPARK_EXECUTOR_MEMORY=512m
SPARK_DRIVER_MEMORY=512m
SPARK_WORKER_MEMORY=512m
SPARK_DAEMON_MEMORY=512m
Save and exit

(e).    We need to start spark with the following command:
/opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
Export spark_home : 
export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/

(f). Change the permissions of the directory:
chmod 777 -R /tmp/hive

(g). Try “spark-shell”, it should work.

Answer 2

尝试此命令

sudo /home/cloudera/cloudera-manager --express --force

Answer 3

请遵循下面的视频，它包含了在Clouedra VM中安装Sprak2所需的所有必要步骤。

youtubue链接- https://www.youtube.com/watch?v=lQxlO3coMxM

另外，为了启动Cloudera Express (免费)，你的虚拟机应该分配至少-8 4GB的内存，或者如果你分配了默认的4 4GB内存，那么你可以强制开始执行下面的命令，然后按照上面的视频进行操作。

sudo /home/cloudera/cloudera-manager --force --express