Fujitsu
Not logged in » Login
X

Please login

Please log in with your Fujitsu Partner Account.

Login


» Forgot password

Register now

If you do not have a Fujitsu Partner Account, please register for a new account.

» Register now
Sep 26 2017

Practical Tips for SPARC: Using Hadoop and Spark on SPARC Servers/Solaris Platform – Configuring Hadoop Cluster Environments (Part 3)

/data/www/ctec-live/application/public/media/images/blogimages/32136_Fujitsu_M10-1_front_view_3D_open_scr.jpg

In part 1 and part 2 of this series, we set up a Hadoop cluster configuration. Today's third and final piece provides step-by-step instructions on how to start and verify operation of the Hadoop cluster environment.

Confirmation of the System: Starting Hadoop Processes

ALERT: Although the message below may appear, it can be ignored, because Hadoop processes will not be affected.

xx/xx/xx xx:xx:xx WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Start "Zookeeper" on "ctrl-node1", "ctrl-node2" and "jrnl-node" first.
# su - hdfs -c '/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg'

Check "Zookeeper" status.
# /usr/java/bin/jps | grep QuorumPeerMain
The message below indicates that it started successfully.
21384 QuorumPeerMain

Initialize HA state in "Zookeeper" on "ctrl-node1".
# su - hdfs -c 'hdfs zkfc -formatZK'

Start "Journalnode" on "ctrl-node1", "ctrl-node2" and "jrnl-node".
# su - hdfs -c 'hadoop-daemon.sh start journalnode'

Check "Journalnode" status.
# /usr/java/bin/jps | grep JournalNode
The message below indicates that it started successfully.
21893 JournalNode

Format Hadoop file system on "ctrl-node1".
# su - hdfs -c 'hdfs namenode -format'
The message below shows that it is formatted successfully.
16/10/24 08:17:36 INFO common.Storage: Storage directory /hdpool/data/1/dfs/nn has been successfully formatted.

Format JournalNode edit data on "ctrl-node1". Without errors, it terminated successfully. The data on "ctrl-node1" and "jrnl-node" will also be formatted at the same time.
# su - hdfs -c 'hdfs namenode -initializeSharedEdits'

Start "NameNode" on "ctrl-node1".
# su - hdfs -c 'hadoop-daemon.sh start namenode'

Check "NameNode" status on "ctrl-node1".
# /usr/java/bin/jps | grep NameNode
The message below indicates that it started successfully.
25852 NameNode

Start "ResourceManager" on "ctrl-node1".
# su - yarn -c 'yarn-daemon.sh start resourcemanager'

Check "ResourceManager" status on "ctrl-node1".
# /usr/java/bin/jps | grep ResourceManager
The message below indicates that it started successfully.
25982 ResourceManager

Start "zkfc" on "ctrl-node1".
# su - hdfs -c 'hadoop-daemon.sh start zkfc'

Check "zkfc" status on "ctrl-node1".
# /usr/java/bin/jps | grep DFSZKFailoverController
The message below indicates that it started successfully.
26860 DFSZKFailoverController

By running the following command on "ctrl-node2", Hadoop file system meta data will be copied from "ctrl-node1".
# su - hdfs -c 'hdfs namenode -bootstrapStandby'
The message below indicates that it copied successfully.
16/10/24 08:17:36 INFO common.Storage: Storage directory /hdpool/data/1/dfs/nn has been successfully formatted.

Start "NameNode" on "ctrl-node2".
# su - hdfs -c 'hadoop-daemon.sh start namenode'

Check "NameNode" status on "ctrl-node2".
# /usr/java/bin/jps | grep NameNode
The message below indicates that it started successfully.
25852 NameNode

Start "ResourceManager" on "ctrl-node2".
# su - yarn -c 'yarn-daemon.sh start resourcemanager'

Check "ResourceManager" status on "ctrl-node2".
# /usr/java/bin/jps | grep ResourceManager
The message below indicates that it started successfully.
25982 ResourceManager

Start "zkfc" on "ctrl-node2".
# su - hdfs -c 'hadoop-daemon.sh start zkfc'

Check "zkfc" status on "ctrl-node2".
# /usr/java/bin/jps | grep DFSZKFailoverController
The message below indicates that it started successfully.
26860 DFSZKFailoverController

Check "NameNode" redundant states. Normally, "nn1" should be "active" and "nn2" should be "standby". The following command can be run on either "ctrl-node1" or "ctrl-node2".
# su - hdfs -c 'hdfs haadmin -getServiceState nn1'
Oracle Corporation SunOS 5.11 11.3 June 2016
16/10/24 14:29:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
active
# su - hdfs -c 'hdfs haadmin -getServiceState nn2'
Oracle Corporation SunOS 5.11 11.3 June 2016
16/10/24 14:29:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
standby

Check "ResourceManager" redundant states. "rm1" should be "active" and "rm2" should be "standby". The following command can be run on either "ctrl-node1" or "ctrl-node2".
# su - yarn -c 'yarn rmadmin -getServiceState rm1'
Oracle Corporation SunOS 5.11 11.3 June 2016
16/10/24 14:29:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
active
# su - yarn -c 'yarn rmadmin -getServiceState rm2'
Oracle Corporation SunOS 5.11 11.3 June 2016
16/10/24 14:29:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
standby

Start "DataNode" and "NodeManager" on "data-node1" and "data-node2".
# su - hdfs -c 'hadoop-daemon.sh start datanode'
# su - yarn -c 'yarn-daemon.sh start nodemanager'

Check "DataNode" status on "data-node1" and "data-node2".
# /usr/java/bin/jps | grep DataNode
The message below indicates that it started successfully.
26240 DataNode

Check "NodeManager" status on "data-node1" and "data-node2".
# /usr/java/bin/jps | grep NodeManager
The message below indicates that it started successfully.
26299 NodeManager

Start "JobHistoryServer" on "ctrl-node1".
# su - mapred -c 'mr-jobhistory-daemon.sh start historyserver'

Check "JobHistoryServer" status on "ctrl-node1".
# /usr/java/bin/jps | grep JobHistoryServer
The message below indicates that it started successfully.
26573 JobHistoryServer

Create HDFS directories.
# (Login to ctrl-node1 as "hdfs" user)
# hadoop fs -mkdir /tmp
# hadoop fs -chmod -R 1777 /tmp
# hadoop fs -mkdir /data
# hadoop fs -mkdir /data/history
# hadoop fs -chmod -R 1777 /data/history
# hadoop fs -chown yarn /data/history
# hadoop fs -mkdir /var
# hadoop fs -mkdir /var/log
# hadoop fs -mkdir /var/log/hadoop-yarn
# hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
# hadoop fs -mkdir /data/spark
# hadoop fs -chown spark /data/spark
# hadoop fs -mkdir /apl
# hadoop fs -mkdir /apl/spark-2.1.0
# hadoop fs -chmod 775 /apl/spark-2.1.0
# hadoop fs -mkdir /apl/spark-2.1.0/jars
# hadoop fs -chmod 775 /apl/spark-2.1.0/jars

On every node, confirm that the HDFS directories can be accessed from all the Hadoop nodes, as follows:
# su - hdfs -c 'hadoop fs -ls -R /'

Confirm that the directories below are displayed on every node.
drwxr-xr-x - hdfs hadoop 0 2016-10-25 07:35 /apl
drwxrwxr-x - hdfs hadoop 0 2016-10-25 07:35 /apl/spark-2.1.0
drwxrwxr-x - hdfs hadoop 0 2016-10-25 07:35 /apl/spark-2.1.0/jars
drwxr-xr-x - hdfs hadoop 0 2016-10-25 07:32 /data
drwxrwxrwt - yarn hadoop 0 2016-10-25 07:26 /data/history
drwxr-xr-x - spark hadoop 0 2016-10-25 07:32 /data/spark
drwxrwxrwt - hdfs hadoop 0 2016-10-25 16:08 /tmp
drwxr-xr-x - hdfs hadoop 0 2016-10-25 07:29 /var
drwxr-xr-x - hdfs hadoop 0 2016-10-25 07:29 /var/log
drwxr-xr-x - yarn mapred 0 2016-10-25 07:29 /var/log/hadoop-yarn

Testing
Connect to http://<target IP address>:50070/ with a web browser. The page shown below indicates that Hadoop is working successfully.

Image

 

Run the Hadoop sample application. When it ends successfully, all the procedures are completed.
# su - spark
$ hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 10 20
Number of Maps = 10
Samples per Map = 20
17/03/01 15:21:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
17/03/01 15:21:56 INFO input.FileInputFormat: Total input paths to process : 10
17/03/01 15:21:57 INFO mapreduce.JobSubmitter: number of splits:10
17/03/01 15:21:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1484800892214_0017
17/03/01 15:21:57 INFO impl.YarnClientImpl: Submitted application application_1484800892214_0017
17/03/01 15:21:57 INFO mapreduce.Job: The url to track the job: http://ctrl-node1:8088/proxy/application_1484800892214_0017/
17/03/01 15:21:57 INFO mapreduce.Job: Running job: job_1484800892214_0017
17/03/01 15:22:08 INFO mapreduce.Job: Job job_1484800892214_0017 running in uber mode : false
17/03/01 15:22:08 INFO mapreduce.Job: map 0% reduce 0%
17/03/01 15:22:23 INFO mapreduce.Job: map 20% reduce 0%
17/03/01 15:22:37 INFO mapreduce.Job: map 20% reduce 7%
17/03/01 15:22:40 INFO mapreduce.Job: map 30% reduce 7%
17/03/01 15:22:43 INFO mapreduce.Job: map 40% reduce 10%
17/03/01 15:22:44 INFO mapreduce.Job: map 50% reduce 10%
17/03/01 15:22:46 INFO mapreduce.Job: map 50% reduce 13%
17/03/01 15:22:49 INFO mapreduce.Job: map 60% reduce 17%
17/03/01 15:22:50 INFO mapreduce.Job: map 80% reduce 17%
17/03/01 15:22:52 INFO mapreduce.Job: map 80% reduce 27%
17/03/01 15:22:53 INFO mapreduce.Job: map 100% reduce 27%
17/03/01 15:22:54 INFO mapreduce.Job: map 100% reduce 100%
17/03/01 15:22:55 INFO mapreduce.Job: Job job_1484800892214_0017 completed successfully
17/03/01 15:22:55 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=226
FILE: Number of bytes written=1339151
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2600
HDFS: Number of bytes written=215
HDFS: Number of read operations=43
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=10
Launched reduce tasks=1
Data-local map tasks=10
Total time spent by all maps in occupied slots (ms)=318683
Total time spent by all reduces in occupied slots (ms)=28545
Total time spent by all map tasks (ms)=318683
Total time spent by all reduce tasks (ms)=28545
Total vcore-milliseconds taken by all map tasks=318683
Total vcore-milliseconds taken by all reduce tasks=28545
Total megabyte-milliseconds taken by all map tasks=326331392
Total megabyte-milliseconds taken by all reduce tasks=29230080
Map-Reduce Framework
Map input records=10
Map output records=20
Map output bytes=180
Map output materialized bytes=280
Input split bytes=1420
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=280
Reduce input records=20
Reduce output records=0
Spilled Records=40
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=1414
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1746141184
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1180
File Output Format Counters
Bytes Written=97
Job Finished in 59.708 seconds
Estimated value of Pi is 3.12000000000000000000

Stay tuned for the last topic of this blog series, which will cover how to configure a Spark cluster environment on Oracle VM for SPARC. Please feel free to post any comments and share your experiences in the comment section below.

Disclaimer
The information contained in this blog is for general information purposes only. While we endeavour to keep the information up-to-date and correct through testing on a practical system, we make no warranties of any kind about the completeness, accuracy, reliability, suitability or availability. Any reliance you place on such information is strictly at your own risk. The information in this blog is subject to change without notice.

Shinichiro Asai

 

About the Author:

Shinichiro Asai

Technical Sales Support, Platform Solution Unit, FUJITSU Japan

SHARE

Comments on this article

No comments yet.

Please Login to leave a comment.