Fujitsu
Not logged in » Login
X

Please login

Please log in with your Fujitsu Partner Account.

Login


» Forgot password

Register now

If you do not have a Fujitsu Partner Account, please register for a new account.

» Register now
Dec 05 2017

Practical Tips for SPARC: Using Hadoop and Spark on SPARC Servers/Solaris Platform – Configuring Apache Spark™ (Part 2)

/data/www/ctec-live/application/public/media/images/blogimages/32136_Fujitsu_M10-1_front_view_3D_open_scr.jpg

In our previous 3-part blog, we explained how to configure a Hadoop Cluster Node environment. The following entry illustrates how to properly set up Apache Spark™ as part of the intended environment. Unfortunately, presenting our case in a single piece would likely provoke many TL;DR reactions, so we opted for another mini-series. Below is part 2, which describes the required software installtion; part 1 is available here; and part 3 will follow by the end of the week.

Required Software Installation: Oracle Developer Studio Installation
Download and install Oracle Developer Studio.
http://www.oracle.com/technetwork/jp/server-storage/developerstudio/downloads/index.html

For details, refer to the installation manual.
http://docs.oracle.com/cd/E60778_01/html/E60743/index.html

Required Software Installation: Hadoop Installation
Download Hadoop, and transfer to the node to install.
http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

Install Hadoop.
# cd /opt
# <extract archive of hadoop>
# ln -s hadoop-2.7.3 hadoop

Change the owner of Hadoop files to root, and change the group ID to hadoop. All permissions of files are "755".
# chown -R root:hadoop /opt/hadoop-2.7.3
# chmod -R 755 /opt/hadoop-2.7.3

Required Software Installation: liblzma and libiconv Installation
Before installing R, liblzma and GNU libiconv should be installed. While Solaris includes its own version of lzma and iconv, compiling R will fail using these libraries. So, the additional libraries should be linked instead of those that comes with Solaris.

Download liblzma, and transfer to the node to install.
https://tukaani.org/xz/xz-5.2.2.tar.gz

To compile in 64-bit mode, specify "-m64" for the environment variable CFLAGS. And then, run "configure".
# cd xz-5.2.2
# export CFLAGS="-m64"
# ./configure

Without errors, run "make install".
# make install

Confirm that lzma.h is in /usr/local/include and that liblzma.so.5.2.2 is in /usr/local/lib. Additionally, confirm that liblzma.so.5 and liblzma.so are symbolic links to liblzma.so.5.2.2.
# cd /usr/local/include
# ls -l lzma.h
-rw-r--r-- 1 root root 9737 Oct 10 08:18 lzma.h
# cd /usr/local/lib
# ls -l liblzma*
-rw-r--r-- 1 root root 579480 Oct 10 08:18 liblzma.a
-rwxr-xr-x 1 root root 949 Oct 10 08:18 liblzma.la
lrwxrwxrwx 1 root root 16 Oct 10 08:18 liblzma.so -> liblzma.so.5.2.2
lrwxrwxrwx 1 root root 16 Oct 10 08:18 liblzma.so.5 -> liblzma.so.5.2.2
-rwxr-xr-x 1 root root 421608 Oct 10 08:18 liblzma.so.5.2.2

Next, download gnuiconv, and transfer to the node to install.
http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.14.tar.gz

Then, specify "-m64" for the environment variable CFLAGS, and run "configure".
# cd libiconv-1.14
# export CFLAGS="-m64"
# ./configure

Without errors, run "make install".
# make install

Confirm that iconv.h is in /usr/local/include and libiconv.so.2.5.1 is in /usr/local/lib. Additionally, confirm that libiconv.so.2 and libiconv.so are symbolic links to libiconv.so.2.5.1.
# cd /usr/local/include
# ls -l iconv.h
-rw-r--r-- 1 root root 9343 Oct 10 14:54 iconv.h
# cd /usr/local/lib
# ls -l libiconv*
-rw-r--r-- 1 root root 929 Oct 10 14:54 libiconv.la
lrwxrwxrwx 1 root root 17 Oct 10 14:54 libiconv.so -> libiconv.so.2.5.1
lrwxrwxrwx 1 root root 17 Oct 10 14:54 libiconv.so.2 -> libiconv.so.2.5.1
-rwxr-xr-x 1 root root 1075256 Oct 10 14:54 libiconv.so.2.5.1

Required Software Installation: R Installation
Download R, and transfer to the node to install.
https://cran.r-project.org/src/base/R-3/R-3.2.5.tar.gz

In a working directory, extract files from archive and move to "R-3.2.5". Set environment variables for compiling R. Specify compile options for optimizing to SPARC64X processor in environment variables. Since R source files are written in C, C++ and Fortran, the compile options for compiler and linker for each language should be specified. Processor name should be specified to compiler option "-xtarget". Specify "sparc64xplus" for SPARC64X+, and "sparc64x" for SPARC64X. For more details about compiler options, refer to the Oracle Developer Studio manual.

# export CFLAGS="-m64 -xO5 -fma=fused -xlibmieee -xlibmil -xmemalign=8s -xtarget=sparc64xplus -xvector=simd,lib -xpagesize=4M"
# export CC="cc -std=c99"
# export FFLAGS="-m64 -xO5 -fma=fused -xlibmil -dalign -xtarget=sparc64xplus -xvector=simd,lib -xpagesize=4M"
# export F77="f95"
# export CXXFLAGS="-m64 -xO5 -fma=fused -xlibmil -xmemalign=8s -xtarget=sparc64xplus -xvector=simd,lib -xpagesize=4M"
# export CXX="CC -library=stlport4"
# export FC="f95"
# export FCFLAGS=$FFLAGS
# export FCLIBS=""
# export LDFLAGS="-m64 -xO5 -xmemalign=8s -xvector=simd,lib -xpagesize=4M -L/usr/local/lib/sparcv9"

Below are the environment variables and their setting for running R.
# export R_LD_LIBRARY_PATH="/usr/local/lib"
# export JAVA_HOME=/usr/java
# export LD_LIBRARY_PATH_64=/usr/local/lib:/opt/R/lib/R/lib

Below is only for compiling.
# export LD_LIBRARY_PATH_64=/usr/local/lib:'pwd'/lib

Next, run "configure" with the following options.
--prefix=/opt/R-3.2.5 Install compiled modules to "/opt/R-3.2.5".
--with-ICU=no Do not use ICU library.
--enable-R-shlib=yes Compile R library modules as a dynamic link object.
--with-blas='-library=sunperf' Use Sun Performance Library for BLAS function.
--with-lapack='-library=sunperf' Use Sun Performance Library for LAPACK function.
--with-system-xz=no Do not use OS attached lzma library.
--disable-long-double Do not use double extended precision floating point number.
# ./configure --prefix=/opt/R-3.2.5 --with-ICU=no --enable-R-shlib=yes --with-blas='-library=sunperf' --with-lapack='-library=sunperf' --with-system-xz=no --disable-long-double

Without errors, compile sources. Since there are so many files, use "gmake" with "-j" option to compile in parallel.
# gmake -j

While compiling sources, an error occurs when generating R documents due to the lack of "pdflatex" program. The message is as follows. However, since documents are not necessary in this blog, it can be ignored.
'pdflatex' is needed to make vignettes but is missing on your system.
Once compilation is complete, copy execution modules to the specified directory. Since documents are not generated in this blog, run "gmake" command with option "-i" to ignore the documents copy error.
# gmake -i install

Run "R" command with the runtime environment variables settings previously indicated. The message below indicates that it started successfully.
# R

R version 3.2.5 (2016-04-14) -- "Very, Very Secure Dishes"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: sparc-sun-solaris2.11 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

Disclaimer
The information contained in this blog is for general information purposes only. While we endeavor to keep the information up-to-date and correct through testing on a practical system, we make no warranties of any kind about the completeness, accuracy, reliability, suitability or availability. Any reliance you place on such information is strictly at your own risk. The information in this blog is subject to change without notice.

 

Shinichiro Asai

 

About the Author:

Shinichiro Asai

Technical Sales Support, Platform Solution Unit, FUJITSU Japan

SHARE

Comments on this article

No comments yet.

Please Login to leave a comment.