User Tools

Site Tools


handbook:handbook:filesystem:gpfs:the_general_parallel_file_system

Install, Create & Configure GPFS-Cluster

Install required packages:

# rpm -ivh gpfs.base-3.4.0-0.x86_64.rpm or update : rpm -Uvh gpfs.base-3.4.0-5.x86_64.update.rpm

# rpm -ivh gpfs.docs-3.4.0-5.noarch.rpm
# rpm -ivh gpfs.gpl-3.4.0-5.noarch.rpm
# rpm -ivh gpfs.msg.en_US-3.4.0-5.noarch.rpm

Verify your GPFS installation:
# rpm -qa | grep gpfs

# cd /usr/lpp/mmfs/src
# make Autoconfig
# make World
# make InstallImages
# mkdir /tmp/GPFSdump   < create a dump dir for GPFS
# mkdir /var/mmfs/config   < Create a configuration-directory for GPFS
# vi /var/mmfs/config/nodes.cluster    < Create a Node Config-File for creating a cluster

gpfsnode01:quorum
gpfsnode02:quorum
gpfsnode03
……
gpfsnodeXX

# cp /var/mmfs/config/nodes.cluster /var/mmfs/config/nodes.list  < and remove “:quorum” entries in that list

# vi /var/mmfs/config/disk.list   < Create disk config file

/dev/sdb:::dataAndMetadata:1:nsd01gpfs
/dev/sdc:::dataAndMetadata:1:nsd02gpfs
……
/dev/sdx:::dataAndMetadata:1:nsd0xgpfs
/dev/sdy:::dataAndMetadata:1:nsd0ygpfs


**Cluster with 3 nodes**

1. #mmcrcluster -N /var/mmfs/config/nodes.cluster -p gpfsnode01-gpfs -s gpfsnode02-gpfs -r /usr/bin/ssh -R /usr/bin/scp -C whatever –A

if anything goes wrong while creating the cluster, just delete whats inside /var/mmfs/gen

2. #mmchlicense server --accept -N 1,2

3. #mmchlicense client --accept -N 3

4. #mmchconfig dataStructureDump=/tmp/GPFSdump,pagepool=512M

5. #mmchconfig maxFilesToCache=4000,maxMBpS=200

6. #mmcrnsd -F /var/mmfs/config/disk.list

7. #mmchconfig tiebreakerDisks=”nsd01gpfs;nsd02gpfs”

8. #mmstartup –a

9. #mmgetstate –a

10. #mmcrfs /gpfsFS1 /dev/gpfsFS1 -F /var/mmfs/config/disk.list -A yes -B 256K -N 20000000 -v no -m 1 -M 2 -r 1 -R 2

11. #mmmount /gpfsFS1 -a


**GPFS Commands and Examples**

Startup GPFS
#mmstartup ; starts GPFS at own node
#mmstartup -n x ; starts GPFS at Clusternode Nr. x
#mmstartup -a ; starts GPFS at all defined Nodes in Cluster


Stop GPFS
#mmshutdown ; stops GPFS at own node
#mmshutdown -n x ; stops GPFS at Clusternode Nr. x
#mmshutdown -a ; stops GPFS at all defined Nodes in Cluster



#mmlscluster      <    GPFS Cluster Definition


#mmlsconfig    <  GPFS Cluster Configuration


#mmgetstate -a   < Clusterstatus

Now, pagepool is a very weird and interesting stuff when you have lots of RAM and you use big database. For example, if you have 100GB database, and you want to have very quick access to it, and we assume you have at least 128GB RAM, you should setup the pagepool to at least 120GB RAM . Pagepool is reserved memory on system, so the hole database will be cached. If you are using the cluster for BWA, stick with the above setup/config .

#mmlsconfig    <    view the configuration parameters 

#mmchconfig -i pagepool=120G    <   change the pagepool setting on all nodes whiteout bringing down the gpfs cluster ( -i ) 
handbook/handbook/filesystem/gpfs/the_general_parallel_file_system.txt · Last modified: 2011/10/07 12:38 by eduard