High Performance Computing on Hydra

University Computing Systems

High Performance Computing on Hydra

General Information

Applications
Disk Quotas
Job Queueing
Libraries
Manual Pages
Printing

Mailing List

ganglia

Notice on SGE 5.3 -> 6.1 Upgrade

General Information

Specifications
The cluster management software in use is warewulf, which provides a framework for managing clusters. A single image is created with only the essential components needed for operation. This image is stored on the master node and each slave node pulls a copy of this image at boot time, making each slave node essentially diskless. After each node boots up, a RAM disk stores an image of the OS, which is a small 45MB.
Logins to hydra are via AFS username and password. However, users logging into hydra will be placed into their local hydra home directory ( /home/users/{faculty, students}/<ucid> ), rather than their AFS home directory.
Users can use klog to authenticate to AFS, and thus access anything in AFS for which they have the proper permission.
Direct connections to hydra are made using ssh only, and must be done from within the NJIT network. If a connection to hydra must be made from outside of the NJIT network, then a VPN client must be used, or a connection to one of the public AFS systems afs<n>.njit.edu must be used as a jump-in point to hydra. Information on obtaining and using the VPN client can be found at : http://telecom.njit.edu/vpnc/index.html
All local software applications that installed by UCS are placed in /export/apps and associated wrapper scripts are placed in /export/site/bin . The MPI/Myrinet libraries as well as other specialized libraries used in computational runs are installed in /opt .
Detailed Documentation

Applications

The following applications are available :

IMSL 6.0

/opt/vni/CTT6.0/
/opt/vni/CTT6.0/lib/lib.lopt64

Matlab 14.3

/opt/site/bin/matlab

Matlab 14.1

/opt/site/bin/matlab -v matlab-14.1

Matlab versions available

/opt/site/bin/matlab -a

Mathematica 5.1

/opt/site/bin/mathematica

NAMD 2.6

/opt/site/bin/namd2

The following compilers are available :

GNU C, C++, Fortran 77; Version : 3.4.6-3.1; Located in /usr/bin
PGI (Portland Group, Inc) C, C++, Fortran 77, Fortran 90; Version : 5.2 and 6.2

Disk Quotas

All user accounts on hydra are assigned a 5GB disk quota.

The quota command (/usr/bin/quota) can be used to check the user's available disk quota.

	/usr/bin/quota 

	Disk quotas for user andym (uid 43407): 
 	    Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
 	      /dev/sda 4327164  5000000 5000000          123996       0       0

The above shows quota details for user andym. The blocks column shows the amount of disk space in use (approximately 4.3GB). The first quota column shows the total quota available for user andym. The files column shows the number of files in andym’s home directory.

Faculty who require quotas larger than the default 5GB should send email to sys@oak.njit.edu requesting the amount of disk space that is needed. Students who require a quota increase should contact their faculty advisor and have them send email to sys@oak.njit.edu on their behalf to request the needed quota.

Job Queueing

All jobs, both serial and parallel, must be submitted to a job queue. The queuing system in use on hydra is Sun Grid Engine (SGE). For detailed information on how to submit jobs and interact with SGE, see the Sun Grid Engine page : http://web.njit.edu/all_topics/HPC/sge.html

Libraries

All package libraries are under /opt.
As of 16 February 2007 :

/opt/blas/gnu3/lib
/opt/blas/pgi/lib

/opt/lapack/gnu3/lib
/opt/lapack/pgi/lib

/opt/mpi/tcp/lam-gnu3-7.1.2/lib
/opt/mpi/tcp/mpich-gnu3-1.2.7/lib

/opt/mpi/myrinet/mpichgm-pgi-1.2.7..15/lib
/opt/mpi/myrinet/mpichgm-gnu3-1.2.7..15/lib

/opt/sge/lib

/opt/pgi/linux86-64/6.2/lib
/opt/pgi/linux86-64/6.2/jre/lib
/opt/pgi/linux86-64/5.2/lib
/opt/pgi/linux86-64/5.2/jre/lib

/opt/atlas/Linux_HAMMER64SSE2_2/lib

/opt/goto/lib

/opt/vni/CTT6.0/lib

/opt/local/acml2.5.1/gnu64/lib
/opt/local/acml2.5.1/pgi64/lib
/opt/local/acml2.5.1/pgi64_mp/lib

/opt/local/acml2.5.3/gnu64/lib
/opt/local/acml2.5.3/pgi64/lib
/opt/local/acml2.5.3/pgi64_mp/lib

/opt/local/acml2.6.0/gnu32/lib
/opt/local/acml2.6.0/gnu32_nosse/lib
/opt/local/acml2.6.0/gnu32_nosse2/lib
/opt/local/acml2.6.0/gnu64/lib

/opt/local/acml2.6.0/pgi32/lib
/opt/local/acml2.6.0/pgi32_mp/lib
/opt/local/acml2.6.0/pgi32_nosse/lib
/opt/local/acml2.6.0/pgi32_nosse2/lib
/opt/local/acml2.6.0/pgi64/lib
/opt/local/acml2.6.0/pgi64_mp/lib

/opt/local/lib

Manual Pages

All packages have man pages. The availablity of man pages is dependent on the correspding module having being loaded. Modules loaded by default are :

mpi
mpichgm-pgi
pgi62
sge

To access man pages of packages not loaded by default :

module load <package>

Printing

There are three printers defined in hydra's print queue :

math615bw - HP Laserjet 4050, located in Cullimore 615 (default)
math615co - HP Color Laser 4500, located in Cullimore 615
math511bw - HP Laserjet 2300, located in Cullimore 512

A user may print directly to their office printer which is locally connected to their office workstation, providing that their system has been setup to accept print requests from remote systems. A user may request that their system be configured to accept remote print jobs by sending mail to sys@oak.njit.edu

Once a user's workstation is properly configured to accept remote print requests, jobs may be printed to the user's locally attached printer by using the lprw script.

This script requires that two environment variables be set :

RPRINTER - set this to the remote print queue
RHOST - set this to the remote host that the print queue defined in RPRINTER is located on.

Example usage of lprw :

	setenv RPRINTER office
	setenv RHOST	remote
	lpwr file.ps

The above example prints the file "file.ps" on the office print queue, which is located on the machine remote.

Mailing List

The hydra-users mailing list, hydra-users@listserver.njit.edu, is used for communication amongst hydra users and administrators.

List archives are available. To access them, you must log into the hydra-users list, at http://listserver.njit.edu/mailman/options/hydra-users. To do this, you need your list password. To get that, at the URL above, enter your email address, then scroll to "Remind" at the bottom of the page and select that. Your list password will then be mailed to you. Use that password at http://listserver.njit.edu/mailman/options/hydra-users, then go to http://listserver.njit.edu/mailman/listinfo/hydra-users to access the archives.