Using Grid Engine on Mills

The Grid Engine job scheduling system is used to manage and control the computing resources for all jobs submitted to a cluster. This includes load balancing, reconciling requests for memory and processor cores with availability of those resources, suspending and restarting jobs, and managing jobs with different priorities. Grid Engine is also known as Oracle/Sun Grid Engine or SGE.

Grid Engine job scheduling system provides an excellent overview of Grid Engine which is the job schedule system used on Mills.

In order to schedule any job (interactively or batch) on a cluster, you must set your workgroup to define your cluster group or investing-entity compute nodes.

See Scheduling Jobs and Managing Jobs for general information about getting started with scheduling and managing jobs on a cluster using Grid Engine.

Each investing-entity on a cluster has four owner queues that exclusively use the investing-entity's compute nodes. (They do not use any nodes belonging to others.) Grid Engine allows those queues to be selected only by members of the investing-entity's group.

There are also node-wise queues, standby, standby-4h, spillover-24core, spillover-48core and idle. Grid Engine allows users to use nodes belonging to other investing-entities. (The idle queue is currently disabled.)

When submitting a batch job to Grid Engine, you specify the resources you need or want for your job. You don't actually specify the name of the queue. Instead, you include a set of directives that specify your job's characteristics. Grid Engine then chooses the most appropriate queue that meets those needs.

The queue to which a job is assigned depends primarily on six factors:

• Whether the job is serial or parallel
• Which parallel environment (e.g., openmpi, threads) is needed
• Which or how much of a resource is needed (e.g., max clock time, max memory)
• Whether the job can be suspended and restarted by the system.
• Whether the job is non-interactive or interactive
• Whether you want to use idle nodes belonging to others.

For each investing-entity, the owner-queue names start with the investing-entity's name:

You may give a resource request list in the form -l resource=value. A list of available resources with their associated valid value specifiers can be obtained by the command:

qconf -sc

Each named complex or shortcut can be a resource. There can be multiple, comma separated, resource=value pairs. The valid values are determined by the type. Examples, MEMORY type could be 5G (5 GigaBytes), or a TIME type could be 1:30:00 (1 hour 30 minutes).

In a cluster as large a Mills, the two most important resources are cores (CPUs) and memory. The number of cores is called slots. It is listed as a "requestable" and "consumable" resource. Parallel jobs, by definition, will use multiple cores. Thus, the slots resource is handled by the parallel environment option -pe, and you do not need to put it in a resource list.

There are several complexes relating to memory and you will be concerned about how much is free. Memory resources come as both consumable and sensor driven (not consumable). For example:

It is usually a good idea to add both resources. The mem_free complex is sensor driven, and is more reliable for choosing a node for your job. The ram_free is consumable, which means you are reserving the memory for future use. Other jobs, using ram_free, may be barred from starting on the node. If you are specifying memory resources for a parallel environment job, the requested memory is multiplied by the slot count.

Consider 30 serial jobs, which each require 20 Gbytes of memory. Use the command

qsub -l mem_free=20G,ram_free=20G -t 1-30 myjob.qs

This will submit 30 jobs to the queue, with the SGE_TASK_ID variable set for use in the myjobs.qs script (an array job.) The mem_free resource will cause Grid Engine to find a node (or wait until one is available) with 20 Gbytes free. The ram_free resource will tell Grid Engine not to schedule too many jobs on the same node. Without the ram_free resource, an available node, which has all 64G free, will be used to start 24 jobs from the 30. This will clearly not work, since only three 20G jobs can run on one node with 64G of memory. With the ram_free resource, 3 jobs will be started on the first available node. Some of your jobs may have to wait until for earlier jobs to complete, but this is better then all jobs being memory starved.

The ram_free complex works best if everyone in your group uses it to schedule all jobs, but even if others in your group do not properly reserve memory with ram_free, you can use it to spread your large memory jobs to multiple nodes.

The /opt/templates/gridengine directory contains basic prototype job scripts for non-interactive parallel jobs. This section describes the –pe parallel environment option that's required for MPI jobs, openMP jobs and other jobs that use the SMP (threads) programming model.

Type the command:

qconf –spl

to display a list of parallel environments available on a cluster.

The general form of the parallel environment option is:

-pe «parallel_environment» «Nproc»

where «Nproc» is the number of processor slots (cores) requested. Just use a single number, and not a range. Grid Engine tries to locate as many free slots as it can and assigns them to that batch job. The environment variable $NSLOTS is given that value.

The two most used parallel environments are threads and openmpi.

Jobs such as those having openMP directives use the threads parallel environment, an implementation of the shared-memory programming model. These SMP jobs can only use the cores on a single node.

For example, if your group only owns nodes with 24 cores, then your –pe threads request may only ask for 24 or fewer slots. Use Grid Engine's qconf command to determine the names and characteristics of the queues and compute nodes available to your investing-entity group on a cluster.

Threaded jobs do not necessarily complete faster when more slots are made available. Before running a series of production runs, you should experiment to determine how many slots generally perform best. Using that quantity will leave the remaining slots for others in your group to request. Remember: others can see how many slots you're using!

For openMP jobs, add the following bash command to your job script:

export OMP_NUM_THREADS=$NSLOTS

MPI jobs inherently generate considerable network traffic among the processor cores of a cluster's compute nodes. The processors on the compute node may be connected by two types of networks: InfiniBand and Gigabit Ethernet.

IT has developed templates to help with the openmpi parallel environments for a given cluster, targeting different user needs and architecture. You can copy the templates from /opt/templates/gridengine/openmpi and customize them. These templates are essentially identical with the exception of the presence or absence of certain qsub options and the values assigned to MPI_FLAGS based on using particular environment variables. In all cases, the parallel environment option must be specified:

-pe openmpi «NPROC»

where <NPROC> is the number of processor slots (cores) requested. Use a single number, not a range. Grid Engine tries to locate as many free slots as it can and assigns them to that job. The environment variable NSLOTS is given that value.

Using the exclusive access resource option -l exclusive=1 will block any other jobs from making use of resources on that host.

Using the standby resource option -l standby=1 will target the standby queues for your job.