abstract:farber:runjobs:runjobs

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
abstract:farber:runjobs:runjobs [2017-12-05 09:20] – sraskarabstract:farber:runjobs:runjobs [2018-08-09 15:01] (current) – [What is a Job?] anita
Line 1: Line 1:
-<booktoc/> +====== Running applications on Farber ======
-====== Running applications ======+
  
-//This section uses the wiki's [[00_conventions|documentation conventions]].// +====== Introduction ======
-===== Introduction =====+
  
-The Grid Engine job scheduling system is used to manage and control the computing resources for all jobs submitted to a cluster. This includes load balancing, reconciling requests for memory and processor cores with availability of those resources, suspending and restarting jobs, and managing jobs with different priorities. Grid Engine on Farber is Univa Grid Engine but still referred to as SGE.+The Grid Engine job scheduling system is used to manage and control the resources available to computational tasks.  The job scheduler considers each job's resource requests (memory, disk space, processor cores) and executes it as those resources become available.  The order in which jobs are submitted and a //scheduling priority// also dictate how soon the job will be eligible to execute.  The job scheduler may suspend (and later restart) some jobs in order to more quickly complete jobs with higher scheduling priority.
  
-[[:general:jobsched:grid-engine:start|Grid Engine job scheduling system]] provides an excellent overview of Grid Engine which is the job scheduling system used on Farber.+Without a job scheduler, a cluster user would need to manually search for the resources required by his or her job, perhaps by randomly logging-in to nodes and checking for other users' programs already executing thereon.  The user would have to "sign-out" the nodes he or she wishes to use in order to notify the other cluster users of resource availability((Historically, this is actually how some clusters were managed!)).  A computer will perform this kind of chore more quickly and efficiently than a human can, and with far greater sophistication.
  
-In order to schedule any job (interactively or batch) on a cluster, you must set your [[general/userguide/04_compute_environ?&#using-workgroup-and-directories|workgroup]] to define your cluster group or //investing-entity// compute nodes.+An outdated but still mostly relevant description of Grid Engine and job scheduling can be found in the first chapter of the [[http://docs.oracle.com/cd/E19957-01/820-0699/chp1-1|Sun N1™ Grid Engine 6.1 User's Guide]].
  
-See [[general:/userguide:06_runtime_environ?&#scheduling-jobs|Scheduling Jobs]] and [[general:/userguide:06_runtime_environ?&#managing-jobs|Managing Jobs]] for general information about getting started with scheduling and managing jobs on a cluster using Grid Engine. +===== What is a Job? =====
  
-===== Runtime environment =====+In this context, a //job// consists of:
  
-Generally, your runtime environment (path, environment variables, etc.) should be the same as your compile-time environment. Usually, the best way to achieve this is to put the relevant VALET commands in shell scripts. You can reuse common sets of commands by storing them in a shell script file that can be //sourced //from within other shell script files.+  * a sequence of commands to be executed 
 +  * a list of resource requirements and other properties affecting scheduling of the job 
 +  * a set of environment variables
  
-<note important> +For an //[[abstract:farber:runjobs:schedule_jobs#interactive-jobs-qlogin|interactive job]]//, the user manually types the sequence of commands once the job is eligible for execution.  If the necessary resources for the job are not immediately available, then the user must wait; when resources are available, the user must be present at his/her computer in order to type the commands.  Since the job scheduler does not care about the time of day, this could happen anytime, day or night.
-If you are writing an executable script that does not have the **-l** option on the **bash** command, and you want to include VALET commands in your script, then you should include the line: +
-<code bash> +
-source /etc/profile.d/valet.sh +
-</code> +
-You do not need this command when you +
-  - type commands, or source the command file, +
-  - include lines in the file to be submitted to the qsub. +
-</note> +
-===== Job scheduling system =====+
  
-A job scheduling system is used to manage and control the computing resources for all jobs submitted to a cluster. This includes load balancing, limiting resources, reconciling requests for memory and processor cores with availability of those resources, suspending and restarting jobs, and managing jobs with different priorities.+By comparison, a //[[abstract:farber:runjobs:schedule_jobs#batch-jobs-qsub|batch job]]// does not require the user be awake and at his or her computer:  the sequence of commands is saved to a file, and that file is given to the job scheduler.  A file containing a sequence of shell commands is also known as a //script//, so in order to run batch jobs a user must become familiar with //shell scripting//.  The benefits of using batch jobs are significant:
  
-Each investing-entity's group (workgroup) has owner queues that allow the use a fixed number of slots to match the total number of cores purchased.  If a job is submitted that would use more than the slots allowed, the job will wait until enough slots are made available by completed jobs.  There is no time limit imposed on owner queue jobs.  All users can see running and waiting jobs, which allows groups to work out policies for managing purchased nodes.+  * a //job script// can be reused (versus repeatedly having to type the same sequence of commands for each job) 
 +  * when resources are granted to the job it will execute immediately (day or night), yielding increased job throughput
  
-The standby queues are available for projects requiring more slots than purchased, or to take advantage of idle nodes when a job would have to wait in the owner queue.  Other workgroup nodes will be used, so standby jobs have a time limit, and users are limited to a total number of cores for all of their standby jobs.  Generally, users can use 10 nodes for an 8 hour standby job or 40 nodes for a 4 hour standby job.+An individual's increased job throughput is good for all users of the cluster!
  
-A spillover queue may be available for the case where a job is submitted to the owner queue, and there are standby jobs consuming needed slots. Instead of waiting, the jobs will be sent to the spillover queue to start on a similar idle node.+===== Queues =====
  
-A spare queue may be on a cluster to make spare nodes available to users, by special request.+At its most basic, a //queue// represents a collection of computing entities (call them nodes) on which jobs can be executed.  Each queue has properties that restrict what jobs are eligible to execute within it:  a queue may not accept interactive jobs; a queue may place an upper limit on how long the job will be allowed to execute or how much memory it can use; or specific users may be granted or denied permission to execute jobs in a queue.
  
-Each cluster is configured with a particular job scheduling system. General documentation is available for all [[:general:start#job-scheduling-systems|job scheduling systems]] currently in use.+<note>Grid Engine uses a //cluster queue// to embody the common set of properties that define the behavior of a queue.  The cluster queue acts as a template for the //queue instances// that exist for each node that executes jobs for the queue.  The term //queue// can refer to either of these, but in this documentation it will most often imply a //cluster queue//.</note>
  
 +When submitting a job to Grid Engine, a user can explicitly specify which queue to use:  doing so will place that queue's resource restrictions (e.g. maximum execution time, maximum memory) on the job, even if they are not appropriate.  Usually it is easier if the user specifies what resources his or her job requires and lets Grid Engine choose an appropriate queue.
  
 +===== Job scheduling system =====
  
 +A job scheduling system is used to manage and control the computing resources for all jobs submitted to a cluster. This includes load balancing, limiting resources, reconciling requests for memory and processor cores with availability of those resources, suspending and restarting jobs, and managing jobs with different priorities.
  
-==== Array jobs ====+Each investing-entity's group (workgroup) has owner queues that allow the use a fixed number of slots to match the total number of cores purchased.  If a job is submitted that would use more than the slots allowed, the job will wait until enough slots are made available by completed jobs.  There is no time limit imposed on owner queue jobs.  All users can see running and waiting jobs, which allows groups to work out policies for managing purchased nodes.
  
-An [[:general:jobsched:grid-engine:35_parallelism#array-jobs|array job]] essentially runs the same job by generating a new repeated task many times. Each time, the environment variable **SGE_TASK_ID** is set to a sequence number by Grid Engine and its value provides input to the job submission script.+The standby queues are available for projects requiring more slots than purchased, or to take advantage of idle nodes when a job would have to wait in the owner queue.  Other workgroup nodes will be used, so standby jobs have a time limit, and users are limited to a total number of cores for all of their standby jobs.  Generally, users can use 10 nodes for an 8 hour standby job or 40 nodes for a 4 hour standby job.
  
-<note tip> +A spillover queue may be available for the case where a job is submitted to the owner queue, and there are standby jobs consuming needed slots. Instead of waiting, the jobs will be sent to the spillover queue to start on a similar idle node.
-The ''$SGE_TASK_ID'' is the key to make the array jobs useful.  Use it in your bash script, or pass it as a parameter so your program can decide how to complete the assigned task. +
-   +
-For example, the ''$SGE_TASK_ID'' sequence values of 2, 4, 6, ... , 5000 might be passed as an initial data value to 2500 repetitions of a simulation model. Alternatively, each iteration (task) of a job might use a different data file with filenames of ''data$SGE_TASK_ID'' (i.e., data1, data2, data3, ', data2000). +
-</note>+
  
-The general form of the **qsub** option is: 
  
--t   //start_value// - //stop_value// : //step_size// 
  
-with a default step_size of 1. For these examples, the option would be: 
  
--t 2-5000:2     and     -t 1-2000+==== Grid Engine ====
  
-Additional simple how-to examples for [[http://wiki.gridengine.info/wiki/index.php/Simple-Job-Array-Howto|array jobs]].+The Grid Engine job scheduling system is used to manage and control the computing resources for all jobs submitted to a cluster. This includes load balancing, reconciling requests for memory and processor cores with availability of those resources, suspending and restarting jobs, and managing jobs with different priorities. Grid Engine on Farber is Univa Grid Engine but still referred to as SGE.
  
-==== Chaining jobs ====+In order to schedule any job (interactively or batch) on a cluster, you must set your [[abstract/farber/system_access/system_access#logging-on-to-farber|workgroup]] to define your cluster group or //investing-entity// compute nodes.
  
-If you have a multiple jobs where you want to automatically run other job(s) after the execution of another job, then you can use chaining. When you chain jobs, remember to check the status of the other job to determine if it successfully completed. This will prevent the system from flooding the scheduler with failed jobs.  Here is a simple chaining example with three job scripts ''doThing1.qs'', ''doThing2.qs'' and ''doThing3.qs''.+See [[abstract/farber/runjobs/schedule_jobs|Scheduling Jobs]] and [[abstract/farber/runjobs/job_status|Managing Jobs]] on the <html><span style="color:#ffffff;background-color:#2fa4e7;padding:3px 7px !important;border-radius:4px;">sidebar</span></html> for general information about getting started with scheduling and managing jobs on a cluster using Grid Engine. 
  
-<code - doThing1.qs>+===== Runtime environment =====
  
-#$ -N doThing1 +Generally, your runtime environment (path, environment variables, etc.) should be the same as your compile-time environment. Usually, the best way to achieve this is to put the relevant VALET commands in shell scripts. You can reuse common sets of commands by storing them in a shell script file that can be //sourced //from within other shell script files.
-# +
-# If you want an email message to be sent to you when your job ultimately +
-# finishes, edit the -M line to have your email address and change the +
-# next two lines to start with #$ instead of just # +
-# -m eas +
-# -M my_address@mail.server.com +
-# +
-# Setup the environment; add vpkg_require commands after this +
-# line:+
  
-# Now append all of your shell commands necessary to run your program +<note important> 
-# after this line: +If you are writing an executable script that does not have the **-l** option on the **bash** command, and you want to include VALET commands in your script, then you should include the line: 
- ./dotask1+<code bash> 
 +source /etc/profile.d/valet.sh
 </code> </code>
 +You do not need this command when you
 +  - type commands, or source the command file,
 +  - include lines in the file to be submitted to the qsub.
 +</note>
  
-<code - doThing2.qs> 
  
-#$ -N doThing2 
-#$ -hold_jid doThing1 
-# 
-# If you want an email message to be sent to you when your job ultimately 
-# finishes, edit the -M line to have your email address and change the 
-# next two lines to start with #$ instead of just # 
-# -m eas 
-# -M my_address@mail.server.com 
-# 
-# Setup the environment; add vpkg_require commands after this 
-# line: 
  
-# Now append all of your shell commands necessary to run your program +===== Getting Help =====
-# after this line:+
  
-# Here is where you should add a test to make sure +Grid Engine includes man pages for all of the commands that will be reviewed in this document.  When logged-in to a cluster, type
-# that dotask1 successfully completed before running +
-# ./dotask2 +
-# You might check if a specific file(s) exists that you would +
-# expect after a successful dotask1 run, something like this +
-#  if [ -e dotask1.log ]  +
-#      then ./dotask2 +
-#  fi +
-# If dotask1.log does not exist it will do nothing. +
-# If you don't need a test, then you would run the task. +
- ./dotask2 +
-</code>+
  
-<code - doThing3.qs> +<code bash> 
- +[traine@farber ~]$ man qstat
-#$ -N doThing3 +
-#$ -hold_jid doThing2 +
-# +
-# If you want an email message to be sent to you when your job ultimately +
-# finishes, edit the -M line to have your email address and change the +
-# next two lines to start with #$ instead of just # +
-# -m eas +
-# -M my_address@mail.server.com +
-# +
-# Setup the environment; add vpkg_require commands after this +
-# line: +
- +
-# Now append all of your shell commands necessary to run your program +
-# after this line: +
-# Here is where you should add a test to make sure +
-# that dotask2 successfully completed before running +
-# ./dotask3 +
-# You might check if a specific file(s) exists that you would +
-# expect after a successful dotask2 run, something like this +
-#  if [ -e dotask2.log ]  +
-#      then ./dotask3 +
-#  fi +
-# If dotask2.log does not exist it will do nothing. +
-# If you don't need a test, then just run the task. +
- ./dotask3+
 </code> </code>
  
-Now submit all three job scripts. In this example, we are using account ''traine'' in workgroup ''it_css'' on Mills.+to learn more about a Grid Engine command (in this case, ''qstat'').  Most commands will also respond to the ''-help'' command-line option to provide a succinct usage summary:
  
-<code> +<code base> 
-[(it_css:traine)@mills ~]$ qsub doThing1.qs +[traine@farber ~]$ qstat -help 
-[(it_css:traine)@mills ~]$ qsub doThing2.qs +usage: qstat [options] 
-[(it_css:traine)@mills ~]$ qsub doThing3.qs+        [-cb]                             view additional binding specific parameters 
 +        [-ext]                            view additional attributes 
 +           :
 </code> </code>
  
-The basic flow is ''doThing2'' will wait until ''doThing1'' finishes, and ''doThing3'' will wait until ''doThing2'' finishes.  If you test for success, then ''doThing2'' will check to make sure that ''doThing1'' was successful before running, and ''doThing3'' will check to make sure that ''doThing2'' was successful before running. +//This section uses the wiki's [[http://docs-dev.hpc.udel.edu/doku.php#documentation-conventions|documentation conventions]].//
- +
-You might also want to have ''doThing1'' and ''doThing2'' execute at the same time, and only run ''doThing3'' after they finish.  In this case you will need to change ''doThing2'' and ''doThing3'' scripts and tests. +
- +
-<code - doThing2.qs> +
- +
-#$ -N doThing2 +
-# +
-# If you want an email message to be sent to you when your job ultimately +
-# finishes, edit the -M line to have your email address and change the +
-# next two lines to start with #$ instead of just # +
-# -m eas +
-# -M my_address@mail.server.com +
-# +
-# Setup the environment; add vpkg_require commands after this +
-# line: +
- +
-# Now append all of your shell commands necessary to run your program +
-# after this line: +
- ./dotask2 +
-</code> +
- +
-<code - doThing3.qs> +
- +
-#$ -N doThing3 +
-#$ -hold_jid doThing1,doThing2 +
-# +
-# If you want an email message to be sent to you when your job ultimately +
-# finishes, edit the -M line to have your email address and change the +
-# next two lines to start with #$ instead of just # +
-# -m eas +
-# -M my_address@mail.server.com +
-# +
-# Setup the environment; add vpkg_require commands after this +
-# line: +
- +
-# Now append all of your shell commands necessary to run your program +
-# after this line: +
-# Here is where you should add a test to make sure +
-# that dotask1 and dotask2 successfully completed before running +
-# ./dotask3 +
-# You might check if a specific file(s) exists that you would +
-# expect after a successful dotask1 and dotask2 run, something like this +
-#  if [ -e dotask1.log -a -e dotask2.log ]; +
-#      then ./dotask3 +
-#  fi +
-# If both files do not exist it will do nothing. +
-# If you don't need a test, then just run the task. +
- ./dotask3 +
-</code> +
- +
-Now submit all three jobs again. However this time ''doThing1'' and ''doThing2'' will run at the same time, and only when they are both finished, will ''doThing3'' run.  ''doThing3'' will check to make sure ''doThing1'' and ''doThing2'' are successful  +
-before running. +
  
  • abstract/farber/runjobs/runjobs.1512483600.txt.gz
  • Last modified: 2017-12-05 09:20
  • by sraskar