Differences

This shows you the differences between two versions of the page.

--- software:matlab [2017-09-07 16:51] – created sraskar
+++ software:matlab [2017-09-07 16:55] (current) – removed sraskar
@@ Line 1: / Line 1: @@
-====== Using Matlab ======
-MATLAB® is a high-level language and interactive environment for numerical computation, visualization, and programming. Using MATLAB, you can analyze data, develop algorithms, and create models and applications. The language, tools, and built-in math functions enable you to explore multiple approaches and reach a solution faster than with spreadsheets or traditional programming languages, such as C/C++ or Java™
-**Details by clusters**
-  * [[software:matlab:Mills]]
-  * [[software:matlab:Farber]]
-See [[software:matlab:shared-node|Computational model for running Matlab on a shared Cluster]]
-====== License information ======
-MATLAB is product licensed by Mathworks.  The MATLAB product and related toolboxes are usually installed, on campus, to use the UD license server.  When MATLAB is started a license token is checked out from the license server. Each toolbox function also requires the availability of a license token to start during a MATLAB session. The UD license server currently has tokens for [[central licenses|core MATLAB and about 40 toolboxes]].
-The Mathworks R2013b and newer license server is being queried every 6 minutes on each cluster for available seats licensed for Matlab, Simulink and associated toolboxes.  These are pushed into consumable (global, per-job) integer complexes in Grid Engine and can be checked using
-<code>
-        qhost -h global -F
-</code>
-to list number of unused license seats for each product.
-Below is an example representing a snapshot of unused licensed seats for Matlab products on the cluster.
-<code>
-	[traine@mills ~]$ qhost -h global -F
-	HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
-	-------------------------------------------------------------------------------
-	global                  -               -     -       -       -       -       -
-	    gc:MLM.Compiler=50.000000
-            gc:MLM.Aerospace_Blockset=1.000000
-            gc:MLM.RTW_Embedded_Coder=2.000000
-            gc:MLM.Robust_Toolbox=150.000000
-            gc:MLM.Aerospace_Toolbox=1.000000
-            gc:MLM.Identification_Toolbox=50.000000
-            gc:MLM.XPC_Target=2.000000
-            gc:MLM.Econometrics_Toolbox=1.000000
-            gc:MLM.Real-Time_Workshop=2.000000
-            gc:MLM.Fuzzy_Toolbox=50.000000
-            gc:MLM.Video_and_Image_Blockset=1.000000
-            gc:MLM.Neural_Network_Toolbox=50.000000
-            gc:MLM.Fin_Instruments_Toolbox=1.000000
-            gc:MLM.Optimization_Toolbox=44.000000
-            gc:MLM.MATLAB_Coder=2.000000
-            gc:MLM.MATLAB=204.000000
-            gc:MLM.Database_Toolbox=1.000000
-            gc:MLM.SIMULINK=100.000000
-            gc:MLM.PDE_Toolbox=48.000000
-            gc:MLM.GADS_Toolbox=1.000000
-            gc:MLM.Symbolic_Toolbox=46.000000
-            gc:MLM.Signal_Toolbox=146.000000
-            gc:MLM.Financial_Toolbox=1.000000
-            gc:MLM.Data_Acq_Toolbox=2.000000
-            gc:MLM.Image_Acquisition_Toolbox=1.000000
-            gc:MLM.Curve_Fitting_Toolbox=9.000000
-            gc:MLM.Image_Toolbox=143.000000
-            gc:MLM.Distrib_Computing_Toolbox=48.000000
-            gc:MLM.OPC_Toolbox=1.000000
-            gc:MLM.MPC_Toolbox=50.000000
-            gc:MLM.Virtual_Reality_Toolbox=1.000000
-            gc:MLM.Statistics_Toolbox=43.000000
-            gc:MLM.Signal_Blocks=50.000000
-            gc:MLM.Instr_Control_Toolbox=2.000000
-            gc:MLM.MAP_Toolbox=12.000000
-            gc:MLM.Communication_Toolbox=50.000000
-            gc:MLM.Control_Toolbox=150.000000
-            gc:MLM.Wavelet_Toolbox=1.000000
-            gc:MLM.Bioinformatics_Toolbox=1.000000
-	    gc:MLM.Simulink_Control_Design=50.000000
-	    gc:MLM.Real-Time_Win_Target=1.000000
-</code>
-Matlab jobs can be submitted to require a certain number of license seats to be available before a job will run. If there are inter-license dependencies for toolboxes, then you should specify all the licenses including Matlab and/or Simulink.
-For example, if a Matlab job requires the Financial toolbox, then you will also need to specify all the inter-related toolbox licenses required by the Financial toolbox such as the Statistics and Optimization toolboxes as well Matlab itself. See [[http://www.mathworks.com/products/availability|Mathworks System Requirements & Platform Availability by Product
-]] for complete details.
-<code>
-	qsub -l MLM.MATLAB=1,MLM.Financial_Toolbox=1,MLM.Statistics_Toolbox=1,MLM.Optimization_Toolbox=1 ...
-</code>
-Naturally, this isn't a to-the-moment mapping because the license server is not being queried constantly.  However, it's consumable, so it is keeping track of how many seats are unused every 6 minutes.
-This will be most helpful when submitting many Matlab jobs that require a toolbox with a low-seat count. They will wait for a toolbox seat to become available rather than trying to run and having many getting the "**License checkout failed**" message from MATLAB.
-====== Project directory ======
-MATLAB is a language for scientific computing.  A MATLAB project consists of a main program, called a MATLAB script, that when executed will read data, call MATLAB functions, and write data.  This data may be in the form of text files, image files, MATLAB save files (MAT-file), or any of many standard file formats, e.g., HDF5.  To keep your project portable you should put your scripts, functions and data in one directory.  This directory is called the //Current Folder// on the MATLAB desktop.
-MATLAB has tools to help you make your project portable, and you should use them.  For example, you should specify all files using
-[[http://www.mathworks.com/help/matlab/file-name-construction.html| MATLAB file construction functions]].
-<note tip>**MAT-file save format:**
-Your preferences at
-MATLAB -> General -> MAT-files
-may have a older MAT-file save format selected as the default.  You should check to make sure you can save large files (greater than 2GB) for all versions of MATLAB where you may write ''.mat'' files. There is another advantage to using the newest file format:
-[[http://www.mathworks.com/help/matlab/import_export/load-parts-of-variables-from-mat-files.html|You can load parts of variables from a MAT-file]].
-</note>
-When you have a MATLAB job tested and debugged in a project directory, you are ready run on a cluster.
-===== Copy the project folder =====
-Copy the project folder to a directory on the cluster.
-Use  any [[general:userguide:03_filetransfer|file transfer client]] to copy your entire project directory.
-====== Batch job======
-Here are specific details for running MATLAB as an [[general:userguide:06_runtime_environ|batch application]].  You
-should have a copy of your MATLAB [[#project directory]] on the cluster.
-<note important>**Versions of MATLAB**
-MATLAB has a new version twice a year.  It is important to keep the version you use on your desktop the same as the
-one on the cluster.  The command
-<code>
-vpkg_versions matlab
-</code>
-will show you the versions available on a cluster.  Choose the one that matches the version on your desktop.  We recommend you do not upgrade MATLAB in the middle of a project, unless there is a new feature or bug fix you need.
-</note>
-<note tip>**Two directories**
-It is frequently advisable to keep your MATLAB project clean from non-MATLAB files such as the queue
-script file and the script output file.  But you may combine them, and even use the MATLAB editor to
-create the script file and look at the output file.
-If you create the file on a PC, take care to not [[general:userguide:03_filetransfer|transfer the files]] as binary.
-When you have one combined directory do not put the ''cd'' command in the queue script; instead, you change
-to the project directory with
-''cd'' on the command line, before the ''qsub''.
-</note>
-===== Create a queue script file =====
-You should create a queue script file to submit a batch job. Start by modifying a grid-engine template file, for example, to submit a serial job on one core of a compute node, copy the serial template.
-In your copy change the commented ''vpkg_require'' command to
-require MATLAB, and then add your shell commands to the end of the file. Your copy may contain the lines:
-<code>
-# Add vpkg_require commands after this line:
-vpkg_require matlab
-# Now append all of your shell commands necessary to run your program
-# after this line:
-cd project_directory
-matlab -nodisplay -singleCompThread -r main_script
-</code>
-The ''project_directory'' should have a file named ''main_script.m'' with your script.  It could have just
-one line **''display 'Hello World'''**.
-<note tip>**Toolbox dependencies**
-You should include toolbox dependencies in your batch script.  This documents the toolboxes you
-need to run the MATLAB script, and it will avoid a failure, which will occur if the job starts with no [[#license information|licenses]] available.
-For example, to Bioinformatics toolbox only has one seat, and in addition it requires the Statistics and machine learning toolbox as well as the core MATLAB.  So add the line:
-<code>
-#$ -l MLM.MATLAB=1,MLM.Statistics_Toolbox=1,MLM.Bioinformatics_Toolbox=1
-</code>
-</note>
-For more examples, including examples that use multiple computational threads or the Parallel toolbox, see the [[#cluster details]]
-===== Submit batch job =====
-Your shell must be in a [[general:userguide:04_compute_environ?&#using-workgroup-and-directories|workgroup environment]]
-to submit any jobs.
-Use the ''qsub'' command to submit a [[general:jobsched/grid-engine/30_batch|batch job]]
-and note the ''<<JOBID>>'' that is assigned to your job.  For example, if you queue script file name is ''matlab_first.qs'',
-submit the job with:
-<code>
-qsub matlab_first.qs
-</code>
-<note important>**WARNING:  Please choose a workgroup before submitting jobs**
-This is the message you get if you are not in workgroup.  Choose a workgroup with the ''workgroup'' command.
-</note>
-<note warning>**Bash script vs queue script**
-It is true that a queue script file is (usually) a bash script, but it must be executed with the ''qsub'' command instead of the ''sh'' command.  This way the grid engine commands with be processed, and the job will be run on a compute node.
-</note>
-===== Wait for job to complete =====
-You can [[general:userguide:06_runtime_environ?&#checking-job-status|check on the status]] of you job with the ''qstat'' command.
-For example, to list the information for job ''<<JOBID>>'', type
-<code>
-qstat -j <<JOBID>>
-</code>
-For long running jobs, you could change your queue script to notify you via an e-mail message when the job is
-complete.
-===== Post process job =====
-All MATLAB output data files will be in the project directory, but the MATLAB standard output will be in
-the current directory, from which you submitted the job.  Look for a file ending in your assigned JOBID.
-====== Interactive job ======
-Here are specific details for running MATLAB as an [[general:userguide:06_runtime_environ#submitting-interactive-jobs-qlogin|interactive application]] on a compute node.  You should have a copy of your [[#MATLAB project directory]] on the cluster.
-===== Command-line =====
-You should work on a compute node when in command-line MATLAB.
-Your shell must be in a [[general:userguide:04_compute_environ?&#using-workgroup-and-directories|workgroup environment]]
-to submit a single threaded interactive job using ''qlogin''.
-<code>
-qlogin
-vpkg_require matlab
-cd project_directory
-matlab -nodesktop -singleCompThread
-</code>
-This will start a interactive command-line session in your terminal window.  When done type the ''quit'' or ''exit'' to terminated the MATLAB session and then ''exit'' to terminated the qlogin session.
-===== Desktop =====
-You should be on a compute node before you start MATLAB.
-To start a MATLAB desktop (GUI mode) on a cluster, you must be running an X11 server and you must have
-[[general:userguide:02_access| connected using
-X11 tunneling]].
-Your shell must be in a [[general:userguide:04_compute_environ?&#using-workgroup-and-directories|workgroup environment]]
-to submit a job using ''qlogin''.
-<code>
-qlogin -l exclusive=1
-vpkg_require matlab
-cd project_directory
-matlab
-</code>
-This will start a interactive DESKTOP session on you X11 screen.  When done type the ''quit'' or ''exit'' in the command window or just close the window. When back at the terminal bash prompt, type ''exit'' to terminate the qlogin session.
-See [[:software:matlab:interactive:tips|tips on starting Matlab]] in an interactive session without the desktop, including executing a script.
-====== Compiling with MATLAB ======
-We show the three most common ways to work with compilers when using MATLAB.
-  - Compiling your matlab code to run in the MCR (Matlab Compiler Runtime)
-  - Compiling your C or Fortran program to call MATLAB engine.
-  - Compiling your own function in C or Fortran to be used in a MATLAB session.
-<note>Make sure your compiler is newer than the one one required by your MATLAB version. In these examples MATLAB requires gcc 4.7 or newer. You may get the Warning:
-<code>
-Warning: You are using gcc version '4.9.3'. The version currently supported
-with MEX is '4.7.x'. For a list of currently supported compilers see:
-http://www.mathworks.com/support/compilers/current_release.
-</code>
-But the compilation completes successfully.
-</note>
-===== Compiling your MATLAB =====
-There is an example MCR project in the ''/opt/templates/'' directory for you to copy and try.  Copy on the head node and qlogin to compile with MATLAB.  Once your program is compiled you can run it interactively or in batch, without needing a MATLAB license.
-==== Copy dev-projects template ====
-On the head node
-<code>
-cp -r /opt/templates/dev-projects/MCR .
-cd MCR
-</code>
-==== Compile with make ====
-Now compile on the compute node by using
-<code>
-qlogin
-make
-</code>
-<note>Remember you must be in a workgroup before using ''qlogin''
-</note>
-Resulting output from the make command:
-<code>
-Adding package `mcr/r2014b-nojvm` to your environment
-make[1]: Entering directory `/home/work/it_css/traine/matlab/MCR'
-mcc -o maxEig -R "-nojvm,-nodesktop,-singleCompThread" -mv maxEig.m
-Compiler version: 5.2 (R2014b)
-Dependency analysis by REQUIREMENTS.
-Parsing file "/home/work/it_css/traine/matlab/MCR/maxEig.m"
-	(Referenced from: "Compiler Command Line").
-Deleting 0 temporary MEX authorization files.
-Generating file "/home/work/it_css/traine/matlab/MCR/readme.txt".
-Generating file "run_maxEig.sh".
-make[1]: Leaving directory `/home/work/it_css/traine/matlab/MCR'
-</code>
-Take note of the package added, and the files that are generated.  You can remove these files, as they are not needed.
-You must add the package in your batch script or to test interactively.
-==== test interactively ====
-To test interactively on the same compute node.
-<code>
-vpkg_require mcr/r2014b-nojvm
-time ./maxEig 20.8
-</code>
-<note tip>This example is designed as a test for batch computing, and takes about 15 minutes to complete. If you
-change the MATLAB statement dim=10000 to dim=1000, and recompile, it will take about 10 seconds</note>
-==== back to the head node ====
-When done, exit the compute node.
-<code>
-exit
-</code>
-==== Copy array job example ====
-<code>
-cp /opt/templates/gridengine/matlab-mcr.qs .
-vi matlab-mcr.qs
-diff /opt/templates/gridengine/matlab-mcr.qs matlab-mcr.qs
-</code>
-The ''diff'' output shows changes made in the ''vi'' session:
-<code>
-c46
-< # -l m_mem_free=5G
----
-> #$ -l m_mem_free=3G
-c51
-< # -t 1-4
----
-> #$ -t 1-100
-c63,64
-< vpkg_require mcr/r2014b-nojvm
----
-> vpkg_require mcr/r2015a-nojvm
-> let lambda="$SGE_TASK_ID-1"
-c80
-< MCR_EXECUTABLE_FLAGS=("$RANDOM")
----
-> MCR_EXECUTABLE_FLAGS=("$lambda")
-</code>
-To submit a standby array job that has 100 tasks.
-<code>
-qsub -l standby=1 matlab-mcr.qs
-</code>
-Example
-<code>
-[(it_css:traine)@farber MCR]$ qsub -l standby=1 matlab-mcr.qs
-Your job-array 627074.1-100:1 ("matlab-mcr.qs") has been submitted
-[(it_css:traine)@farber MCR]$ date
-Mon Apr 11 14:56:26 EDT 2016
-[(it_css:traine)@farber MCR]$ date
-Mon Apr 11 15:17:33 EDT 2016
-[(it_css:traine)@farber MCR]$ ls -l matlab-mcr.qs.o627074.* | wc -l
-</code>
-There are 100 output files with the names matlab-mcr.qs.o627074.1 to matlab-mcr.qs.o627074.100
-For example file 50:
-<code>
-[CGROUPS] UD Grid Engine cgroup setup commencing
-[CGROUPS] Setting 3221225472 bytes (vmem none bytes) on n106 (master)
-[CGROUPS]   with 1 core =
-[CGROUPS] done.
-Adding package `mcr/r2015a-nojvm` to your environment
-GridEngine parameters:
-          MCR_ROOT = /opt/shared/matlab/r2015a
-    MCR executable = /home/work/it_css/traine/matlab/MCR/maxEig
-             flags = 49
-    MCR_CACHE_ROOT = /tmp/627074.50.standby.q
--- begin maxEig run --
-maxe =
-.0243e+03
--- end maxEig run --
-</code>
-[[more examples]]
-===== Compiling your code to use MATLAB engine ======
-There is an simple example function ''**fengdemo.F**'' coded in Fortran, you can copy and use as a starting point.
-On the head node and in a workgroup shell:
-<code>
-vpkg_require matlab/r2015a gcc/4.9
-cp $MATLABROOT/extern/examples/eng_mat/fengdemo.F .
-export LD_LIBRARY_PATH=$MATLABROOT/bin/glnxa64:$MATLABROOT/sys/os/glnx64:$LD_LIBRARY_PATH
-mex -client engine fengdemo.F
-</code>
-To start MATLAB on a compute node to test this new program:
-<code>
-qlogin
-vpkg_require matlab/r2015a gcc/4.9
-export LD_LIBRARY_PATH=$MATLABROOT/bin/glnxa64:$MATLABROOT/sys/os/glnx64:$LD_LIBRARY_PATH
-./fengdemo
-exit
-</code>
-Step one of the fengdemo should give the plot:
-{{:software:figure_1.png?400|}}
-Step two should give the table:
-<code>
- MATLAB computed the following distances:
-   time(s)  distance(m)
-.00     -4.90
-.00     -19.6
-.00     -44.1
-.00     -78.4
-.00     -123.
-.00     -176.
-.00     -240.
-.00     -314.
-.00     -397.
-.0     -490.
-</code>
-===== Compiling your own MATLAB function ======
-There is an simple example function ''**timestwo.c**'', coded in c, you can copy and use as a starting point.
-On the head node and in a workgroup shell:
-<code>
-vpkg_require matlab/r2015a gcc/4.9
-cp $MATLABROOT/extern/examples/refbook/timestwo.c .
-mex timestwo.c
-</code>
-To start MATLAB on a compute node to test this new function:
-<code>
-qlogin
-vpkg_require matlab/r2015a gcc/4.9
-matlab -nodesktop
-timestwo(4)
-quit
-exit
-</code>
-You should get the answer
-<code>
->> timestwo(4)
-ans =
->>
-</code>
-====== Parallel job ======
-You can use [[http://www.mathworks.com/discovery/matlab-multicore.html|multicore or multiprocessor]] MATLAB for your project.
-There are three ways do this using the cluster scheduler and/or MATLAB toolboxes.
-=====  Array job =====
-Multiple, independent, single-threaded MATLAB scripts can be run on the same node, or different nodes. The only communication between the jobs is through reading and writing files. Your script should be compiled using ''mcc'' so it can be deployed on the cluster to run in a [[http://www.mathworks.com/products/compiler/mcr/index.html|MATLAB Runtime]] environment.
-===== Multiple computational threads =====
-MATLAB will detect you are on a multicore host and use all the cores it can to speed up its internal calculations.  Use MATLAB builtin functions to take advantage of these multiple computational threads.  You should scedule a computing node with exclusive access, so your job does not interfere with other jobs on the node.
-===== Parallel computing toolbox =====
-Using the [[http://www.mathworks.com/products/parallel-computing/| MATLAB Parallel Computing Toolbox]] you can create a pool of workers and take control of the parallelism in your algorithm using MATLAB functions such as ''parfor''.  You can run this on one node using the default local scheduler, but the real benefit comes with a separate Mathworks product called the [[http://www.mathworks.com/products/distriben/|MATLAB Distributed Computing Server]]
-====== Cluster details ======
-**Using MATLAB details by cluster**
-  * [[.matlab:Mills]]
-  * [[.matlab:Farber]]