Table of Contents

Case studies

Two Matlab jobs on same node

  1. traine: The first user starts an interactive session with qlogin. The plan is to run a Matlab script, which is expected to take about 30 minutes. Thus the Matlab environment is set via VALET and the job started using nohup and nodisplay. There is a plot produced at the end of the run, but this is exported to a file. This allows traine to exit the compute node and leave the job running.
  1. dnairn: The second user plans to start a Matlab desktop (GUI mode) and occasional run a computational intensive Matlab function in the command Window on the Matlab desktop. Normal activity in the desktop is not computationally intensive.

Both users are assigned the same compute node, n015. Since there are 24 cores on node 'n015', we may expect both jobs to run without any slowing down. There should be enough cores to handle two Matlab jobs, but his is not entirely true.

Matlab tries to run as fast as possible on the compute node, with no regard for any other users on the system. We plan to monitor Matlab useage of the 24 cores on node n015.

Reference runs

For reference, we ran this script as the only MATLAB job on the node.

Single computational thread

[traine@n015 matlab]$ nohup matlab -nodisplay -singleCompThread -r script &
[1] 6887
[traine@n015 matlab]$ nohup: ignoring input and appending output to `nohup.out'

[1]+  Done                    nohup matlab -nodisplay -singleCompThread -r script
[traine@n015 matlab]$ tail nohup.out
maxe =

   67.8668

Elapsed time is 2286.511284 seconds.

avgMaxEig =

   69.0097

Multiple computational threads (default)

[traine@n015 matlab]$ time matlab -nodisplay -r script

          ....
          
maxe =

   67.8668

Elapsed time is 1091.672589 seconds.

avgMaxEig =

   69.0097


real	18m17.526s
user	125m52.497s
sys	1m46.513s

Session one (user traine)

User traine has a Matlab script to run, which should take about 20 minutes to complete. It loops through 12 function calls, averages the results and the plots the values and average (exporting to a png file.) The Matlab tic and toc commands are used to report the Elapsed time for the loop. The script was executed using the time UNIX comand.

[[traine@n015 matlab]$ date "+%s";time matlab -nodisplay -r script > run.out
1365778449

real	19m58.116s
user	133m31.547s
sys	2m2.448s
[traine@n015 matlab]$ tail run.out

   67.8668

Elapsed time is 1190.968489 seconds.

avgMaxEig =

   69.0097

This was done concurrently with some activity from user dnairn. The Elapsed time reported by tic/toc was close the real time - 1191.00 vs 1198.11. We will take the time reported by Matlab and compare with the times reported in the interactive session. We will take the CPU time on all cores during the loop as the usage time diminished by the extra time spent outside the loop.

Session two (dnairn)

User dnairn starts an interactive GUI Matlab with the command

matlab

On the Matlab desktop dnairn, does the typical activity of looking and help files and editing a function. This function uses builtin Matlab routines and will take about 90 seconds to complete one call. Thus the activity is minutes for relatively light CPU activity with occasional heavy activity for a minute to two.

Here is a copy from the Command window of the Matlab Desktop

system('date "+%s"');maxEig(200);
1365778316

maxe =

   2.6002e+03

Elapsed time is 92.380987 seconds.

The script was started at timestamp 1365778449 by traine

>> system('date "+%s"');maxEig(201);
1365778498

maxe =

   2.6019e+03

Elapsed time is 118.775738 seconds.
>> system('date "+%s"');maxEig(202);
1365778693

maxe =

   2.6009e+03

Elapsed time is 167.737078 seconds.

Some light editing files and browsing the desktop, before the same function call was executed. The Matrix in the function is randomly generated for each call, and that is why the maxe values are slightly different.

>> system('date "+%s"');maxEig(202);
1365778991

maxe =

   2.6006e+03

Elapsed time is 102.941354 seconds.

Timing results

Job Elapsed Time CPU Time
Single comp thread 2286.5 2286.5
Exclusive access 1091.6 7603.1
Competing for FPU 1191.0 8004.4

Top sampling of all threads

$ ssh n015 top -H -b -d 1 -n 10 | egrep '(COMMAND|MATLAB)' | grep -v 'S  0'
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 100.1  1.0  12:36.69 MATLAB            
12401 traine    20   0 3554m 619m  73m S 36.6  1.0   6:56.20 MATLAB             
12403 traine    20   0 3554m 619m  73m S 36.6  1.0   6:55.71 MATLAB             
12409 traine    20   0 3554m 619m  73m S 36.6  1.0   6:55.98 MATLAB             
12402 traine    20   0 3554m 619m  73m S 34.7  1.0   6:55.83 MATLAB             
12404 traine    20   0 3554m 619m  73m S 34.7  1.0   6:55.66 MATLAB             
12405 traine    20   0 3554m 619m  73m S 34.7  1.0   6:56.07 MATLAB             
12406 traine    20   0 3554m 619m  73m S 34.7  1.0   6:55.95 MATLAB             
12407 traine    20   0 3554m 619m  73m S 34.7  1.0   6:55.80 MATLAB             
12408 traine    20   0 3554m 619m  73m S 34.7  1.0   6:55.33 MATLAB             
12410 traine    20   0 3554m 619m  73m S 34.7  1.0   6:55.69 MATLAB             
12400 traine    20   0 3554m 619m  73m S 32.7  1.0   6:55.95 MATLAB             
15054 dnairn    20   0 6521m 388m  64m S  7.7  0.6   0:03.17 MATLAB             
15059 dnairn    20   0 6521m 388m  64m S  1.9  0.6   0:00.46 MATLAB             
15063 dnairn    20   0 6521m 388m  64m S  1.9  0.6   0:07.65 MATLAB   
          
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 100.7  1.0  12:37.72 MATLAB            
12400 traine    20   0 3554m 619m  73m S 39.1  1.0   6:56.35 MATLAB             
12401 traine    20   0 3554m 619m  73m S 39.1  1.0   6:56.60 MATLAB             
12402 traine    20   0 3554m 619m  73m S 39.1  1.0   6:56.23 MATLAB             
12405 traine    20   0 3554m 619m  73m S 39.1  1.0   6:56.47 MATLAB             
12407 traine    20   0 3554m 619m  73m S 39.1  1.0   6:56.20 MATLAB             
12403 traine    20   0 3554m 619m  73m S 38.1  1.0   6:56.10 MATLAB             
12404 traine    20   0 3554m 619m  73m S 38.1  1.0   6:56.05 MATLAB             
12406 traine    20   0 3554m 619m  73m S 38.1  1.0   6:56.34 MATLAB             
12408 traine    20   0 3554m 619m  73m S 38.1  1.0   6:55.72 MATLAB             
12409 traine    20   0 3554m 619m  73m S 38.1  1.0   6:56.37 MATLAB             
12410 traine    20   0 3554m 619m  73m S 38.1  1.0   6:56.08 MATLAB             
15054 dnairn    20   0 6521m 392m  64m S 12.7  0.6   0:03.30 MATLAB             
15063 dnairn    20   0 6521m 392m  64m S  3.9  0.6   0:07.69 MATLAB 
           
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 98.8  1.0  12:38.73 MATLAB             
12406 traine    20   0 3554m 619m  73m R 39.1  1.0   6:56.74 MATLAB             
12400 traine    20   0 3554m 619m  73m R 38.1  1.0   6:56.74 MATLAB             
12404 traine    20   0 3554m 619m  73m R 38.1  1.0   6:56.44 MATLAB             
12408 traine    20   0 3554m 619m  73m R 38.1  1.0   6:56.11 MATLAB             
12409 traine    20   0 3554m 619m  73m R 38.1  1.0   6:56.76 MATLAB             
12410 traine    20   0 3554m 619m  73m R 38.1  1.0   6:56.47 MATLAB             
12401 traine    20   0 3554m 619m  73m R 37.2  1.0   6:56.98 MATLAB             
12402 traine    20   0 3554m 619m  73m R 37.2  1.0   6:56.61 MATLAB             
12403 traine    20   0 3554m 619m  73m R 37.2  1.0   6:56.48 MATLAB             
12405 traine    20   0 3554m 619m  73m R 37.2  1.0   6:56.85 MATLAB             
12407 traine    20   0 3554m 619m  73m R 37.2  1.0   6:56.58 MATLAB  
           
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 101.1  1.0  12:39.77 MATLAB            
12400 traine    20   0 3554m 619m  73m R 51.5  1.0   6:57.27 MATLAB             
12401 traine    20   0 3554m 619m  73m R 51.5  1.0   6:57.51 MATLAB             
12402 traine    20   0 3554m 619m  73m R 51.5  1.0   6:57.14 MATLAB             
12403 traine    20   0 3554m 619m  73m R 51.5  1.0   6:57.01 MATLAB             
12405 traine    20   0 3554m 619m  73m R 51.5  1.0   6:57.38 MATLAB             
12408 traine    20   0 3554m 619m  73m R 51.5  1.0   6:56.64 MATLAB             
12409 traine    20   0 3554m 619m  73m R 51.5  1.0   6:57.29 MATLAB             
12404 traine    20   0 3554m 619m  73m R 50.5  1.0   6:56.96 MATLAB             
12406 traine    20   0 3554m 619m  73m R 50.5  1.0   6:57.26 MATLAB             
12407 traine    20   0 3554m 619m  73m R 50.5  1.0   6:57.10 MATLAB             
12410 traine    20   0 3554m 619m  73m R 50.5  1.0   6:56.99 MATLAB             
15063 dnairn    20   0 6521m 392m  64m S  1.0  0.6   0:07.70 MATLAB  

User dnairn starts a computational demanding function from the GUI.

           
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 99.5  1.0  12:40.80 MATLAB             
15027 dnairn    20   0 6521m 393m  64m R 98.6  0.6   0:17.28 MATLAB             
15180 dnairn    20   0 6521m 393m  64m R 41.5  0.6   0:09.04 MATLAB             
15186 dnairn    20   0 6521m 393m  64m R 41.5  0.6   0:08.68 MATLAB             
15188 dnairn    20   0 6521m 393m  64m R 41.5  0.6   0:08.87 MATLAB             
15189 dnairn    20   0 6521m 393m  64m R 41.5  0.6   0:08.49 MATLAB             
15182 dnairn    20   0 6521m 393m  64m R 40.6  0.6   0:08.91 MATLAB             
15183 dnairn    20   0 6521m 393m  64m R 40.6  0.6   0:08.55 MATLAB             
15187 dnairn    20   0 6521m 393m  64m R 40.6  0.6   0:08.73 MATLAB             
15190 dnairn    20   0 6521m 393m  64m R 40.6  0.6   0:08.46 MATLAB             
15181 dnairn    20   0 6521m 393m  64m R 39.6  0.6   0:09.24 MATLAB             
15184 dnairn    20   0 6521m 393m  64m R 39.6  0.6   0:09.04 MATLAB             
15185 dnairn    20   0 6521m 393m  64m R 38.6  0.6   0:08.47 MATLAB             
12401 traine    20   0 3554m 619m  73m R 30.0  1.0   6:57.82 MATLAB             
12403 traine    20   0 3554m 619m  73m R 29.0  1.0   6:57.31 MATLAB             
12406 traine    20   0 3554m 619m  73m R 29.0  1.0   6:57.56 MATLAB             
12408 traine    20   0 3554m 619m  73m R 29.0  1.0   6:56.94 MATLAB             
12400 traine    20   0 3554m 619m  73m R 28.0  1.0   6:57.56 MATLAB             
12407 traine    20   0 3554m 619m  73m S 28.0  1.0   6:57.39 MATLAB             
12409 traine    20   0 3554m 619m  73m S 28.0  1.0   6:57.58 MATLAB             
12404 traine    20   0 3554m 619m  73m R 24.2  1.0   6:57.21 MATLAB             
12402 traine    20   0 3554m 619m  73m R 23.2  1.0   6:57.38 MATLAB             
12405 traine    20   0 3554m 619m  73m R 23.2  1.0   6:57.62 MATLAB             
12410 traine    20   0 3554m 619m  73m R 22.2  1.0   6:57.22 MATLAB             
15156 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15157 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15158 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15159 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15160 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15162 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15163 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15164 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15165 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15168 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15171 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15172 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15173 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15174 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15175 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15177 dnairn    20   0 6521m 393m  64m S  1.9  0.6   0:00.02 MATLAB             
15059 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.47 MATLAB             
15063 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:07.71 MATLAB             
15161 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.01 MATLAB             
15166 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.01 MATLAB             
15167 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.01 MATLAB             
15169 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.01 MATLAB             
15170 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.01 MATLAB             
15176 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.01 MATLAB             
15178 dnairn    20   0 6521m 393m  64m S  1.0  0.6   0:00.01 MATLAB
            
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 100.7  1.0  12:41.84 MATLAB            
15027 dnairn    20   0 6521m 393m  64m R 100.7  0.6   0:18.32 MATLAB            
12400 traine    20   0 3554m 619m  73m R 35.8  1.0   6:57.93 MATLAB             
15181 dnairn    20   0 6521m 393m  64m R 34.9  0.6   0:09.60 MATLAB             
15184 dnairn    20   0 6521m 393m  64m R 34.9  0.6   0:09.40 MATLAB             
12401 traine    20   0 3554m 619m  73m R 33.9  1.0   6:58.17 MATLAB             
12403 traine    20   0 3554m 619m  73m R 33.9  1.0   6:57.66 MATLAB             
12404 traine    20   0 3554m 619m  73m R 33.9  1.0   6:57.56 MATLAB             
12406 traine    20   0 3554m 619m  73m R 33.9  1.0   6:57.91 MATLAB             
12408 traine    20   0 3554m 619m  73m R 33.9  1.0   6:57.29 MATLAB             
12410 traine    20   0 3554m 619m  73m R 33.9  1.0   6:57.57 MATLAB             
15180 dnairn    20   0 6521m 393m  64m R 33.9  0.6   0:09.39 MATLAB             
15186 dnairn    20   0 6521m 393m  64m R 33.9  0.6   0:09.03 MATLAB             
15188 dnairn    20   0 6521m 393m  64m R 33.9  0.6   0:09.22 MATLAB             
12407 traine    20   0 3554m 619m  73m R 32.9  1.0   6:57.73 MATLAB             
12409 traine    20   0 3554m 619m  73m R 32.9  1.0   6:57.92 MATLAB             
15187 dnairn    20   0 6521m 393m  64m R 32.9  0.6   0:09.07 MATLAB             
12405 traine    20   0 3554m 619m  73m R 31.9  1.0   6:57.95 MATLAB             
15182 dnairn    20   0 6521m 393m  64m R 31.9  0.6   0:09.24 MATLAB             
15185 dnairn    20   0 6521m 393m  64m R 31.9  0.6   0:08.80 MATLAB             
15183 dnairn    20   0 6521m 393m  64m R 31.0  0.6   0:08.87 MATLAB             
12402 traine    20   0 3554m 619m  73m R 30.0  1.0   6:57.69 MATLAB             
15189 dnairn    20   0 6521m 393m  64m R 23.2  0.6   0:08.73 MATLAB             
15190 dnairn    20   0 6521m 393m  64m R 20.3  0.6   0:08.67 MATLAB  
            
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 99.1  1.0  12:42.88 MATLAB             
15027 dnairn    20   0 6521m 392m  64m R 99.1  0.6   0:19.36 MATLAB             
15185 dnairn    20   0 6521m 392m  64m S 83.9  0.6   0:09.68 MATLAB             
15180 dnairn    20   0 6521m 392m  64m S 82.9  0.6   0:10.26 MATLAB             
15181 dnairn    20   0 6521m 392m  64m S 82.9  0.6   0:10.47 MATLAB             
15184 dnairn    20   0 6521m 392m  64m S 82.9  0.6   0:10.27 MATLAB             
15186 dnairn    20   0 6521m 392m  64m S 82.9  0.6   0:09.90 MATLAB             
15188 dnairn    20   0 6521m 392m  64m S 82.9  0.6   0:10.09 MATLAB             
15189 dnairn    20   0 6521m 392m  64m S 82.9  0.6   0:09.60 MATLAB             
15190 dnairn    20   0 6521m 392m  64m S 82.9  0.6   0:09.54 MATLAB             
15187 dnairn    20   0 6521m 392m  64m S 77.2  0.6   0:09.88 MATLAB             
15182 dnairn    20   0 6521m 392m  64m S 76.2  0.6   0:10.04 MATLAB             
15183 dnairn    20   0 6521m 392m  64m S 76.2  0.6   0:09.67 MATLAB             
12403 traine    20   0 3554m 619m  73m R 50.5  1.0   6:58.19 MATLAB             
12401 traine    20   0 3554m 619m  73m R 49.6  1.0   6:58.69 MATLAB             
12405 traine    20   0 3554m 619m  73m R 49.6  1.0   6:58.47 MATLAB             
12407 traine    20   0 3554m 619m  73m R 49.6  1.0   6:58.25 MATLAB             
12408 traine    20   0 3554m 619m  73m R 49.6  1.0   6:57.81 MATLAB             
12400 traine    20   0 3554m 619m  73m R 48.6  1.0   6:58.44 MATLAB             
12406 traine    20   0 3554m 619m  73m R 48.6  1.0   6:58.42 MATLAB             
12409 traine    20   0 3554m 619m  73m R 48.6  1.0   6:58.43 MATLAB             
12410 traine    20   0 3554m 619m  73m R 48.6  1.0   6:58.08 MATLAB             
12404 traine    20   0 3554m 619m  73m R 37.2  1.0   6:57.95 MATLAB             
12402 traine    20   0 3554m 619m  73m R 36.2  1.0   6:58.07 MATLAB  
            
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 100.0  1.0  12:43.91 MATLAB            
15027 dnairn    20   0 6521m 392m  64m R 100.0  0.6   0:20.39 MATLAB            
15187 dnairn    20   0 6521m 392m  64m R 77.7  0.6   0:10.68 MATLAB             
15180 dnairn    20   0 6521m 392m  64m R 76.7  0.6   0:11.05 MATLAB             
15181 dnairn    20   0 6521m 392m  64m R 76.7  0.6   0:11.26 MATLAB             
15182 dnairn    20   0 6521m 392m  64m R 76.7  0.6   0:10.83 MATLAB             
15184 dnairn    20   0 6521m 392m  64m R 76.7  0.6   0:11.06 MATLAB             
15188 dnairn    20   0 6521m 392m  64m R 76.7  0.6   0:10.88 MATLAB             
15189 dnairn    20   0 6521m 392m  64m R 76.7  0.6   0:10.39 MATLAB             
15185 dnairn    20   0 6521m 392m  64m R 75.8  0.6   0:10.46 MATLAB             
15186 dnairn    20   0 6521m 392m  64m R 75.8  0.6   0:10.68 MATLAB             
15190 dnairn    20   0 6521m 392m  64m R 74.8  0.6   0:10.31 MATLAB             
15183 dnairn    20   0 6521m 392m  64m R 71.9  0.6   0:10.41 MATLAB             
12406 traine    20   0 3554m 619m  73m R 41.8  1.0   6:58.85 MATLAB             
12410 traine    20   0 3554m 619m  73m R 41.8  1.0   6:58.51 MATLAB             
12405 traine    20   0 3554m 619m  73m R 40.8  1.0   6:58.89 MATLAB             
12407 traine    20   0 3554m 619m  73m R 40.8  1.0   6:58.67 MATLAB             
12408 traine    20   0 3554m 619m  73m R 40.8  1.0   6:58.23 MATLAB             
12409 traine    20   0 3554m 619m  73m R 40.8  1.0   6:58.85 MATLAB             
12401 traine    20   0 3554m 619m  73m R 39.8  1.0   6:59.10 MATLAB             
12403 traine    20   0 3554m 619m  73m R 39.8  1.0   6:58.60 MATLAB             
12404 traine    20   0 3554m 619m  73m R 38.9  1.0   6:58.35 MATLAB             
12402 traine    20   0 3554m 619m  73m R 29.1  1.0   6:58.37 MATLAB             
12400 traine    20   0 3554m 619m  73m R 26.2  1.0   6:58.71 MATLAB    
          
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 100.6  1.0  12:44.96 MATLAB            
15027 dnairn    20   0 6521m 392m  64m R 99.6  0.6   0:21.43 MATLAB             
15181 dnairn    20   0 6521m 392m  64m R 78.5  0.6   0:12.08 MATLAB             
15184 dnairn    20   0 6521m 392m  64m R 77.6  0.6   0:11.87 MATLAB             
15189 dnairn    20   0 6521m 392m  64m R 77.6  0.6   0:11.20 MATLAB             
15182 dnairn    20   0 6521m 392m  64m R 76.6  0.6   0:11.63 MATLAB             
15186 dnairn    20   0 6521m 392m  64m R 76.6  0.6   0:11.48 MATLAB             
15188 dnairn    20   0 6521m 392m  64m R 76.6  0.6   0:11.68 MATLAB             
15190 dnairn    20   0 6521m 392m  64m R 76.6  0.6   0:11.11 MATLAB             
15180 dnairn    20   0 6521m 392m  64m R 74.7  0.6   0:11.83 MATLAB             
15183 dnairn    20   0 6521m 392m  64m R 73.8  0.6   0:11.18 MATLAB             
15185 dnairn    20   0 6521m 392m  64m R 72.8  0.6   0:11.22 MATLAB             
15187 dnairn    20   0 6521m 392m  64m R 70.9  0.6   0:11.42 MATLAB             
12401 traine    20   0 3554m 619m  73m R 50.8  1.0   6:59.63 MATLAB             
12407 traine    20   0 3554m 619m  73m R 50.8  1.0   6:59.20 MATLAB             
12410 traine    20   0 3554m 619m  73m R 50.8  1.0   6:59.04 MATLAB             
12403 traine    20   0 3554m 619m  73m R 49.8  1.0   6:59.12 MATLAB             
12405 traine    20   0 3554m 619m  73m R 49.8  1.0   6:59.41 MATLAB             
12406 traine    20   0 3554m 619m  73m R 49.8  1.0   6:59.37 MATLAB             
12408 traine    20   0 3554m 619m  73m R 49.8  1.0   6:58.75 MATLAB             
12409 traine    20   0 3554m 619m  73m R 49.8  1.0   6:59.37 MATLAB             
12404 traine    20   0 3554m 619m  73m R 46.0  1.0   6:58.83 MATLAB             
12400 traine    20   0 3554m 619m  73m R 41.2  1.0   6:59.14 MATLAB             
12402 traine    20   0 3554m 619m  73m R 39.3  1.0   6:58.78 MATLAB    
          
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
12323 traine    20   0 3554m 619m  73m R 98.6  1.0  12:46.00 MATLAB             
15027 dnairn    20   0 6521m 392m  64m R 97.7  0.6   0:22.46 MATLAB             
15182 dnairn    20   0 6521m 392m  64m R 95.8  0.6   0:12.64 MATLAB             
15180 dnairn    20   0 6521m 392m  64m R 94.8  0.6   0:12.83 MATLAB             
15184 dnairn    20   0 6521m 392m  64m R 94.8  0.6   0:12.87 MATLAB             
15188 dnairn    20   0 6521m 392m  64m R 94.8  0.6   0:12.68 MATLAB             
15189 dnairn    20   0 6521m 392m  64m R 94.8  0.6   0:12.20 MATLAB             
15190 dnairn    20   0 6521m 392m  64m R 94.8  0.6   0:12.11 MATLAB             
15181 dnairn    20   0 6521m 392m  64m R 93.9  0.6   0:13.07 MATLAB             
15186 dnairn    20   0 6521m 392m  64m R 93.9  0.6   0:12.47 MATLAB             
15187 dnairn    20   0 6521m 392m  64m R 92.9  0.6   0:12.40 MATLAB             
15183 dnairn    20   0 6521m 392m  64m R 86.3  0.6   0:12.09 MATLAB             
15185 dnairn    20   0 6521m 392m  64m R 82.5  0.6   0:12.09 MATLAB             
12405 traine    20   0 3554m 619m  73m S 39.8  1.0   6:59.83 MATLAB             
12406 traine    20   0 3554m 619m  73m S 39.8  1.0   6:59.79 MATLAB             
12408 traine    20   0 3554m 619m  73m S 39.8  1.0   6:59.17 MATLAB             
12409 traine    20   0 3554m 619m  73m S 39.8  1.0   6:59.79 MATLAB             
12401 traine    20   0 3554m 619m  73m S 38.9  1.0   7:00.04 MATLAB             
12403 traine    20   0 3554m 619m  73m S 38.9  1.0   6:59.53 MATLAB             
12404 traine    20   0 3554m 619m  73m S 38.9  1.0   6:59.24 MATLAB             
12407 traine    20   0 3554m 619m  73m S 38.9  1.0   6:59.61 MATLAB             
12410 traine    20   0 3554m 619m  73m S 38.9  1.0   6:59.45 MATLAB             
12402 traine    20   0 3554m 619m  73m S 33.2  1.0   6:59.13 MATLAB             
12400 traine    20   0 3554m 619m  73m S 31.3  1.0   6:59.47 MATLAB  

Top sampling with threads combined

This time the the top is done without the -H option. Thus the %CPU is reported for all threads and can be more than 100% when the threads are on different cores.

$ ssh n015 top  -b -d 1 -n 60 | egrep '(COMMAND|MATLAB)' | grep -v 'S  0'
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1193.5  1.0  31:47.28 MATLAB 
          
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1133.1  1.0  31:58.90 MATLAB           
14985 dnairn    20   0 6616m 449m  64m S  1.0  0.7   9:56.75 MATLAB  
                      
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1183.7  1.0  32:11.03 MATLAB           
14985 dnairn    20   0 6616m 449m  64m S  1.0  0.7   9:56.76 MATLAB    
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1140.7  1.0  32:22.72 MATLAB           
14985 dnairn    20   0 6616m 449m  64m S  1.0  0.7   9:56.77 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1160.9  1.0  32:34.61 MATLAB           
14985 dnairn    20   0 6616m 449m  64m S  1.0  0.7   9:56.78 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1174.0  1.0  32:46.65 MATLAB           
14985 dnairn    20   0 6616m 449m  64m S  1.0  0.7   9:56.79 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1186.6  1.0  32:58.81 MATLAB           
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1163.8  1.0  33:10.74 MATLAB           
14985 dnairn    20   0 6616m 449m  64m S  1.0  0.7   9:56.80 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1204.3  1.0  33:23.09 MATLAB           
14985 dnairn    20   0 6807m 467m  64m S 13.7  0.7   9:56.94 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 1113.1  1.0  33:34.90 MATLAB           
14985 dnairn    20   0 6998m 831m  64m S 288.4  1.3  10:00.00 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3300m 428m  73m S 400.4  0.7  33:39.06 MATLAB            
14985 dnairn    20   0 6998m 831m  64m S 100.1  1.3  10:01.04 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 435.1  1.3  10:05.51 MATLAB            
16367 traine    20   0 3491m 620m  73m S 99.3  1.0  33:40.08 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 366.2  1.3  10:09.24 MATLAB            
16367 traine    20   0 3491m 620m  73m S 254.2  1.0  33:42.67 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 340.0  1.3  10:12.73 MATLAB            
16367 traine    20   0 3491m 620m  73m S 323.5  1.0  33:45.99 MATLAB   
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 398.3  1.0  33:50.05 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 343.3  1.3  10:16.23 MATLAB   
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 356.1  1.3  10:19.86 MATLAB            
16367 traine    20   0 3491m 620m  73m S 340.4  1.0  33:53.52 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 524.8  1.0  33:58.87 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.1  1.3  10:20.88 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 443.4  1.3  10:25.41 MATLAB            
16367 traine    20   0 3491m 620m  73m S 241.8  1.0  34:01.34 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 202.2  1.0  34:03.44 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 98.2  1.3  10:26.43 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 376.9  1.3  10:30.27 MATLAB            
16367 traine    20   0 3491m 620m  73m S 334.7  1.0  34:06.85 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 387.2  1.0  34:10.79 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 285.0  1.3  10:33.17 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 405.6  1.0  34:14.95 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 120.9  1.3  10:34.41 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 525.7  1.0  34:20.30 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 498.2  1.3  10:39.48 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 419.5  1.0  34:24.57 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.2  1.3  10:40.50 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 470.6  1.0  34:29.36 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 428.4  1.3  10:44.86 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 431.3  1.3  10:49.25 MATLAB            
16367 traine    20   0 3491m 620m  73m S 425.4  1.0  34:33.69 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 349.7  1.0  34:37.25 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.2  1.3  10:50.27 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 405.8  1.3  10:54.40 MATLAB            
16367 traine    20   0 3491m 620m  73m S 339.0  1.0  34:40.70 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 450.7  1.0  34:45.32 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 392.1  1.3  10:58.42 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 426.8  1.0  34:49.74 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 168.0  1.3  11:00.16 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 400.0  1.0  34:53.84 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 303.4  1.3  11:03.27 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 439.3  1.0  34:58.34 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 418.8  1.3  11:07.56 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 411.0  1.0  35:02.55 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 99.6  1.3  11:08.58 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 359.7  1.0  35:06.21 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 275.2  1.3  11:11.38 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 449.0  1.0  35:10.78 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 367.4  1.3  11:15.12 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 505.0  1.0  35:15.92 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.2  1.3  11:16.14 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 429.3  1.3  11:20.51 MATLAB            
16367 traine    20   0 3491m 620m  73m S 420.4  1.0  35:20.20 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 401.8  1.0  35:24.29 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.2  1.3  11:21.53 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 406.7  1.0  35:28.43 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 276.1  1.3  11:24.34 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 456.8  1.0  35:33.08 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.2  1.3  11:25.36 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 351.7  1.3  11:28.94 MATLAB            
16367 traine    20   0 3491m 620m  73m S 256.4  1.0  35:35.69 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 345.8  1.0  35:39.21 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.2  1.3  11:29.96 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 390.0  1.0  35:43.18 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 251.5  1.3  11:32.52 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 287.1  1.0  35:46.10 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 206.5  1.3  11:34.62 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 355.1  1.0  35:49.74 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 286.8  1.3  11:37.56 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 388.4  1.0  35:53.69 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 355.0  1.3  11:41.17 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 461.1  1.0  35:58.45 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 99.8  1.3  11:42.20 MATLAB             
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 418.5  1.0  36:02.71 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 347.8  1.3  11:45.74 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 350.7  1.0  36:06.28 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.2  1.3  11:46.76 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 510.9  1.0  36:11.48 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 360.6  1.3  11:50.43 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 509.9  1.0  36:16.67 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 102.2  1.3  11:51.47 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 586.7  1.0  36:22.71 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 319.6  1.3  11:54.76 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 410.4  1.0  36:26.94 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 340.5  1.3  11:58.27 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 388.9  1.0  36:30.94 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.1  1.3  11:59.30 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14985 dnairn    20   0 6998m 865m  64m S 441.2  1.3  12:03.84 MATLAB            
16367 traine    20   0 3491m 620m  73m S 414.9  1.0  36:35.21 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 363.6  1.0  36:38.95 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 100.1  1.3  12:04.87 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 475.6  1.0  36:43.81 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 348.4  1.3  12:08.43 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 455.2  1.0  36:48.48 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 368.5  1.3  12:12.21 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 282.2  1.0  36:51.37 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 155.3  1.3  12:13.80 MATLAB            
                    
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
16367 traine    20   0 3491m 620m  73m S 541.1  1.0  36:56.96 MATLAB            
14985 dnairn    20   0 6998m 865m  64m S 371.7  1.3  12:17.64 MATLAB