====== Case studies ======
===== Two Matlab jobs on same node =====
- **''traine''**: The first user starts an interactive session with ''qlogin''. The plan is to run a Matlab script, which is expected to take about 30 minutes. Thus the Matlab environment is set via VALET and the job started using nohup and nodisplay. There is a plot produced at the end of the run, but this is exported to a file. This allows ''traine'' to exit the compute node and leave the job running.
- **''dnairn''**: The second user plans to start a Matlab desktop (GUI mode) and occasional run a computational intensive Matlab function in the command Window on the Matlab desktop. Normal activity in the desktop is not computationally intensive.
Both users are assigned the same compute node, ''n015''. Since there are 24 cores on node 'n015', we may expect both jobs to run without any slowing down. There should be enough cores to handle two Matlab jobs, but his is **not** entirely true.
Matlab tries to run as fast as possible on the compute node, with no regard for any other users on the system. We plan to monitor Matlab useage of the 24 cores on node ''n015''.
==== Reference runs ====
For reference, we ran this script as the only MATLAB job on the node.
== ==
**Single computational thread**
[traine@n015 matlab]$ nohup matlab -nodisplay -singleCompThread -r script &
[1] 6887
[traine@n015 matlab]$ nohup: ignoring input and appending output to `nohup.out'
[1]+ Done nohup matlab -nodisplay -singleCompThread -r script
[traine@n015 matlab]$ tail nohup.out
maxe =
67.8668
Elapsed time is 2286.511284 seconds.
avgMaxEig =
69.0097
**Multiple computational threads (default)**
[traine@n015 matlab]$ time matlab -nodisplay -r script
....
maxe =
67.8668
Elapsed time is 1091.672589 seconds.
avgMaxEig =
69.0097
real 18m17.526s
user 125m52.497s
sys 1m46.513s
==== Session one (user traine) ====
== ==
User ''traine'' has a Matlab script to run, which should take about 20 minutes to complete. It loops through 12 function calls, averages the results and the plots the values and average (exporting to a png file.) The Matlab ''tic'' and ''toc'' commands are used to report the Elapsed time for the loop. The script was executed using the ''time'' UNIX comand.
[[traine@n015 matlab]$ date "+%s";time matlab -nodisplay -r script > run.out
1365778449
real 19m58.116s
user 133m31.547s
sys 2m2.448s
[traine@n015 matlab]$ tail run.out
67.8668
Elapsed time is 1190.968489 seconds.
avgMaxEig =
69.0097
This was done concurrently with some activity from user ''dnairn''.
The Elapsed time reported by ''tic/toc'' was close the real time - 1191.00 vs 1198.11. We will take the time reported by Matlab and compare with the times reported in the interactive session. We will take the CPU time on all cores during the loop as the usage time diminished by the extra time spent outside the loop.
==== Session two (dnairn) ====
== ==
User ''dnairn'' starts an interactive GUI Matlab with the command
matlab
On the Matlab desktop dnairn, does the typical activity of looking and help files and editing a function. This function uses builtin Matlab routines and will take about 90 seconds to complete one call. Thus the activity is minutes for relatively light CPU activity with occasional heavy activity for a minute to two.
Here is a copy from the Command window of the Matlab Desktop
system('date "+%s"');maxEig(200);
1365778316
maxe =
2.6002e+03
Elapsed time is 92.380987 seconds.
The script was started at timestamp 1365778449 by ''traine''
>> system('date "+%s"');maxEig(201);
1365778498
maxe =
2.6019e+03
Elapsed time is 118.775738 seconds.
>> system('date "+%s"');maxEig(202);
1365778693
maxe =
2.6009e+03
Elapsed time is 167.737078 seconds.
Some light editing files and browsing the desktop, before the same function call was executed. The Matrix in the function is randomly generated for each call, and that is why the ''maxe'' values are slightly different.
>> system('date "+%s"');maxEig(202);
1365778991
maxe =
2.6006e+03
Elapsed time is 102.941354 seconds.
==== Timing results ====
^ Job ^ Elapsed Time^ CPU Time^
| Single comp thread | 2286.5 | 2286.5 |
| Exclusive access | 1091.6 | 7603.1 |
| Competing for FPU | 1191.0 | 8004.4 |
==== Top sampling of all threads ====
$ ssh n015 top -H -b -d 1 -n 10 | egrep '(COMMAND|MATLAB)' | grep -v 'S 0'
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 100.1 1.0 12:36.69 MATLAB
12401 traine 20 0 3554m 619m 73m S 36.6 1.0 6:56.20 MATLAB
12403 traine 20 0 3554m 619m 73m S 36.6 1.0 6:55.71 MATLAB
12409 traine 20 0 3554m 619m 73m S 36.6 1.0 6:55.98 MATLAB
12402 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.83 MATLAB
12404 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.66 MATLAB
12405 traine 20 0 3554m 619m 73m S 34.7 1.0 6:56.07 MATLAB
12406 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.95 MATLAB
12407 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.80 MATLAB
12408 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.33 MATLAB
12410 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.69 MATLAB
12400 traine 20 0 3554m 619m 73m S 32.7 1.0 6:55.95 MATLAB
15054 dnairn 20 0 6521m 388m 64m S 7.7 0.6 0:03.17 MATLAB
15059 dnairn 20 0 6521m 388m 64m S 1.9 0.6 0:00.46 MATLAB
15063 dnairn 20 0 6521m 388m 64m S 1.9 0.6 0:07.65 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 100.7 1.0 12:37.72 MATLAB
12400 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.35 MATLAB
12401 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.60 MATLAB
12402 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.23 MATLAB
12405 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.47 MATLAB
12407 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.20 MATLAB
12403 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.10 MATLAB
12404 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.05 MATLAB
12406 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.34 MATLAB
12408 traine 20 0 3554m 619m 73m S 38.1 1.0 6:55.72 MATLAB
12409 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.37 MATLAB
12410 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.08 MATLAB
15054 dnairn 20 0 6521m 392m 64m S 12.7 0.6 0:03.30 MATLAB
15063 dnairn 20 0 6521m 392m 64m S 3.9 0.6 0:07.69 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 98.8 1.0 12:38.73 MATLAB
12406 traine 20 0 3554m 619m 73m R 39.1 1.0 6:56.74 MATLAB
12400 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.74 MATLAB
12404 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.44 MATLAB
12408 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.11 MATLAB
12409 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.76 MATLAB
12410 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.47 MATLAB
12401 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.98 MATLAB
12402 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.61 MATLAB
12403 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.48 MATLAB
12405 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.85 MATLAB
12407 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.58 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 101.1 1.0 12:39.77 MATLAB
12400 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.27 MATLAB
12401 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.51 MATLAB
12402 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.14 MATLAB
12403 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.01 MATLAB
12405 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.38 MATLAB
12408 traine 20 0 3554m 619m 73m R 51.5 1.0 6:56.64 MATLAB
12409 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.29 MATLAB
12404 traine 20 0 3554m 619m 73m R 50.5 1.0 6:56.96 MATLAB
12406 traine 20 0 3554m 619m 73m R 50.5 1.0 6:57.26 MATLAB
12407 traine 20 0 3554m 619m 73m R 50.5 1.0 6:57.10 MATLAB
12410 traine 20 0 3554m 619m 73m R 50.5 1.0 6:56.99 MATLAB
15063 dnairn 20 0 6521m 392m 64m S 1.0 0.6 0:07.70 MATLAB
User dnairn starts a computational demanding function from the GUI.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 99.5 1.0 12:40.80 MATLAB
15027 dnairn 20 0 6521m 393m 64m R 98.6 0.6 0:17.28 MATLAB
15180 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:09.04 MATLAB
15186 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:08.68 MATLAB
15188 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:08.87 MATLAB
15189 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:08.49 MATLAB
15182 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.91 MATLAB
15183 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.55 MATLAB
15187 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.73 MATLAB
15190 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.46 MATLAB
15181 dnairn 20 0 6521m 393m 64m R 39.6 0.6 0:09.24 MATLAB
15184 dnairn 20 0 6521m 393m 64m R 39.6 0.6 0:09.04 MATLAB
15185 dnairn 20 0 6521m 393m 64m R 38.6 0.6 0:08.47 MATLAB
12401 traine 20 0 3554m 619m 73m R 30.0 1.0 6:57.82 MATLAB
12403 traine 20 0 3554m 619m 73m R 29.0 1.0 6:57.31 MATLAB
12406 traine 20 0 3554m 619m 73m R 29.0 1.0 6:57.56 MATLAB
12408 traine 20 0 3554m 619m 73m R 29.0 1.0 6:56.94 MATLAB
12400 traine 20 0 3554m 619m 73m R 28.0 1.0 6:57.56 MATLAB
12407 traine 20 0 3554m 619m 73m S 28.0 1.0 6:57.39 MATLAB
12409 traine 20 0 3554m 619m 73m S 28.0 1.0 6:57.58 MATLAB
12404 traine 20 0 3554m 619m 73m R 24.2 1.0 6:57.21 MATLAB
12402 traine 20 0 3554m 619m 73m R 23.2 1.0 6:57.38 MATLAB
12405 traine 20 0 3554m 619m 73m R 23.2 1.0 6:57.62 MATLAB
12410 traine 20 0 3554m 619m 73m R 22.2 1.0 6:57.22 MATLAB
15156 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15157 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15158 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15159 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15160 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15162 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15163 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15164 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15165 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15168 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15171 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15172 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15173 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15174 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15175 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15177 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB
15059 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.47 MATLAB
15063 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:07.71 MATLAB
15161 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
15166 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
15167 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
15169 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
15170 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
15176 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
15178 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 100.7 1.0 12:41.84 MATLAB
15027 dnairn 20 0 6521m 393m 64m R 100.7 0.6 0:18.32 MATLAB
12400 traine 20 0 3554m 619m 73m R 35.8 1.0 6:57.93 MATLAB
15181 dnairn 20 0 6521m 393m 64m R 34.9 0.6 0:09.60 MATLAB
15184 dnairn 20 0 6521m 393m 64m R 34.9 0.6 0:09.40 MATLAB
12401 traine 20 0 3554m 619m 73m R 33.9 1.0 6:58.17 MATLAB
12403 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.66 MATLAB
12404 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.56 MATLAB
12406 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.91 MATLAB
12408 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.29 MATLAB
12410 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.57 MATLAB
15180 dnairn 20 0 6521m 393m 64m R 33.9 0.6 0:09.39 MATLAB
15186 dnairn 20 0 6521m 393m 64m R 33.9 0.6 0:09.03 MATLAB
15188 dnairn 20 0 6521m 393m 64m R 33.9 0.6 0:09.22 MATLAB
12407 traine 20 0 3554m 619m 73m R 32.9 1.0 6:57.73 MATLAB
12409 traine 20 0 3554m 619m 73m R 32.9 1.0 6:57.92 MATLAB
15187 dnairn 20 0 6521m 393m 64m R 32.9 0.6 0:09.07 MATLAB
12405 traine 20 0 3554m 619m 73m R 31.9 1.0 6:57.95 MATLAB
15182 dnairn 20 0 6521m 393m 64m R 31.9 0.6 0:09.24 MATLAB
15185 dnairn 20 0 6521m 393m 64m R 31.9 0.6 0:08.80 MATLAB
15183 dnairn 20 0 6521m 393m 64m R 31.0 0.6 0:08.87 MATLAB
12402 traine 20 0 3554m 619m 73m R 30.0 1.0 6:57.69 MATLAB
15189 dnairn 20 0 6521m 393m 64m R 23.2 0.6 0:08.73 MATLAB
15190 dnairn 20 0 6521m 393m 64m R 20.3 0.6 0:08.67 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 99.1 1.0 12:42.88 MATLAB
15027 dnairn 20 0 6521m 392m 64m R 99.1 0.6 0:19.36 MATLAB
15185 dnairn 20 0 6521m 392m 64m S 83.9 0.6 0:09.68 MATLAB
15180 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.26 MATLAB
15181 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.47 MATLAB
15184 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.27 MATLAB
15186 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:09.90 MATLAB
15188 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.09 MATLAB
15189 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:09.60 MATLAB
15190 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:09.54 MATLAB
15187 dnairn 20 0 6521m 392m 64m S 77.2 0.6 0:09.88 MATLAB
15182 dnairn 20 0 6521m 392m 64m S 76.2 0.6 0:10.04 MATLAB
15183 dnairn 20 0 6521m 392m 64m S 76.2 0.6 0:09.67 MATLAB
12403 traine 20 0 3554m 619m 73m R 50.5 1.0 6:58.19 MATLAB
12401 traine 20 0 3554m 619m 73m R 49.6 1.0 6:58.69 MATLAB
12405 traine 20 0 3554m 619m 73m R 49.6 1.0 6:58.47 MATLAB
12407 traine 20 0 3554m 619m 73m R 49.6 1.0 6:58.25 MATLAB
12408 traine 20 0 3554m 619m 73m R 49.6 1.0 6:57.81 MATLAB
12400 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.44 MATLAB
12406 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.42 MATLAB
12409 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.43 MATLAB
12410 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.08 MATLAB
12404 traine 20 0 3554m 619m 73m R 37.2 1.0 6:57.95 MATLAB
12402 traine 20 0 3554m 619m 73m R 36.2 1.0 6:58.07 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 100.0 1.0 12:43.91 MATLAB
15027 dnairn 20 0 6521m 392m 64m R 100.0 0.6 0:20.39 MATLAB
15187 dnairn 20 0 6521m 392m 64m R 77.7 0.6 0:10.68 MATLAB
15180 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:11.05 MATLAB
15181 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:11.26 MATLAB
15182 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:10.83 MATLAB
15184 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:11.06 MATLAB
15188 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:10.88 MATLAB
15189 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:10.39 MATLAB
15185 dnairn 20 0 6521m 392m 64m R 75.8 0.6 0:10.46 MATLAB
15186 dnairn 20 0 6521m 392m 64m R 75.8 0.6 0:10.68 MATLAB
15190 dnairn 20 0 6521m 392m 64m R 74.8 0.6 0:10.31 MATLAB
15183 dnairn 20 0 6521m 392m 64m R 71.9 0.6 0:10.41 MATLAB
12406 traine 20 0 3554m 619m 73m R 41.8 1.0 6:58.85 MATLAB
12410 traine 20 0 3554m 619m 73m R 41.8 1.0 6:58.51 MATLAB
12405 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.89 MATLAB
12407 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.67 MATLAB
12408 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.23 MATLAB
12409 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.85 MATLAB
12401 traine 20 0 3554m 619m 73m R 39.8 1.0 6:59.10 MATLAB
12403 traine 20 0 3554m 619m 73m R 39.8 1.0 6:58.60 MATLAB
12404 traine 20 0 3554m 619m 73m R 38.9 1.0 6:58.35 MATLAB
12402 traine 20 0 3554m 619m 73m R 29.1 1.0 6:58.37 MATLAB
12400 traine 20 0 3554m 619m 73m R 26.2 1.0 6:58.71 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 100.6 1.0 12:44.96 MATLAB
15027 dnairn 20 0 6521m 392m 64m R 99.6 0.6 0:21.43 MATLAB
15181 dnairn 20 0 6521m 392m 64m R 78.5 0.6 0:12.08 MATLAB
15184 dnairn 20 0 6521m 392m 64m R 77.6 0.6 0:11.87 MATLAB
15189 dnairn 20 0 6521m 392m 64m R 77.6 0.6 0:11.20 MATLAB
15182 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.63 MATLAB
15186 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.48 MATLAB
15188 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.68 MATLAB
15190 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.11 MATLAB
15180 dnairn 20 0 6521m 392m 64m R 74.7 0.6 0:11.83 MATLAB
15183 dnairn 20 0 6521m 392m 64m R 73.8 0.6 0:11.18 MATLAB
15185 dnairn 20 0 6521m 392m 64m R 72.8 0.6 0:11.22 MATLAB
15187 dnairn 20 0 6521m 392m 64m R 70.9 0.6 0:11.42 MATLAB
12401 traine 20 0 3554m 619m 73m R 50.8 1.0 6:59.63 MATLAB
12407 traine 20 0 3554m 619m 73m R 50.8 1.0 6:59.20 MATLAB
12410 traine 20 0 3554m 619m 73m R 50.8 1.0 6:59.04 MATLAB
12403 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.12 MATLAB
12405 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.41 MATLAB
12406 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.37 MATLAB
12408 traine 20 0 3554m 619m 73m R 49.8 1.0 6:58.75 MATLAB
12409 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.37 MATLAB
12404 traine 20 0 3554m 619m 73m R 46.0 1.0 6:58.83 MATLAB
12400 traine 20 0 3554m 619m 73m R 41.2 1.0 6:59.14 MATLAB
12402 traine 20 0 3554m 619m 73m R 39.3 1.0 6:58.78 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12323 traine 20 0 3554m 619m 73m R 98.6 1.0 12:46.00 MATLAB
15027 dnairn 20 0 6521m 392m 64m R 97.7 0.6 0:22.46 MATLAB
15182 dnairn 20 0 6521m 392m 64m R 95.8 0.6 0:12.64 MATLAB
15180 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.83 MATLAB
15184 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.87 MATLAB
15188 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.68 MATLAB
15189 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.20 MATLAB
15190 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.11 MATLAB
15181 dnairn 20 0 6521m 392m 64m R 93.9 0.6 0:13.07 MATLAB
15186 dnairn 20 0 6521m 392m 64m R 93.9 0.6 0:12.47 MATLAB
15187 dnairn 20 0 6521m 392m 64m R 92.9 0.6 0:12.40 MATLAB
15183 dnairn 20 0 6521m 392m 64m R 86.3 0.6 0:12.09 MATLAB
15185 dnairn 20 0 6521m 392m 64m R 82.5 0.6 0:12.09 MATLAB
12405 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.83 MATLAB
12406 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.79 MATLAB
12408 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.17 MATLAB
12409 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.79 MATLAB
12401 traine 20 0 3554m 619m 73m S 38.9 1.0 7:00.04 MATLAB
12403 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.53 MATLAB
12404 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.24 MATLAB
12407 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.61 MATLAB
12410 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.45 MATLAB
12402 traine 20 0 3554m 619m 73m S 33.2 1.0 6:59.13 MATLAB
12400 traine 20 0 3554m 619m 73m S 31.3 1.0 6:59.47 MATLAB
==== Top sampling with threads combined ====
This time the the top is done without the -H option. Thus the %CPU is reported for all threads and can be more than 100% when the threads are on different cores.
$ ssh n015 top -b -d 1 -n 60 | egrep '(COMMAND|MATLAB)' | grep -v 'S 0'
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1193.5 1.0 31:47.28 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1133.1 1.0 31:58.90 MATLAB
14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.75 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1183.7 1.0 32:11.03 MATLAB
14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.76 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1140.7 1.0 32:22.72 MATLAB
14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.77 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1160.9 1.0 32:34.61 MATLAB
14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.78 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1174.0 1.0 32:46.65 MATLAB
14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.79 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1186.6 1.0 32:58.81 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1163.8 1.0 33:10.74 MATLAB
14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.80 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1204.3 1.0 33:23.09 MATLAB
14985 dnairn 20 0 6807m 467m 64m S 13.7 0.7 9:56.94 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 1113.1 1.0 33:34.90 MATLAB
14985 dnairn 20 0 6998m 831m 64m S 288.4 1.3 10:00.00 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3300m 428m 73m S 400.4 0.7 33:39.06 MATLAB
14985 dnairn 20 0 6998m 831m 64m S 100.1 1.3 10:01.04 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 435.1 1.3 10:05.51 MATLAB
16367 traine 20 0 3491m 620m 73m S 99.3 1.0 33:40.08 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 366.2 1.3 10:09.24 MATLAB
16367 traine 20 0 3491m 620m 73m S 254.2 1.0 33:42.67 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 340.0 1.3 10:12.73 MATLAB
16367 traine 20 0 3491m 620m 73m S 323.5 1.0 33:45.99 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 398.3 1.0 33:50.05 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 343.3 1.3 10:16.23 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 356.1 1.3 10:19.86 MATLAB
16367 traine 20 0 3491m 620m 73m S 340.4 1.0 33:53.52 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 524.8 1.0 33:58.87 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.1 1.3 10:20.88 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 443.4 1.3 10:25.41 MATLAB
16367 traine 20 0 3491m 620m 73m S 241.8 1.0 34:01.34 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 202.2 1.0 34:03.44 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 98.2 1.3 10:26.43 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 376.9 1.3 10:30.27 MATLAB
16367 traine 20 0 3491m 620m 73m S 334.7 1.0 34:06.85 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 387.2 1.0 34:10.79 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 285.0 1.3 10:33.17 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 405.6 1.0 34:14.95 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 120.9 1.3 10:34.41 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 525.7 1.0 34:20.30 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 498.2 1.3 10:39.48 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 419.5 1.0 34:24.57 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 10:40.50 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 470.6 1.0 34:29.36 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 428.4 1.3 10:44.86 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 431.3 1.3 10:49.25 MATLAB
16367 traine 20 0 3491m 620m 73m S 425.4 1.0 34:33.69 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 349.7 1.0 34:37.25 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 10:50.27 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 405.8 1.3 10:54.40 MATLAB
16367 traine 20 0 3491m 620m 73m S 339.0 1.0 34:40.70 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 450.7 1.0 34:45.32 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 392.1 1.3 10:58.42 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 426.8 1.0 34:49.74 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 168.0 1.3 11:00.16 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 400.0 1.0 34:53.84 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 303.4 1.3 11:03.27 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 439.3 1.0 34:58.34 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 418.8 1.3 11:07.56 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 411.0 1.0 35:02.55 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 99.6 1.3 11:08.58 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 359.7 1.0 35:06.21 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 275.2 1.3 11:11.38 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 449.0 1.0 35:10.78 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 367.4 1.3 11:15.12 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 505.0 1.0 35:15.92 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:16.14 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 429.3 1.3 11:20.51 MATLAB
16367 traine 20 0 3491m 620m 73m S 420.4 1.0 35:20.20 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 401.8 1.0 35:24.29 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:21.53 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 406.7 1.0 35:28.43 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 276.1 1.3 11:24.34 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 456.8 1.0 35:33.08 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:25.36 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 351.7 1.3 11:28.94 MATLAB
16367 traine 20 0 3491m 620m 73m S 256.4 1.0 35:35.69 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 345.8 1.0 35:39.21 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:29.96 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 390.0 1.0 35:43.18 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 251.5 1.3 11:32.52 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 287.1 1.0 35:46.10 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 206.5 1.3 11:34.62 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 355.1 1.0 35:49.74 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 286.8 1.3 11:37.56 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 388.4 1.0 35:53.69 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 355.0 1.3 11:41.17 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 461.1 1.0 35:58.45 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 99.8 1.3 11:42.20 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 418.5 1.0 36:02.71 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 347.8 1.3 11:45.74 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 350.7 1.0 36:06.28 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:46.76 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 510.9 1.0 36:11.48 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 360.6 1.3 11:50.43 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 509.9 1.0 36:16.67 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 102.2 1.3 11:51.47 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 586.7 1.0 36:22.71 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 319.6 1.3 11:54.76 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 410.4 1.0 36:26.94 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 340.5 1.3 11:58.27 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 388.9 1.0 36:30.94 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.1 1.3 11:59.30 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14985 dnairn 20 0 6998m 865m 64m S 441.2 1.3 12:03.84 MATLAB
16367 traine 20 0 3491m 620m 73m S 414.9 1.0 36:35.21 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 363.6 1.0 36:38.95 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 100.1 1.3 12:04.87 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 475.6 1.0 36:43.81 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 348.4 1.3 12:08.43 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 455.2 1.0 36:48.48 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 368.5 1.3 12:12.21 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 282.2 1.0 36:51.37 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 155.3 1.3 12:13.80 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16367 traine 20 0 3491m 620m 73m S 541.1 1.0 36:56.96 MATLAB
14985 dnairn 20 0 6998m 865m 64m S 371.7 1.3 12:17.64 MATLAB