Case studies on Mills
Two Matlab jobs on same node
traine: The first user starts an interactive session withqlogin. The plan is to run a Matlab script, which is expected to take about 30 minutes. Thus the Matlab environment is set via VALET and the job started using nohup and nodisplay. There is a plot produced at the end of the run, but this is exported to a file. This allowstraineto exit the compute node and leave the job running.
dnairn: The second user plans to start a Matlab desktop (GUI mode) and occasional run a computational intensive Matlab function in the command Window on the Matlab desktop. Normal activity in the desktop is not computationally intensive.
Both users are assigned the same compute node, n015. Since there are 24 cores on node 'n015', we may expect both jobs to run without any slowing down. There should be enough cores to handle two Matlab jobs, but his is not entirely true.
Matlab tries to run as fast as possible on the compute node, with no regard for any other users on the system. We plan to monitor Matlab useage of the 24 cores on node n015.
Reference runs
For reference, we ran this script as the only MATLAB job on the node.
Single computational thread
[traine@n015 matlab]$ nohup matlab -nodisplay -singleCompThread -r script & [1] 6887 [traine@n015 matlab]$ nohup: ignoring input and appending output to `nohup.out' [1]+ Done nohup matlab -nodisplay -singleCompThread -r script [traine@n015 matlab]$ tail nohup.out maxe = 67.8668 Elapsed time is 2286.511284 seconds. avgMaxEig = 69.0097
Multiple computational threads (default)
[traine@n015 matlab]$ time matlab -nodisplay -r script
....
maxe =
67.8668
Elapsed time is 1091.672589 seconds.
avgMaxEig =
69.0097
real 18m17.526s
user 125m52.497s
sys 1m46.513s
Session one (user traine)
User traine has a Matlab script to run, which should take about 20 minutes to complete. It loops through 12 function calls, averages the results and the plots the values and average (exporting to a png file.) The Matlab tic and toc commands are used to report the Elapsed time for the loop. The script was executed using the time UNIX comand.
[[traine@n015 matlab]$ date "+%s";time matlab -nodisplay -r script > run.out 1365778449 real 19m58.116s user 133m31.547s sys 2m2.448s [traine@n015 matlab]$ tail run.out 67.8668 Elapsed time is 1190.968489 seconds. avgMaxEig = 69.0097
This was done concurrently with some activity from user dnairn.
The Elapsed time reported by tic/toc was close the real time - 1191.00 vs 1198.11. We will take the time reported by Matlab and compare with the times reported in the interactive session. We will take the CPU time on all cores during the loop as the usage time diminished by the extra time spent outside the loop.
Session two (dnairn)
User dnairn starts an interactive GUI Matlab with the command
matlab
On the Matlab desktop dnairn, does the typical activity of looking and help files and editing a function. This function uses builtin Matlab routines and will take about 90 seconds to complete one call. Thus the activity is minutes for relatively light CPU activity with occasional heavy activity for a minute to two.
Here is a copy from the Command window of the Matlab Desktop
system('date "+%s"');maxEig(200);
1365778316
maxe =
2.6002e+03
Elapsed time is 92.380987 seconds.
The script was started at timestamp 1365778449 by traine
>> system('date "+%s"');maxEig(201);
1365778498
maxe =
2.6019e+03
Elapsed time is 118.775738 seconds.
>> system('date "+%s"');maxEig(202);
1365778693
maxe =
2.6009e+03
Elapsed time is 167.737078 seconds.
Some light editing files and browsing the desktop, before the same function call was executed. The Matrix in the function is randomly generated for each call, and that is why the maxe values are slightly different.
>> system('date "+%s"');maxEig(202);
1365778991
maxe =
2.6006e+03
Elapsed time is 102.941354 seconds.
Timing results
| Job | Elapsed Time | CPU Time |
|---|---|---|
| Single comp thread | 2286.5 | 2286.5 |
| Exclusive access | 1091.6 | 7603.1 |
| Competing for FPU | 1191.0 | 8004.4 |
Top sampling of all threads
$ ssh n015 top -H -b -d 1 -n 10 | egrep '(COMMAND|MATLAB)' | grep -v 'S 0' PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 100.1 1.0 12:36.69 MATLAB 12401 traine 20 0 3554m 619m 73m S 36.6 1.0 6:56.20 MATLAB 12403 traine 20 0 3554m 619m 73m S 36.6 1.0 6:55.71 MATLAB 12409 traine 20 0 3554m 619m 73m S 36.6 1.0 6:55.98 MATLAB 12402 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.83 MATLAB 12404 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.66 MATLAB 12405 traine 20 0 3554m 619m 73m S 34.7 1.0 6:56.07 MATLAB 12406 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.95 MATLAB 12407 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.80 MATLAB 12408 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.33 MATLAB 12410 traine 20 0 3554m 619m 73m S 34.7 1.0 6:55.69 MATLAB 12400 traine 20 0 3554m 619m 73m S 32.7 1.0 6:55.95 MATLAB 15054 dnairn 20 0 6521m 388m 64m S 7.7 0.6 0:03.17 MATLAB 15059 dnairn 20 0 6521m 388m 64m S 1.9 0.6 0:00.46 MATLAB 15063 dnairn 20 0 6521m 388m 64m S 1.9 0.6 0:07.65 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 100.7 1.0 12:37.72 MATLAB 12400 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.35 MATLAB 12401 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.60 MATLAB 12402 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.23 MATLAB 12405 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.47 MATLAB 12407 traine 20 0 3554m 619m 73m S 39.1 1.0 6:56.20 MATLAB 12403 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.10 MATLAB 12404 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.05 MATLAB 12406 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.34 MATLAB 12408 traine 20 0 3554m 619m 73m S 38.1 1.0 6:55.72 MATLAB 12409 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.37 MATLAB 12410 traine 20 0 3554m 619m 73m S 38.1 1.0 6:56.08 MATLAB 15054 dnairn 20 0 6521m 392m 64m S 12.7 0.6 0:03.30 MATLAB 15063 dnairn 20 0 6521m 392m 64m S 3.9 0.6 0:07.69 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 98.8 1.0 12:38.73 MATLAB 12406 traine 20 0 3554m 619m 73m R 39.1 1.0 6:56.74 MATLAB 12400 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.74 MATLAB 12404 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.44 MATLAB 12408 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.11 MATLAB 12409 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.76 MATLAB 12410 traine 20 0 3554m 619m 73m R 38.1 1.0 6:56.47 MATLAB 12401 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.98 MATLAB 12402 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.61 MATLAB 12403 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.48 MATLAB 12405 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.85 MATLAB 12407 traine 20 0 3554m 619m 73m R 37.2 1.0 6:56.58 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 101.1 1.0 12:39.77 MATLAB 12400 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.27 MATLAB 12401 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.51 MATLAB 12402 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.14 MATLAB 12403 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.01 MATLAB 12405 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.38 MATLAB 12408 traine 20 0 3554m 619m 73m R 51.5 1.0 6:56.64 MATLAB 12409 traine 20 0 3554m 619m 73m R 51.5 1.0 6:57.29 MATLAB 12404 traine 20 0 3554m 619m 73m R 50.5 1.0 6:56.96 MATLAB 12406 traine 20 0 3554m 619m 73m R 50.5 1.0 6:57.26 MATLAB 12407 traine 20 0 3554m 619m 73m R 50.5 1.0 6:57.10 MATLAB 12410 traine 20 0 3554m 619m 73m R 50.5 1.0 6:56.99 MATLAB 15063 dnairn 20 0 6521m 392m 64m S 1.0 0.6 0:07.70 MATLAB
User dnairn starts a computational demanding function from the GUI.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 99.5 1.0 12:40.80 MATLAB 15027 dnairn 20 0 6521m 393m 64m R 98.6 0.6 0:17.28 MATLAB 15180 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:09.04 MATLAB 15186 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:08.68 MATLAB 15188 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:08.87 MATLAB 15189 dnairn 20 0 6521m 393m 64m R 41.5 0.6 0:08.49 MATLAB 15182 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.91 MATLAB 15183 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.55 MATLAB 15187 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.73 MATLAB 15190 dnairn 20 0 6521m 393m 64m R 40.6 0.6 0:08.46 MATLAB 15181 dnairn 20 0 6521m 393m 64m R 39.6 0.6 0:09.24 MATLAB 15184 dnairn 20 0 6521m 393m 64m R 39.6 0.6 0:09.04 MATLAB 15185 dnairn 20 0 6521m 393m 64m R 38.6 0.6 0:08.47 MATLAB 12401 traine 20 0 3554m 619m 73m R 30.0 1.0 6:57.82 MATLAB 12403 traine 20 0 3554m 619m 73m R 29.0 1.0 6:57.31 MATLAB 12406 traine 20 0 3554m 619m 73m R 29.0 1.0 6:57.56 MATLAB 12408 traine 20 0 3554m 619m 73m R 29.0 1.0 6:56.94 MATLAB 12400 traine 20 0 3554m 619m 73m R 28.0 1.0 6:57.56 MATLAB 12407 traine 20 0 3554m 619m 73m S 28.0 1.0 6:57.39 MATLAB 12409 traine 20 0 3554m 619m 73m S 28.0 1.0 6:57.58 MATLAB 12404 traine 20 0 3554m 619m 73m R 24.2 1.0 6:57.21 MATLAB 12402 traine 20 0 3554m 619m 73m R 23.2 1.0 6:57.38 MATLAB 12405 traine 20 0 3554m 619m 73m R 23.2 1.0 6:57.62 MATLAB 12410 traine 20 0 3554m 619m 73m R 22.2 1.0 6:57.22 MATLAB 15156 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15157 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15158 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15159 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15160 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15162 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15163 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15164 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15165 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15168 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15171 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15172 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15173 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15174 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15175 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15177 dnairn 20 0 6521m 393m 64m S 1.9 0.6 0:00.02 MATLAB 15059 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.47 MATLAB 15063 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:07.71 MATLAB 15161 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB 15166 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB 15167 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB 15169 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB 15170 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB 15176 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB 15178 dnairn 20 0 6521m 393m 64m S 1.0 0.6 0:00.01 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 100.7 1.0 12:41.84 MATLAB 15027 dnairn 20 0 6521m 393m 64m R 100.7 0.6 0:18.32 MATLAB 12400 traine 20 0 3554m 619m 73m R 35.8 1.0 6:57.93 MATLAB 15181 dnairn 20 0 6521m 393m 64m R 34.9 0.6 0:09.60 MATLAB 15184 dnairn 20 0 6521m 393m 64m R 34.9 0.6 0:09.40 MATLAB 12401 traine 20 0 3554m 619m 73m R 33.9 1.0 6:58.17 MATLAB 12403 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.66 MATLAB 12404 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.56 MATLAB 12406 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.91 MATLAB 12408 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.29 MATLAB 12410 traine 20 0 3554m 619m 73m R 33.9 1.0 6:57.57 MATLAB 15180 dnairn 20 0 6521m 393m 64m R 33.9 0.6 0:09.39 MATLAB 15186 dnairn 20 0 6521m 393m 64m R 33.9 0.6 0:09.03 MATLAB 15188 dnairn 20 0 6521m 393m 64m R 33.9 0.6 0:09.22 MATLAB 12407 traine 20 0 3554m 619m 73m R 32.9 1.0 6:57.73 MATLAB 12409 traine 20 0 3554m 619m 73m R 32.9 1.0 6:57.92 MATLAB 15187 dnairn 20 0 6521m 393m 64m R 32.9 0.6 0:09.07 MATLAB 12405 traine 20 0 3554m 619m 73m R 31.9 1.0 6:57.95 MATLAB 15182 dnairn 20 0 6521m 393m 64m R 31.9 0.6 0:09.24 MATLAB 15185 dnairn 20 0 6521m 393m 64m R 31.9 0.6 0:08.80 MATLAB 15183 dnairn 20 0 6521m 393m 64m R 31.0 0.6 0:08.87 MATLAB 12402 traine 20 0 3554m 619m 73m R 30.0 1.0 6:57.69 MATLAB 15189 dnairn 20 0 6521m 393m 64m R 23.2 0.6 0:08.73 MATLAB 15190 dnairn 20 0 6521m 393m 64m R 20.3 0.6 0:08.67 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 99.1 1.0 12:42.88 MATLAB 15027 dnairn 20 0 6521m 392m 64m R 99.1 0.6 0:19.36 MATLAB 15185 dnairn 20 0 6521m 392m 64m S 83.9 0.6 0:09.68 MATLAB 15180 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.26 MATLAB 15181 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.47 MATLAB 15184 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.27 MATLAB 15186 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:09.90 MATLAB 15188 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:10.09 MATLAB 15189 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:09.60 MATLAB 15190 dnairn 20 0 6521m 392m 64m S 82.9 0.6 0:09.54 MATLAB 15187 dnairn 20 0 6521m 392m 64m S 77.2 0.6 0:09.88 MATLAB 15182 dnairn 20 0 6521m 392m 64m S 76.2 0.6 0:10.04 MATLAB 15183 dnairn 20 0 6521m 392m 64m S 76.2 0.6 0:09.67 MATLAB 12403 traine 20 0 3554m 619m 73m R 50.5 1.0 6:58.19 MATLAB 12401 traine 20 0 3554m 619m 73m R 49.6 1.0 6:58.69 MATLAB 12405 traine 20 0 3554m 619m 73m R 49.6 1.0 6:58.47 MATLAB 12407 traine 20 0 3554m 619m 73m R 49.6 1.0 6:58.25 MATLAB 12408 traine 20 0 3554m 619m 73m R 49.6 1.0 6:57.81 MATLAB 12400 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.44 MATLAB 12406 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.42 MATLAB 12409 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.43 MATLAB 12410 traine 20 0 3554m 619m 73m R 48.6 1.0 6:58.08 MATLAB 12404 traine 20 0 3554m 619m 73m R 37.2 1.0 6:57.95 MATLAB 12402 traine 20 0 3554m 619m 73m R 36.2 1.0 6:58.07 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 100.0 1.0 12:43.91 MATLAB 15027 dnairn 20 0 6521m 392m 64m R 100.0 0.6 0:20.39 MATLAB 15187 dnairn 20 0 6521m 392m 64m R 77.7 0.6 0:10.68 MATLAB 15180 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:11.05 MATLAB 15181 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:11.26 MATLAB 15182 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:10.83 MATLAB 15184 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:11.06 MATLAB 15188 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:10.88 MATLAB 15189 dnairn 20 0 6521m 392m 64m R 76.7 0.6 0:10.39 MATLAB 15185 dnairn 20 0 6521m 392m 64m R 75.8 0.6 0:10.46 MATLAB 15186 dnairn 20 0 6521m 392m 64m R 75.8 0.6 0:10.68 MATLAB 15190 dnairn 20 0 6521m 392m 64m R 74.8 0.6 0:10.31 MATLAB 15183 dnairn 20 0 6521m 392m 64m R 71.9 0.6 0:10.41 MATLAB 12406 traine 20 0 3554m 619m 73m R 41.8 1.0 6:58.85 MATLAB 12410 traine 20 0 3554m 619m 73m R 41.8 1.0 6:58.51 MATLAB 12405 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.89 MATLAB 12407 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.67 MATLAB 12408 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.23 MATLAB 12409 traine 20 0 3554m 619m 73m R 40.8 1.0 6:58.85 MATLAB 12401 traine 20 0 3554m 619m 73m R 39.8 1.0 6:59.10 MATLAB 12403 traine 20 0 3554m 619m 73m R 39.8 1.0 6:58.60 MATLAB 12404 traine 20 0 3554m 619m 73m R 38.9 1.0 6:58.35 MATLAB 12402 traine 20 0 3554m 619m 73m R 29.1 1.0 6:58.37 MATLAB 12400 traine 20 0 3554m 619m 73m R 26.2 1.0 6:58.71 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 100.6 1.0 12:44.96 MATLAB 15027 dnairn 20 0 6521m 392m 64m R 99.6 0.6 0:21.43 MATLAB 15181 dnairn 20 0 6521m 392m 64m R 78.5 0.6 0:12.08 MATLAB 15184 dnairn 20 0 6521m 392m 64m R 77.6 0.6 0:11.87 MATLAB 15189 dnairn 20 0 6521m 392m 64m R 77.6 0.6 0:11.20 MATLAB 15182 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.63 MATLAB 15186 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.48 MATLAB 15188 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.68 MATLAB 15190 dnairn 20 0 6521m 392m 64m R 76.6 0.6 0:11.11 MATLAB 15180 dnairn 20 0 6521m 392m 64m R 74.7 0.6 0:11.83 MATLAB 15183 dnairn 20 0 6521m 392m 64m R 73.8 0.6 0:11.18 MATLAB 15185 dnairn 20 0 6521m 392m 64m R 72.8 0.6 0:11.22 MATLAB 15187 dnairn 20 0 6521m 392m 64m R 70.9 0.6 0:11.42 MATLAB 12401 traine 20 0 3554m 619m 73m R 50.8 1.0 6:59.63 MATLAB 12407 traine 20 0 3554m 619m 73m R 50.8 1.0 6:59.20 MATLAB 12410 traine 20 0 3554m 619m 73m R 50.8 1.0 6:59.04 MATLAB 12403 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.12 MATLAB 12405 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.41 MATLAB 12406 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.37 MATLAB 12408 traine 20 0 3554m 619m 73m R 49.8 1.0 6:58.75 MATLAB 12409 traine 20 0 3554m 619m 73m R 49.8 1.0 6:59.37 MATLAB 12404 traine 20 0 3554m 619m 73m R 46.0 1.0 6:58.83 MATLAB 12400 traine 20 0 3554m 619m 73m R 41.2 1.0 6:59.14 MATLAB 12402 traine 20 0 3554m 619m 73m R 39.3 1.0 6:58.78 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12323 traine 20 0 3554m 619m 73m R 98.6 1.0 12:46.00 MATLAB 15027 dnairn 20 0 6521m 392m 64m R 97.7 0.6 0:22.46 MATLAB 15182 dnairn 20 0 6521m 392m 64m R 95.8 0.6 0:12.64 MATLAB 15180 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.83 MATLAB 15184 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.87 MATLAB 15188 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.68 MATLAB 15189 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.20 MATLAB 15190 dnairn 20 0 6521m 392m 64m R 94.8 0.6 0:12.11 MATLAB 15181 dnairn 20 0 6521m 392m 64m R 93.9 0.6 0:13.07 MATLAB 15186 dnairn 20 0 6521m 392m 64m R 93.9 0.6 0:12.47 MATLAB 15187 dnairn 20 0 6521m 392m 64m R 92.9 0.6 0:12.40 MATLAB 15183 dnairn 20 0 6521m 392m 64m R 86.3 0.6 0:12.09 MATLAB 15185 dnairn 20 0 6521m 392m 64m R 82.5 0.6 0:12.09 MATLAB 12405 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.83 MATLAB 12406 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.79 MATLAB 12408 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.17 MATLAB 12409 traine 20 0 3554m 619m 73m S 39.8 1.0 6:59.79 MATLAB 12401 traine 20 0 3554m 619m 73m S 38.9 1.0 7:00.04 MATLAB 12403 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.53 MATLAB 12404 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.24 MATLAB 12407 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.61 MATLAB 12410 traine 20 0 3554m 619m 73m S 38.9 1.0 6:59.45 MATLAB 12402 traine 20 0 3554m 619m 73m S 33.2 1.0 6:59.13 MATLAB 12400 traine 20 0 3554m 619m 73m S 31.3 1.0 6:59.47 MATLAB
Top sampling with threads combined
This time the the top is done without the -H option. Thus the %CPU is reported for all threads and can be more than 100% when the threads are on different cores.
$ ssh n015 top -b -d 1 -n 60 | egrep '(COMMAND|MATLAB)' | grep -v 'S 0' PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1193.5 1.0 31:47.28 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1133.1 1.0 31:58.90 MATLAB 14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.75 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1183.7 1.0 32:11.03 MATLAB 14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.76 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1140.7 1.0 32:22.72 MATLAB 14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.77 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1160.9 1.0 32:34.61 MATLAB 14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.78 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1174.0 1.0 32:46.65 MATLAB 14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.79 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1186.6 1.0 32:58.81 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1163.8 1.0 33:10.74 MATLAB 14985 dnairn 20 0 6616m 449m 64m S 1.0 0.7 9:56.80 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1204.3 1.0 33:23.09 MATLAB 14985 dnairn 20 0 6807m 467m 64m S 13.7 0.7 9:56.94 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 1113.1 1.0 33:34.90 MATLAB 14985 dnairn 20 0 6998m 831m 64m S 288.4 1.3 10:00.00 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3300m 428m 73m S 400.4 0.7 33:39.06 MATLAB 14985 dnairn 20 0 6998m 831m 64m S 100.1 1.3 10:01.04 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 435.1 1.3 10:05.51 MATLAB 16367 traine 20 0 3491m 620m 73m S 99.3 1.0 33:40.08 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 366.2 1.3 10:09.24 MATLAB 16367 traine 20 0 3491m 620m 73m S 254.2 1.0 33:42.67 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 340.0 1.3 10:12.73 MATLAB 16367 traine 20 0 3491m 620m 73m S 323.5 1.0 33:45.99 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 398.3 1.0 33:50.05 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 343.3 1.3 10:16.23 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 356.1 1.3 10:19.86 MATLAB 16367 traine 20 0 3491m 620m 73m S 340.4 1.0 33:53.52 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 524.8 1.0 33:58.87 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.1 1.3 10:20.88 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 443.4 1.3 10:25.41 MATLAB 16367 traine 20 0 3491m 620m 73m S 241.8 1.0 34:01.34 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 202.2 1.0 34:03.44 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 98.2 1.3 10:26.43 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 376.9 1.3 10:30.27 MATLAB 16367 traine 20 0 3491m 620m 73m S 334.7 1.0 34:06.85 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 387.2 1.0 34:10.79 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 285.0 1.3 10:33.17 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 405.6 1.0 34:14.95 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 120.9 1.3 10:34.41 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 525.7 1.0 34:20.30 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 498.2 1.3 10:39.48 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 419.5 1.0 34:24.57 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 10:40.50 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 470.6 1.0 34:29.36 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 428.4 1.3 10:44.86 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 431.3 1.3 10:49.25 MATLAB 16367 traine 20 0 3491m 620m 73m S 425.4 1.0 34:33.69 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 349.7 1.0 34:37.25 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 10:50.27 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 405.8 1.3 10:54.40 MATLAB 16367 traine 20 0 3491m 620m 73m S 339.0 1.0 34:40.70 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 450.7 1.0 34:45.32 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 392.1 1.3 10:58.42 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 426.8 1.0 34:49.74 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 168.0 1.3 11:00.16 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 400.0 1.0 34:53.84 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 303.4 1.3 11:03.27 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 439.3 1.0 34:58.34 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 418.8 1.3 11:07.56 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 411.0 1.0 35:02.55 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 99.6 1.3 11:08.58 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 359.7 1.0 35:06.21 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 275.2 1.3 11:11.38 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 449.0 1.0 35:10.78 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 367.4 1.3 11:15.12 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 505.0 1.0 35:15.92 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:16.14 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 429.3 1.3 11:20.51 MATLAB 16367 traine 20 0 3491m 620m 73m S 420.4 1.0 35:20.20 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 401.8 1.0 35:24.29 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:21.53 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 406.7 1.0 35:28.43 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 276.1 1.3 11:24.34 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 456.8 1.0 35:33.08 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:25.36 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 351.7 1.3 11:28.94 MATLAB 16367 traine 20 0 3491m 620m 73m S 256.4 1.0 35:35.69 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 345.8 1.0 35:39.21 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:29.96 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 390.0 1.0 35:43.18 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 251.5 1.3 11:32.52 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 287.1 1.0 35:46.10 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 206.5 1.3 11:34.62 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 355.1 1.0 35:49.74 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 286.8 1.3 11:37.56 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 388.4 1.0 35:53.69 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 355.0 1.3 11:41.17 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 461.1 1.0 35:58.45 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 99.8 1.3 11:42.20 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 418.5 1.0 36:02.71 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 347.8 1.3 11:45.74 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 350.7 1.0 36:06.28 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.2 1.3 11:46.76 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 510.9 1.0 36:11.48 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 360.6 1.3 11:50.43 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 509.9 1.0 36:16.67 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 102.2 1.3 11:51.47 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 586.7 1.0 36:22.71 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 319.6 1.3 11:54.76 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 410.4 1.0 36:26.94 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 340.5 1.3 11:58.27 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 388.9 1.0 36:30.94 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.1 1.3 11:59.30 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14985 dnairn 20 0 6998m 865m 64m S 441.2 1.3 12:03.84 MATLAB 16367 traine 20 0 3491m 620m 73m S 414.9 1.0 36:35.21 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 363.6 1.0 36:38.95 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 100.1 1.3 12:04.87 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 475.6 1.0 36:43.81 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 348.4 1.3 12:08.43 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 455.2 1.0 36:48.48 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 368.5 1.3 12:12.21 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 282.2 1.0 36:51.37 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 155.3 1.3 12:13.80 MATLAB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16367 traine 20 0 3491m 620m 73m S 541.1 1.0 36:56.96 MATLAB 14985 dnairn 20 0 6998m 865m 64m S 371.7 1.3 12:17.64 MATLAB