Using Moab Job Priorities – Understanding mdiag -p Output

This entry is part 2 of 3 in the series Using Moab Job Priorities

In part two of the Using Moab Job Priorities blog series, we are going to take a look at the Moab diagnostic command for priority: mdiag -p. This command gives the administrator an instaneous “snapshot” view of the priorities for the different jobs within the system. Additionally, it provides information on how these priorites are being calculated.

Output Example

Let’s start by taking a look at an example output from the command. In this case, we are going to use the verbose output (mdiag -p -v), which will give us the priorities of not only the queued jobs, but of the running jobs as well. While it is not shown by default, running jobs do maintain a priority, which is very important in terms of preemption.

diagnosing job priority information (partition: ALL)

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

             Weights   ––––––––       1(    1:    5:    8)     2(    1:    1:    1:    1:    1)     1(    1:    1)

456038                   155300*   98.2(10000:  0.0:  0.0)   0.3(-33.6:-33.6:-33.6:-33.6:-33.6)   1.5(1461.: 74.1)
456341                     1299     0.0(  0.0:  0.0:  0.0)  99.2(128.9:128.9:128.9:128.9:128.9)   0.8(  9.0:  1.2)
456346                     1030     0.0(  0.0:  0.0:  0.0)  92.0( 94.8: 94.8: 94.8: 94.8: 94.8)   8.0( 81.0:  1.5)
456385                      764     0.0(  0.0:  0.0:  0.0)  98.5( 75.3: 75.3: 75.3: 75.3: 75.3)   1.5( 10.0:  1.2)
456165                      338     0.0(  0.0:  0.0:  0.0)  33.2(-33.6:-33.6:-33.6:-33.6:-33.6)  66.8(672.0:  1.9)
456292                      181     0.0(  0.0:  0.0:  0.0)  24.9( -9.0: -9.0: -9.0: -9.0: -9.0)  75.1(245.0: 25.6)
456382                       24     0.0(  0.0:  0.0:  0.0)   0.0(  0.0:  0.0:  0.0:  0.0:  0.0) 100.0( 19.0:  4.8)
456380                       22     0.0(  0.0:  0.0:  0.0)   0.0(  0.0:  0.0:  0.0:  0.0:  0.0) 100.0( 21.0:  1.0)
456386                        8     0.0(  0.0:  0.0:  0.0)   0.0(  0.0:  0.0:  0.0:  0.0:  0.0) 100.0(  7.0:  1.0)
455814                    -7867     0.0(  0.0:  0.0:  0.0)  94.9(-831.:-831.:-831.:-831.:-831.)   5.1(447.0:  1.3)
455637                    -7895     0.0(  0.0:  0.0:  0.0)  95.2(-831.:-831.:-831.:-831.:-831.)   4.8(419.0:  1.3)
455564                    -8081     0.0(  0.0:  0.0:  0.0)  97.3(-831.:-831.:-831.:-831.:-831.)   2.7(233.0:  1.2)
456295                    -8094     0.0(  0.0:  0.0:  0.0)  97.4(-831.:-831.:-831.:-831.:-831.)   2.6(220.0:  1.1)
456371                    -8282     0.0(  0.0:  0.0:  0.0)  99.6(-831.:-831.:-831.:-831.:-831.)   0.4( 32.0:  1.0)
456375                    -8292     0.0(  0.0:  0.0:  0.0)  99.7(-831.:-831.:-831.:-831.:-831.)   0.3( 22.0:  1.1)
–––Active Jobs–––
456287                    99665    99.7(10000:  0.0:  0.0)   0.3(-33.6:-33.6:-33.6:-33.6:-33.6)   0.0(  0.0:  1.0)
452500                        1     0.0(  0.0:  0.0:  0.0)   0.0(  0.0:  0.0:  0.0:  0.0:  0.0) 100.0(  0.0:  1.0)
456355                        1     0.0(  0.0:  0.0:  0.0)   0.0(  0.0:  0.0:  0.0:  0.0:  0.0) 100.0(  0.0:  1.0)
456077                     -734     0.0(  0.0:  0.0:  0.0)  99.9(-73.5:-73.5:-73.5:-73.5:-73.5)   0.1(  0.0:  1.0)
455918                    -1969     0.0(  0.0:  0.0:  0.0)  99.9(-197.:-197.:-197.:-197.:-197.)   0.1(  0.0:  1.0)
455994                    -1969     0.0(  0.0:  0.0:  0.0)  99.9(-197.:-197.:-197.:-197.:-197.)   0.1(  0.0:  1.0)
456230                    -2234     0.0(  0.0:  0.0:  0.0) 100.0(-223.:-223.:-223.:-223.:-223.)   0.0(  0.0:  1.0)
456048                    -2234     0.0(  0.0:  0.0:  0.0) 100.0(-223.:-223.:-223.:-223.:-223.)   0.0(  0.0:  1.0)
456013                    -8314     0.0(  0.0:  0.0:  0.0) 100.0(-831.:-831.:-831.:-831.:-831.)   0.0(  0.0:  1.0)
454860                    -8314     0.0(  0.0:  0.0:  0.0) 100.0(-831.:-831.:-831.:-831.:-831.)   0.0(  0.0:  1.0)
455609                    -8314     0.0(  0.0:  0.0:  0.0) 100.0(-831.:-831.:-831.:-831.:-831.)   0.0(  0.0:  1.0)
456121                    -8314     0.0(  0.0:  0.0:  0.0) 100.0(-831.:-831.:-831.:-831.:-831.)   0.0(  0.0:  1.0)
456063                    -8314     0.0(  0.0:  0.0:  0.0) 100.0(-831.:-831.:-831.:-831.:-831.)   0.0(  0.0:  1.0)
456176                    -8314     0.0(  0.0:  0.0:  0.0) 100.0(-831.:-831.:-831.:-831.:-831.)   0.0(  0.0:  1.0)

Percent Contribution   ––––––––    99.7(  0.2: 99.5:  0.0)   0.3(  0.1:  0.1:  0.1:  0.1:  0.1)   0.0(  0.0:  0.0)

* indicates absolute/relative system prio set on job

 
At first glance, this is one of the scary client command outputs. It’s basically a huge numeric table, threatening to cause spreadsheet nightmares. But, it really isn’t that bad once one knows where to focus. Let’s break down each of the sections.

Priority Configuration

The first thing this command outputs is the actual priority calculation configuration that was discussed in Part I: Creating a Prioritization Strategy. Columns will be automatically added or removed from this output based on which priority components and sub-components have been enabled in the Moab’s configuration file.

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

             Weights   ––––––––       1(    1:    5:    8)     2(    1:    1:    1:    1:    1)     1(    1:    1)

 
This section shows the priority components and sub-components and their associated weights.

Let’s look at the components first.

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

             Weights   ––––––––       1(    1:    5:    8)     2(    1:    1:    1:    1:    1)     1(    1:    1)

 
This particular system is configured to use three different priority components:

Let’s now look at the configured priority sub-components:

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

             Weights   ––––––––       1(    1:    5:    8)     2(    1:    1:    1:    1:    1)     1(    1:    1)

 
This particular site is configured to use an impressive ten different priority sub-components. While there are over forty different available priority sub-components to choose from, most sites tend to choose five or fewer. However, the beauty of Moab’s priority system is that it can be fine-tuned to meet the exacting needs of a particular organization. This organization has chosen the following:

  • User – Specific User Priority with a weight value of 1
  • QOS – Specific Quality of Service Priority with a weight value of 5
  • Class – Specific Class Priority with a weight value of 8
  • User – User-based Historical Usage with a weight value of 1
  • Group – Group-based Historical Usage with a weight value of 1
  • Accnt – Account-based Historical Usage with a weight value of 1
  • QOS – Quality of Service-based Historical Usage with a weight value of 1
  • Class – Class-based Historical Usage with a weight value of 1
  • QTime – Queue Time (minutes) with a weight value of 1
  • XFctr – Expansion Factor with a weight value of 1

 
This means the effective priority for any given job in the system is calculated using the following formula:

Job Priority  = 1 × (1 × UserPriority + 5 × QoSPriority + 8 × ClassPriority)
 + 2 × (1 × UserFSUsage + 1 × GroupFSUsage + 1 × AccountFSUsage + 1 × QoSFSUsage + 1 × ClassFSUsage)
 + 1 × (1 × QueueTime + 1 × ExpansionFactor)

 
I know in this case it does look a bit complicated, but it’s really just the expanded form of the priority function we’ve seen before:

Job Priority Calculation Function

 
In review, Moab will be running this calculation on every eligible or running job every scheduling iteration. Let’s look at the result of that process in a little more detail.

Priority Calculation

After the section explaining how priority is going to be calculated, mdiag -p lists all of the jobs that are currently eligible to run, their priority and how it was calculated. If the -v switch (verbose) was used, running jobs will also be included in the output.

Here we have the first several highest-priority jobs from our example:

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

             Weights   ––––––––       1(    1:    5:    8)     2(    1:    1:    1:    1:    1)     1(    1:    1)

456038                   155300*   98.2(10000:  0.0:  0.0)   0.3(-33.6:-33.6:-33.6:-33.6:-33.6)   1.5(1461.: 74.1)
456341                     1299     0.0(  0.0:  0.0:  0.0)  99.2(128.9:128.9:128.9:128.9:128.9)   0.8(  9.0:  1.2)
456346                     1030     0.0(  0.0:  0.0:  0.0)  92.0( 94.8: 94.8: 94.8: 94.8: 94.8)   8.0( 81.0:  1.5)
456385                      764     0.0(  0.0:  0.0:  0.0)  98.5( 75.3: 75.3: 75.3: 75.3: 75.3)   1.5( 10.0:  1.2)
456165                      338     0.0(  0.0:  0.0:  0.0)  33.2(-33.6:-33.6:-33.6:-33.6:-33.6)  66.8(672.0:  1.9)
456292                      181     0.0(  0.0:  0.0:  0.0)  24.9( -9.0: -9.0: -9.0: -9.0: -9.0)  75.1(245.0: 25.6)
456382                       24     0.0(  0.0:  0.0:  0.0)   0.0(  0.0:  0.0:  0.0:  0.0:  0.0) 100.0( 19.0:  4.8)

 
We’ll skip the first one right now (and come back to it later), as it’s a special case. Let’s take the second highest priority job, 456341, which has a priority of 1299.

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

456346                     1030     0.0(  0.0:  0.0:  0.0)  92.0( 94.8: 94.8: 94.8: 94.8: 94.8)   8.0( 81.0:  1.5)

 
One of the things that sometimes confuses people when they look at the mdiag -p output is that the top section and center section appear to be similar in layout, but they are actually showing different data types. They are grouped together, so in our example, the entries in the columns under the priority components and sub-components relate to those components and sub-components. The difference is the values are not “weight” values.

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

456346                     1030     0.0(  0.0:  0.0:  0.0)  92.0( 94.8: 94.8: 94.8: 94.8: 94.8)   8.0( 81.0:  1.5)

 
For example, the numbers that appear before the parenthesis are related to the priority components above them in the column(i.e., Cred:0.0, FS:92.0 & Serv:8.0). However, these numbers represent the percentage of the overall priority that is provided by that priority component. So, in this case, credentials provided none of the priority points, while 92% of the priority came from fairshare and the remaining 8% came from service-based items.

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

456346                     1030     0.0(  0.0:  0.0:  0.0)  92.0( 94.894.894.894.894.8)   8.0( 81.0:  1.5)

 
The values inside the parenthesizes match to the priority sub-components at their column head. However, instead of being a percentage, these represent the sub-component values that will be plugged into the priority function. So,

Job Priority  = 1 × (1 × UserPriority + 5 × QoSPriority + 8 × ClassPriority)
 + 2 × (1 × UserFSUsage + 1 × GroupFSUsage + 1 × AccountFSUsage + 1 × QoSFSUsage + 1 × ClassFSUsage)
 + 1 × (1 × QueueTime + 1 × ExpansionFactor)

 
becomes:

        1030  ≅ 1 × (1 × 0.0 + 5 × 0.0 + 8 × 0.0)
 + 2 × (1 × 94.8 + 1 × 94.8 + 1 × 94.8 + 1 × 94.8 + 1 × 94.8)
 + 1 × (1 × 81.0 + 1 × 1.5)

 
One will notice we are using the “approximately equal” sign for the calculation. This is because the view shown in this output is rather limited. Rounding can potentially make numbers not appear to add exactly as expected. Additionally, as each priority sub-component value field only shows a maximum of five numbers, any value over 99,999 will be truncated in the view. The PRIORITY* column does not have that size restriction.

Now that we’ve covered the standard case, let’s return and take a look at the top priority job waiting in the queue.

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

             Weights   ––––––––       1(    1:    5:    8)     2(    1:    1:    1:    1:    1)     1(    1:    1)

456038                   155300*   98.2(10000:  0.0:  0.0)   0.3(-33.6:-33.6:-33.6:-33.6:-33.6)   1.5(1461.: 74.1)

 
One will quickly notice there is an asterisk next to the priority. This means the system administrator has made a manual change to the priority of this particular job, overriding the standard priority calculation process. While this calculation is still shown, it is not being used by Moab unless it was a relative-value modification, as opposed to an absolute modification. This change may make the priority higher or lower than what would be calculated by the priority function, depending on the behavior desired by the administrator. It should be noted that once a job’s priority has been manually changed by the administrator, it can never again be fully handled by the automatic priority function. The administrator is responsible for it from then on.

While this all may feel a bit awkward at first, reading this output can quickly come to feel second nature. It becomes easy to quickly see where priority weights or values may not be configured correctly. For example, in the case we have been looking at for this exercise, it quickly becomes apparent the credential-based values may not be fully configured, as each value is 0.0 in almost every entry.

Job                    PRIORITY*   Cred( User:  QOS:Class)    FS( User:Group:Accnt:  QOS:Class)  Serv(QTime:XFctr)

             Weights   ––––––––       1(    1:    5:    8)     2(    1:    1:    1:    1:    1)     1(    1:    1)

456038                   155300*   98.2(10000:  0.0:  0.0)   0.3(-33.6:-33.6:-33.6:-33.6:-33.6)   1.5(1461.: 74.1)
456341                     1299     0.0(  0.0:  0.0:  0.0)  99.2(128.9:128.9:128.9:128.9:128.9)   0.8(  9.0:  1.2)
456346                     1030     0.0(  0.0:  0.0:  0.0)  92.0( 94.8: 94.8: 94.8: 94.8: 94.8)   8.0( 81.0:  1.5)
456385                      764     0.0(  0.0:  0.0:  0.0)  98.5( 75.3: 75.3: 75.3: 75.3: 75.3)   1.5( 10.0:  1.2)
456165                      338     0.0(  0.0:  0.0:  0.0)  33.2(-33.6:-33.6:-33.6:-33.6:-33.6)  66.8(672.0:  1.9)
456292                      181     0.0(  0.0:  0.0:  0.0)  24.9( -9.0: -9.0: -9.0: -9.0: -9.0)  75.1(245.0: 25.6)
456382                       24     0.0(  0.0:  0.0:  0.0)   0.0(  0.0:  0.0:  0.0:  0.0:  0.0) 100.0( 19.0:  4.8)

 
Something isn’t quite right there. The administrator should probably look into it to see why individual users, QoSes and classes don’t have default priority values.

Finally, we’ll move on to the last bit of output from mdiag -p.

Aggregate Statistics

At the very bottom of the output, there is a single line that provides some additional useful information:

Percent Contribution   ––––––––    99.7(  0.2: 99.5:  0.0)   0.3(  0.1:  0.1:  0.1:  0.1:  0.1)   0.0(  0.0:  0.0)

 
Similar to what was seen previously, each of the numbers match with their column heads. However, here every single number is a percentage. Basically, it is telling the administrator, across all jobs, this is the percentage on average provided by each of the priority components and sub-components to the overall average job priority. It is often used to answer the question:

On average, what is making the most difference to my job priorities?

The percentage values here will tell you. Please note, all percentages listed are relative to the overall priority. So, priority sub-component percentages will numerically add up to their associated priority component percentage. As can be seen in the example, rounding can be a visual issue. However, as before, rest assured the numbers inside of Moab are accurate on this account. It’s the displaying of them that can be less than perfect.

In Conclusion (and Homework)

The output from mdiag -p can be a bit scary to the uninitiated. However, the output isn’t too difficult to read and understand with a few basic pointers, which are covered here in today’s post.

As homework, it is suggested you run this command on your system and study the output. You will want to answer the following questions:

  • Are all of the priority components and sub-components I am expecting listed?
  • Are values showing for each of them?
  • How well are the values balancing in terms of our organizational goals?
  • What tweaks, if any, do I want/need to make to the prioritization strategy and priority function?

It is important to occasionally redo this exercise, as things tend to change over time. What I mean by this is that workload patterns, users, priorities and other things may change, thereby necessitating a change in job prioritization strategy.

Good Luck!

Join me again next time for the third and final entry in this series where we will look at some often unused or under-utilized priority sub-components, and discuss what benefits they could potentially provide for your organization.

Series Navigation<< Using Moab Job Priorities – Creating a Prioritization StrategyUsing Moab Job Priorities – Exploring Priority Sub-Components >>
Facebook Twitter Email

Comments

  1. Trev,
    This was extremely helpful. It might he useful to link this from the main Adaptive documentation page. Some of the items there can use explanation like this.

    I do have question. I was wondering how one sub component is weighted relative to another component. For instance, when I made changes to SERVICEWEIGHT (from 100 to 50) and RESWEIGHT (1 to 50), Qtime contribution changed from 12.4% to 0.8. That seemed bit drastic. I also changed PROCWEIGHT and WALLTIMEWEIGHT.

    Before change:

    Job PRIORITY* Cred( QOS:Class) Serv(QTime:XFctr) Res( Proc:WTime)
    Weights ——– 1( 10: 1) 100( 2: 100) 1( 5: 10)
    Percent Contribution ——– 13.0( 0.0: 13.0) 13.8( 12.4: 1.5) 73.2( 7.4: 65.8)

    After:

    Job PRIORITY* Cred( QOS:Class) Serv(QTime:XFctr) Res( Proc:WTime)
    Weights ——– 1( 10: 1) 50( 2: 100) 50( 10: 2)
    Percent Contribution ——– 0.9( 0.0: 0.9) 0.9( 0.8: 0.1) 98.2( 52.0: 46.2)

Speak Your Mind

*