Torque 6.0 NUMA Use Cases: Introduction

This entry is part 1 of 2 in the series Torque 6.0 NUMA Use Cases

LagoonFerrisWheelTorque 6.0.0 was released in October so it is time to help users understand how to use the new features which have been added. One of the main focuses of Torque 6.0.0 was to enable users to maximize the performance and predictability of jobs. By taking advantage of hwloc and a new addition to the qsub options users are now able to place jobs in hardware configurations which will best suit their workload.

The new addition to the qsub syntax is the -L resource request option. Sometimes trying new things can be a little daunting like David Beer’s turducken. But once you learn how to put things together you find the results are much better. I have to take David’s word on the turducken. He says it was pretty tasty. I have yet to try it. I do know that the new -L syntax is a very powerful tool to get the most out of your hardware.

The -L resource request is an additional option for qsub. It does not replace any other options. The -l resource request option is still available in Torque 6.0.0. To learn about all of the new -L option in Torque 6.0.0 you will need to read  section 12.2 of the Torque documentation. There is simply too much information to put in a single blog. Today’s blog introduces the new syntax, how it is used and gives some examples.

A basic knowledge of the system hardware configuration will help users get the most out of using the -L options. It has been designed to work with NUMA aware systems. Users are able to make resource requests which will run their jobs on cpus/cores which are nearest to memory or GPUs or other resources they wish to use so jobs perform at maximum speed. Even so, it is not required that users know specifics about the hardware configurations in order to use this option.

If you read section 12.2 of the Torque documentation you saw that the -L syntax has several options which are called values. There are two values which are mandatory; tasks and lprocs. (lprocs was not intended to be a mandatory value but as of Torque 6.0.0 it is. This may change in future versions of Torque.) Every -L resource request must start with these two values in this order. qsub -L tasks=x:lprocs=y where x is the number of tasks requested and y is the number of logical processors per task. All other values can appear in any order after these two values.

In the current Torque -l resource request option the notion of a task is ambiguous. For example, a request of qsub -l nodes=2:ppn=4 can be interpreted by Moab as 8 tasks. In the -L resource request a task is one instance of the hardware resources specified in the rest of the request. For example,

qsub -L tasks=2:lprocs=4:memory=3Gb:place=numanode=1

will create two tasks where each task will consume four logical processors. That is it will be bound to four cores in a cpuset. Each task will also be allowed 3Gb of memory. Finally, the place=numanode=1 says that each task will run on its own NUMA node within the hardware. This is quite different from the -l nodes=2:ppn=4 result. Moab is likely to place this request on eight contiguous nodes all in the same cpuset if cpusets are enabled. The -L syntax does a much better job of defining a task and allows for better placement in the hardware.

Unfortunately, this is all I am able to introduce in this first blog about NUMA use cases. I am quickly running out of word space for a single blog. What I hope you understand from this post is that the -L resource request is a new addition to the qsub syntax and not a replacement for any existing options. I also hope you understand that the -L resource request removes the ambiguity about what a task is with Moab and that it gives users much more control over which resources are used and where.

Please continue to watch this space for more information about how to use the new Torque features in Torque 6.0.0. Once users learn about the power of this new interface I believe it will be the preferred way for users to make resource requests. It is that good. Where can I get some turducken?

Series NavigationTorque 6.0 NUMA Use Cases: Task Placement (Numa Nodes) >>
Facebook Twitter Email

Speak Your Mind