Moab Accounting Manager: Better Than Gold

Moab Accounting Manager vs. GoldAdaptive Computing has been an industry leader in providing resource usage showback and chargeback capabilities for High Performance Computing — initially with Gold Allocation Manager, and now with Moab Accounting Manager. With experience gained from the needs of HPC customers while pioneering the usage of Gold Allocation Manager, significant improvements were made in Moab Accounting Manager to address the accounting and allocation needs of modern HPC sites.

Among the many significant enhancements, Moab Accounting Manager: is more adaptable to the variety of customer accounting requirements, has improved administration of allocation management, is faster, is easier to use, and fixes many limitations in Gold Allocation Manager.

More adaptable to your unique accounting needs

Our experience has shown that accounting use cases vary widely between sites. What accounting means to one site is often very different from what it means to another. Moab Accounting Manager (MAM) has introduced the ability for the customer to select one of four accounting modes to specify what level of accounting (e.g. showback or chargeback) and resource usage enforcement is needed for their notion of accounting. The accounting mode modifies the way in which Moab interacts with MAM during the various stages of the job or reservation lifetime (e.g. job submission, job start, job completion, etc.). The accounting mode can be one of usage-tracking, notional-charging, fast-allocation or strict-allocation.

Many sites want to allocate funds strictly on a per-account basis. However, others want to earmark their allocated usage according to other properties such as the group, quality of service or class. Whereas Gold allocations were limited to account, user and cluster, with MAM each fund may be associated with arbitrary job properties (e.g. Class=short, QualityOfService=Premium, Organization=IBM).

The charging options within MAM have been significantly expanded over the charge rate types that were available in Gold. Aside from replacing Gold’s charge rate types with charge operation modifiers, MAM also adds many new charging capabilities including: conditional charge rates, charge rate defaults, integer and floating point charge rate ranges, divisor and time-based charge rate modifiers, charge rate override, and alternate rate plans.

Moab Accounting Manager is much more flexible in its ability to be customized according to a site’s local needs. Gold’s job table has been replaced by a more generic usage record table in MAM that can be customized according to the resource usage properties that you want to collect. Custom fields can be added to the transaction table. Custom properties can be specified in many commands via the extension option. Custom objects support fields with auto-generation and default values. These enhancements allow a customer site to address unique accounting use cases that could not be supported in Gold.

Improvements to allocations

Whereas the expiring allocations in Gold were free-form and required a lot of administration to manage, MAM introduced the concept of discrete allocations and automatic reset to greatly simplify administration.

With discrete allocations, MAM ensures that allocations within a fund are non-overlapping (in time) and non-reusable (each allocation period should use a distinct allocation). This policy helps to improve clarity and consistency when reporting on allocation usage during a particular period. Additionally, discrete allocations are better able to track allocation specific properties such as the initial and current allocation amount, permitting quicker reporting.

MAM allows you to automatically renew accounts on a periodic basis without the hassle of having to create all future allocations ahead of time. Although expiring allocations are still supported in MAM, many sites will prefer the new model that is more similar to automatic deposit in the bank. A default deposit amount may be specified for a fund (account) that you want to be automatically renewed. A MAM event can be created to reset (or end) all such funds (accounts) on a designated period.

In the past, reporting a previous allocation balance has been time consuming because this information was only held in the allocation journal. As of MAM 8.1, each transaction that affects the allocation balance (e.g. a charge, deposit, refund, transfer, etc.) now records the resulting allocation balance in the transaction table. This provides a ledger of intermediate balances that can be displayed from within commands such as glstrans and gstatement.

Many sites want to enforce allocations for some accounts, but allow unlimited usage for others. With MAM, deposits can be made with infinite amounts or infinite credit limits.

Some sites need to use allocations with infinite or long-running jobs or reservations. For this case, Moab and MAM have support for periodic charging where long-running jobs or reservations are charged out at regular intervals. If strictly enforcing allocations, MAM can be configured to make an entire lien up front, reducing it as you go, or it can be configured to obtain a partial lien for each period, and take a designated action if the job or reservation runs out of funds.

Faster

When used with Moab, MAM accounting performs much faster than Gold. The MAM server was modified to be able to handle multiple query requests simultaneously. The server was also modified to preload modules so that they were not incurred for each individual client request. However, more critically, Moab and MAM have been modified to permit non-blocking calls via a separate accounting thread in Moab. Additionally, in MAM 8.1, three higher-throughput accounting modes were introduced that significantly improve performance for many customers.

In Moab and MAM 8.0, a new accounting thread pool allows Moab to make non-blocking calls to MAM for charges instead of blocking the scheduling thread. This functionality is only enabled when you use the MAM accounting manager interface type. This results in faster aggregate scheduling and better client response times; however, individual actions might be shortly delayed in some cases. When using solely non-blocking calls to MAM, Moab responsiveness to client commands is drastically improved.

In Moab and MAM 8.1, three new higher throughput accounting modes were introduced: fast-allocation, notional-charging and usage-tracking. For many sites, one of these accounting modes will better match their accounting requirements than the strict-allocation mode that has been the default. When using one of the new alternative modes, scheduling overhead due to accounting can be reduced from a few tenths of a second per job, to less than a few thousands of a second per job. Depending on your existing scheduling overhead due to Moab and Torque policies, using one of the higher throughput modes has the potential of doubling job throughput on a sustained basis and improving peak job throughput by an order of magnitude. This is achieved by handling charges in the background in a separate thread and the elimination of the blocking lien in the strict-allocation accounting mode.

Easier to Use

Moab Accounting Manager has made many improvements that make it easier to use than Gold. Besides the allocation and charge rate improvements mentioned earlier, MAM has introduced over 20 new command line clients to simplify management and querying of usage records, charge rates, roles, itemized charges and events. Improvements have been made in role based access filtering where commands now show output relative to one’s role rather than simply failing. Smarter default behaviors such as making the first account a user is added to as the default account for that user, reduce administration load. Documentation has been significantly enhanced, including a full up-to-date commands reference. This also includes much improved step-by-step instructions on integrating MAM with Moab and steps for initializing MAM in order for it to be used according to a customer’s preferred mode of operation.

Fixes Problems

Moab Accounting Manager undergoes rigorous nightly tests where over 150 different test cases are validated each night. Thorough installation and upgrade testing is performed before each release for varied customer configurations. Dozens of problems in Gold have been fixed in MAM.

One critical fix that stands out is that Gold uses the Perl setuid program to perform client and server authentication. Perl announced that it would remove support for its setuid functionality in version 5.12 which could leave Gold inoperable. Adaptive Computing anticipated this eventuality and provides gauth as an alternative security promotion method in MAM.

Conclusion

MAM has taken Gold to the next level in functionality, performance, quality and ease-of-use. It has been enhanced to address real-world problems encountered by the most sophisticated HPC sites and customers. Additional information about Moab Accounting Manager can be found at docs.adaptivecomputing.com in the Moab Accounting Manager Administrator Guide for the Enterprise Edition product.

Postscript

The following is a list of new features implemented with Moab Accounting Manager.

MAM 7.0

Resource and service usage can now be reported more generically
Account access is now defined by arbitrary constraints
Account and Allocation filters
Reservation filters
Quote filters
Project and User filters
More refined role privilege filtering
New commands to import and manipulate usage records
New command clients to create and manage charge rates
New command clients to query and manage roles
Charge Rate Override
New gmkres command
Pinned and unpinned Quotes
Quote templates (gmkquote)
Quote and Reservation Grace Periods
Object Auto-Generation
Global Object-based Defaults
Local Attribute-based Defaults
Distinction between charge properties and usage properties
CostOnly Quote Option
Modify Reserve Option
Name Based Charge Rate Defaults
Value Based Charge Rate Ranges
Parallel Request Handling
Improved Client Performance
Optional display of currency in hours in GUI
Custom fields can be added to the Transaction table
Extension option available to query commands
Selection Aliases in goldsh
Manage nested accounts from g{mk,ch}account
Account Priority
Usage Record Instantiators
Default actions are autocreated for custom objects
Failover for GUI
HPC and Cloud bootstrap and initialization
Accounting Context
Reskinned web GUI for inclusion within viewpoint
Documentation

MAM 7.1

Infinite Allocations
Allocation Reset
Alternative Constraints
Floating Point Charge Rate Ranges
Conditional Charge Rates
Itemized Charges
Event Scheduler
Notifications
Site selection
Manage nested accounts via CLI
Constraint Object Auto-generation
Does-Not-Match operator
Charge actions by reference to Usage Record
Gauth security promotion method
Allows designation of separate install subdirectories
Perl module dependencies installed from CPAN
Per Accounting Mode and Context Initialization Scripts

MAM 7.2

Name Consistency with Moab
User Initial Account Default
New commands to manage organizations
Client-only or GUI-only installation option
Database pruning
Payment Handling
Payment Filters
Moab Job Submission Validation

MAM 7.3

Time-based Charge Rate Modifiers
Divisor Charge Rate Modifiers
Charging for Complex Usage Property Values
Charges that fail due to no valid allocations will now update the usage record
New -Q option for glsusage
Moab non-blocking accounting calls enabled for charges via new accounting thread pool
High Availability enabled between Moab and MAM

MAM 7.5

Self-tracking allocations
Default deposit amount
Discrete allocations
CPU time tracking and charging
New command-line clients for creating and modifying events
Init.d and profile.d scripts installed
goldd status option
Ability to specify destination allocation in a transfer
Ability to charge to a specified fund
CSV output option
Ability to specify accounting resume failure action and the resume custom script in Moab
Improved reservation charging policies
Completed non-blocking invocations of progress-dependent accounting calls in Moab
Moab synchronization of account information from MAM
MWS accounting queries from MAM

MAM 8.0

Accounting Mode in MAM
Commands Reference
MAM uses new Adaptive Standard Log Format
Implemented full accounting policies for custom script interface in Moab
Hybrid accounting interface in Moab
Grid Accounting Hardening in Moab

MAM 8.1

New command to query configuration files
Incremental balance now tracked in transaction table
Accounting Mode in Moab
Credential synchronization between Moab and MAM for Native (script) interface
Accounting Manager permits a finer granularity refresh period
Accounting Manager permits a finer granularity charge period
Ability to differentiate between failure action for periodic charging and after resuming a suspended job

Facebook Twitter Email

Speak Your Mind

*