YuniKorn Scheduler Placement Rules

The basics for the placement rules are described in the scheduler design document. Multiple rules can be chained to form a placement policy. Access control lists and rule filters are defined per rule and enforced per rule. This document explains how to build a policy, including the rule usage, that is part of the scheduler with examples.

Configuration

Rules are defined per partition as part of the scheduler queue configuration. The order that the rules are defined in is the order in which they are executed. If a rule matches the policy will stop executing the remaining rules.

A matching rule generates a fully qualified queue name. This means that the name returned starts at the root queue. There is no limit on the number of levels in the queue hierarchy that can be generated.

When a rule is executed the result of rules that have been executed is unknown and not taken into account. Similar for rule that have not been executed yet: rules cannot influence other rules except when they are configured as the parent rule.

If the policy does not generate a queue name and no more rules are left the application will be rejected.

Basic structure for the rule placement definition in the configuration:

placementrules:
  - name: <name of the 1st rule>
  - name: <name of the 2nd rule>

Each rule can take a predefined set of parameters in the configuration. The name of the rules that can be used are given in the rule description. A rule name is not case sensitive in the configuration. Rule name must follow the following naming convention:

• start with a letter: a-z or A-Z
• followed by 0 or more characters a-z, A-Z, 0-9 or _

A rule that is not known, i.e. the name does not map to the rules defined below, will cause a initialisation error of the placement manager. Rules can also throw a parse error or an error during the initialisation if the parameters are incorrect. A rule set with an error can never become active.

A placement manager is considered initialised if it has an active rule set. When the configuration is reloaded a new rule set will be created to replace the active rule set. In the case that a new rule set loaded contains an error, i.e. is broken, the placement manager ignores the new rule set. This means that the placement manager stays in a the state it was in when a broken rule set is loaded. If the placement manager keeps using the existing active rule set in the case that it was already initialised. A message will be logged about the broken and ignored configuration.

Dots "." in the rule result are replaced by the string "_dot_". A dot is replaced because it is used as the hierarchy separator in the fully qualified queue name. Replacing the dot occurs before the full queue hierarchy is build and the result is qualified. This means that we allow user name and or tag values to contain dots without the dots affecting the queue hierarchy. For queues in the configuration that as an example must map to username with a dot you must specify them as follows: A user rule with the user user.name will generate the queue name root.user_dot_name as output. If that "user queue" must be added to the configuration the user_dot_name name should be used.

Create parameter

The create parameter is a boolean flag that defines if a queue that is generated by the rule may be created if it does not exist. There is no guarantee that the queue will be created because the existing queues might prevent the queue creation. If the queue generated by the rule does not exist and the flag is not set to true the result of the rule will be a fail.

Basic yaml entry for a rule with create flag:

placementrules:
  - name: <name of the rule>
    create: true

The default value is false. Allowed values: true or false, any other value will cause a parse error.

Parent parameter

The parent parameter allows specifying a rule that generates a parent queue for the current rule. Parent rules can be nested, a parent rule may contain another parent rule. There is no enforced limit of parent rules that can be nested.

A parent rule is treated as if it was a rule specified at the top level of the list and thus has the same parameters and requirements as a any other rule in the placement definition. The exception is that using a parent rule on a rule that already generates a fully qualified queue is considered a configuration error. This error can only occur on the rule of type fixed, see the rule specification for more details.

NOTE: the rule execution traverses down the list of parent rules and executes the last one in the list first. This means that the last parent rule will generate the queue directly below the root. See the examples for details.

Basic yaml entry for a rule with a parent rule:

placementrules:
  - name: <name of the rule>
    parent:
      name: <name of the parent rule>

The default value is no parent.

Filter parameter

The filter on a rule allows filtering the users that the rule applies to. A filter is a complex configuration object.

The users and groups that can be configured can be one of two types:

• a regular expression
• a list of users or groups.

If the entry for users or groups contains more than 1 entry in the yaml it is always considered a list of either users or groups. Duplicate entries in the lists are ignored and do not cause an error. Specifying a regular expression beside other list elements is not allowed.

User and group names follow the standard linux user and group conventions. For user names:

• start with a letter: a-z or A-Z
• followed by 0 or more characters a-z, A-Z, 0-9 or _ . @ -
• the last character may also be a $

For group names:

• start with a letter: a-z or A-Z
• followed by 0 or more characters a-z, A-Z, 0-9 or _ -

If the list is exactly one entry it can be either a single user or group or a regular expression. When an entry contains a character that is not allowed in a user or group name the entry is considered a regular expression. The regular expression must compile as specified. A regular expression that does not compile is ignored.

Specifically for the group regular expression: matching is performed one group at a time not the against the list of groups.

Basic filter yaml entry:

filter:
  type: deny
  users:
    - <user name or regexp>
    - <user name>
  groups:
    - <group name or regexp>
    - <group name>

The default value is no filter.

Value parameter

This is a generic value that can be used to pass to a rule to implement or alter its behaviour. The value It is used by the fixed and the tag rule. The value is a single value in string form and is not interpreted or manipulated by the system.

Basic yaml entry for a rule with a value set:

placementrules:
  - name: <name of the rule>
    value: "any string"

The default is no value.

Access Control List

Access control lists are not defined in the rules but they impact the outcome of the placement policy. Two access control lists can be defined on a queue:

1. Submit ACL: submitacl
2. Administration ACL: adminacl

The placement rule will only match if the ACL of the queue allows submit access via either ACL. The administrative queue ACL also provides submit access. If the queue does not exist or does not have an ACL set, the ACL of the parent queue is checked. This recursive check is repeated until an ACL provides access or after the ACL of the root queue is checked.

For more detail on the ACL syntax check the ACL documentation.

Rules

Provided Rule

Name to be used in the configuration: provided

Returns the queue provided during the submission of the application. The behaviour of the this rule is to fully qualify the queue provided by the application if the queue is not fully qualified. If a parent rule is set and the queue provided in the application submission is fully qualified then the parent rule will not be executed.

Supported parameters:

• create
• parent
• filter

Example: create the queue passed in by the user if it does not exist under the users name

placementrules:
  - name: provided
    create: true
    parent:
      name: user
      create: true

Application submit request by the user developer, queue in the application on submit: my_special_queue
Result: root.developer.my_special_queue (parent rule set the user name)

Application submit request by the user developer, queue in the application on submit: root.dev_queue
Result: root.dev_queue (parent rule ignored)

User Name Rule

Name to be used in the configuration: user

Returns the queue based on the user name that is part of the submitted application.

Supported parameters:

• create
• parent
• filter

Example: submit to a queue based on the user name, do not create the queue if the queue does not exist:

placementrules:
  - name: user
    create: false

Application submit request by the user finance.test, queue does exist:
Result: root.finance_dot_test (note the replacement of the dot)

Application submit request by the user developer, queue does not exist:
Result: failed, next rule executed

Fixed Rule

Name to be used in the configuration: fixed

Returns the name configured in the rule parameter value. The value configured must be a legal queue name or queue hierarchy. The name does not have to be a fully qualified queue name. The hierarchy in the name uses a dot as a separator for the queue names at the different levels in the hierarchy. The fixed rule can only fail if the queue configured does not exist and the create flag is not set as it will always return the configured queue.

If the configured value is a fully qualified queue name configuring a parent rule will be considered an error.

Supported parameters:

• value (required)
• create
• parent
• filter

Example: a fixed queue returned always, without the create flag set and thus requires the queue to be defined as a leaf queue in the configuration.

placementrules:
  - name: fixed
    value: last_resort

Application submit request by the user developer, queue in the application on submit: my_special_queue
Result: root.last_resort

Tag Rule

Name to be used in the configuration: tag Retrieves the queue name from the applications tags. The tag name that is checked for its value is configured in the rule using the value. Configuring a tag rule without a value set is an error, however an application does not have to have the tag set.

If the tag is not set on the application the rule fails. If the tag value returned from the application is a fully qualified queue name the parent rule, if configured, will not be executed.

Supported parameters:

• value (required)
• create
• parent
• filter

Example: place an application based on the kubernetes namespace which gets sets automatically in the application when submitted:

placementrules:
  - name: tag
    value: namespace
    create: true

Application submit request for a kubernetes based application in the namespace default by the user developer, queue in the application on submit: my_special_queue
Result: root.default

Application submit request for a kubernetes based application in the namespace testing by the user developer
Result: root.testing

Application submit request for a non kubernetes based application by the user developer
Result: failed, next rule executed

Complex examples

In this complex example we chain three rules:

1. a user rule, with a parent rule tag using the kubernetes namespace, to be used only for users that are part of and "dev" group.
2. a tag rule using the kubernetes namespace, with a parent rule fixed using the existing queue root.namespaces.
3. a fixed rule to place everything that reaches this point in the root.default queue.

Example:

placementrules:
  - name: user
    create: true
    filter:
      type: allow
      groups:
        - dev*
    parent:
      - name: tag
        value: namespace
  - name: tag
    value: namespace
    create: true
    parent:
      - name: fixed
        value: root.namespaces
        filter:
          type: allow
          users:
            - john
  - name: fixed
    value: root.default

Application submit request for a kubernetes based application in the namespace testing by the user john
Result: root.namespaces.testing (matched in rule 2)

Application submit request for a kubernetes based application in the namespace newapp by the user sarah with groups membership sarah, test_app, dev_app
Result: root.newapp.sarah (matched in rule 1)

Application submit request for a kubernetes based application in the namespace testapp by the user bob with groups membership bob
Result: root.deault (matched in rule 3)

In this second example we chain two rules:

1. a fixed rule to place everything in the root.production queue
2. a user rule, with the create flag set

In this case however we have ACLs set up on the root.production queue to only allow two specific user to use this queue. So even if the rule matches unless the user is either john or bob the application will not be placed in the production queue. All other users will match the second rule and use their own queue, which is created if it does not exist.

partitions:
  - name: default
    queues:
      - name: production
        submitacl: john,bob
    placementrules:
      - name: fixed
        value: root.production
      - name: user
        create: true