Service Configuration
The official distribution of YuniKorn is deployed via Helm charts to Kubernetes. Configuration for YuniKorn is split into two parts: Helm configuration, and YuniKorn service configuration.
Helm Configuration
Helm configuration is used to configure options for the deployment of YuniKorn to Kubernetes.
The following settings can be configured during YuniKorn installation
via Helm, either via Helm's command-line, as in --set key=value
, or
via an external file: -f file.yaml
. The examples below will be given in
YAML syntax.
Container images
YuniKorn ships as a set of container images. The locations and pull policies can be customized as follows:
# Image information for the standard scheduler
image:
repository: apache/yunikorn
tag: scheduler-1.0.0 # default depends on YuniKorn version
pullPolicy: Always
# Image information for the plugin scheduler
pluginImage:
repository: apache/yunikorn
tag: scheduler-plugin-1.0.0 # default depends on YuniKorn version
pullPolicy: Always
# Image information for the web UI
web:
image:
repository: apache/yunikorn
tag: web-1.0.0 # default depends on YuniKorn version
pullPolicy: Always
# Image information for the admission controller
admissionController:
image:
repository: apache/yunikorn
tag: admission-1.0.0 # default depends on YuniKorn version
pullPolicy: Always
Kubernetes configuration
affinity
Sets the affinity for the YuniKorn scheduler pod.
Default: {}
Example:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- primary1
- primary2
admissionController.affinity
Sets the affinity for the YuniKorn admission controller pod.
Default: {}
Example:
admissionController:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- primary1
- primary2
hostNetwork
Controls whether the scheduler should run in the host network.
Default: false
Example:
hostNetwork: true
admissionController.hostNetwork
Controls whether the admission controller should run in the host network.
Default: true
Example:
admissionController:
hostNetwork: false
imagePullSecrets
Provides secrets needed for pulling YuniKorn images.
Default: []
Example:
imagePullSecrets:
- secret1
- secret2
nodeSelector
Sets a node selector(s) to use for placement of the YuniKorn scheduler pod.
Default: {}
Example:
nodeSelector:
node-role.kubernetes.io/control-plane: "true"
admissionController.nodeSelector
Sets a node selector(s) to use for placement of the YuniKorn admission controller pod.
Default: {}
Example:
admissionController:
nodeSelector:
node-role.kubernetes.io/control-plane: "true"
admissionController.replicaCount
Sets the number of replicas to use for the YuniKorn admission controller. This can be set to greater than 1 for high-availability.
Default: 1
Example:
admissionController:
replicaCount: 2
serviceAccount
Sets an alternate service account for the YuniKorn scheduler.
Changing this value is not recommended, as Helm installs role-based access control (RBAC) policies for the default user that are required for proper functionaliy.
Default: yunikorn-admin
Example:
serviceAccount: my-account
admissionController.serviceAccount
Sets an alternate service account for the YuniKorn admission controller.
Changing this value is not recommended, as Helm installs role-based access control (RBAC) policies for the default user that are required for proper functionaliy.
Default: yunikorn-admission-controller
Example:
admissionController:
serviceAccount: my-account
service.type
Sets the type of service used for the scheduler.
Default: ClusterIP
Example:
service:
type: ClusterIP
admissionController.service.type
Sets the type of service used for the admission controller.
Default: ClusterIP
Example:
admissionController:
service:
type: ClusterIP
service.port
Sets the port exposed in the YuniKorn scheduler service for the REST API. It is not recommended to change this value.
Default: 9080
Example:
service:
port: 9080
service.portWeb
Sets the port exposed in the YuniKorn scheduler service for the Web UI. It is not recommended to change this value.
Default: 9889
Example:
service:
portWeb: 9889
tolerations
Sets the tolerations for the YuniKorn scheduler pod.
Default: []
Example:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Equal
value: "true"
effect: NoSchedule
- key: CriticalAddonsOnly
operator: Exists
admissionController.tolerations
Sets the tolerations for the YuniKorn admission controller pod.
Default: []
Example:
admissionController:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Equal
value: "true"
effect: NoSchedule
- key: CriticalAddonsOnly
operator: Exists
podLabels
Sets the labels for the YuniKorn scheduler pod.
Default: {}
Example:
podLabels:
app.kubernetes.io/name: scheduler
app.kubernetes.io/part-of: yunikorn
admissionController.podLabels
Sets the labels for the YuniKorn admission controller pod.
Default: {}
Example:
admissionController:
podLabels:
app.kubernetes.io/name: admission-controller
app.kubernetes.io/part-of: yunikorn
podAnnotations
Sets the annotations for the YuniKorn scheduler pod.
Default: {}
Example:
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/path: /ws/v1/metrics
prometheus.io/port: 9080
admissionController.podAnnotations
Sets the annotations for the YuniKorn admission controller pod.
Default: {}
Example:
admissionController:
podAnnotations:
example.com/admission: "false"
Resource utilization
The resources requested for YuniKorn pods can be customized as follows:
# Scheduler container resources
resources:
requests:
cpu: 200m
memory: 1Gi
limits:
cpu: 4
memory: 2Gi
# Web UI container resources
web:
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 100m
memory: 500Mi
# Admission controller resources
admissionController:
resources:
requests:
cpu: 100m
memory: 500Mi
limits:
cpu: 500m
memory: 500Mi
Optional features
embedAdmissionController
Controls whether to enable the YuniKorn admission controller.
Default: true
Example:
embedAdmissionController: false
enableSchedulerPlugin
Controls whether to run YuniKorn in scheduler plugin mode.
Default: false
Example:
enableSchedulerPlugin: true
YuniKorn defaults
yunikornDefaults
Sets entries which will be rendered to the yunikorn-defaults
ConfigMap. This
can be used to pre-configure YuniKorn at deployment time. Any settings
declared in YuniKorn configuration may be set here.
Default: {}
Example:
yunikornDefaults:
service.clusterId: yunikorn-01
service.policyGroup: group-01
group-01.yaml: |
partitions:
- name: default
placementrules:
- name: tag
value: namespace
create: true
queues:
- name: root
submitacl: '*'
Deprecated settings
The following settings are deprecated, and will be removed from a future
YuniKorn release. They should now be specified in the yunikorn-configs
ConfigMap
or via the Helm yunikornDefaults
section:
Deprecated setting | ConfigMap replacement |
---|---|
operatorPlugins | - |
placeHolderImage | service.placeholderImage |
admissionController: processNamespaces | admissionController.filtering.processNamespaces |
admissionController: bypassNamespaces | admissionController.filtering.bypassNamespaces |
admissionController: labelNamespaces | admissionController.filtering.labelNamespaces |
admissionController: noLabelNamespaces | admissionController.filtering.noLabelNamespaces |
configuration | queues.yaml |
Deprecated example:
operatorPlugins: general
placeHolderImage: registry.k8s.io/pause:3.7
admissionController:
processNamespaces: "^spark-,^mpi-"
bypassNamespaces: "^kube-system$"
labelNamespaces: "^spark-"
noLabelNamespaces: "^mpi-legacy-"
configuration: |
partitions:
- name: default
placementrules:
- name: tag
value: namespace
create: true
queues:
- name: root
submitacl: '*'
Replacement example:
yunikornDefaults:
service.policyGroup: queues
service.placeholderImage: registry.k8s.io/pause:3.7
admissionController.filtering.processNamespaces: "^spark-,^mpi-"
admissionController.filtering.bypassNamespaces: "^kube-system$"
admissionController.filtering.labelNamespaces: "^spark-"
admissionController.filtering.noLabelNamespaces: "^mpi-legacy-"
queues.yaml: |
partitions:
- name: default
placementrules:
- name: tag
value: namespace
create: true
queues:
- name: root
submitacl: '*'
Currently, if both the deprecated parameter and the replacement ConfigMap entry are specified, the ConfigMap entry will take precedence.
YuniKorn Configuration
Service configuration for YuniKorn is controlled by two Kubernetes ConfigMaps
in the namespace where YuniKorn is installed: yunikorn-defaults
and
yunikorn-configs
.
At runtime, these ConfigMaps are polled by YuniKorn and merged together to form an
effective configuration. If a setting is present in both ConfigMaps, the
yunikorn-configs
setting will override the one present in yunikorn-defaults
.
The purpose of yunikorn-defaults
is to provide a mechanism for Helm to configure
initial service configuration details. It should not be modified directly.
The yunikorn-configs
ConfigMap is completely unmanaged by Helm, and is meant for
configurations which may change over time, such as queue configuration. All changes
to YuniKorn configuration outside of provisioning infrastructure should be made here.
Default ConfigMap
If neither ConfigMap is provided, or if an option is not specified, YuniKorn will use the default values listed here:
apiVersion: v1
kind: ConfigMap
metadata:
name: yunikorn-configs
data:
service.clusterId: "mycluster"
service.policyGroup: "queues"
service.schedulingInterval: "1s"
service.volumeBindTimeout: "10s"
service.eventChannelCapacity: "1048576"
service.dispatchTimeout: "5m"
service.disableGangScheduling: "false"
service.enableConfigHotRefresh: "true"
service.placeholderImage: "registry.k8s.io/pause:3.7"
service.instanceTypeNodeLabelKey: "node.kubernetes.io/instance-type"
health.checkInterval: "30s"
log.level: "INFO"
kubernetes.qps: "1000"
kubernetes.burst: "1000"
admissionController.webHook.amServiceName: "yunikorn-admission-controller-service"
admissionController.webHook.schedulerServiceAddress: "yunikorn-service:9080"
admissionController.filtering.processNamespaces: ""
admissionController.filtering.bypassNamespaces: "^kube-system$"
admissionController.filtering.labelNamespaces: ""
admissionController.filtering.noLabelNamespaces: ""
admissionController.filtering.generateUniqueAppId: "false"
admissionController.filtering.defaultQueue: "root.default"
admissionController.accessControl.bypassAuth: "false"
admissionController.accessControl.trustControllers: "true"
admissionController.accessControl.systemUsers: "^system:serviceaccount:kube-system:"
admissionController.accessControl.externalUsers: ""
admissionController.accessControl.externalGroups: ""
queues.yaml: |
partitions:
- name: default
placementrules:
- name: tag
value: namespace
create: true
queues:
- name: root
submitacl: '*'
Service settings
The following parameters are understood by YuniKorn:
service.clusterId
Sets an identifier for the cluster being configured. This is returned as part of REST API calls.
A change to this setting requires a restart of YuniKorn to take effect.
Default: mycluster
Example:
service.clusterId: "yunikorn-east"
service.policyGroup
Defines the policy group in use by this scheduler. The policy group is used to
choose one of several queue configurations. The value of this setting plus an
extension of .yaml
controls the ConfigMap entry used to retrieve partition
and queue configuration.
A change to this setting requires a restart of YuniKorn to take effect.
Default: queues
Example:
service.policyGroup: group_b
group_b.yaml: |
partitions:
- name: default
placementrules:
- name: tag
value: namespace
create: true
queues:
- name: root
submitacl: '*'
service.schedulingInterval
Controls the frequency with which YuniKorn executes scheduling runs.
A change to this setting requires a restart of YuniKorn to take effect.
Default: 1s
Example:
service.schedulingInterval: "5s"
service.volumeBindTimeout
Controls the timeout before volume binding fails.
A change to this setting requires a restart of YuniKorn to take effect.
Default: 10s
Example:
service.volumeBindTimeout: "30s"
service.eventChannelCapacity
Controls the number of internal scheduling events that YuniKorn will allow to be in-flight at one time. This acts as an out-of-memory guard.
A change to this setting requires a restart of YuniKorn to take effect.
Default: 1048576
Example:
service.eventChannelCapacity: "1000000"
service.dispatchTimeout
Controls how long internal events will reattempt dispatching if the event channel is full. Warnings will be emitted if this timeout is exceeded.
A change to this setting requires a restart of YuniKorn to take effect.
Default: 5m
Example:
service.dispatchTimeout: "10m"
service.disableGangScheduling
Allows global disabling of the gang scheduling feature (not recommended).
A change to this setting requires a restart of YuniKorn to take effect.
Default: false
Example:
service.disableGangScheduling: "true"
service.enableConfigHotRefresh
Controls whether configuration should be hot-reloaded. By default, this
is set to true
, but it can be disabled to avoid changes to the
ConfigMaps from being picked up until a scheduler restart.
A change to this setting will be picked up without a restart of YuniKorn.
NOTE: If this setting is disabled, it may not be re-enabled again without a restart of YuniKorn.
Default: true
Example:
service.enableConfigHotRefresh: "false"
service.placeholderImage
Sets the Pod image that will be used for gang scheduling placeholders.
A change to this setting requires a restart of YuniKorn to take effect.
Default: registry.k8s.io/pause:3.7
Example:
service.placeholderImage: "registry.k8s.io/pause:3.6"
service.instanceTypeNodeLabelKey
Sets the node label that will be used to determine the instance type of node.
A change to this setting requires a restart of YuniKorn to take effect.
Default: node.kubernetes.io/instance-type
Example:
service.instanceTypeNodeLabelKey: "node.kubernetes.io/my-instance-type"
Event system settings
event.trackingEnabled
Enables or disables the event system and event generation.
Default: true
Example:
event.trackingEnabled: "false"
event.ringbufferCapacity
Sets the capacity of the ring buffer which stores Yunikorn generated events.
Default: 10000
Example:
event.ringbufferCapacity: "300000"
event.maxStreams
Sets the maximum number of event stream connections.
Default: 100
Example:
event.maxStreams: "50"
event.maxStreamsPerHost
Sets the maximum number of event stream connections from a given host.
Default: 15
Example:
event.maxStreamsPerHost: "5"
Health settings
health.checkInterval
Sets the time between automatic health checks of YuniKorn.
Setting the value to 0
or a negative interval will disable background health
checking.
A change to this setting will be picked up without a restart of YuniKorn.
Default: 30s
Example:
health.checkInterval: "1m"
Log settings
log.level
Sets the default verbosity that YuniKorn will log at.
A change to this setting will be picked up without a restart of YuniKorn. The available values can be numeric or textual:
-1
/debug
/DEBUG
0
/info
/INFO
1
/warn
/WARN
2
/error
/ERROR
3
/dpanic
/DPANIC
4
/panic
/PANIC
5
/fatal
/FATAL
Default: INFO
Example:
log.level: "DEBUG"
log.{subsystem}.level
Sets the verbosity that YuniKorn subsystem will log at.
Yunikorn allows fine-grained logging configuration in a hierarchical manner. For example,
setting an entry for log.core.level
will configure all loggers that start with core.
(including core.scheduler
, etc.) unless a more specific configuration is present.
Each subsystem1 has its log level.
A change to this setting will be picked up without a restart of YuniKorn. The available values can be numeric or textual:
-1
/debug
/DEBUG
0
/info
/INFO
1
/warn
/WARN
2
/error
/ERROR
3
/dpanic
/DPANIC
4
/panic
/PANIC
5
/fatal
/FATAL
Default: INFO
Example:
The log.level
is the default log level for all loggers.
log.level: "INFO"
log.admission.level: "DEBUG"
log.core.config.level: "INFO"