Run Flink Jobs

It's very easy to run Apache Flink on Kubernetes with YuniKorn. Depending on which mode is used to run Flink on Kubernetes, the configuration is slight different.

Standalone mode

Please follow Kubernetes Setup to get details and examples of standalone deploy mode. In this mode, we can directly add required labels (applicationId and queue) in Deployment/Job spec to run flink application with YuniKorn scheduler, see the overview for the label specification.

Native mode

Please follow Native Kubernetes Setup to get details and examples of native deploy mode. Running flink application with YuniKorn scheduler in native mode is only supported for flink 1.11 or above, we can leverage two flink configurations kubernetes.jobmanager.labels and kubernetes.taskmanager.labels to set the required labels. Examples:

• Start a flink session

./bin/kubernetes-session.sh \
  -Dkubernetes.cluster-id=<ClusterId> \
  -Dtaskmanager.memory.process.size=4096m \
  -Dkubernetes.taskmanager.cpu=2 \
  -Dtaskmanager.numberOfTaskSlots=4 \
  -Dresourcemanager.taskmanager-timeout=3600000 \
  -Dkubernetes.jobmanager.labels=applicationId:MyOwnApplicationId,queue:root.sandbox \
  -Dkubernetes.taskmanager.labels=applicationId:MyOwnApplicationId,queue:root.sandbox

• Start a flink application

./bin/flink run-application -p 8 -t kubernetes-application \
  -Dkubernetes.cluster-id=<ClusterId> \
  -Dtaskmanager.memory.process.size=4096m \
  -Dkubernetes.taskmanager.cpu=2 \
  -Dtaskmanager.numberOfTaskSlots=4 \
  -Dkubernetes.container.image=<CustomImageName> \
  -Dkubernetes.jobmanager.labels=applicationId:MyOwnApplicationId,queue:root.sandbox \
  -Dkubernetes.taskmanager.labels=applicationId:MyOwnApplicationId,queue:root.sandbox \
  local:///opt/flink/usrlib/my-flink-job.jar