Setting Prometheus
YuniKorn exposes its scheduling metrics via Prometheus. Thus, we need to set up a Prometheus server to collect these metrics.
If you don't know what metric can be used, you can use REST API.
1. Download Prometheus release
wget https://github.com/prometheus/prometheus/releases/download/v2.30.3/prometheus-2.30.3.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*
2. Configure prometheus.yml
Prometheus collects metrics from targets by scraping metrics HTTP endpoints.
global:
scrape_interval: 3s
evaluation_interval: 15s
scrape_configs:
- job_name: 'yunikorn'
scrape_interval: 1s
metrics_path: '/ws/v1/metrics'
static_configs:
- targets: ['localhost:9080']
# 9080 is internal port, need port forward or modify 9080 to service's port
3. Start port-forward
Port forwarding for the core's web service on the standard port can be turned on via:
kubectl port-forward svc/yunikorn-service 9080:9080 -n yunikorn
9080
is the default port for core's web service.
4. Execute prometheus
./prometheus --config.file=prometheus.yml
5. Access the Prometheus UI
You should be able to browse to a status page at localhost:9090. Give it a couple of seconds to collect data about itself from its own HTTP metrics endpoint.
You can also verify that Prometheus is serving metrics by navigating to its metrics endpoint:localhost:9090/metrics