码蚁

打码改变人生

Prometheus 监控 Kubernetes 并使用 Grafana 进行展示

Posted at — Jul 3, 2018

本文将总结一下我们目前使用 Prometheus 对 Kubernetes 集群监控的实践。 我们选择 Prometheus 作为监控系统主要在以下各层面实现监控:

基础设施层:监控各个主机服务器资源(包括 Kubernetes 的 Node 和非 Kubernetes 的Node ),如 CPU,内存,网络吞吐和带宽占用,磁盘 I/O 和磁盘使用等指标。 中间件层:监控独立部署于 Kubernetes 集群之外的中间件,例如:MySQL、Redis、RabbitMQ、ElasticSearch、Nginx等。 Kubernetes集群:监控Kubernetes集群本身的关键指标 Kubernetes集群上部署的应用:监控部署在Kubernetes集群上的应用

环境说明

Prometheus 介绍

要实现对 Kubernetes 集群的监控,因为 Kubernetes 的 rbac 机制以及证书认证,当然是把 Prometheus 部署在 Kubernetes 集群上最方便。可是我们目前的监控系统是以 Kubernetes 集群外部的 Prometheus 为主的,Grafana 和告警都是使用这个外部的 Prometheus,如果还需要在 Kubernetes 集群内部部署一个 Prometheus 的话一定要把它桶外部的 Prometheus 联合起来,好在 Prometheus 支持 Federation。

Federation 允许一个 Prometheus 从另一个 Prometheus 中拉取某些指定的时序数据。Federation 是 Prometheus 提供的扩展机制,允许 Prometheus 从一个节点扩展到多个节点,实际使用中一般会扩展成树状的层级结构。下面是 Prometheus 官方文档中对 Federation 的配置示例:

- job_name: 'federate'
  scrape_interval: 15s

  honor_labels: true
  metrics_path: '/federate'

  params:
    'match[]':
      - '{job="prometheus"}'
      - '{__name__=~"job:.*"}'

  static_configs:
    - targets:
      - 'source-prometheus-1:9090'
      - 'source-prometheus-2:9090'
      - 'source-prometheus-3:9090'

这段配置所属的 Prometheus 将从 source-prometheus-1 ~ 3 这 3 个Prometheus 节点的 /federate 端点拉取监控数据。 match[]参数指定了只拉取带有 job="prometheus” 标签的指标,或者名称以 job 开头的度量指标(标签匹配器也可以通过匹配内部__name__标签来应用到度量标准名称 。例如,该表达式 http_requests_total{} 等同于 {__name__="http_requests_total”})。

Prometheus 在 Kubernetes 中部署

前面已经介绍了将使用 Prometheus Federation的形式,Kubernetes 集群外部的 Prometheus 从 Kubernetes 集群中 Prometheus 拉取监控数据,外部的 Prometheus 才是监控数据的存储。Kubernetes 集群中部署 Prometheus 的数据存储层可以简单的使用 emptyDir,数据只保留 24 小时(或更短时间)即可,部署在 Kubernetes 集群上的这个 Prometheus 实例即使发生故障也可以放心的让它在集群节点中漂移。

在 Kubernetes 上部署 Prometheus 十分简单,只需要下面 4 个文件:prometheus.rbac.yml, prometheus.config.yml, prometheus.deploy.yml, prometheus.svc.yml。 下面给的例子中将 Prometheus 部署到 kube-system 命名空间。

prometheus.rbac.yml 定义了 Prometheus 容器访问 Kubernetes apiserver所需的 ServiceAccount 和 ClusterRole 及ClusterRoleBinding,参考 Prometheus 源码中库中的例子:

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics", "/healthz"] # 有权访问的部分网址,如果需要访问其他 url 获取指标,应在此给予权限
  verbs: ["get"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: kube-system

prometheus.config.yml configmap中的prometheus的配置文件,参考 Prometheus 源码中库中的例子:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config-v1
  namespace: kube-system
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
    scrape_configs:
    
    - job_name: 'kubernetes-kubelet'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
    
    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    
    - job_name: 'kubernetes-pod'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        regex: true
        action: keep # 筛选出带有 prometheus.io/scrape: 'true' 注释的 pod 获取指标。
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+) # 获取指标不是默认的 /metrics 自定义路径(别忘记添加上一步的 url 权限)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__ # 指定端口

      - source_labels: ['__meta_kubernetes_pod_label_app', '__meta_kubernetes_pod_node_name']
        regex: 'node-exporter;(.*)'
        action: replace
        target_label: nodename

prometheus.deploy.yml 定义 Prometheus 的部署: 这边有做固定到某个 node,先给这个 node 打标签 kubectl label nodes 172-25-134-47 app=prometheus

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: prometheus-deployment
  name: prometheus
  namespace: kube-system

spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      nodeSelector: # 选择带有这个标签的 node
        app: prometheus

      containers:
      - image: prom/prometheus:v2.2.1
        name: prometheus
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention=4h"
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts:
        - mountPath: "/prometheus" # 挂载数据
          name: data
        - mountPath: "/etc/prometheus" # 挂载配置
          name: config-volume
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 500m
            memory: 2500Mi
      serviceAccountName: prometheus
      imagePullSecrets: 
        - name: regsecret
      volumes:
      - name: data # 声明数据卷
        emptyDir: {}
      - name: config-volume # 声明配置卷
        configMap:
          name: prometheus-config-v1

prometheus.svc.yml 定义 Prometheus 的 Service,需要将 Prometheus 以NodePort , LoadBalancer 或使用 Ingress 暴露到集群外部,这样外部的 Prometheus 才能访问它(这里采用 NodePort):

---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: prometheus
  name: prometheus
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 9090
    targetPort: 9090
    nodePort: 30003
  selector:
    app: prometheus

配置外部 Prometheus Federation

完成 Kubernetes 集群上的 Prometheus 的部署之后,下面将配置集群外部的 Prometheus 使其从集群内部的 Prometheus 拉取数据。 实际上只需以静态配置的形式添加一个 job 就可以:

  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~"kubernetes-.*"}'
    static_configs:
      - targets:
        - '172.25.134.47:30003'

注意上面的配置是外部 Prometheus 拉取 Kubernetes 集群上面的 Prometheus 所有名称以 kubernetes- 的 job 的监控数据。

Kubernetes 集群上部署应用的监控

Kubernetes 集群上部署应用的监控需要从三个方面:

这里将主要介绍 node_exporter 和 kube-state-metrics,而对于应用内部的监控实践后边有时间再单独总结。kube-state-metrics 使用 kubernetes 的 go 语言客户端 client-go 可以从 Kubernetes 集群中获取各种资源对象的指标。

在 Kubernetes 上部署 node_exporter

kind: DaemonSet
apiVersion: apps/v1
metadata: 
  name: node-exporter 
  namespace: kube-system
spec:
  selector:
    matchLabels: 
      app: node-exporter
  template:
    metadata:
      name: node-exporter
      labels: 
        app: node-exporter
      annotations: 
        prometheus.io/scrape: 'true' # 能被集群内 Prometheus 获取指标的关键表示符
        prometheus.io/port: '9100'
        prometheus.io/path: '/metrics'
    spec:
      tolerations: # 因 Master 节点具有 Taints(污点)属性,这里要设置 Tolerations(容忍)使得 Master 节点运行 node_exporter
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      volumes:
      - name: proc
        hostPath: {path: /proc}
      - name: sys
        hostPath: {path: /sys}
      containers:
      - name: node-exporter
        image: quay.io/prometheus/node-exporter:v0.15.2 # 因最新版 0.16 某些指标值有所改变,为了配合后面的 Grafana Dashboard,这边选用 0.15.2
        args: [--path.procfs=/proc_host, --path.sysfs=/host_sys]
        ports:
        - {name: node-exporter, hostPort: 9100, containerPort: 9100}
        volumeMounts:
        - {name: sys, readOnly: true, mountPath: /host_sys}
        - {name: proc, readOnly: true, mountPath: /proc_host}
        imagePullPolicy: IfNotPresent
      restartPolicy: Always
      hostNetwork: true
      hostPID: true

在 Kubernetes 上部署 kube-state-metrics:

$ vi kube-state-metrics.rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: kube-system
  name: kube-state-metrics-resizer
rules:
- apiGroups: [""]
  resources:
  - pods
  verbs: ["get"]
- apiGroups: ["extensions"]
  resources:
  - deployments
  resourceNames: ["kube-state-metrics"]
  verbs: ["get", "update"]
---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-state-metrics
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-state-metrics-resizer
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
  namespace: kube-system
rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
---

apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system


$ vi kube-state-metrics.deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata: 
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels: 
      "app": "kube-state-metrics"
  replicas: 1
  template:
    metadata:
      labels: 
        "app": "kube-state-metrics"
      annotations:
        prometheus.io/scrape: 'true' # 能被集群内 Prometheus 获取指标的关键表示符
        prometheus.io/port: '8080'
        prometheus.io/path: '/metrics'
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v1.3.1
        ports:
        - name: http-metrics
          containerPort: 8080
        readinessProbe:
          httpGet: 
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5

关于 kube-state-metrics 暴露的所有监控指标可以参考 kube-state-metrics 的文档kube-state-metrics Documentation

Grafana Dashboard

关于 Kubernetes Grafana Dashboard 我们安装插件 Grafana-Kubernetes-app

安装好以后配置添加需要监控 Kubernetes 集群。

Grafana-Kubernetes-app

配置如上图,图中的 CA 证书,Client 证书,Client key 使用 kubectl 配置文件中的即可,最后点击 Save 而不是 Deploy,因为 Grafana-Kubernetes-app 需要的监控指标上面我们已经手动部署了。

Grafana-Kubernetes-app 配置完成后就在 Grafana Home Dashboard 看到新增加 Dashboard 了,至此监控图完成

参考

comments powered by Disqus