Skip to content
Kubernetes

Monitoring Applications in Kubernetes

k8sprometheusgrafana
On this page

Prometheus

Key Concepts

Metrics: The basic unit of data collection in Prometheus. Each metric has a specific name and a set of labels, and records changes in its value over time.

Labels: Key-value pairs used to distinguish and query metrics. For example, when collecting metrics from a server, you can use labels such as “server location” and “server name.”

Time Series: A sequence of data points that share the same metric name and set of labels. Prometheus uses this time series data to track changes in metrics over time.

Query: Prometheus uses its own query language, PromQL, to retrieve and calculate stored data. This allows users to write complex queries to extract useful information from metric data.

Alerts: Prometheus can trigger alerts based on defined conditions. Users can set up rules to receive notifications when a specific metric exceeds a given threshold or reaches a specific state.

Exporters: Prometheus uses agents called Exporters to collect metrics from various systems and services. Each Exporter collects metrics for a specific service or system and converts them into a format that Prometheus can understand.

Scraping: The Prometheus server “scrapes” metrics by sending HTTP requests to Exporters or target systems at configured intervals. Through this process, Prometheus regularly collects and stores metric data.

Storage: Prometheus stores the collected data in local storage in the form of a time-series database. This data is later used for querying, visualization, and generating alerts.

Types of Metrics

The metrics used in Prometheus are primarily classified into four types. Each metric type is used to monitor and measure different kinds of information:

Counter:

  • A counter is a metric that simply increases in value; it is primarily used to measure the number of requests, completed tasks, or errors.

  • Counter values never decrease (except in cases such as a restart) and are primarily used to observe cumulative values.

Gauge:

  • A gauge is a metric that measures a value at a specific point in time; the value can increase or decrease.

  • They are primarily used to monitor values that represent the current state, such as memory usage, the number of active sessions, and temperature.

Histogram:

  • A histogram allows you to observe the distribution of samples by observing them and classifying them into predefined buckets.

  • It is primarily used to measure response times, request sizes, and other metrics, with each bucket representing a specific range of values.

  • A histogram provides the total sum of measured values and the cumulative count for each bucket.

Summary:

  • A summary is also used to measure the distribution of samples, but unlike a histogram, it provides observed values at specific quantiles.

  • It is useful for calculating metrics such as the 90th percentile of response time.

  • A summary provides the sum of observed values, the count, and values based on specified quantiles.

These types of metrics can be selected and used according to different monitoring objectives and form the foundation of Prometheus’s flexibility and powerful data collection and query capabilities.

Types of Exporters

  • Node Exporter

  • Client Library

  • DCGM

Jobs and Instances

Jobs:

  • A job refers to a group of metrics collected by Prometheus. Typically, a job is a label used when collecting metrics from multiple instances of the same type.

  • For example, when collecting metrics from the same application running on multiple servers, you can use the "job" label to represent the entire group of servers. All servers can belong to a job named "api-server," and Prometheus collects metrics from this job.

  • A job is defined in the Prometheus configuration. This tells Prometheus which endpoints to collect metrics from and under which job name to group those metrics.

Instances:

  • An instance refers to an entity from which metrics are collected individually within a job. It typically refers to a single application instance or a single server.

  • For example, if the “api-server” job is running on multiple servers, each server is considered a separate instance of the “api-server” job. - Prometheus collects metrics independently from each instance and distinguishes these metrics using instance labels. An instance typically includes a hostname and port number, which indicate the exact location where Prometheus collects metrics.

PromQL

https://prometheus.io/docs/prometheus/latest/querying/basics/

Prometheus Operator

Definition

The Prometheus Operator is a tool designed to make it easier to deploy and manage Prometheus within a Kubernetes cluster. It is tightly integrated with Kubernetes resources and extends the Kubernetes API to automate and simplify Prometheus configuration. The following are the key concepts of the Prometheus Operator:

Custom Resource Definitions (CRDs):

  • The Prometheus Operator uses Custom Resource Definitions (CRDs) to allow Prometheus-related configurations to be managed as native Kubernetes resources. This enables users to configure Prometheus instances, Alertmanager, service monitoring, and more using the same methods used to define and manipulate Kubernetes resources.

Prometheus Custom Resource:

  • The Prometheus resource defines the Prometheus instance to be run within the cluster. Users can use this resource to specify the Prometheus server version, configuration, storage requirements, and more.

ServiceMonitor:

  • The ServiceMonitor resource defines how Prometheus discovers and configures the services it will monitor. It is based on service labels and specifies which services Prometheus will scrape, as well as which endpoints and ports to use.

Alertmanager Custom Resource:

  • The Alertmanager resource allows you to configure Alertmanager instances running within the cluster. This enables you to manage alerts from Prometheus and configure alert routing, grouping, deduplication, and delivery methods.

PrometheusRule:

  • The PrometheusRule resource defines alerting rules and recording rules. Using this resource, users can configure alerting rules based on metrics and rules for generating new time series data using the Prometheus Query Language (PromQL).

Operator Pattern:

  • Prometheus Operator follows the Kubernetes Operator pattern. This pattern implements the logic for running complex applications on Kubernetes as code, making it easier for users to deploy and manage applications.

Grafana

Grafana is a powerful visualization and analytics platform that collects data from various sources and visualizes it in the form of interactive dashboards. Users can easily view, monitor, and analyze data using Grafana. The following are key concepts in Grafana:

Dashboards:

  • A dashboard is a collection of data visualized through multiple widgets or panels. Dashboards allow users to view information from various data sources at a glance.

  • Each dashboard can be configured to reflect a specific purpose or perspective, and users can create and manage multiple dashboards as needed.

Panels:

  • Panels are components within a dashboard that display individual visualizations such as charts, graphs, and tables. Each panel visualizes data based on a specific query and supports various visualization types (e.g., graphs, heatmaps, gauges, etc.).

Data Sources:

  • Grafana can integrate with a variety of data sources. It supports integration with various databases and monitoring tools, such as Prometheus, InfluxDB, Elasticsearch, MySQL, and PostgreSQL.

  • Users can add data sources to Grafana, retrieve data from those sources, and create dashboards.

Query:

  • To visualize data within a panel, Grafana executes queries against the data source. These queries are defined by the user and determine which data is displayed on the dashboard.

Alerts:

  • Grafana provides a feature that sends notifications when data points exceed predefined thresholds. This allows users to quickly identify and respond to system issues.

Plugins:

  • Grafana supports a variety of plugins to extend its functionality. You can install data source plugins, panel plugins, app plugins, and more to expand and customize Grafana’s capabilities.

User and Permissions Management:

  • Grafana supports user account management, user groups, and role-based access control. This enables granular access control over dashboards, panels, and data sources.

  • Through these features, Grafana makes it easy to visualize complex data and enables efficient data analysis and monitoring.

k8s Cluster Summary Dashboard Description

Cluster Summary

Panel NameDescriptionNumber of healthy nodesNumber of nodes with status "Ready"Number of nodes with status other than "Ready"Number of deployments relative to the number of pods that can be createdNumber of currently deployed pods relative to the number of pods that can be createdCluster Pod CapacityNumber of pods that can be createdRequested CPU usage relative to total CPU coresrequested/total Cluster CPU Capacity Total number of CPU cores in the cluster Memory Requested/Total Memory Capacity Total memory capacity of the cluster GPU Requested/Total GPU Capacity Total number of GPUs in the cluster

Node Summary Table

Column NameDescriptionipNode IP addresshostnameNode host nameuptimeTime elapsed since last bootmemoryTotal node memoryCPU CoresCPU usage over the last 5 minutes5m loadTotal CPU usage over the last 5 minutesMemory usedMemory usagePartition usedDisk usageDisk readAverage data read from disk per second Disk write Average data written to disk per second CurrEstab Number of active TCP connections TCP-tw Number of terminated TCP connections Download* Average download traffic per second Upload* Average upload traffic per second

Detailed Information by Node

Panel NameDescriptionUptimeTime elapsed since last bootCPU CoresTotal number of CPU coresTotal RAMTotal memory capacityCPU BusyCPU usageUsage RAMMemory usageUsed Max MountDisk usage based on the largest partitionUsed SWAPSwap memory usage

Disk Space Used Basic

Column NameDescriptionDeviceDevice NameFilesystemFile System TypeMounted onMount Point on the File SystemSizeDisk SizeAvailAvailable SpaceUsedUsage

Panel Name Description Internet traffic per hour Network traffic per hour by Allnic CPU% CPU usage by mode (user, system, io, total) in Basic mode Memory Basic Memory usage (free, used, total)Network Bandwidth Usage per Second: Network bandwidth by AllnicSystem Load: System load in 1m, 5m, and 15m intervals(CPU usage) Disk R/W Data: Data read from/written to disk (in bytes) Disk Space Used%: Disk usage Disk IOps Completed: Number of disk I/O operations per second Time Spent Doing I/Os: Time spent on disk I/O Disk R/W Time: Time spent per I/O operation

Network Sockstat

Item NameDescriptionTCP-tw Number of terminated TCP connections CurrEstab Number of established TCP connections TCP_inuse Number of TCP sockets in use UDP_inuse Number of UDP sockets in use TCP_alloc Number of TCP sockets in the allocated state TCP_passive_opens TCP sockets opened by connecting to ports on which this node is listeningTCP_active_opens: TCP sockets opened by this node TCP_inSegs: Number of received TCP segments TCP_retransSegs: Number of retransmitted TCP segments TCP_outSegs: Number of transmitted TCP segments

Open File Descriptors/Context Switches

Item NameDescriptionfile descriptorNumber of allocated file descriptorscontext switchesNumber of context switches performed

Application Monitoring Using Custom Metrics (Example)

Server.go

package main

import (
	"fmt"
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var pingCounter = prometheus.NewCounter(
	prometheus.CounterOpts{
		Name: "ping_request_count",
		Help: "No of request handled by Ping handler",
	},
)

func ping(w http.ResponseWriter, req *http.Request) {
	pingCounter.Inc()
	fmt.Fprintf(w, "pong")
}

func main() {
	prometheus.MustRegister(pingCounter)

	http.HandleFunc("/ping", ping)
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":8090", nil)
}

Dockerfile

FROM golang:1.22.1-alpine AS build
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN GOARCH=amd64 go build -o server .

FROM alpine:latest
WORKDIR /app
COPY --from=build /app/server .
EXPOSE 8090
CMD ["./server"]

Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prom-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prom-example
      release: qks-monitoring
  template:
    metadata:
      labels:
        app: prom-example
        release: qks-monitoring
    spec:
      containers:
      - name: mymetric
        image: registry.smg.quantumcns.io/qms/prom_example:v0.2
        ports:
        - name: metrics
          containerPort: 8090

Service.yaml

apiVersion: v1
kind: Service
metadata:
  name: prom-example
  labels:
    app: prom-example
    release: qks-monitoring
spec:
  selector:
    app: prom-example  
    release: qks-monitoring
  ports:
    - name: metrics
      protocol: TCP
      port: 8090   
      targetPort: metrics
  type: ClusterIP

ServiceMonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
  labels:
    app: prom-example
    release: qks-monitoring
  name: prom-example
spec:
  endpoints:
  - path: /metrics
    port: metrics
    interval: 15s
  namespaceSelector:
    matchNames:
    - qks-monitoring
  selector:
    matchLabels:
      app: prom-example
      release: qks-monitoring

alertmanagerconfig.yaml

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: config-example
  labels:
    alertmanagerConfig: example
spec:
  route:
    groupBy: ['job']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: 'webhook'
  receivers:
  - name: 'webhook'
    webhookConfigs:
    - url: https://webhook.site/6016d666-7cc1-4f81-bb8f-442ef2a9a400
      send_resolved: false

prometheusrule.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    release: qks-monitoring
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
spec:
  groups:
  - name: Count greater than 5
    rules:
    - alert: CountGreaterThan5
      expr: ping_request_count > 5
      for: 10s

Ref.
https://prometheus-operator.dev
https://prometheus.io
Back to list