Prometheus & Grafana: What, Why & How to Install on GKE Cluster

While Prometheus is the name of a Greek titan who brought mankind fire, its new association should be as your go-to tool for monitoring key data points and metrics.

December 16, 2022 | Technical | Google Cloud | Feature Highlight

Prometheus & Grafana: What, Why & How to Install on GKE Cluster

Join us as we briefly discuss Prometheus and Grafana and how these fantastic open-source software can help elevate your DevOps processes. Additionally, we’ll cover a section exclusively for the techies - installing Prometheus & Grafana on a GKE Cluster.

Prometheus: Brief History & Overview

In its inception, the idea behind Prometheus was to monitor dynamic environments like Kubernetes or docker swarm. Prometheus aims to solve the ever-increasing data complexity of DevOps.

Typically, when you’re dealing with multiple servers and 100s of disparate processes, manual monitoring is cumbersome. Prometheus offers a viable solution: Automation.

With Prometheus, SREs and DevOps engineers can configure automatic monitoring of various critical alerts like backend status, auth-service status, crash errors, etc. Moreover, Prometheus also helps with issue prevention. For instance, Prometheus alerts maintainers when a particular memory usage spikes - and consequently, SREs can take proactive measures to prevent downtime.

Unveiling Grafana

Grafana is a scintillating data visualization tool that queries Prometheus and presents valuable information. Moreover, Grafana serves as a single pane of glass (SPOG) for displaying different kinds of data. Grafana’s highlights include the following:

  • Interactive graphs, charts, and maps
  • Custom dashboards
  • Alerting rules to query multiple data sources
  • User access management

Dashboard Templating

Dashboard templating makes Grafana dashboards a lot more dynamic and interactive. For instance, you can create custom Dashboard template variables and redefine legend names or row titles.

Broadly, there are three types of Template variables:

Query:

The Query template generates a dynamic list of variables from your data source. You can use wildcard entries like prod.servers.* to fetch all possible values in your wildcard entry.

wildcard-entry

Interval:

The interval type helps present data around time ranges (e.g., 1 minute, 1 hour, 1 day).

Custom:

The custom option is about creating user-defined template variables on the Grafana dashboard.

kube-equivalents

Image Source

Prometheus and Grafana are Kube’s equivalents of Romeo and Juliet. They are a match made in cloud computing heaven!

prometheus-vs-grafana

Kubestate Metrics: In a Nutshell

Kube-state-metrics (KSM) comprise a simple service that fetches metrics from k8s deployments. These metrics range from health & resource usage to internal object states.

KSM - at its core, is a service that calls the Kubernetes API to generate metrics regarding Kubernetes object states. The API exposes critical parameters like:

  • Pod lifecycle status
  • Node status
  • Available deployment replicas

kubestate-metrics

Image Source

The critical difference between Kubestate metrics and traditional metrics servers (pre-installed on cloud clusters) is that KSM provides data in a Prometheus format. As a result, SREs and DevOps functions can leverage customizable metric visualization and reporting tools.

Prometheus on Google Cloud

Grafana is an open-source tool that essentially adds a reporting layer to the Prometheus DB.

However, since Grafana is OSS, certain cloud providers might impose restrictions via a vendor lock-in.

The Managed Prometheus service on GCP allows you to leverage open-source tools and bypass such vendor lock-in situations.

As a result, you continue to scale your K8 deployments on Google Cloud - while harnessing the full potential of OSS like Grafana for out-of-the-box reporting and dashboard capabilities.

Installation of OSS version of Prometheus and Grafana

This one’s for all the SREs and Kube platform engineers! You guys rock! Here’s how you can deploy Prometheus and Grafana on your GKE cluster:

Installing Prometheus

We will walk you through the installation of Prometheus Operator on a K8s cluster. Prometheus Operator essentially simplifies the Prometheus deployment and configuration.

Without any further ado, here are the installation steps:

  1. Install Prometheus Operator

  2. Configure RBAC Privileges

    • Create a directory to store K8s manifests.
    • Use these commands
    • mkdir operator_k8s
    • cd operator_k8s
    • Create a YAML file named prom_rbac.yaml. Paste in the Kubernetes manifest
  3. Deploy Prometheus

    • Create a YAML file named Prometheus.YAML
    • Paste in the manifest from the official documentation
    • Define certain key critical parameters like name, label, and memory
    • Save and close the file.
    • Deploy it to your GKE cluster by using
    • kubectl apply -f
      prometheus.monitoring.coreos.com/prometheus created
  4. Creating a Prometheus Service

    • Create another manifest file called prom_svc.yaml and paste in the definitions
    • Save and close the file.
    • Now, create the service by:
    • kubectl apply -f prom_svc.yaml
      service/Prometheus created
  5. Creating a Prometheus Service Monitor

    • Create a new manifest file called prometheus_servicemonitor.yaml and paste in the ServiceMonitor resource definition
    • Define key parameters like Service Monitor, label, scrape interval.
    • Save and close the file.
    • Deploy it to your GKE cluster by using
    • kubectl apply -f prometheus_servicemonitor.yaml
      servicemonitor.monitoring.coreos.com/prometheus-self created
  6. Installing Grafana on Kubernetes

    Here’s how you can deploy Grafana on your GKE cluster:

    1. Create a Grafana Kubernetes manifest

      • Create a manifest file named grafana.yaml and paste in the definitions.
    2. Send the manifest to your Kubernetes API server

      • Key in the following command:
      • kubectl apply -f grafana.YAML
      • Verify by executing this command:
      • kubectl port-forward service/grafana 3000:3000
      • Access localhost:3000 from your browser. You will land on a Grafana login page.
      • Use admin as the username & password for login.

    Check out the official Grafana documentation for a more comprehensive tutorial.

    Advantages of Prometheus & Grafana on GKE Cluster

    Prometheus and Grafana are popular open-source software with a reasonably straightforward deployment and configuration setup.

    Prometheus offers out-of-the-box data monitoring from various sources like Datadog, influxDB, MySQL, etc. Grafana is the ideal tool for visualizing all the data scraped by Prometheus. When these two tools work together, you can collect valuable insights.

    advantages-of-prometheus

    Image Source

    Prometheus integrates seamlessly with Kubernetes by automatically fetching metrics from replicas as services scale up.

    On a side note, here are four benefits of monitoring your Kubernetes using Prometheus + Grafana:

    1. Real-time alerts

      You can detect errors early and reduce your restoration time with proactive monitoring.

    2. Efficient workload management

      Monitoring your k8s cluster can unveil pressing concerns like resource contention or unstable pod distribution. Straightforward tweaks like anti-affinities and setting affinities can improve performance.

    3. Better cost visibility

      Kubernetes can drive up your bills in no time. Real-time monitoring of your clusters will help you keep an eye on rising costs.

    4. Seamless Troubleshooting

      Constant monitoring makes it easy to quickly access logs and conduct an RCA for any issues on your Kubernetes cluster. As a result, you encounter lesser downtimes.

    Wrapping Up

    The bottom line is that both Prometheus and Grafana are game-changing additions to your k8s cluster. As your cloud services scale, it becomes critical to monitor key parameters about the health of your cloud containers. Prometheus does an excellent job of scraping valuable data metrics - while Grafana is a fantastic presentation tool that displays your metrics in various formats.

    Our developers behind this guide hope can be easily followed - though if you or your team need any assistance in this implementation or anything else GCP related, get in touch for a free technical consultation today.

Author

Dheeraj Panyam

Dheeraj Panyam

Principal Cloud Architect

Dheeraj has been working as a Google Cloud Professional Architect for the past 5 years helping design solutions and architecture set up on public cloud platforms. Dheeraj also has 20+ years total experience across App Development, Production Support, QA Automation & CloudOps.

Related Posts

What Our
Clients Are
Saying
Constrained with time and budget, we were in search of an experienced technology partner who could navigate through the migration work quickly and effectively. With D3V, we found the right experts who exceeded our expectations and got the job done in no time.
We used D3V to help us launch our app. They built the front end using React and then pushed to native versions of iOS and Android. Our backend was using AWS and Google Firebase for messaging. They were knowledgeable, experienced, and efficient. We will continue to use them in the future and have recommended their services to others looking for outside guidance.
We had an idea and D3V nailed it. Other vendors that we had worked with did not understand what we were trying to do - which was not the case with D3V. They worked with us through weekly meetings to create what is now the fastest and most accurate steel estimating software in the world. Could not have asked for anything better - what a Team!
Working with D3V was hands down one of the best experiences we’ve had with a vendor. After partnering, we realized right away how they differ from other development teams. They are genuinely interested in our business to understand what unique tech needs we have and how they can help us improve.
Our experience with D3V was fantastic. Their team was a pleasure to work with, very knowledgeable, and explained everything to us very clearly and concisely. We are very happy with the outcome of this project!
Protecting our customers data & providing seamless service to our customers was our top priority, which came at a cost. We are very satisfied with the cost savings & operational efficiency that D3V has achieved by optimizing our current setup. We’re excited about future opportunities for improvements through deriving insights from our 400 million biomechanics data points.
Bedrock Real Property Services Founder

Dr. A. Ason Okoruwa

President, Bedrock Real Property Services

Squirrelit

David Brotton

CEO & Founder, Squirrelit

FabSystems Inc

Terry Thornberg

CEO, Fabsystems Inc.

BLI Rentals

Lee Zimbelman

IT Director, BLI Rentals

Osmix Music

Jared Forman

CEO & Co-Founder, OSMix Music

Dari Motion

Ryan Moodie

Founder, DARI Motion

Schedule a call

Book a free technical consultation
with a certified expert.

Schedule Call

Get an estimate

Fill out our form to hear back with a project's cost estimate. No meeting required.

Get Estimate

Get in touch

Send a message to D3V team.
add some text more here

Let's Talk