Eduardo Medeiros

Eduardo Medeiros

System Engineer.

© 2022

About me

Browse by categories

How to create a monitoring playground by using gitOps principles

Introduction

This is a playground that creates a monitoring environment by using gitOps principles.

Features

  • k8s cluster running on docker via kind
  • deployment managed by fluxcd
  • multi-environment support
  • configmaps uses merge behavior
  • ingress supported out-of-box
  • grafana dashboard as a code (fluxcd and ingress controller)

Installation

For this playground, we gonna create two repositories:

Name Description
fluxcd-monitoring-lab contains fluxcd system manifests files
monitoring-fleet contains all infrastructure & monitoring (nginx ingress, prometheus, alert-manager, grafana) manifests files

1. Install requirements

2. Install k8s playground

curl -o kind.yaml https://raw.githubusercontent.com/eduardolmedeiros/monitoring-fleet/main/kind.yaml
kind create cluster --name monitoring --config kind.yaml
kubectl config set-context kind-monitoring

3. Create a github token

export GITHUB_USER=<username>
export GITHUB_TOKEN=<token>

4. bootstrap fluxcd

flux bootstrap github --owner=$GITHUB_USER \
  --repository=fluxcd-monitoring-lab \
  --path=clusters/dev \
  --personal

5. Deploy monitoring stack

Deployment

Fork the repository monitoring-fleet

gh repo fork --remote https://github.com/eduardolmedeiros/monitoring-fleet

Clone the fluxcd-monitoring-lab repository (previously created by the bootstrap command)

git clone https://github.com/$GITHUB_USER/fluxcd-monitoring-lab

Create all infrastructure & monitoring resources

cd fluxcd-monitoring-lab

flux create source git monitoring-fleet \
  --url=https://github.com/$GITHUB_USER/monitoring-fleet \
  --branch=main \
  --interval=1m \
  --export > ./clusters/dev/monitoring-fleet-repo.yaml

flux create kustomization sources \
  --source=monitoring-fleet \
  --path="./sources" \
  --prune=true \
  --interval=1m \
  --export > ./clusters/dev/sources.yaml

flux create kustomization infrastructure \
  --source=monitoring-fleet \
  --path="./infrastructure/dev" \
  --prune=true \
  --interval=1m \
  --export > ./clusters/dev/infrastructure.yaml

flux create kustomization monitoring \
  --source=monitoring-fleet \
  --path="./monitoring/dev" \
  --depends-on=infrastructure \
  --prune=true \
  --interval=1m \
  --export > ./clusters/dev/monitoring.yaml

5 Add & commit & push all files

cd fluxcd-monitoring-lab
git add .
git commit -m "setup monitoring environment" -a
git pull
git push

6. Update /etc/hosts file (required to access all ingress hosts)

echo "# monitoring lab
127.0.0.1 grafana.k8s.local alert-manager.k8s.local prometheus.k8s.local" >> /etc/hosts

7. Access monitoring resources

http://grafana.k8s.local
http://prometheus.k8s.local
http://alert-manager.k8s.local

fluxcd useful commands

# get all resources and statuses
flux get all 

# force sync for infrastructure resources
flux reconcile kustomization infrastructure

# force sync for monitoring resources
flux reconcile kustomization monitoring

# get all helm releases status in all namespaces
flux get helmreleases --all-namespaces

# get helm release in monitoring namespace
flux get helmreleases -n monitoring 

How to update/manage the monitoring resources?

Just update the yaml files from the monitoring-fleet (fork repository)

Example:

Update the kube-prometheus chart version to the latest version.

cd monitoring-fleet
vim monitoring/dev/prometheus/release-patch.yaml
git commit -m "update kube-prometheus chart version" -a
git pull ; git push

Push the changes and check the fluxcd.

flux logs
flux get helmreleases -n monitoring 

How to uninstall?

kind delete cluster --name monitoring