From local development to Kubernetes — Cluster, Helm, HTTPS, CI/CD, GitOps, Kustomize, ArgoCD — Part[1]

Nowadays you may hear a lot about Kubernetes… You open the browser and search for it and all you get is some resources that are really advanced and you can’t even understand what is happening or basically, you don’t see where is your problem fitting into the frame. At least, this was my experience when I started digging around this topic.

Now I wrote this article to share my knowledge and to make your life easier when it comes to using solving problems with Kubernetes using the best practices.

I will go in an ordered way from explaining what Kubernetes is, how to prepare local development, deployment, etc. each step-by-step and explaining in detail what each component is and why you need it in the bigger picture. Are you ready? — Let's go!

Since the article is going to cover a lot of concepts, it’s separated into two parts to make it easier for you to read and follow along. Please find below the link for the second part of this article. This article assumes that the reader has basic knowledge of programming and Kubernetes.

Tutorial steps

The tutorial series progresses with increased complexity, where:

Part 1(This one):

• What is Kubernetes and its most important components
• Build a basic Python application using FastAPI
• Dockerize and run it locally with docker-compose
• Setting up a Kubernetes Cluster in the cloud
• Deploy our app to Kubernetes in the traditional way
• Deploying a PostgreSQL database in our cluster using Helm
• Configure domain, Ingress Controller, HTTPS

Part 2 includes (click to go to Part 2):

• Introduction to GitOps — ArgoCD installation
• Using Kustomize to write Kubernetes manifest
• Secret management in Kubernetes with SealedSecrets
• Add a basic Continuous integration pipeline with GitHub actions
• Creating and running our services in ArgoCD

If you know Kubernetes architecture, you can skip the following part (What is Kubernetes), even though a refresh is highly recommended

What is Kubernetes

Kubernetes is a system for managing and orchestrating containerized applications. In simple terms, it allows you to define how your applications should run, and then it manages the running of those applications for you. It can run on a variety of different platforms, including on your own infrastructure or in the cloud. Kubernetes is especially useful when you have multiple applications or microservices that need to run together, as it can help you to automate the deployment and scaling of those applications, and it makes it easier to manage the entire system.

Kubernetes Architecture

API Server — The Kubernetes API server is the central way to interact with a Kubernetes cluster. It exposes the Kubernetes API, which is a set of RESTful APIs that you can use to create, update, delete, and list the objects in a cluster

Scheduler — The Kubernetes scheduler is a component of the Kubernetes system that is responsible for assigning pods (the smallest deployable units in Kubernetes) to nodes in the cluster. When a pod is created, the scheduler selects the node where the pod should run based on the resource requirements of the pod, the resource availability on each node, and other constraints that may be specified by the user.

Controller Manager — The Kubernetes controller manager is a daemon that runs on the Kubernetes master node and is responsible for the management and execution of various controllers

etcd — is a distributed key-value store that is used by Kubernetes to store data that needs to be shared across all of the nodes in the cluster. This data includes information about the state of the cluster, such as the configuration of the various applications and services that are running on the cluster.

Pod — a group of one or more containers that are deployed together on a host. Pods are the smallest deployable units in Kubernetes and provide a higher level of abstraction than individual containers.

Node — may be a virtual machine or a physical machine, depending on the cluster. Each node has a Kubernetes agent (also called a “kubelet”) that communicates with the control plane components and runs containers on the node. Docker is also installed on each node as it’s used to manage the containers that run inside a pod.

Now that we know a bit about the architecture of a Kubernetes cluster, let’s dive into code and build a simple Python web application that defines some API endpoints, connects to a Postgres database, has some environment variables and also has Dockerfile and docker-compose to simulate the application locally.

Building a basic Python application locally — Run with Docker & docker-compose

To save time here, I have prepared a basic Python application developed on FastAPI and if you want to follow along you can get the code from GitHub Repository here.

├── app                   # Base directory
│   ├── crud.py           # Crud Logic 
│   ├── database.py       # Database engine
│   ├── main.py           # API endpoints and FastAPI initialization
│   ├── models.py         # Database models
│   └── schemas.py        # Schemas used to exchange data in API
├── docker-compose.yaml   # docker-compose with web and postgres
├── Dockerfile            # Dockerfile for FastAPI application
└── requirements.txt      # Requirements for project

I will focus on the points that are important and that you should keep in mind while developing an application for Kubernetes or other platforms that are related to containerized applications like:

• Developing a service in containerized methodology
• Giving environment variables in a dynamic way
• Running a service locally with docker-compose

I am stopping at this point as this as its the beginning of the whole process as Kubernetes itself is based on containers (running containerized applications).

The sample app in the repo is a basic FastAPI application that I extracted from official docs, but I added a PostgreSQL database to make the whole tutorial a bit more challenging. The application is very simple, it exposes some endpoints to create Users and Items.

To start the application locally, simply copy .env_example to .env and run the application with Docker

$ cp .env_example .env
$ docker-compose up -d --build

The application should be up and running, and you should be able to see the API definitions at http://localhost:8000/api/docs

Let’s make sure the service is working as excepted by making some API requests:

# Create a user

curl -X 'POST' \
  'http://localhost:8000/users/' \
  -H 'Content-Type: application/json' \
  -d '{
  "email": "test@demo.com",
  "password": "somepassword"
}'
------------------------------------------------------------
{"email":"test@demo.com","id":1,"is_active":true,"items":[]}                                        ➜  ~

# Create a item for the user with id 1

curl -X 'POST' \
  'http://0.0.0.0:8000/users/1/items/' \
  -H 'Content-Type: application/json' \
  -d '{
  "title": "Item1",
  "description": "Some Description"
}'

Excepted response:
{"title":"Item1","description":"Some Description","id":1,"owner_id":1}                                  ➜  ~

So as excepted, the users and the respective items are being created and stored in PostgreSQL which is running in Docker. I added a PostgreSQL on purpose to also explain how you connect a service to a database running on Kubernetes.

Setting up Kubernetes Cluster

To setup a Kubernetes cluster you can decide if you want to create locally with minikube or kind, or create it in a cloud provider like AWS, GCP, DigitalOcean, etc. in their respective platforms for Kubernetes like EKS, GKE, DOKS.

For this demo, I want to be as realistic as possible, as I want to also set up:
Domain, HTTPS, Reverse Proxy, etc. so I am going to use a cloud-based cluster in DigitalOcean (similar steps are applicable in other cloud providers) or if you follow along using Minikube or Kind.

Steps to set up a Kubernetes cluster may be different in other cloud providers, but the concept are almost the same, so follow along:

Here are the general steps for setting up a Kubernetes cluster on DigitalOcean:

1. Sign up for a DigitalOcean account here
2. Navigate to Kubernetes
3. Click “Create a Kubernetes Cluster”
4. Fill up the necessary information, and click “Create Cluster”

Setting up a Kubernetes cluster in Digital Ocean

We set up a Kubernetes cluster, which is located in Germany, with 2 nodes, which we named my-cluster while the node pool name is my-node-pool . It’s always recommended to have at least two nodes, as Kubernetes concepts are regarding scaling the services between the nodes, so if one goes down, the pods can be created in the other nodes, so in order to keep your services up and running is highly recommended to have at least two nodes.

After creating the Kubernetes Cluster, you need to configure tools to connect to the cluster. The CLI tool to interact with the Kubernetes cluster is kubectl which you can easily download from here.

kubectl requires an active context in order to know in which cluster to make the request. After the cluster has been provisioned, you can download the kubeconfig, and then you can make a request against the cluster.

For now, navigate to the directory in which you have downloaded the config and export a variable named KUBECONFIG . Remember, if you configure this way, you always need to export this variable when you change the terminal.

$ export KUBECONFIG=name_of_downloaded_file.yaml 

# on windows you can use setx

To verify that we are successfully authenticated to cluster, run a simple command:

$ kubectl get nodes

------------------------------------------------------
NAME                 STATUS   ROLES    AGE     VERSION
my-node-pool-mhag8   Ready    <none>   7m4s    v1.25.4
my-node-pool-mhagu   Ready    <none>   6m54s   v1.25.4

Deploying the application to Kubernetes — Traditional Way

To deploy our services to Kubernetes, we need to write the manifests in a declarative way, speaking of which we need to create:

• Deployment
• Service
• Ingress
• Config, Secrets

Please read each one from the official docs on kubernetes.io

Before we do so, let’s push our docker image to a registry (You can skip this step and use my image instead as I am leaving it as public, but feel free to create your own registry if you want to customize for your needs)

I am using the docker hub registry on which I have created a public repository. (How to create a Docker Hub Repository). After creating the repository, I need to rename and tag the image that I built with docker-compose to the name of the repo that I created on Dockerhub

$ docker login # login to your dockerhub account
$ docker tag fastapi-app_app valonjanuzaj/kubernetes-demo:latest
$ docker push valonjanuzaj/kubernetes-demo:latest

Now that we have the application image on the docker hub, we can start creating the manifest for our application. Let’s start by creating a deployment and service. To simplify, I am using the same file and I am separating the manifest with --- which is valid YAML syntax.

To deploy the application in Kubernetes you need to execute:

$ kubectl apply -f deployment-service.yaml

The application will get deployed, but it won’t work as it tries to access the database which we still don’t have. Remember, in the local application, we defined some environment variables regarding the Postgres database and here we still didn’t add any environment variable for the pod.

Deploying a PostgreSQL Database

There are several ways how you can deploy a database in the cluster (you can also use a remote database hosted in the cloud, but we want to make this practice more challenging so we can touch more concepts regarding Kubernetes). You can manually write all the manifests (deployment, service, secrets, volumes, etc) or you can use Helm to deploy the database and many other services. I highly recommend using helm charts (verified ones) to deploy this kind of service instead of writing these by yourself.

Helm is a package manager for Kubernetes. It is used to automate the installation, upgrade, and management of Kubernetes applications.A Helm package is called a “chart”. Charts are made up of a collection of files that describe a related set of Kubernetes resources. For example, a chart might contain a deployment, a service, and a cluster role.

After installing Helm locally, you need to find the chart that you need to install. I usually go with Bitnami charts. To install any chart with helm, we first need to add the repo from which we want to pull the charts, and then we can install after finding the chart that we want to.

$ helm repo add my-repo https://charts.bitnami.com/bitnami
$ helm install postgres my-repo/postgresql --namespace postgres --create-namespace

After you install the chart, all the instructions to connect to the database will be shown. Basically, you would need to port-forward the service of PostgreSQL and connect to it locally to create a new database or just manipulate it.

The command above will install PostgreSQL in the cluster, it will attach the necessary resources and will make it ready to be used by applications. What is even more interesting is that when the chart gets installed it will assign some persistent volumes for data based on the cloud provider that you’re using, in this case, do-block-storage, so even if we delete this chart, we can still re-wire a database to the volume without losing the data.

Let's connect to the database and create a database named kubernetes-demo for our FastAPI service.

Connect the application to the database

Now that the database is created and running in our cluster, we can set up the environment variables to connect our FastAPI service to this database. To do that, we need to provide some environment variables, just like we did locally on the application with docker-compose, but now to the container that is running on the pod that we deployed, and those are:

• POSTGRES_USER
• POSTGRES_PASSWORD
• POSTGRES_DB
• POSTGRES_SERVER
• POSTGRES_PORT

We’ll these are considered secrets and you need to be careful about the way you store them. Currently, we are going to create a secret and then we’re going to make some changes in deployment-service.yaml to reference the secrets that we have created because currently, we haven’t set any environment variable for the container.

Creating the secret: To create the secret, you need to encode the values with base64 which can be easily done on the terminal:

$ echo -n "value_to_encrypt" | base64
# Example
$ echo -n "SomeSecret" | base64
Output: U29tZVNlY3JldA==

Similarly, you encode all the values and then add them to the secret.yaml:

I’ve encoded real values of the Postgres database that we have created. Now create the secret with kubectl:

$ kubectl apply -f secret.yaml

Be very careful when you add the POSTGRES_SERVER; Since the postgres is deployed on a separate namespace named “postgres” and the app is running on “kubernetes-demo” namespace, you need to add the server like: <name_of_the_service>.<namespace>, which in our case would be something like: postgres-postgresql.postgres

These kinds of manifests are not safe to be pushed on the repository, since base64 can be easily decoded to plain text. We are leaving it like this for now as this is the purpose of the article, to show you the traditional way first then we’ll move to advanced ways of managing these kinds of resources.

Now let’s update deployment-service.yaml to reference the created secrets:

    spec:
      containers:
        - name: kubernetes-demo
         ...
         #######################################
         #Add the following part to the yaml file
          envFrom:
            - secretRef:
                name: demo-secrets

Apply the manifest again with:

kubectl apply -f deployment-service.yaml

And the application should be up and running. To access the app now we need to expose something, but for now, let’s just port forward and access it locally to see if everything is working fine:

$ kubectl port-forward --namespace kubernetes-demo svc/kubernetes-demo 8000:80

# The application should be up and running at http://localhost:8000/api/docs

We port-forwarded the service of the application named kubernetes-demo which has port 80 to local port 8000. Now we can access the application at http://localhost:8000/api/docs

Setting up the domain—Installing Ingress Controller — Securing the traffic with HTTPS

Now let’s expose our application for the whole world to see it. To do that we need a domain (you can use the load-balancer IP if you don’t have a domain and don’t want to set one, so bear with me). If you want to get one domain, go check out Namecheap as you can find cheap domains to play around with or for your personal purposes.

I have already one domain purchased get.tech, so I am going to use that one. To manage the domain from DigitalOcean (and any cloud provider), you need to add the domain to it and then point the nameservers to that provider from your domain registrar. To add the domain in DigitalOcean, navigate to Networking > Domains and add your custom domain.

After you add the domain you need to point the nameservers to DigitalOcean (see here how), which are:

ns1.digitalocean.com.
ns2.digitalocean.com.
ns3.digitalocean.com.

After a bit, your domain should be pointing to the DO nameservers and you can manage your records from the DigitalOcean dashboard.

Setting up Kubernetes ingress controller

Note that the same steps are applicable to other cloud providers

An ingress controller acts as a reverse proxy and load balancer. It implements a Kubernetes Ingress. The ingress controller adds a layer of abstraction to traffic routing, accepting traffic from outside the Kubernetes platform and load balancing it to Pods running inside the platform through internal services. The whole flow is as follows:

A user makes a request to the URL that we have exposed. The load balancer accepts the request and based on the mappings that are defined on the ingress forwards the request to a specific service. The service then forwards the request to one of the pods that are available to handle that request.

Kubernetes cluster with ingress controller and load balancer, source Kubernetes Advocate

Now that we understand a bit about what is ingress controller, we need to install one in our cluster.

You may do these steps manually but it would take a lot of work to implement one of the ingresses that are already out there, so I highly recommend installing one from the versions that are out there using Helm. To do that we can install it again from the Bitnami repo that we added before when installing the PostgreSQL, so let’s go ahead and install the nginx-controller implementation:

➜  ~ helm search repo nginx           
NAME                             CHART VERSION APP VERSION DESCRIPTION                                       
my-repo/nginx                    13.2.21       1.23.3      NGINX Open Source is a web server that can be a...
my-repo/nginx-ingress-controller 9.3.24        1.6.0       NGINX Ingress Controller is an Ingress controll...
my-repo/nginx-intel              2.1.13        0.4.9       NGINX Open Source for Intel is a lightweight se...


$ helm install nginx-controller my-repo/nginx-ingress-controller --namespace nginx-controller --create-namespace

The controller will be installed at nginx-controller namespace in a bit. What happens when we install the controller in a managed Kubernetes service through the helm chart is that a LoadBalancer will automatically get created for you, and you can use that created LoadBalancer to send the traffic to your cluster (see picture above, we are implementing the same idea). If we navigate to Networking > Load Balancers we see that the LoadBalancer is created and is ready to be used. Similarly, a load balancer would be created in other cloud providers such as AWS, GCP, etc.

Now that the domain is created, and the LoadBalancer is ready, we can go ahead and create a new subdomain to use for our application. To do so, we navigate again to:

Networking > Domain > Choose our domain and we create an “A domain”

An A record, also known as an “address record,” is a type of DNS record that maps a hostname to a specific IPv4 address.

Just be careful to redirect to the LoadBalancer IP that we have just created so then that Ingress can take control of the request and redirect as it should. Now we are ready, so let’s create an Ingress and map the subdomain to our FastAPI service.

Creating a Kubernetes Ingress

Create a new file named ingress.yaml and paste the following content.

It’s pretty clear right? We are mapping the host demo.*.tech to the service named kubernetes-demo (which exists in the same namespace), in the port 80, when path / is triggered. The ingressClassName should be the name of the ingress controller that we created.

Create the ingress with kubectl:

kubectl apply -f ingress.yaml

To check if the ingress is created and assigned to the right service, we execute the command belowto get more data about the Kubernetes resource:

kubectl describe ingress -n kubernetes-demo kubernetes-demo-ingress
-------------------------------------------------------------------------

Name:             kubernetes-demo-ingress
Labels:           <none>
Namespace:        kubernetes-demo
Address:          164.92.182.82
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  demo.vjanztutorials.tech  
                            /   kubernetes-demo:80 (10.244.0.123:8000,10.244.0.251:8000)

We can see that ingress is created successfully and it’s redirecting the traffic to the kubernetes-demo service, which is redirecting to the two pods that we have created for that Deployment. Perfect, now let’s see if we can access the application from the exposed host:

curl -X 'GET' \
  'http://demo.vjanztutorials.tech/health' \
  -H 'accept: application/json'
--------------------------------------------
{
  "status": "Running!"
}

Or just visit the service URL at: yourdomain.com/health in my case demo.vjanztutorials.tech/health

Everything working as excepted, but there is only one thing we can do better and that is securing our traffic with HTTPS, so why not, let’s do that! If you have doubts about what HTTPS is and why we need it please take a refresher here, and then let's move on.

Securing the traffic with HTTPS

HTTPS is a way to make sure that when you visit a website, the information you share with that website is private and can’t be seen by anyone else. It’s like a secret code between your computer and the website that makes sure that only you and the website can read what’s being sent.

To do so, we need an SSL/TLS certificate for our application. To retrieve one we need to communicate with a certificate authority (CA) to verify the authenticity of the domain and then issue a certificate that can be used to secure the communication between the application and the user.

We will install Cert-Manager to manage our SSL/TLS certificates in an automated way. Cert-manager is a tool that helps manage and automate the process of obtaining and renewing SSL/TLS certificates for your websites and applications. It works by communicating with certificate authorities (CA) to request and renew certificates and then automatically updating your web server or application to use the new certificate. This eliminates the need for manual intervention and ensures that your certificates are always up-to-date and secure.

# search
➜  ~ helm search repo cert-manager
NAME                 CHART VERSION APP VERSION DESCRIPTION                                       
my-repo/cert-manager 0.8.10        1.10.1      cert-manager is a Kubernetes add-on to automate...

# install
helm install cert-manager my-repo/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true

Now that we have cert-manager installed we need to install an Issuer in our cluster, which can be ClusterIssuer or Issuer.
The differences between them are that the Issuer is a namespaced resource and it is not possible to issue certificates from an Issuer in a different namespace while ClusterIssuer is almost identical to the Issuer resource, however, is non-namespaced so it can be used to issue Certificates across all namespaces.

I am going to use ClusterIssuer and Let’s Encrypt as CA

Substitute the email with your email and that should be it. Now create it:

$ kubectl apply -f cluster-issuer.yaml

To issue a TLS certificate for our domain, we’ll annotate ingress.yaml with the ClusterIssuer that we created, so let’s modify it as:

Apply the changes with kubectl:

$ kubectl apply -f ingress.yaml

And check if the certificate is created:

$ kubectl get certificate -n kubernetes-demo
--------------------------------------------
NAME                  READY   SECRET                AGE
kubernetes-demo-tls   True    kubernetes-demo-tls   1m


$ kubectl describe certificate kubernetes-demo-tls -n kubernetes-demo
---------------------------------------------------------------------
Normal  Issuing    2m   cert-manager-certificates-issuing          The certificate has been successfully issued

We can also see that a secret has been created in the same namespace named kubernetes-demo-tls which contains the tls.crt and tls.key which is used to encrypt the traffic between pods, services, and other resources within the cluster.

Perfect, now let’s visit our application through the browser to see if the connection is being through HTTPS.

Perfect, our application is now running on HTTPS, which provides a secure and encrypted connection for users. This means that any information shared on our website, such as login credentials or personal information, is protected from potential hacking or data breaches. Additionally, HTTPS also improves website performance and is a ranking factor for search engines.

Everything is working smoothly, but let’s start to think on:

• How we are going to work in a team on this project
• How we are going to continuously integrate and deploy our app
• How we are going to build a deployment strategy that is secure and reliable, etc.

Summary

In this chapter, we learned what is Kubernetes and its components, creating and running a simple application locally, installing PostgreSQL in the cluster using Helm, and connecting our running application to it. After that, we exposed the application to the world with Ingress and secured the connections with HTTPS.

In the next part, I am going to discuss some of the downsides of the way that we organized the project and manifests and what we could do better using the most modern tools that are out there regarding Kubernetes and deployment strategies to make it developer friendly, reliable, and secure so stay tuned!

If you have any questions or any problem following along, feel free to reach out to me.

Connect with me on LinkedIn, GitHub

Links and resources

Part 1, Part 2

Github repo for this part:
https://github.com/vjanz/kubernetes-demo-app