Open in app

Sign in

Write

Sign in

Kubernetes tooling and shell setup

Prune
9 min readMar 29, 2022

How to be productive fast with Kubernetes

Updates:
- 20220410: added some Dasel examples
-20220410: I’ll be giving a talk about this post April the 12th at the Canadian CNCF and Kubernetes meetup: https://lnkd.in/ep9yaj6Z
- 20220412: add commands to build a patched kubecolor
- 20220413: recording of the K8s/CNCF CANADA Meetup talk

A ton of writing had been done about how to setup your shell and all the tooling that goes with it to use Kubernetes.
Well, I think it needed another one blog post (and talk) about it !

This one is focussing on Apple Mac setup, and specifically ZSH setup, but beside few small changes, it should work the same for Bash or even on Linux.

It’s also a primer, not an advanced setup. But it should contain all the basics so a new user, or a developper, can get going quickly.

Well, do whatever you want with this blog post. I’m writing it as a reference for myself 🙏

What is Kubectl ?

Come on, we all know that: kubectl is the CLI to interact with the K8s API !

kubectl is taking your human-readable requests and translate them to a REST call against the Kubernetes API server.

Then, the server answer with some information, mostly JSON. Usually, kubectl dump that in a human-readable way, or, quite often, in YAML, because we all love YAML !

Here, kubectl is usually doing the minimum, and things quickly gets messy when you have a large cluster with hundreds of resources.

Because of that, a bunch of talented people started building tooling around kubectl and the K8s API so we can spend less time reading kubectl outputs and spend more time doing the important stuff: snacking, watching Hell’s Kitchen show and plowing snow (last one is true when you live in Quebec City, QC, Canada)

OK, let’s go now.

Kubectl

Of course it all starts with kubectl… it’s up to you, there’s too many ways to intall it to give them all here ! gcloud can install one, brew, curl… find your own way !

Here’s one with curl:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod 755 kubectl

Or using Brew…

brew install kubectl

And here’s a Linux one (don’t expect more Linux example):

curl -LO "https://dl.k8s.io/release/$(curl -L -s
https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod 755 kubectl

Always try to use the same kubectl version as the server you are targeting, or +/- one version.
BTW, this kubectl version command is a great way to check for your server’s version:

kubectl versionClient Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2"}Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.9-gke.1001"}WARNING: version difference between client (1.22) and server (1.20) exceeds the supported minor version skew of +/-1

Kubecolor

KubeColor is used to Colourize your kubectl output. It makes reading all those lines of resources easier !

Install:

brew install dty1er/tap/kubecolor# add in your .zshrc
alias k=kubecolor

Result:

Until the PR 86 is merged, you may reach an issue when using the ctx or ns commands, where the whole output is displayed in yellow and hiding the selected default value. In the meantime, clone my own fork and build the binary yourself:

git clone git@github.com:prune998/kubecolor.gitgit checkout prune/ctx-no-colorcd cmd/kubecolor && go build && cp kubecolor /usr/local/bin

ZSH Setup

Alias, Completion, Tooling… in your .zshrc: use k instead of kubectl and reclaim 1s of your life at every command!

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"alias k=kubecolor
source <(kubectl completion zsh)complete -F __start_kubectl kcompdef kubecolor=kubectlsource <(stern --completion=zsh)ulimit -n 2048          # kubectl opens one cnx (file) per resource# gcloud
source "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.zsh.inc"source "/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.zsh.inc"# AWS
complete -C '/usr/local/bin/aws_completer' aws

ZSH customization:

  • Oh-My-ZSH : lots of features in your shell
    Use plugins !!
plugins=(brew kubectl git python tmux vault terraform)
  • Themes
    - Agnoster ZSH theme: better prompt using Powerline Fonts
    - PowerLevel10k: emphasizes speed, flexibility and out-of-the-box experience
  • Fonts
    - Powerline Font: recommend NerdFonts Inconsolata or Firacode

Here’s an example prompt when customizing all this. Of course, it’s possible to add more Kubernetes related stuff in the prompt, but it’s going to get messy quickly:

Krew

Krew is a plugin manager for kubectl.
Install: https://krew.sigs.k8s.io/docs/user-guide/setup/install/

kubectl krew listPLUGIN  VERSION
ctx     v0.9.4
krew    v0.4.1
ns      v0.9.4
whoami  v0.0.36
kubectl krew searchNAME                            DESCRIPTION                                         INSTALLED
access-matrix                   Show an RBAC access matrix for server resources     no
blame                           Show who edited resource fields.                    no
cert-manager                    Manage cert-manager resources inside your cluster   no
ctx                             Switch between contexts in your kubeconfig          yes
...

Kool Krew Plugins to have:

  • ctx: quick context ( current cluster ) changes
  • ns: quick current namespace changes
  • whoami: who the cluster thinks you are from your authentication
  • who-can: RBAC rules introspection

Example ctx usage to manage your contexts:

# list all the existing context, current one in yellow
k ctxarn:aws:eks:us-east-1:111111111111:cluster/eks-example
gke-dv-st-cluster-1
gke-dev_us-central1_test-gke-cluster# change context “manually”
kubectl config use-context gke-dev_us-central1_test-gke-cluster# change the context using ctx
k ctx gke-dev_us-central1_test-gke-cluster# delete context (why not ?)
k ctx -d gke-dv-st-cluster-1

Example using ns to change default namespace:

# List all namespaces, current NS in yellow (not in Medium blogs...)
k nsdatadog-agents
default         <----
kube-public
kube-system# Set default NS by hand
kubectl config set-context --current --namespace=kube-system# Set default Namespace
k ns kube-system

Cloud provider setup

AWS

First, install the awscli:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Then, setup SSO login:

export AWS_DEFAULT_REGION=us-east-1
export AWS_PAGER="" # prevent aws cli to auto-page = display inline
export BROWSER=echo # Do not open a browser, let you choose which browser to opencomplete -C '/usr/local/bin/aws_completer' aws # add that to .zshrc for completion# remove dandling env variables
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY# configure (may ask questions here)
aws configure sso
aws sso login --profile profile_xxxxxx
export AWS_PROFILE=profile_xxxxxx

Configure kubectl context to use your EKS cluster:

aws eks update-kubeconfig \ 
    --region us-east-1    \
    --name <cluster_name> \
    --alias <friendly_name>

Google

Install Google CLI (gcloud):

brew install --cask google-cloud-sdk
gcloud components install kubectl # Optionalgcloud init
gcloud auth logingcloud config set compute/region us-east1

Configure kubectl to use your GKE cluster:

gcloud container clusters get-credentials <cluster> --project <project>

Stern

Stern allows you to tail the logs of multiple pods on Kubernetes and multiple containers within the pod.
Each result is colour coded for quicker debugging.

Install:

brew install sternstern -n my-namespace dv-oma

Stern will tail the logs of all pods matching dv-oma as a pattern. There’s a ton of options to further filter what you want to display:

Kustomize

Kustomize is a Kubernetes native configuration management (templating)

  • Bundled with kubectl, but not all the features are available
  • Better install the full version for your CI/CD pipelines
  • Only output rendered YAML, you have to apply it later

Ex:

kubectl   kustomize --enable-alpha-plugins /path/to/kustomize/folderkustomize build     --enable-alpha-plugins /path/to/kustomize/folder

Render and apply the kustomization:

kustomize build --enable-alpha-plugins /path/to/kustomize/folder | kubectl apply -f -

Kustomize requires a kustomization.yaml file in the target folder, which link resources together, patch them, use generators to create new resources and plugins to patch/generate new yaml. Ex:

# cat /path/to/kustomize/folder/kustomization.yamlapiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomizationresources:
  - my_resources_file.yaml
  - ../basepatches:
  - ./manifests/my_patch.yamlgenerators:
  - my_generator.yaml

Kubernetes Dashboards

K9s

K9s — Kubernetes CLI To Manage Your Clusters In Style!

  • Open-Source
  • In your terminal, like top
  • CRUD operations on K8s resources
  • nothing to install server-side
  • lightweight
brew install k9sk9s -n <namespace>      # To run K9s in a given namespacek9s --context <context> # Start K9s in an existing KubeConfig contextk9s --readonly          # Start K9s in readonly mode - with all cluster modification
                          commands disabled
K9s

Lens

Lens is the only IDE you’ll ever need to take control of your Kubernetes clusters (well, that’s what they say)

  • Launch on your own desktop, no server-side install
  • Include advanced config to reach remote clusters
  • Manage CustomResourceDefinitions (CRD)
  • Nice UI
  • Multi-cluster

VsCode extensions for Kubernetes

I’m sure there are a lot more than these, but I would consider these as essentials:

  • Kubernetes: Develop, deploy and debug Kubernetes applications
  • YAML: Language Support, with built-in Kubernetes syntax support
  • Indent-Rainbow: helper to better see Yaml indentations
Indent-Rainbow

Local Kubernetes

Here’s a minimal list of solutions to deploy a local Kubernetes cluster on your laptop:

Kind

  • Kind is a local K8s cluster
  • Official Kubernetes tool to create lightweight K8s clusters
  • Support ingress / LB (with some tuning)
  • Work with Docker and Podman (and rootless with some more sweat)
brew install kindkind create cluster --help

K3s

K3s is the Rancher take on local clusters. Both ARM64 and ARMv7 are supported

sudo k3s server &# Kubeconfig is written to /etc/rancher/k3s/k3s.yaml
sudo k3s kubectl get node

Minikube

Minikube is certainly the oldest of all. Still supported, well documented and support different container runtimes, so you can get rid of Docker !

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64
sudo install minikube-darwin-amd64 /usr/local/bin/minikubeminikube start

Bonus tooling

Here’s few more tools to help working with Kubernetes resources (and YAML)

KubePug

Use KubePug to ensure your cluster is not using deprecated resources:

Verifies the current kubernetes cluster or input files, checking whether exists objects in this deprecated API Versions, allowing the user to check before migrating

kubectl krew install deprecationsk deprecations --k8s-version=v1.22.0

Dive

Use Dive to inspect the Docker Images. It’s a terminal app that enables you to deep-dive into your container’s layers and all:

Ensure your Docker (container) images are not too big and does not contain unnecessary data

brew install divedive cilium/cilium:v1.9.10

Dasel

Use Dasel to query and modify data structures using selector strings:

Comparable to jq / yq, but supports JSON, YAML, TOML, XML and CSV with zero runtime dependencies

brew install dasel

Here’s some examples of what you can do quickly:

# Select the image for a container named authdasel select -f deployment.yaml -s "spec.template.spec.containers.(name=auth).image" tomwright/x:v2.0.0# Change the image for a container named authdasel put string -f deployment.yaml -s "spec.template.spec.containers.(name=auth).image" "tomwright/x:v2.0.0"# Update replicas to 3dasel put int -f deployment.yaml -s "spec.replicas" 3
# Add a new env vardasel put object -f deployment.yaml -s "spec.template.spec.containers.(name=auth).env.[]" -t string -t string name=MY_NEW_ENV_VAR value=MY_NEW_VALUE# Update an existing env vardasel put string -f deployment.yaml -s "spec.template.spec.containers.(name=auth).env.(name=MY_NEW_ENV_VAR).value" NEW_VALUE

More Bonus: Containers without Docker

Colima

Colima Container runtimes on macOS (and Linux) with minimal setup

  • Intel and M1 Macs support
  • Simple CLI interface
  • Docker and Containerd support
  • Port Forwarding
  • Volume mounts
  • Kubernetes
  • Replace Docker-for-Desktop
brew install colima
brew install docker
colima start                       # default using Docker runtime
colima start --with-kubernetes     # start kubernetes local cluster
colima start --runtime containerd --with-kubernetes  # remove docker completely
colima status
INFO[0000] colima is running
INFO[0000] runtime: docker       # or containerd
INFO[0000] arch: x86_64
INFO[0000] kubernetes: enabled
cat ~/.colima/colima.yaml
vm:
    cpu: 2
    disk: 60
    memory: 2
    arch: x86_64
    forward_agent: false
    mounts: []
runtime: containerd
kubernetes:
    enabled: true
    version: v1.22.2
colima nerdctl run -- -ti --rm alpine:latest sh
/ # ...
colima nerdctl ps
CONTAINER ID    IMAGE                              COMMAND    CREATED           STATUS    PORTS    NAMES
47e87f00711d    docker.io/library/alpine:latest    "sh"       18 seconds ago    Up                 alpine-47e87
kubectl ctx
colima
kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   coredns-85cb69466-bz5mw                  1/1     Running   0          8m18s
kube-system   local-path-provisioner-64ffb68fd-2g9gz   1/1     Running   0          8m18s
kube-system   metrics-server-9cf544f65-t6tzs           1/1     Running   0          8m18s

Podman

Podman is the container Swiss-Army knife from RedHat:

  • multiple image formats including the OCI and Docker image formats
  • full management of container lifecycle
  • container image management (managing image layers, overlay filesystems, etc)
  • Podman 3.4+ now support M1 Apple Macs
  • Replaces Docker for Desktop
brew install podman
podman machine init
podman machine start
podman infopodman run registry.fedoraproject.org/fedora:latest echo helloalias docker=podman

Conclusion

Well, this list is not 100% complete. Who could build and maintain such a list ?
Kubernetes is still a highly evolving ecosystem and new tools and patterns keeps emerging every week.

Feel free to comment and share your favorite tools and I’ll add them here.

This post is abstracted from a talk that I built for CNCF/Kubernetes Canada Meetups. Please contact me if you’re organizing a meetup and are interested.

Kubernetes
Cncf
K8s
Kubectl

--

--

Prune

Written by Prune

155 Followers

Ops in a world of Dev

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams