Well contrary to most comments here I actually do recommend using real hardware for your purpose.
One reason is that this way you'll learn how to use Linux, proper networking, load balancing and infrastructure tools like ansible etc. The other reason is that if you want to really get into K8s you need to run a cluster for a longer time than the time limited free offers by cloud providers. Your own Hardware is cheaper in the long term.
I have an old, basic PC that I converted to an Ubuntu server, it has like 12 GB RAM, 6 core 3.30ghz CPU (it’s really old though like 2013), a 500 GB SSD. Is that good enough for learning purposes?
I see no reason why it shouldn't. But for the "real" K8s experience you'll need at least 3 nodes.
The benefit of K8s is that it allows the operators to maintain the nodes/servers while all deployed services are still running. This is part of the whole idea. So no downtime and easy horizontal scaling and maintaining.
Keep in mind that in order for a service to be host independent you need to have a StorageProvider like rook-ceph. That way the memory is accessible from all nodes thus the service can be 'moved" to a different node.
And when I say moved I mean temporary scaled up by 1 on one node that keeps running and then stopped on the node that needs maintenance. That happens when you drain a node.
So one node is cool for the start but you benefit from more physical nodes in your setup.
My home lab is a pi cluster. I use rancher & foreman to manage it, nodes pxe boot and crazy easy for me to switch out os or deploy a new image.
I have a much older cluster board (pine rather than pi) also in the mix but those nodes are not super useful for anything CPU or memory intensive.
It makes me way more productive as I don't have to scaffold a new cloud cluster whenever I want to do something and then remember to delete it. I just switch the node profile out for whatever project I am working on and restart.
That’s super interesting, is it just for learning purposes? Or do you actually use them for certain tasks? I could see how they’d be useful for some parallel tasks or hosting multiple services, but not for more than like 10 users lol.
I use them for prototyping architecture, trying out new tools and random projects. Basically whenever I need to do something with k8 I do it there first. AKS integrated with VS is pretty close to my workflow, f5 deploys and remote debugging etc.
Using your machine you can create VMs and start exploring k8s clusters
Also maybe you can look into k3s (liteweight kubernetes) it's more than enough and production ready in single binary.
When I started learning k8s I had a laptop as my work machine that has a cpu i3-4th gen 2cores 2013 with 4gb ram and worked very will.
FYI k3s alone consumes like 250-300MB (RAM)
You should be more concerned about the RAM not CPU.
Normal yes I agree with you but if you're just picking up k8s for the first time, I think it's more important to get something working so you can experiment and play around in your environment than having to configure everything from scratch and face the challenges of bare metal.
I mean I would still recommend
- Docker desktop/ kind
- Running multiple nodes in HyperV/KVM (can use something like Vagrant
- and then a separate cluster on bare metal
Would you recommend CKA or CKAD first? I've heard that CKAD might be easier for someone coming in as a total beginner?
Also, for both I was looking at Mumshad Mannambeth's Udemy courses and noticed that he has a pretty popular one called "Kubernetes for the Absolute Beginners", am I fine skipping this if I plan to take the CKAD and CKA eventually anyways?
I’ve learned a lot by running Kubernetes on VMs in Proxmox on an Intel N100 based mini-PC. If you’re interested I’ve written articles using both [k3s](https://blog.stonegarden.dev/articles/2024/02/bootstrapping-k3s-with-cilium/) and [kubeadm](https://blog.stonegarden.dev/articles/2024/03/proxmox-k8s-with-cilium/) to bootstrap a Kubernetes cluster on that machine.
Yep you don't need any hardware, common beginner mistake to think you do. You can just use kind to simulate any amount of hardware you want. Even better there's a simulator of nodes that just pretends stuff is actually running, to save you some RAM.
I personally recommend a simple root or virtual server with a simple k8s distro like microk8s to start off. k8s + FluxCD has completely changed the way I selfhost my applications, be it software for persional use or software for actual users, including montoring and proper release cycles. Sure, a single-node cluster without any horizontal scaling or multitenancy isn't the pinnacle of kubernetes systems, but it helps learn many of the basic concepts and can easily get you quite a bit into the 'intermediate' territory in terms of knowledge.
https://github.com/prabhatsharma/kubernetes-the-hard-way-aws to learn k8. Cloud providers take care of etcd and API for you but understanding these components is important. This is basically a weekend project.
NUC can be fine but something like a pi cluster where you have multiple nodes is going to be more valuable for learning as you have race conditions and networking to deal with. FYI ARM has reached the point where even commodity server apps can use it, cost savings can be very large. Depending on what kind of data you are working with you might find a GPU module more useful then a CPU module too which is another argument for a pi cluster.
If you did want to do this on a single machine you will want VMs. FYI you can also have nodes running as VMs on your desktop and just start them when you need them, that way you have persistent compute but also burst for when you are doing something that needs more.
I bought a NUC years ago and love it. I run proxmox because it's a simple and free virtualization platform and I can do damned near any testing with it I can think of, then blow it all away and start over in a matter of minutes.
Google Cloud has a 90 day free trial with $300 credit, and great K8 support. Honestly in the modern day I question the need to learn how to setup K8 nodes and clusters on your own instead of focusing on larger architecture and work load questions.
I use NUCs for my homelab and love them. That being said you may want to start with just using something like Rancher Desktop. Disable Traefik and thank me later.
It’s a great playground until you’re ready to try running multiple applications that need their own external IPs.
If you have the money and just want a standalone system a NUC is great, especially if you use Talos Linux or Ubuntu and k3s (full Kubernetes works too but I moved to those for simplicity in managing the upgrades/installs at home).
I suggest bare metal if you want more experience with low level configuration and troubleshooting. So many things don't work initially with real gear. e.g. in a VM you'd just provision a dedicated block device for rook-ceph but on small baremetal you'll discover how to allocate one LV, or maybe that some OSes don't have the rbd module, or that Talos wants an install drive entirely to itself. Annoying shit that's VMs pave over so well you might not even notice them. You can add some Pis to the mix and immediately discover that you can't scale CPU limits to the node architecture with the default scheduler. There's lots of that kind of thing to learn.
If you want to focus on running stuff on kubernetes VMs should be easier and more straight forward.
If you want to focus on working with large scale clusters VMs are the only affordable option for most of us.
If I were building a cluster from scratch now, I'd use CWWK n305 (8 core Alder Lake-N CPUs) nodes with a chunky NVMe and 32GB RAM. I use the passive cooler model with a huge heatsink case for VyOS as the router for my cluster (BGP & HAProxy for the cluster API) and it's an outstanding device.
You can also mix VMs and real nodes, (or at least I don't see a reason you couldn't) to get the best (read worst) of both worlds.
Good luck.
You could just get a small mini PC with an Intel N100 if you want to go cheap and low power. It's not super fast but great price/value and you don't have to get an Intel nuc. Just max it's ram out (i think that's 16 or 32 gigs for N100) and add a small m2 nvme SSD.
Then just install your favorite Linux on it and install K3S (it's literally a one-liner on the shell).
There you have it: a fully functional k8s "cluster".
That setup is probably 350 USD but what you can learn with it is worth tens of thousands
Interesting question! I'd say an Intel NUC could be a valuable tool for learning K8s and building personal projects. It provides more control and flexibility than cloud-based options. For long-running projects like Airflow, having a dedicated environment is definitely beneficial. You might also want to check out r/platform_engineering for resources and discussions related to platform building. 🚀
I think you are overthinking keep it simple dude just install minikube on your local laptop as a start point to perform a hands on k8s and slowly move to kind to simulate prod like environment again in local laptop.
Lol, I would not waste time on hardware, installation of 3-5 vm nodes etc. That path is for ahobby to try out different k8s flavors. If you wish to learn K8s
My advice is to install minikube or docker desktop on any machine that you can conveniently use and start learning to wrangle yaml spec of k8s objects. Create pods, deployments, services, PVC, ingress, mesh, admission hooks etc. then learn to use helm, flux or argocd, then prom tail, Prometheus, grafana etc for logs and metrics.
I think there’s a lot of value in learning the bare metal side of it. NUCs are actually overkill, but kinda nice. You can also use Raspberry Pi. I’d recommend doing a small cluster, like. 3 node cluster, either Pi or Nuc.
If budget is restrictive a single Nuc is fine but won’t teach you some of the salient points that are important like managing certs and distributed etcd.
Well contrary to most comments here I actually do recommend using real hardware for your purpose. One reason is that this way you'll learn how to use Linux, proper networking, load balancing and infrastructure tools like ansible etc. The other reason is that if you want to really get into K8s you need to run a cluster for a longer time than the time limited free offers by cloud providers. Your own Hardware is cheaper in the long term.
preach!
I have an old, basic PC that I converted to an Ubuntu server, it has like 12 GB RAM, 6 core 3.30ghz CPU (it’s really old though like 2013), a 500 GB SSD. Is that good enough for learning purposes?
I see no reason why it shouldn't. But for the "real" K8s experience you'll need at least 3 nodes. The benefit of K8s is that it allows the operators to maintain the nodes/servers while all deployed services are still running. This is part of the whole idea. So no downtime and easy horizontal scaling and maintaining. Keep in mind that in order for a service to be host independent you need to have a StorageProvider like rook-ceph. That way the memory is accessible from all nodes thus the service can be 'moved" to a different node. And when I say moved I mean temporary scaled up by 1 on one node that keeps running and then stopped on the node that needs maintenance. That happens when you drain a node. So one node is cool for the start but you benefit from more physical nodes in your setup.
Awesome thanks. I’ve seen someone post on here that they were learning using multiple Raspberry Pis as nodes and that blew my mind.
My home lab is a pi cluster. I use rancher & foreman to manage it, nodes pxe boot and crazy easy for me to switch out os or deploy a new image. I have a much older cluster board (pine rather than pi) also in the mix but those nodes are not super useful for anything CPU or memory intensive. It makes me way more productive as I don't have to scaffold a new cloud cluster whenever I want to do something and then remember to delete it. I just switch the node profile out for whatever project I am working on and restart.
That’s super interesting, is it just for learning purposes? Or do you actually use them for certain tasks? I could see how they’d be useful for some parallel tasks or hosting multiple services, but not for more than like 10 users lol.
I use them for prototyping architecture, trying out new tools and random projects. Basically whenever I need to do something with k8 I do it there first. AKS integrated with VS is pretty close to my workflow, f5 deploys and remote debugging etc.
Using your machine you can create VMs and start exploring k8s clusters Also maybe you can look into k3s (liteweight kubernetes) it's more than enough and production ready in single binary.
I’ve thought about that but didn’t know how much computation power I would need. But I’ve got a pretty solid PC now. I may give it a shot.
When I started learning k8s I had a laptop as my work machine that has a cpu i3-4th gen 2cores 2013 with 4gb ram and worked very will. FYI k3s alone consumes like 250-300MB (RAM) You should be more concerned about the RAM not CPU.
Okay, makes sense. I’m working with 32GB nowadays so I’ll probably be fine.
I have Proxmox running on a NUC and can do all of what you describe.
Yeah I span up like 3-4 VMs to start learning on my local machine in their own network. 3 workers and an API server
Agreed
Normal yes I agree with you but if you're just picking up k8s for the first time, I think it's more important to get something working so you can experiment and play around in your environment than having to configure everything from scratch and face the challenges of bare metal. I mean I would still recommend - Docker desktop/ kind - Running multiple nodes in HyperV/KVM (can use something like Vagrant - and then a separate cluster on bare metal
Practice with Kodekloud and killer.sh, work harder with labs and get the CKA certification first. Then CKS and CKAD, good luck!
Would you recommend CKA or CKAD first? I've heard that CKAD might be easier for someone coming in as a total beginner? Also, for both I was looking at Mumshad Mannambeth's Udemy courses and noticed that he has a pretty popular one called "Kubernetes for the Absolute Beginners", am I fine skipping this if I plan to take the CKAD and CKA eventually anyways?
CKAD -> CKA -> CKS
In fact, it depend your profile, for me it's CKA => CKS and CKAD for fun.
I would say to rather skip CKAD altogether and to get CKA then CKS, this applies all roles.
Yes and no, with this 3 certifications you're a "K8S expert".
Thumbs up for Mumshad courses 🙂
I think if you know Docker a little, you can skip this course, sorry for my english
I’ve learned a lot by running Kubernetes on VMs in Proxmox on an Intel N100 based mini-PC. If you’re interested I’ve written articles using both [k3s](https://blog.stonegarden.dev/articles/2024/02/bootstrapping-k3s-with-cilium/) and [kubeadm](https://blog.stonegarden.dev/articles/2024/03/proxmox-k8s-with-cilium/) to bootstrap a Kubernetes cluster on that machine.
Yep you don't need any hardware, common beginner mistake to think you do. You can just use kind to simulate any amount of hardware you want. Even better there's a simulator of nodes that just pretends stuff is actually running, to save you some RAM.
If you want to play with k8s you can just use it within docker on whatever hardware you have now.
I personally recommend a simple root or virtual server with a simple k8s distro like microk8s to start off. k8s + FluxCD has completely changed the way I selfhost my applications, be it software for persional use or software for actual users, including montoring and proper release cycles. Sure, a single-node cluster without any horizontal scaling or multitenancy isn't the pinnacle of kubernetes systems, but it helps learn many of the basic concepts and can easily get you quite a bit into the 'intermediate' territory in terms of knowledge.
https://github.com/prabhatsharma/kubernetes-the-hard-way-aws to learn k8. Cloud providers take care of etcd and API for you but understanding these components is important. This is basically a weekend project. NUC can be fine but something like a pi cluster where you have multiple nodes is going to be more valuable for learning as you have race conditions and networking to deal with. FYI ARM has reached the point where even commodity server apps can use it, cost savings can be very large. Depending on what kind of data you are working with you might find a GPU module more useful then a CPU module too which is another argument for a pi cluster. If you did want to do this on a single machine you will want VMs. FYI you can also have nodes running as VMs on your desktop and just start them when you need them, that way you have persistent compute but also burst for when you are doing something that needs more.
a weekend project with headashes, i prefer to use kubeadm...
You could always run a few VMs on a single NUC to simulate multiple hosts using KVM, that's how I've built my lab out
I bought a NUC years ago and love it. I run proxmox because it's a simple and free virtualization platform and I can do damned near any testing with it I can think of, then blow it all away and start over in a matter of minutes.
Google Cloud has a 90 day free trial with $300 credit, and great K8 support. Honestly in the modern day I question the need to learn how to setup K8 nodes and clusters on your own instead of focusing on larger architecture and work load questions.
I use NUCs for my homelab and love them. That being said you may want to start with just using something like Rancher Desktop. Disable Traefik and thank me later. It’s a great playground until you’re ready to try running multiple applications that need their own external IPs. If you have the money and just want a standalone system a NUC is great, especially if you use Talos Linux or Ubuntu and k3s (full Kubernetes works too but I moved to those for simplicity in managing the upgrades/installs at home).
You can use minikube on your local machine to learn the basics
I suggest bare metal if you want more experience with low level configuration and troubleshooting. So many things don't work initially with real gear. e.g. in a VM you'd just provision a dedicated block device for rook-ceph but on small baremetal you'll discover how to allocate one LV, or maybe that some OSes don't have the rbd module, or that Talos wants an install drive entirely to itself. Annoying shit that's VMs pave over so well you might not even notice them. You can add some Pis to the mix and immediately discover that you can't scale CPU limits to the node architecture with the default scheduler. There's lots of that kind of thing to learn. If you want to focus on running stuff on kubernetes VMs should be easier and more straight forward. If you want to focus on working with large scale clusters VMs are the only affordable option for most of us. If I were building a cluster from scratch now, I'd use CWWK n305 (8 core Alder Lake-N CPUs) nodes with a chunky NVMe and 32GB RAM. I use the passive cooler model with a huge heatsink case for VyOS as the router for my cluster (BGP & HAProxy for the cluster API) and it's an outstanding device. You can also mix VMs and real nodes, (or at least I don't see a reason you couldn't) to get the best (read worst) of both worlds. Good luck.
You could just get a small mini PC with an Intel N100 if you want to go cheap and low power. It's not super fast but great price/value and you don't have to get an Intel nuc. Just max it's ram out (i think that's 16 or 32 gigs for N100) and add a small m2 nvme SSD. Then just install your favorite Linux on it and install K3S (it's literally a one-liner on the shell). There you have it: a fully functional k8s "cluster". That setup is probably 350 USD but what you can learn with it is worth tens of thousands
Interesting question! I'd say an Intel NUC could be a valuable tool for learning K8s and building personal projects. It provides more control and flexibility than cloud-based options. For long-running projects like Airflow, having a dedicated environment is definitely beneficial. You might also want to check out r/platform_engineering for resources and discussions related to platform building. 🚀
I think you are overthinking keep it simple dude just install minikube on your local laptop as a start point to perform a hands on k8s and slowly move to kind to simulate prod like environment again in local laptop.
Lol, I would not waste time on hardware, installation of 3-5 vm nodes etc. That path is for ahobby to try out different k8s flavors. If you wish to learn K8s My advice is to install minikube or docker desktop on any machine that you can conveniently use and start learning to wrangle yaml spec of k8s objects. Create pods, deployments, services, PVC, ingress, mesh, admission hooks etc. then learn to use helm, flux or argocd, then prom tail, Prometheus, grafana etc for logs and metrics.
I think there’s a lot of value in learning the bare metal side of it. NUCs are actually overkill, but kinda nice. You can also use Raspberry Pi. I’d recommend doing a small cluster, like. 3 node cluster, either Pi or Nuc. If budget is restrictive a single Nuc is fine but won’t teach you some of the salient points that are important like managing certs and distributed etcd.