📖K8s

Conceptions

k3s is a CNCF-certified Kubernetes distribution originally created by Rancher and now maintained under SUSE. It is packaged as a single binary and is designed for edge devices, IoT, CI environments, and single-node clusters.

The important thing to understand is that k3s is Kubernetes. All the objects, APIs and controllers described below are upstream Kubernetes concepts. What k3s changes is the packaging, the defaults, and which auxiliary components are bundled.

Fundamental ideas

Declarative state

You do not tell Kubernetes to "start a container". You submit an object that describes a desired state, and a controller continuously compares reality against that description and repairs the difference.

This cycle is called *reconciliation*. It is why deleting a Pod does not make it stay deleted: the owning controller notices that the actual state no longer matches the desired state and creates a replacement.

Objects and kinds

Everything stored in Kubernetes is an *object* of some *kind*: a Pod, a Service, a Namespace, a Deployment.

Every object manifest carries:

  • apiVersion — the API group and version, such as v1 or apps/v1

  • kind — the type of object

  • metadata — name, namespace, labels, annotations

  • spec — the desired state, written by you

  • status — the observed state, written by controllers

The split between spec and status is central: you own the first, the system owns the second.

Cluster architecture

Server and agent

k3s uses a two-role model:

  • *Server* — runs the control plane. Installed with k3s server. By default a server also runs an agent, so it can schedule and run workloads on itself. This is what makes a single-machine k3s cluster work.

  • *Agent* — runs only the worker components and joins an existing cluster. Installed with k3s agent, pointing at a server URL and authenticating with a node token.

Control plane

The control plane is the set of processes that make decisions and hold the cluster state.

  • kube-apiserver — the HTTP API and the only component that talks directly to the datastore. Every kubectl command and every controller goes through it.

  • kube-scheduler — watches for Pods that have no node assigned and chooses one for them.

  • kube-controller-manager — runs the built-in reconciliation loops: Deployments, ReplicaSets, Namespaces, Service endpoints, and so on.

  • *Datastore* — the persistent record of every object in the cluster.

Datastore

This is the largest architectural difference between k3s and a textbook Kubernetes installation.

  • *SQLite* — the default. A single local file. Perfectly adequate for one server, and the reason k3s has such a small footprint. It cannot be shared between servers.

  • *Embedded etcd* — a distributed key/value store running inside k3s. Enabled with --cluster-init on the first server; further servers join it. This is what allows a highly available control plane without provisioning an external database.

  • *External datastore* — k3s can also use etcd, MySQL or PostgreSQL over the network via --datastore-endpoint.

Workloads

Pod

A *Pod* is the smallest unit Kubernetes schedules. It is a wrapper around one or more containers that share:

  • a network namespace, and therefore one IP address

  • the ability to reach each other over localhost

  • declared volumes

Containers in different Pods do not share a network namespace, even on the same node.

Pods are ephemeral. Their IPs change, they get killed, they get rescheduled. You almost never manage them directly; you manage a controller that owns them.

Deployment and ReplicaSet

A *Deployment* declares an application: which image, how many replicas, how to roll out changes.

The chain of ownership is:

  • Deployment owns a *ReplicaSet*

  • ReplicaSet owns *Pods*

Each rollout creates a new ReplicaSet. This is what makes rollback possible: the old ReplicaSet still exists with its original Pod template.

Isolation and grouping

Namespace

A *Namespace* partitions one cluster into scoped groups of objects. Names only need to be unique within a namespace.

A Namespace is a real API object of kind Namespace. It must be created explicitly. Setting a namespace field in a build tool or a manifest does not create it — it only assigns objects to it.

Important caveats:

  • Namespaces are not a network boundary by default. Pods in different namespaces can communicate unless you add policy.

  • Namespaces are not a hard security boundary. Isolation requires RBAC and, for networking, NetworkPolicy.

  • Deleting a Namespace cascades deletion to the objects inside it.

Labels and selectors

A *selector* is a query over labels. Controllers and Services use selectors to decide which objects they act on.

This indirection is deliberate: a Service does not know or care which Pods exist, only which labels they carry. You can replace every Pod behind a Service without touching the Service.

Networking

CNI and Flannel

*CNI* stands for Container Network Interface. It is a specification for plugins that give Pods their network interfaces and make Pod-to-Pod traffic work across nodes.

k3s ships *Flannel*. It assigns each node a subnet from the pod CIDR and connects those subnets. The default backend encapsulates traffic in VXLAN tunnels, which is why an overlay network resembles a VPN: Pods address each other with virtual IPs that are meaningless outside the cluster.

Service

A *Service* provides a stable virtual endpoint in front of a changing set of Pods.

Types:

  • *ClusterIP* — a virtual IP reachable only inside the cluster. The default.

  • *NodePort* — reserves a port in the range 30000–32767 on every node. Traffic arriving on any node is forwarded to a backend Pod, even if that Pod runs on a different node.

  • *LoadBalancer* — requests an externally reachable address. In a cloud this provisions a real load balancer; in k3s it is handled by ServiceLB.

Ingress and Traefik

A *Service* is Layer 4. An *Ingress* is a Layer 7 routing object: it maps hostnames and URL paths to Services, and can terminate TLS.

An Ingress object is inert without a controller. k3s bundles *Traefik*, which watches Ingress resources and configures itself accordingly.

Storage

PV, PVC and StorageClass

Three objects work together:

  • *PersistentVolume (PV)* — a concrete piece of storage in the cluster.

  • *PersistentVolumeClaim (PVC)* — a request for storage: a size, an access mode, optionally a storage class.

  • *StorageClass* — a template describing how storage should be dynamically provisioned when a PVC asks for it.

You normally write only a PVC. A *provisioner* watches for unbound PVCs and creates the matching PV automatically. This is called *dynamic provisioning*.

local-path

k3s bundles a *local-path provisioner*, exposed as the default StorageClass local-path.

It simply creates a directory on the node that hosts the Pod, by default under /var/lib/rancher/k3s/storage.

Consequences that matter:

  • The resulting PV has *node affinity*. The Pod can only be scheduled on that node again.

  • Data is not replicated. Losing the node means losing the data unless you have backups.

For a single-node cluster this is entirely appropriate. For multi-node workloads you need either node pinning or a storage system that spans nodes.

k3s-specific machinery

The Helm controller

k3s includes a small built-in controller that installs Helm charts declaratively. It watches for HelmChart and HelmChartConfig custom resources.

This is how the bundled components — Traefik, CoreDNS, metrics-server, local-path — are installed. You do not run helm install yourself; you describe the desired chart and k3s reconciles it.

containerd and image loading

Because k3s uses its own containerd instance, images built into the Docker daemon are invisible to it.

Images must either be pulled from a registry that the node can reach, or explicitly imported into k3s's containerd image store. The k3s containerd socket lives under /run/k3s/containerd, and k3s exposes its own ctr subcommand for image management.

For anything beyond a single-node experiment, a registry is the correct solution.