Chapter 4: Registering with Kubelet

Right now, localdir-csi is a binary you run by hand, on your own laptop, listening on a socket in a directory you created yourself. That's been exactly the right amount of complexity for learning how the Identity service works. It is not, however, a driver Kubernetes knows exists. Nothing about kind create cluster or kubectl has any idea your binary is out there.

This chapter closes that gap: your driver goes into a container image, that image gets deployed to your kind cluster as a real Kubernetes workload, and — with the help of one small sidecar — kubelet finds out it exists. By the end, you'll have watched a real Kubernetes node discover your driver over a socket, the same way it discovers every CSI driver in every real cluster. You'll also hit a genuine wall, on purpose: kubelet wants one more thing from your driver before it'll fully trust it, and we haven't written that thing yet. That wall is exactly what Chapter 5 is for.

One binary, two homes

Open cmd/localdir-csi/main.go as it stands at the end of Chapter 3. Two values are hardcoded as constants: the socket path (/csi/csi.sock) and the health checker's root directory (/data). Those are the right paths for a container — they're where we'll mount things in a moment — but they're the wrong paths to run directly on your laptop, where writing to /csi and /data means writing to your machine's real filesystem root, which will either fail outright (no permission) or, worse, succeed and leave files somewhere you didn't intend.

Chapter 3 sidestepped this by having you hardcode local-friendly values — ./csi/csi.sock and ./data — instead, with a note that we'd deal with the container paths "for real" once we actually had a container to put them in. Now we do, and the honest problem is: one binary needs to listen in two different places, depending on where it's running, and a hardcoded constant can only ever be right in one of them.

This is what command-line flags are for. You've been passing flags to other programs all through this book — kind create cluster --name csi-dev — without needing to write one yourself yet. Go's standard library ships everything for it in the flag package: you declare a named flag, a default value, and a one-line description; call flag.Parse() once, early in main; and from then on you have a value that's whatever the program was actually invoked with, or the default if it wasn't given.

We'll add two: -endpoint, for where the driver listens, and -data-dir, for where the health checker looks. Defaults matter here — default to whatever's right for the container, since that's where this binary spends most of its life, and override explicitly for local runs.

A small, testable piece: turning an endpoint into a socket path

-endpoint needs a default value, and it's worth choosing that value deliberately rather than arbitrarily. Real CSI drivers — the ones this book is modeled on — universally accept their endpoint as a URI with a unix:// scheme, not a bare path: something like unix:///csi/csi.sock. You've already met this exact scheme once, in Chapter 3, typing it yourself on the grpcurl command line. Reusing it here isn't a coincidence — it's the same idea ("this is a filesystem path, not a host:port pair") showing up on the server side instead of the client side, and it means anyone who's used grpcurl against a CSI driver before will recognize the convention immediately.

The catch, also from Chapter 3: unix:// URIs are easy to get subtly wrong. unix:///csi/csi.sock (three slashes) is correct for an absolute path. unix:./csi/csi.sock (one colon, no slashes at all) is correct for a relative one. unix://./csi/csi.sock (two slashes plus a relative path) looks plausible and is silently wrong — it hands . to the URI's host portion and drops the leading . from the path entirely. Since net.Listen has no idea what a unix:// URI is — it wants a plain filesystem path — something in our code has to parse the endpoint string and pull the real path back out, correctly, for all of these forms.

That parsing is a small, pure function: string in, path out, no network, no gRPC, nothing that needs a live server to test. Exactly the kind of thing to write test-first.

Create internal/driver/endpoint_test.go:

package driver

import "testing"

func TestSocketPathFromEndpoint(t *testing.T) {
	tests := []struct {
		name     string
		endpoint string
		want     string
		wantErr  bool
	}{
		{name: "absolute path, three slashes", endpoint: "unix:///csi/csi.sock", want: "/csi/csi.sock"},
		{name: "relative path, one colon, no slashes", endpoint: "unix:./csi/csi.sock", want: "./csi/csi.sock"},
		{name: "wrong scheme is rejected", endpoint: "tcp://localhost:8080", wantErr: true},
		{name: "missing scheme is rejected", endpoint: "/csi/csi.sock", wantErr: true},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			got, err := SocketPathFromEndpoint(tt.endpoint)
			if tt.wantErr {
				if err == nil {
					t.Fatalf("SocketPathFromEndpoint(%q) = %q, want an error", tt.endpoint, got)
				}
				return
			}
			if err != nil {
				t.Fatalf("SocketPathFromEndpoint(%q) returned an error: %v", tt.endpoint, err)
			}
			if got != tt.want {
				t.Errorf("SocketPathFromEndpoint(%q) = %q, want %q", tt.endpoint, got, tt.want)
			}
		})
	}
}

Run it:

go test ./internal/driver/...

Red, the way every first test in this book has been red, because SocketPathFromEndpoint doesn't exist yet:

# github.com/yourname/localdir-csi/internal/driver [github.com/yourname/localdir-csi/internal/driver.test]
./endpoint_test.go:20:16: undefined: SocketPathFromEndpoint
FAIL	github.com/yourname/localdir-csi/internal/driver [build failed]
FAIL

Now write it. Per Single Responsibility — the same principle Chapter 3 pointed at for LocalDirHealthChecker — this doesn't belong crammed into driver.go alongside the Driver struct itself; parsing an endpoint string is a self-contained job with nothing to do with what Driver is or does. Give it its own file, internal/driver/endpoint.go:

package driver

import (
	"fmt"
	"net/url"
)

// SocketPathFromEndpoint extracts a filesystem path from a "unix:"
// endpoint URI. This is deliberately the same parsing rule grpc-go's own
// unix resolver uses internally — url.Path for a "unix://" form with the
// double slash ("unix:///abs/path"), falling back to url.Opaque for the
// single-colon form without one ("unix:relative/path") — so a value that
// behaves correctly as a grpcurl target also behaves correctly here.
func SocketPathFromEndpoint(endpoint string) (string, error) {
	u, err := url.Parse(endpoint)
	if err != nil {
		return "", fmt.Errorf("parsing endpoint %q: %w", endpoint, err)
	}
	if u.Scheme != "unix" {
		return "", fmt.Errorf("endpoint %q has scheme %q, want \"unix\"", endpoint, u.Scheme)
	}
	path := u.Path
	if path == "" {
		path = u.Opaque
	}
	if path == "" {
		return "", fmt.Errorf("endpoint %q has no socket path", endpoint)
	}
	return path, nil
}

go test ./internal/driver/...

ok  	github.com/yourname/localdir-csi/internal/driver	0.002s

Green — all four cases, including both rejected forms.

Updating main.go to use it

With SocketPathFromEndpoint written and tested, wiring it into main.go is mechanical. Replace the whole file:

package main

import (
	"flag"
	"log"
	"net"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"

	"github.com/container-storage-interface/spec/lib/go/csi"
	"github.com/yourname/localdir-csi/internal/driver"
)

const (
	driverName    = "localdir.csi.example.com"
	driverVersion = "0.1.0"
)

func main() {
	endpoint := flag.String("endpoint", "unix:///csi/csi.sock",
		"gRPC endpoint the driver listens on, as a unix:// URI")
	dataDir := flag.String("data-dir", "/data",
		"directory the health check reads and writes to confirm local storage is usable")
	flag.Parse()

	socketPath, err := driver.SocketPathFromEndpoint(*endpoint)
	if err != nil {
		log.Fatalf("invalid -endpoint: %v", err)
	}

	// If a socket file from a previous run is still sitting there, remove
	// it. Without this, trying to listen on the same path a second time
	// fails with "address already in use" — a file on disk, not an
	// actual network port, but Go's net package still treats it that way.
	if err := os.Remove(socketPath); err != nil && !os.IsNotExist(err) {
		log.Fatalf("failed to remove existing socket: %v", err)
	}

	listener, err := net.Listen("unix", socketPath)
	if err != nil {
		log.Fatalf("failed to listen on %s: %v", socketPath, err)
	}

	d := driver.NewDriver(
		driverName,
		driverVersion,
		&driver.LocalDirHealthChecker{Root: *dataDir},
	)

	server := grpc.NewServer()
	csi.RegisterIdentityServer(server, d)
	reflection.Register(server)

	log.Printf("localdir-csi listening on %s", socketPath)
	if err := server.Serve(listener); err != nil {
		log.Fatalf("server stopped: %v", err)
	}
}

Everything below the flag declarations is exactly what Chapter 3 built — the only change is that socketPath and *dataDir are now read from the command line instead of baked in.

Update the run target in your Makefile to pass local-friendly values explicitly, since the defaults are now the container's paths, not your laptop's:

run:
	mkdir -p ./csi ./data
	go run ./cmd/localdir-csi -endpoint=unix:./csi/csi.sock -data-dir=./data

make run and every grpcurl command from Chapter 3 still work exactly as before — the only thing that changed is that they now say so explicitly on the command line instead of it being the only option.

Packaging it: the Dockerfile

Kubernetes doesn't run Go binaries. It runs containers — a container image is a filesystem plus some metadata (what to run, as what user) bundled up so it can be shipped to any machine and run identically there. A Dockerfile is the recipe for building one: a short script of instructions, each producing one layer of that filesystem.

We want two very different things out of this build. First, a Go toolchain, to compile cmd/localdir-csi into a binary — that toolchain is enormous (hundreds of megabytes) and completely useless once the binary exists. Second, the smallest possible image to actually ship and run — every extra file in a running container is one more thing that could theoretically be exploited if the container is ever compromised. A multi-stage build is Docker's answer to wanting both: a Dockerfile with more than one FROM line, where each FROM starts a fresh, separate filesystem, and a later stage can selectively copy specific files out of an earlier one — the compiled binary, and nothing else the compiler needed to produce it.

Each instruction below does one plain thing: FROM picks the starting filesystem for a stage, WORKDIR picks the directory later instructions run in, COPY copies files in from outside the image, RUN runs a command while the image is being built (not later, when it's running), and ENTRYPOINT is the one command that actually runs when a container starts from the finished image.

Create Dockerfile in the project root:

# Stage 1: compile the binary. This image has the full Go toolchain and
# nothing about it ends up in the final image except whatever we
# explicitly COPY out of it below.
FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/localdir-csi ./cmd/localdir-csi

# Stage 2: the image that actually ships. "distroless" means exactly what
# it sounds like — no shell, no package manager, no coreutils, nothing
# beyond the C library our binary is linked against. There's nothing here
# for an attacker to get a shell with even if they found a way in, and
# nothing here to patch for vulnerabilities that don't apply to a program
# that never asked for them.
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/localdir-csi /localdir-csi
ENTRYPOINT ["/localdir-csi"]

CGO_ENABLED=0 matters specifically because of that second stage: distroless/static has no C library at all, so a binary that dynamically links against one (the default, if any dependency ever pulls in cgo) would fail to start with a missing-library error the moment it hit that image. Setting CGO_ENABLED=0 tells Go's compiler to produce a fully self-contained binary instead — nothing to link against at runtime, because nothing was linked in the first place.

make build from Chapter 2 already runs docker build -t localdir-csi:dev . — it just had nothing to find before now. Try it:

make build

What node-driver-registrar is actually for

Before looking at where its socket lives or what it logs, it's worth being clear about the one job node-driver-registrar exists to do, in plain terms: it's the thing that tells kubelet your driver exists at all.

Think of kubelet as a building's front-desk security guard, deciding which companies are allowed to serve people on that floor. Your driver is a brand-new company that just moved in — it's up and running, fully capable of doing its job, but nobody at the front desk has heard of it yet. It's just a process quietly listening on a socket file, and a socket file sitting on disk isn't something kubelet automatically notices.

node-driver-registrar's entire purpose is to walk up to that front desk on your driver's behalf and say "there's a new company here, this is its name, and this is the door it's behind." Concretely, it does that in two steps: first it asks your own driver "what's your name?" (a gRPC call to GetPluginInfo, the exact method Chapter 3 wrote), and then it writes that name, plus the path to your driver's socket, into a directory kubelet is constantly watching. The moment that new information shows up, kubelet notices it and goes to introduce itself to your driver directly.

Without node-driver-registrar, none of that introduction happens. localdir-csi could be running perfectly, listening on its socket, completely healthy — and kubelet would simply never know to knock on its door, the same way a security guard has no reason to check a floor for new tenants nobody told them about.

Where the driver actually lives on a node

Chapter 1 described sidecars as sharing a Unix socket with your driver over an emptyDir volume — true, and it's exactly how the Controller side will work starting in Chapter 7, where external-provisioner and your Controller service are two containers in the very same pod. The Node side, and specifically node-driver-registrar, needs one more layer of precision on that idea, because of who has to reach the socket.

An emptyDir volume is only visible to containers inside the same pod. Kubelet is not a container in your pod — it's a process running directly on the node itself, outside of Kubernetes entirely, the same way dockerd or sshd might be. For kubelet to ever open your driver's socket, that socket has to live somewhere kubelet can already see without going through Kubernetes at all: a real directory on the node's own filesystem. That's a hostPath volume — instead of Kubernetes conjuring up empty, pod-scoped storage, it hands your container a bind mount straight into a path that already exists on the host.

By convention, every CSI driver's socket lives at /var/lib/kubelet/plugins/<driver-name>/ on the host — kubelet expects one subdirectory per driver, named after that driver's own name. Mount that same host directory into both your driver container and the registrar sidecar, at whatever path each one is configured to look for it, and both containers — and kubelet itself, reading straight off the host — are looking at the exact same file.

The DaemonSet

Chapter 1 already told you why the Node service runs as a DaemonSet rather than an ordinary Deployment: it needs to run on every node, because "attached to a node" and "mounted into a pod on that node" are both inherently per-node facts, not cluster-wide ones. A DaemonSet is Kubernetes's object for exactly that shape — one pod per node, automatically, including on any node added to the cluster later.

Create deploy/node.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: localdir-csi-node
  labels:
    app: localdir-csi
spec:
  selector:
    matchLabels:
      app: localdir-csi
  template:
    metadata:
      labels:
        app: localdir-csi
    spec:
      containers:
        - name: localdir-csi
          image: localdir-csi:dev
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: data-dir
              mountPath: /data
        - name: node-driver-registrar
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/localdir.csi.example.com/
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: data-dir
          hostPath:
            path: /var/lib/localdir-csi/data
            type: DirectoryOrCreate

A few things worth reading carefully here, since they're easy to get subtly wrong:

imagePullPolicy: IfNotPresent matters specifically because of how you're getting this image onto the cluster at all. There's no registry involved — make load (which make deploy depends on, from Chapter 2) runs kind load docker-image, which copies your locally built image directly into every kind node. Without IfNotPresent, kubelet's default behavior would be to try pulling localdir-csi:dev from a real registry, fail, and never discover the image already sitting right there on the node.

socket-dir is mounted at /csi in both containers, backed by the exact same hostPath. Your driver listens at /csi/csi.sock inside its own container (that's the default we just wired into -endpoint); the registrar's --csi-address=/csi/csi.sock points at the same path inside its container — same file, reached through two different containers' mount points, because both mount points resolve back to the same directory on the host.

--kubelet-registration-path, by contrast, is not a path inside any container — it's the path kubelet itself, running on the host, will use to reach that socket. That's why it's the host path (/var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock), matching socket-dir's hostPath.path plus the socket filename, rather than /csi/csi.sock. Mixing these two up — giving the registrar its own container path where kubelet expects a host path — is a common mistake, and the failure mode is unhelpful: kubelet will simply never find a working socket at the path it was told, with no error pointing at the actual cause.

registration-dir uses type: Directory, not DirectoryOrCreate, on purpose: /var/lib/kubelet/plugins_registry/ is a directory kubelet itself manages and already creates on every node it runs on. If it's missing, something more fundamental than your driver's manifest is wrong, and creating it out from under kubelet isn't the right fix. socket-dir and data-dir, on the other hand, are specific to this driver and won't exist until something creates them — DirectoryOrCreate lets Kubernetes do that the first time this DaemonSet runs.

data-dir isn't used by anything in this chapter — nothing calls Probe against the deployed driver yet — but it's the same /data path LocalDirHealthChecker has depended on since Chapter 3, now given somewhere real on the host to actually write to, instead of silently failing inside an empty container filesystem.

Telling Kubernetes about the driver, cluster-wide: the CSIDriver object

The DaemonSet gets your driver's code running on every node. Kubernetes also wants a small, separate, cluster-scoped object describing facts about the driver itself — not "where is it running," but "what does it need from the rest of the system." That object is CSIDriver, and unlike the DaemonSet, there's exactly one of these per driver, cluster-wide, not one per node.

Create deploy/csidriver.yaml:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: localdir.csi.example.com
spec:
  attachRequired: false
  podInfoOnMount: false
  volumeLifecycleModes:
    - Persistent

attachRequired is the one to get right immediately, and it's worth understanding why rather than just copying the value. When true (the default, if this field is left out entirely), Kubernetes will wait for a separate ControllerPublishVolume call — "attach this volume to this node" — to succeed before it ever tries to mount that volume into a pod. That's the right behavior for storage that genuinely has an attach step, like an EBS volume being attached to an EC2 instance. localdir-csi never will — Chapter 8 implements ControllerPublishVolume for the sake of completeness and to show the pattern, but our storage is already local to every node by definition, so there's nothing to attach. Setting attachRequired: false now tells Kubernetes not to wait for a step this driver — as of this chapter, and permanently in spirit — doesn't need.

podInfoOnMount: false (also the field's default) means Kubernetes won't bother passing pod metadata — name, namespace, service account — into NodePublishVolume calls. Some drivers need that context to make mount-time decisions; ours doesn't, so there's no reason to ask for it.

volumeLifecycleModes: [Persistent] states the obvious but required fact that this driver's volumes are the ordinary kind — created ahead of time via a PersistentVolumeClaim, existing independently of any one pod — as opposed to Ephemeral inline volumes, which this book doesn't cover.

Deploying

Everything's in place. Build, load, and apply:

make deploy

Watch the pods come up:

kubectl get pods -l app=localdir-csi -w

You should see one pod per node in your cluster (just one, for the single-node csi-dev cluster from Chapter 2), with 2/2 containers — localdir-csi and node-driver-registrar — reported as Running.

What actually happens when kubelet finds your socket

Registration isn't one gRPC call — it's a small handshake, and it's worth walking through concretely, because you're about to watch part of it fail, honestly, for a reason that makes complete sense once you see it.

node-driver-registrar's job splits into two halves. First, it acts as a gRPC client to your driver: it dials --csi-address and calls GetPluginInfo, the exact method you wrote in Chapter 3, purely to learn your driver's name. Check its logs to see this happen for real:

kubectl logs -l app=localdir-csi -c node-driver-registrar --tail=50

Somewhere in there you should find lines close to:

Attempting to open a gRPC connection  csiAddress="/csi/csi.sock"
Calling CSI driver to discover driver name
CSI driver name  csiDriverName="localdir.csi.example.com"

Second, having learned its own driver's name, the registrar becomes a gRPC server itself — a tiny one, implementing a completely different, Kubernetes-internal service (not CSI at all) whose only job is answering "what plugin are you, and where's its socket?" It listens for that on its own socket, inside registration-dir. Kubelet's plugin watcher polls that directory continuously; the moment a new socket shows up there, it connects and asks exactly that question.

Getting an answer to that question is where kubelet's own work begins, separately from anything node-driver-registrar does. Kubelet dials your driver's socket directly — the real, actual /var/lib/kubelet/plugins/localdir.csi.example.com/csi.sock, the same file, not through the registrar at all — and, among other calls, tries to invoke NodeGetInfo. That's a method on csi.NodeServer. We haven't written a Node service yet. Not one method of it exists, and main.go never even calls csi.RegisterNodeServer — the whole service is simply absent from what your driver exposes.

You can watch kubelet discover exactly that, from your own laptop, without touching the cluster at all — it's the same failure, over the same kind of socket, that Chapter 3 already gave you the tools to reproduce:

make run

and, in a second terminal:

grpcurl -plaintext unix:./csi/csi.sock csi.v1.Node/NodeGetInfo

ERROR:
  Code: Unimplemented
  Message: unknown service csi.v1.Node

That's not "not implemented" in the polite, UnimplementedNodeServer sense Chapter 3 walked through for Identity — there's no csi.v1.Node service registered on this server at all, so gRPC can't even find a stub to answer with. It's the plainest, earliest kind of "this doesn't exist yet" a gRPC server can report.

Back in the cluster, kubelet gets that exact same error when it tries NodeGetInfo for real, and it treats that as a failed registration — which node-driver-registrar takes seriously enough to exit and let Kubernetes restart it, rather than silently limping along half-registered. Watch it happen:

kubectl get pods -l app=localdir-csi -w

Give it a few seconds and you should see the RESTARTS count on the pod start climbing. Because the container that just crashed is gone by the time you'd normally check its logs, you need --previous to see what it said on its way out:

kubectl logs -l app=localdir-csi -c node-driver-registrar --previous

Look for a line mentioning registration failing — the exact wording varies by version, but the cause underneath it is always this same missing NodeGetInfo.

One more angle on the same fact, using an object you haven't met yet. Kubernetes tracks which CSI drivers are actually, successfully registered on each node in a CSINode object — one per node, cluster-scoped, not something your driver creates or touches directly; kubelet manages it entirely on its own, updating it only once a driver's registration fully succeeds.

kubectl get nodes
kubectl get csinode <node-name> -o yaml

Look under spec.drivers. localdir.csi.example.com isn't there. As far as Kubernetes is concerned, your driver introduced itself, and kubelet still doesn't trust it as a working node plugin — because it genuinely isn't one yet.

That's the honest, unglamorous state to end this chapter on: a driver that's discoverable, but not yet complete. NodeGetInfo is the very first method of the Node service, and it's next.

What you should have now

• internal/driver/endpoint.go with a tested SocketPathFromEndpoint helper, parsing unix: endpoint URIs the same way grpc-go's own resolver does
• cmd/localdir-csi/main.go reading its socket path and health-check directory from -endpoint and -data-dir flags, defaulting to the container's paths
• A Dockerfile building a small, distroless image via a multi-stage build
• deploy/node.yaml: a DaemonSet running your driver alongside node-driver-registrar, sharing a hostPath-backed socket directory
• deploy/csidriver.yaml: a CSIDriver object declaring attachRequired: false and volumeLifecycleModes: [Persistent]
• A driver pod running in your kind cluster, its identity confirmed by node-driver-registrar's logs, and node-driver-registrar itself crash-looping for a reason you've reproduced locally and fully understand
• Direct, hands-on proof — via csi.v1.Node/NodeGetInfo and kubectl get csinode — of exactly what kubelet still needs from you

Chapter 5 writes NodeGetInfo and NodeGetCapabilities: the first two methods of the Node service, test-first, the same way every method in Chapter 3 was written. Once NodeGetInfo exists, the exact same DaemonSet you just wrote, completely unchanged, will register successfully — you'll rerun the same kubectl get csinode command from this chapter and watch the answer change.