Chapter 12: Integration Testing

Every real-cluster proof in this book so far has the same shape: type a kubectl command, read the output, decide whether it matches what the prose predicted. Chapter 8 proved ControllerPublishVolume this way. Chapter 10 proved CreateSnapshot's idempotency this way — and also, on a live cluster, proved a real bug this way, the one where a genuine retry collided with a file os.CopyFS had already written. That bug was caught by a human happening to run the same command twice and noticing the second response wasn't 0 OK. A human doesn't always happen to do that. A test does it every time, on purpose, without needing anyone to remember to check.

This chapter doesn't add anything to the driver. It takes proofs this book has already done by hand — provisioning a volume, mounting it into a pod, snapshotting it, retrying that snapshot — and writes them down as real Go tests that run against a real kind cluster. The payoff is concrete: the exact CreateSnapshot regression from Chapter 10 gets a test that would have caught it the day it happened, instead of relying on whoever's at the keyboard to run the same call twice and read the diff.

client-go, not kubectl

Every object this chapter touches — creating a PersistentVolumeClaim, watching it bind, running a command inside a pod — has a real, first-class API for it in k8s.io/client-go, the same library Kubernetes' own controllers (and kubectl itself, underneath) are built on. Reaching for that library instead of shelling out to kubectl means every check in this chapter reads a real typed Go value (pvc.Status.Phase, not a string parsed out of kubectl get ... -o jsonpath=...), and a failure a step earlier — a client that never connected, a malformed object — surfaces as a real Go error at the exact call that caused it, not as an opaque non-zero exit code from a subprocess three layers removed from the test that's checking it.

One operation in this chapter has no client-go equivalent, and is named honestly rather than faked: checking the kind node's own disk directly, the same docker exec csi-dev-control-plane cat ... proof Chapters 8 and 10 ran by hand. That was never a Kubernetes operation even when kubectl did it — kubectl has no subcommand for it either, which is why those chapters reached for docker exec instead of kubectl exec at exactly that step. client-go talks to the Kubernetes API server; the API server has no concept of "the file sitting on a kind node's host filesystem outside any container." That one check stays os/exec-wrapped docker exec, unavoidably, for the same reason it always was.

Where these tests live, and how they're kept separate

Real-cluster tests need a real cluster — they can't run in the same go test ./... that Chapters 3–11 have relied on for everything else, because that command runs on every commit, on any machine, with no kind cluster guaranteed to exist. Go's answer to "these tests need something extra" is a build tag. test/integration/integration_test.go starts with one:

//go:build integration

package integration

go test ./... (no tags) never even compiles this file — the ordinary test suite Chapters 3–11 built stays exactly as fast and as cluster-independent as it's always been. Running these tests needs an explicit opt-in:

go test -tags=integration ./test/integration/... -v

Add a Makefile target so nobody has to remember that flag, the same way Chapter 9 added sanity-run for csi-sanity:

integration-test:
	@kubectl config current-context 2>/dev/null | grep -qx kind-csi-dev || \
		{ echo "current kubectl context is not kind-csi-dev — run 'make kind-up deploy' or 'kubectl config use-context kind-csi-dev' first" >&2; exit 1; }
	go test -tags=integration ./test/integration/... -v -timeout 5m

The context check above matters for a reason that has nothing to do with correctness of the tests themselves: requireCluster's own context-and-driver checks (below) are deliberately built to t.Skip, not fail, because a stray go test ./... with no cluster around should report "nothing to test here," not a false failure. But that same t.Skip means make integration-test — the thing a reader (or CI) actually runs on purpose, expecting real coverage — would otherwise print ok with every test skipped and zero real assertions run, indistinguishable from a genuinely green suite. The kubectl config current-context check above closes exactly that gap: once someone has explicitly asked for the integration suite to run, "there's no cluster to run it against" is a failure of that request, not a quiet no-op.

client-go is a new real dependency — nothing in internal/driver needed it, so it isn't in go.mod yet. Pin all three of client-go, apimachinery, and api to the same Kubernetes minor version — mixing versions across these three modules is unsupported and will surface as confusing type-mismatch errors, not a version-solver warning:

go get k8s.io/client-go@v0.34.0
go get k8s.io/apimachinery@v0.34.0
go get k8s.io/api@v0.34.0

k8s.io/api/core/v1 shows up directly in this chapter's imports (corev1.PersistentVolumeClaim, corev1.Pod, ...) even though nothing above named it explicitly — go mod tidy after writing the test files below would catch this too, but it's worth adding up front so go build doesn't stall on a missing module mid-chapter.

Setting up the cluster

Every test in this chapter needs the driver actually running on a real kind cluster first — none of them stand up a cluster themselves, the same way Chapters 8 and 10's grpcurl proofs never did either. This is the same sequence those chapters walked through by hand, run once before touching any test file. From the project root:

make kind-up
make deploy

Creating cluster "csi-dev" ...
 ✓ Ensuring node image (kindest/node:v1.36.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-csi-dev"
...
kubectl apply -f deploy/
serviceaccount/localdir-csi-controller created
clusterrole.rbac.authorization.k8s.io/localdir-csi-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/localdir-csi-provisioner created
clusterrole.rbac.authorization.k8s.io/localdir-csi-snapshotter created
clusterrolebinding.rbac.authorization.k8s.io/localdir-csi-snapshotter created
deployment.apps/localdir-csi-controller created
csidriver.storage.k8s.io/localdir.csi.example.com created
daemonset.apps/localdir-csi-node created
storageclass.storage.k8s.io/localdir-csi created
error: resource mapping not found for name: "localdir-csi-snapshot" ... no matches for kind "VolumeSnapshotClass"
ensure CRDs are installed first

That last error is expected, and safe to ignore for this chapter: deploy/volumesnapshotclass.yaml (Chapter 10) needs the external-snapshotter CRDs installed cluster-wide first, and none of this chapter's tests go through a VolumeSnapshot object — TestCreateSnapshot_IdempotentRetry calls CreateSnapshot directly by grpcurl, the same as Chapter 10's own grpcurl proof did before its VolumeSnapshot section. Everything else applied cleanly. Confirm both pods came up:

kubectl get pods

NAME                                       READY   STATUS    RESTARTS   AGE
localdir-csi-controller-7f56558f88-tqxhk   3/3     Running   0          15s
localdir-csi-node-9s6bz                    2/2     Running   0          15s

One more manual check worth running once, by hand, before trusting the suite to run it automatically: confirm grpcurl itself actually works against this pod, the same kubectl cp step Chapters 8 and 10 ran by hand. (TestCreateSnapshot_IdempotentRetry below doesn't depend on this having been run — it uploads its own copy of grpcurl every time, so a pod restart between now and then can't silently break it — but seeing a real response here first is worth the thirty seconds.)

CTRL_POD=$(kubectl get pods -l app=localdir-csi-controller -o jsonpath='{.items[0].metadata.name}')
kubectl cp ./grpcurl-linux $CTRL_POD:/tmp/grpcurl -c localdir-csi
kubectl exec $CTRL_POD -c localdir-csi -- chmod +x /tmp/grpcurl
kubectl exec $CTRL_POD -c localdir-csi -- /tmp/grpcurl -plaintext unix:///csi/csi.sock csi.v1.Identity/GetPluginInfo

{
  "name": "localdir.csi.example.com",
  "vendorVersion": "0.1.0"
}

A real response from a real running driver — the cluster is ready for every test below. kind clusters survive between go test runs (they aren't torn down by anything in this chapter), so this setup normally only needs doing once per session; make kind-down tears it back down when you're done for good.

Connecting: one client, shared by every test

Every test in this chapter needs the same thing first — a *kubernetes.Clientset built from whatever kubeconfig kubectl itself would use, so "the cluster this test talks to" is always the same cluster kubectl would have talked to by hand. client-go's clientcmd package resolves that the same way kubectl does: KUBECONFIG, then ~/.kube/config. Rather than hand-write that resolution in every test, one helper, in test/integration/cluster_test.go, that everything else in this chapter goes through — the same "one seam" shape Chapter 11's interceptors already taught.

"Whatever kubeconfig kubectl would use" is convenient, but it cuts both ways: this suite creates Pods and PVCs, and it deletes them in cleanup. If your current context happens to point at some other cluster — one you were debugging five minutes ago, say — "convenient" becomes "this test just created and deleted real objects somewhere it had no business touching." requireCluster doesn't just check that a cluster is reachable; it checks that the reachable cluster is this tutorial's disposable kind cluster, by name and by the presence of the driver these tests are actually exercising, before returning anything a test could use to mutate it:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"testing"
	"time"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
)

// wantContext is the only kubeconfig context these tests will run
// against. `kind create cluster --name csi-dev` (Chapter 2) always
// names its context kind-<cluster-name>, so this matches this book's
// own setup exactly — change it if you named your cluster something
// else.
const wantContext = "kind-csi-dev"

// wantCSIDriverName is the CSIDriver object Chapter 4 registers. Its
// presence is this package's proof that the reachable cluster is
// running this tutorial's driver, not just any kind cluster that
// happens to be named the same thing.
const wantCSIDriverName = "localdir.csi.example.com"

// requireCluster builds a Clientset from whatever kubeconfig kubectl
// itself would use (KUBECONFIG, then ~/.kube/config), and refuses to
// return one at all unless that kubeconfig's current context is
// wantContext and the cluster it points at is actually running
// wantCSIDriverName. Every test in this package calls this first —
// and skips, rather than fails confusingly three calls later, or
// silently mutates the wrong cluster — if either check fails.
func requireCluster(t *testing.T) (*kubernetes.Clientset, *rest.Config) {
	t.Helper()

	loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
	rawConfig, err := loadingRules.Load()
	if err != nil {
		t.Skipf("no usable kubeconfig (%v) — run against a real kind cluster with the driver deployed", err)
	}
	if rawConfig.CurrentContext != wantContext {
		t.Skipf("current kubeconfig context is %q, want %q — these tests create and delete real objects and must not run against the wrong cluster; run `kubectl config use-context %s` first",
			rawConfig.CurrentContext, wantContext, wantContext)
	}

	restConfig, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
		loadingRules, &clientcmd.ConfigOverrides{},
	).ClientConfig()
	if err != nil {
		t.Skipf("no usable kubeconfig (%v) — run against a real kind cluster with the driver deployed", err)
	}

	clientset, err := kubernetes.NewForConfig(restConfig)
	if err != nil {
		t.Fatalf("kubernetes.NewForConfig: %v", err)
	}

	if _, err := clientset.Discovery().ServerVersion(); err != nil {
		t.Skipf("cluster unreachable (%v) — run against a real kind cluster with the driver deployed", err)
	}

	if _, err := clientset.StorageV1().CSIDrivers().Get(context.Background(), wantCSIDriverName, metav1.GetOptions{}); err != nil {
		t.Skipf("context %q is reachable, but CSIDriver %q isn't registered on it (%v) — deploy this tutorial's driver first with `make deploy`",
			wantContext, wantCSIDriverName, err)
	}

	return clientset, restConfig
}

// uniqueName appends a nanosecond timestamp to base, so re-running this
// suite immediately after a previous run — or a previous run's cleanup
// only partially finishing, PVC finalizers included — never collides
// with an object the previous run created. Every test below calls this
// once per object it creates, rather than hardcoding a fixed name.
func uniqueName(base string) string {
	return fmt.Sprintf("%s-%d", base, time.Now().UnixNano())
}

Both new checks fail closed with t.Skip, not t.Fatal — the same "there's nothing wrong with your code, there's nothing here to test yet" signal the reachability check already used. A context-name mismatch or a missing CSIDriver isn't this suite's problem to fix; it's a sign the reader hasn't pointed kubectl at the right place yet, and the right response is to say so and stop, not to touch anything.

restConfig gets returned alongside clientset because building an authenticated connection into a running pod (needed further down, for the grpcurl and file-write proofs) reuses the exact same credentials clientcmd already resolved — no second round of "where's the kubeconfig" logic anywhere else in this package.

Waiting for a status to change

Nothing in the Kubernetes API blocks until a PersistentVolumeClaim says Bound — a Get call right after Create almost always still says Pending. k8s.io/apimachinery's own wait package is the real library idiom for exactly this shape of problem, the same package Kubernetes' own controllers use to poll for a condition. Add it to test/integration/cluster_test.go, alongside requireCluster — they're both small, cluster-connection-adjacent helpers every other test in this package calls. Add "context", "time", and "k8s.io/apimachinery/pkg/util/wait" to that file's existing import block (alongside "testing" and the three k8s.io/client-go imports requireCluster already needed), then add the function itself:

// waitFor polls condition every 500ms until it returns true, or fails
// the test once timeout has passed. condition returning a non-nil error
// stops the poll immediately and fails the test with that error, the
// same way a real setup problem (not just "not ready yet") should.
func waitFor(t *testing.T, timeout time.Duration, what string, condition wait.ConditionWithContextFunc) {
	t.Helper()
	if err := wait.PollUntilContextTimeout(context.Background(), 500*time.Millisecond, timeout, true, condition); err != nil {
		t.Fatalf("timed out waiting for %s: %v", what, err)
	}
}

condition's real type, wait.ConditionWithContextFunc, is func(ctx context.Context) (done bool, err error) — every call site below hands waitFor a closure matching that shape, reading whatever object it's waiting on fresh off the API server each time.

Running a command inside a pod

kubectl exec has a client-go equivalent too, though it's the one piece in this chapter with real ceremony behind it: exec runs over a separate, upgraded HTTP connection (SPDY), not an ordinary REST call, because the pod's stdout is a live stream, not a JSON response body. One helper hides that ceremony from every test that needs it, in test/integration/exec_test.go:

//go:build integration

package integration

import (
	"bytes"
	"context"
	"os"
	"testing"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/kubernetes/scheme"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/remotecommand"
)

// execInPod runs command inside container of pod podName, in namespace
// ns, and returns its combined stdout+stderr. It fails the test on a
// transport error (couldn't even reach the pod) or a non-zero exit
// inside the container, both surfaced as a real Go error from
// remotecommand rather than a parsed shell exit code. ctx is taken as
// an explicit parameter rather than reaching for t.Context() inside
// the function — more on why right after this listing.
func execInPod(ctx context.Context, t *testing.T, restConfig *rest.Config, clientset *kubernetes.Clientset, ns, podName, container string, command []string) string {
	t.Helper()

	req := clientset.CoreV1().RESTClient().Post().
		Resource("pods").
		Namespace(ns).
		Name(podName).
		SubResource("exec")
	req.VersionedParams(&corev1.PodExecOptions{
		Container: container,
		Command:   command,
		Stdout:    true,
		Stderr:    true,
	}, scheme.ParameterCodec)

	executor, err := remotecommand.NewSPDYExecutor(restConfig, "POST", req.URL())
	if err != nil {
		t.Fatalf("building exec request for pod %q: %v", podName, err)
	}

	var stdout, stderr bytes.Buffer
	err = executor.StreamWithContext(ctx, remotecommand.StreamOptions{
		Stdout: &stdout,
		Stderr: &stderr,
	})
	combined := stdout.String() + stderr.String()
	if err != nil {
		t.Fatalf("exec %v in pod %q: %v\n%s", command, podName, err, combined)
	}
	return combined
}

// uploadGrpcurl copies the local grpcurl-linux binary into podName's
// container at /tmp/grpcurl, over the same exec stream execInPod uses
// — just with Stdin attached instead of Stdout/Stderr captured, the
// same way `kubectl cp` itself streams a file through exec underneath.
// Any test that needs grpcurl calls this first, every run, rather than
// assuming a copy from a previous session survived — a controller pod
// restart (a new rollout, a node reboot) wipes /tmp along with it.
func uploadGrpcurl(ctx context.Context, t *testing.T, restConfig *rest.Config, clientset *kubernetes.Clientset, ns, podName, container string) {
	t.Helper()

	f, err := os.Open("grpcurl-linux")
	if err != nil {
		t.Fatalf("opening local grpcurl-linux binary (build it per Chapter 8's grpcurl setup): %v", err)
	}
	defer f.Close()

	req := clientset.CoreV1().RESTClient().Post().
		Resource("pods").
		Namespace(ns).
		Name(podName).
		SubResource("exec")
	req.VersionedParams(&corev1.PodExecOptions{
		Container: container,
		Command:   []string{"sh", "-c", "cat > /tmp/grpcurl && chmod +x /tmp/grpcurl"},
		Stdin:     true,
		Stdout:    true,
		Stderr:    true,
	}, scheme.ParameterCodec)

	executor, err := remotecommand.NewSPDYExecutor(restConfig, "POST", req.URL())
	if err != nil {
		t.Fatalf("building exec request for pod %q: %v", podName, err)
	}

	var stdout, stderr bytes.Buffer
	if err := executor.StreamWithContext(ctx, remotecommand.StreamOptions{
		Stdin:  f,
		Stdout: &stdout,
		Stderr: &stderr,
	}); err != nil {
		t.Fatalf("uploading grpcurl to pod %q: %v\n%s", podName, err, stdout.String()+stderr.String())
	}
}

os.Open("grpcurl-linux") resolves relative to go test's own working directory, which is the package directory — so this expects the binary at test/integration/grpcurl-linux, not the project root. Copy or symlink it there once; go test doesn't change directories between runs, so this, unlike the pod-side copy, only needs doing once per checkout.

scheme.ParameterCodec (from k8s.io/client-go/kubernetes/scheme) is what turns the typed PodExecOptions struct — container name, the command to run, which streams to attach — into the URL query parameters the API server's exec subresource actually expects; it's the same encoding kubectl exec relies on internally, just reached directly instead of through a subprocess.

Two things about this listing are corrections, not stylistic choices, and both were found the same way every real bug in this book has been found: by running the thing against a real cluster and watching it misbehave.

The first: ctx is a parameter here, not t.Context() called directly inside the function, because execInPod needs to work correctly from inside a t.Cleanup closure too — Chapter's own TestCreateSnapshot_IdempotentRetry below calls grpcurl (which wraps execInPod) from both the test body and its cleanup, to delete what it created. t.Context() is documented to be "canceled just before Cleanup-registered functions are called" — always, every run — so a version of execInPod that called t.Context() internally would work fine everywhere except cleanup, where every call would fail with operation was canceled. Confirmed live: an earlier version of this exact helper did exactly that, and its cleanup's DeleteSnapshot/DeleteVolume calls failed with precisely that error the first time this chapter's snapshot test ran for real. Passing ctx in means callers decide: t.Context() from the test body, context.Background() from inside t.Cleanup — the same fix TestCreateVolume_ViaPVC's own cleanup already needed, below.

The second is subtler, and was the real cause of a much stranger symptom: an early version of this helper pointed both Stdout and Stderr at the same bytes.Buffer. remotecommand streams stdout and stderr back on separate channels, read by separate goroutines — and bytes.Buffer is not safe for concurrent writes. Running TestCreateSnapshot_IdempotentRetry for real against this book's kind cluster reproduced this exactly: the two CreateSnapshot calls occasionally came back with one of them holding a real, correct response and the other holding an empty string, with no error from either call — a plain data race, not a driver bug, not a network flake, confirmed by re-running the same two grpcurl calls by hand with kubectl exec (no client-go involved) and seeing identical, correct output both times. go test -race on the fixed version — two separate buffers, concatenated only after StreamWithContext returns, once both goroutines are known to be done writing — ran clean across several real, repeated runs against the cluster. The lesson generalizes past this one helper: any time two ends of a concurrent stream get pointed at the same non-thread-safe destination, "sometimes empty, never errors" is exactly the shape of bug to expect.

Two object builders, shared instead of duplicated

Two tests below both need a PersistentVolumeClaim shaped exactly like Chapter 8's demo-pvc.yaml, and one needs a Pod shaped exactly like demo-pod.yaml. Writing that struct literal twice would be exactly the kind of repetition Chapter 6 already taught this book to notice and factor out — so both live once, in test/integration/objects_test.go:

//go:build integration

package integration

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/api/resource"
)

// demoPVC builds a PersistentVolumeClaim matching Chapter 8's
// demo-pvc.yaml exactly — same storage class, access mode, and size.
func demoPVC(name string) *corev1.PersistentVolumeClaim {
	storageClassName := "localdir-csi"
	return &corev1.PersistentVolumeClaim{
		ObjectMeta: metav1.ObjectMeta{Name: name},
		Spec: corev1.PersistentVolumeClaimSpec{
			StorageClassName: &storageClassName,
			AccessModes:      []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
			Resources: corev1.VolumeResourceRequirements{
				Requests: corev1.ResourceList{
					corev1.ResourceStorage: resource.MustParse("1Gi"),
				},
			},
		},
	}
}

// demoPod builds a Pod matching Chapter 8's demo-pod.yaml exactly —
// same image, same mount path, referencing whichever PVC name is passed
// in so TestNodePublishVolume_ViaPod and any future test can reuse it
// without hardcoding "demo-claim" a second time.
func demoPod(name, pvcName string) *corev1.Pod {
	return &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{Name: name},
		Spec: corev1.PodSpec{
			Containers: []corev1.Container{{
				Name:    "app",
				Image:   "busybox",
				Command: []string{"sleep", "3600"},
				VolumeMounts: []corev1.VolumeMount{{
					Name:      "data",
					MountPath: "/data",
				}},
			}},
			Volumes: []corev1.Volume{{
				Name: "data",
				VolumeSource: corev1.VolumeSource{
					PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{
						ClaimName: pvcName,
					},
				},
			}},
		},
	}
}

Provisioning: CreateVolume, triggered by a PersistentVolumeClaim

Chapter 8's proof: create the claim, watch it go Pending → Bound, confirm a PersistentVolume now exists. As a test, in test/integration/provisioning_test.go:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func TestCreateVolume_ViaPVC(t *testing.T) {
	clientset, _ := requireCluster(t)
	ctx := t.Context()
	pvcs := clientset.CoreV1().PersistentVolumeClaims("default")
	claimName := uniqueName("it-demo-claim")

	if _, err := pvcs.Create(ctx, demoPVC(claimName), metav1.CreateOptions{}); err != nil {
		t.Fatalf("creating PVC: %v", err)
	}
	t.Cleanup(func() {
		bg := context.Background()
		if err := pvcs.Delete(bg, claimName, metav1.DeleteOptions{}); err != nil {
			t.Errorf("deleting PVC %q: %v", claimName, err)
			return
		}
		waitFor(t, 30*time.Second, fmt.Sprintf("PVC %s to be gone", claimName), func(ctx context.Context) (bool, error) {
			_, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
			if apierrors.IsNotFound(err) {
				return true, nil
			}
			return false, err
		})
	})

	var bound *corev1.PersistentVolumeClaim
	waitFor(t, 30*time.Second, fmt.Sprintf("PVC %s to bind", claimName), func(ctx context.Context) (bool, error) {
		pvc, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
		if err != nil {
			return false, err
		}
		if pvc.Status.Phase == corev1.ClaimBound {
			bound = pvc
			return true, nil
		}
		return false, nil
	})

	if bound.Spec.VolumeName == "" {
		t.Fatal("PVC bound, but Spec.VolumeName is empty")
	}

	pv, err := clientset.CoreV1().PersistentVolumes().Get(ctx, bound.Spec.VolumeName, metav1.GetOptions{})
	if err != nil {
		t.Fatalf("getting PersistentVolume %q: %v", bound.Spec.VolumeName, err)
	}
	if pv.Status.Phase != corev1.VolumeBound {
		t.Errorf("PersistentVolume %q status = %q, want %q", pv.Name, pv.Status.Phase, corev1.VolumeBound)
	}
}

t.Cleanup runs even if an assertion above it fails — the same reason Chapters 6–10's own newTestDriverInTempDir helper uses it — so a failed run doesn't leave the claim (and the real directory CreateVolume made for it) behind for the next run to trip over. uniqueName means a leftover claim from an earlier failed run can never collide with this run's name either way, but the cleanup still checks the Delete call's own error with t.Errorf — a silently swallowed error here would hide a real problem (RBAC, a finalizer stuck forever) behind a green test — and then waitFors the claim actually disappearing rather than firing the delete and assuming it worked; PersistentVolumeClaim carries the pvc-protection finalizer, so Get can still return the object, Terminating, for a moment after Delete returns.

The cleanup deliberately uses context.Background(), not ctx, and this one is worth being exact about, because the bug it avoids is real and easy to reintroduce by reaching for the context already in scope: Go's own testing package documents t.Context() as "canceled just before Cleanup-registered functions are called" — not "eventually," not "maybe," but always, on every run, pass or fail. A Delete call made with ctx inside t.Cleanup is handed an already-cancelled context before it even reaches the API server, and client-go returns immediately with a context-cancellation error — which an unchecked cleanup wouldn't even notice, so the delete would silently never happen. Confirmed live, against a real cluster, while drafting this chapter: a first version using ctx in t.Cleanup left the claim sitting there, Bound, after a passing test run. context.Background() in cleanup, always — the same lesson TestNodePublishVolume_ViaPod below already applies.

Run it now, against the cluster from the setup section above:

go test -tags=integration ./test/integration/... -run TestCreateVolume_ViaPVC -v

=== RUN   TestCreateVolume_ViaPVC
--- PASS: TestCreateVolume_ViaPVC (0.52s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	0.530s

Real 0 OK, against a real cluster — and worth checking the thing the test itself can't check about its own cleanup:

kubectl get pvc

No resources found in default namespace.

Gone, confirming the context.Background() fix actually took — exactly the same proof-that-cleanup-really-cleaned-up instinct every docker exec ... ls check since Chapter 8 has used, just aimed at a test's own teardown this time instead of the driver's.

Mounting: NodePublishVolume, triggered by a Pod

Chapter 8's other half: a pod that references the claim gets kubelet to call NodePublishVolume with no grpcurl involved, and a file written inside the pod is provably the same file sitting in the node's own directory. In test/integration/mount_test.go:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"os/exec"
	"strings"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

const kindNodeContainer = "csi-dev-control-plane"

func TestNodePublishVolume_ViaPod(t *testing.T) {
	clientset, restConfig := requireCluster(t)
	ctx := t.Context()
	pvcs := clientset.CoreV1().PersistentVolumeClaims("default")
	pods := clientset.CoreV1().Pods("default")
	claimName := uniqueName("it-mount-claim")
	podName := uniqueName("it-mount-pod")

	if _, err := pvcs.Create(ctx, demoPVC(claimName), metav1.CreateOptions{}); err != nil {
		t.Fatalf("creating PVC: %v", err)
	}
	if _, err := pods.Create(ctx, demoPod(podName, claimName), metav1.CreateOptions{}); err != nil {
		t.Fatalf("creating Pod: %v", err)
	}
	t.Cleanup(func() {
		bg := context.Background()
		if err := pods.Delete(bg, podName, metav1.DeleteOptions{}); err != nil {
			t.Errorf("deleting pod %q: %v", podName, err)
		} else {
			waitFor(t, 30*time.Second, fmt.Sprintf("pod %s to be gone", podName), func(ctx context.Context) (bool, error) {
				_, err := pods.Get(ctx, podName, metav1.GetOptions{})
				if apierrors.IsNotFound(err) {
					return true, nil
				}
				return false, err
			})
		}
		if err := pvcs.Delete(bg, claimName, metav1.DeleteOptions{}); err != nil {
			t.Errorf("deleting PVC %q: %v", claimName, err)
			return
		}
		waitFor(t, 30*time.Second, fmt.Sprintf("PVC %s to be gone", claimName), func(ctx context.Context) (bool, error) {
			_, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
			if apierrors.IsNotFound(err) {
				return true, nil
			}
			return false, err
		})
	})

	waitFor(t, 60*time.Second, fmt.Sprintf("pod %s to become Running", podName), func(ctx context.Context) (bool, error) {
		pod, err := pods.Get(ctx, podName, metav1.GetOptions{})
		if err != nil {
			return false, err
		}
		return pod.Status.Phase == corev1.PodRunning, nil
	})

	execInPod(ctx, t, restConfig, clientset, "default", podName, "app",
		[]string{"sh", "-c", "echo integration-test > /data/hello.txt"})

	fromPod := execInPod(ctx, t, restConfig, clientset, "default", podName, "app",
		[]string{"cat", "/data/hello.txt"})
	if strings.TrimSpace(fromPod) != "integration-test" {
		t.Fatalf("cat inside pod = %q, want %q", fromPod, "integration-test")
	}

	pvc, err := pvcs.Get(ctx, claimName, metav1.GetOptions{})
	if err != nil {
		t.Fatalf("getting PVC: %v", err)
	}
	nodePath := "/var/lib/localdir-csi/data/" + pvc.Spec.VolumeName + "/hello.txt"
	out, err := exec.Command("docker", "exec", kindNodeContainer, "cat", nodePath).Output()
	if err != nil {
		t.Fatalf("docker exec cat %s: %v", nodePath, err)
	}
	if fromNode := strings.TrimSpace(string(out)); fromNode != "integration-test" {
		t.Errorf("cat on node disk = %q, want %q", fromNode, "integration-test")
	}
}

Same proof-from-both-sides technique every hand-run chapter since Chapter 6 has used, just run by go test instead of a person: the pod sees the file it wrote, and the node's own disk — reached through a completely separate path, docker exec into the kind container rather than a pod exec stream — sees the exact same content. If the driver only pretended to bind-mount (say, it silently wrote into a scratch directory pods never actually see), this test would still pass its first assertion and fail its second — which is exactly why the second one exists.

strings.TrimSpace on both sides of that comparison is not decoration — cat's output over a real exec stream carries the trailing newline echo wrote, same as cat always would from a real terminal. Compare fromPod (or fromNode) without trimming it first and this test fails on a difference that was never actually a difference in the data, only in whitespace neither side cares about — a real, easy mistake, not a hypothetical one.

Run it now:

go test -tags=integration ./test/integration/... -run TestNodePublishVolume_ViaPod -v

=== RUN   TestNodePublishVolume_ViaPod
--- PASS: TestNodePublishVolume_ViaPod (2.14s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	2.143s

One more real wrinkle worth naming, hit while confirming this test back-to-back with itself in a tight loop: a PersistentVolumeClaim doesn't finish deleting the instant Delete returns — a pvc-protection finalizer holds it in Terminating until the underlying volume is actually gone, which for this driver means DeleteVolume has to run first. Reusing a fixed claim name across runs, with a cleanup that fires Delete and moves on without checking whether the object is actually gone, can transiently hit object is being deleted: ... already exists on the next run's Create if the previous run's Terminating claim hasn't finished clearing yet. That's exactly why both this test's and TestCreateVolume_ViaPVC's cleanups call waitFor on the delete, not just the create — a cleanup that returns before the object is actually gone isn't really "cleaned up," it's "asked to clean up and hoped." Combined with uniqueName, a Terminating leftover from a genuinely interrupted run (a killed test process, say) still can't collide with anything a later run creates, even if it clears too slowly to matter to that later run at all.

The regression test: CreateSnapshot, called twice

This is the test this chapter exists to write. Chapter 10's live incident, replayed exactly, as TestCreateSnapshot_IdempotentRetry in test/integration/snapshot_test.go. Calling the driver directly by grpcurl — run inside the controller pod via the same execInPod helper above, rather than through a VolumeSnapshot object — keeps this test aimed precisely at the bug — a VolumeSnapshotContent retry loop would eventually paper over a slow second call in ways that would make a real regression harder to catch, not easier. Finding the controller pod itself is one more client-go list call, by the same label kubectl logs -l app=localdir-csi-controller has used by hand since Chapter 7:

//go:build integration

package integration

import (
	"context"
	"testing"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
)

func controllerPodName(t *testing.T, clientset *kubernetes.Clientset) string {
	t.Helper()
	pods, err := clientset.CoreV1().Pods("default").List(t.Context(), metav1.ListOptions{
		LabelSelector: "app=localdir-csi-controller",
	})
	if err != nil {
		t.Fatalf("listing controller pods: %v", err)
	}
	if len(pods.Items) == 0 {
		t.Fatal("no localdir-csi-controller pod found — is the driver deployed?")
	}
	return pods.Items[0].Name
}

With that in place, the regression test itself:

//go:build integration

package integration

import (
	"context"
	"fmt"
	"testing"
)

func TestCreateSnapshot_IdempotentRetry(t *testing.T) {
	clientset, restConfig := requireCluster(t)
	ctx := t.Context()
	podName := controllerPodName(t, clientset)
	uploadGrpcurl(ctx, t, restConfig, clientset, "default", podName, "localdir-csi")

	volumeID := uniqueName("it-snap-source")
	snapshotID := uniqueName("it-snap-demo")
	grpcurl := func(ctx context.Context, data, method string) string {
		return execInPod(ctx, t, restConfig, clientset, "default", podName, "localdir-csi", []string{
			"/tmp/grpcurl", "-plaintext", "-d", data,
			"unix:///csi/csi.sock", "csi.v1.Controller/" + method,
		})
	}

	t.Cleanup(func() {
		bg := context.Background()
		grpcurl(bg, fmt.Sprintf(`{"snapshot_id": %q}`, snapshotID), "DeleteSnapshot")
		grpcurl(bg, fmt.Sprintf(`{"volume_id": %q}`, volumeID), "DeleteVolume")
	})

	grpcurl(ctx, fmt.Sprintf(`{
		"name": %q,
		"capacity_range": {"required_bytes": 1048576},
		"volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
	}`, volumeID), "CreateVolume")

	execInPod(ctx, t, restConfig, clientset, "default", podName, "localdir-csi",
		[]string{"sh", "-c", fmt.Sprintf("echo real data > /data/%s/hello.txt", volumeID)})

	first := grpcurl(ctx, fmt.Sprintf(`{
		"source_volume_id": %q,
		"name": %q
	}`, volumeID, snapshotID), "CreateSnapshot")

	second := grpcurl(ctx, fmt.Sprintf(`{
		"source_volume_id": %q,
		"name": %q
	}`, volumeID, snapshotID), "CreateSnapshot")

	if first != second {
		t.Errorf("CreateSnapshot retry returned a different response:\nfirst:  %s\nsecond: %s", first, second)
	}
}

uploadGrpcurl replaces the manual kubectl cp step from the cluster setup section above — that step is still worth knowing (it's exactly what uploadGrpcurl automates), but this test no longer depends on it having been run, or on its result having survived whatever's happened to the controller pod since.

grpcurl itself takes ctx as a parameter, for the exact same reason execInPod does — this closure is called from both the test body (where ctx is t.Context()) and from t.Cleanup (where it has to be context.Background() instead). A closure that captured a single fixed ctx from its enclosing scope, rather than taking one as a parameter, would silently pick whichever context was in scope when the closure was defined — the test body's, canceled by the time cleanup runs — the same bug in a slightly more hidden shape.

first != second is the entire regression check, and it's worth being precise about why that one comparison is enough. grpcurl's JSON output includes creationTime, down to the nanosecond. Chapter 10's real incident didn't fail this way — the pre-fix bug returned a real gRPC error (Internal: ... file exists) on the second call, which execInPod's t.Fatalf on a non-zero exit would already catch, before the string comparison ever runs. The string comparison catches the other failure mode a fix like this can quietly introduce: a second call that "succeeds" but doesn't actually short-circuit — say, one that recomputes sizeBytes from a directory listing instead of returning the recorded metadata. That would return 0 OK both times, pass a check that only asserted "no error," and still be wrong. Comparing the full response, creationTime included, is what makes this test check the same snapshot, not just no error — the exact distinction Chapter 10's real live confirmation drew by hand when it checked that the retry's creationTime matched the original down to the nanosecond.

Run it now, the same way as the two tests above:

go test -tags=integration ./test/integration/... -run TestCreateSnapshot_IdempotentRetry -v

=== RUN   TestCreateSnapshot_IdempotentRetry
--- PASS: TestCreateSnapshot_IdempotentRetry (0.24s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	0.252s

Real 0 OK twice, real identical creationTime, confirmed by go test, on the exact bug this chapter set out to catch — worth re-running a few times in a row rather than trusting one green result, precisely because this section's own two fixes above were both found by a previously passing-looking run turning out to be unreliable. Five repeated runs, with go test -race (to keep the buffer fix honest) and a manual cleanup between each:

go test -tags=integration -race ./test/integration/... -run TestCreateSnapshot_IdempotentRetry -v -count=1

--- PASS: TestCreateSnapshot_IdempotentRetry (0.31s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.31s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.31s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.30s)
--- PASS: TestCreateSnapshot_IdempotentRetry (0.29s)

Five for five, no race detected — the shared-buffer bug is really gone, not just quiet this particular run.

Running the whole suite together

Once all three tests exist, make integration-test runs them in one pass — the same Makefile target added earlier in this chapter:

make integration-test

go test -tags=integration ./test/integration/... -v -timeout 5m
=== RUN   TestNodePublishVolume_ViaPod
--- PASS: TestNodePublishVolume_ViaPod (2.62s)
=== RUN   TestCreateVolume_ViaPVC
--- PASS: TestCreateVolume_ViaPVC (0.51s)
=== RUN   TestCreateSnapshot_IdempotentRetry
--- PASS: TestCreateSnapshot_IdempotentRetry (0.24s)
PASS
ok  	github.com/yourname/localdir-csi/test/integration	3.384s

go test runs a package's tests in the order they appear in each file, sorted by filename — TestNodePublishVolume_ViaPod (mount_test.go) before TestCreateVolume_ViaPVC (provisioning_test.go), alphabetically. All three real, all three green, in one pass, against one real kind cluster.

Three tests, each skipping cleanly if run directly with go test and no matching cluster around, but failing loudly if run via make integration-test with the wrong context or no cluster at all: TestCreateVolume_ViaPVC, TestNodePublishVolume_ViaPod, and TestCreateSnapshot_IdempotentRetry — the last one doing, on every run, exactly what a human happened to do once by accident back in Chapter 10.

What you should have now

• test/integration/, a //go:build integration-tagged package, invisible to the ordinary go test ./... every earlier chapter's suite runs
• requireCluster, waitFor, and execInPod: three small helpers everything else in the package builds on, the same "one seam" shape as Chapter 11's interceptors — plus demoPVC/demoPod, two object builders shared by every test that needs one, instead of the same struct literal written out twice
• k8s.io/client-go and k8s.io/apimachinery as new real dependencies — the first time this driver's own go.mod has needed a Kubernetes client library, since every earlier chapter's Kubernetes interaction was either a sidecar's job or typed by hand
• TestCreateVolume_ViaPVC: Chapter 8's PersistentVolumeClaim provisioning proof, automated
• TestNodePublishVolume_ViaPod: Chapter 8's pod-mount proof, automated, keeping the same proof-from-both-sides technique — a client-go exec stream inside the pod, docker exec on the node's own disk (the one operation with no client-go equivalent, named honestly rather than faked)
• TestCreateSnapshot_IdempotentRetry: a real regression test for the exact bug Chapter 10 hit live, comparing the full response of two identical CreateSnapshot calls rather than just checking for an error, so a future regression that "succeeds twice, quietly wrong" gets caught too, not just one that errors outright
• An integration-test Makefile target, the same convenience Chapter 9 added for csi-sanity
• Two real bugs in the test helpers themselves, found only by actually running this chapter's tests against a real cluster repeatedly, not by reading the code: execInPod (and, following the same pattern, TestCreateSnapshot_IdempotentRetry's grpcurl closure) taking ctx as an explicit parameter instead of reaching for t.Context() internally, because t.Context() is canceled before any t.Cleanup runs; and execInPod using two separate buffers for stdout and stderr instead of one shared bytes.Buffer, because two goroutines writing to the same non-thread-safe buffer is a real data race, confirmed with go test -race clean across five repeated runs once fixed. Neither bug was hypothetical — both reproduced live, against this book's own kind cluster, before being fixed
• requireCluster rejecting any kubeconfig context other than kind-csi-dev, and any cluster missing this tutorial's own CSIDriver, before any test can create or mutate anything — the suite no longer trusts "some cluster is reachable" to mean "this is the disposable one"
• uniqueName, used for every PVC, Pod, and CSI volume/snapshot name this chapter creates, so re-running the suite immediately — or after a previous run's cleanup only partially finished — can't collide with a leftover from before
• Every cleanup now checks its own Delete call's error with t.Errorf and waitFors the object actually disappearing, instead of firing a delete and assuming it worked — pvc-protection's finalizer means "deleted" and "gone" aren't the same moment
• uploadGrpcurl, replacing a manually-copied /tmp/grpcurl that a controller pod restart would silently wipe — the snapshot regression test now provisions its own copy of the binary it depends on, every run
• make integration-test failing outright, with a clear message, when the current kubectl context isn't kind-csi-dev — the tests themselves still skip cleanly for an ad hoc go test run with no cluster around, but the one command a reader (or CI) actually invokes on purpose no longer reports ok having silently run zero real assertions

Two honest gaps remain. These tests aren't wired into any CI — they need a real kind cluster with the driver already deployed, and this book has never set up a CI pipeline to provide one; running them today means running them by hand, the same way every other real-cluster proof in this book has been run, just with go test instead of a person reading kubectl output line by line. And coverage here is deliberately narrow — three tests, not a full conformance suite (that's what Chapter 9's csi-sanity already is) — chosen specifically to cover the two flows this book has proven by hand the most (provisioning and mounting) plus the one real bug this book has actually hit live. A fuller integration suite covering every RPC this driver has is possible with exactly these same helpers; it just isn't written yet.