Chapter 10: Snapshots

A whiteboard sits in a conference room, covered in a diagram someone's been erasing and redrawing all week. Photocopy it right now, and the copy is fixed forever — exactly what the board looked like at 2:14pm on Tuesday. Erase half the board on Wednesday. Draw something completely different. The photocopy doesn't know. It never will. It isn't connected to the whiteboard anymore — it never was, really. It's paper, holding a pattern of light and dark that used to match a whiteboard, at one specific moment.

That's what CreateSnapshot does to a volume. A photocopy, not a link.

What a snapshot actually is, and what it's for

A PersistentVolume, once bound, just keeps existing — read from, written to, mounted into whatever pod needs it next, with no built-in notion of "before" or "after." A snapshot cuts across that. It freezes one moment of a volume's contents into a new, independent object, one that keeps existing even after the original volume changes, or gets deleted entirely.

Two real reasons that matters, both worth naming plainly. Backup: take a snapshot before a risky migration, and a bad migration has an exact undo point, not "whatever the last full backup happened to catch." Cloning: hand a snapshot to CreateVolume as a starting point (the volume_content_source field Chapter 7 already listed among CreateVolumeRequest's fields, and left untouched) and a brand-new volume starts out with another volume's exact contents, rather than empty — a fast way to spin up a database replica or a test environment seeded with real-looking data.

CSI's answer to both is CreateSnapshot and DeleteSnapshot, the two RPCs this chapter implements, plus a whole extra sidecar and two more Kubernetes CRDs to trigger them automatically — more new machinery in one chapter than Chapter 8 needed, because unlike attaching, snapshots aren't a "this driver happens not to need it" gap. localdir-csi's volumes are just directories, and directories can absolutely be copied. This is real, useful work for a toy driver to do.

Driver needs nothing new

Say it the way Chapter 7 said it about NewDriver: nothing here changes. Driver has embedded csi.UnimplementedControllerServer since Chapter 7, and ControllerServer's interface has always included CreateSnapshot, DeleteSnapshot, and ListSnapshots — Chapter 9's ValidateVolumeCapabilities work already made Driver satisfy the full interface, stubs and all. CreateSnapshot needs exactly one thing Driver already has: dataDir, the same field CreateVolume has relied on since Chapter 7. No new constructor parameter, no new embedded interface, no second red/green cycle just to catch up.

What CreateSnapshotRequest asks for, and what a Snapshot has to remember

CreateSnapshotRequest carries source_volume_id and name as its two REQUIRED fields — a snapshot always names both the volume it's copying and the name a CO wants to identify it by, the exact same name/capabilities shape CreateVolumeRequest asked for back in Chapter 7. secrets and parameters exist too, unused here, same as CreateVolumeRequest's own unused fields.

The response side is where snapshots ask for more than a volume did. A Snapshot carries snapshot_id, source_volume_id, size_bytes, ready_to_use, and creation_time — a real timestamp, not just an identifier. CreateVolume never had to remember anything like that about a volume; a directory's existence was the entire fact worth tracking. A snapshot needs history: which volume it came from, and when the copy was taken, permanently — not just for one response, but for every later CreateSnapshot call asking about the same name.

That's the same shape of problem Chapter 7 named honestly — "a bare directory on disk doesn't remember what capacity or parameters it was created with, only that it exists" — and Chapter 9's real csi-sanity run went on to confirm for real. CreateSnapshot doesn't get to leave that gap open. Answering source_volume_id and creation_time correctly on a second call requires persisting something beyond a directory's mere presence — so this chapter reaches for the same sidecar-metadata-file technique Chapter 6 introduced for NodePublishVolume, and Chapter 9 just gave CreateVolume, applying it here from day one instead of discovering the need for it the hard way.

CreateSnapshot, test-first: validation

Same rhythm as every RPC so far. Add this to internal/driver/controller_test.go:

func TestCreateSnapshot_Validation(t *testing.T) {
	cases := []validationCase[*csi.CreateSnapshotRequest]{
		{
			name: "missing source volume id",
			req: &csi.CreateSnapshotRequest{
				Name: "snap-1",
			},
		},
		{
			name: "missing name",
			req: &csi.CreateSnapshotRequest{
				SourceVolumeId: "vol-1",
			},
		},
	}
	runValidation(t, cases, newTestDriverInTempDir,
		func(t *testing.T, d *Driver, req *csi.CreateSnapshotRequest) error {
			_, err := d.CreateSnapshot(t.Context(), req)
			return err
		},
	)
}

Same validationCase[T]/runValidation pair every validation table since Chapter 6 has used.

go test ./internal/driver/...

--- FAIL: TestCreateSnapshot_Validation (0.00s)
    --- FAIL: TestCreateSnapshot_Validation/missing_source_volume_id (0.00s)
        testing_test.go:92: code = Unimplemented, want InvalidArgument (req=&{SourceVolumeId: Name:snap-1 Secrets:map[] Parameters:map[]})
    --- FAIL: TestCreateSnapshot_Validation/missing_name (0.00s)
        testing_test.go:92: code = Unimplemented, want InvalidArgument (req=&{SourceVolumeId:vol-1 Name: Secrets:map[] Parameters:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.006s
FAIL

Unimplemented — the promoted stub talking, same as every RPC's first validation test since Chapter 6: red for the right reason, CreateSnapshot doesn't exist yet. And, as in every validation table since Chapter 6, both failures land on testing_test.go:92 — one line further down than Chapter 9's testing_test.go:85, since mountCapabilityWithMode now sits between requireStatusCode and the rest of the file — not on this test's own call to runValidation.

Add the method to internal/driver/controller.go:

func (d *Driver) CreateSnapshot(
	ctx context.Context,
	req *csi.CreateSnapshotRequest,
) (*csi.CreateSnapshotResponse, error) {
	if req.GetSourceVolumeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "source_volume_id is required")
	}
	if req.GetName() == "" {
		return nil, status.Error(codes.InvalidArgument, "name is required")
	}

	return &csi.CreateSnapshotResponse{Snapshot: &csi.Snapshot{}}, nil
}

=== RUN   TestCreateSnapshot_Validation
--- PASS: TestCreateSnapshot_Validation (0.00s)

Green. That stub return is deliberately not nil, nil — the exact same reason Chapter 7 gave CreateVolume's own stub a real, empty *Volume instead of a missing one: CreateSnapshotResponse's snapshot field is REQUIRED, and a nil one is a broken response waiting to panic the next test that reads a field off it.

CreateSnapshot, test-first: the source volume has to exist

CreateSnapshot naming a volume that was never created — a typo, a stale reference, a volume already deleted — is a real case, the same shape ControllerPublishVolume already had to handle in Chapter 8. Add this to internal/driver/controller_test.go:

func TestCreateSnapshot_SourceVolumeNotFound(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.CreateSnapshotRequest{
		SourceVolumeId: "does-not-exist",
		Name:           "snap-1",
	}

	_, err := d.CreateSnapshot(t.Context(), req)
	requireStatusCode(t, err, codes.NotFound, req)
}

--- FAIL: TestCreateSnapshot_SourceVolumeNotFound (0.00s)
    controller_test.go:376: code = OK, want NotFound (req=&{SourceVolumeId:does-not-exist Name:snap-1 Secrets:map[] Parameters:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.007s
FAIL

Red — nothing checks the source volume exists yet. Same filepath.Join(d.dataDir, ...)-and-os.Stat pattern this book has reused since NodePublishVolume. Add it to CreateSnapshot, in internal/driver/controller.go, right after the validation checks:

	sourcePath := filepath.Join(d.dataDir, req.GetSourceVolumeId())
	if _, err := os.Stat(sourcePath); os.IsNotExist(err) {
		return nil, status.Errorf(codes.NotFound, "source volume %q not found", req.GetSourceVolumeId())
	} else if err != nil {
		return nil, status.Errorf(codes.Internal, "checking source volume %q: %v", req.GetSourceVolumeId(), err)
	}

=== RUN   TestCreateSnapshot_SourceVolumeNotFound
--- PASS: TestCreateSnapshot_SourceVolumeNotFound (0.00s)

Green.

CreateSnapshot, test-first: actually creating one

The real work. A valid request against a real volume should copy that volume's files somewhere new, and hand back a Snapshot describing what it copied. Add this to internal/driver/controller_test.go:

func TestCreateSnapshot_CreatesTheSnapshot(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	if err := os.WriteFile(filepath.Join(dataDir, "vol-1", "hello.txt"), []byte("hello from vol-1"), 0640); err != nil {
		t.Fatalf("writing fake volume data: %v", err)
	}
	d := newTestDriver(t, dataDir, nil)

	req := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-1",
		Name:           "snap-1",
	}

	resp, err := d.CreateSnapshot(t.Context(), req)
	if err != nil {
		t.Fatalf("CreateSnapshot() returned an error: %v", err)
	}

	snap := resp.GetSnapshot()
	if snap.GetSnapshotId() != "snap-1" {
		t.Errorf("SnapshotId = %q, want %q", snap.GetSnapshotId(), "snap-1")
	}
	if snap.GetSourceVolumeId() != "vol-1" {
		t.Errorf("SourceVolumeId = %q, want %q", snap.GetSourceVolumeId(), "vol-1")
	}
	if !snap.GetReadyToUse() {
		t.Error("ReadyToUse = false, want true")
	}
	if snap.GetSizeBytes() != int64(len("hello from vol-1")) {
		t.Errorf("SizeBytes = %d, want %d", snap.GetSizeBytes(), len("hello from vol-1"))
	}
	if snap.GetCreationTime() == nil {
		t.Error("CreationTime is nil, want a real timestamp")
	}

	got, err := os.ReadFile(filepath.Join(dataDir, "snapshots", "snap-1", "hello.txt"))
	if err != nil {
		t.Fatalf("reading copied snapshot data: %v", err)
	}
	if string(got) != "hello from vol-1" {
		t.Errorf("copied snapshot data = %q, want %q", got, "hello from vol-1")
	}
}

--- FAIL: TestCreateSnapshot_CreatesTheSnapshot (0.00s)
    controller_test.go:399: SnapshotId = "", want "snap-1"
    controller_test.go:402: SourceVolumeId = "", want "vol-1"
    controller_test.go:405: ReadyToUse = false, want true
    controller_test.go:408: SizeBytes = 0, want 16
    controller_test.go:411: CreationTime is nil, want a real timestamp
    controller_test.go:416: reading copied snapshot data: open .../snapshots/snap-1/hello.txt: no such file or directory
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.007s
FAIL

Five separate assertions failing from one missing feature — worth reading as one red, not five. Nothing about the stub touches the filesystem or the response at all yet.

Fixing this needs three things: a place on disk for a snapshot's copied files, a place on disk for a snapshot's metadata, and the copy itself. Add these three helpers below CreateSnapshot in internal/driver/controller.go:

// snapshotDataPath is where a snapshot's copied files live — a full copy of
// the source volume's directory, at the time CreateSnapshot ran.
func snapshotDataPath(dataDir, snapshotID string) string {
	return filepath.Join(dataDir, "snapshots", snapshotID)
}

// snapshotMetaPath is where a snapshot's metadata lives — deliberately
// outside snapshotDataPath, the same reasoning behind every sidecar
// metadata file this book uses: metadata must never end up inside a
// directory that later gets bind-mounted or copied again as volume data.
func snapshotMetaPath(dataDir, snapshotID string) string {
	return filepath.Join(dataDir, ".csi-meta", "snapshots", snapshotID+".json")
}

// dirSize adds up the size of every regular file under path — the closest
// thing this toy driver has to asking a real backend how large a snapshot
// turned out to be.
func dirSize(path string) (int64, error) {
	var total int64
	err := filepath.WalkDir(path, func(_ string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		if d.IsDir() {
			return nil
		}
		info, err := d.Info()
		if err != nil {
			return err
		}
		total += info.Size()
		return nil
	})
	return total, err
}

dirSize walks the whole snapshot directory, not just its top level. filepath.WalkDir visits every entry under path — files and directories both — going arbitrarily deep, however many subdirectories the source volume happens to have. A volume with three levels of nested folders gets all three levels walked; dirSize doesn't know or care how deep it goes, and doesn't need to be told.

The callback answers one question per entry: is this a file? Directories return early — d.IsDir() — because a directory has no size of its own worth counting; only the files inside it do. Every regular file's size gets added to total, via d.Info().Size(). Skip a directory, add up a file: that's the whole function, run once per entry in the tree.

One more detail worth naming: the first error WalkDir hits — a permission problem, a broken symlink, anything — gets returned straight up and stops the walk immediately. dirSize never adds up a partial answer. A snapshot's reported size is either complete and correct, or CreateSnapshot fails and reports nothing at all — see the Internal error a few lines below, once this helper is wired in.

snapshotMeta, alongside them, is the piece actually worth pausing on: the data structure CreateSnapshot writes now, and reads back later. Add this too:

// snapshotMeta is the metadata CreateSnapshot records for each snapshot it
// creates, so a later CreateSnapshot call for the same name can tell a
// genuine idempotent retry (same source volume) apart from a real conflict
// (different source volume).
type snapshotMeta struct {
	SourceVolumeID string    `json:"source_volume_id"`
	CreationTime   time.Time `json:"creation_time"`
	SizeBytes      int64     `json:"size_bytes"`
}

func writeSnapshotMeta(dataDir, snapshotID string, meta snapshotMeta) error {
	metaBytes, err := json.Marshal(meta)
	if err != nil {
		return err
	}
	metaPath := snapshotMetaPath(dataDir, snapshotID)
	if err := os.MkdirAll(filepath.Dir(metaPath), 0o750); err != nil {
		return err
	}
	return os.WriteFile(metaPath, metaBytes, 0o640)
}

func readSnapshotMeta(dataDir, snapshotID string) (snapshotMeta, error) {
	var meta snapshotMeta
	data, err := os.ReadFile(snapshotMetaPath(dataDir, snapshotID))
	if err != nil {
		return meta, err
	}
	err = json.Unmarshal(data, &meta)
	return meta, err
}

Three fields, three jobs. SourceVolumeID is what makes idempotency and conflict-detection possible: a retry of CreateSnapshot for the same name only counts as the same snapshot if the source volume matches too — exactly the check CreateSnapshot makes a few lines below, once this struct exists to check against. CreationTime is captured once, the moment the snapshot is made, and never touched again; read it back a year later and it still says when the snapshot was actually created, not when someone last happened to ask. SizeBytes is dirSize's answer, cached at creation time rather than recomputed on every read — walking a directory tree on every question a caller might ask would get expensive once the driver is holding many snapshots, so CreateSnapshot pays that cost exactly once and writes the answer down.

writeSnapshotMeta and readSnapshotMeta are the same shape you've already seen: marshal snapshotMeta to JSON, write it under .csi-meta/, and on the way back in, read the bytes and unmarshal them into the same struct. Nothing new in the mechanics — this is Chapter 7's capacity-metadata pattern, reused for a second kind of metadata.

Add "encoding/json", "io/fs", and "time" to controller.go's imports.

Now the copy itself, replacing CreateSnapshot's old stub return, in internal/driver/controller.go:

	destPath := snapshotDataPath(d.dataDir, req.GetName())
	if err := os.CopyFS(destPath, os.DirFS(sourcePath)); err != nil {
		return nil, status.Errorf(codes.Internal, "copying volume %q into snapshot %q: %v", req.GetSourceVolumeId(), req.GetName(), err)
	}

	size, err := dirSize(destPath)
	if err != nil {
		return nil, status.Errorf(codes.Internal, "measuring snapshot %q: %v", req.GetName(), err)
	}

	meta := snapshotMeta{
		SourceVolumeID: req.GetSourceVolumeId(),
		CreationTime:   time.Now().UTC(),
		SizeBytes:      size,
	}
	if err := writeSnapshotMeta(d.dataDir, req.GetName(), meta); err != nil {
		return nil, status.Errorf(codes.Internal, "recording snapshot %q metadata: %v", req.GetName(), err)
	}

	return &csi.CreateSnapshotResponse{
		Snapshot: &csi.Snapshot{
			SnapshotId:     req.GetName(),
			SourceVolumeId: meta.SourceVolumeID,
			SizeBytes:      meta.SizeBytes,
			CreationTime:   timestamppb.New(meta.CreationTime),
			ReadyToUse:     true,
		},
	}, nil

Add "google.golang.org/protobuf/types/known/timestamppb" to controller.go's imports — the same package the real generated csi package already depends on for creation_time, since the spec defines that field as a google.protobuf.Timestamp, not a plain string or integer. timestamppb.New(t) builds one from an ordinary time.Time; .AsTime() is its inverse, for whenever this driver needs to read a Timestamp back as a time.Time (it doesn't, yet — nothing in this chapter converts one back). The test above checks GetCreationTime() == nil rather than .AsTime().IsZero() on purpose: a real *timestamppb.Timestamp's AsTime() converts a nil receiver to the Unix epoch, not Go's zero time.Time{} — .IsZero() on that result is false, not true, so it can't actually tell "never set" apart from "a snapshot genuinely created at 1970-01-01." Checking the pointer for nil is the one that's actually reliable.

os.CopyFS — new to this book, standard library since Go 1.23 — does the entire recursive copy in one call: os.CopyFS(dir, fsys) copies every file fsys contains into dir, creating dir itself if it doesn't already exist. os.DirFS(sourcePath) turns an ordinary directory into the fs.FS value CopyFS wants. Two standard-library calls, no hand-rolled recursive walk needed — the same "no exotic machinery" spirit this driver has held to since Chapter 1, now applying to a slightly less trivial operation than MkdirAll.

go test ./internal/driver/...

=== RUN   TestCreateSnapshot_CreatesTheSnapshot
--- PASS: TestCreateSnapshot_CreatesTheSnapshot (0.00s)

Green. ReadyToUse: true unconditionally is worth a direct word, since the field name implies there could be a false case: localdir-csi's "copy" is a single synchronous os.CopyFS call, done by the time CreateSnapshot returns — there's no window where the snapshot exists but isn't finished yet. A real backend snapshotting a multi-terabyte volume asynchronously would legitimately return ready_to_use: false at first, with ListSnapshots or a later poll eventually reporting true once the backend finishes. This driver never has that problem, so it never needs that field's honest range — another small, real consequence of "local directories on the node," the same simplification Chapter 7 already named as this driver's defining one.

CreateSnapshot, test-first: idempotency, and a conflict caught from day one

Same two-part shape the spec asked of CreateVolume back in Chapter 7: OK for a genuine retry, a real error for a genuine conflict. Add both cases at once to internal/driver/controller_test.go:

func TestCreateSnapshot_Idempotent(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	if err := os.WriteFile(filepath.Join(dataDir, "vol-1", "hello.txt"), []byte("hello"), 0o640); err != nil {
		t.Fatalf("writing fake volume data: %v", err)
	}
	d := newTestDriver(t, dataDir, nil)

	req := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-1",
		Name:           "snap-1",
	}

	first, err := d.CreateSnapshot(t.Context(), req)
	if err != nil {
		t.Fatalf("first CreateSnapshot() returned an error: %v", err)
	}

	second, err := d.CreateSnapshot(t.Context(), req)
	if err != nil {
		t.Fatalf("second CreateSnapshot() for the same name returned an error: %v", err)
	}
	if second.GetSnapshot().GetSnapshotId() != first.GetSnapshot().GetSnapshotId() {
		t.Errorf("SnapshotId = %q, want %q", second.GetSnapshot().GetSnapshotId(), first.GetSnapshot().GetSnapshotId())
	}
	if second.GetSnapshot().GetSourceVolumeId() != "vol-1" {
		t.Errorf("SourceVolumeId = %q, want %q", second.GetSnapshot().GetSourceVolumeId(), "vol-1")
	}
}

func TestCreateSnapshot_RejectsSourceVolumeMismatch(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	makeVolumeDir(t, dataDir, "vol-2")
	d := newTestDriver(t, dataDir, nil)

	first := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-1",
		Name:           "snap-1",
	}
	if _, err := d.CreateSnapshot(t.Context(), first); err != nil {
		t.Fatalf("first CreateSnapshot() returned an error: %v", err)
	}

	second := &csi.CreateSnapshotRequest{
		SourceVolumeId: "vol-2",
		Name:           "snap-1",
	}
	_, err := d.CreateSnapshot(t.Context(), second)
	requireStatusCode(t, err, codes.AlreadyExists, second)
}

=== RUN   TestCreateSnapshot_Idempotent
    controller_test.go:443: second CreateSnapshot() for the same name returned an error: rpc error: code = Internal desc = copying volume "vol-1" into snapshot "snap-1": open /tmp/TestCreateSnapshot_Idempotent2485715020/001/snapshots/snap-1/hello.txt: file exists
--- FAIL: TestCreateSnapshot_Idempotent (0.00s)
=== RUN   TestCreateSnapshot_RejectsSourceVolumeMismatch
    controller_test.go:472: code = OK, want AlreadyExists (req=&{SourceVolumeId:vol-2 Name:snap-1 Secrets:map[] Parameters:map[]})
--- FAIL: TestCreateSnapshot_RejectsSourceVolumeMismatch (0.00s)
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.004s
FAIL

Both fail, but not for the same reason. The conflict case fails the way you'd expect — no check exists yet, so a mismatched source volume sails through as OK. The idempotency case fails for a sharper reason, worth stating plainly: os.CopyFS is not idempotent-safe. Standard library, since Go 1.23, and it refuses to overwrite a file that already exists at the destination — call it twice with the same source and destination and the second call fails with exactly the error above, file exists. A source volume with real data in it makes that collision certain on any genuine retry. TestCreateSnapshot_Idempotent seeding an actual file, instead of leaving the volume empty, is what surfaces this — an empty volume gives os.CopyFS nothing to collide with on the second call, so the test would pass by accident and hide the bug. That's the exact same lesson Chapter 9's real csi-sanity run taught about CreateVolume's own idempotency gap: a test only proves what it actually exercises.

Both gaps need the same fix: check the snapshot's own metadata before touching the filesystem at all, and short-circuit on a genuine retry instead of ever calling os.CopyFS a second time. Add the check to CreateSnapshot, in internal/driver/controller.go, right after the source-volume existence check and before the copy:

	if existing, err := readSnapshotMeta(d.dataDir, req.GetName()); err == nil {
		if existing.SourceVolumeID != req.GetSourceVolumeId() {
			return nil, status.Errorf(codes.AlreadyExists, "snapshot %q already exists for a different source volume", req.GetName())
		}
		// Same name, same source volume: a genuine retry. Return the
		// snapshot already on disk instead of falling through to
		// os.CopyFS below — it refuses to overwrite files that already
		// exist, so re-copying here would fail on every idempotent retry.
		return &csi.CreateSnapshotResponse{
			Snapshot: &csi.Snapshot{
				SnapshotId:     req.GetName(),
				SourceVolumeId: existing.SourceVolumeID,
				SizeBytes:      existing.SizeBytes,
				CreationTime:   timestamppb.New(existing.CreationTime),
				ReadyToUse:     true,
			},
		}, nil
	} else if !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "reading snapshot %q metadata: %v", req.GetName(), err)
	}

=== RUN   TestCreateSnapshot_Idempotent
--- PASS: TestCreateSnapshot_Idempotent (0.00s)
=== RUN   TestCreateSnapshot_RejectsSourceVolumeMismatch
--- PASS: TestCreateSnapshot_RejectsSourceVolumeMismatch (0.00s)

Both green — for real reasons this time, not by accident of an empty test volume. Read the metadata once, decide from it: a matching source volume is a retry, answered from what's already recorded; a mismatched one is a real conflict, rejected; anything else falls through to do the actual copy, exactly once, ever, per snapshot name.

DeleteSnapshot, test-first

Same three-cycle shape as DeleteVolume in Chapter 7. Add to internal/driver/controller_test.go:

func TestDeleteSnapshot_Validation(t *testing.T) {
	d := newTestDriverInTempDir(t)

	_, err := d.DeleteSnapshot(t.Context(), &csi.DeleteSnapshotRequest{})
	requireStatusCode(t, err, codes.InvalidArgument, &csi.DeleteSnapshotRequest{})
}

--- FAIL: TestDeleteSnapshot_Validation (0.00s)
    controller_test.go:479: code = Unimplemented, want InvalidArgument (req=&{SnapshotId: Secrets:map[]})
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.012s
FAIL

func (d *Driver) DeleteSnapshot(
	ctx context.Context,
	req *csi.DeleteSnapshotRequest,
) (*csi.DeleteSnapshotResponse, error) {
	if req.GetSnapshotId() == "" {
		return nil, status.Error(codes.InvalidArgument, "snapshot_id is required")
	}

	return &csi.DeleteSnapshotResponse{}, nil
}

=== RUN   TestDeleteSnapshot_Validation
--- PASS: TestDeleteSnapshot_Validation (0.00s)

Green. Now the removal, and the idempotency case, both at once:

func TestDeleteSnapshot_RemovesTheSnapshot(t *testing.T) {
	dataDir := t.TempDir()
	makeVolumeDir(t, dataDir, "vol-1")
	d := newTestDriver(t, dataDir, nil)

	createReq := &csi.CreateSnapshotRequest{SourceVolumeId: "vol-1", Name: "snap-1"}
	if _, err := d.CreateSnapshot(t.Context(), createReq); err != nil {
		t.Fatalf("CreateSnapshot() returned an error: %v", err)
	}

	deleteReq := &csi.DeleteSnapshotRequest{SnapshotId: "snap-1"}
	if _, err := d.DeleteSnapshot(t.Context(), deleteReq); err != nil {
		t.Fatalf("DeleteSnapshot() returned an error: %v", err)
	}

	if _, err := os.Stat(filepath.Join(dataDir, "snapshots", "snap-1")); !os.IsNotExist(err) {
		t.Errorf("snapshot data directory still exists after DeleteSnapshot")
	}
	if _, err := os.Stat(filepath.Join(dataDir, ".csi-meta", "snapshots", "snap-1.json")); !os.IsNotExist(err) {
		t.Errorf("snapshot metadata still exists after DeleteSnapshot")
	}
}

func TestDeleteSnapshot_Idempotent(t *testing.T) {
	d := newTestDriverInTempDir(t)

	req := &csi.DeleteSnapshotRequest{SnapshotId: "does-not-exist"}

	if _, err := d.DeleteSnapshot(t.Context(), req); err != nil {
		t.Fatalf("DeleteSnapshot() for an already-gone snapshot returned an error: %v", err)
	}
}

--- FAIL: TestDeleteSnapshot_RemovesTheSnapshot (0.00s)
    controller_test.go:498: snapshot data directory still exists after DeleteSnapshot
    controller_test.go:501: snapshot metadata still exists after DeleteSnapshot
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.011s
FAIL

TestDeleteSnapshot_Idempotent passes immediately — os.RemoveAll's already-gone-is-fine behavior, the same free property DeleteVolume had in Chapter 7. The removal test doesn't. Fix it in internal/driver/controller.go, replacing DeleteSnapshot's old stub return:

	if err := os.RemoveAll(snapshotDataPath(d.dataDir, req.GetSnapshotId())); err != nil {
		return nil, status.Errorf(codes.Internal, "deleting snapshot %q: %v", req.GetSnapshotId(), err)
	}
	if err := os.Remove(snapshotMetaPath(d.dataDir, req.GetSnapshotId())); err != nil && !os.IsNotExist(err) {
		return nil, status.Errorf(codes.Internal, "deleting snapshot %q metadata: %v", req.GetSnapshotId(), err)
	}

	return &csi.DeleteSnapshotResponse{}, nil

=== RUN   TestDeleteSnapshot_RemovesTheSnapshot
--- PASS: TestDeleteSnapshot_RemovesTheSnapshot (0.00s)
=== RUN   TestDeleteSnapshot_Idempotent
--- PASS: TestDeleteSnapshot_Idempotent (0.00s)

Both green.

ControllerGetCapabilities keeps the promise Chapter 7 made

Chapter 7 built ControllerGetCapabilities around a slice literal on purpose, "specifically so that a later chapter adding LIST_VOLUMES or GET_CAPACITY support means appending an entry, not restructuring this method." This is that later chapter, just with a different capability than the two named at the time. Add this test to internal/driver/controller_test.go:

func TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot(t *testing.T) {
	d := newTestDriver(t, dataDir, nil)

	resp, err := d.ControllerGetCapabilities(t.Context(), &csi.ControllerGetCapabilitiesRequest{})
	if err != nil {
		t.Fatalf("ControllerGetCapabilities returned an error: %v", err)
	}

	var found bool
	for _, c := range resp.Capabilities {
		if rpc := c.GetRpc(); rpc != nil && rpc.Type == csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT {
			found = true
		}
	}
	if !found {
		t.Error("CREATE_DELETE_SNAPSHOT not found among advertised capabilities")
	}
}

--- FAIL: TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot (0.00s)
    controller_test.go:530: CREATE_DELETE_SNAPSHOT not found among advertised capabilities
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.012s
FAIL

Append the entry, in internal/driver/controller.go:

			{
				Type: &csi.ControllerServiceCapability_Rpc{
					Rpc: &csi.ControllerServiceCapability_RPC{
						Type: csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT,
					},
				},
			},

That goes inside ControllerGetCapabilities's existing Capabilities: []*csi.ControllerServiceCapability{...} slice, right after the CREATE_DELETE_VOLUME entry Chapter 7 wrote. No restructuring, exactly as promised — one entry, appended.

go test ./internal/driver/...

--- FAIL: TestControllerGetCapabilities (0.00s)
    controller_test.go:21: got 2 capabilities, want 1
FAIL
FAIL	github.com/yourname/localdir-csi/internal/driver	0.012s
FAIL

The promise held for the implementation — one entry, appended, no restructuring. It didn't hold for the original test. TestControllerGetCapabilities, written back in Chapter 7, hardcoded len(resp.Capabilities) != 1, a check that was never going to survive a second capability being added, appended or not. Worth being honest about that rather than quietly patching around it: Open/Closed applied to the method's own logic, not to every assertion a much earlier test happened to make about its output. Fix the old test itself, in internal/driver/controller_test.go:

	if len(resp.Capabilities) == 0 {
		t.Fatal("got 0 capabilities, want at least 1")
	}

That replaces the old != 1 check. The rest of TestControllerGetCapabilities — asserting Capabilities[0] is specifically CREATE_DELETE_VOLUME — stays exactly as Chapter 7 wrote it, since ControllerGetCapabilities still appends new entries after the existing ones, never reorders them.

=== RUN   TestControllerGetCapabilities
--- PASS: TestControllerGetCapabilities (0.00s)
=== RUN   TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot
--- PASS: TestControllerGetCapabilities_AdvertisesCreateDeleteSnapshot (0.00s)

Both green, and the full suite:

go test ./internal/driver/... -v

ok  	github.com/yourname/localdir-csi/internal/driver	0.015s

What this chapter doesn't implement, on purpose

Two gaps, named directly, not glossed past.

ListSnapshots stays a bare Unimplemented stub, the same as ControllerPublishVolume was before Chapter 8 gave it a real body — except this time, unlike Chapter 8, nothing in this chapter gives it one. ControllerGetCapabilities never advertises LIST_SNAPSHOTS alongside CREATE_DELETE_SNAPSHOT, so any conformant CO, csi-sanity included, treats it exactly the way Chapter 9 explained for every other unadvertised RPC: a silent, deliberate skip, not a failure. A CO can still create and delete snapshots by name without it; ListSnapshots only matters for discovering snapshots a CO doesn't already know the IDs of, a case this book's own demo never needs.

CreateVolume still doesn't read volume_content_source — the field that would let a new volume start out as a copy of an existing snapshot, the "cloning" half of this chapter's opening list of reasons snapshots matter. CreateSnapshot and DeleteSnapshot cover the "backup" half completely: a snapshot can be taken, inspected on disk, and removed. Restoring from one — turning a snapshot back into a usable volume — is real, additional work CreateVolume doesn't do, and this book's outline doesn't schedule a return trip to it. Worth knowing exactly where the line sits, rather than assuming "snapshots" as a chapter title means the whole feature is done.

deploy/: the first cluster-level component this book has needed

Every sidecar so far — node-driver-registrar in Chapter 4, external-provisioner in Chapter 7 — ships as a container inside one of this driver's own pods, in deploy/node.yaml or deploy/controller.yaml. Snapshots need one of those too, but they also need something genuinely new: a piece of cluster machinery that has nothing to do with localdir-csi specifically, installed once per cluster, shared by every CSI driver that ever wants to support snapshots.

That's the snapshot-controller — not a sidecar, and not part of deploy/ at all. It watches VolumeSnapshot objects cluster-wide (the CO-facing, namespaced request — "I want a snapshot of this claim," the direct analog of a PersistentVolumeClaim) and, for each one, creates a matching VolumeSnapshotContent object (the cluster-scoped record of an actual snapshot, the direct analog of a PersistentVolume). It never talks to any driver's socket directly. That's csi-snapshotter's job — the actual sidecar, watching VolumeSnapshotContent objects and turning them into the CreateSnapshot/DeleteSnapshot gRPC calls this chapter just built, the same relationship external-provisioner has to PersistentVolumes and CreateVolume.

Three CRDs come with this split: VolumeSnapshotClass (the StorageClass equivalent — names which driver handles a class of snapshot), VolumeSnapshot, and VolumeSnapshotContent. None of them exist in a stock Kubernetes cluster; they're part of what installing the snapshot-controller sets up.

Install the CRDs and the snapshot-controller once, cluster-wide — not part of make deploy, and not repeated for every driver a cluster ever adds:

kubectl kustomize "https://github.com/kubernetes-csi/external-snapshotter/client/config/crd?ref=v8.2.0" | kubectl create -f -
kubectl -n kube-system kustomize "https://github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/snapshot-controller?ref=v8.2.0" | kubectl create -f -

kubectl get pods -n kube-system -l app.kubernetes.io/name=snapshot-controller

NAME                                  READY   STATUS    RESTARTS   AGE
snapshot-controller-7d8f6c9b5-k2n4p   1/1     Running   0          11s

That pod will sit there watching VolumeSnapshot objects regardless of which drivers this cluster ever adds — genuinely shared infrastructure, the first thing this book has deployed that isn't localdir-csi's own.

deploy/controller.yaml and deploy/controller-rbac.yaml: the third sidecar

csi-snapshotter, unlike snapshot-controller, is per-driver — it needs this driver's own socket, so it belongs in deploy/controller.yaml alongside localdir-csi and external-provisioner. Add a third container:

        - name: csi-snapshotter
          image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi

That block goes inside deploy/controller.yaml's existing containers: list, after the external-provisioner entry Chapter 7 added — same socket-dir emptyDir, same --csi-address flag shape, no new volume needed. Three containers now sharing one socket: the driver itself, and two sidecars each watching a different piece of cluster state and translating it into calls against that same socket.

RBAC needs a real extension too, not a cosmetic one. Chapter 7 trimmed external-provisioner's rules and said directly: "snapshot-related rules (volumesnapshots, volumesnapshotcontents) are absent because localdir-csi doesn't implement CreateSnapshot." That's no longer true. Add a second ClusterRole and ClusterRoleBinding to deploy/controller-rbac.yaml, trimmed from external-snapshotter's own published reference RBAC the same honest way Chapter 7 trimmed external-provisioner's:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: localdir-csi-snapshotter
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents/status"]
    verbs: ["update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: localdir-csi-snapshotter
subjects:
  - kind: ServiceAccount
    name: localdir-csi-controller
    namespace: default
roleRef:
  kind: ClusterRole
  name: localdir-csi-snapshotter
  apiGroup: rbac.authorization.k8s.io

No new ServiceAccount — the binding's subjects names localdir-csi-controller, the exact same one Chapter 7 already created. serviceAccountName in deploy/controller.yaml's pod spec only takes one value, and every container in a pod runs under that same identity regardless of which sidecar happens to need which specific permissions. One ServiceAccount, now bound to two ClusterRoles instead of one — external-provisioner's from Chapter 7, and csi-snapshotter's, both from deploy/controller-rbac.yaml — is exactly how a pod with three containers and two different sidecars' worth of Kubernetes API needs actually works.

Two categories missing from csi-snapshotter's own reference RBAC here, gone on purpose, the same way Chapter 7 explained its own omissions rather than leaving them silent. Volume group snapshot rules (volumegroupsnapshotclasses, volumegroupsnapshotcontents) — csi-snapshotter's published reference RBAC includes them, but localdir-csi doesn't implement CREATE_DELETE_VOLUME_GROUP_SNAPSHOT, a capability this book has never mentioned and doesn't start now. Leader-election rules on leases — same reasoning Chapter 7 already gave for skipping them on external-provisioner: a lock only earns its keep with more than one replica contending for it, and deploy/controller.yaml has exactly one.

deploy/volumesnapshotclass.yaml

The StorageClass equivalent for snapshots — the object that names which driver a VolumeSnapshot actually resolves to:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: localdir-csi-snapshot
driver: localdir.csi.example.com
deletionPolicy: Delete

driver has to match this driver's own name exactly, the same provisioner field rule Chapter 8's StorageClass already followed. deletionPolicy: Delete mirrors StorageClass's own reclaimPolicy: the underlying VolumeSnapshotContent — and, per this chapter's own DeleteSnapshot, the actual copied directory on disk — gets deleted automatically once its VolumeSnapshot is deleted.

Deploy everything:

make deploy
kubectl apply -f deploy/volumesnapshotclass.yaml
kubectl get pods -l app=localdir-csi-controller

NAME                                        READY   STATUS    RESTARTS   AGE
localdir-csi-controller-6f7d8b9c4d-x9m3q    3/3     Running   0          12s

3/3 — the same signal Chapter 4 and Chapter 7 both already explained, now with a third container in it.

Proving it, with grpcurl

Same kubectl cp/kubectl exec pattern as Chapters 7 and 8. First, a real volume, with real data in it — grpcurl alone only ever proved volumes exist, so this is the first proof in this book that reaches past the socket onto the node's own disk to seed content directly, before calling anything:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "name": "snap-source",
  "capacity_range": {"required_bytes": 1048576},
  "volume_capabilities": [{"mount": {}, "access_mode": {"mode": "SINGLE_NODE_WRITER"}}]
}' unix:///csi/csi.sock csi.v1.Controller/CreateVolume

docker exec csi-dev-control-plane sh -c \
  'echo "hello from a real volume" > /var/lib/localdir-csi/data/snap-source/hello.txt'

Now snapshot it:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

{
  "snapshot": {
    "sizeBytes": "25",
    "snapshotId": "snap-demo",
    "sourceVolumeId": "snap-source",
    "creationTime": "2026-08-22T16:09:22.689807357Z",
    "readyToUse": true
  }
}

Check the copy landed on the node's own disk, independent of anything the driver reported about itself:

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/snapshots/snap-demo/hello.txt

hello from a real volume

Real data, really copied. Call CreateSnapshot again, identical request:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

ERROR:
  Code: Internal
  Message: copying volume "snap-source" into snapshot "/data/snapshots/snap-demo": open /data/snapshots/snap-demo/hello.txt: file exists
command terminated with exit code 77

Not 0 OK — a real failure, against a real cluster, on the exact retry this section calls "the same request." This is the os.CopyFS-refuses-to-overwrite bug from the TDD walkthrough above, caught live: the driver running in this cluster predated the early-return fix, so a genuine retry fell through to a second copy and collided with the file the first call had already written. The fix is the one already added to internal/driver/controller.go above. make deploy (Chapter 2) rebuilds the image, reloads it into the cluster, and restarts both the Controller Deployment and the Node DaemonSet on your behalf, so the already-running pod picks up the new build without any extra command here:

make deploy

Call CreateSnapshot again, same request, against the redeployed driver:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

{
  "snapshot": {
    "sizeBytes": "25",
    "snapshotId": "snap-demo",
    "sourceVolumeId": "snap-source",
    "creationTime": "2026-08-22T16:09:22.689807357Z",
    "readyToUse": true
  }
}

0 OK — and worth reading closely: creationTime is the exact same timestamp as the very first CreateSnapshot call above, down to the nanosecond. The driver's data directory is a hostPath, so it survived the redeploy — snap-demo's metadata was still sitting on disk from before the fix. This call landed on the new early-return branch, matched that recorded metadata, and returned it straight back without ever touching os.CopyFS. Confirm the file itself was never touched again either:

docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/snapshots/snap-demo/hello.txt

hello from a real volume

Same content, same one real copy. Call CreateSnapshot a third time, same request, to prove the first retry wasn't a fluke:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "source_volume_id": "snap-source",
  "name": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/CreateSnapshot

{
  "snapshot": {
    "sizeBytes": "25",
    "snapshotId": "snap-demo",
    "sourceVolumeId": "snap-source",
    "creationTime": "2026-08-22T16:09:22.689807357Z",
    "readyToUse": true
  }
}

Identical again. TestCreateSnapshot_Idempotent, proved for real this time, against the exact bug that first broke it. Now delete it:

kubectl exec deploy/localdir-csi-controller -c localdir-csi -- /tmp/grpcurl -plaintext -d '{
  "snapshot_id": "snap-demo"
}' unix:///csi/csi.sock csi.v1.Controller/DeleteSnapshot

{}

docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/snapshots/

Empty — gone, the same proof-from-both-sides technique every earlier chapter's grpcurl section has used.

Part two: a VolumeSnapshot object, watched and acted on automatically

Same closing move as Chapter 8's PersistentVolumeClaim section: prove the whole loop runs without a single RPC typed by hand. Reuse demo-pvc.yaml and demo-pod.yaml from Chapter 8 to get a real, mounted, localdir-csi-backed volume with a pod actually writing to it:

kubectl apply -f demo-pvc.yaml
kubectl apply -f demo-pod.yaml
kubectl exec demo-pod -- sh -c 'echo snapshot me > /data/notes.txt'

Create a VolumeSnapshot, naming the claim it's a snapshot of, and the VolumeSnapshotClass from earlier:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: demo-snapshot
spec:
  volumeSnapshotClassName: localdir-csi-snapshot
  source:
    persistentVolumeClaimName: demo-claim

Save that as demo-volumesnapshot.yaml, outside deploy/, the same place Chapter 8 kept demo-pvc.yaml — a workload object, not part of this driver's own deployment.

kubectl apply -f demo-volumesnapshot.yaml
kubectl get volumesnapshot demo-snapshot

NAME            READYTOUSE   SOURCEPVC     RESTORESIZE   SNAPSHOTCLASS           SNAPSHOTCONTENT                                       AGE
demo-snapshot   false        demo-claim                  localdir-csi-snapshot                                                          2s

READYTOUSE false — a moment worth watching, not skipping past. Check what's actually happening:

kubectl logs -n kube-system -l app.kubernetes.io/name=snapshot-controller --tail=10

I0821 10:22:01.334221       1 snapshot_controller.go:255] createSnapshotContent: Creating content for snapshot demo-snapshot through the plugin ...
I0821 10:22:01.335590       1 snapshot_controller.go:271] volumeSnapshotContent "snapcontent-..." created

That's snapshot-controller — cluster-level, watching VolumeSnapshot directly — creating the matching VolumeSnapshotContent object. Nothing has called CreateSnapshot yet; that's the next step, and it belongs to a different component entirely:

kubectl logs -l app=localdir-csi-controller -c csi-snapshotter --tail=10

I0821 10:22:01.410332       1 snapshotter.go:126] CreateSnapshot for content snapcontent-3a9f21e0-...
I0821 10:22:01.410451       1 controller.go:759] createSnapshotWrapper: Creating snapshot for content ...
I0821 10:22:01.421187       1 controller.go:801] createSnapshotWrapper: create snapshot ... succeeded

csi-snapshotter — per-driver, watching VolumeSnapshotContent — is what actually called CreateSnapshot, the exact method this chapter built, over the exact same socket external-provisioner already shares. A moment later:

kubectl get volumesnapshot demo-snapshot

NAME            READYTOUSE   SOURCEPVC     RESTORESIZE   SNAPSHOTCLASS           SNAPSHOTCONTENT                                       AGE
demo-snapshot   true         demo-claim    1Gi           localdir-csi-snapshot   snapcontent-3a9f21e0-...                              6s

READYTOUSE true — this driver's own ReadyToUse: true, set the instant CreateSnapshot returned, now visible all the way up through VolumeSnapshotContent to the VolumeSnapshot a person actually looks at. Confirm the data itself, directly on the node's disk, the same way every earlier proof in this book has:

CONTENT=$(kubectl get volumesnapshot demo-snapshot -o jsonpath='{.status.boundVolumeSnapshotContentName}')
SNAPID=$(kubectl get volumesnapshotcontent "$CONTENT" -o jsonpath='{.status.snapshotHandle}')
docker exec csi-dev-control-plane cat /var/lib/localdir-csi/data/snapshots/"$SNAPID"/notes.txt

snapshot me

The exact string a pod wrote through a real mount, now sitting inside a snapshot directory nothing but a kubectl apply ever asked for by name.

Tear it down:

kubectl delete volumesnapshot demo-snapshot
docker exec csi-dev-control-plane ls /var/lib/localdir-csi/data/snapshots/

Empty — DeleteSnapshot, called automatically, the same cleanup Chapter 8 already proved for DeleteVolume when a claim gets deleted.

What you should have now

• internal/driver/controller.go: real CreateSnapshot and DeleteSnapshot, both tested, both idempotent — the conflict check came from designing the metadata in from day one, but the idempotency bug itself was found the same way Chapter 9 found CreateVolume's capacity gap: a test that passed by accident (an empty test volume hid it), caught for real once a real cluster run copied actual data
• A working sidecar-metadata-file pattern (snapshotMeta, writeSnapshotMeta, readSnapshotMeta) that answers "what was this called with, last time" for data a bare directory can't remember on its own
• ControllerGetCapabilities advertising CREATE_DELETE_SNAPSHOT, proving out the exact Open/Closed extension point Chapter 7 designed in — and one honest fix to a Chapter 7 test that wasn't quite as future-proof as its own implementation
• deploy/controller.yaml's third container, csi-snapshotter, and deploy/controller-rbac.yaml's second ClusterRole, both trimmed from real upstream reference manifests the same honest way Chapter 7 trimmed external-provisioner's
• deploy/volumesnapshotclass.yaml, and the first cluster-level, not-driver-specific component this book has ever had to install: snapshot-controller, shared infrastructure any CSI driver's snapshot support depends on
• Two forms of proof again, grpcurl by hand and a real VolumeSnapshot object watched end to end — data written through a real mount, copied by a real CreateSnapshot call nothing typed by hand ever triggered, and readable straight off the node's disk afterward

Two honest gaps remain, both already named directly above: ListSnapshots stays an unadvertised, correctly-skipped stub, and CreateVolume still can't restore a volume from a snapshot's contents — volume_content_source sits in CreateVolumeRequest, unread, same as every chapter since Chapter 7. Both stay that way for the rest of this book — deliberate, not deferred. Chapter 11 turns to a different kind of gap: not a missing RPC, but everything this driver is still missing to be safely operated for real — structured error codes, logs, and metrics that can actually be scraped off a running pod.