Storage, multi-GPU, and learning what 'network' actually means

PodWarden Team

Vancouver, February 26 to 28, 2026. The first daffodils are out on Commercial Drive. Vancouver in late February is famous for two days of false spring followed by another month of rain, and we're in the false-spring days. Making them count!

This was a week of unglamorous, heavy-lifting infrastructure. The kind of work that doesn't make a great launch tweet but makes the difference between a prop and the real thing. If you remember nothing else from this article, remember this: state is the hardest part of any system, and we spent three days getting it right.

What we shipped this week

New functionality

  • Persistent storage, the right way. PodWarden now understands three storage worlds:

    • PVC (PersistentVolumeClaim), the K8s-native way, for things like Postgres or Redis that want a single mount.
    • NFS, for the very common "I have a NAS and I want my pods to share files" case.
    • S3 and S3-compatible (MinIO, Wasabi, Cloudflare R2, and so on), for object storage workloads.

    Each of those gets its own concept of a "storage connection", so you can register, say, "the office NAS" once and reuse it across every app you deploy.

  • Multi-GPU scheduling. If your nodes have more than one GPU (and a lot of our clients are running RTX A4000s or 3090s in pairs), PodWarden knows how to ask K8s for "this app wants GPU 0 and GPU 1," or "any 2 GPUs on this node." Sounds boring. Took two days. Worth it.

  • DB-encrypted secrets. All the app secrets we introduced two weeks ago? We tightened the loop with a per-deployment data-encryption key and proper key rotation. If you compromise the database file alone, you still don't get the secrets.

  • Network awareness: public, mesh, lan. Every host now declares which networks it lives on (public internet, Tailscale mesh, local LAN), and PodWarden uses that when picking a path. The UI shows the chips so you can see at a glance which way the connection is going.

Changed and refactored

  • The whole K8s deployer was rewritten. The first version was a thin wrapper around kubectl apply. The new one composes manifests internally, does its own diffing, and gives us proper structured errors when something refuses to land.
  • Storage volumes used to be tied 1:1 to deployments. Now they're independent, can be reattached, and survive a redeploy.

Bugfixes

  • A weirdly persistent issue where multi-replica deployments would race on first start and one replica would get an empty PVC. Solved by making the volume-bind step idempotent and waiting for it explicitly.
  • Network detection was occasionally classifying a Tailscale IP as "public" if it sat in a particular CIDR slice. Fixed, and added a unit test.

Experimenting

  • We're prototyping a "dry run before deploy" view that shows you the manifests PodWarden is about to apply. Not shipped yet, but we have a rewritten deployer that will make it possible!

Was this hard?

About 220 commits across the three days. Two of those were "fix typo in changelog." The rest were hard work. The K8s deployer rewrite was a single MR with 48 files in it. We held our breath through code review and shipped it on Friday afternoon, wayy later than we usually like, but the test suite was green and we wanted to head into the weekend with a clean main branch.

How this helps our users

We keep saying: "PodWarden should be the boring layer." The exciting work is your app. PodWarden's job is to remember where your data lives, hand the right GPU to the right pod, and never lose track of which network it's on. We were able to do all three this week.

If you've ever lost a Postgres volume because Helm decided to recreate a PVC, or spent an entire afternoon figuring out why your two-GPU model server only saw one GPU, you understand what we did here.

Notes from the room

The Vancouver tech crowd is starting to talk about KubeCon EU prep. There's a meetup happening at Notch8 next week, and one of our members is doing a 5-minute lightning talk on Tailscale-aware K3s clusters. We agreed that "shipped the K8s deployer rewrite" is the line we're going to toast at the team dinner on Saturday.

Mood: focused, kinda underslept, but pushing through!!