← Blog

Orchestrating On-Demand Minigame Servers with Redis Pubsub

The Problem

Every minigame on our Minecraft network runs as its own ephemeral Kubernetes pod. They spawn on demand when a player queues, run their round, and disappear. Players shouldn't have to think about any of this, instead they queue up, wait a moment, and land in a game.

This means coordinating between two systems: a Velocity plugin that handles the player-facing queue and a Go controller that talks to the Kubernetes API. Both exist in the same cluster. Both need to agree on a server name, a game type, and a set of players before anything happens.

We chose to do this entirely via Redis pubsub to keep things quick and keep surfaces minimal. The controller that talks to Kube was explicitly removed from Velocity and turned into its own thing for the purposes of security boundaries. We didn't want Velocity, a public facing service, having write access to Kube itself.

The Flow

  1. Player queues up with e.g. /queue spleef
  2. plugin-minigame publishes minigame:start to Redis
  3. controller-minigame picks up the message
    1. Checks cluster capacity to ensure that we can allocate a pod, with a 20% buffer
    2. Generates a unique server name
    3. Provisions the game pod
    4. Publishes minigame:start:result
  4. plugin-minigame receives the result and either queues the players or backs off the queue
  5. When the server registers with Velocity, the players in the related queue get sent on their way

The Wire Protocol

The start message is simple JSON:

{
  "game": "spleef",
  "queue_id": "bbdc8590-eb25-4c00-b0f1-f81e6c0ec6ef",
  "players": [
    "8597a593-b063-4446-b78e-f9574b3a461c",
    "ebcabf3d-ec98-4d58-91fd-e1cdb8236b9e",
    "455d2c13-2ba5-460e-85e7-5424244be8cf"
  ]
}

The controller responds on a separate channel with:

{
  "server": "spleef-xk9d",
  "game": "spleef",
  "queue_id": "bbdc8590-eb25-4c00-b0f1-f81e6c0ec6ef",
  "accepted": true,
  "reason": ""
}

The plugin doesn't care about delivery guarantees. If the response doesn't arrive within a reasonable time, it retries. The controller doesn't track state, instead, the pods are tagged as minigames in Kube. When a minigame pod terminates and lands in a terminal state, the controller deletes the pod and cleans up.

Server Name Generation

The controller generates server names like spleef-xk9d or siege-a3f8. Kubernetes requires unique names for pods.

  • We retry up to 10 times if the name collides or contains a banned substring
  • We fall back to a longer suffix (spleef-7f3a2b1c) if all 10 attempts fail
  • Names are checked against actual existing pods via kcli.Get() before creation

Capacity Checking

Before creating a pod, the controller checks whether any node has room:

  • Lists all nodes and their allocatable CPU/memory
  • Subtracts a 20% buffer from allocatable (nodes should never run at capacity)
  • If no node can fit the game's resource request, the controller publishes a rejection with reason "Cluster does not have enough resources"

This prevents the "pod stuck in Pending" scenario where the game never starts and players sit in an infinite loading screen. Instead, rejected players get put in a queue waiting for availability, and are told their position in line.

Pod Construction

The minigame OCI images are built from a base image containing all our tools with the game-specific image layered on top. Labels carry metadata for the K8s informer-based service registry on Velocity:

minigame/game-type: spleef
minigame/queue-id: bbdc8590-eb25-4c00-b0f1-f81e6c0ec6ef
haus.gamer.k8s/enable-server-discovery: "true"
haus.gamer.k8s/server-name: spleef-xk9d

The participant list is injected as an environment variable MINIGAME_PARTICIPANTS (comma-separated UUIDs). The game plugin reads this during startup and knows exactly who belongs in the match. Players who join without being in the list are left in spectator mode, though we currently don't allow players to join, other than admins porting in. The pod uses RestartPolicy: Never to ensure it gets reaped when it's complete.

Cold Start

Paper dominates startup time. We average about 30 seconds from a queue firing a server request to players landing in the game. We've optimized what we can. The image is pulled from a local registry, so there's no WAN download. Pre-warming is the obvious next step, keep a pod of each game type running in an idle state. That adds operational complexity (how many pre-warmed pods? how long to keep them? how to communicate to them?) that we don't yet think is worth the effort.

Why This Approach Works

No REST API: The plugin and controller don't need to know about each other's HTTP endpoints, health checks, or TLS config. They share nothing but a Redis instance that's already part of the infrastructure.

Fire-and-forget semantics: If Redis goes down, the plugin retries on the next queue tick. The controller doesn't track outstanding requests. Redis is the authoritative state store for the entire minigame system.

Clean language boundaries: Java handles what Java is good at, Go handles what Go is good at, and write access to Kube is kept out of Minecraft itself.

Ephemeral by design: When the game ends, the pod terminates itself. The controller reaps Failed and Succeeded pods automatically.

Come check it out

Join us in Minecraft and try the system out! The minigames are pretty fun, too.