Audio and Networking

The second video is up.

Until recently the engine was completely silent. Fires, rain and waterfalls all happened without a sound. And the whole thing ran in one process, so only one person could be in a world. Fixing both was this stretch of work, and the two ended up tied together: the audio is built on the same event stream the networking sends.

Sound that comes from the world

The easy way to do ambient sound is a looping track per area: a forest loop, a rain loop and so on. It sounds fine until the world changes. Burn down half the forest and the forest loop keeps playing.

Here the sound is driven by the simulation instead. Every frame the audio layer reads the state of the wind, rain, fire and water and synthesises from it. Wind comes from the wind strength and what it's blowing through, so grass and leaves sound different, and walking in under the trees changes it. Rain is worked out by what each drop lands on, so the same storm sounds different on the pond, on stone and on a roof. Fire comes from the burning cells themselves, which means a fire that grows gets louder and wider, and one that rain is putting out hisses. The waterfall and any water you pour come from the water simulation's flow.

Recordings are used only where synthesis isn't worth it: birdsong, the fox, the dog, and a quiet background bed for day and for night. There are about forty of them, all public domain, played with variation so repeats aren't obvious.

All of this runs on the player's machine and reads only what that player's copy of the world already knows. So a server never runs audio at all, and every client hears the world it actually has.

Knocks, thuds and breaks

Every material has a sound class, and anything that hits something (a footstep, a block landing, a dig) is synthesised rather than played back. The method is modal synthesis. A struck object rings at a handful of resonant frequencies, each dying away at its own rate. Model each one as a tiny resonator, hit them all at once with a short burst of noise, and you get something that sounds like a knock on that material. Bigger objects ring lower, so a large block sounds heavier than a small one with no extra data.

Each mode is a two-pole resonant filter, which costs two multiplies and two adds per sample:

// One resonant mode as a two-pole filter.
// Feed it a short burst of noise and it rings at freq, dying away over decaySeconds.
struct Mode
{
    float a1, a2, y1, y2;

    public Mode(float freq, float decaySeconds, float sampleRate)
    {
        float r = MathF.Exp(-1f / (decaySeconds * sampleRate)); // how much survives each sample
        a1 = 2f * r * MathF.Cos(2f * MathF.PI * freq / sampleRate);
        a2 = -r * r;
    }

    public float Tick(float input)
    {
        float y = input + a1 * y1 + a2 * y2;
        y2 = y1;
        y1 = y;
        return y;
    }
}

A knock is a few of these summed, with the frequencies, decays and loudness coming from the material's class.

The first listening pass turned up a problem: digging into stone went "ping". A resonator bank is right for a knock and wrong for something breaking. A fracture is mostly broadband noise rather than a few clean resonances, so stone, rock and glass now have a fracture sound: a sharp broadband crack, a few more as the fracture runs, gritty debris trickling after it and a low thud underneath, with the ringing turned right down. Glass keeps a bit more of it and shatters into shards that ring. Placing a stone block or walking on it still knocks.

Hearing the shape of the world

A voxel world can change shape at any moment, so nothing about the acoustics can be baked in ahead of time. Instead, the engine fires rays from the listener through a coarse copy of the world and works out the room from what they hit: how enclosed you are, how far a sound travels on average before it hits something, and how much of it escapes to open sky. From that it estimates how long the space should echo, separately for low, mid and high frequencies, and feeds a reverb built from a feedback delay network, a standard design where a few delay lines feed back into each other through a mixing matrix.

That gives a reverb time of about a quarter of a second under a tree and nearly two seconds deep in a dug tunnel.

My first version worked, but it had five separate propagation paths, each a special case. Point sounds got a ray or two to check for walls. Wind and rain got scaled by how open each direction was. The canopy had its own gain. The background beds went through a filter based on how outdoors you were. Scattered bird calls had their own rule again. They didn't agree with each other, and caves showed it: the rustle of leaves got quieter in a cave but never got muffled.

I replaced them with a single propagation model. Every source is now an emitter somewhere in the world: clusters of leaves at the height they actually are, patches of grass, the air above the ground, the surfaces rain is landing on, the waterfall, each fire, and a ring of sky emitters far out for the background. Each emitter reaches you in up to three ways:

• Straight through, losing energy for every metre of solid it crosses, more at high frequencies than low. Foliage lets most of it through.
• Through openings. The listener's rays already know which directions escape, so those get grouped into a few openings, like a cave mouth or a doorway. Sound goes from the emitter to the opening and turns the corner to you, and the sharper the turn, the more it loses, highs much more than lows. That's ordinary diffraction, and it's why you hear the bass of something round a corner before the detail.
• Into the reverb, in proportion to how much energy arrives, so rain falling outside a cave fills the cave with its echo.

The apparent direction of each sound is where its energy comes from, so standing in a tunnel you hear the outdoors through the mouth, and its direction shifts as you walk. Wall the mouth up and only what gets through the rock is left. Seal a small pit with one 5 cm block and the outdoors goes nearly silent. Carve the block away and it comes straight back.

All of this costs a couple of hundred rays per audio step, traced on the CPU across a few worker threads. GPU ray traversal would be faster, but it isn't guaranteed to give the same answer twice, and the sound has to be deterministic: the tests render audio offline and compare it, and every player's copy of the world has to hear the same thing from the same inputs.

Still open: there's no HRTF yet, so on headphones you can tell left from right but not front from back. And the occasional slow frame from the sound layer is over the budget I set for it, which I still need to fix.

Splitting the engine in two

Before any networking, the simulation had to be separated from the renderer. Until now the window owned the world: it ran the simulation, applied your edits and drew the result.

Now there's a simulation that owns the world, and everything else is a client of it. Commands go in: move, dig, place, pour water, light a fire, change the time. Three streams come out: a log of every edit, the events that happened (something caught fire, something landed), and the changing state of things like water, fire and animals. The window is just a client that sends commands and draws what comes back.

To prove the split, a second copy of the world is built from the same scene file and seed, following only those streams. After every step its world is compared with the original, every brick of it. If anything leaks around the streams, the two drift apart and the test fails at the step it happened.

Same maths on every machine

Multiplayer builds on that: the server doesn't send you the world. Your machine builds the same world from the same seed, and the server only sends what's changed. That only works if both machines generate exactly the same terrain, down to the last voxel.

Two Windows machines matched. Then I generated the same world on Linux and it came out slightly different. The terrain generator uses sin, cos and atan2 everywhere, and the built-in versions aren't guaranteed to give bit-identical results across platforms. On a test set of 64 inputs, sine disagreed on four and the float atan2 on fourteen. The differences were in the last bit, but run that through noise and a threshold and a voxel ends up solid on one machine and empty on the other.

So nothing that affects the simulation uses the platform's maths any more. I ported a set of well-known reference implementations of sin, cos, tan, their inverses, exp, log, pow and a few others to plain C#, with care over how the float versions round, and switched generation, creatures, fire and the audio's direction tables to them. They're several times slower per call than the built-ins, which made no measurable difference to how long the world takes to build. A Linux server and a Windows client now produce identical worlds.

Playing together

The server is the same engine with no window and no graphics card. It builds the world, runs the simulation 60 times a second and serves it over UDP.

To join, your machine builds the world from the seed, which is the slow part, and checks a hash of it against the server's. Then it takes a compressed snapshot of everything that's changed since the world began (a few tens of KB for the Court), which takes a second or two, and from then on applies the server's steps as they arrive.

Your own player is predicted. When you press forward, you move straight away, and the input goes to the server at the same time. When the server's answer comes back, your machine throws away the inputs the server has already used and replays the rest on top of the server's state:

// Each frame: move now, remember the input, send it.
void Tick(Input input)
{
    input.Sequence = ++_sequence;
    _state = Motion.Step(_state, input, _world);
    _pending.Enqueue(input);
    Send(input);
}

// When the server's state arrives, tagged with the last input it used:
void OnServerState(PlayerState server, uint lastInput)
{
    while (_pending.Count > 0 && _pending.Peek().Sequence <= lastInput)
        _pending.Dequeue();                 // already included in the server's state

    var state = server;
    foreach (var input in _pending)         // replay what the server hasn't seen yet
        state = Motion.Step(state, input, _world);

    _smoother.Correct(_state, state);       // ease out any difference over a few frames
    _state = state;
}

This is standard client-side prediction with server reconciliation. Corrections are only invisible if the client's movement code gives exactly the same answer as the server's. Mine does, bit for bit, including collision with the voxels. My first collision check read the world one voxel at a time and was over twenty times slower than it needed to be. Querying whole bricks brought a walking step down to tens of microseconds.

Edits aren't predicted. Every dig, block, pour and fire is a command the server runs, and you see it when the server says it happened. On a LAN that's instant. Over a slow connection it'll feel late. Showing an edit straight away and correcting it if the server disagrees is on the list for later.

Other players are drawn from a short buffer of their past positions, so they move smoothly between updates, with a name tag above their heads.

To test it without a room full of people, the engine can run a server and a few scripted bots in one process over real UDP, walking, building, lighting fires and pouring water. Every bot compares its whole world with the server's at every step it applies. Through a simulated link with 150 ms of latency and 5% packet loss, they all stay exact.

Sending less

The first version sent everyone everything, about 250 KB a second to a player doing nothing. Fine on a LAN, not fine for a hundred players.

The fix is interest management: each client is sent only what's near it. The world is split into cells about 13 metres across. You get the animals within 60 metres of you, and they stop being sent once they're past 64, so something walking along the edge doesn't flicker in and out. Water and fire within reach are sent as they change. A fire that changes far away is held back for you and caught up when you come back within range. Nearby players are updated 30 times a second and distant ones about five. Edits, events and chat still go to everyone.

What is sent is now delta-encoded against what the client already holds. Together that roughly halved it, to about 120 KB a second standing at the spawn.

That's still over the budget I set. Most of what's left is the waterfall, which sits about 14 metres from the spawn and changes dozens of times a second, and it's about half of what an idle player receives. Distance can't fix something that close. Sending its far side less often, or the animals less often and letting the client fill the gaps, would work, but both would be visible, so I haven't picked one yet.

How many players

To find out what a server can hold, the engine can run a swarm of lightweight bots that walk around and send moves without building a world of their own.

The budget for a step is about 17 ms. A hundred players walking around the glade fit with plenty of room, using about one core.

The limit is fire. Five forest fires with 25 people around them and the server can't keep up with 60 steps a second. Most of the time went into sending the fire's changes to every client separately. Encoding them once and sending the same bytes to everyone brought it down a lot, but not enough. A big fire with a crowd watching is the case I still don't handle.

The server's memory needed work too. The default garbage collector setup is tuned for throughput, and it paused for up to about 200 ms every so often. At 60 steps a second that's a dozen steps dropped at once. I measured five configurations, and the one I kept uses less memory and its longest pause is under 10 ms.

The server also saves the world now, every few minutes and when it's stopped. The whole glade with water flowing, about 60 MB in memory, compresses to around 230 KB on disk. And it runs on Linux, in a container, with a Windows client joining it and seeing exactly the same world.

That's audio and networking for now. More soon, here and on YouTube.