Muse / WHAM: building a world model with creators

The problem

How do you build and evaluate a foundational generative AI model when its capabilities are emergent and its use cases are undefined?

This is the defining challenge of research product management in frontier AI research. You can’t write requirements for a capability nobody has seen yet — and you can’t wait for the model to be finished before asking who it’s for, because by then the training data and the evaluation metrics have already made that decision for you.

This case study shows how I answered it while building Muse, the first World and Human Action Model (WHAM), trained on seven years of human gameplay from Ninja Theory’s Bleeding Edge.

World and Human Action Models - Muse

Muse is a generative model of a video game. Given a sequence of gameplay footage and a set of controller actions, it generates what happens next — both the visuals and the actions themselves. It learned the dynamics of Bleeding Edge from over a billion frames of human play, paired with the controller inputs that produced them.

The clips below show gameplay sequences generated by Muse (based on WHAM-1.6B). Each was produced by prompting the model with 10 initial frames (equivalent of one second) of human gameplay, plus the controller actions for the whole sequence.

Figure 1. Example gameplay sequences generated by Muse (WHAM-1.6B). The game world, player behaviour and NPC interactions are all generated.

Process

Phase 1 — Establish the possibility space

Internal technical workshop. Before talking to any user, I ran an internal workshop to map the input data, the possible output formats, and the constraints researchers could foresee. This teases out the boundaries of innovation: what the model absolutely cannot do, and what it might be able to do.

Market analysis. The team had a hunch this model would be revolutionary for game development, so I analysed the game development landscape to identify its biggest challenges — signal on the most ambitious problem worth tackling.

Workshop board titled 'Current workflow of a level designer', mapping the six stages a designer moves through to build a Bleeding Edge level — develop the concept, create the walkthrough, gather visual references, greybox the scene, prototype the interaction, and polish — annotated with the artefacts and prototypes produced at each stage and a set of discussion questions about the model's possible inputs and outputs. — Figure 2. Mapping current practice — the stages, artefacts and open questions a level designer moves through, used to bound what the model would need to support.

Phase 2 — Communicate abstract concepts with concrete examples

Lo-fi design probes. I made envisionment videos depicting how a game creator might use these capabilities inside a real creation process. These were probes, not proposals — aimed at aligning visions and grounding an abstract capability in concrete examples.

Interactive prototype: Cygnus. In early 2023 the model was still early in development, so I worked with a research engineer to build a design probe called Cygnus — a simulated experience of creating a game level using potential AI capabilities, such as generating an environment from text or editing one through image examples. It was deliberately imaginative and provocative: not a roadmap, but an artefact designed to elicit reactions and let creators respond to something concrete rather than a description.

Figure 3. Cygnus — a design probe used to elicit responses from game creators, here generating an environment from text.

Phase 3 — Co-design with creators

I ran co-design workshops with 27 game creators from studios ranging from solo indie to AAA. Since most AI innovation is built in the Global North, we deliberately recruited creators from underrepresented backgrounds and geographies. Our goal was to create a technology that benefits everyone — not just those already in positions of privilege.

Eight workshops, each with a creative team of 3–4 people in complementary roles — designers alongside engineers, artists alongside producers — so we could see how these tools might change collaboration across disciplines, not just individual work. Each session was a simulated walk-through of Cygnus followed by discussion.

A diagram of the workshop protocol: a simulated walk-through of Cygnus followed by a structured group discussion. — Figure 4. Workshop protocol — a simulated walk-through of Cygnus, followed by discussion.

The eight teams: four indie studios (Belgium, USA, UK, Cameroon), one AAA studio (USA), an accessibility consultant (UK), the Xbox Accessibility Team (USA), and a creative studio for the blind (UK / India) — including three blind creatives.

What came back wasn’t a feature list. Creators described their practice as iterative prototyping and divergent thinking — quickly trying out an idea, tweaking it, and iterating as they go.

“As game developers, sometimes we want to evoke a certain feeling or a certain atmosphere, but we’re not even sure where to start. And that’s just been a hole that’s been very nicely filled with AI.” — P9, Level Designer, Indie Studio 3

“We’re not even conscious of what we want to create as game designers, but we just know something’s off. We tweak. Ok. Yeah, that feels good… But the details are what make really amazing game experiences.” — P3, Technical Artist, Indie Studio 1

Phase 4 — Translation and bridging disciplinary gaps

Turning creator insight into machine learning requirements, and model behaviour back into language the design team could reason about.

A finding like “players shouldn’t be able to walk through a wall” is not actionable to an ML researcher. It becomes actionable only when it’s restated as a property that can be measured, such as the generated worlds should maintain consistent physics. Additionally, measurement is what determines what the team optimises for.

Phase 5 — Build together to find out what you have

Informed by the creator insights and the technical groundwork, the machine learning research team and the human-centred AI research team came together to run a hackathon to build the WHAM Demonstrator — an interface for prompting the model, branching between continuations, and steering them with a game controller.

Figure 5. The WHAM Demonstrator — prompting, branching and steering generated sequences.

Building it together did four things months of discussion hadn’t. It crystallised the insights into a shared format — the first artefact both disciplines could read without me in the middle translating. It forced the trade-off conversation: with something concrete on screen, the team could argue about what a real product would take. It surfaced what needed optimising — nothing reveals generation speed as a constraint faster than a person sitting there waiting.

And it showed us what we actually had. With an emergent capability, nobody can fully answer what the model can do — not the designers, not the researchers who trained it. Offline metrics only measure what you already thought to ask. Every affordance was a new question, and one of them — what if you edit the frame and keep going? — found persistency: paste a character into a visual and the model adapts it into everything that follows. You only find that once the interface exists.

The Demonstrator shipped publicly alongside the paper, and became the team’s own working instrument for probing model behaviour.

What it produced

The workshops identified iterative prototyping and divergent thinking as the traits of creative practice the system had to support. Hands-on time in the Demonstrator showed us which of those the model could already deliver — and where it fell short. Read together, those translated into three machine learning requirements: the model’s outputs needed to be consistent, diverse and persistent.

Consistent — a generated sequence should be consistent with the established game dynamics and remain coherent throughout, with no sudden changes to characters or objects: characters should not pass through walls, and objects should not disappear without cause. Consistency is what lets a creator iterate and build on a sequence, which makes it the requirement underpinning iterative practice. We measured it with Fréchet Video Distance, using human gameplay as the ground truth.

Diverse — the model should generate material that reflects a range of different potential outcomes, because diverse options are what spark new ideas. This is the requirement behind divergent thinking. From the same starting frames the model should produce genuinely different futures — in one the player circles the spawn point, in another they head straight for a jump pad. We measured it as the distance between the distribution of the model’s actions and the actions real players took.

Persistent — persistency gives creatives control over what the model produces, and is what enables iterative tweaking. The model has to be flexible enough to accept a modification to the game state and assimilate it into what it generates next: insert an object, a character or a map element into a frame, and it should still be there in what follows.

Figure 6. What the model produced against the three requirements the co-design work identified: consistency, diversity and persistency.

Communication

Finally, publishing in Nature is not the same as being understood. I worked with the PR team to craft the PR strategy across Microsoft Research and Xbox, building a story that forefronts creators’ needs and highlights the co-design process. The narrative acknowledges the sensitivities around job displacement that AI could bring to the game development space, while highlighting new possibilities unlocked by AI game generation that could enable new forms of storytelling.

I also produced audiovisual assets to help communicate the capabilities in a straightforward way, helping the public understand the research. The published paper drew 136K+ views, with 55 outlets reporting on it, including Bloomberg and Reuters.

A collage of overlapping press clippings covering the Muse announcement: Eurogamer, Ars Technica ('Microsoft shows progress toward real-time AI-generated game worlds'), Bloomberg UK ('Microsoft Debuts Generative AI That…'), ABC Tecnología ('Microsoft presenta Muse, la primera IA capaz de desarrollar videojuegos'), and Sina Tech. — Figure 7. Selected coverage of the Muse announcement across games, technology and international press.

What I’d take to the next project

Make it concrete before it exists. Emergent capabilities can’t be specified, but they can be probed. A lo-fi walkthrough video, an interactive prototype, or even a sketched storyboard costs a day and changes what a research team argues about.
Constant translation aligns perspectives and consolidates expertise. To unleash the magic of an interdisciplinary team, keep translating concepts across disciplines — it brings everyone onto the same page instead of letting each side quietly assume.
Ground evaluation metrics in real-world use cases. If a user need doesn’t become something the team measures, it will quietly disappear during optimisation.
Build together to think together. A hackathon aligned disciplines faster than months of meetings, because a shared artefact forces disagreements into the open.
Who you recruit decides who the technology serves. That decision should be made early and intentionally. I always ask: who am I excluding, and on whose behalf am I making these assumptions?