In brief: Despite the controversy surrounding generative AI, some game designers use the technology to brainstorm concepts during development. Microsoft’s new open-source model aims to make this practice feasible for interactive content. It focuses on maintaining consistency across many frames while recognizing and integrating player input.
Microsoft researchers recently introduced Muse, a generative AI model designed to extrapolate interactive video game scenarios from images, clips, and recorded player input. The tool aims to streamline game development while upholding ethical training practices. The weights, sample data, and interactive interface for Muse, which Microsoft calls a “World and Human Action Model,” are now available on Azure AI Foundry. The developers discussed details of the technology on the company’s research blog and in a newly published study in Nature.
Following training on one million updates, Muse can faithfully predict up to half a minute of gameplay based on one second of actual footage and nine seconds of player input. Unlike prior models, where game details disappear or become distorted as soon as they leave the visible frame, Muse remembers details like terrain, characters, and game mechanics.
In a video interview with Xbox head Phil Spencer, Dom Matthews from Ninja Theory explained that the company doesn’t plan to use Muse to build content for end users. Instead, the studio plans to use it to help draft and iterate ideas by rapidly generating snapshots of a cohesive vision. It remains unclear whether other developers would follow a similar philosophy, but Microsoft won’t mandate the use of its models across its studios.
Some developers immediately expressed disinterest. David Goldfarb, who worked on Battlefield, Mass Effect, and Killzone, offered a brief, dismissive response. Marc Burrage from Creative Assembly told Wired that even restricting generative AI to the prototyping stage can harm game development by robbing employees of hands-on experience.
Click to enlarge
Spencer optimistically speculated that technologies like Muse might facilitate game preservation by helping developers port older titles to modern devices. Microsoft has frequently promised to preserve its customers’ digital libraries across hardware generations, but it remains to be seen whether regenerating them with AI can produce satisfactory results. Frank Cifaldi from the Video Game History Foundation isn’t as optimistic, comparing the idea to a photocopy of a painting.
Microsoft also faced harsh criticism regarding Muse’s training data. Much of the harsh controversy facing AI models stems from companies using material from across the internet without the original creators’ consent. Microsoft sidestepped the issue by training Muse on games it already owns.
The initial demonstration used player telemetry data from Ninja Theory’s multiplayer shooter, Bleeding Edge. Users who agreed to the game’s EULA have already consented to help train Muse. Theoretically, the company could extend the practice to popular titles like Minecraft, Call of Duty, Halo, Overwatch, StarCraft, Doom, or Forza.
