2020 · Conference paper
Learning to Simulate Dynamic Environments with GameGAN
GameGAN learns an action-conditioned neural simulator of a game from recorded frames and the actions taken, without access to the game’s logic or engine. An action-conditioned LSTM dynamics engine with a stochastic latent variable predicts the next state, an optional external memory module builds an internal map so that revisited locations stay consistent, and a rendering engine decodes frames while separating static from dynamic components. Training is end-to-end with adversarial losses, including an action-conditioned discriminator, and a temporal cycle loss for memory consistency. The authors evaluate on a modified Pac-Man, a ghost-free Pac-Man-Maze and VizDoom “take cover”, training RL agents inside the learned simulators and testing them in the real games, and measure long-term consistency with a come-back-home task; a human-playable model of the official Pac-Man is shown qualitatively.
Scope and limitations
GameGAN imitates a game’s pixels and dynamics; it does not recover game code, symbolic rules, assets or an executable. Training needs large logged datasets with actions: 40K Pac-Man episodes from a high-entropy DQN agent that sees the full environment, and 10K random-policy VizDoom episodes, at 84×84 and 64×64 resolution. Quantitative results cover small grid-based Pac-Man variants and one VizDoom scenario, with an external classifier supplying game outcomes. Agents trained in the simulators can exploit modeling errors: the full model sometimes let Pac-Man pass through walls, and its Pac-Man agents scored below those of the memoryless variant, while on VizDoom the World Model baseline scored higher than GameGAN. The released code supports VizDoom training only, relies on a deprecated WorldModels data setup, and is under the non-commercial NVIDIA Source Code License; Pac-Man data and the official Pac-Man model are not included.
Why it fits this collection
Adds a neural approach to reconstructing a playable game simulator from observed frames and actions, complementing the section’s program-synthesis and black-box game-replication research.
Publication and review details
- Venue
- CVPR 2020
- Status
- Conference paper
- Publication / posting date
- June 2020
- Date source
- IEEE’s Crossref deposit and an archived copy of the IEEE Xplore page give June 2020, represented at month precision. The May 2020 arXiv submission and the conference dates are not substituted.
- First arXiv submission
- May 25, 2020
- arXiv
- 2005.12126
- Reviewed arXiv version
- v1
- DOI
- 10.1109/CVPR42600.2020.00131
Crossref and the archived IEEE Xplore page give the five authors, CVPR 2020 and pages 1228–1237; IEEE metadata capitalizes the title as “Learning to Simulate Dynamic Environments With GameGAN”, while the CVF open-access paper and arXiv print “with”. The 10-page CVF PDF is the Computer Vision Foundation’s open-access version; IEEE Xplore holds the version of record. arXiv has only v1, submitted 2020-05-25 with the comment “CVPR 2020”. The repository README at the reviewed 2021-11-11 commit gives BibTeX pages 1231–1240, which differ from IEEE, and its Project Page link points to an unrelated lift-splat-shoot page. The README lists Ubuntu 18.04 and Python 3.8.3, and requirements pin torch 1.4.0 and torchvision 0.5.0. LICENSE section 3.3 limits use to non-commercial research or evaluation.
Sources reviewed
Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.
Primary sources
- https://ieeexplore.ieee.org/document/9156900
- https://web.archive.org/web/20240914161135/https://ieeexplore.ieee.org/document/9156900/
- https://api.crossref.org/works/10.1109/CVPR42600.2020.00131
- https://openaccess.thecvf.com/content_CVPR_2020/papers/Kim_Learning_to_Simulate_Dynamic_Environments_With_GameGAN_CVPR_2020_paper.pdf
- https://arxiv.org/abs/2005.12126
- readme.md GitHub · nv-tlabs/GameGAN_code · Revision 1195c39
- LICENSE GitHub · nv-tlabs/GameGAN_code · Revision 1195c39
- requirements.txt GitHub · nv-tlabs/GameGAN_code · Revision 1195c39