1 What are HAMs?
Heterogeneous Agent Models drop the representative-agent fiction (that an entire economy can be summarized by a single representative agent) and instead model an economy as a population of agents who differ along some dimension - wealth, productivity, beliefs, information, or behavioral rule - and interact through prices or aggregate market-clearing conditions. Two traditions use the term:
Macro/finance HAMs (Hommes tradition). Boundedly rational agents use simple heuristics or rule-of-thumb strategies (e.g., fundamentalist vs. trend-following traders), switching between strategies based on past performance. These models are often low-dimensional and can be analyzed with nonlinear dynamics tools, but they reproduce stylized facts of financial data - excess volatility, clustered volatility, bubbles and crashes - that representative-agent, rational-expectations models struggle with.
Macro HAMs with aggregate risk (Krusell–Smith tradition). A continuum of rational, forward-looking agents face uninsurable idiosyncratic risk (e.g., income shocks) on top of aggregate shocks, and choose savings/consumption subject to a borrowing constraint. Because markets are incomplete, the wealth/asset distribution itself becomes a state variable of the aggregate economy - this is the source of the curse of dimensionality that motivates almost all of the numerical-methods literature below. This is the branch most relevant to our Nash-equilibrium/ mean-field-game numerics work, since the distribution as state variable + fixed-point over policy/price functions structure is essentially the same object we are computing in the groundwater market game.
The core numerical challenge in both traditions, but especially the second, is that the cross-sectional distribution of agents is itself infinite-dimensional, so an equilibrium is a fixed point in a space of functions (policy function, price function, law of motion for the distribution) rather than a finite vector - directly analogous to searching for a Nash equilibrium over strategy functions in a mean-field or N-player stochastic game.
2 Key foundational papers
[[krusell1998|Krusell and Smith (1998)]] - the canonical HA model with aggregate uncertainty and the original “Krusell-Smith algorithm”: approximate the infinite- dimensional distribution with a small number of moments (e.g., mean capital), solve individual policies by value function iteration conditional on a guessed law of motion for those moments, simulate, and update the law of motion until it’s consistent. The KS method is efficient for simple HA models but becomes less effective for complex HA models with multiple shocks or multiple endogenous states, since computational cost grows with the number of moments needed.
Hommes (2005/2006), “Heterogeneous Agent Models in Economics and Finance” — the standard survey of the behavioral/bounded-rationality HAM tradition. Surveys work on dynamic heterogeneous agent models where boundedly rational agents use heuristics or rule-of-thumb strategies, producing highly nonlinear dynamics — from a unique stable steady state to complex, chaotic behavior — that can explain stylized facts like excess volatility, bubbles, crashes, and volatility clustering. SSRN link · ScienceDirect chapter
Den Haan (2010), “Comparison of Solutions to the Incomplete Markets Model with Aggregate Uncertainty” — systematically compares numerical solutions to the Krusell-Smith model generated by different algorithms, finding the algorithms agree closely on correlations but diverge more on unconditional moments, tail behavior in bad times, and higher-order distributional moments. Good entry point before diving into specific methods. ScienceDirect
Algan, Allais, Den Haan & Rendahl, “Solving and Simulating Models with Heterogeneous Agents and Aggregate Uncertainty” (Handbook of Computational Economics ch.) — surveys the full menu of solution families: perturbation-only methods, projection methods, and hybrid approaches combining moment-based, value-iteration, and simulation techniques, using the Krusell-Smith model as the common testbed. ScienceDirect
Bayer, Broer, and coauthors and more recently Ma, Nolan (2025-ish EJ paper), “The Trouble with Rational Expectations in Heterogeneous Agent Models” — reviews existing numerical solution methods for HA models and situates the bounded-rationality tradition (Den Haan 1996, Krusell-Smith 1998) as closest in spirit to newer proposals, in the specific case where the “Krusell-Smith moments” coincide with prices themselves. Useful as a recent (2025) critical survey bridging both HAM traditions. Oxford Economic Journal
3 Non-RL numerical methods (the “classical” toolkit)
| Method family | Idea | Representative paper |
|---|---|---|
| Moment-based / Krusell-Smith | Approximate the distribution with a few moments; iterate policy ↔︎ law of motion | Krusell & Smith (1998) |
| Perturbation around steady state | Linearize (or higher-order expand) the full distribution’s response to aggregate shocks | Reiter (2009, 2010) |
| Sequence-space Jacobian | Compute the linear response (“Jacobian”) of aggregate sequences to shocks directly, without ever parameterizing the distribution’s law of motion; extremely fast, now the workhorse in HANK modeling | Auclert, Bardóczy, Rognlie & Straub (2021), Econometrica |
| Model reduction | Reduce the distribution’s dimensionality via basis functions / PCA before solving | Winberry (2018); Ahn, Kaplan, Moll, Winberry & Wolf (2018); Bayer & Luetticke (2020) |
| MIT shocks | Treat a one-time aggregate shock as a deterministic transition path (no need for a full law of motion) | Boppart, Krusell & Mitman (2018) |
The sequence-space Jacobian paper is worth reading closely for your purposes: it reports a large speed-up from a “fake news” algorithm relative to a direct algorithm for computing Jacobians across a Krusell-Smith model, a one-asset HANK model, and a two-asset HANK model, exploiting the fact that Jacobians of heterogeneous-agent problems become asymptotically time-invariant. That “compute a linear operator around the fixed point rather than solving the full nonlinear problem” trick has a flavor similar to Newton-based or stochastic-approximation methods for computing best-response fixed points in your fictitious-play setting. NBER working paper · Auclert’s PDF
4 Reinforcement learning & deep learning methods
This is the fastest-moving part of the literature and the most relevant to your best-response / fictitious-play work.
Jirnyi & Lepetyuk (2011) — an early and notable application of reinforcement learning to solving the Krusell-Smith (1998) model, predating the deep-learning wave; useful as a historical anchor. Referenced in Maliar, Maliar & Winant (2021), JME, “Deep learning for solving dynamic economic models”
Maliar, Maliar & Winant (2021), “Deep Learning for Solving Dynamic Economic Models,” Journal of Monetary Economics - demonstrates that deep neural networks can approximate policy and value functions in dynamic economic models with accuracy comparable to established projection methods, while scaling to problems with dozens of state variables. This is probably the single best starting point for deep-learning solution methods generally (not RL-specific, but closely related — it uses simulation-based training rather than grid-based DP). PDF
Han, Yang & E (2021) and related “Master equation” papers (Azinovic, Gaegauf & Scheidegger 2022 “Deep Equilibrium Nets”; Kahou, Fernández- Villaverde, Perla & Sood 2021; Duarte et al. 2024) — this strand tackles the mean-field “Master equation” directly using deep neural networks, learning value and policy functions of heterogeneous agents via simulation and exploiting agents’ structural knowledge of their own individual dynamics — a “structural RL” approach. This line is the closest existing analogue to mean-field-game numerics in the economics literature - worth reading in parallel with the MFG/fictitious-play papers you already have (Guéant, Lasry-Lions, Ruimeng Hu’s deep fictitious play).
Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics (Moll and coauthors, 2025) — differs from the Han et al.
- line by sidestepping the Master equation rather than taming its curse of dimensionality: policy functions are made to depend only on current prices (or a short price history), keeping the state space low-dimensional enough to use tabular, grid-based RL instead of deep networks. Implemented in JAX and demonstrated on a Huggett model with aggregate risk, plus a joint household/firm HANK application where the same policy-gradient algorithm handles both consumption-saving and forward-looking price-setting decisions. Directly useful if you’re weighing “full distributional state” vs. “reduced/low-dimensional state” tradeoffs in your own best-response algorithm design. arXiv HTML
“Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learning” (arXiv 2303.04833) — notes that deep RL has been used successfully for heterogeneous agent models with aggregate shocks and for discrete-continuous choice dynamic models, and has also proven effective for finding micro-founded general equilibria in many-agent macro models. The paper draws an explicit bridge to mean-field games: MFGs extend naturally to settings with infinitely many players via a mean-field term, and convergence of RL algorithms to MFG equilibria has been studied directly, including entropy-regularized MFGs on discrete action spaces with finite time horizons. This is probably the most directly relevant paper to your dissertation work of everything in this list — it’s explicitly at the RL/MFG/competitive-equilibrium intersection. arXiv PDF
DeepHAM (arXiv 2112.14377) — a global solution method combining generalized moments (replacing the full distribution) with a deep-learning solver for the reduced model; framed explicitly as an alternative to Krusell-Smith that avoids the degradation in accuracy KS suffers with multiple shocks or endogenous states. Good technical read for how to combine moment-reduction with neural approximation, which may generalize to your continuum-field groundwater model. arXiv PDF
Multi-agent deep RL for heterogeneous macro (2024–2025 wave) — a cluster of very recent papers applying MARL directly to N heterogeneous agents rather than a mean-field reduction: “Heterogeneous RBCs via Deep Multi-Agent Reinforcement Learning” (Gabriele, Glielmo & Taboga, 2025) and “Simulating the Economic Impact of Rationality through Reinforcement Learning and Agent-Based Modelling” (2024). These sit closer to your finite-player groundwater game setup than the mean-field papers above, since they keep a genuinely finite population of distinct agents rather than passing to the continuum limit. Related work in this space includes a heterogeneous agent model of mortgage servicing with income-based relief analysis and reinforcement learning for combining search methods in calibrating economic agent-based models. Heterogeneous RBCs, arXiv · Simulating Economic Impact, arXiv
“A Survey of Reinforcement Learning for Economics” (arXiv 2603.08956) — a broad, fairly recent survey with a dedicated macro/HA section: covers how heterogeneous agent models generate state spaces that grow with the number of agent types and asset positions, and reviews how deep RL/deep learning methods (Maliar et al. 2021; Fernández-Villaverde et al. 2023, who solve a model with financial frictions and an endogenous wealth distribution where perturbation methods fail due to strong nonlinearities) address this. Good as a map of the whole RL-for-econ landscape before you decide which specific method to adapt. arXiv PDF
5 How this connects to your work
The throughline across all of these papers is the same tension you’re navigating in the groundwater market game: an equilibrium concept defined over an infinite-dimensional object (distribution, or field of strategies) that must be reduced to something numerically tractable — either by (a) approximating the distribution with moments (Krusell-Smith, DeepHAM), (b) linearizing around a fixed point (sequence-space Jacobian), or (c) learning value/policy functions directly via simulation-based RL/deep learning while exploiting structural knowledge of individual dynamics (Han et al., Structural RL). The RL/MFG intersection papers (2303.04833, the Structural RL paper) are probably the two most worth reading closely first, since they’re explicitly working at the RL × mean-field-equilibrium boundary rather than treating RL as just another function approximator inside a classical algorithm.
6 Suggested reading order
- Hommes survey (context/history) → Krusell & Smith (1998) (canonical model)
- Den Haan (2010) (method comparison) → Auclert et al. (2021) sequence-space Jacobian (modern classical benchmark)
- Maliar, Maliar & Winant (2021) (deep learning bridge)
- Han et al. (2021)-style Master equation papers / Structural RL (2025)
- “Finding Regularized Competitive Equilibria… with RL” (2303.04833) — most directly tied to your MFG/RL interests