🧮

Two theories of how people learn games competed for years: reinforcement (repeat what paid off) and belief learning (best-respond to what you have seen). It turned out they are the same model with a dial, and real people sit in between.

Learning and adaptation

Fictitious play was one learning rule; reinforcement learning is another. Experience-weighted attraction unifies them. Two parameters, one for memory and one for how much you learn from options you did not take, span the whole space, and fitting them to data lands squarely between the two classic extremes.

From earlier lessons

Two quick ones before we start.

Fictitious play has players best-respond to the opponent's...

Reinforcement learning updates the value of...

Predict, then play

A learner with two dials: memory (how much past experience persists) and imagination (how much it learns from payoffs it did not actually receive). Watch how fast it finds the better action.

Why did that happen?

Take a guess, then reveal the reason

Reinforcement updates only the action you actually chose, so you learn about the road not taken only by eventually trying it, which is slow. Belief learning imagines the payoff every action would have given and updates them all, so it corrects course quickly. EWA puts a dial (imagination) between these: turn it to zero for reinforcement, to one for belief learning. Fitting the dial to human data lands around a half, meaning people do weigh what they missed, but not as heavily as what they experienced.

The math

Definition, experience-weighted attraction

EWA maintains an attraction $A_j$ for each action, updated as $A_j(t) = \dfrac{\phi\,N(t-1)\,A_j(t-1) + [\delta + (1-\delta)\mathbf{1}\{j\text{ played}\}]\,\pi_j(t)}{N(t)}$ , with $N(t) = \rho\,N(t-1) + 1$ , and choice by logit over attractions. The dial $\delta$ is imagination: $\delta = 0$ updates only the played action (reinforcement), while $\delta = 1$ with $\phi = \rho$ updates every action by its foregone payoff (belief learning / fictitious play). The dial $\phi$ is memory.

Setting $\delta = 0$ recovers reinforcement learning: an unchosen action receives no payoff term at all, so you only ever learn about what you actually played. (Its attraction still drifts, because $N$ keeps growing and the attractions are experience-weighted averages, so an action you stop sampling fades rather than freezing. What it never gets is news about its own payoff.) Setting $\delta = 1$ with $\phi = \rho$ recovers belief learning: every action is credited with its counterfactual payoff, exactly fictitious play. Estimates from experiments put $\phi \approx 0.8$ to $1$ and $\delta \approx 0.5$ : people remember most of the past and weigh foregone payoffs about half as much as realized ones. One model, tuned by data, subsumes both traditions.

Work it out

Turn the imagination dial

Suppose action A pays $6$ and action B would have paid $4$ this round, and you played A. Under $\delta = 0$ (reinforcement), only A’s attraction takes in the payoff; B’s carries no new information and simply fades as unsampled experience ages, so you learn nothing about B unless you try it. Under $\delta = 1$ (belief learning), both attractions update: A by $6$ , B by $4$ , so you immediately compare the two and lean toward A.

In this fixed-payoff example, the reinforcement learner may take many rounds to discover B is worse because it only updates what it samples. The belief learner sees both counterfactual payoffs each round. In changing or strategic environments, higher $\delta$ need not always produce faster or better learning, so the graph demonstrates the mechanism rather than a general ranking.

Key idea

Experience-weighted attraction nests reinforcement and belief learning in one model with two dials, memory and imagination. Fit to data, people weigh foregone payoffs about half as much as realized ones.

Check yourself

Check yourselfapply

What does the imagination parameter delta = 0 correspond to?

Check yourselfanalyse

Why does a belief learner (high delta) adapt faster than a reinforcement learner?

Check yourselfbrings back: learning in games

How does EWA relate to the fictitious play from the learning lesson?

Sources

Camerer, C., and Ho, T.-H. (1999). Experience-weighted attraction learning in normal form games. Econometrica, 67(4), 827-874.