Source

Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree

When actions have uncertain outcomes, a rational agent combines two things:

  • a utility function: how good each outcome is
  • outcome probabilities: how likely each outcome is, given the action

Together these give the expected utility.

Expected Utility

Where:

  • is the action and the agent's evidence about the environment
  • is the th possible outcome of doing
  • is the utility of that outcome

Principle of maximum expected utility: choose the action with the highest .

Example: Robot

ActionOutcomesExpected utility
Turn righthits wall (, ); finds target (, )
Turn leftfalls in water (, ); finds target (, )

Choose turn right.

Agent Environment Interface

Agent and environment interact in discrete time steps :

  • the agent observes state
  • it produces action
  • it receives reward and moves to the next state

When the utility depends on a whole sequence of such decisions, the problem is a sequential decision problem, modelled as a Markov decision process.