Overview
- a POMDP is the standard model of an embodied agent, its world and its task
- "partially observable" means the agent never sees the full state, only an observation such as one camera image
- it starts from the MDP loop (goal, action, state, reward, observation), then lists the parts of the POMDP tuple
- the agent's aim is the policy that earns the most discounted reward
A Partially Observable Markov Decision Process (POMDP) is the usual mathematical model of an embodied agent, its environment and its task. "Partially observable" means the agent never sees the full state of the world. It only gets an observation, such as one camera image.
Starting point: the MDP loop

A Markov Decision Process (MDP) describes a loop:
- the agent receives a goal
- it takes an action
- the environment moves to a new state and returns a reward
- the agent receives an observation , which is some function of the state, and the loop repeats
POMDP tuple
A POMDP is written as:
where:
- is the state space, every possible state of the world
- is the observation space, everything the agent can observe
- is the action space, every action the agent can take
- is the observation distribution, which says what the agent is likely to observe in state
- is the dynamics, which says how the state changes after an action
- is the reward function
- is the initial state distribution, where episodes start, which is a distribution because the starting state changes each episode
- is the discount factor, a number between 0 and 1 that makes later rewards count less
- is the finite horizon, the maximum number of steps in an episode
Goal of the agent
The agent looks for the best policy:
where:
- is a policy, the rule that chooses an action from what the agent has seen
- is the best policy
- is a trajectory, one full episode of states, actions and rewards produced by following
- is the reward at step , and is that reward after discounting
- the sum adds up all discounted rewards from step 0 to step , and averages this total over many episodes
Note
This model is the formal base for every task in Embodied AI.