What is a POMDP?

📌 POMDP Components (Formula-Based)

A POMDP is defined by the following 7 elements:

$$ \text{POMDP} = \langle S, A, T, R, O, Z, \gamma \rangle $$

📌 1. belief state

Since the agent cannot directly know the current state, it maintains a probability distribution over states as follows:

$$ b(s) = P(\text{current state is } s) $$

This is called the belief state. In other words, it has the following probability distribution over the state space $S$:

$$ b: S \rightarrow [0, 1], \quad \sum_{s \in S} b(s) = 1 $$

📌 2. Belief Update Formula (Bayes Filter)

The formula for updating the belief after receiving observation $o$ following action $a$ is:

$$ b’(s’) = \eta \cdot Z(o \mid s’, a) \cdot \sum_{s \in S} T(s’ \mid s, a) \cdot b(s) $$

Here:

• $b(s)$: prior belief
• $T(s’ \mid s, a)$: state transition probability
• $Z(o \mid s’, a)$: observation probability
• $\eta$: normalization constant (so that the total sum is 1)

📌 3. Value Function

The value function based on the belief is defined as follows:

$$ V(b) = \max_{a \in A} \left[ \sum_{s \in S} b(s) R(s, a) + \gamma \sum_{o \in O} P(o \mid b, a) V(b_{a,o}) \right] $$

Here:

• $P(o \mid b, a)$: probability of observing $o$ after taking action $a$ in belief state $b$
• $b_{a,o}$: result of updating the belief after action $a$ and observation $o$
• $V(b_{a,o})$: value of the updated belief

📌 4. Calculating Observation Probabilities

$$ P(o \mid b, a) = \sum_{s’} Z(o \mid s’, a) \sum_{s} T(s’ \mid s, a) b(s) $$

🧠 Key Summary

❗ Challenges

• Belief space is a continuous, high-dimensional probability distribution → computational cost is very high
• Therefore, approximation algorithms (e.g., Particle Filter, Point-based Value Iteration) are often used