EIN 6935 Stochastic Decision Models II

Objective: Get exposed to the theory of stochastic dynamic programming and Markov decision processes and build foundations for their decision support applications.

Text:Dynamic probabilistic systems, R. Howard, 2007.

Topics

I. Review of Markov processes
(absolute & transition probabilities; irreducibility; steady-state probabilities; long-run expected average reward)

II. Principles of dynamic programming
(sequential decision making; forward & backward recursion; finite acyclic directed networks; principle of optimality; myopic policies; solving LP problems using DP; applications)

III. Stochastic dynamic programming
(finite-stage models; discounted dynamic programming; negative dynamic programming; positive dynamic programming; applications)

IV. Markov decision processes
(stationary policies; exhaustive enumeration; policy iteration methods w/ & w/out discounting; value iteration methods; LP solutions; applications)

V. Elements of risk & utility theory
(stochastic dominance of reward distributions; concept of utility; utility functions: properties and assessment; certainty equivalence & risk premium; risk attitudes; risk aversion & risk tolerance; common families of utility functions; decreasigly/increasingly/constant risk averse utility functions; multi-attribute utility)

VI. Approximate dynamic programming
(various topics; recent advances)