Bhatnagar, Shalabh and Panigrahi, Ranjan J (2006) Actor-critic algorithms for hierarchical Markov decision processes. In: Automatica, 42 (4). pp. 637-644.
Restricted to Registered users only
Download (256Kb) | Request a copy
We consider the problem of control of hierarchical Markov decision processes and develop a simulation based two-timescale actor-critic algorithm in a general framework. We also develop certain approximation algorithms that require less computation and satisfy a performance bound. One of the approximation algorithms is a three-timescale actor-critic algorithm while the other is a two-timescale algorithm, however, which operates in two separate stages. All our algorithms recursively update randomized policies using the simultaneous perturbation stochastic approximation (SPSA) methodology. We briefly present the convergence analysis of our algorithms. We then present numerical experiments on a problem of production planning in semiconductor fabs on which we compare the performance of all algorithms together with policy iteration. Algorithms based on certain Hadamard matrix based deterministic perturbations are found to show the best results.
|Item Type:||Journal Article|
|Additional Information:||This Copyright belongs to Elsevier.|
|Keywords:||Hierarchical decision making;Learning algorithms;Markov decision processes;Stochastic approximation;Optimal control|
|Department/Centre:||Division of Electrical Sciences > Computer Science & Automation (Formerly, School of Automation)|
|Date Deposited:||20 Apr 2006|
|Last Modified:||19 Sep 2010 04:25|
Actions (login required)