ePrints@IIScePrints@IISc Home | About | Browse | Latest Additions | Advanced Search | Contact | Help

Parametrized actor-critic algorithms for finite-horizon MDPs

Abdulla, Mohammed Shahid and Bhatnagar, Shalabh (2007) Parametrized actor-critic algorithms for finite-horizon MDPs. In: American Control Conference 2007, JUL 09-13, 2007, New York,.

[img] PDF
yas.pdf - Published Version
Restricted to Registered users only

Download (272Kb) | Request a copy
Official URL: http://ieeexplore.ieee.org/search/srchabstract.jsp...

Abstract

Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FHMDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.

Item Type: Conference Paper
Additional Information: Copyright 2007 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Department/Centre: Division of Electrical Sciences > Computer Science & Automation (Formerly, School of Automation)
Date Deposited: 11 Jun 2010 05:40
Last Modified: 22 Feb 2012 06:51
URI: http://eprints.iisc.ernet.in/id/eprint/26811

Actions (login required)

View Item View Item