Multi-Time-Scale Markov Decision Processes for Organizational Decision-Making

Transcription

1 Muli-Time-Scale Markov Decision Processes for Organizaional Decision-Making Chrisian Wernz Grado Deparmen of Indusrial and Sysems Engineering, Virginia Tech, Blacksburg, VA Phone: Fax: Absrac Decision-makers in organizaions and oher hierarchical sysems inerac wihin and across muliple organizaional levels and ake inerdependen acions over ime. The challenge is o idenify incenive mechanisms ha align agens ineress and o provide hese agens wih guidance for heir decision processes. To his end, we developed a muliscale decision-making model ha combines game heory wih muli-ime-scale Markov decision processes o model agens muli-level, muli-period ineracions. For a wo-level, muli-period, muli-agen problem, we derived closed-form soluions ha deermine opimal incenive levels, provide agens wih decision rules, and idenify informaion needs. We show he difference beween a single-imescale model, where agens make decisions wih he same frequency, and a muli-ime-scale model, where superior agens make decisions less frequenly han heir subordinaes. Our resuls show ha in he single-ime-scale case, agens can make opimal decisions wih informaion merely from he curren and subsequen period. In he muli-ime-scale case, daa from he enire decision horizon are necessary. This paper conribues o muliscale decision heory by generalizing soluions for muli-period ineracions ha were previously limied o a few ime periods. We apply our model o a managemen problem wih an accoun manager and muliple cusomer represenaives. Subjec classificaion: Muliscale decision heory, Markov decision processes, non-cooperaive game heory, managemen applicaion 1

2 1. Inroducion Decision-making over muliple periods is a challenging ask for agens. The decision process is furher complicaed when agens across muliple hierarchical levels affec each oher s decisions and oucomes, and when uncerainies need o be considered. We model and analyze he ineracions beween decision-making agens across wo hierarchical levels, where he higher-level agen relies on cooperaive decisions by muliple lowerlevel agens. A conflic of ineres exiss beween he wo levels as he preferred acions of he lower-level agens reduce he superior agen s likelihood o achieve is goal. To align ineress, he higher-level agen can offer a share of is reward as an incenive o he lower level agens. A sufficienly large incenive resuls in cooperaive behavior by he lower-level agens. In his paper, we deermine agens opimal decisions and incenives given heir organizaional inerdependencies and muli-period ineracions. Decisions by higher and lower-level agens occur on differen ime scales. The higher-level agen makes decisions on a slower ime scale, i.e., less frequenly, han lower-level agens. In oher words, he decisions of he higher-level agens are more sraegic, whereas he decisions by he lower-level agens are more operaional. To analyze his ype of ineracion and o deermine opimal decisions for agens, we combine a game-heoreic model wih muli-ime-scale Markov decision processes o accoun for he muli-period, muli-ime-scale inerdependencies. This paper conribues o muliscale decision heory by generalizing soluions for muli-period ineracions ha were previously limied o few periods. In general, analyical models have been limied o eiher muli-agen one-period problems, or singleagen muli-period problems. Muli-agen, muli-period problems have no ye been solved analyically. This paper develops a unified model for simulaneously solving hierarchical and emporal decision challenges. Lasly, his paper makes a conribuion o he lieraure on muli-ime scale Markov decision processes (e.g., Chang e al., 2003) by proposing a novel modeling approach and formulaion echnique. 2

3 The paper is organized as follows: Secion 2 discusses he relaed lieraure and Secion 3 inroduces he model. In Secion 4, he agen ineracion is analyzed, firs for a single-ime-scale siuaion and hen for he muli-ime-scale case. Secion 5 provides a numerical example of he model. Secion 6 presens he conclusions. 2. Lieraure Review We review he lieraure on hierarchical and organizaional decision-making ha has influenced he developmen of muliscale decision heory (MSDT). We discuss he saeof-he-ar mehods in MSDT and show heir relaionship wih he lieraure on muli-imescale modeling and muli-organizaional scale modeling. 2.1 Hierarchical and Organizaional Decision-Making The phenomenon of organizaions and heir hierarchical srucures was firs sudied by sociologiss and economiss. In his 1922 poshumously published work on Economy and Sociey, Max Weber views bureaucracies as hierarchical srucures composed of raional decision-makers (Weber 1978). Williamson s (1979) seminal work on ransacion-cos economics linked he exisence of organizaions and heir hierarchical srucures o ransacion coss associaed wih asse specificiy, uncerainy, and ransacion frequency. Transacion coss are he reasons why organizaions are formed as opposed o single consumer-producer ineracions in he marke place. Williamson (1967, 1970) had earlier developed a quaniaive model o deermine he opimal number of hierarchical levels in organizaions. Hierarchy wihin organizaions can be explained wih informaion processing limiaions and social preferences of humans. Radner (1992) views decenralizaion as a necessiy o overcome he limiaions of individuals in processing informaion, bu aribues he fac ha decenralizaion akes he form of hierarchy o sociological and psychological reasons raher han pure economic reasons. The concep of bounded raionaliy reconciled economiss raionaliy assumpion wih behavioral observaions of he limiaions of human decision-making. Bounded raionaliy is mos prominenly 3

4 associaed wih he work of Simon (1957, 1972), who received he Noble Memorial Prize in Economics for his work in his area. Earlier discussion of he bounded raionaliy concep can be raced back o Savage s (1954) Foundaions of Saisics. Raional decision-making of inerdependen individuals in organizaion can be formulaed mahemaically using cooperaive and non-cooperaive game heory. The heory of eams by Marschak (1955) and Marschak and Radner (1972) formulaes he problems of collaboraing individuals who have he same ineress and beliefs, bu do no share he same informaion. In conras o his cooperaive game heory approach, noncooperaive game heory is used o describe conflics of ineress beween individuals in organizaions. Personal and/or eam objecives can be in conflic wih ha of oher individuals, eams, or he higher-level objecives, paricularly in large organizaions. Incenives can be used o align conflicing ineress and moivae work ha suppors higher-level, and hus more imporan, organizaional objecives (Wernz 2008). This paper deermines effecive incenives for objecive alignmen, and our model has similariies o principal-agen models. Principal-agen heory describes a hierarchical ineracion beween wo decision-makers wih incomplee and asymmeric informaion. The superior decision-maker (principal) offers a conrac o he subordinae decisionmaker (agen). The conrac is designed o moivae he agen o perform work in he ineres of he principal (Milgrom and Robers 1992). Alhough a number of papers, e.g., Besanko (1985), Plambeck and Zenios (2000), discuss principal-agen models for muliperiods, also called dynamic principal agen models, no publicaion has ye discussed he muli-ime-scale aspec in which he principal makes decisions on a sraegic level, i.e. less frequenly han he subordinae agen. Organizaional agen ineracions have also been sudied in fields ouside of economics, paricularly in operaions research, managemen science, compuer science, and sysems engineering. In he following paragraphs, we presen conribuions ha have mos significanly influenced he developmen of muliscale decision heory. 4

5 A axonomy o classify and formally describe hierarchical agen ineracions was developed by Schneeweiss (1995, 2003a, 2003b). A unified noaion for various hierarchical agen ineracion siuaions was proposed. The auhor acknowledges he differen ime scales presen in organizaional sysems (sraegic, acical, operaional), bu he models only describe one-period ineracions. Mesarovic e al. (1970) developed a mahemaical framework o describe and analyze muli-level hierarchical sysems, bu again only for single period ineracions. A mahemaical heory of coordinaion was developed based on he concepual and formal aspecs of hierarchical sysems. From heir work, we adoped he ineracion relaionship beween lower and higher levels, in which he success of he supremal agen depends on he performance of he infimal agen. To accoun for uncerainy and emporal dynamics in sraegic agen ineracions, sochasic game heory was developed (Shapley 1953). Sochasic games are a combinaion of Markov decision processes (MDPs) (Puerman 1994) and classical game heory. Shapley (1953) was he firs o propose an algorihm ha solves sochasic games. Zachrisson (1964) coined he erm Markov games o emphasize he connecion o MDPs. For survey papers on sochasic games and soluion algorihms see Howard (1960), Pollaschek and Avi-Izhak (1969), Raghavan and Filar (1991), Merens (1992), Filar and Vrieze (1996), and Bowling and Veloso (2000). We are using an MDP formulaion in our model o accoun for he sochasic, muli-period agen ineracions. In he following secion, we furher discuss his opic wih an emphasis on muli-ime-scale aspecs. 2.2 Muli-Time-Scale Decision-Making Models Researchers have used various approaches o model muli-ime-scale phenomena via MDPs. One can disinguish beween wo classes of conribuions: (1) muli-ime-scale sysems ha are modeled wih MDPs and (2) one-ime-scale sysems wih imposed hierarchical, muli-ime-scale srucures. The laer is mosly used o increase he compuaional efficiency of soluion algorihms; e.g., Suon (1995), Hauskrech e al. 5

6 (1998), Parr (1998), and Suon e al. (1999). Our model falls ino class (1) of muli-imescale sysems, and he relaed lieraure is discussed in he following paragraphs. Chang e al. (2003) proposed muli-ime-scale Markov decision processes (MMDPs). The auhors model is an exension of Muppala e al. (1996) and Goseva- Popsojanova and Trivedi (2000), in which he lower level is modeled as a Markov chain and he upper level is described via a Markov reward process. MMDP has been applied o producion planning in semiconducor fabricaion (Panigrahi and Bhanagar 2004, Bhanagar and Panigrahi 2006), managemen of hydro power plans (Zhu e al. 2006), and has influenced work on arge racking in sensor neworks (Yeow e al. 2005, 2007) and reverse supply chains (Wonghasanekorn e al. 2010). Chang (2004) exended his earlier work o include game-heoreic ineracions beween agens wih conflicing ineress. In Chang s model, agens make heir decisions sequenially. In our model, we assume ha agens do no know he acions aken by he oher agens, which we recognize by applying a non-cooperaive game-heoreic model wih simulaneous decision-making. Jacobson e al. (2003) proposed periodically ime-inhomogeneous Markov decision processes (PTMDPs). The evoluion of he sysem is described by (N+1)- periodic sequences of reward funcions and ransiion probabiliies. The firs N epochs are fas-scale epochs, while he inerval N+1 is a slow-scale cycle. Their approach presens an alernaive o our approach of modeling muli-ime-scale evaluaion, ye wihou including sraegic agen ineracions. In heir book, Sehi and Zhang (1994) focused on hierarchical decision-making in sochasic manufacuring sysems. Differen hierarchical levels have differen ime scales associaed wih heir decisions. Hierarchy is discussed from a emporal perspecive, bu organizaional ineracions across hierarchies are no explicily modeled. Delebecque and Quadra (1981) analyzed a muli-ime scale phenomenon in hydroelecric power generaion wih singularly perurbed MDPs. In singularly perurbed MDPs, small perurbaions on fas ime scales affec he aggregae behavior observable on slower ime scales. Singularly perurbed MDPs have been exensively sudied in he 6

7 lieraure; see Yin and Zhang (1998, 2005) for an overview. In our model, fas scale decisions also affec higher-level phenomena, bu he link is hrough organizaional hierarchies, no perurbaions. 2.2 Muliscale Decision Theory The model ha describes he ype of hierarchical agen ineracion considered in his paper was inroduced by Wernz and Deshmukh (2007a,b). Rewards are passed down from a supremal agen o moivae cooperaive decisions by infimal agens. A cooperaive decision by an infimal agen improves he supremal agen s chances of success, i.e., increases he probabiliy of a ransiion o a sae associaed wih higher rewards. Wernz and Deshmukh (2010b) buil upon his iniial model in heir developmen of he muliscale decision-making framework. Their model was he firs comprehensive approach ha could derive closed-form, analyic soluions for one period, muliorganizaional scale ineracions in large, hierarchical organizaions. The muliscale decision-making framework has been applied o a wo-level managemen challenge (Wernz and Deshmukh 2007b), a wo-level producion planning problem (Wernz and Deshmukh 2007a), a hree-level service operaions applicaion (Wernz and Henry 2009), and a hree-sage supply chain problem (Henry and Wernz 2013). Furher deails on hese applicaions are presened a he end of he example problem in Secion 3.1. Wernz and Deshmukh (2012) exended a hree-level model from one o hree periods. The nex sep in he heory s developmen is o exend he muliscale decisionmaking model o accoun for ineracions over many periods. The firs effors in his direcion were documened in conference proceedings by Wernz and Deshmukh (2009, 2010a). This paper exends heir work by generalizing he model from wo agens o n+1 agens, by formulaing heorems and developing proofs, by applying he model o a challenge problem, by providing a numerical example, and by discussing he resuls and heir implicaions in a decision processes conex. 7

8 3. Model We moivae our model wih an example problem from he cusomer service indusry. The purpose of he example is o illusrae he applicabiliy of he muliscale decisionmaking model and o pu he mahemaical resul ino a decision-relevan conex. 3.1 Example Problem In a cusomer service division of an organizaion, an accoun manager is suppored by a number of cusomer represenaives. The manager is responsible for renewing cusomer conracs in regular ime inervals. The day-o-day cusomer care is handled by he represenaives, who answer cusomer phone calls o solve cusomer problems. When conacing cusomers for service conrac renewals, he manager decides beween wo approaches: (1) she calls he cusomer o discuss new conrac erms, or (2) sends a small gif along wih a leer wih deails of he new erms. Boh acions have he same coss for he organizaion. As a resul of he manager s acion, cusomers eiher renew or disconinue heir conracs. Boh oucomes are possible for a given acion, bu a call is more likely o resul in a renewal han when a leer wih a gif is sen. The represenaives, when receiving a cusomer s phone call wih a service reques, can also ake one of wo acions. To address a cusomer reques, represenaives can (1) be sric and insis on he conracual agreemen when idenifying resources o solve he cusomer s problem, or (2) be lenien and use addiional resources o ry o saisfy he cusomer. As a resul of he ineracion, he cusomer is eiher saisfied or dissaisfied. Saisfying he cusomer is associaed wih addiional cos o he organizaion. A lenien response o a cusomer reques mos likely resuls in a saisfied, high-cos cusomer, while a sric response mos likely resuls in an unsaisfied, low-cos cusomer. Sill, boh oucomes are possible for a given acion. The represenaive s pay depends on he cos he produces; lower coss are preferred and resul in a higher salary from he organizaion. Consequenly, he represenaive prefers o be sric. However, since cusomer saisfacion has an influence 8

9 on conrac renewal raes, he manager migh be willing o compensae he payoff difference for he represenaives. If i is financially advanageous for her, he manager will offer an incenive o he represenaives. We assume ha he incenive is a proporional share of he manager s pay. The higher he cusomer renewal rae, he higher he manager s pay, and he higher he incenive for he represenaives. A number of quesions arise for he manager and he represenaives: how high does he incenive for he represenaives have o be o swich from he iniially preferred acion of being sric o he cooperaive acion of being lenien? Wha is he maximum share of her pay he manager is willing o offer before he cos of incenives exceeds he benefis? Wha role do he ransiion probabiliies ha describe he link beween acions and oucomes play? How can he manager and he represenaives compue heir opimal decisions? Wha informaion do hey need o deermine an opimal decision? How do he emporal dynamics of represenaives ineracing muliple imes wih a cusomer before conrac renewal affec all of hese aspecs? The emporal dynamics will be modeled via Markov decision processes (MDPs). To accoun for he fac ha he manager only makes decisions from ime o ime, while he represenaives make heir decisions more frequenly, we model he manager s decision as ime-invarian unil he nex conrac renewal phase. This means ha he manager s decision (i.e., o call or send a leer) and he incenive level she offers o he represenaives does no change over ime. This difference in decision frequency accouns for he muli-ime-scale naure of he problem, and he fac ha he ime-scale is coupled o he organizaional scale. Figure 1 illusraes he model and he relaionship beween organizaional scale and ime scale. On he lef side of Figure 1, a dependency graph (Dolgov and Durfee 2004) depics he muual influence of he decision-makers. Corresponding o he organizaional scale, he righ side of Figure 1 shows he muli-period and muli-imescale aspecs across boh hierarchical levels. 9

10 organizaional scale Cusomer Rep 1 Accoun Manager Cusomer Rep x reward influence rans. prob. influence Cusomer Rep X ime scale one ime-invarian decision made a =1 x X 1 decision a every epoch ime Figure 1: Muliscale model for wo hierarchical levels and many periods The following secion will describe he example problem mahemaically and in general erms. The mahemaical model descripion is applicable o a wide range of hierarchical decision-making problems. The cusomer service example serves he purpose of illusraing resuls and providing he conex for he decision process sudied. As menioned in he lieraure review, similar examples wih applicaions in producion planning, mainenance service operaions, and supply chain managemen have been analyzed. In producion planning, a hierarchical chain consising of operaions manager, producion planner, and maerial handler have conflics of ineres relaed o profiabiliy, meeing producion deadlines, and invenory coss (Wernz and Deshmukh 2007a, 2010b). In a hree-level mainenance service scenario, an accoun manager uses incenives o align he ineress of her subordinae mainenance supervisor, who in urn seeks o align he acions of his mainenance workers (Wernz and Henry 2009, Wernz and Deshmukh 2012). Muliscale decision-making has also been used o coordinae he decisions beween organizaions. In a supply chain, a reailer, a wholesaler, and a producer can use incenives o coordinae heir chain-wide decisions (Henry and Wernz 2013). Furher applicaions in healh care and hospial managemen are currenly being explored (Sudhaakar and Wernz 2013). 3.2 Model Formulaion In he general model, he accoun manager will be referred o as he supremal agen, or agen, and he represenaives as infimal agens or agens INF1, INF2 ec., generally, wih x 1,..., X. 10

11 We formulae he muli-period model as a discree ime problem wih N epochs ha demarcae N 1 periods following he MDP noaion of Puerman (1994). Time begins wih decision epoch 1, he sar of period 1, and ends wih period N-1 followed by he final epoch N. A every decision epoch 1,..., N 1, agens carry ou acions a where, nx is he index ha denoes he chosen acion from agen s acion space. No acion is carried ou a he final epoch N. Agen decides in he firs decision epoch o ake an acion indexed by m for his and all fuure periods, i.e., a m, wih 1,..., N 1. By limiing agen o only one decision ha governs he enire ime horizon, we accoun for he differen decision frequencies beween agen and agens. In our example, he manager makes one decision (call cusomer or send leer), while he represenaives have muliple ineracions and make a new decision (sric or lenien) each ime. nx Associaed wih each agen are saes ȘUP si and k x, s for every period. Index i and index funcion kx ( ) denoe he agens possible saes given period. Depending on heir curren saes ȘUP s i and s k x,, and heir acions a m, and a n, x, agens move o saes s j, 1 and, 1 lx, 1 jx, nx, s wih probabiliy p l x sj, 1 si,, am, and p s s, a in he following period. The sae o which he agen ransiions deermines he agen s reward (referred o as pay in he example), which is noaed by j, 1 INF INF r s and r slx, respecively. This process repeas for every, 1 period 1,..., N 1. We analyze a siuaion in which each agen has a disinc se of wo acions and wo saes in each period. The acion spaces for agens and a a given decision epoch are denoed by : a1,, a2,, : a1,, a2, heir sae spaces by : 1,, s s2,, : 1,, s s2, wih 1,..., N 1 and wih 1,..., N. 11

12 In he conex of our example, acion a 1ȘUP corresponds o he manager s phone call, and a 2, o sending a leer o he cusomer. The corresponding oucomes are s 1ȘUP for conrac renewal and s 2, for no conrac renewal. For he cusomer represenaives, a corresponds o being lenien and a 2, o being sric. The oucomes are s 1, 1,, a saisfied, high-cos cusomer, and s 2, a dissaisfied, low-cos cusomer. The iniial rewards for agens and are :, r s 1, 1 1, r s :, (1) 2, 1 2, :, r s 1, 1 1, r s :. (2) 2, 1 2, The daa can alernaively be represened in vecor noaion: R : 1, 2,, R : 1, 2,. (3) The iniial sae-dependen ransiion probabiliies for agen are, :, p s s a 1, 1 i, 1, i.1, p s s, a : 1, (4) 2, 1 i, 1, i.1,, : 1, p s s a 1, 1 i, 2, i.2, p s s, a :, (5) 2, 1 i, 2, i.2, wih i 1, 2, m 1,2 and 0 im., 1. In marix noaion, we can compacly represen he daa of agen as 1 i.1, i.1, si, : 1i.2, i.2, P. (6) The rows of he marices correspond o he acions and he columns correspond o he saes o which he agen ransiions. For agens, he noaion can be represened accordingly, and in marix noaion he ransiion probabiliies are 1 jx.1, jx.1, s, : j x 1 jx.2, jx.2, P (7) wih index funcion jx 1, 2. 12

13 The hierarchical ineracions beween agen and agens consis of boom-up and a op-down influences. The boom-up influence is he effec agens have on agen s expeced oucome. The saes o which agens ransiion influence agen s chances of success, i.e., is ransiion probabiliy p. Muliscale decision heory models his influence using an addiive influence funcion f x,. The final ransiion probabiliy of agen is based on he iniial ransiion probabiliy and he influence funcions such ha X final, k,,,, 1 i, m, l x, 1 k, 1 i, m, x, k, 1 l, 1 1,..., X x1 p s s a s p s s a f s s (8) for indices,,, ' 1,2 INF ikm j x, and wih 1 INF 2 INFX s : s, s,..., s. l x, 1 l 1, 1 l 2, 1 l X, 1 1,..., X We choose he influence funcion o be a consan and define i as, if, cx k l x fx, sk, 1 s, 1: l x cx, if k l x, wih cx, 0. (9) Consan c x, is referred o as he change coefficien. We denoe he aggregae influence of all agens in period as C wih X C : f s s x, k, 1 lx, 1 x1. Since probabiliies can neiher be negaive nor exceed uniy, p final, 0 1 mus hold, which bounds he aggregae influence coefficien o 0C min,,1,1. (10) i.1, i.2, i.1, i.2, The meaning of he chosen change coefficien srucure in (9) is as follows: in each period, saes s 1, increase he probabiliy of sae s 1ȘUP, and reduce he probabiliy of sae s 2, accordingly. The effec on agen s ransiion probabiliies change in opposie direcion for agens s saes s 2,, i.e., sae s 2, becomes more likely and sae s 1, becomes less likely. This effec on ransiion probabiliies applies o siuaions where he saes of infimal agens lead o agen reaching a specific sae wih higher 13

14 or lower probabiliy. The aggregae change coefficien C describes he influence of all infimal agens combined. In he conex of our example, represenaives who can achieve high cusomer saisfacion conribue o a higher conrac renewal rae for he manager. The op-down influence of he hierarchical ineracion beween agen and agens is an incenive paymen by agen o all agens. Each agen receives a share b x, of agen s reward. We refer o b x, as he share coefficien. The final reward in period for an agen is is iniial reward plus he incenive, i.e., final, k,, : 1, 1, 1 x, k, 1, x, l x l x l x k, r s s r s b r s b. (11) Agen s iniial reward is reduced by he reward share given o agens. Agen s final reward is X rfinal, sk, 1 : 1 bx, k,. (12) x1 Figure 2 provides a graphical summary of he model. I shows he ineracion beween agen and an agen over wo periods. Saes Acions Transiions Agen s 1, s 2, a a 2, 1, p final, s1, 1 s2, 1 s1, 2 s2, 2 Agen Influence on rewards and ransiion probabiliies a1, s1, s1, 1 p s 2, a 2, s2, 1 bx, 1 f c x, 1 x, 1 s1, 2 s2, 2 Time periods +1 Figure 2: Schemaic model represenaion The following assumpions abou he model s parameers are made: 14

15 , 1, 2, (13) 1, 2, 1 im.,, (14) j x. n, 2, 1.1, 2.1, (15) 1.1, 2.1, Inequaliies in (13) express ha agen prefers sae s 1, 1 over s 2, 1 and agens iniially prefer s 2, 1 over 1, 1. This corresponds o he manager s s preference for conrac renewal and he represenaives goal of low coss. One can see ha a conflic of ineres exiss beween agen and agens. Agens iniially prefer o reach sae s 2, 1, which would reduce agen s chance of aaining is preferred sae s1, 1. Expression (14) saes ha an acion is linked o he sae wih he same index. In oher words, here is an index-corresponding acion for every sae, which is he mos likely consequence of he respecive acion. This resricion circumvens redundan cases in he analysis. Inequaliies in (15) express ha, given acion a 1,, he ransiion probabiliy 1.1, from sae s 1, o s 1, 1 is greaer han 2.1,, which denoes he probabiliy of ransiioning from sae s 2, o s 1, 1. Tha means i is more likely o remain in sae 1 han o swich from sae 2 o sae 1, given he corresponding acion a 1,. Applied o our example, his implies ha he probabiliy of conrac renewal is greaer han non-renewal when he cusomer has planned o renew and he manager makes he phone call. A he represenaive level, his means ha a saisfied cusomer will more likely remain saisfied as opposed o dissaisfied when he represenaive is lenien. Agens seek o maximize heir rewards. For each agen in period he expeced reward is: final, i, jx, m, nx, 1,..., X 1,..., X E r s, s, a, a : 15

16 k1 l 1 1 lx1 lx1 final, k, 1, 1 l x final, k, 1 i, m, lx, 1 1,..., X r s, s p s s, a, s p s s, 1, a l x j x, nx (16), x1,..., X The expeced reward for agen can be calculaed in he same way by replacing rfinal,, sk, s 1 lx wih r,, 1 final sk slx in (16).,, 1, 1 1,..., X Agens seek o maximize he sum of all r final, over he ime horizon, i.e., heir cumulaive rewards. The cumulaive rewards from period o period N 1 for agens and are N 1 final() i,,, :,, 1, j x final k lx, 1 r s s r s s, (17) N 1 final() i,,, :,, 1, j x final k lx, 1 1,..., X 1,..., X r s s r s s. (18) To calculae he expeced cumulaive reward, which agens need o calculae o deermine heir opimal course of acion, he backward inducion principle (Bellman 1957) is applied, saring in he las period of he ime horizon and working backwards o period 1. The expeced cumulaive reward from period o period N 1 is E r s, s, a, a : final i, jx, 1,..., X m, n x, 1,..., X INF final, i,,,,,, j x m nx, 1,..., X 1,..., X,, 1,, l x, 1 lx, 1 jx, nx, 1,..., X E r s s a a pfinal sk si, am, s p s s, a k1 l 1 1 lx1 lx1 x1,..., X final 1 k, 1 lx, 1 1,..., X E r s, s (19) for i, j, m, n 1,2, 1,..., N 1 and wih E r skn,, s 0. final N l x, N 1,..., X In he following analysis, we deermine opimal incenive levels and opimal acions for he agens. 16

17 4. Analysis We assume ha agens are risk-neural and raional, i.e., agens maximize heir expeced uiliies, or equivalenly, heir expeced rewards. Raional agens are able o calculae boh heir own expeced rewards, as well as he oher agens expeced rewards. They can decide which decisions yield he highes expeced rewards for hemselves. Hence, agens will engage in a game-heoreic reasoning process, recognizing he dependency of each oher s decisions. The Nash equilibrium concep is used o deermine which decisions he raional agens ake for a given incenive. The game occurs in wo sages. In he firs sage, agen deermines he incenive level, i.e., he share coefficien. In he second sage, agens ake he respecive acions in response o he incenive level chosen. To deermine an opimal acion, agens evaluae heir and he oher agens cumulaive expeced rewards of he enire ime horizon, i.e., E r and final 1 final 1 E r. The Nash equilibrium is he se of agens acions ha are he bes responses o oher agens preferred acions. Expressed mahemaically, his means ha he Nash equilibrium is a sraegy profile * * INF 1* INFX a a * m,, an(1),,..., an( X), for a given 1,..., N1 * b such ha * final 1 final 1 E r, a E r, a and (20) * final 1 final 1 E r, a E r, a x (21) for all a, where a refers o any sraegy profile differen from * a. I is easy o show ha agen has a dominan acion a 1,, i.e., a bes response regardless of oher agens acions. Since agen s preferred sae is s 1ȘUP, i will choose acion a 1, as i has he highes probabiliy of reaching sae s 1,. This resul holds for all levels of incenive. For a formal proof, see Wernz and Henry (2009). 17

18 Agen s chances of success, i.e., reaching saes s 1, agens swich from heir iniially preferred acions a 2, o a 1,, are improved when high level of incenives, agens will choose he cooperaive acions a 1,. For a sufficienly. Agen will offer his incenive if is expeced gain in reward is larger han he cos of he incenive. In he following secions, we will deermine he condiions for which a collaboraive Nash equilibrium exiss. Firs, we deermine he Nash equilibrium for a single-ime-scale model, where agens make decisions a he same frequency, before analyzing he muli-ime-scale case. The single-ime-scale model provides imporan resuls ha are he basis for muli-ime-scale analysis. In he muli-ime-scale model, hierarchical ineracions occur on muliple ime scales, i.e., agen and agens make decisions a differen frequencies. 4.1 Single Time-Scale Model In he single-ime-scale model, agen can vary is decision from period o period and can selec differen incenive levels (share coefficiens) in each period. In conras, agen can choose only one decision and one share coefficien for all periods in he muliime-scale. We begin he analysis by deermining he share coefficien b x, for which agens would swich from non-cooperaive o cooperaive acions. Using backward inducion, we sar in he final decision epoch, and hen move o earlier periods. Theorem 1: The share coefficiens bxn, 1 of he final period N 1 ha are cosminimal for agen and ha moivae agens o choose he cooperaive acions a 1, N 1 are b 2, N1 1, N1 xn, 1 2cxN, 1 1, N1 2, N1. (22) Proof: See Appendix. 18

19 The resul of Theorem 1 is surprising insofar as alhough all agens affec he expeced reward of agen, only daa from he respecive infimal agen plays a role in he opimal share coefficien. Furhermore, neiher he agens prior saes nor he ransiion probabiliies affec he resul. Transiion probabiliies and prior saes do no affec he share coefficien, because a he opimal incenive level b xn, 1 agens are indifferen beween heir acions (sill, weakly prefer he cooperaive acion) and he ransiion probabiliy variables cancel each oher ou. Nex, we will invesigae wheher agen is willing o give he share b of xn, 1 is reward o agens in reurn for heir cooperaion. The so-called paricipaion condiion checks if agen receives a higher expeced reward if an incenive according o b xn, 1 is offered o agens. Mahemaically, he paricipaion condiion is,, Er si,, s, a1,, a 1,, a x, 1,..., X final b j x n x 1,..., X 1,..., x1, x1,... X Erfinal () s,,,,,, 1,, x, 0, i s a a a b b j x 2, n x. (23) x, 1,..., x1, x1,..., X 1,..., X 1,..., x1, x1,... X Solving his inequaliy resuls in b xn, 1 2cxN, 1 1, N1 2, N1 1.1, N1 1.2, N1 1 c 2 1 INF 1, N1 2, N1 x, N1 1.1, N1 1.1, 2, N1. (24) * An incenive level ha induces cooperaion only exiss if b saisfies (24). In xn, 1 he final secion of his paper, we will illusrae his paricipaion condiion wih a numerical example. Nex, we deermine he share coefficiens for he earlier periods. Theorem 2: The cos-minimal share coefficiens b x, for 1,..., N 2 ha moivae agens o choose he cooperaive acion a 1, only depends on daa from he curren period and he nex period 1. The share coefficien is 19

20 b 1.1, 1 2.1, 1 2, 1 1, 1 c * 2, 1, x, 2cx, 1, 2, 2 x, 1 1, 2, for 1,..., N 2. (25) Proof: See Appendix. Theorem 2 shows ha agens have o look merely one period ahead o deermine heir opimal decisions. Surprisingly, periods furher ino he fuure do no affec he curren decisions. This fac can be explained by he anicipaed choices of opimal share coefficiens b x, in fuure periods. The opimal share coefficiens make agens indifferen beween boh of heir acions in period, i.e., he expeced rewards are he same. Consequenly, rewards of periods beyond he nex period do no play a role in he decision process a he curren epoch. For share coefficiens bx, bx, his is no he case, and daa from all fuure periods are decision-relevan. We will show his effec in he nex secion, where agen selecs one share coefficien value for he enire ime horizon. 4.2 Muli-Time-Scale Decision Making In he muli-ime-scale case, agen commis o one ype of acion and one share coefficien value ha applies o all periods. This case resembles a sraegic decision. Agen does no have he flexibiliy o change is iniial decision. Anoher way o inerpre his siuaion is ha agen makes one decision ha conrols he Markov chain over subsequen periods. In our example, he manager eiher calls or sends a leer, which affecs he final sae specifying conrac renewal or no renewal. The ime-invarian share coefficien is denoed by bx,1.. N 1 and he ime-invarian acion by am,1.. N 1 wih he subscrip indicaing ha hey apply o periods 1 hrough N 1. The muli-ime-scale aspec does no affec agen s preferred decision, which is sill acion wih index 1, i.e., a. 1,1.. N 1 To deermine he opimal share coefficiens, we firs calculae inermediae, auxiliary share coefficien values in each period. We inroduce an auxiliary share coefficien b x,[ ], which is he basis for selecing he opimal value b x,1.. N 1 for each agen 20

21 . Agen hen chooses he larges share b x,[ ] among he auxiliary share coefficiens over he ime horizon o ensure ha each agen chooses he cooperaive acion in every period, i.e., b max b. (26) x,1.. N1 1,..., N1 x,[ ] For he following heorem, we inroduce a shorhand for he reward and ransiion probabiliy differences, which simplifies he represenaion of resuls: 1, 2,, 2, 1,, (27) 1.1, 2.1,, 1.1, 2.1,. (28) Theorem 3: The auxiliary share coefficien b x,[ ], which is required in (26) o deermine he opimal share coefficien b x,1.. N 1, can be calculaed as follows: b 1 num x, x,[ ] for 1,..., N 1 2 denx, (29) where numeraor ( num x, ) and denominaor ( den x, ) are compued recursively as follows: num, (30) xn, 1 N1 num num for 1,..., N 2 ; (31) x, 1 1 and den c, (32) xn, 1 N1 xn, 1 den den c x, 1 x, 1 x, N N N1 1 N 1 1 denx, 1 cx, 1 1 for 1,..., N 2. (33) Proof: See Appendix. The resuls of Theorem 3 show ha in order o make decisions in he muli-imescale case, agens and need informaion from all fuure periods. In conras, for 21

22 he single-ime-scale case (Theorem 2), agens only needed informaion from he curren and nex period. Besides he effec of higher informaion needs, here is a second effec associaed wih he muli-ime-scale model: agen has o pay a larger share of is reward o incenivize agens. In he single-ime-scale case, agen had more flexibiliy o adjus incenives, which allowed agen o reain more reward for iself. In he final sep, we deermine if agen wans o offer he opimal share coefficiens, ha resul in cooperaive behavior by agens, or raher no b x N pay an incenive. No paying an incenive allows agen o reain all of is rewards, bu resuls in non-cooperaive behavior by agens. Agen commis o share coefficiens, according o (26) and Theorem 3, if is paricipaion condiion is b x N me. Agen decides on he level of incenive in he firs period and informs agens of is selecion of he share coefficiens. The paricipaion condiion can be deermined as a closed-form analyic resul similar o (24), bu he size of he equaion is oo large o presen effecively. Sill, he paricipaion condiion can be easily calculaed and evaluaed for specific daa using mahemaical sofware. The following secion provides a numerical example. 5. Numerical Example To provide furher insighs and apply he resuls of he model, we analyze a numerical example. The example illusraes he difference beween he single-ime-scale and he muli-ime-scale model. In addiion, we compare he expeced rewards of cooperaive infimal agens o he no cooperaion case, where agen does no offer incenives and agens choose non-cooperaive acions. The example is based on he following ime-invarian daa of a four-period ineracion ( N 5 ) beween agen and one infimal agen. R 60 5, INF1 1 R 3, 22

23 P y s y 1, for 1, 2,3, 4 y y, P s2, C c1, 0.15, , , y, INF1. We assume ha agens are in saes wih index 1 in he firs period, i.e., s 1,1 and s INF1 1,1. The following able shows he resuls of he analysis. final(1) Table 1: Numerical resuls of analysis Single-ime-scale Muli-ime-scale No cooperaion b1,1 9.70% b1,[1] 9.75% b1,1 0 b1,2 9.70% b1,[2] 9.89% b1,2 0 b1,3 9.70% b1,[3] 10.39% b1,3 0 b1, % b1,1..5 b1,[4] 12.12% b1,4 0 E r E r final(1) A number of observaions can be made based on hese resuls. Firs, agen is beer off choosing he opimal, non-zero share coefficiens in boh he muli-ime-scale and single-ime-scale case. These incenives induce cooperaive behavior by agen INF1 since is expeced reward is higher han if i did no cooperae. In oher words, he paricipaion condiion is me for he chosen daa. Furhermore, agen INF1 significanly benefis from he incenive and is cooperaive acions, as is cumulaive expeced reward abou riples from 9.63 o and 31.21, respecively. The second observaion is ha in he single-ime-scale case, he share coefficien values are idenical for periods 1, 2, and 3. The values are idenical since he daa of all hree periods is he same. The share coefficien in he fourh and las period is larger han he prior ones. This deviaion can be explained as follows: in earlier periods, agen INF1 benefied from ransiioning wih a higher probabiliy o is preferred sae. Thus, agen INF1 received a benefi beyond he curren period s reward, which allowed for a lower 23

24 share coefficien as compensaion. In he las period, his effec no longer exiss, hence he larger share coefficien of 12.12% compared o 9.70%. The hird observaion is ha in he muli-ime-scale model, he auxiliary share coefficien values differ in each period. The las period requires he larges share coefficien value of 12.12%, which is applied o all periods. Wih daa ha is differen for differen periods, any period, no jus he las, could have been he period wih he highes auxiliary share coefficien. Exending he example from one o muliple infimal agens yields similar resuls, since agens, heir acions, and incenives are independen from one anoher. However, he cumulaive influence C mus no violae condiion (10), which implies ha individual change coefficiens c x, will have o be smaller for a larger number of infimal agens. The consequence of smaller change coefficiens is ha incenives for agens need o increase o moivae cooperaion, while he upper limi agen is willing o pay for incenives (paricipaion condiion), goes down. Thus, he likelihood of cooperaion decreases if rewards remain he same. We conduced a sensiiviy analysis wih respec o he agens rewards. We found ha reward difference has o be muliple imes larger han o resul in siuaions in which agen benefis from paying he necessary incenives ha moivae cooperaive behavior from agens. This effec inensifies as he number of infimal agens increases. This means ha incenives will only be offered when he difference in payoffs beween preferred and no preferred oucomes for he superior is sufficienly larger han ha of heir subordinaes. For example, for 40 insead of 55 (wihou any oher changes) or for 5 insead of 2, agen prefers o no offer an incenive. Furhermore, infimal agens acions need o have a significan effec on agen via change coefficien C. In he conex of our example, his means ha he financial benefis of cusomer conrac renewal mus be larger han he cos of providing 24

25 cusomer care for he manager o offer incenives o her cusomer represenaives. Furhermore, cusomer care mus have a sufficienly srong influence on he chance of conrac renewal. Oherwise, he manager will prefer no o pay an incenive and no o moivae cooperaive behavior. For example, a reducion of he change coefficien o c1, 0.12 from 0.15 would resul in no incenive paymens. The advanage of analyical soluions for sensiiviy analysis is ha one can readily see from equaions which daa influence he resuls and how. By differeniaing he share coefficien equaions and paricipaion condiions wih respec o model parameers, one can analyically describe he effec of changes in daa. For an example of a comprehensive sensiiviy analysis of muliscale decision making models see Wernz and Deshmukh (2007a) and Henry and Wernz (2013). 6. Discussion and Conclusions We developed a muli-ime-scale decision-making model ha allows hierarchically ineracing agens o make opimal decisions over many periods. The supremal agen, agen, chooses one ype of decision and one level of incenive ha is fixed over he enire decision horizon. If he paricipaion condiion is me, i.e., if profiable, agen will offer a share of is pay (reward) o moivae cooperaive acions by agens. We deermined he opimal levels of incenives, which ake ino accoun muli-period agen inerdependencies. In he conex of he moivaional example, his means ha he accoun manager can deermine a cos-minimal incenive level ha moivaes cooperaive behavior by cusomer represenaives, which can resolve he represenaives conflics of ineres. Furhermore, we showed he difference beween a muli-ime-scale model and a single-ime scale model. In he single-ime scale case, agen (he manager) can change is decisions and level of incenives from period o period. The advanage of his flexibiliy is ha he opimal incenive levels and decisions can be deermined wih a subse of daa from he curren and nex period. In conras, in he muli-ime-scale, where 25

26 agen makes a sraegic decision ha applies o all periods, i needs informaion from all periods o find an opimal decision. In addiion, agen has o pay more, i.e., offer higher incenive levels, o obain cooperaive behavior from is infimal agens. The comparison of he single- and muli-ime-scale models shows he cos of sraegic managemen and he benefis of flexibiliy in erms of rewards and daa requiremens. The paper conribues o he unificaion of he emporal and organizaional scale in muli-agen, muli-period models. Furhermore, he paper provides an alernaive formulaion for muli-ime-scale MDPs. Mos resuls in his paper can be described analyically as opposed o being described only numerically. The advanage of analyic soluions is ha agens know which informaion hey need in order o make opimal decisions, he effec of parameers and he sensiiviy of resuls can be readily deermined, and opimal decisions can be calculaed wih lile compuaional effor. In he descripion of he example problem, we lised a number of quesions ha can now be answered. 1. How high does he incenive for he represenaives have o be o swich from he iniially preferred acion of being sric o he cooperaive acion of being lenien? Theorems 1-3 answer his quesion by providing equaions ha calculae he minimum level of incenive necessary o moivae cooperaive acion. In he numerical example, he incenive ha induces cooperaive acion from cusomer represenaives, expressed in erms of he share of he manager s reward, was 9.7% and 12.12%, respecively. 2. Wha is he maximum share of her pay he manager is willing o offer before he cos of he incenives exceeds he benefis? The paricipaion condiion formulaed in (23) answers his quesion. For he single-ime-scale case, equaion (24) is he analyic soluion for he maximum incenive level in he las decision epoch. For earlier periods, and for he muli-ime-scale case, resuls can be obained for specific daa, as shown in he numerical example. The sensiiviy analysis showed ha for changes in parameers, such as 40, 5, or c1, 0.12, he paricipaion condiion is no longer me. 26

27 Wih an increasing number of cusomer represenaives (agens ) ha each individually exer a smaller influence on cusomers decisions o renew heir conracs, he manager s propensiy o offer incenives goes down. The managerial implicaion of his insigh is ha incenives should be paid o small eams ha effecively suppor heir superiors work. Once he eam grows larger and he individual s conribuion becomes smaller, incenives are no longer cos-effecive for superiors. 3. Wha role do he ransiion probabiliies ha describe he link beween acions and oucomes play? How can he manager and he represenaives compue heir opimal decisions? Wha informaion do hey need o deermine an opimal decision? The ransiion probabiliies in he single-ime-scale case play only a limied role in deermining opimal incenives. For he incenives in he final period, ransiion probabiliies do no affec he resul (Theorem 1). For earlier periods, only he probabiliies a he manager level (agen ) are needed o deermine he opimal incenive levels (Theorem 2). This implies ha cusomer represenaives (agens ) can deermine heir opimal acions using parial or incomplee informaion. In paricular, he incenive paymens are independen of he acion-oucome uncerainies (ransiion probabiliies) a he cusomer represenaives level. In conras, he manager needs daa on all model parameers o deermine wheher she wans o pay he opimal levels of incenives. Furhermore, all daa is needed by all agens in he muli-ime-scale case. However, aggregaed daa in form of reward and probabiliy differences is sufficien; see (27)-(28). 4. How do he emporal dynamics of represenaives ineracing muliple imes wih a cusomer before conrac renewal affec all of hese aspecs? The difference in decision frequency is capured by he muli-ime-scale model. When compared o he single-ime-scale model, incenives for cusomer represenaives have o be larger in he muli-ime-scale case. This resuls primarily from he inflexibiliy of he manager o adjus he incenives from period o period. Consequenly, cusomer represenaives 27

28 receive higher incenives and he manager has o share a higher percenage of her reward wih her subordinaes. The muli-ime-scale model considered only one decision horizon a he level of agen (manager). An expansion o muliple decision horizons, i.e., a long-erm perspecive on ineracions of manager, represenaives and cusomers, is possible. Each decision horizon is coupled o he nex only by he facor of wheher a cusomer has renewed heir conrac or no. The analysis can hus be broken down ino individual decision horizons and merely he iniial saes would differ. To explore a long or infinie decision horizon, he limi behavior of model parameers on opimal share coefficiens would provide insighs. Wihou much analysis, once can see ha he effec of far-ou periods on curren periods is small, geing smaller wih more emporal disance. Fuure work should explore he effec of more hierarchical levels, more oucome saes, and more acions. Wernz and Deshmukh (2012) modeled a 3-level, 3-period model, which illusraes he complexiy of adding jus one level in a muli-ime-scale case. Sill, a muli-level expansion ha uses an algorihmic rule o capure all possible permuaions and auomaically analyzes all ineracion scenarios seems possible. Sudhaakar and Wernz (2013) explored he effec of muliple acions and oucomes for a muli-level, one-period model. They showed ha wih more acions and oucomes, he ransiion probabiliies, which previously were no necessary, are now needed o calculae he incenive levels. In a muli-period exension of his work, we expec o lose he abiliy o presen he resuls as analyic soluions, bu soluions based on daa using analyical mehods would sill be possible. 28

29 References Bellman RE (1957) Dynamic Programming. Princeon Universiy Press, Princenon, NJ Besanko D (1985) Muli-Period Conracs Beween Principal and Agen wih Adverse Selecion. Economics Leers 17 (1-2):33-37 Bhanagar S, Panigrahi JR (2006) Acor-Criic Algorihms for Hierarchical Markov Decision Processes. Auomaica 42 (4): Bowling M, Veloso M (2000) An Analysis of Sochasic Game Theory for Muliagen Reinforcemen Learning. School of Compuer Science, Carnegie Mellon Universiy, Chang HS (2004) A Model for Muli-Timescaled Sequenial Decision-Making Processes wih Adversary. Mah Compu Model Dyn Sys 10 (3-4): doi: / Chang HS, Fard PJ, Marcus SI, Shayman M (2003) Muliime Scale Markov Decision Processes. IEEE Transacions on Auomaic Conrol 48 (6): Delebecque F, Quadra JP (1981) Opimal Conrol of Markov Chains Admiing Srong and Weak Ineracions. Auomaica 17 (2): Dolgov D, Durfee E (2004) Graphical Models in Local, Asymmeric Muli-Agen Markov Decision Processes. Proceedings of he Third Inernaional Join Conference on Auonomous Agens and Muliagen Sysems-Volume 2: Filar JA, Vrieze K (1996) Compeiive Markov Decision Processes. Springer, New York Goseva-Popsojanova K, Trivedi KS (2000) Sochasic Modeling Formalisms for Dependabiliy, Performance and Performabiliy. In: Haring G, Lindemann C, Reiser M (eds) Performance Evaluaion - Origins and Direcions, vol Lecure Noes in Compuer Science. Springer, New York, pp Hauskrech M, Meuleau N, Kaelbling LP, Dean T, Bouilier C Hierarchical Soluion of Markov Decision Processes Using Macro-Acions. In: Proceedings of he Foureenh Conference on Uncerainy in Arificial Inelligence, Universiy of Wisconsin Business School, Madison, WI, July pp Henry A, Wernz C (2013) Revenue-Sharing in a Three-Sage Supply Chain wih Uncerainy: A Muliscale Decision Theory Approach. Under Review Howard RA (1960) Dynamic Programming and Markov Process. MIT Press, Cambridge, MA Jacobson M, Shimkin N, Shwarz A (2003) Markov Decision Processes wih Slow Scale Periodic Decisions. Mahemaics of Operaions Research 28 (4): Marschak J (1955) Elemens for a Theory of Teams. Managemen Science 1 (2): Marschak J, Radner R (1972) Economic Theory of Teams. Yale Universiy Press, New Haven Merens JF (1992) Sochasic Games. In: Aumann RJ, Har S (eds) Handbook of Game Theory wih Economic Applicaions. Norh-Holland, Amserdam, Mesarovic MD, Macko D, Takahara Y (1970) Theory of Hierarchical, Mulilevel, Sysems. Academic Press, New York Milgrom PR, Robers J (1992) Economics, Organizaions and Managemen. Prenice Hall, Englewood Cliffs, NJ Muppala JK, Malhora M, Trivedi KS (1996) Markov Dependabiliy Models of Complex Sysems: Analysis Techniques. In: Özekici S (ed) Reliabiliy and Mainenance of Complex Sysems. Springer, Berlin, Germany, pp Panigrahi JR, Bhanagar S Hierarchical Decision Making in Semiconducor Fabs Using Muli-Time Scale Markov Decision Processes. In: Proceedings of IEEE 29