Vaneet Aggarwal

Reinforcement Learning (RL) serves as a versatile framework for sequential decision-making, finding applications across diverse domains such as robotics, autonomous driving, recommendation systems, supply chain optimization, biology, mechanics, and finance. The primary objective of these applications is to maximize the average reward. Real-world scenarios often necessitate adherence to specific constraints during the learning process. This monograph focuses on the exploration of various model-based and model-free approaches for Constrained RL within the context of average reward Markov Decision Processes (MDPs). The investigation commences with an examination of model-based strategies, delving into two foundational methods – optimism in the face of uncertainty and posterior sampling. Subsequently, the discussion transitions to parametrized model-free approaches, where the primal dual policy gradient-based algorithm is explored as a solution for constrained MDPs. The monograph provides regret guarantees and analyzes constraint violation for each of the discussed setups. For the above exploration, the authors assume the underlying MDP to be ergodic. Further, this monograph extends its discussion to encompass results tailored for weakly communicating MDPs, thereby broadening the scope of its findings and their relevance to a wider range of practical scenarios.

The advent of Big Data analytics and cloud computing has resulted in an unprecedented increase in the demand for distributed data storage demand. Companies are constantly on the lookout for ways of reducing this cost and improving reliability. Erasure coding has emerged as a promising technique to achieve these goals and most major tech companies have adopted it. However, one major issue in such systems is the characterization and optimization of access latency when data objects are erasure coded in distributed storage. In this monograph, the authors provide a review of recent theoretical and practical progress on systems that employ erasure codes for distributed storage. Starting with an overview the key challenges and research problems, the authors give an overview of different models and approaches that have been developed to quantify latency of erasure-coded storage. They also extend the discussions to video streaming from erasure-coded distributed storage systems. Practical implementations of erasure-coded storage are then discussed in real-world storage systems such as in content delivery and caching. This monograph is aimed at students, researchers and practitioners in information theory active in the research and development of modern day distributed storage systems.