AMS 550: Stochastic Models

Includes Poisson processes, renewal theory, discrete-time and continuous-time Markov processes, Brownian motion, applications to queues, statistics, and other problems of engineering and social sciences.

Required Textbook: Introduction to Probability Models by Sheldon M. Ross, 12th edition, 2019, Elsevier

Supplementary Textbooks:

1. Stochastic Processes by Sheldon M. Ross, 2nd edition, 1995, Wiley
2. A First Course in Stochastic Processes by Samuel Karlin and Howard M. Taylor, 2nd edition, 1975, Academic Press

Learning Outcomes

1. Familiarity with the double expectation formula and demonstrate an ability to apply the formula in calculating expectations and probabilities involving two or more random variables.
2. Demonstrate an understanding of the concepts in Poisson processes:
   • Familiarity with the memoryless property of exponential random variables;
   • State the three alternative definitions of a Poisson process;
   • Understand and be able to prove/derive the various properties of Poisson processes;
   • Understand nonhomogeneous Poisson processes and compound Poisson processes;
   • Model simple real-life problems using the Poisson process and its generalizations.
3. Demonstrate an understanding of the concepts in Discrete Time Markov Chains:
   • Appreciate the range of applications of Markov chains;
   • Apply the Chapman-Kolmogorov equations to compute the marginal distributions of a Markov chain (Transient behavior);
   • Classify the states of a Markov chain;
   • Determine the transience and recurrence of a Markov chains;
   • Calculate steady state distributions for ergodic Markov chains;
   • Familiarity with the techniques used to study the first passage time and absorption probabilities.
4. Demonstrate an understanding of the concepts in Renewal Processes:
   • Characterize a renewal process;
   • Derive integral equations using the renewal argument;
   • Understand the limiting behavior of a renewal process;
   • Define a renewal reward process;
   • Familiarity with the elementary renewal theorem and the renewal reward theorem;
   • Knowledge of the key renewal theorem and an intuitive understanding of the inspection paradox;
   • Model simple real-life problems using the renew process and its generalizations such as the renewal reward process and the alternating renewal process.
5. Demonstrate an understanding of the concepts in Continuous Time Markov Chains:
   • Understand the sample path properties of a continuous time Markov chain;
   • Understand and be able to derive the properties of the transition matrix (forward and backward Kolmogorov equations);
   • Classify the states of a continuous time Markov chain;
   • Determine the transience and recurrence of a continuous time Markov chains;
   • Calculate the limiting distributions of ergodic continuous time Markov chains by solving the balance equations.
6. Demonstrate an understanding of the basic concepts in Queueing Theory:
   • Understand the properties of general queueing systems such as PASTA and Little’s law;
   • Apply continuous time Markov chain theory to study limiting properties of birth and death queues, e.g., stationary probabilities, averaging waiting time, expected number of customers in system;
   • Understand the basic concepts and properties of open and closed Jackson queueing networks and be able to solve simple problems on queueing networks;
   • Use the theory of discrete time Markov chain to analyze M/G/1 and G/M/1 queues.