# Bayesian and Frequentist Regression Methods by Jon Wakefield

By Jon Wakefield

This booklet presents a balanced, glossy precis of Bayesian and frequentist tools for regression analysis.

Cover

Bayesian and Frequentist Regression Methods

ISBN 9781441909244 ISBN 9781441909251

Preface

Contents

Chapter 1 advent and Motivating Examples

1.1 Introduction
1.2 version Formulation
1.3 Motivating Examples
1.3.1 Prostate Cancer
1.3.4 Pharmacokinetic Data
1.3.5 Dental Growth
1.3.6 Spinal Bone Mineral Density
1.4 Nature of Randomness
1.5 Bayesian and Frequentist Inference
1.7 Bibliographic Notes

Part I

bankruptcy 2 Frequentist Inference
2.1 Introduction
2.2 Frequentist Criteria
2.3 Estimating Functions
2.4 Likelihood
o 2.4.1 greatest probability Estimation
o 2.4.2 variations on Likelihood
o 2.4.3 version Misspecification
2.5 Quasi-likelihood 2.5.1 greatest Quasi-likelihood Estimation
o 2.5.2 A extra complicated Mean-Variance Model
2.6 Sandwich Estimation
2.7 Bootstrap Methods
o 2.7.1 The Bootstrap for a Univariate Parameter
o 2.7.2 The Bootstrap for Regression
o 2.7.3 Sandwich Estimation and the Bootstrap
2.8 collection of Estimating Function
2.9 speculation Testing
o 2.9.1 Motivation
o 2.9.2 Preliminaries
o 2.9.3 ranking Tests
o 2.9.4 Wald Tests
o 2.9.5 probability Ratio Tests
o 2.9.6 Quasi-likelihood
o 2.9.7 comparability of try Statistics
2.10 Concluding Remarks
2.11 Bibliographic Notes
2.12 Exercises
bankruptcy three Bayesian Inference
3.1 Introduction
3.2 The Posterior Distribution and Its Summarization
3.3 Asymptotic homes of Bayesian Estimators
3.4 past Choice
o 3.4.1 Baseline Priors
o 3.4.2 sizeable Priors
o 3.4.3 Priors on significant Scales
o 3.4.4 Frequentist Considerations
3.5 version Misspecification
3.6 Bayesian version Averaging
3.7 Implementation
o 3.7.1 Conjugacy
o 3.7.2 Laplace Approximation
o 3.7.4 built-in Nested Laplace Approximations
o 3.7.5 significance Sampling Monte Carlo
o 3.7.6 Direct Sampling utilizing Conjugacy
o 3.7.7 Direct Sampling utilizing the Rejection Algorithm
3.8 Markov Chain Monte Carlo 3.8.1 Markov Chains for Exploring Posterior Distributions
o 3.8.2 The Metropolis-Hastings Algorithm
o 3.8.3 The city Algorithm
o 3.8.4 The Gibbs Sampler
o 3.8.5 Combining Markov Kernels: Hybrid Schemes
o 3.8.6 Implementation Details
o 3.8.7 Implementation Summary
3.9 Exchangeability
3.10 speculation checking out with Bayes Factors
3.11 Bayesian Inference according to a Sampling Distribution
3.12 Concluding Remarks
3.13 Bibliographic Notes
3.14 Exercises
bankruptcy four speculation trying out and Variable Selection
4.1 Introduction
4.2 Frequentist speculation Testing
o 4.2.1 Fisherian Approach
o 4.2.2 Neyman-Pearson Approach
o 4.2.3 Critique of the Fisherian Approach
o 4.2.4 Critique of the Neyman-Pearson Approach
4.3 Bayesian speculation trying out with Bayes components 4.3.1 evaluate of Approaches
o 4.3.2 Critique of the Bayes issue Approach
o 4.3.3 A Bayesian View of Frequentist speculation Testing
4.5 trying out a number of Hypotheses: normal Considerations
4.6 checking out a number of Hypotheses: fastened variety of Tests
o 4.6.1 Frequentist Analysis
o 4.6.2 Bayesian Analysis
4.7 checking out a number of Hypotheses: Variable Selection
4.8 techniques to Variable choice and Modeling
o 4.8.1 Stepwise Methods
o 4.8.2 All attainable Subsets
o 4.8.3 Bayesian version Averaging
o 4.8.4 Shrinkage Methods
4.9 version construction Uncertainty
4.10 a realistic Compromise to Variable Selection
4.12 Bibliographic Notes
4.13 Exercises

Part II

bankruptcy five Linear Models
5.1 Introduction
5.2 Motivating instance: Prostate Cancer
5.3 version Specifiation
5.4 A Justificatio for Linear Modeling
5.5 Parameter Interpretation
o 5.5.1 Causation as opposed to Association
o 5.5.2 a number of Parameters
o 5.5.3 facts Transformations
5.6 Frequentist Inference 5.6.1 Likelihood
o 5.6.2 Least Squares Estimation
o 5.6.3 The Gauss-Markov Theorem
o 5.6.4 Sandwich Estimation
5.7 Bayesian Inference
5.8 research of Variance
o 5.8.1 One-Way ANOVA
o 5.8.2 Crossed Designs
o 5.8.3 Nested Designs
o 5.8.4 Random and combined results Models
5.10 Robustness to Assumptions
o 5.10.1 Distribution of Errors
o 5.10.2 Nonconstant Variance
o 5.10.3 Correlated Errors
5.11 overview of Assumptions
o 5.11.1 assessment of Assumptions
o 5.11.2 Residuals and In uence
o 5.11.3 utilizing the Residuals
5.12 instance: Prostate Cancer
5.13 Concluding Remarks
5.14 Bibliographic Notes
5.15 Exercises
bankruptcy 6 normal Regression Models
6.1 Introduction
6.2 Motivating instance: Pharmacokinetics of Theophylline
6.3 Generalized Linear Models
6.4 Parameter Interpretation
6.5 probability Inference for GLMs 6.5.1 Estimation
o 6.5.2 Computation
o 6.5.3 speculation Testing
6.6 Quasi-likelihood Inference for GLMs
6.7 Sandwich Estimation for GLMs
6.8 Bayesian Inference for GLMs
o 6.8.1 previous Specification
o 6.8.2 Computation
o 6.8.3 speculation Testing
o 6.8.4 Overdispersed GLMs
6.9 overview of Assumptions for GLMs
6.10 Nonlinear Regression Models
6.11 Identifiabilit
6.12 chance Inference for Nonlinear types 6.12.1 Estimation
o 6.12.2 speculation Testing
6.13 Least Squares Inference
6.14 Sandwich Estimation for Nonlinear Models
6.15 The Geometry of Least Squares
6.16 Bayesian Inference for Nonlinear Models
o 6.16.1 previous Specification
o 6.16.2 Computation
o 6.16.3 speculation Testing
6.17 review of Assumptions for Nonlinear Models
6.18 Concluding Remarks
6.19 Bibliographic Notes
6.20 Exercises
bankruptcy 7 Binary facts Models
7.1 Introduction
7.2 Motivating Examples 7.2.1 final result After Head Injury
o 7.2.2 plane Fasteners
o 7.2.3 Bronchopulmonary Dysplasia
7.3 The Binomial Distribution 7.3.1 Genesis
o 7.3.2 infrequent Events
7.4 Generalized Linear types for Binary facts 7.4.1 Formulation
7.5 Overdispersion
7.6 Logistic Regression types 7.6.1 Parameter Interpretation
o 7.6.2 probability Inference for Logistic Regression Models
o 7.6.3 Quasi-likelihood Inference for Logistic Regression Models
o 7.6.4 Bayesian Inference for Logistic Regression Models
7.7 Conditional probability Inference
7.8 review of Assumptions
7.9 Bias, Variance, and Collapsibility
7.10 Case-Control Studies
o 7.10.1 The Epidemiological Context
o 7.10.2 Estimation for a Case-Control Study
o 7.10.3 Estimation for a Matched Case-Control Study
7.11 Concluding Remarks
7.12 Bibliographic Notes
7.13 Exercises

Part III

bankruptcy eight Linear Models
8.1 Introduction
8.2 Motivating instance: Dental development Curves
8.3 The Effciency of Longitudinal Designs
8.4 Linear combined types 8.4.1 the final Framework
o 8.4.2 Covariance types for Clustered Data
o 8.4.3 Parameter Interpretation for Linear combined Models
8.5 probability Inference for Linear combined Models
o 8.5.1 Inference for fastened Effects
o 8.5.2 Inference for Variance elements through greatest Likelihood
o 8.5.3 Inference for Variance elements through limited greatest Likelihood
o 8.5.4 Inference for Random Effects
8.6 Bayesian Inference for Linear combined types 8.6.1 A Three-Stage Hierarchical Model
o 8.6.2 Hyperpriors
o 8.6.3 Implementation
o 8.6.4 Extensions
8.7 Generalized Estimating Equations 8.7.1 Motivation
o 8.7.2 The GEE Algorithm
o 8.7.3 Estimation of Variance Parameters
8.8 overview of Assumptions 8.8.1 evaluate of Assumptions
o 8.8.2 techniques to Assessment
8.9 Cohort and Longitudinal Effects
8.10 Concluding Remarks
8.11 Bibliographic Notes
8.12 Exercises
bankruptcy nine common Regression Models
9.1 Introduction
9.2 Motivating Examples
o 9.2.1 birth control Data
o 9.2.2 Seizure Data
o 9.2.3 Pharmacokinetics of Theophylline
9.3 Generalized Linear combined Models
9.4 probability Inference for Generalized Linear combined Models
9.5 Conditional chance Inference for Generalized Linear combined Models
9.6 Bayesian Inference for Generalized Linear combined versions 9.6.1 version Formulation
o 9.6.2 Hyperpriors
9.7 Generalized Linear combined versions with Spatial Dependence 9.7.1 A Markov Random box Prior
o 9.7.2 Hyperpriors
9.8 Conjugate Random results Models
9.9 Generalized Estimating Equations for Generalized Linear Models
9.10 GEE2: hooked up Estimating Equations
9.11 Interpretation of Marginal and Conditional Regression Coeffiients
9.12 creation to Modeling established Binary Data
9.13 combined versions for Binary info 9.13.1 Generalized Linear combined versions for Binary Data
o 9.13.2 probability Inference for the Binary combined Model
o 9.13.3 Bayesian Inference for the Binary combined Model
o 9.13.4 Conditional chance Inference for Binary combined Models
9.14 Marginal versions for based Binary Data
o 9.14.1 Generalized Estimating Equations
o 9.14.2 Loglinear Models
o 9.14.3 extra Multivariate Binary Models
9.15 Nonlinear combined Models
9.16 Parameterization of the Nonlinear Model
9.17 chance Inference for the Nonlinear combined Model
9.18 Bayesian Inference for the Nonlinear combined Model
o 9.18.1 Hyperpriors
o 9.18.2 Inference for capabilities of Interest
9.19 Generalized Estimating Equations
9.20 evaluation of Assumptions for basic Regression Models
9.21 Concluding Remarks
9.22 Bibliographic Notes
9.23 Exercises

Part IV

bankruptcy 10 Preliminaries for Nonparametric Regression
10.1 Introduction
10.2 Motivating Examples
o 10.2.1 mild Detection and Ranging
o 10.2.2 Ethanol Data
10.3 The optimum Prediction
o 10.3.1 non-stop Responses
o 10.3.2 Discrete Responses with ok Categories
o 10.3.3 common Responses
o 10.3.4 In Practice
10.4 Measures of Predictive Accuracy
o 10.4.1 non-stop Responses
o 10.4.2 Discrete Responses with ok Categories
o 10.4.3 common Responses
10.5 a primary examine Shrinkage Methods
o 10.5.1 Ridge Regression
o 10.5.2 The Lasso
10.6 Smoothing Parameter Selection
o 10.6.1 Mallows CP
o 10.6.2 K-Fold Cross-Validation
o 10.6.3 Generalized Cross-Validation
o 10.6.4 AIC for normal Models
o 10.6.5 Cross-Validation for Generalized Linear Models
10.8 Bibliographic Notes
10.9 Exercises
bankruptcy eleven Spline and Kernel Methods
11.1 Introduction
11.2 Spline equipment 11.2.1 Piecewise Polynomials and Splines
o 11.2.2 usual Cubic Splines
o 11.2.3 Cubic Smoothing Splines
o 11.2.4 B-Splines
o 11.2.5 Penalized Regression Splines
o 11.2.6 a short Spline Summary
o 11.2.7 Inference for Linear Smoothers
o 11.2.8 Linear combined version Spline illustration: chance Inference
o 11.2.9 Linear combined version Spline illustration: Bayesian Inference
11.3 Kernel Methods
o 11.3.1 Kernels
o 11.3.2 Kernel Density Estimation
o 11.3.3 The Nadaraya-Watson Kernel Estimator
o 11.3.4 neighborhood Polynomial Regression
11.4 Variance Estimation
11.5 Spline and Kernel tools for Generalized Linear Models
o 11.5.1 Generalized Linear types with Penalized Regression Splines
o 11.5.2 A Generalized Linear combined version Spline Representation
o 11.5.3 Generalized Linear types with neighborhood Polynomials
11.7 Bibliographic Notes
11.8 Exercises
bankruptcy 12 Nonparametric Regression with a number of Predictors
12.1 Introduction
12.2 Generalized Additive types 12.2.1 version Formulation
o 12.2.2 Computation through Backfittin
12.3 Spline tools with a number of Predictors
o 12.3.1 ordinary skinny Plate Splines
o 12.3.2 skinny Plate Regression Splines
o 12.3.3 Tensor Product Splines
12.4 Kernel equipment with a number of Predictors
12.5 Smoothing Parameter Estimation 12.5.1 traditional Approaches
o 12.5.2 combined version Formulation
12.6 Varying-Coefficien Models
12.7 Regression bushes 12.7.1 Hierarchical Partitioning
o 12.7.2 a number of Adaptive Regression Splines
12.8 Classificatio
o 12.8.1 Logistic versions with ok Classes
o 12.8.2 Linear and Quadratic Discriminant Analysis
o 12.8.3 Kernel Density Estimation and Classificatio
o 12.8.4 Classificatio Trees
o 12.8.5 Bagging
o 12.8.6 Random Forests
12.10 Bibliographic Notes
12.11 Exercises

Part V

Appendix A Differentiation of Matrix Expressions
Appendix B Matrix Results
Appendix C a few Linear Algebra
Appendix D likelihood Distributions and producing Functions
Appendix E features of standard Random Variables
Appendix F a few effects from Classical Statistics
Appendix G simple huge pattern Theory

References

Index

Best probability books

Quality Control and Reliability, Volume 7

Hardbound. This quantity covers a space of records facing complicated difficulties within the creation of products and prone, upkeep and service, and administration and operations. the hole bankruptcy is through W. Edwards Deming, pioneer in statistical qc, who was once eager about the standard keep an eye on circulation in Japan and helped the rustic in its swift business improvement.

Aspects of multivariate statistical theory

A classical mathematical remedy of the innovations, distributions, and inferences according to the multivariate basic distribution. Introduces noncentral distribution idea, determination theoretic estimation of the parameters of a multivariate general distribution, and the makes use of of round and elliptical distributions in multivariate research.

Time Series Analysis, Fourth Edition

A modernized new version of 1 of the main relied on books on time sequence research. seeing that e-book of the 1st variation in 1970, Time sequence research has served as some of the most influential and well-known works at the topic. This new version keeps its balanced presentation of the instruments for modeling and interpreting time sequence and likewise introduces the most recent advancements that experience happened n the sector over the last decade via functions from parts comparable to enterprise, finance, and engineering.

Extra info for Bayesian and Frequentist Regression Methods

Sample text

Inferential recipes, such as specific estimators, are assessed with respect to their performance under repeated sampling of the data, with model parameters viewed as fixed, albeit unknown, constants. By contrast, in the Bayesian approach that is described in this book, probabilities are viewed as subjective and are interpreted conditional on the available information. As a consequence, assigned probabilities concerning the same event may differ between individuals. In this sense probabilities do not exist as they vary as a function of the available information.

1. Suppose that θn is a solution to the estimating equation Gn (θ) = 1 n n G(θ, Yi ) = 0, i=1 that is, Gn (θn ) = 0. 9) (asymptotic normality), where A = A(θ) = E ∂ G(θ, Y ) ∂θ T and B = B(θ) = E[G(θ, Y )G(θ, Y )T ] = var [G(θ, Y )] . Outline Derivation We refer the interested reader to van der Vaart (1998, Sect. 2) for a proof of consistency and present an outline derivation of asymptotic normality, based on van der Vaart (1998, Sect. 3). For simplicity we assume that θ is univariate. We expand Gn (θ) in a Taylor series around the true value θ: 0 = Gn (θn ) = Gn (θ) + (θn − θ) dGn dθ d2 Gn 1 + (θn − θ)2 2 dθ2 θ where θ is a point between θn and θ.

This development illustrates that an aim of regression modeling is often to “explain” the error terms using observed covariates. In general, error terms represent not only unmeasured variables but also data anomalies, such as inaccurate recording of responses and covariates, and model misspecification. Clearly the nature of the randomness, and the probabilities we attach to different events, is conditional upon the information that we have available and, specifically, the variables we measure. Similar considerations can be given to other types of random variables.