Recursions for the Moments of Some Continuous Distributions

This post follows on from my recent one, Recursions for the Moments of Some Discrete Distributions . I'm going to assume that you've read the previous post, so this one will be shorter. What I'll be discussing here are some useful recursion formulae for computing the moments of a number of continuous distributions that are widely used in econometrics. The coverage won't be exhaustive, by any means. I provide some motivation for looking at formulae such as these in the previous post, so I won't repeat it here. When we deal with the Normal distribution, below, we'll make explicit use of Stein's Lemma. Several of the other results are derived (behind the scenes) by using a very similar approach. So, let's begin by stating this Lemma. Stein's Lemma (Stein, 1973): "If X ~ N[ θ , σ 2 ], and if g (.) is a differentiable function such that E| g '( X )| is finite, then E[ g ( X )( X - θ )] = σ 2 E[ g '( X )]." It's worth noting that although this lemma relates to a single Normal random variable, in the bivariate Normal case the lemma generalizes to: "If X and Y follow a bivariate Normal distribution, and if g (.) is a differentiable function such that E| g '( Y )| is finite, then Cov.[ g ( Y ) , X ] = Cov.( X , Y ) E[ g '( Y )]." In this latter form, the lemma is useful in asset pricing models. There are extensions of Stein's Lemma to a broader class univariate and multivariate distributions. For example, see Alghalith (undated), and Landsman et al . (2013), and the references in those papers. Generally, if a distribution belongs to an exponential family , then recursions for its moments can be obtained quite easily. Now, let's get down to business............ Recall that t he r th. "raw moment" (or "moment about zero") for the random variable X , with distribution function F , is defined as μ r ' = E[ X r ] = ∫ x r dF ( x ) , for r = 1, 2, .....; and the "central (centered) moments" of X are defined as μ r = E[( X - μ 1 ' ) r ] , for r = 1, 2, 3, ....... Also, we can express one of these sets of moments in terms of the other set by using the two relationships: μ r = Σ{[ r ! / ( i ! ( r - i )!)] μ i ' (- μ 1 ' ) r-i } ,                                              (1) where the range of summation is from i = 0 to i = r ; and μ r ' = Σ{[ r ! / ( i ! ( r - i )!)] μ i ( μ 1 ' ) r-i } ,                                                (2) where, again, the range of summation is from i = 0 to i = r . Normal distribution If X ~ N[ θ , σ 2 ], then we know that μ 1 ' = E[ X ] = θ , and μ 2 ' = E[ X 2 ] = σ 2 + θ 2 . Also, note that we can write: μ 3 ' = E[ X 3 ] = E[ X 2 ( X - θ + θ )] = E[ X 2 ( X - θ )] + θ E[ X 2 ] . Applying Stein's Lemma, with g ( X ) = X 2 , we obtain: μ 3 ' = 2 σ 2 E[ X ] + θ E[ X 2 ] = θμ 2 ' + 2 σ 2 μ 1 ' = θ 3 + 3 θ σ 2 . Similarly, if we apply the Lemma with g ( X ) = X k ; k =3, 4, ......, we obtain: μ 4 ' = E[ X 4 ] = E[ X 3 ( X - θ )] + θ E[ X 3 ]  = θμ 3 ' + 3 σ 2 μ 2 ' = θ 4 + 6 θ 2 σ 2 + 3 σ 4 . μ 5 ' = E[ X 5 ] = E[ X 4 ( X - θ )] + θ E[ X 4 ] = θμ 4 ' + 4 σ 2 μ 3 ' = θ 5 + 10 θ 3 σ 2 + 15 θσ 4 ; and so on. So, each moment is obtained recursively from all of the lower-order moments, through the repeated use of Stein's Lemma. More particularly, you can see immediately that following recursion formula holds for the raw moments  of the Normal distribution (Bain, 1969; Willink, 2005): μ r +1 ' = θ μ r ' + rσ 2 μ r -1 '  ; r = 1, 2, 3, ......... Chi-square distribution We'll use the following standard result for the Chi-square distribution with " v " degrees of freedom: "For any real-valued function, h ,  E[ h ( χ 2 v )] = v E[ h ( χ 2 v +2 ) / χ 2 v +2 ], provided that the expectations exist." (This result can be generalized to the case of a non-central Chi-square distribution. See Appendix B of Judge and Bock (1978). Let h ( x ) = x k , for some integer, k . Then, applying the above result repeatedly, we get: If k = 1, then μ 1 ' = E[ h ( χ 2 v )] = v E[ χ 2 v +2 / χ 2 v +2 ] = v . If k = 2, then μ 2 ' = E[( χ 2 v )2] = v E[( χ 2 v +2 ) 2 / χ 2 v +2 ] = v E[ χ 2 v +2 ] = v ( v + 2). Immediately, we see that μ 2 = Var.[ χ 2 v ] = μ 2 ' - ( μ 1 ') 2 = 2 v . If k = 3, then μ 3 ' = E[( χ 2 v ) 3 ] = v E[( χ 2 v +2 ) 3 / χ 2 v +2 ] = v E[( χ 2 v +2 ) 2 ] = v ( v + 2)( v + 4) . I'll bet that you can see right away what the expressions are for μ 4 ', μ 5 ', etc .! In terms of a genuine recurrence relationship, we see from the above that we can write: μ r ' = μ r -1 ' ( v + 2( r -1))   ; r = 1, 2, 3, ........ Although we can then use equation (1) to obtain the central moments of X , there's also a separate recursion formula for these moments in the case of the Chi-square distribution This is discussed in the section on the Gamma distribution, below. Student-t distribution Suppose that X follows a Student-t distribution, with v degrees of freedom. Then the moments of X can be summarized as follows: μ r ' = 0                                              ; if r is odd , and 0 < r < v μ r ' = v ( r /2) Π[(2 i - 1) / ( v - 2 i )]       ; if r is even , and 0 < r < v (3) where the product is for i = 1 to i = ( r / 2). The inequality, v > r , ensures that the moment "exists" - that is, it is finite.  If v = 1, then X follows a Cauchy distribution, and none of its moments "exist". Equation (3) gives us a nice, tidy, and direct formula for obtaining any of the even-order moments of X . For instance, we see that: μ 2 ' = v [(2- 1) / ( v - 2)] = v / ( v - 2)  ;          if v > 2. Obviously, μ 2 = Var.[ X ] = μ 2 ', because E[ X ] = 0. Also, μ 4 ' = v 2 [1 / ( v - 2)][3 / ( v - 4)] = 3 v 2 / [( v - 2)( v - 4)]  ;          if v > 4. μ 6 ' = v 3 [1 / ( v - 2)][3 / ( v - 4)][5 / ( v - 6)] = 15 v 3 / [( v - 2)( v - 4)( v - 6)]  ;          if v > 6. etc. This set of results isn't in the form of a recurrence relationship, but we can obtain one very easily. Note that: μ 4 ' = [ v / ( v - 2)][3 v / ( v - 4)] = μ 2 ' [3 v / ( v - 4)]                 ; if v > 4. μ 6 ' = [ v / ( v - 2)][3 v / ( v - 4)][5 v / ( v - 6)] = μ 4 ' [5 v / ( v - 6)] ; if v > 6, and so on. Clearly, in general, the following recurrence formula holds - μ r ' = μ r-2 ' [( r - 1) / ( v - r )]      ;    if r is even , and 0 < r < v . Trust me - this formula is much more appealing than integrating functions of the density, or working with the characteristic function, for this particular distribution. Finally, note that because E[ X ] = μ 1 ' = 0, μ r = μ r ', for all (even) r . F distribution Suppose that X follows an F distribution, with numerator and denominator degrees of freedom, v 1 and v 2 respectively. The general formula for the raw moments of X is: μ r ' = ( v 2 / v 1 ) r Γ[( v 1 / 2) + r ] Γ ([ v 2 / 2) - r ] / (Γ [ v 1 / 2] Γ[ v 2 / 2]) .                    (4) So, μ 1 ' = ( v 2 / v 1 )Γ[( v 1 / 2) + 1] Γ ([( v 2 / 2) - 1] / (Γ [ v 1 / 2] Γ[ v 2 / 2] . Using the result that Γ[ x + 1] = x Γ[ x ], and cancelling terms, this simplifies to μ 1 ' = v 2 / ( v 2 - 2)  ;     if v 2 > 2 Similarly, μ 2 ' = ( v 2 / v 1 ) 2 Γ[( v 1 / 2) + 2] Γ ([( v 2 / 2) - 2] / (Γ [ v 1 / 2] Γ[ v 2 / 2]. Again, noting that Γ[( v 1 / 2) + 2] = (( v 1 / 2) + 1)Γ[( v 1 / 2) + 1]; that Γ[( v 2 / 2) - 2] = Γ[( v 2 / 2) - 1] / (( v 2 / 2) - 2); and then simplifying, we get: μ 2 ' = μ 1 ' ( v 2 / v 1 )[( v 1 + 2) / ( v 2 - 4)]   ;    if v 2 > 4 Proceeding in a similar manner, we find that: μ 3 ' = μ 2 ' ( v 2 / v 1 )[( v 1 + 4) / ( v 2 - 6)]   ;   if v 2 > 6 μ 4 ' = μ 1 ' ( v 2 / v 1 )[( v 1 + 6) / ( v 2 - 8)]   ;   if v 2 > 8 etc. So, in general you can see that we have the following recursion formula for the moments of X : μ r +1 ' = μ r ' ( v 2 / v 1 )[( v 1 + 2 r ) / ( v 2 - 2( r + 1))]   ; r = 0, 1, 2, .............; if v 2 > 2(r + 1) . This recurrence relationship avoids us having to deal with the gamma functions in (4), let alone having to perform any integration, or deal with this distribution's messy characteristic function. (As with the Student-t distribution, the moment generating function isn't defined here, because of the above conditions on the existence of the moments.) The distributions that we've considered so far are ones that you use every day in your econometrics work. Now let's consider a couple more distributions that arise a bit less frequently. Gamma distribution One situation where the Gamma distribution comes up in econometrics is in the context of "count data". This might seem a bit odd, because count data are non-negative integers , and we're talking about continuous random variables here. However, the Gamma distribution comes into play when we generalize the Poisson distribution to a particular form of the Negative Binomial distribution. Both of these distributions were discussed in my previous, related, post . Looking back at that post, you'll be able to see that the variance of a Poisson random variable equals its mean. We say that the distribution is "equi-dispersed". This is very restrictive, and isn't realistic with a lot of count data in practice. On the other hand, the variance of the Negative Binomial distribution exceeds its mean. We say that this distribution is "over-dispersed", and in practice this is often more reasonable. To construct the Negative Binomial distribution in the form that we usually use it in econometrics, we take a Poisson random variable and then add in an unobserved random effect to its (conditional) mean. If this random effect follows a Gamma distribution, we end up with the Negative Binomial distribution for the count data. (See Greene, 2012, pp.806-807 for details.) With this by way of motivation, consider a random variable, X that follows a Gamma distribution with a shape parameter ' a ', and a scale parameter ' b '. (Be careful here - there are two forms of the Gamma distribution. The other one has the shape parameter ' a ' and the rate parameter, θ = 1 / b .) Willink (2003) shows that the central moments ( moments about the mean ) for X can be obtained from the following recursion - μ r = ( r - 1)( bμ r -1 + ab 2 μ r -2 )     ; r = 2, 3, .......... There are two special cases of the Gamma distribution that we might note. First, if a = ( v / 2), and b = 2, then X follows a Chi-square distribution with v degrees of freedom. So, the central moments of the Chi-square distribution follow the recursion relationship: μ r = 2( r - 1)( μ r -1 + vμ r -2 )        ; r = 2, 3, .......... Second, if a = b = 1, then X follows an Exponential distribution, and its central moments satisfy: μ r = ( r - 1)( μ r -1 + μ r -2 ) ; r = 2, 3, ......... In addition, Withers (1992) shows that for the Exponential distribution we have the following simpler recursion: μ r = rμ r -1 + (-1) r ; r = 1, 2, 3, ....... Beta distribution One of the important features of the Beta distribution is that its density's support is the unit interval (although this can be generalized). It's one of the few distributions used by econometricians that has a finite support. This suggests that it might be useful when modelling data that are continuous in nature, and are in the form of proportions. Indeed, this is the case. I discuss this in the context of consumer demand analysis in this old post . More generally, the Beta regression model is discussed in detail by Ferrari and Cribari-Neto (2004) and Cribari-Neto and Zeileia (2010). The density function for X , that follows a Beta distribution is: f ( x | α , β ) = [Γ( α + β ) / (Γ( α )Γ( β ))] x α -1 (1 - x ) β -1 ; 0 < x < 1  ; α , β > 0 The general formula for the r th. raw moment of X is μ r ' = E[ X r ] = [ ( α + r - 1)( α + r - 2) ...... α ] / [( α + β + r - 1)( α + β + r - 2) .....( α + β )]  ; r = 1, 2, 3, ...... Immediately, we have: μ 1 ' = α / ( α + β ) μ 2 ' = [ α ( α + 1)] / [( α + β + 1)( α + β )] = [( α + 1) / ( α + β + 1)] μ 1 ' μ 3 ' = [ α ( α + 1)( α + 2)] / [( α + β + 2)( α + β + 1)( α + β )] = [( α + 2) / ( α + β + 2)] μ 2 ' etc . So, a general recursion formula for the raw moments of the Beta distribution is: μ r ' = [( α + r - 1) / ( α + β + r - 1)] μ r- 1 '   ; r = 1, 2, 3, ..... Numerical examples Remember, the whole point abut these recursion formulae is they help us to rapidly compute all of the moments of a distribution, up to some pre-specified maximum order, using one general formula. There is an R script file on the code page for this blog that illustrates this point, first for X ~ N[ θ , σ 2 ]; and second for X ~ F[ v 1 , v 2 ]. In the first case the first ten raw moments for X when θ = 1 and σ 2 = 4 are:    1,  5,  13,  73,  281,  1741,  8485,  57233,  328753,  2389141,  .... In the second case, the first ten raw moments for X when v 1 = 7 and v 2 = 24 are:  1.09,  1.68,  3.5265,  9.82,  36.09,  175.28,  1141.85,  10276.63,  135064.30,  2894235.00, .... References Alghalith, M. , undated. A note on generalizing and extending Stein's lemma. Bain, L. J ., 1969. Moments of non-central t and on-central F distribution. American Statistician , 23, 33-34. Cobb, L., P. Koppstein, & N. H. Chen , 1983. Estimation and moment recursion relations for multimodal distributions of the exponential family. Journal of the American Statistical Association , 78, 124-130. Cribari-Neto F.  & A. Zeileis , 2010. Beta regression in R. Journal of Statistical Software , 1–24. Ferrari, S. L. P. & F. Cribari-Neto , 2004. Beta regression for modelling rates and proportions. Journal of Applied Statistics , 31, 799–815. Greene, W. E ., 2012. Econometric Analysis , 7th. ed.. Prentice Hall, Upper Saddle River, NJ. J udge, G. G. & M. E. Bock , 1978. The Statistical Implications of Pre-Test and Stein-Rule Estimators in Econometrics . North-Holland, New York. Landsman, Z., S. Vanduffel, & J. Yao , 2013. A note on Stein's lemma for multivariate elliptical distributions. Journal of Statistical Planning and Inference , 11, 2016-2022. Stein, C. M., 1973. Estimation of the mean of a multivariate normal distribution. Proceedings of the Prague Symposium on Asymptotic Statistics , 345-381. Willink, R. , 2003. Relationships between central moments and cumulants, with formulae for the central moments of gamma distributions. Communications in Statistics - Theory and Methods , 32, 701-704. Willink, R. , 2005. Normal moments and Hermite polynomials. Statistics and Probability Letters , 73, 271-275. Withers, S. W. , 1992. A recurrence relation for the moments of the exponential. New Zealand Statistician , 27, 13-14. Woodland, A. D. , 1979. Stochastic specification and the estimation of share equations. Journal of Econometrics , 10, 361-383. © 2019, David E. Giles