is co-authored by Felipe Bandeira, Giselle Fretta, Thu Than, and Elbion Redenica. We additionally thank Prof. Carl Scheffler for his help.
Introduction
Parameter estimation has been for many years one of the necessary subjects in statistics. Whereas frequentist approaches, akin to Most Probability Estimations, was the gold customary, the advance of computation has opened house for Bayesian strategies. Estimating posterior distributions with Mcmc samplers grew to become more and more widespread, however dependable inferences depend upon a activity that’s removed from trivial: ensuring that the sampler — and the processes it executes beneath the hood — labored as anticipated. Preserving in thoughts what Lewis Caroll as soon as wrote: “For those who don’t know the place you’re going, any highway will take you there.”
This text is supposed to assist information scientists consider an usually missed facet of Bayesian parameter estimation: the reliability of the sampling course of. All through the sections, we mix easy analogies with technical rigor to make sure our explanations are accessible to information scientists with any degree of familiarity with Bayesian strategies. Though our implementations are in Python with PyMC, the ideas we cowl are helpful to anybody utilizing an MCMC algorithm, from Metropolis-Hastings to NUTS.
Key Ideas
No information scientist or statistician would disagree with the significance of strong parameter estimation strategies. Whether or not the target is to make inferences or conduct simulations, having the capability to mannequin the information technology course of is a vital a part of the method. For a very long time, the estimations have been primarily carried out utilizing frequentist instruments, akin to Most Probability Estimations (MLE) and even the well-known Least Squares optimization utilized in regressions. But, frequentist strategies have clear shortcomings, akin to the truth that they’re centered on level estimates and don’t incorporate prior data that would enhance estimates.
As a substitute for these instruments, Bayesian strategies have gained recognition over the previous a long time. They supply statisticians not solely with level estimates of the unknown parameter but in addition with confidence intervals for it, all of that are knowledgeable by the information and by the prior data researchers held. Initially, Bayesian parameter estimation was performed by means of an tailored model of Bayes’ theorem centered on unknown parameters (represented as θ) and recognized information factors (represented as x). We will outline P(θ|x), the posterior distribution of a parameter’s worth given the information, as:
[ P(theta|x) = fractheta) P(theta){P(x)} ]
On this formulation, P(x|θ) is the probability of the information given a parameter worth, P(θ) is the prior distribution over the parameter, and P(x) is the proof, which is computed by integrating all attainable values of the prior:
[ P(x) = int_theta P(x, theta) dtheta ]
In some circumstances, as a result of complexity of the calculations required, deriving the posterior distribution analytically was not attainable. Nonetheless, with the advance of computation, operating sampling algorithms (particularly MCMC ones) to estimate posterior distributions has turn into simpler, giving researchers a robust device for conditions the place analytical posteriors are usually not trivial to seek out. But, with such energy additionally comes a considerable amount of duty to make sure that outcomes make sense. That is the place sampler diagnostics are available in, providing a set of beneficial instruments to gauge 1) whether or not an MCMC algorithm is working properly and, consequently, 2) whether or not the estimated distribution we see is an correct illustration of the true posterior distribution. However how can we all know so?
How samplers work
Earlier than diving into the technicalities of diagnostics, we will cowl how the method of sampling a posterior (particularly with an MCMC sampler) works. In easy phrases, we are able to consider a posterior distribution as a geographical space we haven’t been to however have to know the topography of. How can we draw an correct map of the area?
Certainly one of our favourite analogies comes from Ben Gilbert. Suppose that the unknown area is definitely a home whose floorplan we want to map. For some cause, we can’t instantly go to the home, however we are able to ship bees inside with GPS gadgets hooked up to them. If every little thing works as anticipated, the bees will fly round the home, and utilizing their trajectories, we are able to estimate what the ground plan seems like. On this analogy, the ground plan is the posterior distribution, and the sampler is the group of bees flying round the home.
The explanation we’re writing this text is that, in some circumstances, the bees received’t fly as anticipated. In the event that they get caught in a sure room for some cause (as a result of somebody dropped sugar on the ground, for instance), the information they return received’t be consultant of the complete home; moderately than visiting all rooms, the bees solely visited a couple of, and our image of what the home seems like will in the end be incomplete. Equally, when a sampler doesn’t work accurately, our estimation of the posterior distribution can be incomplete, and any inference we draw based mostly on it’s more likely to be improper.
Monte Carlo Markov Chain (MCMC)
In technical phrases, we name an MCMC course of any algorithm that undergoes transitions from one state to a different with sure properties. Markov Chain refers to the truth that the subsequent state solely is dependent upon the present one (or that the bee’s subsequent location is simply influenced by its present place, and never by all the locations the place it has been earlier than). Monte Carlo implies that the subsequent state is chosen randomly. MCMC strategies like Metropolis-Hastings, Gibbs sampling, Hamiltonian Monte Carlo (HMC), and No-U-Flip Sampler (NUTS) all function by setting up Markov Chains (a sequence of steps) which are near random and regularly discover the posterior distribution.
Now that you just perceive how a sampler works, let’s dive right into a sensible situation to assist us discover sampling issues.
Case Examine
Think about that, in a faraway nation, a governor needs to know extra about public annual spending on healthcare by mayors of cities with lower than 1 million inhabitants. Relatively than sheer frequencies, he needs to know the underlying distribution explaining expenditure, and a pattern of spending information is about to reach. The issue is that two of the economists concerned within the mission disagree about how the mannequin ought to look.
Mannequin 1
The primary economist believes that every one cities spend equally, with some variation round a sure imply. As such, he creates a easy mannequin. Though the specifics of how the economist selected his priors are irrelevant to us, we do have to take into account that he’s making an attempt to approximate a Regular (unimodal) distribution.
[
x_i sim text{Normal}(mu, sigma^2) text{ i.i.d. for all } i
mu sim text{Normal}(10, 2)
sigma^2 sim text{Uniform}(0,5)
]
Mannequin 2
The second economist disagrees, arguing that spending is extra complicated than his colleague believes. He believes that, given ideological variations and funds constraints, there are two sorts of cities: those that do their greatest to spend little or no and those that aren’t afraid of spending lots. As such, he creates a barely extra complicated mannequin, utilizing a mix of normals to mirror his perception that the true distribution is bimodal.
[
x_i sim text{Normal-Mixture}([omega, 1-omega], [m_1, m_2], [s_1^2, s_2^2]) textual content{ i.i.d. for all } i
m_j sim textual content{Regular}(2.3, 0.5^2) textual content{ for } j = 1,2
s_j^2 sim textual content{Inverse-Gamma}(1,1) textual content{ for } j=1,2
omega sim textual content{Beta}(1,1)
]
After the information arrives, every economist runs an MCMC algorithm to estimate their desired posteriors, which can be a mirrored image of actuality (1) if their assumptions are true and (2) if the sampler labored accurately. The primary if, a dialogue about assumptions, shall be left to the economists. Nonetheless, how can they know whether or not the second if holds? In different phrases, how can they make sure that the sampler labored accurately and, as a consequence, their posterior estimations are unbiased?
Sampler Diagnostics
To guage a sampler’s efficiency, we are able to discover a small set of metrics that mirror completely different components of the estimation course of.
Quantitative Metrics
R-hat (Potential Scale Discount Issue)
In easy phrases, R-hat evaluates whether or not bees that began at completely different locations have all explored the identical rooms on the finish of the day. To estimate the posterior, an MCMC algorithm makes use of a number of chains (or bees) that begin at random places. R-hat is the metric we use to evaluate the convergence of the chains. It measures whether or not a number of MCMC chains have combined properly (i.e., if they’ve sampled the identical topography) by evaluating the variance of samples inside every chain to the variance of the pattern means throughout chains. Intuitively, which means that
[
hat{R} = sqrt{frac{text{Variance Between Chains}}{text{Variance Within Chains}}}
]
If R-hat is near 1.0 (or beneath 1.01), it implies that the variance inside every chain is similar to the variance between chains, suggesting that they’ve converged to the identical distribution. In different phrases, the chains are behaving equally and are additionally indistinguishable from each other. That is exactly what we see after sampling the posterior of the primary mannequin, proven within the final column of the desk beneath:
The r-hat from the second mannequin, nevertheless, tells a distinct story. The actual fact we have now such giant r-hat values signifies that, on the finish of the sampling course of, the completely different chains had not converged but. In observe, which means that the distribution they explored and returned was completely different, or that every bee created a map of a distinct room of the home. This basically leaves us and not using a clue of how the items join or what the entire flooring plan seems like.
Given our R-hat readouts have been giant, we all know one thing went improper with the sampling course of within the second mannequin. Nonetheless, even when the R-hat had turned out inside acceptable ranges, this doesn’t give us certainty that the sampling course of labored. R-hat is only a diagnostic device, not a assure. Typically, even when your R-hat readout is decrease than 1.01, the sampler may not have correctly explored the total posterior. This occurs when a number of bees begin their exploration in the identical room and stay there. Likewise, in case you’re utilizing a small variety of chains, and in case your posterior occurs to be multimodal, there’s a chance that every one chains began in the identical mode and didn’t discover different peaks.
The R-hat readout displays convergence, not completion. So as to have a extra complete concept, we have to verify different diagnostic metrics as properly.
Efficient Pattern Measurement (ESS)
When explaining what MCMC was, we talked about that “Monte Carlo” refers to the truth that the subsequent state is chosen randomly. This doesn’t essentially imply that the states are totally unbiased. Despite the fact that the bees select their subsequent step at random, these steps are nonetheless correlated to some extent. If a bee is exploring a lounge at time t=0, it’ll in all probability nonetheless be in the lounge at time t=1, despite the fact that it’s in a distinct a part of the identical room. Because of this pure connection between samples, we are saying these two information factors are autocorrelated.
Because of their nature, MCMC strategies inherently produce autocorrelated samples, which complicates statistical evaluation and requires cautious analysis. In statistical inference, we regularly assume unbiased samples to make sure that the estimates of uncertainty are correct, therefore the necessity for uncorrelated samples. If two information factors are too related to one another, the correlation reduces their efficient data content material. Mathematically, the formulation beneath represents the autocorrelation perform between two time factors (t1 and t2) in a random course of:
[
R_{XX}(t_1, t_2) = E[X_{t_1} overline{X_{t_2}}]
]
the place E is the anticipated worth operator and X-bar is the complicated conjugate. In MCMC sampling, that is essential as a result of excessive autocorrelation implies that new samples don’t train us something completely different from the previous ones, successfully decreasing the pattern measurement we have now. Unsurprisingly, the metric that displays that is known as Efficient Pattern Measurement (ESS), and it helps us decide what number of actually unbiased samples we have now.
As hinted beforehand, the efficient pattern measurement accounts for autocorrelation by estimating what number of actually unbiased samples would offer the identical data because the autocorrelated samples we have now. Mathematically, for a parameter θ, the ESS is outlined as:
[
ESS = frac{n}{1 + 2 sum_{k=1}^{infty} rho(theta)_k}
]
the place n is the full variety of samples and ρ(θ)okay is the autocorrelation at lag okay for parameter θ.
Usually, for ESS readouts, the upper, the higher. That is what we see within the readout for the primary mannequin. Two widespread ESS variations are Bulk-ESS, which assesses mixing within the central a part of the distribution, and Tail-ESS, which focuses on the effectivity of sampling the distribution’s tails. Each inform us if our mannequin precisely displays the central tendency and credible intervals.
In distinction, the readouts for the second mannequin are very dangerous. Usually, we need to see readouts which are a minimum of 1/10 of the full pattern measurement. On this case, given every chain sampled 2000 observations, we should always count on ESS readouts of a minimum of 800 (from the full measurement of 8000 samples throughout 4 chains of 2000 samples every), which isn’t what we observe.
Visible Diagnostics
Other than the numerical metrics, our understanding of sampler efficiency may be deepened by means of using diagnostic plots. The primary ones are rank plots, hint plots, and pair plots.
Rank Plots
A rank plot helps us establish whether or not the completely different chains have explored all the posterior distribution. If we as soon as once more consider the bee analogy, rank plots inform us which bees explored which components of the home. Subsequently, to judge whether or not the posterior was explored equally by all chains, we observe the form of the rank plots produced by the sampler. Ideally, we would like the distribution of all chains to look roughly uniform, like within the rank plots generated after sampling the primary mannequin. Every coloration beneath represents a series (or bee):
Beneath the hood, a rank plot is produced with a easy sequence of steps. First, we run the sampler and let it pattern from the posterior of every parameter. In our case, we’re sampling posteriors for parameters m and s of the primary mannequin. Then, parameter by parameter, we get all samples from all chains, put them collectively, and get them organized from smallest to largest. We then ask ourselves, for every pattern, what was the chain the place it got here from? This can enable us to create plots like those we see above.
In distinction, dangerous rank plots are simple to identify. In contrast to the earlier instance, the distributions from the second mannequin, proven beneath, are usually not uniform. From the plots, what we interpret is that every chain, after starting at completely different random places, received caught in a area and didn’t discover the whole lot of the posterior. Consequently, we can’t make inferences from the outcomes, as they’re unreliable and never consultant of the true posterior distribution. This is able to be equal to having 4 bees that began at completely different rooms of the home and received caught someplace throughout their exploration, by no means protecting the whole lot of the property.
KDE and Hint Plots
Just like R-hat, hint plots assist us assess the convergence of MCMC samples by visualizing how the algorithm explores the parameter house over time. PyMC supplies two sorts of hint plots to diagnose mixing points: Kernel Density Estimate (KDE) plots and iteration-based hint plots. Every of those serves a definite function in evaluating whether or not the sampler has correctly explored the goal distribution.
The KDE plot (often on the left) estimates the posterior density for every chain, the place every line represents a separate chain. This permits us to verify whether or not all chains have converged to the identical distribution. If the KDEs overlap, it means that the chains are sampling from the identical posterior and that mixing has occurred. Then again, the hint plot (often on the suitable) visualizes how parameter values change over MCMC iterations (steps), with every line representing a distinct chain. A well-mixed sampler will produce hint plots that look noisy and random, with no clear construction or separation between chains.
Utilizing the bee analogy, hint plots may be regarded as snapshots of the “options” of the home at completely different places. If the sampler is working accurately, the KDEs within the left plot ought to align carefully, exhibiting that every one bees (chains) have explored the home equally. In the meantime, the suitable plot ought to present extremely variable traces that mix collectively, confirming that the chains are actively transferring by means of the house moderately than getting caught in particular areas.
Nonetheless, in case your sampler has poor mixing or convergence points, you will note one thing just like the determine beneath. On this case, the KDEs won’t overlap, that means that completely different chains have sampled from completely different distributions moderately than a shared posterior. The hint plot may also present structured patterns as a substitute of random noise, indicating that chains are caught in numerous areas of the parameter house and failing to totally discover it.
Through the use of hint plots alongside the opposite diagnostics, you’ll be able to establish sampling points and decide whether or not your MCMC algorithm is successfully exploring the posterior distribution.
Pair Plots
A 3rd sort of plot that’s usually helpful for diagnostic are pair plots. In fashions the place we need to estimate the posterior distribution of a number of parameters, pair plots enable us to watch how completely different parameters are correlated. To know how such plots are shaped, suppose once more in regards to the bee analogy. For those who think about that we’ll create a plot with the width and size of the home, every “step” that the bees take may be represented by an (x, y) mixture. Likewise, every parameter of the posterior is represented as a dimension, and we create scatter plots exhibiting the place the sampler walked utilizing parameter values as coordinates. Right here, we’re plotting every distinctive pair (x, y), ensuing within the scatter plot you see in the course of the picture beneath. The one-dimensional plots you see on the sides are the marginal distributions over every parameter, giving us extra data on the sampler’s habits when exploring them.
Check out the pair plot from the primary mannequin.
Every axis represents one of many two parameters whose posteriors we’re estimating. For now, let’s give attention to the scatter plot within the center, which reveals the parameter mixtures sampled from the posterior. The actual fact we have now a really even distribution implies that, for any explicit worth of m, there was a variety of values of s that have been equally more likely to be sampled. Moreover, we don’t see any correlation between the 2 parameters, which is often good! There are circumstances after we would count on some correlation, akin to when our mannequin includes a regression line. Nonetheless, on this occasion, we have now no cause to imagine two parameters must be extremely correlated, so the very fact we don’t observe uncommon habits is optimistic information.
Now, check out the pair plots from the second mannequin.
Provided that this mannequin has 5 parameters to be estimated, we naturally have a higher variety of plots since we’re analyzing them pair-wise. Nonetheless, they give the impression of being odd in comparison with the earlier instance. Particularly, moderately than having a good distribution of factors, the samples right here both appear to be divided throughout two areas or appear considerably correlated. That is one other approach of visualizing what the rank plots have proven: the sampler didn’t discover the total posterior distribution. Beneath we remoted the highest left plot, which accommodates the samples from m0 and m1. In contrast to the plot from mannequin 1, right here we see that the worth of 1 parameter drastically influences the worth of the opposite. If we sampled m1 round 2.5, for instance, m0 is more likely to be sampled from a really slim vary round 1.5.
Sure shapes may be noticed in problematic pair plots comparatively continuously. Diagonal patterns, for instance, point out a excessive correlation between parameters. Banana shapes are sometimes related to parametrization points, usually being current in fashions with tight priors or constrained parameters. Funnel shapes would possibly point out hierarchical fashions with dangerous geometry. When we have now two separate islands, like within the plot above, this will point out that the posterior is bimodal AND that the chains haven’t combined properly. Nonetheless, take into account that these shapes would possibly point out issues, however not essentially achieve this. It’s as much as the information scientist to look at the mannequin and decide which behaviors are anticipated and which of them are usually not!
Some Fixing Methods
When your diagnostics point out sampling issues — whether or not regarding R-hat values, low ESS, uncommon rank plots, separated hint plots, or unusual parameter correlations in pair plots — a number of methods may also help you deal with the underlying points. Sampling issues sometimes stem from the goal posterior being too complicated for the sampler to discover effectively. Complicated goal distributions may need:
- A number of modes (peaks) that the sampler struggles to maneuver between
- Irregular shapes with slim “corridors” connecting completely different areas
- Areas of drastically completely different scales (just like the “neck” of a funnel)
- Heavy tails which are troublesome to pattern precisely
Within the bee analogy, these complexities characterize homes with uncommon flooring plans — disconnected rooms, extraordinarily slim hallways, or areas that change dramatically in measurement. Simply as bees would possibly get trapped in particular areas of such homes, MCMC chains can get caught in sure areas of the posterior.
To assist the sampler in its exploration, there are easy methods we are able to use.
Technique 1: Reparameterization
Reparameterization is especially efficient for hierarchical fashions and distributions with difficult geometries. It includes remodeling your mannequin’s parameters to make them simpler to pattern. Again to the bee analogy, think about the bees are exploring a home with a peculiar structure: a spacious lounge that connects to the kitchen by means of a really, very slim hallway. One facet we hadn’t talked about earlier than is that the bees must fly in the identical approach by means of the complete home. That implies that if we dictate the bees ought to use giant “steps,” they’ll discover the lounge very properly however hit the partitions within the hallway head-on. Likewise, if their steps are small, they’ll discover the slim hallway properly, however take eternally to cowl the complete lounge. The distinction in scales, which is pure to the home, makes the bees’ job tougher.
A traditional instance that represents this situation is Neal’s funnel, the place the size of 1 parameter is dependent upon one other:
[
p(y, x) = text{Normal}(y|0, 3) times prod_{n=1}^{9} text{Normal}(x_n|0, e^{y/2})
]
We will see that the size of x relies on the worth of y. To repair this drawback, we are able to separate x and y as unbiased customary Normals after which remodel these variables into the specified funnel distribution. As an alternative of sampling instantly like this:
[
begin{align*}
y &sim text{Normal}(0, 3)
x &sim text{Normal}(0, e^{y/2})
end{align*}
]
You’ll be able to reparameterize to pattern from customary Normals first:
[
y_{raw} sim text{Standard Normal}(0, 1)
x_{raw} sim text{Standard Normal}(0, 1)
y = 3y_{raw}
x = e^{y/2} x_{raw}
]
This method separates the hierarchical parameters and makes sampling extra environment friendly by eliminating the dependency between them.
Reparameterization is like redesigning the home such that as a substitute of forcing the bees to discover a single slim hallway, we create a brand new structure the place all passages have related widths. This helps the bees use a constant flying sample all through their exploration.
Technique 2: Dealing with Heavy-tailed Distributions
Heavy-tailed distributions like Cauchy and Scholar-T current challenges for samplers and the best step measurement. Their tails require bigger step sizes than their central areas (just like very lengthy hallways that require the bees to journey lengthy distances), which creates a problem:
- Small step sizes result in inefficient sampling within the tails
- Giant step sizes trigger too many rejections within the heart
Reparameterization options embody:
- For Cauchy: Defining the variable as a metamorphosis of a Uniform distribution utilizing the Cauchy inverse CDF
- For Scholar-T: Utilizing a Gamma-Combination illustration
Technique 3: Hyperparameter Tuning
Typically the answer lies in adjusting the sampler’s hyperparameters:
- Enhance complete iterations: The best strategy — give the sampler extra time to discover.
- Enhance goal acceptance charge (adapt_delta): Cut back divergent transitions (strive 0.9 as a substitute of the default 0.8 for complicated fashions, for instance).
- Enhance max_treedepth: Enable the sampler to take extra steps per iteration.
- Prolong warmup/adaptation part: Give the sampler extra time to adapt to the posterior geometry.
Do not forget that whereas these changes might enhance your diagnostic metrics, they usually deal with signs moderately than underlying causes. The earlier methods (reparameterization and higher proposal distributions) sometimes supply extra elementary options.
Technique 4: Higher Proposal Distributions
This answer is for perform becoming processes, moderately than sampling estimations of the posterior. It mainly asks the query: “I’m presently right here on this panorama. The place ought to I bounce to subsequent in order that I discover the total panorama, or how do I do know that the subsequent bounce is the bounce I ought to make?” Thus, selecting a superb distribution means ensuring that the sampling course of explores the total parameter house as a substitute of only a particular area. proposal distribution ought to:
- Have substantial chance mass the place the goal distribution does.
- Enable the sampler to make jumps of the suitable measurement.
One widespread alternative of the proposal distribution is the Gaussian (Regular) distribution with imply μ and customary deviation σ — the size of the distribution that we are able to tune to resolve how far to leap from the present place to the subsequent place. If we select the size for the proposal distribution to be too small, it would both take too lengthy to discover the complete posterior or it’ll get caught in a area and by no means discover the total distribution. But when the size is simply too giant, you would possibly by no means get to discover some areas, leaping over them. It’s like enjoying ping-pong the place we solely attain the 2 edges however not the center.
Enhance Prior Specification
When all else fails, rethink your mannequin’s prior specs. Obscure or weakly informative priors (like uniformly distributed priors) can generally result in sampling difficulties. Extra informative priors, when justified by area data, may also help information the sampler towards extra affordable areas of the parameter house. Typically, regardless of your greatest efforts, a mannequin might stay difficult to pattern successfully. In such circumstances, take into account whether or not an easier mannequin would possibly obtain related inferential objectives whereas being extra computationally tractable. One of the best mannequin is commonly not essentially the most complicated one, however the one which balances complexity with reliability. The desk beneath reveals the abstract of fixing methods for various points.
| Diagnostic Sign | Potential Concern | Really useful Repair |
| Excessive R-hat | Poor mixing between chains | Enhance iterations, modify the step measurement |
| Low ESS | Excessive autocorrelation | Reparameterization, enhance adapt_delta |
| Non-uniform rank plots | Chains caught in numerous areas | Higher proposal distribution, begin with a number of chains |
| Separated KDEs in hint plots | Chains exploring completely different distributions | Reparameterization |
| Funnel shapes in pair plots | Hierarchical mannequin points | Non-centered reparameterization |
| Disjoint clusters in pair plots | Multimodality with poor mixing | Adjusted distribution, simulated annealing |
Conclusion
Assessing the standard of MCMC sampling is essential for making certain dependable inference. On this article, we explored key diagnostic metrics akin to R-hat, ESS, rank plots, hint plots, and pair plots, discussing how every helps decide whether or not the sampler is performing correctly.
If there’s one takeaway we would like you to remember it’s that you must all the time run diagnostics earlier than drawing conclusions out of your samples. No single metric supplies a definitive reply — every serves as a device that highlights potential points moderately than proving convergence. When issues come up, methods akin to reparameterization, hyperparameter tuning, and prior specification may also help enhance sampling effectivity.
By combining these diagnostics with considerate modeling selections, you’ll be able to guarantee a extra strong evaluation, decreasing the chance of deceptive inferences attributable to poor sampling habits.
References
B. Gilbert, Bob’s bees: the significance of utilizing a number of bees (chains) to evaluate MCMC convergence (2018), Youtube
Chi-Feng, MCMC demo (n.d.), GitHub
D. Simpson, Perhaps it’s time to let the previous methods die; or We broke R-hat so now we have now to repair it. (2019), Statistical Modeling, Causal Inference, and Social Science
M. Taboga, Markov Chain Monte Carlo (MCMC) strategies (2021), Lectures on chance concept and mathematical Statistics. Kindle Direct Publishing.
T. Wiecki, MCMC Sampling for Dummies (2024), twecki.io
Stan Consumer’s Information, Reparametrization (n.d.), Stan Documentation
