In a seminal however underappreciated e-book titled Common Synthetic Intelligence: Sequential Choices Based mostly on Algorithmic Likelihood, Marcus Hutter tried a mathematical formulation of common synthetic intelligence, shortened to AIXI. This text goals to make AIXI accessible to knowledge scientists, technical lovers and basic audiences each conceptually and formally.
We start with a short overview of the axioms of likelihood idea. Subsequently we delve into conditional likelihood, whose calculation is ruled by Bayes’s theorem. Whereas Bayes’s theorem offers the framework for updating beliefs, it leaves open the query of assign priors. To deal with this, we flip to algorithmic info idea, which connects Kolmogorov complexity, outlined because the size of the shortest program that outputs a string, with the project of Bayesian priors. The bridge between these two concepts is the Solomonoff prior, also referred to as the common prior. The common prior offers us with the required scaffolding to discover the AIXI formalism, which integrates sequential determination idea, the Solomonoff prior and Occam’s Razor. Within the last part, we briefly focus on the restrictions of AIXI and different approaches to formalizing an common agent, whereas acknowledging that the time period “common agent” carries important philosophical ambiguity. Particularly, we focus on Energetic Inference as a philosophical different to AIXI, whereby the previous fashions an embodied predictive agent, whereas the latter, fashions a disembodied utility maximization algorithm.
Observe: All pictures within the weblog are created by the creator.
Likelihood Axioms
The Kolmogorov axioms outline probabilistic area as a triple (Ω , 𝒜, 𝓟), the place Ω defines the overall pattern area, 𝒜 the gathering of subsets of occasions of curiosity, and 𝓟 the operate that assigns a likelihood to every occasion normalized to the unit interval.
- If 𝑨 ϵ 𝒜, then 𝓟(𝑨) ≥ 0
- If 𝑨, B ϵ 𝒜 and A ∩ B = ∅, then P (A⋃B) = P(A) + P(B)
- P(Ω) = 1
The primary axiom, non-negativity, ensures that possibilities are significant as real-valued measures of perception or frequency. The second, additivity, formalizes the precept that the likelihood of disjoint outcomes is the sum of their particular person possibilities. And the third, normalization, ensures that the overall likelihood assigned to the whole pattern area equals one.
Nonetheless, whereas the likelihood axioms specify the structural guidelines of likelihood, they don’t prescribe how possibilities must be up to date in mild of latest proof. On this sense, the Kolmogorov framework is analytic and a priori: it defines what a likelihood measure should fulfill, however not how such a measure is revised by means of proof. To maneuver from static likelihood assignments to dynamic inference, we’d like a technique to relate new knowledge to current hypotheses, specifically conditional likelihood. This epistemic hole is addressed in frequentist statistics by decoding conditional possibilities by means of long-run frequencies of repeated occasions, sometimes below the belief that such occasions are independently and identically distributed (i.i.d.), whereas Bayes’s theorem offers a normative rule for updating beliefs about hypotheses in mild of latest proof, helpful particularly when observations are made incrementally or pattern areas are usually not well-defined.
Bayesian Inference
First formalized by a Scottish monk, Bayes’ theorem is an algebraic derivation of conditional likelihood. As soon as we perceive how conditional likelihood is calculated, Bayes’ Theorem will be derived by means of just a few algebraic operations. Let’s recall how we compute conditional likelihood:
This states that the likelihood of the speculation H given the proof D is computed because the joint likelihood of the speculation and the proof divided by the likelihood of the proof.
Why would we compute conditional likelihood this manner? Let’s stroll by means of it by the use of an instance. The likelihood that it rained — the speculation — provided that the bottom is moist — the knowledge — assumes that these two occasions are dependent. In the event that they had been unbiased, then the likelihood of their joint incidence can be computed by their product P(H) · P(D). It is because the likelihood of P(H|D) = P(H) and the likelihood of P(D|H)=P(D) assume that the occasion of the bottom being moist is unbiased of it raining. Discover what we’ve simply asserted: P(H∩D) = P(H) · P(D). Which means the intersection of unbiased occasions is computed by the product of their particular person possibilities.
However how is the intersection P(H∩D) of dependent occasions computed? Set-theoretically, the joint likelihood is outlined because the intersection of the 2 units:
To be able to perceive the proportion of the distribution of the pattern areas, we are able to visualize the conditional likelihood we search to calculate as follows:
However in observe we virtually by no means have prior information of the joint likelihood distribution. That is the place we are able to use a easy algebraic step to assist us approximate the joint likelihood. We multiply each side of the equation by the denominator to resolve for the joint likelihood P(H∩D):
Now conversely, if we needed to compute the likelihood that the bottom is moist provided that it rained, our conditional likelihood formulation can be the next:
The identical transformation give us:
Discover that the joint likelihood of the 2 occasions in query is the time period each conditional possibilities share. Since P(H∩D) is a symmetrical relation and consequently an algebraic fixed between the equations, we get the next essential equality:
Subsequently, we if need to check the speculation that “it rained” provided that the “floor is moist”, we are able to rearrange this equality to acquire Bayes’ formulation:
In Bayesian terminology, we confer with P(H|D) because the posterior (specifically, the likelihood we want to verify), P(H) because the prior, P(D|H) because the probability, and P(D) because the marginal.
This standard nomenclature is essential when coping with Bayesian statistics. The probability provides the conditional possibilities for the info factors below the speculation (offering the values used to replace our beliefs), whereas the marginal normalizes the posterior to the pattern area of the situation or knowledge.
Since we’d like approximations of values for all of the phrases in Bayes’ formulation, a significant hurdle in Bayesian statistics is greatest assign these values. Particularly, specifying the prior will be difficult since we don’t at all times have the requisite information prematurely. Some methods in approximating the prior embrace utilizing a uniformly distributed prior, the place we assign the identical likelihood to all potential outcomes, often called the Laplacian precept of indifference, and different methods embrace utilizing an informative prior, specifically a previous that goals to approximate the precise likelihood distribution of the occasion. In our instance, this might be the Poisson distribution of day by day rain.
As we shift from viewing hypotheses as mounted values to treating them as random variables, the purpose turns into to deduce the total posterior distribution moderately than a single estimate. Accordingly, we are able to now transfer from the treating hypotheses as level estimates towards statistical inference over random variables with corresponding likelihood distributions. To do that, we mannequin the prior P(H) and the probability P(D|H) as likelihood distributions and compute the marginal likelihood of the info P(D), which is obtained both as a sum (the likelihood mass operate for discrete variables), or an integral (the likelihood density for steady variables). These elements enable us to use Bayes’ theorem to acquire the posterior distribution: P(H|D).
The likelihood mass operate (PMF) is computed because the sum of all values of the distribution, equaling to at least one, whereas the likelihood density operate (PDF) is the integral of the world below the curve of the distribution approaching one because the restrict. For steady variables we combine as a result of we’re coping with infinite values within the distribution. Under is the formulation for the marginal for steady variables, specifically the legislation of whole likelihood expressed because the likelihood density operate:
Bayesian statistics types another framework to the extra established frequentist method in statistics. Despite the fact that its historic roots are as deep because the frequentist formulation, computational intractability restricted its adoption for a lot of the 20 th century. With advances in computational energy, Bayesian strategies have undergone speedy growth and elevated utility. At the moment, Bayesian statistics performs a central function in machine studying (ML), significantly probabilistic modeling, uncertainty estimation and decision-making below uncertainty.
Kolmogorov Complexity
We noticed that our Kolmogorov axioms equipped an analytic framework for computing possibilities which included defining the union of disjoint units as sums and their intersection as merchandise. Nevertheless it didn’t inform us compute joint units. For this we invoked Bayes’ theorem, which makes use of set intersection to derive a common formulation of conditional likelihood.
Nonetheless, in our explication of Bayesian likelihood we recognized the project of priors as an issue for the framework: what info ought to we encode within the prior? Ought to we make it detached, as per the precept of indifference, or make it informative?
That is the place the notion of Kolmogorov Complexity is available in. Whereas Kolmogorov complexity doesn’t entail likelihood, by means of the Coding Theorem (which we’ll explicate under), it encodes an a posteriori meta-theoretic assumption as a mathematical choice bias. This meta-theoretic generalization states that simplicity encodes larger likelihood. If we’re confronted with the datum or end result that the bottom is moist, what speculation from the obtainable retailer of all potential hypotheses ought to we choose? Intuitively we would like the speculation that assigns the best likelihood to the noticed end result. However with out further info, how are we to know which speculation maximizes the likelihood of the result? Kolmogorov answered this throughout the bounds of algorithmic info idea: the best speculation is the speculation that encodes the least info or the sequence with shortest size.
To be able to perceive the motivation behind this, let’s first state the issue inside algorithmic info idea, then circle again to its utility inside much less summary, real-world situations.
In algorithmic info idea, we encode symbolic sequences or strings based on some symbolic base akin to base-2, binary strings. We outline a Common Turing Machine, U as a (partial — since p can’t be outlined for all outputs) operate from a program p to an output x , i.e. U(p) = x. Consider this as loosely cognate to: f(x) = y. This system p represents the speculation or idea, whereas the output x represents the info or the proof. This mapping is essential to grasp the intuitive thrust of the speculation.
The Kolmogorov Complexity of an info object defines because the shortest size of an algorithmic sequence that outputs that object, the place Ok(x) defines the size of this system in bits:
This expression tells us that out of all of the applications p that produce x as output, we choose the shortest one, specifically the minimal . Kolmogorov complexity is outlined over finite binary strings x ϵ {0,1}
What we now have to do is join Kolmogorov complexity to likelihood idea in order that it may inform our Bayesian priors. To do that, we discover a connection, at the very least superficial at first, between between Kolmogorov complexity and Shannon info entropy. Each appear to quantify some measure of data content material: Ok(x) defines size in bits whereas info entropy H defines the common quantity of data required to encode the distribution of a random variable, the place info is outlined as uncertainty and quantified because the anticipated worth of -log P(x) over all potential outcomes in bits. The better the uncertainty, the bigger the quantity of data required to encode the occasion. Each Ok(x) and H(X) are measured in bits, so what’s the distinction?
Ok(x) describes the size of the shortest program in bits that outputs the string x, whereas H(X) computes the variety of bits wanted on common to encode this system drawn from the likelihood distribution of potential values of x, over the pattern area of X. It looks like some deep connection should maintain between these two measures. What then is the connection between Kolmogorov complexity and Shannon entropy? We want a bridge from uncooked size values to their possibilities.
If we isolate a single end result from the Shannon distribution, we are able to outline it because the self-information of x, the output of our program:
This says that the self-information (consider this because the entropy measure for a single end result) is proportional to the log-inverse likelihood of x occurring. I(x) is an occasion of the total distribution that defines Shannon entropy for a selected occasion:
Now, the Coding Theorem states that Kolmogorov Complexity is roughly equal to the Shannon info contained in a string.
This states that the shortest program that outputs x is roughly so long as the log-inverse of the overall likelihood that outputs x. In different phrases, our Shannon distribution accommodates the shortest program because the one assigned with the best likelihood! Now we have now linked uncooked program size with likelihood idea: the extra compressible a program output is, the extra seemingly it’s to happen.
That is how we join algorithmic compressibility, specifically program size outlined for an occasion, to likelihood and data idea, enabling us to deal with compressibility as a Bayesian probability. On a sidenote, the rationale we don’t have a exact equality within the equation above is that the postulated relationship will not be actual as much as an additive fixed c, which relies on the selection of Common Turing Machine (UTM), making Ok(x) machine-dependent previous to an higher certain c throughout UTMs:
Now you’re in all probability questioning, what sort of distribution allows us to assign possibilities to all potential program lengths? That distribution is the Solomonoff Common Prior.
Solomonoff Induction
As we mentioned, the selection of prior impacts the posterior particularly when pattern sizes are small enough. This begs the query: what if we had a previous operate that might be utilized to all potential occasions within the pattern area? That is what Solomonoff’s prior encodes. Exactly, the Solomonoff prior encodes the likelihood of observing an output sequence x as the overall likelihood {that a} random program outputs x on a common Turing machine.
Now, let’s check out Solomonoff’s common prior formulation, which ought to click on into place our earlier assertion that algorithmic likelihood is intimately linked with simplicity. Solomonoff outlined the common prior P(x) because the sum of the chances of all finite binary prefix-free applications p, the place the likelihood that p is outlined by its simplicity, 2^-|p|.
As a result of we outline the likelihood of a program as shrinking by half for each further bit it accommodates, the extra info bits in this system, the smaller its weight within the distribution. Subsequently, the overall likelihood over all prefix-free binary applications might be dominated by shortest applications.
We acknowledged that the Solomonoff prior is outlined for prefix-free finite binary sequences of strings. Let’s guarantee we perceive every of those qualifiers. We use binary sequences as a result of a Common Turing machine will be outlined by way of binary inputs and outputs, the place we are able to signify any info object with binary encoding. We outline the prior over finite sequences so as to meet the situations of computability: infinite sequences are Turing incomputable.
A set of strings is prefix free if no string within the set is a prefix of one other:
This yields units of disjoint finite binary string sequences. In different phrases, we’d like disjoint units to compute their union. Per the Kolmogorov axiom of additivity, the likelihood of the union of the members of the set will be expressed because the sum of their possibilities.
Disjointness ensures that the likelihood related to every speculation or prior string obeys Kraft’s inequality, which states that the sum of all possibilities doesn’t exceed the unit interval:
This tells us that for some some prefix-free string C, the likelihood of that sequence is expressed as 2 raised to the unfavorable exponent, the place the exponent describes the size. As a result of all of the sequences are disjoint, their sum can’t exceed 1 (although it may be lower than one, making it a semi-measure). This allows us to deal with code weights as possibilities and consequently to compute the likelihood mass operate of the whole distribution by summing over string weights.
Accordingly, Solomonoff’s prior is outlined because the sum of the weights or possibilities of finite binary applications p:
Subsequently, so as to compute the likelihood of getting some output x from a potential program, we situation that likelihood to the sum of possibilities of all potential applications:
As a result of p is deterministic, the likelihoods and the posterior are outlined as delta capabilities: both a program outputs x or it doesn’t.
Additional, as a result of the prior is outlined over prefix-free binary strings, we are able to categorical the conditionalization by way of bitwise strings:
As a substitute of a joint distribution over occasions, now we have a weighted sum over bitstrings that syntactically generate x as a stand-in for all potential occasions. This reveals a few of the limitations of the formalism: does formal compressibility suffice as an explanatory bias for sure applications or theories over others? We’ll delve into these limitations later akin to the shortage of structural bias and representational alignment.
Along with the Coding Theorem, the Solomonoff prior tenders a deep connection between induction and compressibility: generalization is revealed to be formally equal to info compression such that the extra compressible a dataset is, the extra possible this system that produces it. In the true world, nevertheless, we all know that essentially the most “compressible” theories are usually not at all times those with the best explanatory or predictive energy, although the preponderance of greatest theoretical approximations have a tendency towards simplicity.
The formulation under expresses our notions of algorithmic complexity, the common prior, and data entropy as roughly equal to one another (below particular ranges of x):
AIXI
Because it stands, our idea of common induction which mixes the Solomonoff prior and Bayesian posteriors will not be outlined for a constrained agent. What if we mix Solomonoff induction with sequential determination idea?
That is the place Marcus Hutter’s AIXI idea is available in: it integrates Solomonoff induction, determination idea, and reinforcement studying in order that our common prior can do work for an agent.
Transitioning from Solomonoff induction into the territory of determination idea and reinforcement studying requires increasing our ontology to actions, observations, and rewards. AIXI fashions a common agent whose interplay with any computable setting allows it to decide on the motion that maximizes anticipated rewards. In additional formal phrases, AIXI selects an motion at every time step and in return receives an commentary and reward from the setting. How does AIXI choose the optimum motion? As we’ll see, as a result of AIXI encodes a super Bayesian agent, it constitutes a model-based agent. Nonetheless, not like a typical Bellman-based deterministic agent (which solves the Bellman equations to determinate optimality, see my earlier article on reinforcement studying for that), AIXI maintains uncertainty over all potential environments. It does so by computing the sum of merchandise of the likelihoods, specifically the chances of environmental suggestions given the area of actions, and the Solomonoff weights assigned to each computable setting (or program), known as the common combination.
Put succinctly, the common combination constitutes a time period inside AIXI that defines the probabilistic prediction of the subsequent commentary and reward pair given the present motion. It’s computed because the sum of merchandise of the weighted distribution of each potential setting. The common combination exhausts the setting area by summing over the product of every setting weight and its likelihood distribution the place every setting mannequin μ assigns possibilities to observation-reward sequences given hitherto actions. The common combination is given by the formulation under:
The common combination accumulates a predictive distribution of the setting by means of every motion that it takes. Consider ξ as assigning possibilities to each potential future trajectory of observations and rewards given a sequence of actions.
The common combination offers us with the likelihood of commentary and reward pairs below the motion historical past, nevertheless it doesn’t inform us which of those trajectories is essentially the most values or reward maximizing. For this we sum the reward per setting or trajectory:
To be able to discover out which trajectory to decide on, we multiply the sum of rewards per trajectory by the likelihood assigned to that trajectory by the common combination:
As such, we compute expectation by weighting every trajectory by its cumulative rewards.
After we compute expectations, the ultimate step entails deciding on the motion that maximizes anticipated return given the weighting of rewards and setting possibilities. For this we make use of the arg max operate as follows:
Arg max selects the motion that maximizes cumulative returns throughout all potential trajectories:
AIXI goals to formalize a common agent whose tools with the Solomonoff prior biases it towards environments with minimal Kolmogorov complexity. Aside from assuming all potential computable environments and associating compressibility with larger likelihood, the AIXI agent acquires structural bias from interplay with the setting. This ensures that AIXI ought to be capable to navigate any setting (supplied that it’s structured/computable to start with).
Whereas AIXI formalizes a common agent, it doesn’t sufficiently bias this agent in order that it may effectively navigate precise environments. That’s to say, the procedures for optimum motion or determination that AIXI formalizes don’t encode environment friendly structural biases akin to domain-specific heuristics or architectural constraints that may speed up studying. Nonetheless, this function is a consequence of AIXI’s scope of setting common bounds on determination optimality throughout all potential environments. In precept, AIXI acquires environment friendly structural biases not directly by means of Bayesian updating. With enough environmental sampling, AIXI asymptotically converges to the true setting throughout infinite interactions, assuming the setting is computable and has non-zero prior. Nonetheless, in observe convergence will be inefficient because of overspreading over posterior weights leaving the agent’s actions suboptimal for an indefinite period of time.
Noncomputability & Structural Bias
In its common kind AIXI will not be algorithmically implementable as a result of Kolmogorov complexity and the Solomonoff prior are each not computable. The category of all applications that halt and produce legitimate (computable) environments is uncountable or not recursively enumerable. Computing the common prior requires an infinite simulation of environments whereas computing future expectations requires infinite foresight, all of that are mathematically intractable.
For that reason, computable approximations of AIXI exist akin to AIXItl. AIXItl introduces time and program-length bounds (which is what the tl stands for) limiting the setting area Mtl to ≤ t time steps and l bits lengthy. Nonetheless, AIXItl continues to be inefficient for the reason that combinatorial prospects of environments is exponential: O(2^l). Mannequin-free alternate options akin to DQN and gradient-based alternate options akin to Dreamer and World Fashions signify alternate options within the seek for a common agent. These later alternate options use heuristics and sampling-based strategies for exploration akin to Monte Carlo Tree Search for optimum determination making. Essentially, the competition lies between model-based and model-free strategies, the latter which derive their biases fully from interplay with the setting.
Representational Alignment
As we already acknowledged, AIXI treats the universe as a computable sequence represented by means of finite binary strings. The belief that the setting is Turing computable will not be entailed within the Church-Turing Thesis and thereby represents a further assumption of AIXI. The reality of this assumption, in precept, that’s, not with respect to a realizable machine, is an open query despite the fact that there’s good purpose to assume it false.
As we noticed AIXI treats observations as bitstrings, whereas actual world knowledge require structured representations akin to causal construction, temporal relationships, and spatial dimensions, to call just a few. To be able to encode richer buildings inside AIXI we would wish priors that encode structured representations akin to graphs, tensors, differential equations and so forth. Encoding structural biases would make AIXI extra environment friendly, however on the expense of its universality. The price of encoding real-world representational buildings inside a Bayesian mannequin is due to this fact specializing the mannequin on the expense of environment-generalizability. On condition that in observe an agent that realizes AIXI will not be potential, we should always due to this fact look to brokers that encode real-world representations akin to AIXItl, model-free brokers or deep-learning-based brokers.
Energetic Inference
We noticed that AIXI incorporates a maximally informative prior, however this prior is fully unstructured and embodies no prior information in regards to the world besides the meta-selection bias for brief or compressible applications as being the almost certainly. We additionally noticed that this makes each AIXI and the Solomonoff prior computationally intractable, which precludes its implementation in its full kind.
One other strand of agent modeling, extra not too long ago branded energetic inference, whose centerpiece is the free-energy minimization precept, goals to combine the modeling lineages of utility maximization, reinforcement studying, Bayesian inference, predictive coding, statistical mechanics, and much from thermodynamic equilibrium dynamics into the unified mannequin of a hierarchical generative Bayesian agent. Optimality within the energetic inference generative Bayesian mannequin consists of minimizing free vitality, the place free vitality is outlined because the anticipated shock in regards to the joint incidence of sensations and their inferred causes.
Put in colloquial phrases, the generative mannequin predicts future perceptions given actions by means of a ahead mannequin that estimates the causes of these sensations by means of the prior, and it conversely estimates or predicts the actions required to result in most popular states by means of an inverse mannequin. The agent dynamically navigates the setting by means of loops of ahead and inverse fashions or, extra merely, loops notion and motion. Free vitality outcomes from mismatch between predictions and environmental suggestions, minimized by means of hierarchical Bayesian updating of the mannequin’s priors.
The formulation under formally expresses the computation of variational free vitality because the divergence between the popularity density (the approximate posterior) and the conditional density (the true posterior), the place ỹ stands for noticed enter, 𝜗 for latent causes, p( ỹ, 𝜗) defines the generative mannequin because the joint likelihood density of perceptions and latent causes, whereas q(𝜗) defines the approximate posterior:
The primary expression within the formulation defines the Kullback-Leibler divergence between the approximate posterior q(𝜗) and the true posterior p(𝜗| ỹ) minus the log mannequin proof log p(ỹ). Typically, the Kullback-Leibler divergence quantifies the geometric divergence between a mannequin distribution Q and the precise distribution P. Free vitality outcomes from the information-theoretic divergence between the approximate and true posterior offset by the log mannequin proof. We compute variational free vitality by integrating the log ratio between the approximate and true joint distributions over latent causes. The second time period expresses this similar amount because the sum of the entropy of the approximate posterior q(𝜗) and the cross-entropy between the posterior and the generative mannequin p( ỹ, 𝜗). Minimizing free vitality quantities to lowering the divergence between the popularity mannequin and the conditional density.
Each AIXI and Energetic Inference provide optimum Bayesian brokers in numerous methods. However whereas AIXI is formally non-computable in its unbounded kind, Energetic Inference allows tractable approximations through variational Bayesian fashions. Optimization in AIXI consists of maximizing rewards, whereas in Energetic Inference in minimizing free vitality. Within the former, mannequin accuracy outcomes implicitly from maximizing rewards, whereas within the latter maximizing rewards outcomes implicitly from minimizing anticipated shock or free vitality. On this regard, Energetic Inference constitutes a structured generative mannequin that hierarchically estimates latent causes, guiding motion choice by means of inference moderately than AIXI’s enumeration, which selects the reward maximizing motion from all potential environments. But, Energetic Inference stays an incomplete framework because it glosses over many particulars about concrete brokers, akin to goal-setting, mannequin studying (the place this can be very obscure), and a viable description of agent boundary (the Markov blanket formulation is inadequate and unable to differentiate organic brokers from bodily techniques that don’t quantity to precise brokers).
References
Friston, Ok., Kilner, J., & Harrison, L. (2006). A free vitality precept for the mind. Journal of Physiology, Paris, 100(1–3), 70–87. https://doi.org/10.1016/j.jphysparis.2006.10.001
Hutter, M. (2005). Common Synthetic Intelligence: Sequential Choices Based mostly on Algorithmic Likelihood (1st ed.). Springer Berlin Heidelberg. https://doi.org/10.1007/b138233 SpringerLink+13SpringerLink+13Google Books+13
Sommaruga, G. (Ed.). (2009). Formal Theories of Info: From Shannon to Semantic Info Idea and Common Ideas of Info (Lecture Notes in Pc Science, Vol. 5363). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-00659-3
Zabell, S. (2009). Philosophy of inductive logic: The Bayesian perspective. In L. Haaparanta (Ed.), The event of recent logic (pp. 725–774). Oxford College Press. https://doi.org/10.1093/acprof:oso/9780195137316.003.0044
