Backoff DOP: Parameter Estimation by Backoff

Size: px

Start display at page:

Download "Backoff DOP: Parameter Estimation by Backoff"

Regina Hudson
5 years ago
Views:

1 Backoff DOP: Parameter Estimation by Backoff Luciano Buratto and Khalil ima an Institute for Logic, Language and Computation (ILLC) University of Amsterdam, Amsterdam, The Netherlands Abstract. The Data Oriented Parsing (DOP) model currently achieves state-ofthe-art parsing on benchmark corpora. However, existing DOP parameter estimation methods are known to be biased, and ad hoc adjustments are needed in order to reduce the effects of these biases on performance. This paper presents a novel estimation procedure that exploits a unique property of DOP: different derivations can generate the same parse-tree. We show that the different derivations represent different Markov orders that the DOP model interpolates together. The idea behind the present method is to combine the different derivation orders by backoff instead of interpolation. This allows for a novel estimation procedure that employs Katz backoff for estimation. We report on experiments showing error reduction of up to 15% with respect to earlier methods. 1 The DOP model The Data Oriented Parsing (DOP) model currently exhibits state-of-the-art performance on benchmark corpora [1]. Like other treebank models, DOP extracts a finite set of rewrite productions, called subtrees, from the training treebank together with probabilities. A connected subgraph of a treebank tree Ø is called a subtree iff it consists of one or more Context-Free productions 1 from Ø. Following [2], the set of rewrite productions of DOP consists of all the subtrees of the treebank trees. Figure 3 exemplifies the set of subtrees extracted from the treebank of Figure 1. The DOP model employs the set of subtrees as a tochastic Tree-ubstitution Grammar (TG): an TG is a rewrite system similar to Context-Free Grammars (CFGs), 1 Note that a non-leaf node labeled Ô in tree Ø dominating a sequence of nodes labeled ½ Ò consists of a graph that represents the Context-Free production: Ô ½ Ò. John Æ Æ Æ John Æ John = Æ John = John Fig. 1. A toy treebank Fig. 2. Two different derivations of the same parse

2 (s1) (s2) (s3) (s4) (s5) (s6) (s7) (s8) (s9) (s10) John John John (s11) (s12) (s13) (s14) (s15) (s16) (s17) John John John Fig. 3. The subtrees of the treebank in Figure 1 with the only difference that the productions of a TG are subtrees of arbitrary depth 2. A TG derivation proceeds by combining subtrees using the substitution operation Æ starting from the start symbol Ë of the TG. In contrast with CFG derivations, multiple TG derivations may generate the same parse. For example, the parse in Figure 1 can be derived at least in two different ways as shown in Figure 2. In this sense, the DOP model deviates from other contemporary models, e.g. [3, 4], that belong to the so-called History-Based tochastic Grammar (HBG) family [5]. The latter models generate every parse-tree through a unique stochastic derivation. An tochastic TG (TG) is a TG extended with a probability mass function È over the set of subtrees: the probability of subtree Ø, that has root label Ê Ø, is given by È Ø Ê Ø µ, i.e. for every non-terminal : È Ø Ê Ø È Ø µ ½. Given a probability function È, the probability of a derivation Ë ÆØ ½ Æ ÆØ Ò is defined by È Ëµ É Ò È Ø ½ Ê Ø µ. The probability of a parse is defined by the sum of the probabilities of all derivations in the TG that generate that parse. As a down side to the multiple derivations per parse property, the algorithms for selecting the parse with the highest probability are known to be intractable [6]. In this paper we address another difficulty that arises from the property of the multiple derivations per parse with regard to the DOP model: how to estimate the model parameters (subtree probabilities) from a treebank? 2 Existing DOP estimators The problem of how to estimate the probabilities of the subtrees from a treebank is not as straightforward as originally thought. o far, there exist three estimation procedures [2, 7, 1]. As shown in [7 9], all three estimation procedures turn out to be biased in an unintuitive manner. Next we provide an overview of the existing methods. (1) Relative frequency Wrong bias The first instantiation of a DOP model is due to [2] and is referred to as ÇÈ Ö. In this model, the probability estimates of subtrees 2 The depth of a tree is the number of edges along the longest path from the root to a leaf node.

3 extracted from a treebank are given by a relative frequency estimator. Let Øµ represent the number of times Ø occurred in the bag of subtrees extracted from the treebank. Then Øµ the probability of Ø in ÇÈ Ö is estimated as: È Ø Ê Ø µ ÈØ. ¼ ÊØ ¼ Ê Ø Ø¼ µ ÇÈ Ö s performance on benchmark corpora has been promising, e.g. 89.7% labelled recall and precision on the Wall treet Journal treebank [1]. Despite good performance, ÇÈ Ö estimator has been shown to be biased and inconsistent, meaning that the sequence of probability distributions obtained with the estimator does not converge to the distribution that generated the data [8]. Furthermore, it has been shown that ÇÈ Ö s good performance can be attributed to formal constraints on the set of subtrees extracted from the treebank, e.g. an upperbound on subtree depth and another on the number of leaf nodes labeled with a non-terminal symbol in a subtree. These constraints limit ÇÈ Ö s bias, leading to improved performance ([7]). (2) Bonnema s estimator The estimation procedure introduced in [7] assumes that every treebank parse stands for a uniform distribution over all possible derivations that generate that parse in the DOP model. Hence, prior to parameter estimation, every treebank parse is expanded to all derivations that generate it in the DOP model. ubsequently, counting subtree frequency proceeds by decomposing every derivation into the subtrees that participate in it leading to the following estimate: È Ø Ê Ø µ ¾ Æ Øµ Ø Ê Ø µ, where Æ Øµ is the number of non-root nodes of subtree Ø and is the original ÇÈ Ö s relative frequency estimator. This can be interpreted as a ÇÈ Ö estimator with a correction factor ¾ Æ Øµ for subtree size. Large subtrees have their estimate downgraded, and small subtrees are (relatively) upgraded. The estimator defines a new DOP model which we refer to as ÇÈ ÓÒ model. As shown in [9], the Bonnema procedure is also biased. Briefly stated, while the original relative frequency estimator ÇÈ Ö gives too much probability mass to large subtrees, ÇÈ ÓÒ gives an excessive amount of probability mass to small subtrees. (3) Maximum-likelihood Overfitting One might say that ÇÈ Ö estimator is biased because it is not a Maximum-Likelihood (ML) estimator. This is in fact the approach taken in [1], where the Inside-Outside algorithm is used for estimation of the DOP model parameters from a treebank under the assumption that the model has a hidden element (the derivations that generated the parses of the treebank). However, as [7] note, ML for DOP always results in a model that overfits the treebank. Let be given a treebank with trees ½ ¾, both having the same root label. The ML probability assignment to the subtrees extracted from this treebank is given by È ½ Ê ½ µ È ¾ Ê ¾ µ ½ ¾ and zero to all other subtrees. This parameter assignment is ML because it gives the treebank trees the largest possible likelihood of exactly 1. However, this probability assignment yields a DOP model that overfits the treebank, such that probability zero is given to all parses not present in the treebank. 3 A new estimator for DOP In this section, we develop a completely different approach to parameter estimation for DOP than earlier work. Consider the common situation where a subtree 3 Ø is equal to 3 The term subtree is reserved for the tree-structures that DOP extracts from the treebank.

4 a tree generated by a derivation Ø ½ Æ Æ Ø Ò involving multiple subtrees Ø ½ Ø Ò. For example, subtree s17 (Figure 3) can be constructed by different derivations such as (s16 Æ s2), (s14 Æ s1) and (s15 Æ s1 Æ s3). We will refer to subtrees that can be constructed from derivations involving other subtrees with the term complex subtrees. For every complex subtree Ø, we restrict 4 our attention only to the derivations involving pairs of subtrees; in other words, we focus on subtree Ø such that there exist subtrees Ø ½ and Ø ¾ such that Ø Ø ½ Æ Ø ¾ µ. In DOP, the probability of Ø is given by È Ø Ê Ø µ. In contrast, the derivation probability is given by È Ø ½ Ê Ø½ µè Ø ¾ Ê Ø¾ µ. However, according to the chain rule È Ø ½ Æ Ø ¾ Ê Ø½ µ È Ø ½ Ê Ø½ µè Ø ¾ Ø ½ µ. Therefore, the derivation Ø ½ ÆØ ¾ embodies an independence assumption realized by the approximation 5 : È Ø ¾ Ø ½ µ È Ø ¾ Ê Ø¾ µ. This approximation involves a so-called backoff, i.e. a weakening of the conditioning context from È Ø ¾ Ø ½ µ to È Ø ¾ Ê Ø¾ µ. Hence, we will say that the derivation Ø ½ Æ Ø ¾ constitutes a backoff of subtree Ø and we will write (Ø Ø ½ Æ Ø ¾ ) to express this fact. The backoff relation between a subtree and a pair of other subtrees allows for a partial order between the derivations of the subtrees extracted from a treebank. A graphical representation of this partial order is a directed acyclic graph which consists of a node for each pair of subtrees Ø Ø that constitute a derivation of another complex subtree. A directed edge points from a subtree Ø in a node 6 to another node containing a pair of subtrees Ø Ø iff Ø Ø ÆØ. We refer to this graph as the backoff graph. A portion of the backoff graph for the subtrees of Figure 3 would look as follows (where ¼ stands for a subtree consisting of a single node labeled Ë the start symbol): s9 s1 s4 s3 s8 s2 s6 s7 s0 s17 s11 s3 s13 s3 s4 s2.. s16. s2. s14. s1. We distinguish two sets of subtrees: initial and atomic. Initial subtrees are subtrees that do not participate in a backoff derivation of any other subtree. In Figure 3, subtree ½ is the only initial subtree.atomic subtrees are subtrees for which there are no backoffs. In Figure 3, these are subtrees of depth one (double circled in the backoff graph). In the DOP model (under any estimation procedure discussed in section 2), the probablity of a parse-tree is defined as the sum of the probabilities of all derivations that generate this parse-tree. This means that DOP linearly interpolates derivations involving subtrees from different levels of the backoff graph; this is similar to the way 4 Because DOP takes all subtrees of the treebank, if complex subtree Ø has a derivation Ø½ Æ Ø¾ Æ Æ Ø Ò, then the tree resulting from Ø½ Æ Ø¾ is a complex subtree also. For example, in Figure 3, ½ can be derived through (s15 Æ s1 Æ s3); ½ Æ ½ generates subtree ½. Hence, derivations of Ø that involve more than two subtrees can be separarted into (sub)derivations that involve pairs of subtrees, each leading to a complex subtree. Therefore, for any complex subtree Ø, we may restrict our attention to derivations involving only pairs of subtrees i.e., Ø Ø½ Æ Ø¾. 5 Note that Ê Ø¾ is part of Ø½ (the label of the substitution site). 6 In a pair Ø Ø or Ø Ø that constitutes a node.

5 Hidden Markov Models interpolate different Markov orders over, e.g. words, for calculating sentence probability. Hence, we will refer to the different levels of subtrees in the backoff graph as the Markov orders. Backoff DOP Crucially, the partial order over the subtrees, embodied in the backoff graph, can be exploited for turning DOP into a backedoff model as follows. A subtree is generated by a sequence of derivations ordered by the backoff relation. This is in sharp contrast with existing DOP models that consider the different derivations leading to the same subtree as a set of disjoint events. Next we present the estimation procedure that accompanies this new realization of DOP as a recursive backoff over the different Markov orders. Estimation vs. smoothing It is common in probabilistic modeling to smooth a probability distribution È Ø µ by a backoff distribution thereof e.g. È Ø µ. The smoothing of È Ø µ aims at dealing with the problem of sparse-data (whenever the probability È Ø µ is zero). The backoff value È Ø µ can be used as an approximation of È Ø µ under the assumption that Ø and are independent. moothing, then, aims at enlarging the space of non-zero events in the distribution È Ø µ. Hence, the goal of smoothing differs from our goal. While smoothing aims at filling the zero gaps in a distribution, our goal is to estimate the distribution (a priori to smoothing it). Despite these differences, we employ a backoff method for parameter estimation by redistributing probability mass among DOP model subtrees. Katz Backoff The Katz Backoff method [10, 11] is a smoothing technique based on the discounting method of Good-Turing (GT) [12, 11]. Given a higher order distribution È Ø µ, Katz backoff employs the GT formula for discounting from this distribution È leading to È Ì Ø µ. Then, the probability mass that was discounted (½ È Ø Ì Ø µ) is distributed over the lower order distribution È Ø µ. Backoff DOP We assume initial probability estimates È based on frequency counts, e.g. as in ÇÈ Ö or ÇÈ ÓÒ. The present backoff estimation procedure operates topdown, stepwisely, over the backoff graph, starting with the initial and moving down to the atomic subtrees. In essense this procedure transfers, stepwisely, probability mass from complex subtrees to their backoffs. Let È represent the current probability estimate resulting from the previous step (initialy È È ). For every (Ø Ø ½ Æ Ø ¾ ) in the backoff graph, we know that (1) È Ø Ê Ø µ È Ø ½ Ê Ø½ µè Ø ¾ Ø ½ µ and (2) È Ø ½ Æ Ø ¾ µ È Ø ½ Ê Ø½ µè Ø ¾ Ê Ø¾ µ (note that Ê Ø Ê Ø½ ). This mean that È Ø ¾ Ø ½ µ is backedoff to È Ø ¾ Ê Ø¾ µ. Hence, we may adapt the Katz method to estimate the Backoff DOP probability È Ó as follows: È È Ó Ø ¾ Ø Ø Ì ¾ Ø ½ µ «Ø ½ µ È Ø ¾ Ê Ø¾ µ ½ µ «Ø ½ µ È Ø ¾ Ê Ø¾ µ È Ø ¾ Ø ½ µ ¼ ÓØ ÖÛ where «Ø ½ µ is a normalization factor that guarantees that the È sum of the probabilities of subtrees with the same root label is one: «Ø ½ µ ½ È Ø Ø ¾ Ø ½ Ø ¾µ ¼ Ì ¾ Ø ½ µ. Using the above estimate of È Ó Ø ¾ Ø ½ µ, the backoff estimates are calculated as follows: È Ó Ø Ê Ø µ È Ø ½ Ê Ø½ µ È Ø Ì ¾ Ø ½ µ, and È Ó Ø ½ Ê Ø½ µ ½ «Ø ½ µ µè Ø ½ Ê Ø½ µ. Then the current probabilities are updated before the next step of Katz backoff takes place over the next layer in the backoff graph as follows: È Ø ½ Ê Ø½ µ È Ó Ø ½ Ê Ø½ µ.

6 È Note that È Ó is a proper distribution in the sense that for all nonterminals : Ø È Ø µ ½. This is guaranteed by the redistribution of the reserved probability mass at every step of the procedure over the layers of the backoff graph. Furthermore, we note that the present method is not a smoothing method since it applies Katz Backoff for redistributing probability mass only among subtrees that did occur in the treebank. The present method does not address probability estimation for unknown/unseen events. Current implementation The number of subtrees extracted from a tree-bank is extremely large. In this paper, we choose to apply the Katz backoff only to Ø Ø ½ Æ Ø ¾ iff Ø ¾ is a lexical subtree i.e., Ø ¾ Û where is a Part of peech (Po) tag and Û a word. Our choice has to do with the importance of lexicalized subtrees and the overestimation that accompanies their relative frequency. All experiments reported here pertain to applying the backoff estimation procedure to this limited set of subtrees (while the probabilities of all other subtrees are left untouched). 4 Empirical results OI corpus and evaluation metrics OI is a Dutch, speech-based, dialogue system that provides railway time-table information to human users over ordinary telephone lines. The OI corpus contains 10,049 syntactically and semantically annotated utterances which are answers given by users to the system s questions (e.g. From where to where do you want to travel? ). The answers are used to fill in a number of slots that are typical of travel information, such as origin, destination and time. The semantic content of the utterances, expressed in an update language [13], is used to update the system s information state. The OI treebank utterances are annotated by a phrase-structure scheme with syntactic-semantic labels. The corpus was randomly split into two sets: i) a training set with 9,049 trees; ii) a test set with 1,000 trees. The experiments were carried out using the same train/test split. We report results for sentences that are at least two-word long (as 1 word setences are easy). Without 1-word sentences, the average sentence length is 4.6 words/sentence. Three accuracy measures were employed: exact match and recall/precision (F-score) of labeled bracketing [14], to assess individual model performance, and error reduction ratio, the ratio between the percent point improvement of model 1 over model 2 normalized by the global error of model 2, to evaluate inter-model performance. ubtree space was reduced by means of three upper bounds on their shape: 1) depth (d), 2) number of lexical items (l) and 3) number of substitution sites (n). Most Probable Derivation (MPD) was used as the maximization entity to select the preferred parse. Models We tested the new estimator under two different counting strategies: ÇÈ Ö and ÇÈ ÓÒ. The following naming convention was used: ÇÈ Ö (as in [2]), ÇÈ ÓÒ (as in [7]), BO- ÇÈ Ö (backoff estimator applied to ÇÈ Ö frequencies), and BO- ÇÈ ÓÒ (backoff estimator applied to ÇÈ ÓÒ frequencies). Accuracy vs. depth upper bound Figure 4 shows exact match results as a function of subtree depth upper bound. The subtrees were restricted to at most 2 words and 4 substitution nodes (l2n4). This yields small subtree spaces: for depth 7, this corresponds to 172,050 subtrees. For all depth upper bounds, BO- ÇÈ Ö achieved the best results

7 % Exact Match BO-DOP rf DOP rf DOP Bon BO-DOP Bon ubtree depth upper bound Discounted probability mass Total (N1/N) 2-word subtrees 1-word subtrees ubtree depth upper bound Fig. 4. Exact match as a function of subtree depth upper bound (l2n4,mpd). Fig. 5. Probability mass discounted from subtrees as a function of depth upper bound (l2n4). followed by ÇÈ Ö, ÇÈ ÓÒ and BO- ÇÈ ÓÒ. BO- ÇÈ Ö improved on ÇÈ Ö by 1.72 percent points at depth 6, an increase of 2.02%. This corresponds to an error reduction of 11.4%. When compared to ÇÈ ÓÒ, error reduction rose to about 15%. F-score results followed the same pattern. At depth 6, BO- ÇÈ Ö reached 95.33%; ÇÈ Ö, 94.73%; ÇÈ ÓÒ, 94.5% and BO- ÇÈ ÓÒ, 94.33%. Error reduction of BO- ÇÈ Ö with respect to ÇÈ Ö reached 11.3% and with respect to ÇÈ ÓÒ, 15.7%. Probability mass transfer Figure 5 shows discounted probability mass as a function of subtree depth upper bound. The probability mass discounted from 2-word subtrees is bigger than the mass discounted from 1-word subtrees 7. This happens because the number of hapax legomena (subtrees that occur just once) tends to increase for higher d and l upper bounds, since larger subtrees with rare word combinations are allowed into the distribution. The more hapax legomena the higher the discounting rates will be (according to the Good-Turing method). Thus, the probability mass discounted from Ò-word subtrees is, in general, bigger than the mass discounted from (Ò-1)-subtrees. Consequently, the magnitude of the probability transfer across Markov orders gradually decreases as the recursive estimation procedure approaches atomic subtrees. The property of decreasing discounts avoids the pitfall of overestimating small subtrees (c.f. ÇÈ ÓÒ ) and reduces the overestimation of large subtrees (c.f. ÇÈ Ö ). The performance pattern exhibited above remained steady across four different training/test splits. The average over the four splits shows that BO- ÇÈ Ö reduces error compared to ÇÈ Ö and ÇÈ ÓÒ by % Conclusions This paper presented a new estimator for the DOP model which uses the Katz method for redistributing frequency-based probability mass among the subtrees of the model. 7 Ò-word subtrees are subtrees having exactly Ò words in their leaf nodes.

8 We have seen empirical evidence for the improved performance of this estimator (over other existing estimators). An interesting side effect is that the problem of selecting the preferred parse can be achieved in deterministic polynomial-time, since this estimator treats the different derivations of the same parse as backoff alternatives. Future work will address (1) formal aspects of the new estimator (bias and inconsistency questions), (2) a Maximum-Likelihood variant for DOP that incorporates the observations discussed in this paper, and (3) further experiments. References 1. Bod, R.: What is the minimal set of fragments that achieves maximal parse accuracy? In: Proceedings of the 39th Annual Meeting of the ACL (ACL 2001). (2001) 2. Bod, R.: Enriching Linguistics with tatistics: Performance models of Natural Language. PhD dissertation. ILLC dissertation series , University of Amsterdam (1995) 3. Chelba, C., Jelinek, F.: Exploiting syntactic structure for language modeling. In Boitet, C., Whitelock, P., eds.: Proceedings of the Thirty-ixth Annual Meeting of the Association for Computational Linguistics and eventeenth International Conference on Computational Linguistics, an Francisco, California, Morgan Kaufmann Publishers (1998) Charniak, E.: A maximum entropy inspired parser. In: Proceedings of the 1st Meeting of the North American Chapter of the ACL (NAACL-00), eattle, Washington, UA (2000) Black, E., Jelinek, F., Lafferty, J., Magerman, D., Mercer, R., Roukos,.: Towards Historybased Grammars: Using Richer Models for Probabilistic Parsing. In: Proceedings of the 31st Annual Meeting of the ACL (ACL 93), Columbus, Ohio (1993) 6. ima an, K.: Computational complexity of probabilistic disambiguation. Grammars 5(2) (2002) Bonnema, R., Buying, P., cha, R.: A new probability model for data oriented parsing. In Dekker, P., ed.: Proceedings of the Twelfth Amsterdam Colloquium. ILLC/Department of Philosophy, University of Amsterdam, Amsterdam (1999) Johnson, M.: The DOP estimation method is biased and inconsistent. Computational Linguistics 28(1) (2002) Buratto, L.: Back-off as parameter estimation for DOP models. In de Jongh, D., ed.: Master of Logic eries (MoL ). ILLC cientific Publications, Institute for Logic, Language and Computation (ILLC), Amsterdam, The Netherlands (2002) 10. Katz,.: Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, peech and ignal Processing (AP) 35(3) (1987) Chen,., Goodman, J.: An empirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Harvard University (1998) 12. Good, I.: The population frequencies of species and the estimation of population parameters. Biometrika 40 (1953) eldhuijzen van Zanten, G.: emantics of update expressions. Technical report #24, Netherlands Organization for cientific Research (NWO), Priority Programme for peech and Language Technology, ØØÔ Ö Ð Ø ÖÙ ÒÐ ¾½ (1996) 14. Black et al., E.: A procedure for Quantitatively Comparing the yntactic Coverage of English Grammars. In: Proceedings of the February 1991 DARPA peech and Natural Language Workshop, an Mateo, CA., Morgan Kaufman (1991)

Solutions of Implication Constraints yield Type Inference for More General Algebraic Data Types

Solutions of Implication Constraints yield Type Inference for More General Algebraic Data Types Peter J. Stuckey NICTA Victoria Laboratory Department of Computer Science and Software Engineering The University