Announcements. HW3 Due tonight HW4 posted No class Thursday (Thanksgiving) 2017 Kevin Jamieson

Transcription

1 Announcements HW3 Due tonight HW4 posted No class Thursday (Thanksgiving) 2017 Kevin Jamieson 1

2 Mixtures of Gaussians Machine Learning CSE546 Kevin Jamieson University of Washington November 20, 2016 Kevin Jamieson 2017 Kevin Jamieson

3 Mixture models Y Z = {y i } n i=1 is observed data = { i } n i=1 is unobserved data If (x) is Gaussian density with parameters =(µ, 2 )then `( ; Z, ) = nx (1 i) log[(1 ) 1 (y i )] + i log( 2 (y i )] i=1 i( ) =E[ i, Z] = Kevin Jamieson 2017 Kevin Jamieson

4 Mixture models Kevin Jamieson 2017 Kevin Jamieson

5 Gaussian Mixture Example: Start Kevin Jamieson 2017 Kevin Jamieson

6 After first iteration Kevin Jamieson 2017 Kevin Jamieson

7 After 2nd iteration Kevin Jamieson 2017 Kevin Jamieson

8 After 3rd iteration Kevin Jamieson 2017 Kevin Jamieson

9 After 4th iteration Kevin Jamieson 2017 Kevin Jamieson

10 After 5th iteration Kevin Jamieson 2017 Kevin Jamieson

11 After 6th iteration Kevin Jamieson 2017 Kevin Jamieson

12 After 20th iteration Kevin Jamieson 2017 Kevin Jamieson

13 Some Bio Assay data Kevin Jamieson 2017 Kevin Jamieson

14 GMM clustering of the assay data Kevin Jamieson 2017 Kevin Jamieson

15 Resulting Density Estimator Kevin Jamieson 2017 Kevin Jamieson

16 Expectation Maximization Algorithm Observe data x 1,...,x n drawn from a distribution p( ) for some 2 nx log(p(x i )) = i=1 = = i=1 i=1 b MLE = arg max 0 1 nx X p(x i,z i = j ) A j 0 nx X j nx X q i (z i = j 0 ) log i=1 j nx log(p(x i )) i=1 1 q i (z i = j 0 ) p(x i,z i = j ) A q i (z i = j 0 ) p(xi,z i = j ) q i (z i = j 0 ) nx X q i (z i = j 0 ) log (p(x i,z i = j )) + i=1 j (Introduce hidden data zi) (Introduce dummy distribution qi, variable θ ) nx X i=1 j (Jensen s inequality, log() is concave) q i (z i = j 0 ) log( 1 q i (z i =j 0 ) ) Does not depend on θ! Kevin Jamieson 2017 Kevin Jamieson

17 Expectation Maximization Algorithm Observe data X =[x 1,...,x n ] drawn from a distribution p( ) for some 2 nx log(p(x i )) i=1 b MLE = arg max nx log(p(x i )) i=1 nx X q i (z i = j 0 ) log (p(x i,z i = j )) i=1 j True for any choice of 0 and distribution q i (z i = j 0 ) Set q i (z i = j 0 )=p(z i = j 0, X) Kevin Jamieson 2017 Kevin Jamieson

18 Expectation Maximization Algorithm Observe data x 1,...,x n drawn from a distribution p( ) for some 2 nx log(p(x i )) i=1 b MLE = arg max nx log(p(x i )) i=1 nx X p(z i = j 0, X) log (p(x i,z i = j )) =: Q(, 0 ) i=1 j Initial guess for (0), for each step k: E-step: compute Q(, (k) )= nx i=1 i E zi hlog (p(x i,z i )) (k), X M-step: find (k+1) = arg max Q(, (k) ) Kevin Jamieson 2017 Kevin Jamieson

19 Expectation Maximization Algorithm Initial guess for (0), for each step k: E-step: compute M-step: find Q(, (k) )= nx i=1 (k+1) = arg max i E zi hlog (p(x i,z i )) (k), X Q(, (k) ) Example: Observe x 1,...,x n (1 )N (µ 1, 2 1 )+ N (µ 2, 2 2 ) z i = j if i is in mixture component j for j 2 {1, 2} =(,µ 1, E zi [log(p(x i,z i ) (k), X] = p(z i =1 (k),x i ) log p(x i,z i =1 ) = p(z i =1 (k),x i ) log p(x i z i =1, )p(z i =1 ) + p(z i =2 (k),x i ) log p(x i,z i =2 ) + p(z i =2 (k),x i ) log p(x i z i =2, )p(z i =2 ) 2 1,µ 2, 2 2) = (x i µ (k) 1, 2(k) 1 ) (x i µ (k) 1, 2 (k) 1 )+ (x i µ (k) 2, 2 2 (k) ) log (x i µ 1, 2 1)(1 ) + (x i µ (k) 2, 2(k) 2 ) (x i µ (k) 1, 2 (k) 1 )+ (x i µ (k) 2, 2 2 (k) ) log (x i µ 2, 2 2) Kevin Jamieson 2017 Kevin Jamieson

20 Expectation Maximization Algorithm - EM used to solve Latent Factor Models - Also used to solve missing data problems - Also known as Baum-Welch algorithm for Hidden Markov Models - In general, EM is non-convex so it can get stuck in local minima. Kevin Jamieson 2017 Kevin Jamieson

21 Density Estimation Machine Learning CSE546 Kevin Jamieson University of Washington November 20, 2016 Kevin Jamieson 2017 Kevin Jamieson

22 Kernel Density Estimation A very lazy GMM Kevin Jamieson 2017 Kevin Jamieson

23 Kernel Density Estimation Kevin Jamieson 2017 Kevin Jamieson

24 Kernel Density Estimation What is the Bayes optimal classification rule? Predict arg max m br im Kevin Jamieson 2017 Kevin Jamieson

25 Generative vs Discriminative Kevin Jamieson 2017 Kevin Jamieson

26 Basic Text Modeling Machine Learning CSE4546 Kevin Jamieson University of Washington November 20, 2017 Kevin Jamieson 26

27 Bag of Words n documents/articles with lots of text Questions: - How to get a feature representation of each article? - How to cluster documents into topics? Bag of words model: ith document: x i 2 R D x i,j = proportion of times jth word occurred in ith document Kevin Jamieson 27

28 Bag of Words n documents/articles with lots of text Questions: - How to get a feature representation of each article? - How to cluster documents into topics? Bag of words model: ith document: x i 2 R D x i,j = proportion of times jth word occurred in ith document Given vectors, run k-means or Gaussian mixture model to find k clusters/topics Kevin Jamieson 28

29 Nonnegative matrix factorization (NMF) A 2 R m n A i,j = frequency of jth word in document i Nonnegative Matrix factorization: min ka WH T k 2 F W 2 R m d +,H 2 R n d + d is number of topics Also see latent Dirichlet factorization (LDA) Kevin Jamieson 29

30 Nonnegative matrix factorization (NMF) A 2 R m n A i,j = frequency of jth word in document i Nonnegative Matrix factorization: min ka WH T k 2 F W 2 R m d +,H 2 R n d + d is number of topics Each column of H represents a cluster of a topic, Each row W is some weights a combination of topics Also see latent Dirichlet factorization (LDA) Kevin Jamieson 30

31 Word embeddings, word2vec Previous section presented methods to embed documents into a latent space Alternatively, we can embed words into a latent space This embedding came from directly querying for relationships. word2vec is a popular unsupervised learning approach that just uses a text corpus (e.g. nytimes.com) Kevin Jamieson 31

32 Word embeddings, word2vec slide: Kevin Jamieson 32

33 Word embeddings, word2vec Training neural network to predict co-occuring words. Use first layer weights as embedding, throw out output layer slide: Kevin Jamieson 33

34 Word embeddings, word2vec e hx ants,y car i X i e hx ants,y i i Training neural network to predict co-occuring words. Use first layer weights as embedding, throw out output layer slide: Kevin Jamieson 34

35 word2vec outputs king - man + woman = queen country - capital slide: Kevin Jamieson 35

36 TF*IDF n documents/articles with lots of text How to get a feature representation of each article? 1. For each document d compute the proportion of times word t occurs out of all words in d, i.e. term frequency TF d,t 2. For each word t in your corpus, compute the proportion of documents out of n that the word t occurs, i.e., document frequency DF t 3. Compute score for word t in document d as 1 TF d,t log( ) DF t Kevin Jamieson 36

37 BeerMapper - Under the Hood Algorithm requires feature representations of the beers {x 1,...,x n } R d Two Hearted Ale - Input ~2500 natural language reviews Reviews for each beer Bag of Words weighted by TF*IDF Get 100 nearest neighbors using cosine distance Non-metric multidimensional scaling Embedding in d dimensions

38 BeerMapper - Under the Hood Algorithm requires feature representations of the beers {x 1,...,x n } R d Two Hearted Ale - Weighted Bag of Words: Reviews for each beer Bag of Words weighted by TF*IDF Get 100 nearest neighbors using cosine distance Non-metric multidimensional scaling Embedding in d dimensions

39 BeerMapper - Under the Hood Algorithm requires feature representations of the beers {x 1,...,x n } R d Weighted count vector for the ith beer: z i 2 R 400,000 Cosine distance: d(z i,z j )=1 z T i z j z i z j Two Hearted Ale - Nearest Neighbors: Bear Republic Racer 5 Avery IPA Stone India Pale Ale (IPA) Founders Centennial IPA Smuttynose IPA Anderson Valley Hop Ottin IPA AleSmith IPA BridgePort IPA Boulder Beer Mojo IPA Goose Island India Pale Ale Great Divide Titan IPA New Holland Mad Hatter Ale Lagunitas India Pale Ale Heavy Seas Loose Cannon Hop3 Sweetwater IPA Reviews for each beer Bag of Words weighted by TF*IDF Get 100 nearest neighbors using cosine distance Non-metric multidimensional scaling Embedding in d dimensions

40 BeerMapper - Under the Hood Algorithm requires feature representations of the beers {x 1,...,x n } R d Find an embedding {x 1,...,x n } R d such that x k x i < x k x j whenever d(z k,z i ) <d(z k,z j ) for all 100-nearest neighbors. (10 7 constraints, 10 5 variables) distance in 400,000 dimensional word space Solve with hinge loss and stochastic gradient descent. (20 minutes on my laptop) (d=2,err=6%) (d=3,err=4%) Could have also used local-linear-embedding, max-volume-unfolding, kernel-pca, etc. Reviews for each beer Bag of Words weighted by TF*IDF Get 100 nearest neighbors using cosine distance Non-metric multidimensional scaling Embedding in d dimensions

41 BeerMapper - Under the Hood Algorithm requires feature representations of the beers {x 1,...,x n } R d Reviews for each beer Bag of Words weighted by TF*IDF Get 100 nearest neighbors using cosine distance Non-metric multidimensional scaling Embedding in d dimensions

42 BeerMapper - Under the Hood Algorithm requires feature representations of the beers {x 1,...,x n } R d Pilsner IPA Sanity check: styles should cluster together and similar styles should be close. Light lager Pale ale Blond Amber Brown ale Doppelbock Belgian light Wit Belgian dark Lambic Wheat Porter Stout Reviews for each beer Bag of Words weighted by TF*IDF Get 100 nearest neighbors using cosine distance Non-metric multidimensional scaling Embedding in d dimensions

43 BeerMapper - Under the Hood Algorithm requires feature representations of the beers {x 1,...,x n } R d Pilsner IPA Sanity check: styles should cluster together and similar styles should be close. Light lager Pale ale Blond Amber Brown ale Doppelbock Belgian light Wit Belgian dark Lambic Wheat Porter Stout Reviews for each beer Bag of Words weighted by TF*IDF Get 100 nearest neighbors using cosine distance Non-metric multidimensional scaling Embedding in d dimensions

44 Feature generation for images Machine Learning CSE4546 Kevin Jamieson University of Washington November 20, 2017 Kevin Jamieson 44

45 Contains slides from LeCun & Ranzato Russ Salakhutdinov Honglak Lee Google images Kevin Jamieson 45

46 Convolution of images (Note to EEs: deep learning uses the word convolution to mean what is usually known as cross-correlation, e.g., neither signal is flipped) Image I Filter K I K Slide credit: Kevin Jamieson 46

47 Convolution of images (Note to EEs: deep learning uses the word convolution to mean what is usually known as cross-correlation, e.g., neither signal is flipped) K I K Image I Kevin Jamieson 47

48 Convolution of images Input image X flatten into vector 2 3 vec(h 1 X) 6 4 vec(h 2 X) 7 5. filters H k convolved image H k X Kevin Jamieson 48

49 Stacking convolved images filters Kevin Jamieson 49

50 Stacking convolved images filters Apply Non-linearity to the output of each layer, Here: ReLu (rectified linear unit) Other choices: sigmoid, arctan Kevin Jamieson 50

51 Pooling Pooling reduces the dimension and can be interpreted as This filter had a high response in this general region 27x27x64 14x14x64 Kevin Jamieson 51

52 Pooling Convolution layer 27 14x14x filters Convolve with 64 6x6x3 filters MaxPool with 2x2 filters and stride 2 Kevin Jamieson 52

53 Full feature pipeline 27 14x14x filters Convolve with 64 6x6x3 filters MaxPool with 2x2 filters and stride 2 Flatten into a single vector of size 14*14*64=12544 How do we choose all the hyperparameters? How do we choose the filters? - Hand design them (digital signal processing, c.f. wavelets) - Learn them (deep learning) Kevin Jamieson 53

54 Some hand-created image features SIFT Spin Image HoG RIFT Texton GLOH Slide from Honglak Lee Kevin Jamieson 54

55 ML Street Fight Machine Learning CSE546 Kevin Jamieson University of Washington November 20, 2017 Kevin Jamieson 2017 Kevin Jamieson

56 Mini case study Inspired by Coates and Ng (2012) Input is CIFAR-10 dataset: examples of 32x32x3 images 1. Construct set of patches by random selection from images 2. Standardize patch set (de-mean, norm 1, whiten, etc.) 3. Run k-means on random patches 4. Convolve each image with all patches (plus an offset) 5. Push through ReLu 6. Solve least squares for multiclass classification 7. Classify with argmax Kevin Jamieson 56

57 Mini case study Methods of standardization: Kevin Jamieson 57

58 Mini case study Dealing with class imbalance: Kevin Jamieson 58

59 Mini case study Dealing with outliers: Kevin Jamieson 59

60 Mini case study Dealing with outliers: `huber (z) = ( 1 2 z2 if z apple 1 1 z 2 otherwise arg min 0 X j i=1 k(x i,x j ) j y i 1 A 2 + X i,j i j k(x i,x j ) arg min nx i=1 `huber X j k(x i,x j ) j y i 1 A + X i,j i j k(x i,x j ) Kevin Jamieson 60

61 Mini case study Dealing with hyperparameters: Kevin Jamieson 61

62 Hyperparameter Optimization Machine Learning CSE546 Kevin Jamieson University of Washington November 20, 2017 Kevin Jamieson 2017 Kevin Jamieson

63

64 N out = 10 N hid Training set Eval \ set N in = 784 hyperparameters learning rate 2 [10 3, 10 1 ] `2-penalty 2 [10 6, 10 1 ] #hiddennodesn hid 2 [10 1, 10 3 ]

65 N out = 10 N hid Training set Eval \ set N in = 784 bf Hyperparameters (10 1.6, , ) (10 1.0, , ) (10 1.2, , ) (10 2.4, , ) (10 2.6, , ) (10 2.7, , ) (10 1.8, , ) (10 1.4, , ) (10 1.9, , ) (10 1.8, , ) Eval-loss hyperparameters learning rate 2 [10 3, 10 1 ] `2-penalty 2 [10 6, 10 1 ] #hiddennodesn hid 2 [10 1, 10 3 ]

66 N out = 10 N hid Training set Eval \ set N in = 784 bf Hyperparameters (10 1.6, , ) (10 1.0, , ) (10 1.2, , ) (10 2.4, , ) (10 2.6, , ) (10 2.7, , ) (10 1.8, , ) (10 1.4, , ) (10 1.9, , ) (10 1.8, , ) Eval-loss hyperparameters learning rate 2 [10 3, 10 1 ] `2-penalty 2 [10 6, 10 1 ] #hiddennodesn hid 2 [10 1, 10 3 ]

67 N out = 10 N hid Training set Eval \ set N in = 784 Hyperparameters (10 1.6, , ) (10 1.0, , ) (10 1.2, , ) (10 2.4, , ) (10 2.6, , ) (10 2.7, , ) (10 1.8, , ) (10 1.4, , ) (10 1.9, , ) (10 1.8, , ) Eval-loss hyperparameters learning rate 2 [10 3, 10 1 ] `2-penalty 2 [10 6, 10 1 ] #hiddennodesn hid 2 [10 1, 10 3 ]

68 N out = 10 N hid Training set Eval \ set N in = 784 Hyperparameters (10 1.6, , ) (10 1.0, , ) (10 1.2, , ) (10 2.4, , ) (10 2.6, , ) (10 2.7, , ) (10 1.8, , ) (10 1.4, , ) (10 1.9, , ) (10 1.8, , ) Eval-loss How do we choose hyperparameters to train and evaluate?

69 How do we choose hyperparameters to train and evaluate? Grid search: Hyperparameters on 2d uniform grid

70 How do we choose hyperparameters to train and evaluate? Grid search: Hyperparameters on 2d uniform grid Random search: Hyperparameters randomly chosen

71 How do we choose hyperparameters to train and evaluate? Grid search: Hyperparameters on 2d uniform grid Random search: Hyperparameters randomly chosen Bayesian Optimization: Hyperparameters adaptively chosen 10 2

72 Bayesian Optimization: How does it work? Hyperparameters adaptively chosen

73 Recent work attempts to speed up hyperparameter evaluation by stopping poor performing settings before they are fully trained. Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arxiv: , Alekh Agarwal, Peter Bartlett, and John Duchi. Oracle inequalities for computationally adaptive model selection. COLT, Domhan, T., Springenberg, J. T., and Hutter, F. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In IJCAI, András György and Levente Kocsis. Efficient multi-start strategies for local search algorithms. JAIR, 41, Li, Jamieson, DeSalvo, Rostamizadeh, Talwalkar. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. ICLR Hyperparameters Eval-loss (10 1.6, , ) (10 1.0, , ) (10 1.2, , ) (10 2.4, , ) (10 2.6, , ) eval-loss How computation time was spent? (10 2.7, , ) (10 1.8, , ) (10 1.4, , ) (10 1.9, , ) (10 1.8, , ) epochs

74 Hyperparameter Optimization In general, hyperparameter optimization is non-convex optimization and little is known about the underlying function (only observe validation loss) Your time is valuable, computers are cheap: Do not employ grad student descent for hyper parameter search. Write modular code that takes parameters as input and automate this embarrassingly parallel search. Use crowd resources (see pywren) Tools for different purposes: - Very few evaluations: use random search (and pray) or be clever - Few evaluations and long-running computations: see refs on last slide - Moderate number of evaluations (but still exp(#params)) and high accuracy needed: use Bayesian Optimization - Many evaluations possible: use random search. Why overthink it?