ICLR14: O Alsharif: End-to-End Text Recognition with Hybrid HMM Maxout Models

What this episode is about

The paper presents a novel approach to end-to-end text recognition in natural images using hybrid HMM Maxout models, demonstrating that careful engineering in deep learning can achieve state-of-the-art accuracy. It highlights the complexities of text detection and recognition in natural scenes compared to structured documents, aiming to bridge this gap while addressing challenges related to high accuracy and low inference complexity.

Key takeaways

• Bridging document and natural image text recognition is ambitious but essential for practical machine learning applications.
• The true challenge: real-world text often defies our neat, document-based assumptions—it's messy and unpredictable.
• Character confusion, especially between upper and lowercase, highlights the need for context in text recognition.
• Modern models blend speed and accuracy, but lexicon reliance can complicate inference in text recognition tasks.
• Segmentation strategies in recognition—dependent versus independent—show that approach matters as much as the algorithm.

Who speaks in this episode?

• O Alsharif (Host) · 12 min speaking
• Ian (Guest) · 1 min speaking
• Aaron (Guest) · 1 min speaking
• Joshua (Guest) · 1 min speaking

What is discussed, and when

1. 0:00 – 3:00: SVM
2. 3:00 – 6:00: ICML 2013
3. 9:00 – 12:00: CRF
4. 12:00 – 13:28: CRF, GTM, Attorney General

Notable quotes from "ICLR14: O Alsharif: End-to-End Text Recognition with Hybrid HMM Maxout Models"

• Really do such a good job. So there's a bunch of classifiers that you could select from. You could have SPMs.

  about SVM · at 2:28

• So there's been lots of hybrid models going back to the early 90s for speech recognition handwriting recognition various other things, including CRF and HM and what you see.

  about CRF · at 11:06

• Globally as well with you know two different the GTM paper of course yeah We were not exactly sure on the GT.

  about GTM · at 11:55

Who and what is mentioned in this episode?

Concepts & topics

• SVM — Concept · Key argument · 1 mention · first at 2:34
  Discussed as one of the classifiers considered for character recognition, highlighting its limitations in providing natural probabilities.
• CRF — Concept · Framework · 2 mentions · first at 11:57
  Mentioned as a method used in earlier works for text recognition, contrasting with the current approach.
• GTM — Concept · Framework · 1 mention · first at 12:02
  Referenced as a model considered but not used due to lack of related works on image recognition.
• Attorney General — Concept · Referenced dataset · 1 mention · first at 12:11
  Mentioned in passing as a dataset where GTMs were tested, but clarified to be different from the current work's focus.
  GTMs. So, we were not sure if the results would work as well. And they were not tested on these data sets. I think GTMs were tested on MS. No, they're they're tested on check images.

Places & events

• ICML 2013 — Event · Referenced event · 1 mention · first at 3:08
  Mentioned as the conference where the max out activation function paper was presented.
  Activation function that groups that kind of pulls different nodes with a max function. Recent it's an ICML 2013 paper from Ian and other guys at interested in what and it can…

Full transcript with timestamps

0:03 All right. So, uh this is work that I did with Joel a while earlier and uh it's a pure application paper. The gist of it is with some careful engineering in deep models, you can get state-of-the-art results on character recognition, general text recognition in natural images. So, uh I find it hard to convince people that this is a difficult problem by if you take a look at text, I mean usually it's quite difficult to tell. Well, it's quite difficult to recognize task and that's for a variety of reasons. A bunch of machine learning problems are here. It's difficult to detect and once it's difficult to detect, there's a sequencing problem kind of related to voice recognition and after and to do that you need some very good object recognizers which are I mean combats are going to appear in this talk at one point. So usually people ask me if this is solved or not and of course the answer is no.

0:52 It's kind of considered to be soft in the document state where we can make assumptions about the way text is positioned. We can make assumptions about the way it looks. It's black on white backgrounds is structured in lines etc etc. In natural images we cannot make assumptions well we do make some implicit assumptions that text is horizontal. Other than that we cannot make many. So the goal in for this work is to try and bridge the gap between documents and natural scenes. Of course that's overly ambitious and for that we will need to do three these three main tasks.

1:27 We will need to detect text. We will need to recognize text and for that we will need to recognize characters really well. The technical goal of this is that we would like high accuracy but not just that. Usually when people do text recognition they have lexicons because well it's a very confusing problem and we would like low inference complexity relative to lexicon size. So our work is well at least as as far as I understand it bridges the point between these two. You have models of literature that are highly accurate. Models are fast. This is highly accurate and fast. So I'll start from the bottom up. I'll start presenting the character recognizer, the word recognizer and then the text detector and how these things work. So character recognition, it's a standard 62 classification problem. It just tends to be so confusing because well uppercase and lowerase characters tend to be to look the same and without

2:21 context information which is are kind of not so permissible when you're testing on benchmark data sets uh you cannot really do such a good job. So there's a a bunch of classifiers that you could select from. You could have SPMs. Recently there's a paper on deformable parts models. You can have comp nets. We would have liked something that provides natural probabilities. And natural just means that it's not like an SPM where you where you fit a logistic on top of the SVM to get probabilities. And that because we're going to plug this in into another model later. And for our particular compound, we made use of max out for pragmatic reasons. So if you're not familiar with max out, it's an activation function that groups that kind of pulls uh different nodes with a max function. Recent it's an ICML 2013 paper from uh Ian and other guys at uh interested in what and it can be thought of as uh linear layers followed by

3:20 cooling layers which is uh the easiest way I find it to explain to people and the variant particular variant we use has convolutions and drawout. So I mean how well does that work? Well, it's a bit better than the state-ofthe-art. So to get that, we're testing on the standard 2013 2003 data set. So it has like 6,000 characters. There's as minimal pre-processing as you can get away with. And this is like the optimum razor of stuff. We experimented with a ton of pre-processing segmentation whatever you can, grab cuts, everything you can think of. And usually just standard contrast normalization works best.

3:59 uh we did data augmentation that all the other methods did that so it's a fair comparison and if you do the standard trick you chop off the last layer and you replace it with something like an SVM with an RVF kernel you could uh inch accuracy a bit higher it seems to be neat if you look at the confusion matrix you can see that these two lines essentially that say most of the confusion is coming from uppercase and lowerase characters being confused with each other so we're not sure how to get around that without context So based on that we would like to go for query recognition which is the bulk of this. So we would like to recognize images and here's a you know a scary instance and there are many methods that people go about doing this. Some people would like to segment. Some people like segmentation free method in which they segment and recognize simultaneously.

4:49 Others like segmentation dependent. You just segment and then recognize. And usually people depend a lot on lexicons again because this is highly confusing and people do other kinds of things. People have done have used complex for this problem. People have used conditional random fields and pictorial structures. So our particular approach is a segmentation dependent and it's initially lexicon free but we phrase our method such that you can plug in lexicons while you're doing inference or as opposed to inference tool and it turns out it works really really well.

5:22 So the stages to get this to work, we would like to first segment the word into characters and we do that with a hybrid HML model which you're going to see next. We build what we call cascade and then we have a beam search variant on top. So the hybrid HML model um quite famous in early word recognition early uh speech recognition works except in speech recognition people try to recognize characters im uh phone names immediately while they're looking at the speech sequences. here it didn't work out for us so well. So what we did our hybrid model segments the word segments a word into a character versus no character sections and then when you do that you will get well this is just using standard base rule this is improper but it works and people don't.

6:09 So one one instance of output from such a hybrid HML model is this middle line and you can see that sometimes the hybrid model confuses things and we see things that we refer to as over segmentation or under segmentation. Now under segmentation happens when you have two V's and the segment thinks oh it's a W and over segmentation happens in the other way around. So how do we get around that? We just use the simplest rule you can get away with. We chop up every every character from the recognized things into two and we combine adjacent notes in the graph if the probability of the joint the joint image is higher as a character than the probability of either of its con constituents. And that forms a graph an adjacent an adjacency graph on the word which we would like to do inference on.

6:58 So to formalize this we have a bunch alphabet character we have a bunch of intervals and an interval is one of these things. This is an interval for instance and uh we would like to compute the probability of an interval character pair a word in uh ending in an interval and a character and for that without a language model or with a language model you could have a very simple dynamic programming approach. Problem is this doesn't really scale well when you want engram language models and our solution for that which a variant of beam search and here's the algorithm. And it's just a picture. It's not meant to be read.

7:37 What what what you should take away from this is that this does inference in a time constant to the size of your lexicon because it's bounded by the beam width by the things you're allowed to expect and it allows us to use higher order language models and it's quite tunable and we have some graphs that I might be able to show later. So how well does this work? Well, we test this on two data sets on they're in SBT and we have different lexicon sizes which is standard except the large. Nobody uses that for pragmatic reasons. And uh we have two testing schemes. The one we allow a language model to be used. We use the for in here and uh the other standard technique is that we do nearest neighbor search on edit distance after it prints without any language model.

8:23 And in both cases, we beat the previous state-of-the-art. Uh so here are the results down the bottom comparing language model and edit distance neighbor and the distance worked best, but the language model still outperforms the state of the previous state. And it's nice that we're consistent. So we outperform on SBT and PDR, which the previous one, the Wang Ital 2012 doesn't. So then plugging this into the end to end system, we would like to localize text and we have a pure vision approach because we were not sure how to make this fast enough and I guess you'll hear more about that from the Google presentation tomorrow. We would like to split sentences into words, recognize words and then prune. All of these are kind of standard steps. More or less this particular pipeline is not. We start by extracting potential characters using maximally stable extreal regions.

9:12 And you can think of these as kind of sort of things that maybe work which are local minimas and maximus in the image manifold. We then group these things together with a clustering algorithm DB scan essentially. You get things like this. These are possible sentences. We have to chop off sentences into words. And for that we train another hybrid HML model trained on a different data set train to sequence words from sentences. We recognize each of these possible words and then we prune on the recognition accuracy and we do not match suppression. And how well does that work?

9:48 Well, works pretty well so far. And here are some examples. We tested also on the end to end data set and SVT which is also standard and uh it seems to be working neatly. This was optimized on the F measure. So sometimes there are some weird things that happens like this word not being detected. There's a bunch of other examples here where for instance you can see that some neat mistakes like here not being able to recognize this or that. So if the gist of this talk is very simple with some careful engineering and deep models we can get state-of-the-art text recognition with hybrid HML models we could use them for text recognition similar to speech but not exactly as is. And that means that we could probably use recurrent neural lines and bid birectional recurrent neural lines. Well, we just haven't tried it yet. And our work does very large lexicons pretty easily. So that we thought that was pretty neat. And uh

10:48 like to thank Ian, Aaron, and Joshua for pointing out some things that are were inconsistent and uh well for their helpful comments. So thank you. So there's been lots of hybrid models going back to the early 90s for speech recognition handwriting recognition various other things, you know, including CRF and HM and what you see. So what is there any sort of difference in terms of the methods or is it just the application that's different? Uh so we were trying to dig deep into these hybrid models and it seems like for this particular natural X repetition thing, nobody really tried them for some reason. And uh if you look at the literature, they tried it with handwriting stuff. um and the particular structure of the hybrid model diff difference diff difference diff difference diff difference differs so some people were trying voiceike approach where you have multiple characters and you sequence immediately

11:38 and that didn't work for us so I'm not sure what you mean by that so for example what you describe as the the graph where you you know you combine the probabilities of pieces and decide which uh we call this a segmentation graph so the 1998 paper with Joshua and Patrick Hefner uses this technique is trained globally as well uh with you know two different the GTM paper of course uh yeah We were not exactly sure on the GT. Well, we considered the GTM paper. The problem was that we haven't seen any other work related to that. So, we couldn't find anyone else that used GTMs. So, we were not sure if the results would work as well. And they were not tested on these data sets. I think GTMs were tested on MS. No, they're they're tested on check images.

12:18 So, basically introduce strings of uh this is not mis it's different. So they're also tested on um online handwriting recog checks and online handwriting recognition. So we did not find any works I guess on images and for some reason well this gap apparently was needed to be bridged. So we kind of hoped that we did that also. Well yeah it's kind of interesting because there's a bunch of earlier words doing CRFs and whatnot and if this predates CRFs by the way. Uh, I know, I know, I know. Um, but then that makes all of these obsolete, which is kind of interesting.

12:55 Um, it's the way things are, I guess. So perhaps you could say something about how this relates to the Google tomorrow. Yes. Um, I'm not sure exactly. I'd like to see that. I'd like to hear their explanation. Looking forward to that. Any other questions? Okay. Well, let's thank We again

Questions about this episode

Who speaks in "ICLR14: O Alsharif: End-to-End Text Recognition with Hybrid HMM Maxout Models"?

O Alsharif (Host), Ian (Guest), Aaron (Guest), Joshua (Guest).

What is "ICLR14: O Alsharif: End-to-End Text Recognition with Hybrid HMM Maxout Models" about?

The paper presents a novel approach to end-to-end text recognition in natural images using hybrid HMM Maxout models, demonstrating that careful engineering in deep learning can achieve state-of-the-art accuracy. It highlights the complexities of text detection and recognition in natural scenes compared to structured documents, aiming to bridge this gap while addressing challenges related to high…

What is mentioned in this ICLR episode?

5 people, works and ideas are mentioned, including SVM, CRF, ICML 2013, GTM, Attorney General. Each one on this page links to its own page with every episode that mentions it.

How long is "ICLR14: O Alsharif: End-to-End Text Recognition with Hybrid HMM Maxout Models"?

The episode runs 13 min; the transcript is about 2,400 words in 18 timestamped sections.

More from ICLR

• All analyzed episodes
• People mentioned on ICLR
• Topics mentioned on ICLR