A Novel Speech to Text Model for Children

A paper I wrote a while ago when we were developing our automatic speech engine. We never submitted it for publication. Thought I would share it here and anyone is free to use anything from it.

1. Introduction

Today, the speech recognition technology is facing challenges for the development of children’s applications in entertainment and education [1]. It has been described that error rates in word recognition can be 100% worse for children’s speech when compared with adults’ speech [2]. Especially, the speech of very young children is more varying [3] and performance degradation because of bandwidth reduction is more marked in case of children’s speech [4]. Since last few years, Speech recognition for grown-ups has been improved drastically. However, there has been very less advancement in the speech recognizing for children [5, 6]. There is a number of factors which make identification of the children’s speech very challenging. The ability of children to realize speech sounds accurately alters as they learn to speak [7, 8].

Spectrally, smaller vocal tracts of children contribute to high-pitched formant and fundamental frequencies. Children’s overall speaking rate is slower compared to adults, and they have unevenness in speech spontaneity, the rate of speaking, and vocal effort [9]. Children are more likely to use “incorrect pronunciations, ungrammatical phrases, and imaginative words” [10]. It was shown that variation in performance could be decreased on a digit recognition task by training on children’s speech. However, accuracy is still not as good as adults [11]. Adult-targeted automatic speech recognition (ASR) has advanced significantly in these years and can develop speech-to-text (STT) output with much-decreased word-error-rate. This is available for multiple languages and in different types of noisy conditions, such as living-room, outdoor-noise, car noise, etc. Although, when it concerns to child speech, there is little advancement compared to the adult targeted ASR.

The main difference between the speech of children and adults is the length of the vocal track. The reason behind this is bodily differences between both age groups. Also, some researchers studied the effects of age on the quality of sound [12]. They found that the quality of sound differs in 7, 10, 13 and 15-year-old children and adults in the first two sound spectrum’s frequency band, F1 and F2. It proves that there is a linear decrease in values of F1 and F2 bands as age increases. To overcome this challenge, developers have started using frequency warping or Vocal Tract Length Normalisation (VTLN) on the condition of no bandwidth restrictions [12, 13]. This is used to manipulate the speech quality differences between different age groups.

1.1 Generic Adaptation

ASR is more upgraded for adult systems because the number of users in this age group and most of the research has been performed in the area of ASR for adults. Nowadays, generic adaptation techniques are applied for the best performance of children’s speech recognition. For example, Maximum A Posteriori (MAP) adaptation [14] or Maximum Likelihhod Linear Regression (MLLR) adaptation [15] or variants or combinations of both. These techniques are mainly useful for reducing the error rates of ASR for children.

1.2 Challenges in ASR for Children’s Speech

Even though the availability of frequency warping, MAP, and MLLR techniques, they have their limitations like both MAP and MLLR are simply eliminate mismatches between the theoretical model and observed speech date. These techniques do not consider the actual causes between mismatches. The success rate is strictly dependent on the error rates correction obtained with these techniques when compared between trained models on adults’ speech and trained models on ‘matched’ children’s speech. However, the performance is still poorer in case of ‘matched’ models.

The answer to the above challenges is the use of deep neural networks (DNNs). Google and Microsoft have started the use of DNNs in their application development and production systems. Various ASR frameworks have also been introduced for the purpose of research in recent years. These frameworks are easily accessible at small scale research systems such as HTK [16], SONIC [17, 18], SPHINX [19, 20], RWTH [21], JULIUS [22], KALDI [23], the more recent ASR framework SIMON [24], and the relatively novel system known as BAVIECA [25]. Still, there is an urge for the huge database in order to develop an ASR for spontaneous, continuous, and naturally spoken Deep Neural Networks (DNNs) for child speech. Mainly, there is a scarcity of the training data available which was a major problem with bi-directional LSTM RNN models. Even though, scientists have reported that these models could be trained with a large output vocabulary without the need of any speech decoding [26]. Researchers have demonstrated that bi-directional LSTM RNN models allow us to build a neural speech recognizer when combined with the large training dataset. Still, it cannot be compared with the sub-word phone-based recognizer.

Kadho utilizes DNNs in order to educate and prepare children speech ASR experiments. We have also incorporated cross utterance word history data and dynamic n-gram language modeling system. The present report addresses Kadho neural speech engine on English words and phrases recognition by native Chinese children.

The DNN is aimed to cater posterior probability guesses for the HMM states in a DNN-HMM hybrid system. Especially, for an observation o_ut standing for the time t in utterance u, the output y_ut(s) of the DNN for the HMM states is found by the softmax activation function as below.

The recognizer utilizes a pseudo log of state ‘s’ given observation o_ut as below,

The maximum mutual information (MMI) criterion utilized in ASR [9] is the common information between the probability distributions of word sequences and the observation. With O_u={O_u1,...,O_uT_u} as the all observations’ sequence, and W_u as the word sequence in the reference for utterance (u), the MMI criterion is as below:

Differentiating (3) with respect to the log-likelihood log p(o_ut|r) for state r, we get:

The required gradient with respect to the activations is found as:

2. Kadho Neural Network Speech Engine

Karel Vesely [27] has used three techniques for the execution of DNNs; 1) layer-wise pre-training based on Restricted Boltzmann Machines (RBMs) [28]; (2) per-frame cross-entropy training; and (3) sequence-discriminative training [29-30]. It is important to note that these systems are developed based on LDA, MFCC, MLLT, and fMLLR methods. A single GPU using Compute Unified Device Architecture is used to process whole DNN training. Still, the CUDA matrix library is projected to run on machines also without a GPU at very slower rates.

It has been described that the acoustic models of one language can be used for training of models for another language which is known as cross-language bootstrapping [31]. Here, phonemes in the source and target language are mapped to retrain the source language acoustic model on new transliterated training audio data in the target language. The main advantage is a smaller amount of acoustic training data required. In the above-mentioned report, two principle ways of gaining at a phoneme mapping have been demonstrated. We can extrapolated this for children’s speech and adult’s speech as two different languages. Here, the source and target language mapping can be done manually or can be extracted directly from the available data with the help of a target language phoneme recognizer. Manually constructed mapping is very simple to use, so it has been preferred over the automatically derived mapping even though having a better performance. This nominal difference can be overcome after a number of cycles of retraining. Mapping is required to preserve the symbols of both the languages. Once the mapping is performed acoustic models of both languages can be used together for retraining. Unsupervised Training is mainly aimed to amend the model by recognition and retraining with the use of training data in case of absence of manual data. Notably, it is crucial to utilize the confidence measures in order to measure the contributions of correctly accredited audio data for the modeling. This is required to have an efficient use of available acoustic data.

2.1 Pre-training phase

The execution of layer-wise Restricted Boltzmann Machine (RBM) pre-training is complying “www.cs.toronto.edu/~hinton/absps/guideTR.pdf” [28]. The training algorithm is incompatible Divergence with one-step of the Markov Chain Monte Carlo sampling (CD-1). The hyperparameters of the recipe were adjusted on the 100 hours Switchboard subset. Mainly the number of epochs (N) requires adjusting to 100 hours/set_size if smaller databases are used. It is adequate to furnish single data-directory with input features as the training is unattended. There is a high risk of weight-explosion when training the RBM with Gaussian-Bernoulli units. Especially thousands of hidden neurons and with larger learning rates. A mechanism has been executed to avoid weight-explosion, which equates the variance of the reconstruction data with the variance of training data in a mini-batch. The weights are reduced, and the learning rate is temporarily decreased if the reconstruction variance is > 2x larger, t.

2.2 Frame-level cross-entropy training

A DNN which separates frames into triphone-states is trained in this phase. This is performed by mini-batch Stochastic Gradient Descent⁵. The default is to use Softmax output units, fully connected layers AffineTransform and Sigmoid hidden units. The learning rate is 0.008 by default with the size of mini-batch 256, and no regularization of momentum is utilized. The optimal learning-rate varies with the type of hidden units, the value for tanh is 0.00001, and sigmoid is 0.008. The input_transform and the pre-trained Deep Belief Network (i.e., DBN, a stack of RBMs) is communicated into to the script practicing the options ‘– dbn’ and ‘–input-transform’; only the output layer is formatted indiscriminately. An early stopping criterium is utilized to forbid over-fitting. The objective function on the held-out set (i.e., cross-validation set) is assessed for this. Hence, two pairs of feature-alignment dirs are required to execute the supervised training.

2.3 Sequence-discriminative training

Here, the neural network is aimed to relegate correctly the complete sentences which is similar to the general ASR objective than frame-level training. The sequence-discriminative training aims to maximize the anticipated accuracy of state labels deduced from reference transcriptions: lattice framework is used to present competing hypothesis. The training is done by Stochastic Gradient Descent (SGD) with small learning rate: 1e-5 kept constant, per-utterance updates, and 3-5 epochs. Faster convergence when regenerating lattices after a 1st epoch is detected. MMI, sMBR⁶, BMMI, and MPE training are all supported (see [32] for all acronyms references). Even if sMBR is a little bit better, all the techniques execute very similarly. In sMBR optimization, silence frames are omitted from gathering approximate accuracies⁷[27]. LDA: Linear Discriminant Analysis; CMN: Cepstral Mean Normalization; fMLLR: feature space Maximum Likelihood Linear Regression; MFCC: Mel-Frequency Cepstral Coefficients; MLTT: Maximum Likelihood Linear Transform.

2.4 Data Collection and Language Modeling

We have used our lexicon on CMUdict with stress markers eliminations, and the pronunciations for out-of-vocabulary (OOV) words were generated with the Sequitur G2P toolkit. The audio sections were divided into small parts of up to 30 minutes in length to prevent problems with long audio recordings. The gmm-decode-faster decoder from the Kaldi toolkit was used for recognizing the audio, and the VoxForge dataset was used for the training purpose. We have used triphone model trained with Boosted MMI depending on MFCC features followed by LDA and STC transform. The whole data contains 1000 hours of speech sampled at 16 kHz.

We have used the Smith-Waterman alignment algorithm for the best single region of alignment between the audio of children and adult. It has changed rather than fixed weights for the various errors. Then we took the largest single region of matching and discarded the remaining parts. With a focus on that similarity region, we have matched it with reference and then split the audio into smaller parts using a dynamic programming algorithm. We aimed to create a better model with different types of children related queries to report a child-friendly experience. The whole training data was highly assured machine transcriptions of spoken audio. A small set of sites of no interest to children was chosen randomly as negative scored seeds. A small set of sites of high interest to children was chosen as positive scored seeds. Web sites that were clicked in the same session where a seeded site was clicked and received some of the seed’s score. On each of the above sources, separate 5-gram language models were trained and then merged using Bayesian interpolation [33] to give a 100M n-gram model. The vocabulary is a list of 4.6M tokens extracted from written domain text.

3. Experimental setup

3.1 Training Data

Training data for speech recognition engine can be found in two ways. One way is to use existing speech data and transliterate it, and a second way is to produce a text corpus and have speakers speak the text. Our approach was closer to the second way. Children were taught what to speak during English lessons, and this data was recorded and transcribed. The recording was done using a single channel (mono) and at a 16 kHz sampling rate. Speech data was gathered for children between 3 and ten years old. A total of 500 children were used (9 ten-year-olds, 11 nine-year-olds, 50 eight-year-olds, 30 seven-year-olds, 190 five-year-olds, 100 six-year-olds, 100 four-year-olds, 10 three-year-olds). Data collection was carried out during a 9-month period with a total of 1000 hours of record speech with 2000 utterance and 10,000 words (1,013 unique words).

3.2 Test Data

The test data was broken up as follow: 198 utterances for 3 year olds; 221 utterances for 4 year olds; 243 utterances for 5 year olds; 265 utterances for 6 year olds; 225 utterances for 7 year olds; 232 utterances for 8 year olds; 217 utterances for 9 year olds; 233 utterances for 10 year olds. We have compared the following services: Google, Nuance, Microsoft, Iflytek, IBM Watson, and Human listeners.

4. Results

Human recognition surmounts all technologies. Our system outperforms the leading commercial systems. The champion among all available commercial systems for young children is Google. However, iFlyTek executes very well for older Chinese children speaking English. We did not reproduce the 14.5% WER claimed by Google for their Youtube Kids Speech-to-text (STT), but the results are still significant and impressive.

Table 1: Word Error Rate (WER) % for different age groups compared to different systems

Age	Kadho	Human	Google	Iflytek	Microsoft	IBM
3	30.2	11.6	64.1	68.1	78.1	89
4	22.1	8.2	55.6	56.8	66.3	73.2
5	18.5	7.9	42.1	38.9	67.8	68.9
6	15.4	5.8	41.2	45.1	58.1	67.3
7	13.2	3.8	34.5	33.5	45.1	56.1
8	12.1	3.1	20.2	28.1	42.1	52.1
9	11.2	3.2	23.5	20.8	38.1	52.1
10	9.4	2.3	25.8	16.7	36.2	38.1

Figure 1: Comparison of word error rate among six-year-olds

5. Conclusion

Kadho is far the best solution as per our findings and test criteria of exact recognized phrases and word error rate. However, Google performs extremely well and this is not surprising given their history of developing and proving voice search, but unfortunately, it is not commercially available for now. Still, Google’s quality could be used as a benchmark for the commercially available products as many of them have features and tools for tailoring search experience. Iflytek executes very well with older children, and this is also not surprising either because their English STT has a lot of data from adult native Chinese speakers.

6. References

[1] Mostow, J., Roth, S.F., Hauptmann, A.G. & Kane, M. (1994). “Aprototype reading coach that listens”, Proc. 12th National Conference on Artificial Intelligence (AAAI’94), Seattle, WA, pp 785-792, 1994.

[2] Wilpon Jay G., Jacobsen Claus N. (1996). “A Study of Speech Recognition for Children and the Elderly”, In Proceedings of ICASSP 1996, Vol. 1, pp. 349-352.

[3] Lee, S., Potamianos, A. & Narayanan, S. (1999). “Acoustics of children’s speech: Developmental changes of temporal and spectral parameters,” Journal of the Acoustical Society of America, pp. 1455-1468.

[4] Li, Q. & Russell, M. (2002). “An Analysis of the Causes of Increased Error Rates in Children’s Speech Recognition”, In Proceedings of ICSLP.

[5] Gerosa, M., Giuliani, D., Narayanan, S., & Potamianos, A. (2009). “A review of ASR technologies for children’s speech,” in Proc. Workshop on Child, Computer, and Interaction (WOCCI).

[6] Shivakumar, P. G., Potamianos, A., Lee, S., & Narayanan, S. (2014). “Improving speech recognition for children using acoustic adaptation and pronunciation modeling,” in Proc. Workshop on Child, Computer, and Interaction (WOCCI), 2014.

[7] Russell, M. & D’Arcy, S. (2007). “Challenges for computer recognition of children’s speech,” in Proc. Speech and Language Technologies in Education (SLaTE).

[8] Hämäläinen, A., Candeias, S., Cho, H., Meinedo, H., Abad, A., Pellegrini, T., Tjalve, M., Trancoso, I., & Dias, M. S. (2014). “Correlating ASR errors with developmental changes in speech production: A study of 3-10-year-old European Portuguese children’s speech,” in Proc. Workshop on Child, Computer, and Interaction (WOCCI).

[9] Potamianos, A., Narayanan, S., & Lee, S. (1997). “Automatic speech recognition for children,” In Proceedings of Eurospeech.

[10] Gray, S. S., Willett, D., Lu, J., Pinto, J., Maergner, P., & Bodenstab, N. (2014). “Child automatic speech recognition for US English: Child interaction with living-room-electronic-devices,” in Proc. Workshop on Child, Computer, and Interaction (WOCCI).

[11] Elenius, D., & Blomberg, M. (2005). “Adaptation and normalization experiments in speech recognition for 4 to 8-year-old children”. In Proceedings of Interspeech.

[12] Narayanan, S., & Potamianos, A. (2002). Creating conversational interfaces for children, IEEE Trans. Speech and Audio Proc., Vol. 10, No. 2, 65-78.

[13] Das, S., Nix, D. & Picheny, M. (1998). Improvements in children’s speech recognition performance, Proc. ICASSP’98, Seattle, WA, USA.

[14] Gauvain, J.L. & Lee, C.H. (1994). Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains, IEEE Trans. on Speech and Audio Proc., Vol. 2, 291-298.

[15] Leggetter, C.J. & Woodland, P.C. (1995). Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models, Computer Speech and Language, vol. 9, pp.171-185.

[16] Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Liu, X., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V. & Woodland, P. (2009). The HTK Book (version 3.4). Cambridge Univ. Eng. Dept.

[17] Pellom, B. (2001). “SONIC: The University of Colorado Continuous Speech Recognizer”. In Technical Report TR-CSLR-2001-01, Center for Spoken Language Research, University of Colorado, USA.

[18] Pellom, B. & Hacioglu, K. (2003). Recent Improvements in the CU SONIC ASR System for Noisy Speech: The SPINE Task. In Proceedings of ICASSP.

[19] Lee, K.F., Hon, H.W. & Reddy, R. (1990). An overview of the SPHINX speech recognition system. In IEEE Transactions on Acoustics, Speech and Signal Processing g 38.1, 35-45.

[20] Walker, W., Lamere, P., Kwok, P., Raj, B., Singh, R., Gouvea, E., Wolf, P. & Woelfel, J. (2004). “Sphinx-4: A flexible open source framework for speech recognition,” Sun Microsystems Inc., Technical Report SML1 TR2004-0811.

[21] Rybach, D., Gollan, C., Heigold, G., Hoffmeister, B., Lööf, J., Schlüter, R. & Ney, H. (2009). The RWTH Aachen University Open Source Speech Recognition System. In Proceedings of INTERSPEECH 2009, 2111-2114.

[22] Lee, A., Kawahara, T. & Shikano, K. (2001), JULIUS - an open source real-time large vocabulary recognition engine. In Proceedings of INTERSPEECH 2001, 1691-1694.

[23] Povey, D., Burget, L., Agarwal, M., Akyazid, P., Kaie, F., Ghoshalf, A., Glembekb, O., Goelg, N., Karafi´atb, M., Rastrowh, A., Rosei, R., Schwarzb, P., & Thomash, S. (2011). The subspace Gaussian mixture model-A structured model for speech recognition. In Computer Speech & Language, vol. 25, n. 2, 404-439.

[24] Simon. WEB: http://www.simon-listens.com.

[25] Bolaños, D. (2012). The Bavieca Open-Source Speech Recognition Toolkit. In Proceedings of IEEE Workshop on Spoken Language Technology (SLT), Miami, FL, USA.

[26] Soltau, H. Liao, H. Sak, H. (2016). Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition. CoRR, abs/1610.09975.

[27] Vesely, K., Ghoshal, A., Burget, L. & Povey, D. (2013). Sequence-discriminative training of deep neural networks. In Proceedings of INTERSPEECH 2013.

[28] Hinton, G. (2010). A Practical Guide to Training Restricted Boltzmann Machines. UTML TR 2010:003, Momentum, 9:1.

[29] Povey, D., & Kingsbury, B. (2007). Evaluation of proposed modifications to MPE for large scale discriminative training. In Proceedings of ICASSP.

[30] Gibson, M., & Hain, T. (2006). Hypothesis Spaces For Minimum Bayes Risk Training In Large Vocabulary Speech Recognition. In Proceedings of INTERSPEECH.

[31] Schultz, T. and Waibel, A., “Experiments on cross-language acoustic modeling,” in Proc. European Conf. on Speech Communication and Technology, Aalborg, Denmark, Sep. 2001, pp. 2721–2724.

[32] Rath, S.P., Povey, D., Vesely, K. & Cernocky, J. (2013). Improved feature processing for Deep Neural Networks. In Proceedings of INTERSPEECH 2013, 109-113.

[33] Allauzen, C. & Riley, M. (2011). Bayesian language model interpolation for mobile speech input. In Proceedings of INTERSPEECH 2011.

Preprint

Preprint

A Novel Speech to Text Model for Children