View source for Pronunciation assessment

{{Short description|Use of speech recognition to verify pronunciation}}

Automatic '''pronunciation assessment''' is the use of [[Speech recognition|speech recognition]] to verify the correctness of pronounced speech,<ref>{{cite journal |last1=Ehsani |first1=Farzad |last2=Knodt |first2=Eva |title=Speech technology in computer-aided language learning: Strengths and limitations of a new CALL paradigm |journal=Language Learning & Technology |date=July 1998 |volume=2 |issue=1 |pages=54–73 |url=https://www.lltjournal.org/item/10125-25032/ |access-date=11 February 2023 |publisher=University of Hawaii National Foreign Language Resource Center; Michigan State University Center for Language Education and Research |language=en}}</ref> as distinguished from manual assessment by an instructor or proctor.<ref>{{cite journal |last1=Isaacs |first1=Talia |last2=Harding |first2=Luke |title=Pronunciation assessment |journal=Language Teaching |date=July 2017 |volume=50 |issue=3 |pages=347–366 |doi=10.1017/S0261444817000118 |s2cid=209353525 |language=en |issn=0261-4448|doi-access=free }}</ref> Also called '''speech verification, pronunciation evaluation''', and '''pronunciation scoring''', the main application of this technology is '''computer-aided pronunciation teaching (CAPT)''' when combined with computer-aided instruction for [[Software:Computer-assisted language learning|computer-assisted language learning]] (CALL), speech [[Social:Remedial education|remediation]], or [[Social:Accent reduction|accent reduction]]. Pronunciation assessment does not determine unknown speech (as in [[Engineering:Digital dictation|dictation]] or automatic transcription) but instead, knowing the expected word(s) in advance, it attempts to verify the correctness of the learner's pronunciation and ideally their [[Physics:Intelligibility (communication)|intelligibility]] to listeners,<ref>{{citation |last1=Loukina |first1=Anastassia |last2=Lopez |first2=Melissa |display-authors=1 |contribution=Pronunciation accuracy and intelligibility of non-native speech |contribution-url=https://www.isca-speech.org/archive/pdfs/interspeech_2015/loukina15_interspeech.pdf |title=INTERSPEECH 2015 |pages=1917–1921 |publisher=[[Organization:International Speech Communication Association|International Speech Communication Association]] |location=Dresden, Germany |date=September 6, 2015 |quote=only 16% of the variability in word-level intelligibility can be explained by the presence of obvious mispronunciations.}}</ref><ref name="obrien">{{cite journal |last1=O’Brien |first1=Mary Grantham |last2=Derwing |first2=Tracey M. |display-authors=1 |title=Directions for the future of technology in pronunciation research and teaching |journal=Journal of Second Language Pronunciation |date=31 December 2018 |volume=4 |issue=2 |pages=182–207 |doi=10.1075/jslp.17001.obr |s2cid=86440885 |language=en |issn=2215-1931 |quote=pronunciation researchers are primarily interested in improving L2 learners’ intelligibility and comprehensibility, but they have not yet collected sufficient amounts of representative and reliable data (speech recordings with corresponding annotations and judgments) indicating which errors affect these speech dimensions and which do not. These data are essential to train ASR algorithms to assess L2 learners’ intelligibility.|doi-access=free |hdl=2066/199273 |hdl-access=free }}</ref> sometimes along with often inconsequential [[Social:Prosody (linguistics)|prosody]] such as [[Social:Intonation (linguistics)|intonation]], [[Physics:Pitch (music)|pitch]], [[Tempo|tempo]], rhythm, and stress.<ref>{{cite journal |last1=Eskenazi |first1=Maxine |title=Using automatic speech processing for foreign language pronunciation tutoring: Some issues and a prototype |journal=Language Learning & Technology |date=January 1999 |volume=2 |issue=2 |pages=62–76 |url=https://www.lltjournal.org/item/10125-25043/ |access-date=11 February 2023 |language=en}}</ref> Pronunciation assessment is also used in reading tutoring, for example in products such as [[Software:Microsoft Teams|Microsoft Teams]]<ref>{{cite news |last1=Tholfsen |first1=Mike |title=Reading Coach in Immersive Reader plus new features coming to Reading Progress in Microsoft Teams |url=https://techcommunity.microsoft.com/t5/education-blog/reading-coach-in-immersive-reader-plus-new-features-coming-to/ba-p/3734079 |access-date=12 February 2023 |work=Techcommunity Education Blog |publisher=Microsoft |date=9 February 2023 |language=en}}</ref> and from Amira Learning.<ref>{{cite news |last1=Banerji |first1=Olina |title=Schools Are Using Voice Technology to Teach Reading. Is It Helping? |url=https://www.edsurge.com/news/2023-03-07-schools-are-using-voice-technology-to-teach-reading-is-it-helping |access-date=7 March 2023 |work=EdSurge News |date=7 March 2023 |language=en}}</ref> Automatic pronunciation assessment can also be used to help diagnose and treat speech disorders such as apraxia.<ref>{{cite book |last1=Hair |first1=Adam |last2=Monroe |first2=Penelope |title=Proceedings of the 17th ACM Conference on Interaction Design and Children |chapter=Apraxia world: A speech therapy game for children with speech sound disorders |display-authors=1 |date=19 June 2018 |pages=119–131 |doi=10.1145/3202185.3202733 |isbn=9781450351522 |s2cid=13790002 |url=https://psi.engr.tamu.edu/wp-content/uploads/2018/04/hair2018idc.pdf}}</ref>

The earliest work on pronunciation assessment avoided measuring genuine listener intelligibility,<ref>{{citation |last1=Bernstein |first1=Jared |last2=Cohen |first2=Michael |display-authors=1 |contribution=Automatic Evaluation and Training in English Pronunciation |title=First International Conference on Spoken Language Processing (ICSLP 90) |date=November 18, 1990 |pages=1185–1188 |contribution-url=https://www.isca-speech.org/archive/pdfs/icslp_1990/bernstein90_icslp.pdf |access-date=11 February 2023 |publisher=[[Organization:International Speech Communication Association|International Speech Communication Association]] |location=Kobe, Japan |quote=listeners differ considerably in their ability to predict unintelligible words.... Thus, it seems the quality rating is a more desirable... automatic-grading score.}} (Section 2.2.2.)</ref> a shortcoming corrected in 2011 at the Toyohashi University of Technology,<ref>{{citation |last1=Hiroshi |first1=Kibishi |last2=Nakagawa |first2=Seiichi |contribution=New feature parameters for pronunciation evaluation in English presentations at international conferences |pages=1149–1152 |title=INTERSPEECH 2011 |publisher=[[Organization:International Speech Communication Association|International Speech Communication Association]] |date=August 28, 2011 |location=Florence, Italy |contribution-url=https://www.isca-speech.org/archive/pdfs/interspeech_2011/kibishi11_interspeech.pdf |access-date=11 February 2023 |quote=we investigated the relationship between pronunciation score / intelligibility and various acoustic measures, and then combined these measures.... As far as we know, the automatic estimation of intelligibility has not yet been studied.}}</ref> and included in the Versant high-stakes English fluency assessment from Pearson<ref>{{cite web |last1=Bonk |first1=Bill |title=New innovations in assessment: Versant's Intelligibility Index score |url=https://www.english.com/blog/intelligibility-index-score-versant/ |website=Resources for English Language Learners and Teachers |publisher=Pearson English |access-date=11 February 2023 |date=25 August 2020 |archive-url=https://web.archive.org/web/20230127122339/https://www.english.com/blog/intelligibility-index-score-versant/ |archive-date=2023-01-27 |quote=you don’t need a perfect accent, grammar, or vocabulary to be understandable. In reality, you just need to be understandable with little effort by listeners.}}</ref> and mobile apps from 17zuoye Education & Technology,<ref>{{citation |last1=Gao |first1=Yuan |last2=Srivastava |first2=Mohan Lal Srivastava |display-authors=1 |contribution=Spoken English Intelligibility Remediation with PocketSphinx Alignment and Feature Extraction Improves Substantially over the State of the Art |date=May 25, 2018 |title=2nd IEEE Advanced Information Management, Communication, Electronic and Automation Control Conference (IMCEC 2018) |doi=10.1109/IMCEC.2018.8469649 |arxiv=1709.01713 |s2cid=31125681 }}</ref> but still missing in 2023 products from [[Software:Google Search|Google Search]],<ref>{{cite web |last1=Snir |first1=Tal |title=How do you pronounce quokka? Practice with Search |url=https://blog.google/products/search/how-do-you-pronounce-quokka-practice-search/ |website=The Keyword |publisher=Google |access-date=11 February 2023 |language=en-us |date=14 November 2019}}</ref> [[Company:Microsoft|Microsoft]],<ref>{{cite web |title=Pronunciation assessment tool |url=https://speech.microsoft.com/portal/pronunciationassessmenttool |website=Azure Cognitive Services Speech Studio |publisher=Microsoft |access-date=11 February 2023}}</ref> Educational Testing Service,<ref>{{cite book |last1=Chen |first1=Lei |last2=Zechner |first2=Klaus |display-authors=1 |title=Automated Scoring of Nonnative Speech: Using the SpeechRater v. 5.0 Engine |date=December 2018 |publisher=Educational Testing Service |location=Princeton, NJ |url=https://onlinelibrary.wiley.com/doi/epdf/10.1002/ets2.12198 |doi=10.1002/ets2.12198 |series=ETS Research Report Series |volume=2018 |issue=1 |pages=1–31 |s2cid=69925114 |issn=2330-8516 |access-date=11 February 2023 |language=en}}</ref> Speechace,<ref>{{citation |last1=Alnafisah |first1=Mutleb |contribution=Technology Review: Speechace |series=no. 40 |title=Proceedings of the 12th Pronunciation in Second Language Learning and Teaching Conference (Virtual PSLLT) |location=St. Catharines, Ontario |date=September 2022 |volume=12 |contribution-url=https://www.iastatedigitalpress.com/psllt/article/14315/galley/14157/view |issn=2380-9566 |access-date=14 February 2023}}</ref> and ELSA.<ref>{{Cite AV media |last1=Gorham |first1=Jon |last2=Raine |first2=Paul |display-authors=1 |title=Speech Recognition for English Language Learning |url=https://www.youtube.com/watch?v=SdC16s-GEEo&t=1325s |date=March 10, 2022 |publisher=Education Solutions |work=Technology in Language Teaching and Learning |format=video |language=en |access-date=2023-02-14}}</ref> Assessing authentic listener intelligibility is essential for avoiding inaccuracies from [[Social:Accent (sociolinguistics)|accent]] bias, especially in high-stakes assessments;<ref>{{cite news |title=Computer says no: Irish vet fails oral English test needed to stay in Australia |url=https://www.theguardian.com/australia-news/2017/aug/08/computer-says-no-irish-vet-fails-oral-english-test-needed-to-stay-in-australia |access-date=12 February 2023 |agency=Australian Associated Press |work=The Guardian |date=8 August 2017}}</ref><ref>{{cite news |last1=Ferrier |first1=Tracey |title=Australian ex-news reader with English degree fails robot's English test |url=https://www.smh.com.au/technology/australian-exnews-reader-with-english-degree-fails-robots-english-test-20170809-gxsjv2.html |access-date=12 February 2023 |work=The Sydney Morning Herald |date=9 August 2017 |language=en}}</ref><ref>{{cite news |last1=Main |first1=Ed |last2=Watson |first2=Richard |title=The English test that ruined thousands of lives |url=https://www.bbc.com/news/uk-60264106 |access-date=12 February 2023 |work=BBC News |date=9 February 2022}}</ref> from words with multiple correct pronunciations;<ref>{{cite web |last1=Joyce |first1=Katy Spratte |title=13 Words That Can Be Pronounced Two Ways |url=https://www.rd.com/list/words-that-can-be-pronounced-two-ways/ |publisher=Reader's Digest |access-date=23 February 2023 |date=January 24, 2023}}</ref> and from phoneme coding errors in machine-readable pronunciation dictionaries.<ref>E.g., [[Software:CMU Pronouncing Dictionary|CMUDICT]], {{cite web |title=The CMU Pronouncing Dictionary |url=http://www.speech.cs.cmu.edu/cgi-bin/cmudict |website=www.speech.cs.cmu.edu |access-date=15 February 2023}} Compare "four" given as "F AO R" with the vowel AO as in "caught," to "row" given as "R OW" with the vowel OW as in "oat."</ref> In 2022, researchers found that some newer speech to text systems, based on [[End-to-end reinforcement learning|end-to-end reinforcement learning]] to map audio signals directly into words, produce word and phrase confidence scores very closely correlated with genuine listener intelligibility.<ref>{{cite conference |last1=Tu |first1=Zehai |last2=Ma |first2=Ning |last3=Barker |first3=Jon |title=Unsupervised Uncertainty Measures of Automatic Speech Recognition for Non-intrusive Speech Intelligibility Prediction |date=2022 |book-title=Proc. Interspeech 2022 |conference=INTERSPEECH 2022 |pages=3493–3497 |doi=10.21437/Interspeech.2022-10408 |url=https://www.isca-speech.org/archive/pdfs/interspeech_2022/tu22b_interspeech.pdf |access-date=17 December 2023 |publisher=ISCA}}</ref> In the Common European Framework of Reference for Languages (CEFR) assessment criteria for "overall phonological control", intelligibility outweighs formally correct pronunciation at all levels.<ref>{{Cite book |url=https://rm.coe.int/cefr-companion-volume-with-new-descriptors-2018/1680787989 |title=Common European framework of reference for languages learning, teaching, assessment: Companion volume with new descriptors |date=February 2018 |publisher=Language Policy Programme, Education Policy Division, Education Department, Council of Europe |oclc=1090351600 |page=136}}</ref>

Although there are as yet no industry-standard [[Benchmark (computing)|benchmarks]] for evaluating pronunciation assessment accuracy, researchers occasionally release evaluation [[Speech corpus|speech corpus]]es for others to use for improving assessment quality.<ref>{{citation |last1=Vidal |first1=Jazmín |last2=Ferrer |first2=Luciana |display-authors=1 |contribution=EpaDB: A Database for Development of Pronunciation Assessment Systems |title=Interspeech 2019 |date=15 September 2019 |pages=589–593 |doi=10.21437/Interspeech.2019-1839 |hdl=11336/161618 |s2cid=202742421 |contribution-url=https://www.isca-speech.org/archive/pdfs/interspeech_2019/vidal19_interspeech.pdf |access-date=19 February 2023|hdl-access=free }}; [https://drive.google.com/file/d/1zEFNnyp8gmgJbOuSUwh1avccAw00FN5z/view database .zip file.]</ref><ref>{{citation |last1=Zhang |first1=Junbo |last2=Zhang |first2=Zhiwen |display-authors=1 |contribution=speechocean762: An Open-Source Non-Native English Speech Corpus for Pronunciation Assessment |title=Interspeech 2021 |date=30 August 2021 |pages=3710–3714 |doi=10.21437/Interspeech.2021-1259 |arxiv=2104.01378 |s2cid=233025050 |contribution-url=https://www.isca-speech.org/archive/pdfs/interspeech_2021/zhang21x_interspeech.pdf |access-date=19 February 2023}}; [https://github.com/jimbozhang/speechocean762 GitHub corpus repository.]</ref> Such evaluation databases often emphasize formally unaccented pronunciation to the exclusion of genuine intelligibility evident from [[Blinded experiment|blinded]] listener transcriptions.<ref name="obrien" /> Some promising areas for improvement being developed in 2023 include articulatory [[Feature extraction|feature extraction]]<ref>{{cite arXiv |mode=cs2 |last1=Wu |first1=Peter |last2=Chen |first2=Li-Wei |display-authors=1 |title=Speaker-Independent Acoustic-to-Articulatory Speech Inversion |date=14 February 2023 |class=eess.AS |eprint=2302.06774 }}</ref> and [[Transfer learning|transfer learning]] to suppress unnecessary corrections.<ref>{{cite book |last1=Sancinetti |first1=Marcelo |last2=Vidal |first2=Jazmin |title=ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) |chapter=A Transfer Learning Approach for Pronunciation Scoring |display-authors=1 |date=23 May 2022 |pages=6812–6816 |doi=10.1109/ICASSP43922.2022.9747727 |arxiv=2111.00976 |isbn=978-1-6654-0540-9 |s2cid=249437375 }}</ref> Other interesting advances under development include "[[Augmented reality|augmented reality]]" interfaces for mobile devices using [[Optical character recognition|optical character recognition]] to provide pronunciation training on text found in user environments.<ref>{{cite journal |last1=Che Dalim |first1=Che Samihah |last2=Sunar |first2=Mohd Shahrizal |display-authors=1 |title=Using augmented reality with speech input for non-native children's language learning |journal=International Journal of Human-Computer Studies |date=February 2020 |volume=134 |pages=44–64 |doi=10.1016/j.ijhcs.2019.10.002 |s2cid=208098513 |url=https://core.ac.uk/download/pdf/233047951.pdf |access-date=28 February 2023}}</ref><ref>{{cite journal |last1=Tolba |first1=Rahma M. |last2=Elarif |first2=Taha |display-authors=1 |title=Mobile Augmented Reality for Learning Phonetics: A Review (2012–2022) |journal=Extended Reality and Metaverse |series=Springer Proceedings in Business and Economics |date=2023 |pages=87–98 |doi=10.1007/978-3-031-25390-4_7 |url=https://link.springer.com/chapter/10.1007/978-3-031-25390-4_7 |access-date=28 February 2023 |publisher=Springer International Publishing |isbn=978-3-031-25389-8 |language=en}}</ref>

== See also ==
* [[Social:Phonetics|Phonetics]]
* [[Speech segmentation]] — often called "forced alignment" (of audio to its expected phonemes) in this context<ref>{{cite book |last1=Mathad |first1=Vikram C. |last2=Mahr |first2=Tristan J. |title=22nd Annual Conference of the International Speech Communication Association (INTERSPEECH 2021) |chapter=The Impact of Forced-Alignment Errors on Automatic Pronunciation Evaluation |display-authors=1 |date=2021 |pages=176–180 |chapter-url=https://www.isca-speech.org/archive/pdfs/interspeech_2021/mathad21_interspeech.pdf |doi=10.21437/interspeech.2021-1403 |access-date=10 March 2023 |publisher=International Speech Communication Association |isbn=9781713836902|s2cid=239694157 }}</ref>
* [[Statistical classification]]

== References ==
{{reflist}}

== External links ==
* [[Organization:International Speech Communication Association|International Speech Communication Association]] (ISCA) Special Interest Group on [https://www.isca-speech.org/iscaweb/index.php/sigs?layout=edit&id=121 Speech and Language Technologies in Education (SLaTE)]

{{Natural language processing}}

[[Category:Educational technology]]
[[Category:Language learning software]]
[[Category:Natural language processing]]
[[Category:Speech recognition]]
[[Category:Statistical classification]]

{{Sourceattribution|Pronunciation assessment}}