Generative Models for Indic Languages: Evaluating Content Generation Capabilities

2Citations
Citations of this article
25Readers
Mendeley users who have this article in their library.

Abstract

Large language models (LLMs) and generative AI have emerged as the most important areas in the field of natural language processing (NLP). LLMs are considered to be a key component in several NLP tasks, such as summarization, question-answering, sentiment classification, and translation. Newer LLMs, such as Chat-GPT, BLOOMZ, and several such variants, are known to train on multilingual training data and hence are expected to process and generate text in multiple languages. Considering the widespread use of LLMs, evaluating their efficacy in multilingual settings is imperative. In this work, we evaluate the newest generative models (ChatGPT, mT0, and BLOOMZ) in the context of Indic languages. Specifically, we consider natural language generation (NLG) applications such as summarization and questionanswering in monolingual and cross-lingual settings. We observe that current generative models have limited capability for generating text in Indic languages in a zero-shot setting. In contrast, generative models perform consistently better on manual quality-based evaluation in Indic languages and English language generation. Considering limited generation performance, we argue that these LLMs are not intended to use in zero-shot fashion in downstream applications.

Cite

CITATION STYLE

APA

Bhat, S., Varma, V., & Pedanekar, N. (2023). Generative Models for Indic Languages: Evaluating Content Generation Capabilities. In International Conference Recent Advances in Natural Language Processing, RANLP (pp. 187–195). Incoma Ltd. https://doi.org/10.26615/978-954-452-092-2_021

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free