Ashish Mittal

I'm a research scientist at IBM Research India, where I work on multimodal large language models and extending LLM capabilities for Indic languages. My recent work focuses on aligning speech and text modalities for Granite Speech models. At IBM, I have previously contributed to Watson Speech and developed natural language interfaces for Cognos.

I am nearing completion of my PhD at IIT Bombay, advised by Sunita Sarawagi, Preethi Jyothi, and George Saon. My doctoral research focuses on text-only contextualization of ASR models through retrieval, adaptation and LLM integration.

Email  /  CV  /  Scholar  /  Twitter  / 

Research

I'm interested in multimodal language models, with a focus on fusing speech for building accessible interfaces to world knowledge. I also work on adapting LLMs to new and low-resource languages, and on inducing reasoning abilities in multilingual settings.

Research Papers

This paper introduces Granite-speech LLMs, compact and efficient speech-language models (2B and 8B parameters) for English ASR and AST, which outperform competitors trained on more data and maintain pace on English-to-X AST for major languages, while preserving text LLM capabilities and being freely available on HuggingFace.

This paper introduces culturally adapted variants of the GSM8K math benchmark for five non-Western regions, evaluating six LLMs and revealing a consistent performance drop on culturally adapted versions, though models with stronger reasoning capabilities are more resilient.

RECAST is a lightweight, retrieval-augmented method for ASR contextual biasing that uses decoder states to retrieve keywords from large dictionaries, significantly improving rare term recognition and overall WER across diverse languages.

HLM is a hybrid model that couples a multilingual encoder with a lightweight decoder to deliver efficient, high-accuracy sentiment analysis on code-mixed text.

This paper identifies a tokenization mismatch issue (token fertility gap) in SALSA, an ASR-LLM integration method, particularly affecting low-resource languages by starving the LLM of audio context; to address this, they propose SKIP-SALSA, which adaptively skips ASR decoder states to synchronize with the LLM, leading to significant ASR performance improvements in low-resource languages.

This paper introduces Indic-QA, a new large benchmark for context-grounded question answering in 11 major Indian languages, revealing that multilingual LLMs exhibit an English bias and perform weakly in low-resource settings, while a Translate-Test paradigm proves more effective.

This paper introduces SALSA, a novel method for improving ASR systems for low-resource languages by efficiently coupling the ASR and LLM decoders with a simple projection, addressing tokenizer mismatches through cascading tokenization, and achieving up to 38% WER reduction on the FLEURS benchmark.

This paper proposes a novel inference algorithm that enhances rare word prediction in state-of-the-art ASR models by using a nearest-neighbor-based matching on an inference-time word list, indexed in a memory as a trie to prevent spurious matches.

This paper introduces a technique for RNN-Transducer ASR models that improves text representations by looking ahead in the audio input, thereby reducing language model biasing and multi-step hallucination, resulting in a 5%-20% relative reduction in word error rate.

This paper introduces a novel text-only adaptation technique for RNN-T ASR models that imputes speech representations to enable in-situ adaptation, significantly reducing word error rate (up to 35%) and mitigating catastrophic forgetting without runtime overhead or retraining the ASR model from scratch.

This paper provides a theoretical and empirical analysis of Soft Random Sampling (SRS), a simple and effective data sampling method for large-scale deep neural network training, demonstrating its strong convergence and generalization, as well as its superior accuracy-efficiency trade-off compared to coreset-based methods in image and speech recognition tasks.

This paper proposes Partitioned Gradient Matching (PGM), a novel distributable data subset selection algorithm for RNN-T ASR systems, which significantly reduces training time (3x-6x speedup) with minimal accuracy degradation (< 1% WER difference) even with noisy data, addressing the high computational cost of training state-of-the-art ASR models.

This paper introduces a lightweight adaptive language model discounting technique for RNN-Transducer ASR models, which improves rare word performance by masking prediction network output and dynamically discounting the implicit language model, achieving significant WER and PER reductions on a Hindi-English code-mixed task.

This paper introduces a novel cross-lingual transfer learning strategy for ASR, pretraining high-resource language speech with transliterated text in the low-resource target language, proving effective even for unrelated language families and significantly reducing Word Error Rate (WER) in low-resource scenarios.

This paper details a challenge to build multilingual and code-switching ASR systems for seven Indian languages, providing 600 hours of transcribed data and baseline recipes.

The paper introduces an automated technique for bootstrapping chatbot interfaces for question answering on relational databases, utilizing natural language classifiers from industrial chatbot platforms to translate natural language into structured queries.

This paper presents novel algorithms that extend the ATHENA system, using Functional Partitioning of Ontology and Lazy Inclusion to significantly improve the automated generation of precise interpretations for natural language database queries by a factor of at least 400%.

A few-shot spoken intent classification method that uses a meta-learning approach to create task-agnostic representations of utterances, enabling accurate classification of new intents with limited examples.

A system that translates complex natural language queries into nested SQL queries by combining linguistic patterns with deep domain reasoning using ontologies.

Unified semantic parser trained via weak supervision from multiple domain-specific teachers without requiring domain labels.

An ontology-based two-stage architecture enables NLQ systems to translate user questions first into ontology-level queries and then into executable database queries to support intuitive querying over structured data.

A tooling framework to easily instantiate NLQ systems across diverse structured data sources and formats, enabling natural language querying with minimal human configuration.

Proposed the first query completion framework for NLIDB systems using functional partitioning of ontologies to generate accurate and semantically meaningful completions without relying on query logs.

Built a system to answer contextual natural language queries on tabular data by combining semantic parsing with program execution.

Demonstrated a system for answering natural language questions over enterprise knowledge graphs using semantic parsing and graph querying.

Co‑developed ATHENA, an ontology-driven natural language interface that enables users to query complex relational databases using natural language.

Built a rule-based system and a graphical model to extract factual triples (e.g., (India, Population, 1.3 billion)) from sentences expressing numeric facts.

Patents

Adapted from Jon Barron's personal website (see the source code on GitHub).