Mehbubul Hasan Al-Quvi

Automated mobile application testing faces two critical challenges: (1) executing natural language test specifications without manual scripting, and (2) reliably verifying test outcomes across diverse application states. Existing approaches often require specialized scripting languages or rely on fragile record-and-replay mechanisms, which increase maintenance costs and reduce verification reliability. To address these issues, we introduce LELANTE, a novel framework that leverages large language models (LLMs) to automate both execution and verification of natural language test cases. LELANTE employs a two-stage pipeline: an LLM interprets and executes test steps via interaction with the device's graphical user interface (GUI), while another language model validates execution outcomes against expected conditions. In experiments across 390 test cases spanning 10 popular Android applications, LELANTE achieved an 85% execution success rate and an 85% verification accuracy using open-source models on standard hardware. These results show that LLMs can effectively bridge the gap between natural language test specifications and automated test execution, making mobile testing more accessible without compromising reliability.

Large Language Models (LLMs) have shown remarkable potential as autonomous agents in diverse environments, but their agentic capabilities are strongly contingent on their parameter count and context window size. While closed-source LLMs such as GPT-4 and Claude are already quite capable, smaller open-source models struggle with autonomous UI navigation tasks. To bridge this gap, we propose DistilAgent, a novel framework that distills the capabilities of long-context closed-source LLMs into smaller models , significantly improving their performance in autonomous UI navigation. DistilAgent incorporates three key components: task discretization to break down long-horizon tasks into smaller sub-tasks that fit within the context window of smaller models, explicit reasoning enforcement to ensure the LLM reasons about grounding and the current state of the environment, and exploration encouragement to prompt the LLM to explore the UI when required actions are not immediately available on the current screen. Our results demonstrate that DistilAgent improves the task completion rate of GPT4 from 50% to 90%. We collected AutoNav1K - a synthetic dataset of 1,000 GPT4 task executions using DistilAgent. After finetuning on AutoNav1K, the task completion rate of LLaMA-3.1-8B improved from 0% to 60%. DistilAgent showcases the potential for distilling the agentic capabilities of large, closed-source LLMs into smaller models, democratizing access to high-performance autonomous UI navigation.