KPIs for AI Agents and Generative AI: A Rigorous Framework for Evaluation and Accountability

Sunkara, Vivek Lakshman Bhargav (2024) KPIs for AI Agents and Generative AI: A Rigorous Framework for Evaluation and Accountability. International Journal of Scientific Research and Modern Technology, 3 (4): 572. pp. 22-29. ISSN 2583-4622

Abstract

AI agents and generative AI systems are increasingly becoming integral across sectors such as healthcare, finance, and creative industries. However, the rapid evolution of these systems has outpaced traditional evaluation methods, leaving gaps in evaluating them. This paper proposes a comprehensive Key Performance Indicator (KPI) framework spanning across five vital dimensions – Model Quality, System Performance, Business Impact, Human-AI Interaction, and Ethical and Environmental Considerations – to holistically evaluate these systems. Drawing insights from multiple studies, benchmarks like MLPerf, AI Index and standards like the EU AI Act [1] and NIST AI RMF, this framework blends established metrics like accuracy, latency and efficiency with novel metrics like “ethical drift” and “creative diversity” for tracking AI’s moral compass in real time. Evaluated on systems like GPT-4, DALL-E 3 and MidJourney, and validated through case studies such as Waymo [1] and Claude3, this framework addresses technical, operational, and ethical dimensions to enhance accountability and performance.

Documents
65:311
[thumbnail of KPIs+for+AI+Agents+and+Generative+AI.pdf]
Preview
KPIs+for+AI+Agents+and+Generative+AI.pdf - Published Version

Download (612kB) | Preview
Information
Library
Metrics

Altmetric Metrics

Dimensions Matrics

Statistics

Downloads

Downloads per month over past year

View Item