Semantic Action Graph: A Shared Representation for Agent Grounding and Human Interpretation of Sports Highlights
Tica Lin, Deepak Chandran, Gauri Jagatap, Chen Chen, Andrea Fanelli, David Gunawan, Josh Kimball
VIS 2026 Workshop on GenAI, Agents, and the Future of VIS
We introduce Semantic Action Graph, a structured domain schema representing sports matches through performer, action, recipient, moment, and state components connected by specific edge types. This framework enables two parallel uses: powering an automated pipeline that generates narrated highlights and providing a visual query interface for viewer interaction.
We developed SportSAGE, pairing a four-module highlight system with a graph interface, and gathered responses from twelve soccer enthusiasts. Participants found the generated highlights and narratives satisfactory and leveraged the graph interface to explore and comprehend match events. The study suggests that one small, human-readable schema can ground agent generation and support human interpretation at the same time.
