Abstract by Xuan Cao
A Sample-Based Approach for Measuring Autonomous Agents Competency: Preliminary Results
Autonomous agents are designed to achieve goals, whatever the goals are. The competency of an autonomous agent tells how likely the agent will achieve its goal given limited time. We report a sample-based method to measure competency, as well as some preliminary results derived from a basic maze environment where an agent follows the policy computed by the Value Iteration algorithm to reach a specific “goal” position from random starting positions. We ran multiple simulations for the agent and plotted curves representing the relationships between success rate and time limitation, which is referred to as “competency curves”. We applied various sigmoid curves to fit the plot and found the 3-parameter Gompertz curve was the best. We explored how the estimated parameters of Gompertz curves vary with the starting positions of the agent. These preliminary results help to build intuition about competency and enhance the basis of competency measurement for more complicated agents.