Modern perception systems for autonomous vehicles are often dependent on deep neural networks, however, such networks are unfortunately susceptible to subtle perturbations to their inputs. Due to the interconnected nature of perception/control systems in autonomous vehicles, it is quite difficult to evaluate the autonomy stack’s robustness in different scenarios. Numerous tools have been developed to assist developers increase the robustness of these algorithms for on-road driving, but little has been accomplished for off-road driving. This work aims to bridge this gap by presenting a reinforcement learning framework to identify unsuspecting off-road scenes that confuse a custom autonomy stack with a DNN-based perception algorithm to ultimately lead the vehicle into a collision.