Efficient Bounded Exhaustive Input Generation from Program APIs

Abstract

Bounded exhaustive input generation (BEG) is an effective approach to reveal software faults. However, existing BEG approaches require a precise specification of the valid inputs, i.e., a repOK, that must be provided by the user. Writing repOKs for BEG is challenging and time consuming, and they are seldom available in software. In this paper, we introduce BEAPI, an efficient approach that employs routines from the API of the software under test to perform BEG. Like API-based test generation approaches, BEAPI creates sequences of calls to methods from the API, and executes them to generate inputs. As opposed to existing BEG approaches, BEAPI does not require a repOK to be provided by the user. To make BEG from the API feasible, BEAPI implements three key pruning techniques: (i) discarding test sequences whose execution produces exceptions violating API usage rules, (ii) state matching to discard test sequences that produce inputs already created by previously explored test sequences, and (iii) the automated identification and use of a subset of methods from the API, called builders, that is sufficient to perform BEG. Our experimental assessment shows that BEAPI’s efficiency and scalability is competitive with existing BEG approaches, without the need for repOKs. We also show that BEAPI can assist the user in finding flaws in repOKs, by (automatically) comparing inputs generated by BEAPI with those generated from a repOK. Using this approach, we revealed several errors in repOKs taken from the assessment of related tools, demonstrating the difficulties of writing precise repOKs for BEG.