Bias, arising from inductive assumptions, is necessary for successful artificial learning, allowing algorithms to generalize beyond training data and outperform random guessing. We explore how bias relates to algorithm flexibility (expressivity). Expressive algorithms alter their outputs as training data changes, allowing them to adapt to changing situations. Using a measure of algorithm flexibility rooted in the information-theoretic concept of entropy, we examine the trade-off between bias and expressivity, showing that while highly biased algorithms may outperform uniform random sampling, they cannot also be highly expressive. Conversely, maximally expressive algorithms necessarily have performance no better than uniform random guessing. We establish that necessary trade-offs exist in trying to design flexible yet strongly performing learning systems.
CITATION STYLE
Montañez, G. D., Bashir, D., & Lauw, J. (2021). Trading Bias for Expressivity in Artificial Learning. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 12613 LNAI, pp. 332–353). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-71158-0_16
Mendeley helps you to discover research relevant for your work.