To decode task-relevant information from sensory observations, the brain must eliminate nuisance variables that affect those observations. For natural tasks, this generally requires nonlinear computation. Here we contribute new concepts and methods to characterize behaviorally relevant nonlinear computation downstream of recorded neurons. Linear decoding weights can be inferred from correlations between neurons and behavior. However, these weights do not adequately describe the neural code when, due to nuisance variation, mean neural responses are poorly tuned to the task while higher-order statistics of neural responses are well tuned. The task-relevant stimulus information can then be extracted only by nonlinear operations. For example, detecting an object boundary in an image requires contrast invariance: an edge appears when the foreground object is darker lighter than the background, yet any linear function will exhibit opposite responses in these two cases. We generalize past weight-inference methods to determine the brain's nonlinear neural computations from joint higher-order statistics of neural activity and behavioral choices in perceptual tasks. This method is based on a new statistical measure we call nonlinear choice correlation, defined as the correlation coefficient between behavioral choices and nonlinear functions of measured neural responses. Importantly, the exact neural transformations may not be uniquely identifiable, since many neural nonlinearities can generate the same behavioral output. This is expected when sensory signals are expanded into a larger cortical response space, creating a redundant code. We exploit this redundancy to define a new concept of equivalence classes for neural transformations. We then demonstrate how to quantify essential properties of these equivalence classes, and provide simulations that show how these properties can be extracted using neural data from behaving animals. Finally, we explain the functional importance of these nonlinearities in specific perceptual tasks.
