Improving Fairness with Ensemble Combination: Margin-Dependent Bounds

The concern about hidden discrimination in machine learning models is growing, as their widespread real-world applications increasingly impact human lives. Various techniques, including commonly used group fairness measures and several fairness-aware ensemble-based methods, have been developed to enhance fairness. However, existing fairness measures typically focus on only one aspect -- either group or individual fairness, and the compatibility difficulty among these measures indicates a possibility of remaining biases even when one of them is satisfied. Moreover, existing mechanisms to boost fairness usually present empirical results to show validity, yet few of them discuss whether fairness can be boosted with certain theoretical guarantees. To address these issues, we propose a fairness quality measure named `discriminative risk' by only perturbing protected attributes in instances, to express both individual and group fairness aspects. Furthermore, we investigate its properties and establish the first- and second-order oracle bounds and their relaxations, which show that fairness is possibly improved via ensemble combination with margin-dependent bounds. The analysis is suitable for both binary and multi-class classification. A few ensemble pruning methods are also proposed to utilise our proposed measure and obtain both accurate and fair sub-ensembles; comprehensive experiments are conducted to evaluate the effectiveness of the proposed fairness measure and pruning methods.