Using Stability Selection to combat feature instability

Methodology

I have been thinking about the common frustration where a variable looks significant in one model run but vanishes the moment you change the random seed or tweak the training set. It suggests that the "best" feature set might often be a lucky snapshot rather than a robust discovery. One approach to this is Stability Selection. Instead of relying on a single model run, you resample your data hundreds of times. For each subsample, you run your feature selection process and track how often each variable is chosen. You only keep features that exceed a specific threshold, such as 80 percent. To be fair, there is a strong argument for the traditional single-run approach. If a dataset is sufficiently powered and the model is correctly specified, the resulting feature set should be representative. Adding the computational overhead of 500 runs could be seen as overkill if the initial sample is already a high-fidelity reflection of the population. However, consider a hypothetical scenario where a small cluster of outliers in one specific slice of data is driving a variable's significance. In a single run, that variable appears essential. In a stability framework, those outliers only appear in a fraction of the subsamples. The variable fails the selection threshold, and you avoid a false positive. The implementation is relatively direct: 1. Define a subsampling rate (for example, 50 percent of the data). 2. Run your selection algorithm, such as LASSO, on 100 to 1,000 different random subsamples. 3. Calculate the selection probability for each feature (total times selected divided by total runs). 4. Retain only the features that meet your stability threshold (e.g., 0.8). This shifts the goal from finding the mathematically optimal set for one specific dataset to finding the features that are most resilient to perturbations in the data.