Authors: Augusto Cerqua, Marco Letta, Gabriele Pinto
studying (ML) has gained a central position in economics, the social sciences, and enterprise decision-making. Within the public sector, ML is more and more used for so-called prediction coverage issues: settings the place policymakers purpose to establish models most liable to a unfavorable consequence and intervene proactively; for example, concentrating on public subsidies, predicting native recessions, or anticipating migration patterns. Within the personal sector, related predictive duties come up when companies search to forecast buyer churn, or optimize credit score threat evaluation. In each domains, higher predictions translate into extra environment friendly allocation of sources and simpler interventions.
To attain these objectives, ML algorithms are more and more utilized to panel knowledge, characterised by repeated observations of the identical models over a number of time intervals. Nevertheless, ML fashions weren’t initially designed to be used with panel knowledge, which characteristic distinctive cross-sectional and longitudinal dimensions. When ML is utilized to panel knowledge, there’s a excessive threat of a delicate however significant issue: knowledge leakage. This happens when info unavailable at prediction time by chance enters the mannequin coaching course of, inflating predictive efficiency. In our paper “On the Mis(Use) of Machine Studying With Panel Knowledge” (Cerqua, Letta, and Pinto, 2025), lately printed within the Oxford Bulletin of Economics and Statistics, we offer the primary systematic evaluation of knowledge leakage in ML with panel knowledge, suggest clear tips for practitioners, and illustrate the implications via an empirical software with publicly accessible U.S. county knowledge.
The Leakage Drawback
Panel knowledge mix two buildings: a temporal dimension (models noticed throughout time) and a cross-sectional dimension (a number of models, resembling areas or companies). Normal ML apply, splitting the pattern randomly into coaching and testing units, implicitly assumes impartial and identically distributed (i.i.d.) knowledge. This assumption is violated when default ML procedures (resembling a random cut up) are utilized to panel knowledge, creating two major sorts of leakage:
- Temporal leakage: future info leaks into the mannequin throughout the coaching section, making forecasts look unrealistically correct. Moreover, previous info can find yourself within the testing set, making ‘forecasts’ retrospective.
- Cross-sectional leakage: the identical or very related models seem in each coaching and testing units, that means the mannequin has already “seen” a lot of the cross-sectional dimension of the info.
Determine 1 reveals how completely different splitting methods have an effect on the chance of leakage. A random cut up on the unit–time degree (Panel A) is probably the most problematic, because it introduces each temporal and cross-sectional leakage. Alternate options resembling splitting by models (Panel B), by teams (Panel C), or by time (Panel D), mitigate one kind of leakage however not the opposite. Consequently, no technique utterly eliminates the issue: the suitable selection will depend on the duty at hand (see under), since in some circumstances one type of leakage will not be an actual concern.
Determine 1 | Coaching and testing units underneath completely different splitting guidelines
Two Varieties of Prediction Coverage Issues
A key perception of the examine is that researchers should clearly outline their prediction objective ex-ante. We distinguish two broad courses of prediction coverage issues:
1. Cross-sectional prediction: The duty is to map outcomes throughout models in the identical interval. For instance, imputing lacking knowledge on GDP per capita throughout areas when just some areas have dependable measurements. The most effective cut up right here is on the unit degree: completely different models are assigned to coaching and testing units, whereas all time intervals are saved. This eliminates cross-sectional leakage, though temporal leakage stays. However since forecasting isn’t the objective, this isn’t an actual concern.
2. Sequential forecasting: The objective is to foretell future outcomes primarily based on historic knowledge—for instance, predicting county-level revenue declines one 12 months forward to set off early interventions. Right here, the proper cut up is by time: earlier intervals for coaching, later intervals for testing. This avoids temporal leakage however not cross-sectional leakage, which isn’t an actual concern because the identical models are being forecasted throughout time.
The unsuitable strategy in each circumstances is the random cut up by unit-time (Panel A of Determine 1), which contaminates outcomes with each sorts of leakage and produces misleadingly excessive efficiency metrics.
Sensible Tips
To assist practitioners, we summarize a set of do’s and don’ts for making use of ML to panel knowledge:
- Select the pattern cut up primarily based on the analysis query: unit-based for cross-sectional issues, time-based for forecasting.
- Temporal leakage can happen not solely via observations, but in addition via predictors. For forecasting, solely use lagged or time-invariant predictors. Utilizing contemporaneous variables (e.g., utilizing unemployment in 2014 to foretell revenue in 2014) is conceptually unsuitable and creates temporal knowledge leakage.
- Adapt cross-validation to panel knowledge. Random k-fold CV present in most ready-to-use software program packages is inappropriate, because it mixes future and previous info. As a substitute, use rolling or increasing home windows for forecasting, or stratified CV by models/teams for cross-sectional prediction.
- Be sure that out-of-sample efficiency is examined on actually unseen knowledge, not on knowledge already encountered throughout coaching.
Empirical Utility
For instance these points, we analyze a balanced panel of three,058 U.S. counties from 2000 to 2019, focusing completely on sequential forecasting. We think about two duties: a regression drawback—forecasting per capita revenue—and a classification drawback—forecasting whether or not revenue will decline within the subsequent 12 months.
We run lots of of fashions, various cut up methods, use of contemporaneous predictors, inclusion of lagged outcomes, and algorithms (Random Forest, XGBoost, Logit, and OLS). This complete design permits us to quantify how leakage inflates efficiency. Determine 2 under stories our major findings.
Panel A of Determine 2 reveals forecasting efficiency for classification duties. Random splits yield very excessive accuracy, however that is illusory: the mannequin has already seen related knowledge throughout coaching.
Panel B reveals forecasting efficiency for regression duties. As soon as once more, random splits make fashions look much better than they are surely, whereas right time-based splits present a lot decrease, but real looking, accuracy.
Determine 2 | Temporal leakage within the forecasting drawback
Panel A – Classification job
Panel B – Regression job
Within the paper, we additionally present that the overestimation of mannequin accuracy turns into considerably extra pronounced throughout years marked by distribution shifts and structural breaks, such because the Nice Recession, making the outcomes notably deceptive for coverage functions.
Why It Issues
Knowledge leakage is greater than a technical pitfall; it has real-world penalties. In coverage functions, a mannequin that appears extremely correct throughout validation might collapse as soon as deployed, resulting in misallocated sources, missed crises, or misguided concentrating on. In enterprise settings, the identical concern can translate into poor funding selections, inefficient buyer concentrating on, or false confidence in threat assessments. The hazard is particularly acute when machine studying fashions are supposed to function early-warning programs, the place misplaced belief in inflated efficiency can lead to expensive failures.
In contrast, correctly designed fashions, even when much less correct on paper, present sincere and dependable predictions that may meaningfully inform decision-making.
Takeaway
ML has the potential to remodel decision-making in each coverage and enterprise, however provided that utilized accurately. Panel knowledge provide wealthy alternatives, but are particularly weak to knowledge leakage. To generate dependable insights, practitioners ought to align their ML workflow with the prediction goal, account for each temporal and cross-sectional buildings, and use validation methods that stop overoptimistic assessments and an phantasm of excessive accuracy. When these ideas are adopted, fashions keep away from the entice of inflated efficiency and as an alternative present steering that genuinely helps policymakers allocate sources and companies make sound strategic decisions. Given the fast adoption of ML with panel knowledge in each private and non-private domains, addressing these pitfalls is now a urgent precedence for utilized analysis.
References
A. Cerqua, M. Letta, and G. Pinto, “On the (Mis)Use of Machine Studying With Panel Knowledge”, Oxford Bulletin of Economics and Statistics (2025): 1–13, https://doi.org/10.1111/obes.70019.
