Case study · 2025
S&P 500 ML Predictor
A full-stack web app predicting S&P 500 directional movements with ensemble models, honest walk-forward backtesting, and an interactive dashboard.
Role
Solo project
Stack
Python, Flask, Scikit-learn, Plotly
Links
Highlights
- Built a full-stack app using Random Forest and Gradient Boosting to predict S&P 500 directional movements from historical OHLCV data.
- Engineered a pipeline transforming raw OHLCV data into 25+ technical indicators (RSI, MACD, Bollinger Bands, lag features), then tested feature subsets to isolate the most predictive signals.
- Implemented walk-forward validation to prevent temporal data leakage, reaching 57.3% directional accuracy — 7.3% above a random baseline.
- Built interactive Plotly dashboards and modular experimentation workflows for comparing model performance and interpreting results.
Context
I built this project to apply what I had learned in my first-year AI module to a real problem. We had covered core machine learning models such as linear models and random forests, and I wanted practical experience using them beyond classwork. I chose financial time-series data specifically because it is more challenging than a standard static dataset: patterns shift over time, noise is high, and evaluation has to be handled carefully to avoid misleading results. That made it a strong test of whether I could apply machine learning properly in a realistic setting, while also pushing me to explore methods like gradient boosting beyond the module.
Solution
I developed a full-stack web application using Python, Flask, Scikit-learn, and Plotly to predict S&P 500 directional movements from historical market data. I built a pipeline that transformed raw OHLCV data into 25+ technical indicators, including RSI, MACD, Bollinger Bands, and lag-based features, then used these to train and compare models such as Random Forest and Gradient Boosting. A key part of the project was not just generating features, but testing which combinations were actually useful. Rather than assuming that more features would lead to better performance, I experimented with different feature subsets and compared their results systematically — testing groups of indicators, observing how they changed model behaviour, and using repeated evaluation to identify which sets were most predictive and which added noise. To keep this realistic, I implemented walk-forward validation, ensuring the models were always tested on future data. I also built interactive dashboards to visualise results, compare models, and make the experimentation process easier to interpret and extend.
Challenges
The main challenge was that financial forecasting is inherently noisy, so building a model was much easier than building one that performed reliably. One of the hardest parts was deciding how to test features properly, because a feature that looked helpful in one setup could easily prove unstable over time. That meant thinking carefully about methodology as well as modelling: how to compare feature sets fairly, avoid data leakage, and judge whether apparent improvements were meaningful or just random variation. Another challenge was moving from a machine learning model on its own to a complete application, which meant building a more robust data pipeline, handling real market data cleanly, and presenting results usefully rather than just technically correctly.
What I learned
The project taught me that applied machine learning is not just about training a model and reporting accuracy. Using walk-forward validation, the system achieved around 57% directional accuracy, which outperformed a random baseline but was still lower than I had initially hoped. That was an important outcome in itself, because it taught me to evaluate technical work honestly: a sound pipeline and a promising idea do not automatically mean a strong real-world solution. Just as importantly, experimenting with feature subsets showed me that model performance depends as much on the quality and relevance of inputs as on the model itself.
Future roadmap
If I developed the project further, I would focus on improving robustness rather than simply chasing a higher accuracy figure. That would include testing richer feature sets, comparing additional models, and exploring whether external signals such as financial news or text data could add useful context. I would also improve the product side by making the experimentation workflow more modular and strengthening monitoring so model behaviour could be tracked more effectively over time.