Electricity Forecasting Framework#

Overview#

This skill provides end-to-end support for electricity load/demand forecasting projects, from data preprocessing to model deployment. It covers traditional statistical methods, modern machine learning approaches, and state-of-the-art deep learning architectures.

Quick Start#

1. Define Your Forecasting Task#

HorizonTypeTypical Use1-48 hoursShort-term (STLF)Grid operations, unit commitment1 week - 1 monthMedium-termMaintenance scheduling, fuel planning1-12 monthsLong-term (LTLF)Capacity planning, infrastructure investment

2. Prepare Your Data#

bash

# Run the data preparation script
python scripts/prepare_data.py --input raw_load.csv --output processed/

Required data columns:

• timestamp: Datetime index (hourly or sub-hourly)
• load: Target variable (MW or kWh)
• temperature: Weather feature (°C)
• Optional: humidity, wind_speed, solar_radiation, holiday_flag

3. Select Your Model#

See references/model-selection.md for detailed guidance.

Quick recommendation:

• Baseline: Start with persistence or seasonal-naive
• Production STLF: Use XGBoost or LightGBM with weather features
• Research/SOTA: Try Temporal Fusion Transformer (TFT) or iTransformer

4. Train and Evaluate#

bash

python scripts/train_model.py --model xgboost --data processed/ --horizon 24

Key metrics to track:

• MAPE (%): Mean Absolute Percentage Error - business interpretability
• RMSE (MW): Root Mean Square Error - penalizes large errors
• MAE (MW): Mean Absolute Error - robust to outliers
• Coverage (%): Prediction interval coverage probability

Core Workflows#

Data Preprocessing#

1. Load raw data with proper datetime parsing

2. Handle missing values: Forward-fill for short gaps, interpolate for longer

3. Feature engineering:

   • Temporal: hour, day_of_week, month, is_weekend, is_holiday
   • Lag features: load_t-1, load_t-24, load_t-168 (weekly)
   • Rolling stats: rolling_mean_24h, rolling_std_7d
   • Weather: temperature, humidity, apparent_temperature

4. Normalization: RobustScaler or MinMaxScaler for deep learning models

See references/feature-engineering.md for complete feature list.

Model Training#

python

# Example training workflow
from electricity_forecasting import ForecastPipeline

pipeline = ForecastPipeline(
    model_type="xgboost",
    horizon=24,
    lookback=168  # 1 week of history
)

pipeline.fit(train_data, val_data)
predictions, uncertainty = pipeline.predict(test_data)
metrics = pipeline.evaluate(predictions, actuals)

Hyperparameter Tuning#

Use scripts/hyperparameter_search.py for automated tuning:

bash

python scripts/hyperparameter_search.py \
  --model lightgbm \
  --data processed/ \
  --n-trials 50 \
  --study-name stlf-tuning

Uncertainty Quantification#

For risk-aware decision making:

• Quantile Regression: Predict multiple quantiles (0.1, 0.5, 0.9)
• Conformal Prediction: Distribution-free uncertainty bounds
• Ensemble Methods: Model disagreement as uncertainty proxy
• Monte Carlo Dropout: For neural networks

See references/uncertainty.md for implementation details.

Model Reference#

Statistical Models#

ModelBest ForProsConsARIMAStable seriesInterpretable, fastAssumes linearitySARIMAStrong seasonalityCaptures daily/weekly patternsManual parameter tuningProphetMultiple seasonalitiesHandles holidays wellLess accurate for STLFTBATSComplex seasonalityAutomatic parameter selectionSlower training

Machine Learning Models#

ModelBest ForProsConsXGBoostProduction STLFFast, accurate, handles missingNo native uncertaintyLightGBMLarge datasetsFaster than XGBoost, memory efficientSensitive to hyperparametersRandom ForestBaseline MLRobust, easy to tuneLower accuracy than boostingCatBoostCategorical featuresHandles categoricals nativelySlower training

Deep Learning Models#

ModelBest ForProsConsLSTMSequential patternsCaptures long-term dependenciesSlow training, hard to tuneGRUSimilar to LSTMFaster convergenceSimilar limitationsTransformerLong sequencesParallel training, attentionData-hungry, complexTFTMulti-horizonInterpretable attention, uncertaintyComplex implementationN-BEATSPure deep learningStrong baseline, interpretableLess flexible than TFTiTransformerSOTA performanceInverted transformer architectureRecent, less battle-tested

See references/deep-learning-models.md for architecture details and PyTorch implementations.

Evaluation Best Practices#

Time Series Cross-Validation#

Never use random k-fold! Use expanding or sliding window:

python

# Expanding window CV
from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5, test_size=168)  # 1 week test
for train_idx, test_idx in tscv.split(data):
    train, test = data[train_idx], data[test_idx]
    # Train and evaluate

Backtesting Framework#

bash

python scripts/backtest.py \
  --model xgboost \
  --data processed/ \
  --cv-splits 5 \
  --horizon 24 \
  --metrics mape,rmse,mae

Benchmark Comparison#

Always compare against:

1. Persistence: load_t = load_t-1
2. Seasonal Naive: load_t = load_t-24 (for hourly data)
3. Weekly Naive: load_t = load_t-168

Deployment#

Production Pipeline#

1. Model serialization: Save with joblib or ONNX
2. Feature pipeline: Ensure identical preprocessing at inference
3. Scheduling: Cron or Airflow for automated forecasts
4. Monitoring: Track forecast drift and retrain triggers

See references/deployment.md for MLOps patterns.

Real-time Inference#

python

from electricity_forecasting import DeploymentModel

model = DeploymentModel.load("models/xgboost-stlf.joblib")
features = prepare_features(latest_data)
prediction = model.predict(features, return_uncertainty=True)

Common Pitfalls#

1. Data leakage: Ensure no future information in features
2. Holiday handling: Special days need explicit modeling
3. Temperature nonlinearity: Use heating/cooling degree days
4. Concept drift: Retrain quarterly or when MAPE degrades >20%
5. Peak prediction: Models often under-predict peaks - consider quantile loss

Resources#

• Feature Engineering Guide
• Model Selection Guide
• Deep Learning Architectures
• Uncertainty Quantification
• Deployment Patterns
• Datasets Reference

Scripts#

ScriptPurposescripts/prepare_data.pyData cleaning and feature engineeringscripts/train_model.pyModel training with validationscripts/hyperparameter_search.pyAutomated hyperparameter optimizationscripts/backtest.pyTime series cross-validationscripts/evaluate.pyComprehensive metric calculationscripts/deploy_model.pyExport model for production

Example Usage#

bash

# Complete workflow example
# 1. Prepare data
python scripts/prepare_data.py --input data/load_2024.csv --output data/processed/

# 2. Train model
python scripts/train_model.py --model lightgbm --data data/processed/ --horizon 48

# 3. Hyperparameter tuning
python scripts/hyperparameter_search.py --model lightgbm --data data/processed/ --n-trials 100

# 4. Backtest
python scripts/backtest.py --model lightgbm-best --data data/processed/ --cv-splits 5

# 5. Deploy
python scripts/deploy_model.py --model lightgbm-best --output models/production/