Based on the work by D. Beretta et al., Sustainable Energy, Grids and Networks, Volume 21, March 2020, 100308. Readers are encouraged to consult the manuscript to master the methodology.
These tools are designed for scientists, researchers, and engineers working in all fields of electrical grid optimization. They provide capabilities to apply FPCA to daily electricity load curves and predict future consumption patterns based on historical data.
The fpca-load-tools can be installed either via pip or by cloning this repository.
To install the tools via pip, use the following commands based on your operating system:
- On Unix/Mac: Open a terminal and run:
pip install fpca-load-tools
- On Windows: Open Command Prompt or PowerShell and run:
pip install fpca-load-tools
To install the app by cloning the repository to your local machine, follow these steps:
- Clone the repository:
git clone https://github.com/BerriesLab/fpca-load-tools.git
- Navigate to the project directory:
cd project-name - Install the required dependencies:
pip install -r requirements.txt
fpca-load-tools is designed around three main classes:
ElectricityLoadTimeSeries: Manages time series data within a Pandas DataFrame, including pre-processing operations such as filtering for complete time series and augmenting time series with calendar information.ElectricityLoadFPCA: Applies Functional Principal Component Analysis (FPCA) to daily electricity load curves. This class requires an ElectricityLoadTimeSeries object as attribute.- Note 1: The ElectricityLoadTimeSeries object is stored as a reference in ElectricityLoadFPCA. Therefore, any changes made to the ElectricityLoadTimeSeries object outside ElectricityLoadFPCA will also affect the data being processed by the FPCA.
- Note 2: The results of the FPCA, including the scores, are stored in the results attribute of the class.
ElectricityLoadRegression: Trains a model and predicts daily electricity load curves using FPCA data. It requires an ElectricityLoadFPCA object as attribute.- Note 1: The ElectricityLoadFPCA object is stored as a reference in ElectricityLoadRegression.Therefore, any modifications to the ElectricityLoadFPCA object outside ElectricityLoadRegression will affect the data being processed by the regressor.
- Note 2: The results of the training are stored in the class attributes 'model' and 'scaler'.
The following figure is a graphical representation of the classes with their own attributes and methods.
classDiagram
class ElectricityLoadTimeSeries {
• ts: pd.DataFrame
• filter_complete_data()
• filter_complete_days()
• filter_complete_months()
• filter_complete_years()
• filter_non_null_entries()
• resample_days()
• augment_time_series_with_day_of_the_week()
• augment_time_series_with_year_month_day()
• drop_year_month_day()
• convert_utc_to_local_timestamp()
• sort()
• save_time_series()
• load_time_series()
• load_example_entsoe_transparency()
}
class ElectricityLoadFPCA {
• ts: ElectricityLoadTimeSeries
• results: ElectricityLoadFPCAResults
• apply_fpca_to_all_days_grouped_by_date()
• apply_fpca_to_all_days_grouped_by_weekday()
• apply_fpca_to_all_days_grouped_by_month()
• plot_scores_vs_day_of_the_week()
• plot_scores_vs_month_of_the_year()
• plot_cdf_of_explained_variability()
• plot_fpc()
• plot_functional_boxplot()
• save_fpca_results()
• load_fpca_results()
}
class ElectricityLoadFPCAResults {
• day = None
• day_of_the_week = None
• month_of_the_year = None
}
class ElectricityLoadRegression {
• fpca: ElectricityLoadFPCA
• model: dict[LinearRegression]
• scaler: StandardScaler
• train_linear_model()
• predict_daily_eletricity_load_curve()
• save_model()
• load_model()
}
ElectricityLoadFPCAResults --|> ElectricityLoadFPCA
ElectricityLoadFPCA --|> ElectricityLoadRegression
ElectricityLoadTimeSeries --|> ElectricityLoadFPCA
Users can load time series from CSV files using the load_time_series() method of the
ElectricityLoadTimeSeries class. The expected data structure in the CSV file is:
| utc_timestamp | load | feature_1 | feature_2 | ... | feature_n |
|---|---|---|---|---|---|
| ... | ... | ... | ... | ... | ... |
- utc_timestamp: The timestamp in Coordinated Universal Time (UTC) or Greenwich Mean Time (GMT).
- load: The electricity load measurement.
- feature_1 to feature_n: Additional features for analysis and/or prediction.
Upon loading, the CSV file is converted into a Pandas DataFrame with a DateTimeIndex based on the utc_timestamp.
Users can load multiple files and features as needed. The method automatically merges new CSV files with the existing
DataFrame in memory on the DateTimeIndex. Users should ensure that only one column named load is present in
memory. To help the user, load_time_series() allows to select which columns to load from
teh CSV file and to choose the names for these columns in the destination DataFrame. If multiple columns with the same
name are loaded, Pandas will handle them by renaming the new columns with suffixes (e.g., column_name_r).
An example of meteorological time series data that could be merged with the electricity load time series is:
| utc_timestamp | temperature | radiation | relative_humidity |
|---|---|---|---|
| ... | ... | ... | ... |
To save time series data to CSV files, users can use the save_time_series() method.
When studying electricity load time series, the choice between using UTC (Coordinated Universal Time) and local time depends on the objectives of the analysis and the nature of the data. For standardization purposes, such as comparing electricity load across different time zones, using UTC provides a uniform time reference and simplifies time zone conversions. However, if the goal is to investigate consumer behavior, local time may be more relevant since electricity load often correlates with human activities and routines, which follow local time patterns (e.g., peak load times during mornings and evenings). Similarly, for operational planning, such as scheduling generation or demand response activities, local time aligns better with the actual timing of events and conditions experienced by consumers and grid operators.
To facilitate this, the ElectricityLoadTimeSeries class includes the
method convert_utc_to_local_timestamp, which converts the UTC DateTimeIndex to the
corresponding local timestamp. This method requires the user to specify the geographical area in Olson Timezone format.
It is important to note that converting from UTC to local time, including accounting for daylight saving time, can result in days with duplicate entries or missing values. To address these issues, you can either resample the days with duplicates or missing entries or remove days that do not meet completeness and integrity requirements. For detailed guidance on handling these issues, see the section on Filtering complete time series
To execute FPCA and predict future electricity load curves, it is essential that the dataset is complete. The completeness criteria are as follows:
- Year Completeness: A year is considered complete if the number of months with non-null entries meets or exceeds a tolerance percentage of the expected number of months, which is 12. By default, this tolerance level is set to 11/12.
- Month Completeness: A month is considered complete if the number of days with non-null entries meets or exceeds a tolerance percentage of the expected number of calendar days. By default, this tolerance level is set to 95% of the month's calendar days.
- Day Completeness: A day is considered complete if the number of non-null entries meets or exceeds a tolerance percentage of the expected number of entries. By default, this tolerance level is set to 100% of the expected entries.
To filter a complete dataset, the user can use the filter_complete_data() method
from the ElectricityLoadTimeSeries class. This method utilizes four sequentially
executed methods:
filter_non_null_entries(): Delete all rows with at least oneNonevalue.filter_complete_years(): Remove incomplete years. The default tolerance is set to 11/12.filter_complete_months(): Remove incomplete months. The default tolerance is set to 95% of the month’s calendar days.filter_complete_days(): Remove incomplete days. The default tolerance is set to 100% of the mode of the time series grouped by date.
Note 1: As the first step, entries with null data are dropped by the filter_non_null_entries method. This is
essential because the subsequent methods only evaluate the DateTimeIndex values, regardless of the columns actual values.
Note 2: When filtering days with a tolerance level less than 100% or converting timestamps from UTC to local time, the
resulting time series may include missing values. To address this, the user can use
the resample_days() method. This method resamples the time series daily with a
user-defined frequency (defaulting to one hour). Missing values are linearly interpolated between their nearest
neighbors, and any remaining None values at the beginning or end of an interpolated period are filled with the nearest
neighbor value.
The standard PCA in scikit-learn expects a 2D data matrix where each row represents a sample and each column represents a feature. In the context of FPCA, the "features" are values of the functions at discretized points.
The class ElectricityLoadFPCA offers three methods to apply three different types of FPCAs:
apply_fpca_to_all_days_grouped_by_date(): Applies FPCA to daily curves grouped by date.apply_fpca_to_all_days_grouped_by_weekday(): Applies FPCA to daily curves grouped by day of the week.apply_fpca_to_all_days_grouped_by_month(): Applies FPCA to daily curves grouped by month of the year.
The results from each FPCA are stored in an instance of the ElectricityLoadFPCAResults
class, which is the 'results' attribute of ElectricityLoadFPCA.Note that only one result
per FPCA type can be stored at a time: performing an analysis again will overwrite any previous results. For example,
running apply_fpca_to_all_days_grouped_by_date() a second time will replace the results from the
first analysis.
FPCA results can be saved to and loaded from a pickle file on disk using the following methods:
The ElectricityLoadFPCAResults class provides several plotting methods for visualizing
FPCA results, similar to the visualizations reported in D. Beretta et al., Sustainable Energy, Grids and Networks,
Volume 21, March 2020, 100308. These methods include:
plot_functional_boxplot(): Plots a functional boxplot that overlays all daily load curves with median and interquartile bands.
Figure 1: Functional boxplot for a representative dataset.
plot_fpc(): Plots the Functional Principal Components (FPCs), rescaled according to their explained variance ratio.
Figure 2: FPCs of a representative dataset rescaled by their explained variance ratio.
plot_cdf_of_explained_variability(): Plots the Cumulative Distribution Function (CDF) of the explained variability percentage as a function of the number of FPCs.
Figure 3: CDF of a representative dataset.
plot_scores_vs_day_of_the_week(): Plots a boxplot of FPC scores versus the day of the week for the first n FPCs.
Figure 4: Scores boxplot of a representative dataset vs day of the week.
plot_scores_vs_month_of_the_year(): Plots a boxplot of FPC scores versus the month of the year for the first n FPCs.
Figure 5: Scores boxplot of a representative dataset vs month of the year.
Note: All above methods collect the data to plot from the ElectricityLoadFPCA class' attributes.
FPCA can be integrated into any time-series predictive model to predict daily electricity load curves. Unlike traditional time-series models that predict actual data, FPCA-based models predict the scores of a selected number of Functional Principal Components (FPCs). This approach balances model complexity and explained variability. For more details on this methodology, refer to D. Beretta et al., Sustainable Energy, Grids and Networks, Volume 21, March 2020, 100308.
The functional decomposition allows to cast the electricity load curve of a given day in the form:
where
where
Note: Since the model predicts the FPCs scores, and since the FPCs are daily time series, the features must be averaged over the day, e.g. the average temperature of the day.
The class ElectricityLoadRegression handles the prediction process. It can be instantiated with or
without passing an instance of ElectricityLoadFPCA.
The ElectricityLoadRegression class provides a method for training a linear model and a method for
predicting the electricity load curves. Specifically:
train_linear_model(): Trains the model described in section The model iteratively on the first n FPCs using scikit-learn LinearRegression. This results in n weight matrices, one for each FPC, which are stored as part of the respective objects in the class 'model' attribute.
Figure 6: Actual vs predicted scores of FPC1 for a representative dataset.
predict_daily_electricity_load_curve(): Predicts the electricity load curve for a specified future date, and returns a list of prediction metrics, including the percentage power error.
Figure 7: Actual vs predicted electricity load curve.
The ElectricityLoadRegression class includes methods for saving and loading the model parameters and the feature scaler:
save_model(): Saves the model and feature scaler to pickle file.load_model(): Loads a previously saved model and feature scaler from pickle file.
Please follow the tutorial to learn how to use fpca-load-tools in practice.
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests.
This app has been developed by D. Beretta, building on the work by D. Beretta et al., Sustainable Energy, Grids and Networks, Volume 21, March 2020, 100308. Please refer to CREDITS.md and CITATION.md for more details.
This project is licensed under the GNU License - see the LICENSE file for details.