NASA’s C-MAPSS (dataset to assess the engine’s lifetime and engine degradation. We will compare the SARIMAX results with other models with hyperparameter search.
This dataset and calculations are for educational purposes only solely based on author's analysis.
To better understand the behaviour of our data, we divided our samples into 4 quartiles to check if there is a trend in every quarter of the whole time cycle. The techniques and discussion will not be shown here for brevity.
Backward elimination with Elastic Net is a feature selection technique used in building statistical models. It involves starting with all candidate variables and successively removing the least significant variables until only those that have a significant impact on the regression model remain. This process can help identify the most effective predictors and simplify the model without sacrificing predictive power.
Notice that a high condition number typically indicates the presence of multicollinearity among the predictor variables in our model. Multicollinearity occurs when one or more predictors are highly linearly dependent on others. This can make the regression coefficients unstable and very sensitive to small changes in the model or the data despite the fact that we a chose the best predictors where p-values are less than 0.05. Note of the Bypass Duct Pressure was not captured by backward regression and permutation shuffling as one of the important predictor unless plotted. Each units that passed through the optimal 'dip' has longer time cycle. Traditional feature importance methods in predictive models often look at features in isolation or in the context of a specific predictive framework, potentially missing interactions that only become apparent when data is visualized or grouped in high-dimensional space. Clustering algorithms excels in capturing these kinds of interactions by nature of its approach to grouping similar observations.
Summary Report After Hyperparameter Search with Elastic Net
Table of overlapping time-cycle for analysis.
Bypassduct Pressure Setting for Low/High Performance Engines
HPC Outlet Temperature Setting for Low/High Performance Engines
Absolute Area computed (17.73) for a unit under observation from optimal line condition. Higher values means the unit will tend to approach the failure curve rather than the optimal curve.
Absolute Area computed (4.367) for a unit under observation with failure curve. This line has shorter RUL as it approached the failure cuve.
Computing feature importance (RUL and unit no. included) via permutation shuffling for 20 features using random 5000 rows with 5 shuffle sets.
For our scenario where we are considering both binary classification and regression for predicting Remaining Useful Life (RUL), the decision on which model to train first depends on our specific goals and the nature of our data.
1. Binary Classification First, Then Regression:
Approach: First, classify whether an engine is near failure (e.g., will fail within a certain number of cycles). If classified as near failure, then use a regression model to predict the exact RUL or create survival probability model.
When to Use: This approach is useful if we have a lot of variance in the RUL across our data set, or if it’s more critical to identify engines that are nearing failure as a priority. By filtering out the samples not near failure, we can focus our regression model on a narrower, possibly more homogenous subset of the data where the prediction of exact RUL is more relevant and possibly more accurate.
2. Regression First, Then Binary Classification:
Approach: Start by predicting the RUL for all engines. Based on the predicted RUL, classify engines into categories such as 'near failure' or 'not near failure' based on a threshold (e.g., engines with a predicted RUL under a specific number of cycles are near failure).
When to Use: This method might be preferable if our primary goal is to maintain a continuous monitoring system that predicts the lifespan of all engines regardless of their current state. The classification post-regression can help in decision-making processes or prioritizing maintenance schedules without missing out on the broader context of engine health.
Recommended Strategy: Given that we are working with engines where early identification of potential failures could be crucial, starting with binary classification to focus on potentially failing engines might be more practical. This way, we can apply more complex and possibly resource-intensive regression models on a targeted subset of our data where accuracy in predicting exact RUL is most critical. This approach can potentially improve the performance of our regression model by reducing noise from the engines that are far from failure.
FIRST MODEL TRAINING WITH REGRESSIVE TASK PER QUANTILE:
quantile_1 MSE: 0.3355726799710202
quantile_2 MSE: 0.337087532108279
quantile_3 MSE: 0.309688467364816
quantile_4 MSE: 0.19429625238753873 (As we approached the End of Cycle, our prediction is becoming more accurate).
Cycle Dynamics: The decreasing MSE might reflect a pattern in our data where the target variable becomes more predictable or less variable towards the end of the cycle.
Model Strengths: Our initial model is well-suited for predicting values in the upper quantiles of the target distribution. This could be valuable if we are primarily interested in understanding or forecasting the behavior of the target variable when it takes on higher values, however, our model limitation which have higher MSE values for the lower quantiles suggest that the model struggles to accurately predict the target variable when it takes on lower values which in this case, short life cycle engines as samples in our previous plots.
FIRST MODEL TRAINING WITH CLASSIFICATION TASK FOR 10,000 RANDOM SAMPLES using stacking of base models using Random Forest Classifier, Gradient Boosting and Support Vector Machine at L3 level which is quite long time (7-9 hrs) of inference.
Accuracy: 0.922
Log Loss: 0.1739271976730883
F1 Score: 0.9220018720299524
ROC AUC: 0.9815661566156616
Precision: 0.9220701282805132
Recall: 0.922
Balanced Accuracy: 0.922052205220522
