Article by: Antonio Rubio, Project Engineer, Braking Systems in Applus IDIADA
Review Part One | Review Part Two | Review Part Three
The field of artificial intelligence (AI) has made significant progress in recent years, with applications ranging from natural language processing to computer vision. In recent years, Applus IDIADA Brakes department has presented several studies about artificial intelligence application for detection of brake noises. In this paper, Applus IDIADA presents the research done in this area, but focusing on the development of an AI model for predicting subjective ratings for squeal brake noises based on objective measurements collected through the instrumentation in a typical Brake Noise Durability programme. Subjective ratings are based on human opinions and can be challenging to quantify. Objective measurements, on the other hand, can be objectively quantified and provide a more reliable basis for prediction.
The first part of the article introduced the data processing, whereas the second and third parts focused on the AI model creation and validation, respectively. This fourth part, on the other hand, summarizes the main results and draws the conclusions.
Other drivers’ evaluations
Subjective ratings from two different highly skilled drivers were used (different from the reference driver selected for the model trained). With that, the noises and conditions of noises should be similar, but drivers’ evaluations are different. Dataset per rating used to evaluate other driver’s evaluations is shown in table 9.
Using different drivers for validation, we are validating at the same time:
- Model prediction accuracy
- Drivers’ subjective difference criteria
Ideally, model prediction accuracy should be similar to the accuracy result that comes from the validation performed on the model with the reference driver. Differences between accuracy of the model of the reference driver and the accuracy with the data set of other drivers, could be attributed to differences in subjective criteria between reference driver and the driver evaluated.
It can be seen that there are more subjective ratings available in the data set with high ratings than for low ratings.
Similar to the validation of the model for the reference driver, results for each driver are presented in terms of accuracy. Results can be checked in table 10 and accuracy per driver/rating in table 11.
Accuracy is calculated comparing the subjective rating prediction from the model with the actual ones of the drivers, meaning a 100% of accuracy a correct prediction (same as driver) of the model for all subjective ratings. In addition, the % of ratings not correctly assigned with a difference error of 1 rating, 2 rating and 3 rating is calculated.
It can be seen that:
- Both drivers have similar accuracy between them.
- Around 50% of prediction rating are same as the driver rating.
- Rating discrepancies between model and driver rating are mainly with a 1 rating error.
- Rating discrepancies between model and driver rating more than 2 points are around 6%.
It can be seen that:
- Accuracy for rating 9, rating 8 and rating 7 is over 50%.
- Accuracy for rating 6 decreases to 30%.
- Accuracy for rating 5, rating 4 and rating 3 is lower than 8%.
Summary results
Regarding reference driver validation, close to 70% of prediction ratings are the same as the reference driver rating. Rating discrepancies between model and driver rating are mainly with a 1 rating error. Rating discrepancies between model and driver rating more than 2 points are minimal. Accuracy for rating 9, rating 8 and rating 7 is around 70%. Accuracy for rating 6 or lower decrease to 50% or lower.
Regarding other drivers’ evaluations, the accuracy is around 50% for both of them. Same tendency in comparison with reference driver results can be shown. There is an increase of rating discrepancies mainly of 1 rating. The decrease of accuracy can be explained with the difference of subjective criteria of the drivers in comparison with the reference driver.
Conclusion
The goal of the project is to replicate the evaluation of brake noise annoyance performed by an expert driver using a model. Data containing noise samples collected during several years of testing at Applus IDIADA from a reference driver and their corresponding subjective ratings are provided for this purpose.
The data analysis revealed that there is a feasible opportunity to clean and preprocess the dataset by removing variables that do not contribute value to the model. Outliers were removed from the dataset. Data has been split in three parts: 70% noise events for training, 20% for test and 10% for validation.
Two artificial intelligence models were trained with the dataset: a classification and a regression model. According to the test phase results of training, it is shown that models achieve a good knowledge of the dataset. Finally, according to the different trials, the final model involves combining the classification and regression models. A threshold is set to determine when to rely on the classification model’s prediction and when to prioritize the rounded output from the regression model.
The model underwent validation by comparing its results with evaluations from the reference driver using different vehicles in conditions that were used for training. An accuracy of 68.5% was achieved, with rating discrepancies between model and driver rating mainly with a 1 rating.
In addition, predicted ratings from different drivers with model from the reference driver have been compared. It can be seen that accuracy in comparison with the reference driver decreased, but it can be explained as differences in subjective criteria with the other drivers.
The results of the study were promising, obtaining with the model an important level of accuracy in predicting subjective ratings based on objective measurements, indicating that the model’s predictions were close to the actual subjective ratings. Actually, it can be seen during models training that characterization of the subjective criteria is learnt by the models. Main rating discrepancies between model and driver rating are mainly with a 1 rating error that it could be explained as some uncertainty in the subjective criteria of the reference driver. This uncertainty in the subjective criteria of the driver could be explained by a variety of uncontrolled variables which can result in different subjective ratings for the same noise event. These differences appear mainly for low rating below rating 6. In addition, dataset contained a smaller number of lower rating 6 or below than above 6.
In conclusion, the development of an AI model for predicting subjective ratings based on objective measurements is an important step towards the understanding of subjective ratings and objective measurements for brake squeal noise. Prediction results from the current artificial intelligence model are based in objective measurements from 20 variables at the same time that characterize the most important features of the noise as frequency, amplitude, duration or corner source. Furthermore, the results of this study demonstrate the potential of AI models to be implemented in the near-to-medium future on autonomous vehicles providing more accurate subjective rating based on objective data. Future work in this area could involve expanding the model to include additional variables or incorporating other machine learning techniques to further improve performance.
About Applus IDIADA
With over 25 years’ experience and 2,450 engineers specializing in vehicle development, Applus IDIADA is a leading engineering company providing design, testing, engineering, and homologation services to the automotive industry worldwide.
Applus IDIADA is located in California and Michigan, with further presence in 25 other countries, mainly in Europe and Asia.
