Comparative Analysis of Transformer Models for Emotion Classification in Indonesian Text Data
DOI:
https://doi.org/10.30741/jid.v5i1.2082Keywords:
Emotion Classification, Transformer, BERT, RoBERTa, DistilBERTAbstract
This study presents a comparative analysis of three transformer-based models, BERT, RoBERTa, and DistilBERT, for emotion classification on Indonesian text. The Indo4B dataset, which consists of five emotion labels (anger, fear, happiness, love, and sadness), is used as the benchmark corpus. All three models were trained using fine-tuning and hyperparameter-tuning procedures. The evaluation was conducted using accuracy, F1-score, model size, and training time as the primary performance metrics. The results indicate that RoBERTa achieves the best overall performance, with an accuracy of 90.83% and an F1-score of 91. Meanwhile, DistilBERT demonstrates approximately 60% faster training time compared to BERT, with only a 0.61% decrease in accuracy. These findings suggest that DistilBERT offers a highly efficient alternative while maintaining competitive classification performance.
References
Acheampong, F. A., Nunoo-Mensah, H., & Chen, W. (2021). Transformer Models for Text-based Emotion Detection: A Review of BERT-based Approaches. Springer. https://doi.org/https://doi.org/10.1007/s10462-021-09958-2
Aman, S., & Szpakowicz, S. (2007). Identifying Expressions of Emotion in Text. LNAI, 4629, 196–205. https://doi.org/10.1007/978-3-540-74628-7_27
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv. https://doi.org/http://arxiv.org/abs/1810.04805
Dong, M. (2018). Emotion Classification on Indonesian Twitter Dataset. IEEE. https://doi.org/10.1109/IALP.2018.8629262
Hastomo, W., Bayangkari Karno, A. S., Kalbuana, N., Meiriki, A., & Sutarno. (2021). Characteristic Parameters of Epoch Deep Learning to Predict Covid-19 Data in Indonesia. Journal of Physics: Conference Series, 1933(1). https://doi.org/10.1088/1742-6596/1933/1/012050
He, F., Liu, T., & Tao, D. (2019). Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence.
Hidayatullah, A. F., & Ma’Arif, M. R. (2017). Pre-processing Tasks in Indonesian Twitter Messages. Journal of Physics: Conference Series, 801(1). https://doi.org/10.1088/1742-6596/801/1/012072
Huan, H., Guo, Z., Cai, T., & He, Z. (2022). A text classification method based on a convolutional and bidirectional long short-term memory model. Connection Science, 34(1), 2108–2124. https://doi.org/10.1080/09540091.2022.2098926
Huang, J., Li, Y.-F., Xie, M., Huang, J., Xie, M., & Li, Y. F. (2015). A Systematic Analysis of Data Preprocessing for Machine Learning- based Software Cost Estimation. HAL, 67. https://doi.org/10.1016/j.infsof.2015.07.004ï
Kaur, H., Pannu, H. S., & Malhi, A. K. (2019). A systematic review on imbalanced data challenges in machine learning: Applications and solutions. ACM Computing Surveys, 52(4). https://doi.org/10.1145/3343440
Lee, K., Palsetia, D., Narayanan, R., Patwary, M. M. A., Agrawal, A., & Choudhary, A. (2011). Twitter trending topic classification. Proceedings - IEEE International Conference on Data Mining, ICDM, 251–258. https://doi.org/10.1109/ICDMW.2011.171
Li, W., & Xu, H. (2014). Text-based emotion classification using emotion cause extraction. Expert Systems with Applications, 41(4 PART 2), 1742–1749. https://doi.org/10.1016/j.eswa.2013.08.073
Lin, C. H., & Nuha, U. (2023). Sentiment analysis of Indonesian datasets based on a hybrid deep-learning strategy. Journal of Big Data, 10(1). https://doi.org/10.1186/s40537-023-00782-9
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv. https://doi.org/https://doi.org/10.48550/arXiv.1907.11692
Markoulidakis, I., Rallis, I., Georgoulas, I., Kopsiaftis, G., Doulamis, A., & Doulamis, N. (2021). Multiclass Confusion Matrix Reduction Method and Its Application on Net Promoter Score Classification Problem. Technologies, 9(4), 81. https://doi.org/10.3390/technologies9040081
Medhat, W., Hassan, A., & Korashy, H. (2014). Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal, 5(4), 1093–1113. https://doi.org/10.1016/j.asej.2014.04.011
Nissa, N. K., & Yulianti, E. (2023). Multi-label text classification of Indonesian customer reviews using bidirectional encoder representations from transformers language model. International Journal of Electrical and Computer Engineering, 13(5), 5641–5652. https://doi.org/10.11591/ijece.v13i5.pp5641-5652
Rahmad, F., Suryanto, Y., & Ramli, K. (2020). Performance Comparison of Anti-Spam Technology Using Confusion Matrix Classification. IOP Conference Series: Materials Science and Engineering, 879(1). https://doi.org/10.1088/1757-899X/879/1/012076
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2020). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv. https://doi.org/https://doi.org/10.48550/arXiv.1910.01108
Shaver, P. (1987). Emotion knowledge: Further exploration of a prototype approach. Journal of Personality and Social Psychology. https://doi.org/https://doi.org/10.1037/0022-3514.52.6.1061
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to Fine-Tune BERT for Text Classification? Springer. https://doi.org/10.1007/978-3-030-32381-3_16
Wiciaputra, Y. K., Young, J. C., & Rusli, A. (2021). Bilingual text classification in english and indonesian via transfer learning using XLM-RoBERTa. International Journal of Advances in Soft Computing and Its Applications, 13(3), 72–87. https://doi.org/10.15849/ijasca.211128.06
Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., & Purwarianti, A. (2020). IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding. ArXiv. https://doi.org/https://doi.org/10.48550/arXiv.2009.05387
Xu, J., Zhang, Y., & Miao, D. (2020). Three-way confusion matrix for classification: A measure driven view. Information Sciences, 507, 772–794. https://doi.org/10.1016/j.ins.2019.06.064
Yanuar, M., & Shiramatsu, S. (2020). Aspect Ectractionfor Tourist Spot Review in Indonesian Language Using BERT.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
