Digital Dialogues Under the Microscope: LLM Chatbots vs. Human Interaction
DOI:
https://doi.org/10.5281/zenodo.15478074Keywords:
Deberta v3, Machine learning, ChatbotsAbstract
In recent times, Large Language Model (LLM)-driven chatbots have emerged as a focal point in artificial intelligence research. These intelligent systems, leveraging state-of-the-art neural network architectures, represent a significant advancement in natural language processing capabilities. The construction of such chatbots commences with data exploration, where statistical summaries and distribution visualizations are employed to uncover hidden patterns within the dataset. Subsequently, the text undergoes an intensive preprocessing pipeline, including tokenization, stop word removal, and normalization, to ensure data quality for model training. A cutting-edge RoBERTa-based framework is then utilized to generate contextually relevant chatbot responses, followed by fine-tuning to enhance semantic coherence. To assess the authenticity of generated text, a gradient boosting classifier is implemented, trained on a diverse corpus of human and machine-generated utterances. The experimental evaluation reveals that the model attains an accuracy rate of 82%, a precision of 58%, a recall of 60%, and an F1 measure of 0.59. While the high accuracy reflects the model's proficiency in distinguishing between chatbot and human language to a certain extent, the relatively low precision and recall values highlight persistent challenges in accurately classifying text origin. This suggests that there remains room for improvement in refining the model's ability to produce outputs that closely mimic human language while maintaining clear differentiability.
References
Li, Keqin, et al. "Exploring the Impact of Quantum Computing on Machine Learning Performance." (2024).
Wang, Zixiang, et al. "Research on Autonomous Driving Decision-making Strategies based Deep Reinforcement Learning." arXiv preprint arXiv:2408.03084 (2024).
Yan, Hao, et al. "Research on Image Generation Optimization based Deep Learning." (2024).
Tang, Xirui, et al. "Research on Heterogeneous Computation Resource Allocation based on Data-driven Method." arXiv preprint arXiv:2408.05671 (2024).
Su, Pei-Chiang, et al. "A Mixed-Heuristic Quantum-Inspired Simplified Swarm Optimization Algorithm for scheduling of real-time tasks in the multiprocessor system." Applied Soft Computing 131 (2022): 109807.
Zhao, Yuwen, Baojun Hu, and Sizhe Wang. "Prediction of Brent crude oil price based on LSTM model under the background of low-carbon transition."arXiv preprint arXiv:2409.12376(2024).
Diao, Su, et al. "Ventilator pressure prediction using recurrent neural network." arXiv preprint arXiv:2410.06552 (2024).
Zhao, Qinghe, Yue Hao, and Xuechen Li. "Stock Price Prediction Based on Hybrid CNN-LSTM Model." (2024).
Yin, Ziqing, Baojun Hu, and Shuhan Chen. "Predicting Employee Turnover in the Financial Company: A Comparative Study of CatBoost and XGBoost Models." (2024).
Xu, Q., Wang, T., & Cai, X. (2024). Energy Market Price Forecasting and Financial Technology Risk Management Based on Generative AI. Preprints. https://doi.org/10.20944/preprints202410.2161.v1
Wu, X., Xiao, Y., & Liu, X. (2024). Multi-Class Classification of Breast Cancer Gene Expression Using PCA and XGBoost. Preprints. https://doi.org/10.20944/preprints202410.1775.v2
Wang, H., Zhang, G., Zhao, Y., Lai, F., Cui, W., Xue, J., Wang, Q., Zhang, H., & Lin, Y. (2024). RPF-ELD: Regional Prior Fusion Using Early and Late Distillation for Breast Cancer Recognition in Ultrasound Images. Preprints. https://doi.org/10.20944/preprints202411.1419.v1
Min, L., Yu, Q., Zhang, Y., Zhang, K., & Hu, Y. (2024, October). Financial Prediction Using DeepFM: Loan Repayment with Attention and Hybrid Loss. In 2024 5th International Conference on Machine Learning and Computer Application (ICMLCA) (pp. 440-443). IEEE.
Accurate Prediction of Temperature Indicators in Eastern China Using a Multi-Scale CNN-LSTM-Attention model
Rao, Jiarui, Qian Zhang, and Xinqiu Liu. "Applications Analyzing E-commerce Reviews with Large Language Models (LLMs): A Methodological Exploration and Application Insight." Journal of Artificial Intelligence General science (JAIGS) ISSN: 3006-4023 7.01 (2024): 207-212.
Zhang, Qian, et al. "Sea MNF vs. LDA: Unveiling the Power of Short Text Mining in Financial Markets." International Journal of Engineering and Management Research 14.5 (2024): 76-82.
Rao, Jiarui, et al. "Machine Learning in Action: Topic-Centric Sentiment Analysis and Its Applications." (2024).
Rao, Jiarui, et al. "Integrating Textual Analytics with Time Series Forecasting Models: Enhancing Predictive Accuracy in Global Energy and Commodity Markets." Innovations in Applied Engineering and Technology (2023): 1-7.
Zhang, Qian, and Jiarui Rao. "Enhancing Financial Forecasting Models with Textual Analysis: A Comparative Study of Decomposition Techniques and Sentiment-Driven Predictions." Innovations in Applied Engineering and Technology (2022): 1-6.
Rao, Jiarui. "Machine Learning in Action: Topic-Centric Sentiment Analysis and Its Applications." Available at SSRN (2024).
Rao, Jiarui, Qian Zhang, and Xinqiu Liu. "Applications Analyzing E-commerce Reviews with Large Language Models (LLMs): A Methodological Exploration and Application Insight." Journal of Artificial Intelligence General science (JAIGS) ISSN: 3006-4023 7.01 (2024): 207-212.
Li, Chao, Jiarui Rao, and Qian Zhang. "LLM-Enhanced XGBoost-Driven Fraud Detection and Classification Framework." (2025).
Rao, Jiarui, and Qian Zhang. "Deep Learning with LLM: A New Paradigm for Financial Market Prediction and Analysis." (2025).
Rao, Jiarui, and Zeyu Wang. "Optimizing Stock Market Return Forecasts with Uncertainty Sentiment: Leveraging LLM-based Insights." Available at SSRN 5117562 (2024).
Bo, Shi, and Minheng Xiao. "Time-Series K-means in Causal Inference and Mechanism Clustering for Financial Data." Proceedings of the 2024 7th International Conference on Computer Information Science and Artificial Intelligence. 2024.
Rao, Jiarui, and Qian Zhang. "Deconstructing Digital Discourse: A Deep Dive into Distinguishing LLM-Powered Chatbots from Human Language." Journal of Theory and Practice in Education and Innovation 2.2 (2025): 18-25.
Qian, Chenghao, et al. "WeatherDG: LLM-assisted procedural weather generation for domain-generalized semantic segmentation." arXiv preprint arXiv:2410.12075 (2024).
Qin, Gaoyuan, et al. "Application of Convolutional Neural Network in Multimodal Emotion Recognition." 2024 9th International Symposium on Computer and Information Processing Technology (ISCIPT). IEEE, 2024.
Wang, Randi, Vadim Shapiro, and Morad Mehandish. "Model consistency for mechanical design: Bridging lumped and distributed parameter models with a priori guarantees." Journal of Mechanical Design 146.5 (2024): 051710.
Wang, Randi, and Morad Behandish. "Surrogate modeling for physical systems with preserved properties and adjustable tradeoffs." arXiv preprint arXiv:2202.01139 (2022).
Wang, Randi, and Vadim Shapiro. "Topological semantics for lumped parameter systems modeling." Advanced Engineering Informatics 42 (2019): 100958.
Liu, Yanming, et al. "Bridging context gaps: Leveraging coreference resolution for long contextual understanding." arXiv preprint arXiv:2410.01671 (2024).
Privacy-Preserving Hybrid Ensemble Model for Network Anomaly Detection: Balancing Security and Data Protection
Dai, Y., Wang, Y., Xu, B., Wu, Y., & Xian, J. (2020). Research on image of enterprise after-sales service based on text sentiment analysis. International Journal of Computational Science and Engineering, 22(2-3), 346-354.
Cui, Wendi, et al. "Phaseevo: Towards unified in-context prompt optimization for large language models." arXiv preprint arXiv:2402.11347 (2024).
Cui, Wendi, et al. "Divide-Conquer-Reasoning for Consistency Evaluation and Automatic Improvement of Large Language Models." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2024.
Xiao, Minheng, Shi Bo, and Zhizhong Wu. "Multiple greedy quasi-newton methods for saddle point problems." 2024 6th International Conference on Data-driven Optimization of Complex Systems (DOCS). IEEE, 2024.
Zhang, Jiaxin, et al. "Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models." arXiv preprint arXiv:2410.09629 (2024).
Bo, Shi, and Minheng Xiao. "Data-Driven Risk Measurement by SV-GARCH-EVT Model." 2024 6th International Conference on Data-driven Optimization of Complex Systems (DOCS). IEEE, 2024.
Li, Zhuohang, et al. "Towards statistical factuality guarantee for large vision-language models." arXiv preprint arXiv:2502.20560 (2025).
Survival of the Safest: Towards Secure Prompt Optimization through Interleaved Multi-Objective Evolution
Bo, Shi, and Minheng Xiao. "Root cause attribution of delivery risks via causal discovery with reinforcement learning." Algorithms 17.11 (2024): 498.
SCE: Scalable Consistency Ensembles Make Blackbox Large Language Model Generation More Reliable
Wang, Yu, et al. "Gradient-guided Attention Map Editing: Towards Efficient Contextual Hallucination Mitigation." arXiv preprint arXiv:2503.08963 (2025).
Automatic Prompt Optimization via Heuristic Search: A Survey
Zhang, Shiwen. "On the importance of source text embedding in text-guided image editing." (2024).
Zhang, Shiwen. "Vector projection in text embedding space for text-guided image editing." (2024).
Zhang, Shiwen. "Hyper-parameter tuning for text guided image editing." arXiv preprint arXiv:2407.21703 (2024).
Zhang, Shiwen. "On the coefficient of model merging in forgetting mechanism." (2024).
Xiao, Minheng, and Shi Bo. "Electroencephalogram emotion recognition via auc maximization." Algorithms 17.11 (2024): 489.
Qian, Chenghao, et al. "WeatherDG: LLM-assisted procedural weather generation for domain-generalized semantic segmentation." IEEE Robotics and Automation Letters (2025).
Li, Wanxin. "User-Centered Design for Diversity: Human-Computer Interaction (HCI) Approaches to Serve Vulnerable Communities." Journal of Computer Technology and Applied Mathematics 1.3 (2024): 85-90.
Wang, Yang, Yojiro Mori, and Hiroshi Hasegawa. "Resource assignment based on core-state value evaluation to handle crosstalk and spectrum fragments in SDM elastic optical networks." 2020 Opto-Electronics and Communications Conference (OECC). IEEE, 2020.
Wang, Yang, et al. "Optimizing multi-criteria k-shortest paths in graph by a natural routing genotype-based genetic algorithm." 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA). IEEE, 2018.
Wang, Yang, Yojiro Mori, and Hiroshi Hasegawa. "Dynamic Routing and Spectrum Allocation Based on Actor-critic Learning for Multi-fiber Elastic Optical Networks." Photonics in Switching and Computing. Optica Publishing Group, 2021.
Zhu, Yuanjing, and Yunan Liu. "Innovative Prompting Strategies and Holistic Evaluation of LLM Movie Recommender." 2024 International Conference on Intelligent Computing and Next Generation Networks (ICNGN). IEEE, 2024.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Mark Ouyang
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
