EVALUATION AND QUALITY ASSURANCE OF MIGRATED ABAP CODE USING AN INTEGRAL METRIC AND GENERATIVE ARTIFICIAL INTELLIGENCE MODELS
DOI:
https://doi.org/10.15588/1607-3274-2026-1-8Keywords:
software quality, integral metrics, large language models, migration of legacy custom code, LLM fine-tuningAbstract
Context. Migration automation of legacy custom code when transitioning to the new version of the SAP S/4HANA system using large language models (LLMs) is a promising option. However, the generated code quality assessment remains an unresolved issue, since existing approaches utilize fragmented metrics which do not allow for a comprehensive software code quality assessment and assurance for further use without additional revision.
Objective. The objective of this work is to improve the efficiency of the process of intelligent reengineering of a computer system based on the method of comprehensive assessment and quality assurance of migrated ABAP custom code.
Method. The developed method is based on two key components. The Integral ABAP Quality Score (IAQS) comprehensively takes into account the syntactic, functional, and semantic characteristics of the code and is based on the provisions of the international software quality standards ISO/IEC 25010, ISO/IEC 25040, as well as the theory of composite indicators. The threestage approach to LLM fine-tuning (Qwen 2.5 Coder 14B) includes continuous pre-training (CPT), parameter-efficient fine-tuning (PEFT), and alignment based on preferences using the ORPO algorithm. At the same time, the use of the developed IAQS metric to form a set of preference data at the alignment stage creates a mechanism for controlled improvement, namely, it determines the direction of LLM adaptation.
Results. The results of experimental studies demonstrate that the implementation of the developed method allows improving both individual indicators of software code quality and the integral metric of IAQS quality assessment as a whole. The final model, trained on the basis of the proposed three-stage approach, achieved a high IAQS value (0.756), which demonstrates a significant improvement compared to the baseline model (0.117).
Conclusions. The study presents a new problem-oriented approach to automated migration of ABAP code during intelligent reengineering of computer systems. The proposed IAQS integral metric is the basis for creating a formalized and objective system for evaluating the quality of software generated by LLM in the context of legacy custom code migrating. It has been demonstrated that consistent fine-tuning of LLM based on a three-stage approach using IAQS provides a significant improvement in the generated software code integral quality indicator.
References
Hardy P. Migrating сustom сode to SAP S/4HANA. Boston, Rheinwerk Publishing Inc., 2020, 333 p.
Ktern AI. SAP S/4HANA 2022: The ultimate custom code migration guide [Electronic resource]. 2025. Access mode: https://ktern.com/article/sap-custom-code-migration-guide-2024/.
SAPinsider. Technical guide: using ABAP Test Cockpit for SAP S/4HANA Transition [Electronic resource], 2017. Access mode: https://sapinsider.org/articles/technical-guideusing-abap-test-cockpit-for-sap-s-4hana-transition/.
Pozdnyakov O. A., Parkhomenko A. V. Migration of custom code to new versions of complex computer systems using methods and models of intelligent reengineering, Scientific Works of DonNTU. Series “Informatics, Cybernetics and Computer Engineering”, 2025, Vol. 2 (41), pp. 86–98 (in Ukranian).
Pozdnyakov O. A., Parkhomenko A. V. Research and selection of large learning modes for automation of ABAPcode migration, Management of the development of complex systems, 2025, Vol. 63, pp. 191–200 (in Ukrainian).
ISO/IEC 25010:2023. Systems and software engineering – systems and software quality requirements and evaluation (SQuaRE). Product quality model. [Electronic resource]. Access mode: https://www.iso.org/ru/standard/78176.html.
ISO/IEC 25040:2024. Systems and software engineering – systems and software quality requirements and evaluation (SQuaRE). Evaluation process. [Electronic resource]. Access mode: https://www.iso.org/standard/83467.html.
Nardo M., Saisana M., Saltelli A., Tarantola S., Hoffman A., Giovannini E. Handbook on constructing composite indicators: Methodology and user guide. Paris, OECD Publishing, 2008, 162 p.
Yurchyshyn V. M. Methodological Approaches to Assessing Software Quality for Oil and Gas Industry Facilities, Methods and Instruments of Quality Control, 2020, Vol. 2 (45), pp. 40–57. DOI:10.31471/1993-9981-2020-2(45)-40-57 (in Ukranian).
Prokofiev I. Method of Static Analysis of Code Quality Using Machine Learning, Measuring and Computing Technology in Technological Processes, 2025, Vol. 3, pp. 126–133. DOI:10.31891/2219-9365-2025-83-17 (in Ukrainian).
Qwen Team. Qwen2.5-Coder series: Powerful, Diverse, Practical [Electronic resource]. 2024. Access mode: https://qwenlm.github.io/blog/qwen2.5-coder-family/.
Rozière B. , Gehring J., Gloeckle F. et al. Code Llama: Open foundation models for code, Meta AI Technical Report, 2023, 38 p. DOI:10.48550/arXiv.2308.12950.
Weyssow M., Zhou X., Kim K. et al. Exploring parameterefficient fine-tuning techniques for code generation with large language models, ACM Transactions on Software Engineering and Methodology, 2024, Vol. 33(6), pp. 1–25. DOI:10.1145/3714461.
Ray J. Teach an old dog new tricks: LLM continual pretraining (CPT) [Electronic resource], 2025. Access mode: https://medium.com/better-ml/teach-an-old-dog-new-tricksllm-continual-pre-training-cpt-684cfb931247.
Han Z., Gao C., Liu J. et al. Parameter-efficient fine-tuning for large models: a comprehensive survey, Transactions on Machine Learning Research, 2024, pp. 1–44. DOI:10.48550/arXiv.2403.14608.
Hu E. J., Shen Y., Wallis P. et al. LoRA: Low-rank adaptation of large language models, International Conference on Learning Representations (ICLR 2022), pp. 1–13. DOI:10.48550/arXiv.2106.09685.
Dettmers T., Pagnoni A., Holtzman A., Zettlemoyer L. QLoRA: Efficient finetuning of quantized LLMs, 37th Conference on Neural Information Processing Systems (NeurIPS 2023), 2023, pp. 1–28. DOI:10.48550/arXiv.2305.14314.
Hong J., Lee N., Thorne J. ORPO: Monolithic preference optimization without reference model, 2024 Conference onEmpirical Methods in Natural Language Processing, 2024, pp. 11170–11189. DOI:10.18653/v1/2024.emnlp-main.626.
Ren S., Guo D., Lu S. et al. CodeBLEU: a method for automatic evaluation of code synthesis, Computer Science. Software Engineering, 2020, pp. 1–8. DOI:10.48550/arXiv.2009.10297.
Microsoft. CodeXGLUE: CodeBLEU evaluation metric [Electronic resource]. 2025. Access mode: https://github.com/microsoft/CodeXGLUE/tree/main/CodeCode/code-to-code-trans/evaluator/CodeBLEU.
Bhattacharjee A., Dwyer C. Analyzing and mitigating surface bias in code evaluation metrics, Computer Science. Software Engineering, 2025, pp. 1–22. DOI:10.48550/arXiv.2509.15397.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 O. A. Pozdnyakov, A. V. Parkhomenko
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Creative Commons Licensing Notifications in the Copyright Notices
The journal allows the authors to hold the copyright without restrictions and to retain publishing rights without restrictions.
The journal allows readers to read, download, copy, distribute, print, search, or link to the full texts of its articles.
The journal allows to reuse and remixing of its content, in accordance with a Creative Commons license СС BY -SA.
Authors who publish with this journal agree to the following terms:
-
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License CC BY-SA that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
-
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
-
Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) as it can lead to productive exchanges, as well as earlier and greater citation of published work.
