MedPro: A Hybrid Recommendation System with Advanced OCR for Medical Applications

Abstract

In a clinical setting, appropriate medication management is vital for minimizing adverse drug events and optimizing patient outcomes. This paper describes MedPro, a new hybrid recommendation system that fuses an advanced optical character recognition (OCR) technique with a recommendation engine. A unique feature of MedPro is that a custom convolutional recurrent neural network (CRNN)-based OCR model has been combined with the open-source EasyOCR engine as the main text extractor, supported by Gemini API from Google for resolving ambiguous cases. The processing stage involves images of handwritten prescriptions and straight text inputs, extraction of the drug names, and querying within a curated dataset to offer alternatives for allopathic and homoeopathic brands with a wide array of associated side effects. Experimental validation of our method shows impressive improvements in terms of text recognition accuracy achieved by our hybrid OCR pipeline, while the recommendation engine yields better precision and recall compared to traditional methods. The full-fledged web interface based on Flask allows end-user interaction making it an appropriate and useful tool for clinical decision support. The present paper advocates MedPro, a unique hybrid recommendation system that incorporates advanced optical character recognition (OCR) techniques with a strong recommendation engine. MedPro essentially combines a custom CRNN-based OCR model with the open-source EasyOCR engine as the primary text extractor, while Google's Gemini API resolves intractable ambiguities if they arise. The system processes handwritten prescription images as well as text written directly. The names of the medicines are extracted and a query on the curated medicine dataset is run to suggest alternative drugs with side effects therein. These experimental evaluations show that our hybrid OCR pipeline leads to text recognition accuracy that is much improved, and the outcomes from the recommendation engine do outperform traditional methods in regard to precision and recall. A Flask-based web interface offers seamless user communication, making MedPro an adaptable and practical clinical decision-making tool.