AllergoAssist 2.0: a Specialized Artificial Intelligence for Pediatric Allergology

Introduction: Integrating artificial intelligence (AI) in medicine offers significant potential to enhance healthcare. AllergoAssist 2.0, based on ChatGPT-4, supports pediatricians in managing allergic diseases in children. This study explores its development, implementation, and performance.

Materials and Methods: AllergoAssist 2.0 was created by training on specialized data, including guidelines from European Academy of Allergy and Clinical Immunology (EAACI), Global Initiative for Asthma (GINA), and Allergic Rhinitis and its Impact on Asthma (ARIA), as well as pediatric allergology literature. Using OpenAI’s “custom GPT” feature, the model incorporated specific clinical data. Performance was evaluated with multiple-choice questions from pediatric allergy and immunology case collections.

Results: AllergoAssist 2.0 showed superior accuracy compared to ChatGPT-4, achieving 83% correct answers in pediatric allergy and 84.9% in pediatric immunology, significantly higher than ChatGPT-4 65.72% and 70.44% (p < 0.001). This demonstrates its superior capability to provide accurate, evidence-based recommendations.

Discussion: The specialized training of AllergoAssist 2.0 allows effective handling of clinical terminologies and protocols, enhancing its utility. While the model improves the quality of care and operational efficiency, it relies on the quality of the input data and thus highlights the challenges of AI decision-making. Future improvements aim to enhance transparency and provide continuous data updates.

Conclusion: AllergoAssist 2.0 exemplifies the potential of customized AI in specialized medical fields, suggesting accurate, evidence-based support to pediatricians. This advancement is a significant step towards personalized and efficient healthcare, highlighting the transformative impact of AI in medicine

INTRODUCTION

Innovation in artificial intelligence (AI) has transformed various sectors, including medicine, where it offers unprecedented potential to improve healthcare ¹. In particular, AI has found a significant niche in specialized medicine, as demonstrated by the case of AllergoAssist 2.0. This AI model is a specialized version of ChatGPT-4 (OpenAI, California, 2023), designed to support pediatricians in managing allergic diseases in children. The growing prevalence of pediatric allergies globally necessitates more accurate and accessible diagnostic and therapeutic tools to facilitate rapid, evidence-based clinical decisions ².

AllergoAssist 2.0 (https://chatgpt.com/g/g-FRlrfONrZ-allergoassist-2-0) stands out for its ability to process and synthesize updated guidelines from major health authorities and research institutions. These standards are essential to ensure a consistent and up-to-date approach to allergy treatment. The specific training with clinical data and guidelines enables AllergoAssist 2.0 to suggest personalized and detailed advice, such as selecting specific allergens for IgE testing, interpreting results, and planning treatment, including dosing and timing of allergen immunotherapy.

Thus, the model could be a valuable resource for specialists, improving the quality of care and operational efficiency of medical practices. This introduction to AllergoAssist 2.0 highlights the potential of highly specialized AI to address specific challenges in well-defined medical fields, marking an important step toward increasingly personalized and evidence-based medicine.

MATERIALS AND METHODS

Development and Architecture of AllergoAssist 2.0

AllergoAssist 2.0 is built on the architecture of ChatGPT-4, an advanced generative language model developed by OpenAI. However, unlike the generic version, AllergoAssist 2.0 has been adapted to specifically meet the needs of pediatricians.

Training and Data

The creation of AllergoAssist 2.0 is distinguished by its focus on highly specialized data. The dataset includes:

• Guidelines: The model has been trained on various documents, including the latest updates from the European Academy of Allergy and Clinical Immunology (EAACI) Molecular Allergology User’s Guide 2.0 ³, Global Initiative for Asthma (GINA) Strategy for Asthma Management and Prevention report for asthma ⁴, and Allergic Rhinitis and its Impact on Asthma (ARIA) ⁵ Italian document, which provide detailed recommendations for the treatment of allergies and asthma;
• Medical Literature: Articles covering a wide range of pediatric allergology and immunology topics have been provided for the model;
• Reference Values: We have loaded reference values with age ranges for pediatric allergology and immunology’s most critical blood tests. The reference data come from the Mayo Clinic’s database⁶.

Training Methodology

To create AllergoAssist 2.0, we leveraged the creative potential of “custom GPTs” ⁷. OpenAI offers this exclusive feature to plus subscribers, allowing of ChatGPT versions to be customized to individual users’ requests and needs. Thanks to the of the integrated GPT Build capabilities, we can train our custom GPT on data of interest, updating ChatGPT-4’s information on specific topics. This also allows the average user to relatively easily create RAG (Retrieval Augmented Generation) structures⁸ without needing the deep computer science knowledge required to program such algorithms. Figure 1 illustrates a simplified functioning of the model. To our knowledge, AllergoAssist 2.0 is the first example of a custom Italian medical GPT in the allergology and immunology field. Starting from May 13th, 2024, OpenAI made custom GPTs available to users who use the free version of the ChatGPT service ⁹.

Implementation and Integration

Once trained and validated, AllergoAssist 2.0 can be integrated into the clinical activities of pediatrician allergists/immunologists and general pediatricians. This integration allows specialists to query the model in real time to obtain evidence-based advice during clinical visits, improving treatment effectiveness and clinical time efficiency. The advantage of this tool lies in its usability on any internet-connected device without the need for significant expenses in updating IT infrastructures.

In conclusion, the rigor in data selection and the specificity of the training make AllergoAssist 2.0 an advanced example of how AI can be adapted to address specific challenges in highly specialized medical fields.

RESULTS

To test our model’s capabilities in resolving clinical cases, we tested it with multiple-choice questions based on clinical cases from two texts: “Pediatric Allergy A Case-Based Collection with MCQs” ¹⁰ and “Pediatric Immunology A Case-Based Collection with MCQs” ¹¹, which cover pediatric allergology and pediatric immunology, respectively. We submitted 219 pediatric allergy questions and 309 pediatric immunology questions, achieving correct answers in 83% and 84.9% of cases, respectively. We compared the custom GPT’s results using the same test set with ChatGPT-4, or the plus version, obtaining 65.72% correct answers in pediatric allergy and 70.44% in pediatric immunology (Tab. I).

We calculated the p-value (setting it < 0.01) for both pediatric allergology and pediatric immunology by comparing the rate of correct answers between ChatGPT-4 and AllergoAssist to demonstrate that the responses from our model were not given randomly but relied on the knowledge we provided.

Below are examples of the tool’s functionality in clinical cases:

1. Spirometry: in Figure 2, we request the interpretation of a spirometry from Medscape ¹²;
2. Pork-cat syndrome: in Figure 3, we describe a clinical case of possible Pork-Cat syndrome, asking the model which specific IgE to look for in the case;
3. Asthma: in Figure 4, we present a clinical case of asthma, asking which therapy to recommend.

DISCUSSION

Unlike ChatGPT-4, a general-purpose generative AI platform, AllergoAssist 2.0 is specialized and optimized for specific applications in pediatric allergology. This specialization allows the model to understand and process specific clinical terminologies and protocols, offering a valuable resource for specialists. One of the most crucial aspects of training AllergoAssist 2.0 has been incorporating updated guidelines, e.g., from EAACI, GINA, and ARIA. The accuracy of the information provided by the system reflects the ongoing commitment to keep the model up to date.

These results are comparable with other AI models in the medical field. For instance, PubMedGPT 2.7B, a model specific to the biomedical sector, has demonstrated high accuracy in question-answering tasks in the medical context, achieving 50.3% accuracy on MedQA, a benchmark derived from the United States Medical Licensing Examination ¹³.

AllergoAssist 2.0 exemplifies how AI can be specialized to address specific needs in the medical field. Unlike ChatGPT-4, designed for general applications, AllergoAssist 2.0 is carefully calibrated for pediatric allergology and immunology. This level of customization allows the model to suggest solutions to complex issues, such as interpreting individual variations in allergic responses and managing therapies, which are particularly critical in the pediatric population.

Despite its significant benefits, AllergoAssist 2.0 also presents limitations. The reliance on input data means that incomplete or inaccurate clinical data can compromise the quality of the model’s recommendations. Additionally, while the model effectively handles many clinical scenarios, extremely rare situations or new allergy manifestations may escape its analytical capabilities, requiring human intervention for evaluation. One aspect that warrants further discussion is the role of a general pediatrician in utilizing a tool like AllergoAssist 2.0. The question arises whether a pediatrician, without specific training in pediatric allergology and immunology, would be able of supervising and double-checking the AI-based suggestions in all cases.

It is plausible that a general pediatrician could manage common cases with the assistance of AllergoAssist 2.0, given the model’s ability to interpret and apply updated guidelines and standardized protocols. However, in more complex or atypical situations, where deeper knowledge and a more nuanced understanding of allergological issues are required, the expertise of a specialist remains irreplaceable. In such cases, the oversight of a specialist in pediatric allergology and immunology would be crucial to ensure the accuracy of therapeutic decisions.

Therefore, while AllergoAssist 2.0 represents a significant advancement in the use of AI in medicine, its safe and effective implementation necessitates careful consideration of the level of competence of the medical personnel using it. A general pediatrician may benefit from using AllergoAssist 2.0 as a decision-support tool, but clear guidelines should be established on when and how to consult a specialist for cases that exceed the general practitioner’s expertise.

Another significant challenge in using AI like AllergoAssist 2.0 is the “black box” nature of many models¹⁴. This term refers to the difficulty in understanding and tracing how decisions are made within the model. Indeed, although the model can provide accurate recommendations, the internal decision-making process often remains opaque, creating potential trust and transparency issues for users. This aspect is particularly critical in the medical field, where understanding the basis for a recommendation is essential for evaluation and acceptance by healthcare professionals ¹⁵. Improving the transparency and explainability of AI models remains a key goal for future research and development in AI applied to medicine.

As the model is based on the ChatGPT-4 engine, error rates may be independent of the data used for training and depend on the status of OpenAI’s servers. For improved usability, we recommend visiting the OpenAI status website to check the availability of their services ¹⁶.

CONCLUSION

The implementation of AllergoAssist 2.0 represents a significant advancement in the use of AI in medicine. It demonstrates how generative language models, such as ChatGPT-4, can be adapted to meet specific and complex needs. With its ability to analyze and synthesize detailed clinical information, AllergoAssist 2.0 offers valuable support to pediatricians, thus improving the quality of care and operational efficiency.

The potential of customized ChatGPT-4 models is vast, allowing for of personalization beyond the capabilities of general-purpose AI platforms. This approach enables the creation of highly specialized tools, like AllergoAssist 2.0, which improve the accuracy of diagnoses and treatments and facilitate access to evidence-based care in various and often complex clinical settings.

Despite the significant potential, it is essential to consider the intrinsic limitations related to the quality and completeness of input data the need for continuous human supervision to manage rare or unforeseen clinical scenarios. In conclusion, integrating AllergoAssist 2.0 into clinical practice should be accompanied by a supervisory framework that considers the complexity of the clinical case and the level of specialization required for an accurate interpretation of the model’s recommendations. This approach ensures the optimal and safe use of AI in treating pediatric allergies. Moreover, AllergoAssist 2.0 marks a critical step toward more personalized and evidence-based medicine, exemplifying how AI can be strategically utilized to address specific challenges in highly specialized medical fields. By combining rigorous oversight with advanced AI tools, we can enhance the precision and effectiveness of medical interventions, particularly in complex areas like pediatric allergology.

Acknowledgements

None.

Conflicts of interest statement

The authors certify that they have NO affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

Ethical considerations

Not applicable.

Funding

This research no external funding.

Authors’ contribution

GM: contributes with: conceptualization, methodology, supervision, writing - review & editing, and creation of the project. SC, VF, AP: contributes with: methodology, investigation, data curation, writing - original draft. CI, MMG: contributes with: validation and data curation.

History

Received: June 20, 2024

Published: October 7, 2024

Figures and tables

FIGURE 1. How the custom GPT works.

FIGURE 2. A clinical case of a potential Pork-Cat syndrome.

FIGURE 3. A clinical case of a potential Pork-Cat syndrome.

FIGURE 4. What should be done for this asthmatic patient?

TABLE I. p¹ derives from the comparison between number of correct answers ChatGPT4 and AllergoAssist 2.0 in the field of Pediatric Allergology; p² derives from the comparison between number of correct answers ChatGPT4 and AllergoAssist 2.0 in the field of Pediatric Immunology.

	Pediatric Allergology	Pediatric Immunology
ChatGPT 4	144/219 (65.72%)	217/309 (70.44%)
AllergoAssist2.0	182/219 (83%)	262/309 (84.9%)
p-value	p1 < 0.001	p2 < 0.001