A hybrid multi-corpus RAG framework for faithful and source-grounded MUI fatwa question answering

Abstract

The increasing use of large language models (LLMs) for Islamic legal information retrieval presents substantial challenges related to hallucinations, source relevance, and the limited coverage of single-corpus retrieval-augmented generation (RAG) systems. This study proposes a sequential multi-corpus RAG architecture for a hybrid MUI fatwa chatbot that prioritises authoritative fatwa documents while providing controlled fallback retrieval from a curated hadith corpus. The system employs dense semantic retrieval using FAISS for the primary MUI fatwa corpus, followed by similarity threshold filtering and LLM-based relevance verification. When no sufficiently relevant fatwa context is identified, the query is redirected to sparse retrieval of over 12,370 hadith records from Sahih al-Bukhari and Sahih Muslim. The proposed system was compared with a conventional single-corpus RAG configuration using the BERTScore and Giskard’s RAG evaluation toolkit (RAGET). BERTScore evaluation on ten questions showed that the proposed architecture increased the average F1-score from 0.6664 to 0.7546, representing a relative improvement of 13.23%. On 41 successfully paired evaluation records, the Giskard faithfulness score increased from 0.8537 to 0.9439, equivalent to a 10.57% relative improvement. However, both systems produced an identical abstain rate of 26.83%, indicating that the proposed architecture primarily improved answer relevance and contextual grounding rather than overall question coverage. These findings demonstrate that hierarchically organising knowledge sources according to their authority and function can enhance the reliability of RAG-based Islamic legal information systems. Nevertheless, the chatbot should be regarded as a source-grounded information-access tool and not as a substitute for fatwas or qualified Islamic scholars.