Rav Dicta wants Semicha. But is it good enough?
New LLM just dropped so we're gonna do our thing. But also Gemini Deep Research does some magic.
A few days ago, a new specialized LLM, called Rav Dicta, was released. Here’s how the creators describe it:
Rav Dicta is a virtual rabbi, created with AI technology. He is designed to answer users' halachic questions based on classical rabbinic literature.
This is how Rav Dicta forms answers:
1. Comprehensive Source Review: In editing the response, Rav Dicta examines a wide range of sources, such as Tannaitic and Amoraic texts, Rishonim, Acharonim, and the most recent Acharonim. Depending on the subject of the question, Rav Dicta focuses on the appropriate literary genres, such as study, halacha, Chassidus, Jewish philosophy, and ethics. It is important to emphasize that every source presented is checked and verified as authentic.
2. Analyzing Relevance: Using an AI algorithm, the system identifies sources that explicitly discuss the user's question, applies relevant categories or rules, and presents in-depth discussions with multiple perspectives.
3. Summary and Conclusion: The system utilizes an AI algorithm to identify sources that explicitly address the user's question, applying relevant categories and rules. It then presents comprehensive discussions from multiple perspectives. Each cited source is thoroughly reviewed for authenticity, ensuring a well-founded response. Ultimately, the system synthesizes the insights and reaches a reasoned conclusion based on the most relevant and credible sources.
Through the use of AI technologies, Rav Dicta offers users a comprehensive and authoritative perspective on their questions, ensuring clarity and academic depth.
Bold claim! Let’s see how it does on the Rabbanut test we used earlier. It is a very difficult test if you did not study specifically for it.
Instructions:
Question:
Rav Dicta Answer, which can appear both in Hebrew and English:
Regarding the matter of "Davar She'yesh Lo Matirin" (something that will become permitted) and the law of a vessel that absorbed prohibited matter and became mixed with others:
It is stated in the Gemara Beitzah (3b) that anything that will become permitted does not become nullified even in a thousand-fold mixture. This requires extensive explanation. Indeed, our early and later sages, may their merit protect us, have elaborated on this, and we shall explain the matters with Heaven's help.
The Rambam, of blessed memory, in Hilchot Ma'achalot Asurot (15:9) explained the reasoning: since there is a way to permit it, the law of nullification does not apply at all. This is indeed logical, for why should we rely on nullification when there is a permissible way available?
The Taz (Yoreh De'ah 102:1) wrote that although biblically one-in-two is nullified, the rabbis were stringent in this matter, for since it will become permitted later, we do not allow it to be eaten through nullification. The Aruch HaShulchan (YD 102:1) similarly concluded that since it will be permissible to eat later legitimately, one should not eat it now through nullification.
However, all this applies only when it is mixed with its own kind, but when mixed with a different kind, the Shulchan Aruch (YD 102:1) ruled that it is nullified in sixty. This is also found in the Tur (ibid.) in the name of Rabbeinu Tam.
Regarding a vessel that absorbed prohibited matter and became mixed with others, the Beit Yosef (YD 102) cited the Rashba that although it's possible to kasher it through hagalah (boiling), we don't consider it a davar she'yesh lo matirin. He ruled accordingly in the Shulchan Aruch that it is nullified by majority, because kashering requires expense.
The Knesset HaGedolah (YD 102) added two additional reasons: First, we only apply the law of davar she'yesh lo matirin when the prohibition is inherent, not when it absorbed prohibition from elsewhere. Second, it's only considered davar she'yesh lo matirin when the permission comes automatically.
In practice, we hold that a vessel that absorbed prohibited matter and became mixed with others is nullified by majority, and it is permitted to use all the vessels.
This is what appears correct to my limited understanding, and if I have erred, may God forgive.
Humbly, Artificial Intelligence, Residing here in the holy city of Jerusalem, may it be rebuilt
We have a relatively short answer with a list of provided sources (only in Hebrew though for some reason). Let’s see how it does.
Question 1: 2/10
No mention of Rashi or Ran by name, which is the main machloket and the correct answer to the question.
Rav Dicta (RD from now on) quotes a Rambam (Maachalot Asurot 15:9) instead, which is about chametz, which although the Rambam holds it is a davar she yesh lo matirin, that is subject to a machloket as the Mordechai does not think it is a davar she yesh lo matirin.
RD then quotes/translates the (Taz 102:1) with a svara, which in actuality, quotes rashi in the text, but that is left out as being rashi’s svara by RD. So close!
Question 2: 3/10
Partially correct. Correct citation of the Shulchan aruch and the Tur in the name of Rabbeinu Tam, but doesn’t quote the two reasons of the Taz for explaining this din. 3/10
Question 3: 3/10
Correct citation of the Shulchan Aruch/Beit Yosef, who quotes the rashbah, essentially giving a translation of the Shulchan Aruch, that it is batel.
Doesn’t quote the Taz who provides a few reasons for why it’s batel, nor the Shach, who quotes a few reasons to argue on the Rashbah. Instead RD quotes the Knesset HaGedolah, which is on topic, although not a primary source on the topic.
Correctly quotes how we paskin that it is batel, but doesn’t quote any sources and reason is imprecise.
Summary:
How’d Rav Dicta do? Not bad. While other LLMs like Claude and ChatGPT hallucinated more on parts of this question and made up stuff, RD was able to stay on topic, not hallucinate and give cogent answers. Were those answers that good? Not really. Were they incomplete? Very much so. But at least they weren’t made up and for the era of LLMs that’s pretty good! In other words, it has a high specificity and somewhat middling sensitivity for finding sources, which is something new in the world of LLMs for Jewish learning.
But not enough for Israeli Rabbanut Semicha. Time to join Claude in Yeshiva, Reb Dikta.
What’s going on with RD?
Let’s see RD’s thought process. RD’s style is kind of fun; it sort of mimics and maintains the form of a classic teshuva, with the characteristic beginning, end, humble flourishes regarding possible mistakes, give and take reasoning etc.
In terms of content, it correctly starts at the gemara and gives a good summary, quotes a Rambam, which is then contrasted with the Taz and Aruch Hashulchan. That’s a good methodology for someone trying to answer this question. There’s clearly much thought given how to structure an answer and which sources to quote.
It goes without saying, that learning how to answer halakhic questions is much harder than that. It’s really hard for an experienced learner to identify the most critical sources in each sugya, and even harder to program an algorithm to actually find, organize, and answer using the most correct sources.
Going back to the creator’s about page, let’s grade RD.
Comprehensive source review: Not comprehensive, but not entirely wrong either.
Analyzing Relevance: Yes!
Summary and Conclusion: Not really but c’mon that’s really hard and subjective, but it didn’t do a terrible job.
How to use Rav Dicta?
Find sources: The sources it quotes are almost likely correct, albeit possibly incomplete. But it’s a good place to start.
Provide a first pass analysis: It’s barebones but can get you started on a topic.
Easy questions with a quick/basic answer: I imagine this model excels at this, but we’ll have to test this out
Learning how responsa work: Educational tool for students to learn how shutim work. Try to write your own and see what RD says
Have fun: playing with RD is a ton on fun! It’s like watching a precocious 11th grader copy and paste from the internet for his homework. But 9th graders grow up real fast, especially when they’re silicon-based.
Many people will immediately type in a question to RD, like “I’m a doctor in a hospital with a case of possible brain death on shabbat yom kippur, and the patient is an aguna- what’s the psak?” and expect to get a phenomenal, gadol-level answer. That is the wrong way to view this model. This model is a fun, proof of concept that an LLM can do some pretty sophisticated, specialized tasks in the world of Jewish text learning.
Overall, I’m impressed by the first gen model. The sources quoted are largely correct without anything blatantly wrong. Like the original release of ChatGPT to the public, it’s smart but has some mistakes/omissions. This is the first of presumably many updates and likely the first of many other models of incredible abilities.
On the other hand, the billion dollar models such as Claude, Gemini, and ChatGPT are releasing updates to models quicker than post Yom Kippur Maariv, including advanced research modes. For example, I just put the same exact prompt into Google Gemini’s Deep Research and it gave an incredible answer. On first glance, it quotes more sources than RD, and gives a more nuanced approach. It’s truly incredible. It might be the first model to get semicha. We’ll have to investigate further.
What does this mean? Large billion dollar companies building generalized models might beat out more specialized, smaller models, even those specifically trained on Jewish texts, which raises questions as to whether research groups should try to build their own models vs just use the big ones.
But I don’t know. I’m just trying to figure out the meaning of davar she’yesh lo matirin.
Great review. I've been playing with this new tool (Rav dicta) myself, great to see a review of it