Also, wanted to point out that I had this exact idea for test testing almost two years ago, at the beginning of June 2023, great to see someone testing it :)
Testing ChatGPT-4’s current capabilities to provide pesak
We can rigorously test ChatGPT-4’s current capabilities to provide pesak. In general, rigorous testing has been done extensively with other tests, including by the company behind ChatGPT-4 (Open AI). I wanted to do this myself. In my opinion, the best test for this would be the Semichah and Dayanut tests given by the official Israeli rabbinate. These are available online, going back many years, here and here.
I put in the first question from the most recent Issue Ve’Heter test (2023) [...]
-we definitely at some point will try to upload entire seforim to a project. I think right now claude's context window is currently limited to a few hundred pages of text, so we'd have to find a more focused/appropriate topic, ie go thru all of rav chaim's chidushim on the rambam, and list every type of chakirah, frequency used, etc. I think google's LLM has a pretty large context window so uploading a significant portion of texts is totally possible and I assume will be done be someone at some point
-prescient article :) We're only at the very beginning of the LLM and jewish text era so still so much work to be done!
Maybe try loading up some relevant seforim into its context window, via a "project"? Can download entire text files from sefaria
Also, wanted to point out that I had this exact idea for test testing almost two years ago, at the beginning of June 2023, great to see someone testing it :)
https://www.ezrabrand.com/p/the-intersection-of-halacha-and-artificial
I wrote there:
"
Testing ChatGPT-4’s current capabilities to provide pesak
We can rigorously test ChatGPT-4’s current capabilities to provide pesak. In general, rigorous testing has been done extensively with other tests, including by the company behind ChatGPT-4 (Open AI). I wanted to do this myself. In my opinion, the best test for this would be the Semichah and Dayanut tests given by the official Israeli rabbinate. These are available online, going back many years, here and here.
I put in the first question from the most recent Issue Ve’Heter test (2023) [...]
"
Hey, thanks for the comments!
-we definitely at some point will try to upload entire seforim to a project. I think right now claude's context window is currently limited to a few hundred pages of text, so we'd have to find a more focused/appropriate topic, ie go thru all of rav chaim's chidushim on the rambam, and list every type of chakirah, frequency used, etc. I think google's LLM has a pretty large context window so uploading a significant portion of texts is totally possible and I assume will be done be someone at some point
-prescient article :) We're only at the very beginning of the LLM and jewish text era so still so much work to be done!