Hey GPT, check yourself...
text
Hey GPT, check yourself...
new_line
new_line
Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥
text
Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥
new_line
new_line
💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.
text
💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.
new_line
new_line
This method is called SelfCheckGPT with Prompt and shows very nice results. 👀 
text
This method is called SelfCheckGPT with Prompt and shows very nice results. 👀 
new_line
new_line
The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃
text
The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃
new_line
new_line
More details and variations of this method are in the paper: 
text
More details and variations of this method are in the paper: 
SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning (2308.00436)
https://huggingface.co/papers/2308.00436
2308.00436
paper
resource
https://huggingface.co/papers/2308.00436
Hey GPT, check yourself...

Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥

💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.

This method is called SelfCheckGPT with Prompt and shows very nice results. 👀 

The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃

More details and variations of this method are in the paper: https://huggingface.co/papers/2308.00436
👍
aa08281 osanseviero clem samusenps victor pulkitmehtawork Gvangelista yjernite
❤️
clem samusenps