Using Logprobs to Gauge GPT’s Confidence in OCR Result

In my recent blog post on parsing complex PDFs using GPT, I explored the potential of incorporating confidence scores to quantify GPT’s accuracy in the results. By integrating confidence scores, we can establish a scalable and efficient document processing and data extraction workflow that requires minimal oversight and review, thereby optimizing automation and reliability. For this purpose, we will be using the Log probabilities or logprobs.

Language models, such as GPT, rely heavily on log probabilities to efficiently manage and compute the vast array of token probabilities inherent in natural language processing.

In generative NLP tasks, log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is log(p), where p = probability of a token occurring at a specific position based on the previous tokens in the context.

Some key points about logprobs, taken from OpenAI Cookbook:

• Higher log probabilities suggest a higher likelihood of the token in that context. This allows users to gauge the model’s confidence in its output or explore alternative responses the model considered.
• Range of Values, logprobs can be any negative number or 0.0. 0.0 corresponds to 100% probability.