How to Evaluate an AI Model

This article provides both visual and written instructions for evaluating the performance of an AI model in scoring document content.


Video Tutorial

HubSpot Video

AI models are used by Reveal to score documents based upon decisions by reviewers in coding Prediction Enabled tags (referred to as Supervised Learning) or by reference to published AI Model Library classifiers added to a project. Reveal can use these models to score documents automatically to assign documents and to compare, report and update its calculations based upon Supervised Learning (a process called Active Learning).

This article will address the process of evaluating an AI Model.

  1. With a project open, select Supervised Learning
    __Supervised Learning Button-1
  2. The Classifiers screen will open, displaying a card for each Classifier in the current project. Users have the option to switch the display into Dark Mode, shown here. 31 - 01 - Classifiers Screen in Dark Mode (no button)-1
  3. To examine the status of a current model, click on View Details on the Classifier card below its title. 
    31 - 02 - Classifier card - view details
  4.  The details window for the selected classifier will open. 31 - 03 - Classifier details - Tagging Scoring-1
  5. The above image shows a graph of Tagging & Scoring results. This graph shows actual tagging for this classifier, broken out as document counts of Positive and Negative reviewer assessments of Responsiveness as compared with the related AI Model's assessment of likely responsiveness. As of 14 rounds in which 138 documents have been tagged so far, 94 Positive and 44 Negative. In this way, a project manager can see at a glance how user coding compares with the model's prediction. The documents tagged Negative while scored 100 (assessed positively responsive) will clearly need a look in our demo project! NOTE that negative tagging is extremely important in training AI models, in that these help define document language that fails to address the subject matter of this tag, which is Responsiveness.
  6. The details to the right of the graph provide further information, including how many documents have not yet been tagged and the number of documents whose scoring according to current training is Uncertain.
  7. Further training will help to stabilize the classifier and reduce the number of Uncertain documents. Reveal will include Uncertain documents in batching to gain a better understanding of the factors involved in scoring for this classifier.
  8. At the bottom of the details is a link to Download Score History, which will export the numbers as a table to CSV, as in the example below. 31 - 04 - Download Score History
  9. The lower graph in the Classifier Details screen show the Control Set activity for the Current Round and for prior rounds. 31 - 05a - Classifier Details - Control Set (no Stability)-1
    1. Details for the graph, including Documents to Review, are again presented at the right.
    2. The graph may be adjusted for different document counts using the slider indicated by the arrow in the above illustration.
    3. At the bottom of the details is a link to Download Control Set History, which will export the numbers as a table to CSV, as in the example below. 31 - 06 - Classifier Details - Download Control Set History

To exit the Classifier Details screen, go to the top and click Classifiers in the breadcrumb path.

 

Last Updated 9/26/2022