Rapid Pilot Project Creation

Document Purpose

The purpose of this document is to provide the instructions to create the "Rapid Pilot Project". This project consists of an Enron subset, and a handful of documents to showcase our Review Accelerators (Translation, Image Labeling, and Transcription) totaling ~40K documents. This allows us to have a project for sandbox or training purposes.

Technical Documentation

To access all technical documentation within the Reveal Discovery Platform, please go to our Online Help Portal.

High Level Process Overview

Below provides the high-level illustration of this entire process.

Access to Load Machine

All Load Machine access is administered through Perimeter 81. To access a Load Machine, you can follow the high-level steps below.

1. https://landing.revealdata.com
2. Use your Reveal Universal ID to login
3. Find/click on the icon for your Load Machine by searching for the MSA.
4. If you cannot find or access your Load Machine, please email support@revealdata.com and ask for access.

Stage Test Data

There are two steps to stage the test data.

1. Download the test data to the Load Machine.
2. Copy/Unzip the data to the root of the Z:\ drive on the Load Machine.

Download Test Data

Follow the steps below to download the Test Data.

1. On the Pilot’s Load Machine, use the following link in Chrome to download the Test Data,
   
   1. https://revealdata-public-test-data.s3.us-east-1.amazonaws.com/_RapidPilot.zip

Copy/Unzip the Data

1. After downloading the zip file, copy it to the Z:\ drive of the Load Machine.
2. Unzip this file to the root of the Z:\ drive so it looks like the below.
   1. 
   2. 

Project Creation

Reveal Review Manager is used to administer your instance of Reveal Review. In Review Manager, you can do any number of functions associated with the administration of Review such as create databases, manage users, load large amounts of data, set analytics and more. To see more information about this, please see Reveal Review Manager Overview.

Create Company

The Reveal Company may already exist. If it does exist, please skip this step, otherwise create the Company Reveal using the steps below. If it does not exist, or you want to create a new Company, follow the steps below.

A Company is the highest level of organization in our permission structure. Please create a Company named Reveal.

1. Open Review Manager on the Load Machine
2. Expand Instance Setup
3. Click Companies
4. Click the Add New tab
5. Enter Reveal (or preferred name) as the Company name, and click Insert
   1. 

Create Client

The Reveal Client may already exist. If it does exist, please skip this step, otherwise create the Client Reveal using the steps below. If it does not exist, or you want to create a new Client, follow the steps below.

A Client is only associated with one Company. A Client can have multiple projects associated with it and is a way to keep a logical structure to the projects.

1. Open Review Manager on the Load Machine
2. Expand Instance Setup
3. Click Clients
4. Click the Add New tab, enter Reveal (or preferred name) as the Client name, enter a Client Number (typically a number, recommended to use date created in YYYYMMDD format), associate the Client with a Company, and Click Insert
   1. 

Create Rapid Pilot (RP) Enron Project

After creating or using a previously created Company and Client, follow the steps below to create a new Rapid Pilot.

1. Open Review Manager on the Load Machine
2. Expand Instance Setup, select Projects
3. Click the New tab
4. Enter the Project Name (use ‘RP_’ as the prefix, the client's MSA_Enron as the project name, i.e. RP_81010001_Enron), enter a Project ID (typically a number, recommended to use date created in YYYYMMDD format), associate the Company/Client, choose the Reveal InControl Standard Template, and UTC for Time Zone.
5. Click Create Case
   1. 

User Setup

After creating the project, there are a few things that must be done prior to importing the data.

Create Review Account and Add to Rapid Pilot Project

Create a Review account for yourself and make your user a Company Admin as detailed below.

1. Open Review Manager on the Load Machine
2. Expand Project Setup
3. Click Users
4. Click Add
5. Click Create New, fill in the Create a new User ID form, and click Create
   1. 
6. Choose the Review Account, and click Add
   1. 
7. Change Group to Administrators
   1. 

Company Admin Rights

A user must have Company Admin rights to create users or a project within Review. To make a user a Company Admin follow the steps below.

1. Open Review Manager on the Load Machine
2. Expand Instance Setup, and select Companies
3. Click the Users By Company tab
4. Click Add User, and choose your user
5. After adding the user select the checkbox Is Admin.
   1. 

Checking for Company Admin Rights in Review

If a user does not have access to the Load Machine, but wants to check for Company Admin rights, the user can check within Review.

1. Log into Review
   1. You can get the URL to Review to any environment in Review Manager by expanding Advanced Options, click System Settings, and in the Value column type 'indexserverserviceclient'. When copying this URL result, exclude '/review_indexserver' from the result.
2. Click Admin
   1. 
3. Click the Menu button in the upper left-hand corner and look for Company Admin.
   1. 

Import Data

We will import data into the project. This requires importing the DAT file into the system and creating indexes. To do this, follow the steps below. 

Field Creation

To load the data set change the below settings.

1. Open Review Manager on the Load Machine
2. Expand Project Setup
3. Click Fields
4. Change DOMAINS from 4000 to 0 which is the max characters allowed for a field.
   1. 

Import Mappings

A new field mapping will also need to be created to load the data.

1. Open Review Manager on the Load Machine
2. Expand Project Setup
3. Click Import Mappings
4. Browse to the following DAT file
   1. Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0001_EXPORT01\EXPORT01_loadfile.dat
5. Click Match All Fields 
   1. 
6. Sort by the field Table Name, click in one of the fields that say -- Select -- and find/remap only the fields that show here as the did not map properly (there will not be that many). When this is complete, there will be no more fields remaining in the drop down. 
   1. 
7. After mapping the unmapped fields, save this mapping as the Rapid Pilot Mapping
   1. 

Documents Import

To import the DAT file, follow the steps below.

1. Open Review Manager on the Load Machine
2. Expand Import
3. Click Documents
4. Choose the Rapid Pilot project
5. Choose the Rapid Pilot Mapping
6. Import 1
   1. Import Data File: Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0001_EXPORT01\EXPORT01_loadfile.dat
   2. Click Import Data
   3. This will load 40,656 files.
      1. 
      2. 
      3. 
         1. You can ignore this error and click OK.
7. Import 2
   1. Repeat the same steps from Import 1
   2. Use Import Data File: Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0002_EXPORT02\EXPORT02_loadfile.dat
   3. This will load 86 files.

Documents Overlay

When this data set was created, there were a few fields that were not within the DAT file that are essential for visual analytics and AI. To get these fields populated, we will need to do an overlay or Update Data for the project in Review Manager. To overlay these fields, follow the steps below.

1. Open Review Manager on the Load Machine
2. Expand Import
3. Click Documents
4. Choose the Rapid Pilot project
5. Choose Update Data
6. Choose CSV
7. Overlay
   1. Import Data File: Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0003_BRS_OVERLAY\BRS_OVERLAY.CSV
      1. Please note that this is a CSV file and not a DAT file.
   2. Click Update Data
   3. Click Match All
   4. We are only overlaying 3 fields. You will need to find the following two fields and deselect them from the overlay, as Match All will return 5 fields.
      1. Find the field SENT_DATE and deselect it (remove the checkbox from Import for the row) as we should be overlaying the field SENT_DATETIME and not SENT_DATE
      2. Find the field SUBJECT and deselect it (remove the checkbox from Import for the row) as we should be overlaying the field SUBJECT_OTHER and not SUBJECT
   5. Click OK
   6. This will update all files (40,742)
      1. 

Create Indexes

After importing the data, you need to index the data. Please only do this after Store Pending = 0 as seen below. As part of this process, we grab the native and text files from the load file, and move them to S3 in AWS, or the “store”. Also note there is 1 file that is a Store Error. This was a virus that was removed by AV, so this is expected. This process will take some time, so this may be a good time to take a break for about an hour.

Please note the project in the screen shot is for illustration purposes only, and you should be choosing your project you created.

1. Open Review Manager on the Load Machine
2. Expand Create
3. Click Indexes
4. Choose the Rapid Pilot project
5. Select both indexes
6. Click Index/Re-Index
7. Click Native / HTML, Extracted, OCR / Loaded, Document_Metadata, and click OK
   1. 

Create Document Folders

After kicking off indexing, run the following process in the Create section of Review Manager.

1. Document Folders
   1. Choose the Rapid Pilot project
   2. Select both data sets
   3. Change Document Folder Field to RELATIVEPATHPARENT
   4. Click Run
      1. 

Monitor Progress

1. You can monitor the indexing progress in the Review Manager, or you can go to Menu -> Jobs -> Index, select View All Jobs, and you will see the indexing process of the import. Make sure the counts in your project match the counts in the screen shot below except for Total Errors.
   
   1. 
2. Run the term depart* to make sure 5,260 documents are returned.
   1. 
3. Make sure the data loaded properly into AI by going to Menu -> Jobs -> AI Document Sync. If this shows complete
   1. 
4. Everything will be complete when the Candy Bar displays counts on the Dashboard, and that the index Brainspace connector overlay completed as well.  
   1. 
   2. 

Run Final Processes

There are 3 processes to run if you need to showcase the Review Accelerators (i.e., Image Label, Translation, and Transcription). Please note the project in the screen shots is for illustration purposes only, and you should be running these processes in the project you created. When the Label, Translation, and Transcription jobs complete, you will receive an email for each job.

Field Profile Management

By default, field profiles are not assigned to roles out of the box. This makes the Default Field Profile inaccessible for all users. To fix this, follow the below steps, and you can use the Default field profile. 

1. Login to your project via your browser in Review
2. Select Admin 
3. Click Fields -> select Default -> click Edit
   1. 
4. Select Original Administrators -> click UPDATE
   1. 
5. Switch to Grid, and choose Default if not selected
   1. 

Label

Run Image Label on the following images for demo purposes.

1. Login to your project via your browser in Review
2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Images, and this should return 49 files
3. Click the Label button
4. Click LABEL
5. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.
   1. 

Translate

There are two sets of documents for translation purposes of Japanese and Korean. Below illustrates how to translate the Japanese files. The process should be repeated for the Korean files as well.

1. Login to your project via your browser in Review.
2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Foreign Languages -> Japanese, and this should return 30 files
3. Click the Translate button
4. Next to Destination Text Set click the + Create new button, enter the following items, and click Submit.
   1. 
5. Back on the Translate Documents settings, choose the following options.
   1. 
   2. If the Destination Text Set that was just created does not show up in the list of text sets to choose from, logout/login, and the text set will be available. 
6. Repeat this process for the Korean files (Documents Folder Documents -> Dropbox -> Group Share -> Foreign Languages -> Korean), but translate from Korean to English. Please add the Korean files to the same Text Set of Translation, and do not create a new Text Set.
7. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

Transcribe

There is 1 deposition video to transcribe.

1. Login to your project via your browser in Review.
2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Videos, and this should return 1 file
3. Click the Transcribe, and click TRANSCRIBE (leaving all default settings)
   1. 
4. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

Show Transcription

After the deposition video completes, turn on the transcription to show it during a demo.

1. Login to your project via your browser in Review.
2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Videos, and this should return 1 file
3. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.
4. Launch this file in Review by clicking on the row.
5. Click the CC Show Transcript button , and this will show the text next to the video when it is played. This setting will stick for all future demos of this file.