The "Three Golden Rules" of Reviewer Assignments: 

  • Keep already saved/submitted reviews assigned and visible to reviewers as long as possible, typically until there is intentional individual intervention.
  • If applications need to be re-assigned, do this in such a way as to be the least disruptive and allow all reviewers to hold on to existing (started or unstarted) reviews as much as possible.
  • If applications need to be re-assigned, start with the most over-burdened reviewer and work your way down.  

What happens to saved and submitted reviews if a reviewer is removed from a group?

For Auto-Assigning Groups:

  • The reviewer will no longer see this information in their portal
  • The score will be removed from all calculations
  • If that user is later added back to the group:
    • The existing saved/submitted reviews (with scores) will be restored to the portal
    • The scores will be restored to all calculations, including the All Reviews grid with no additional intervention

For Manually Assigning Groups:

  • The reviewer will no longer see this information in their portal
  • The score will be removed from all calculations
  • If that user is later added back to the group:
    • Formerly assigned applications not yet begun will be restored
    • The scores will be restored to all calculations, including the All Reviews grid with no intervention 
    • The existing saved/submitted reviews (with scores) will be restored to the portal


How do I shuffle review assignments?

Let's say there is a group of reviewers sitting around a table playing a card game. One reviewer stands up and quits the game, leaving his cards on the table. The remaining players keep both their played and unplayed cards, but scoop up only the cards the former player left behind (including played and unplayed cards) and redistributes them to the remaining players as equally as possible so that the game can continue.

If a user leaves a group, only the reviews that are no longer assigned (to the absent reviewer) are gathered up and dealt back out to the remaining group members. For remaining group members, they will keep all started and unstarted reviews but may gain additional applications from the former group member.

Another option is to lock the reviewer that will not be finishing their reviews (gets up and leaves the game) so that they will keep any of their saved reviews, and any unsaved reviews will be distributed to the remaining reviewers as evenly as possible.  If a new reviewer is added during the same transaction that a former reviewer is locked, that new reviewer will receive all of the unsaved or unstarted reviews that the locked reviewer had, but the new one must be added and the old one locked before updating the group for this one to one assignment to occur.

Note: Recalculations that involve reducing the number of assignments (ex. at most 5 reviews per application changes to at most 2 reviews per application, or additional reviewers are added to the group) for reviewers follow a “communist” system – that is, if there is a reviewer with more assignments than other reviewers in the group, the system will take assignments from him first and redistribute those assignments (the wealth) equally among the other reviewers in the group. This is the only standard logic when redistribution of applications occurs.

 

How does the Reviewer Email work?

Upon the review period openingReviewer emails are triggered by the opening of the review period. At 3 AM CST on the opening date, all reviewers for the opportunity (or general applications) whether they are new reviewers, or reviewers that are returning from previous cycles. If a review period start date is set after the trigger time of 3 AM CST, then the emails will be marked as already sent and will not be triggered. Once this occurs, setting a review period to a date in the future will not cause the emails to be triggered, as they have already been marked as sent.

Seen: If a reviewer is assigned new applications, and the user authenticates into the system (“Become User” by an admin DOES NOT count) and “sees” these reviews (must click into the opportunity or General Application listing and view either applicant names or ID#s) then the system registers this as “notified” and NO reviewer email will be triggered at the next 3 AM CST job.

Unseen: If new applications are assigned and the reviewer does not authenticate into the system to “see” the new applications before 3 AM CST, a reviewer email will be generated to complete the notification to the reviewer of these newly assigned applications.

A note on count: If a user loses reviews and then is assigned additional new ones, but the overall number of reviews is the same (10 before and 10 after) and does not “see” them, these are still considered “new” assignments and will still trigger the reviewer email. Regardless of how the final number ended up (was 10, still 10) the email will note the number of “unseen” new applications (ex: reviewer has 10 reviews. He loses 3, gains 4, for a total of 11. The email would note there are 4 “new” applications, although he only appears to have gained 1 additional assignment.)  

What happens when the Review Period End Date arrives?

Reviewers are able to edit their reviews until the 'End Review Period' date is met, and then once the End Date passes will be able to see the reviews, but not make any changes, until the opportunity is archived. This will give better control over when reviews should cease, while still allowing reviewers visibility into their evaluations, which can be useful during group review sessions.

More tips on Cloning and Archiving Review Groups

In the past, if an administrator removed a reviewer from a group, it would hide that reviewers score in the archived opportunities and would only restore the scores if the reviewer was added back to the group.  Now administrators can clone a review group, and archive the old groups so that the reviewer can be removed from the newly cloned group without affecting the scores in the archived opportunities. Archiving groups will also clean up the list of review groups so that only the current groups are displayed on the default tab and archived groups will be held on another tab under reviewer groups.

Cloning must be performed before archiving, and it is important that any opportunities that the group has been assigned to are archived first, otherwise all of the reviewer scores will be hidden when the group is archived.

 

What happens if you auto assign reviews and then unclick auto assign and do nothing further? What happens if you then make manual assignments? What happens to completed reviews? Drafted reviews? Not yet saved reviews?

All unstarted application assignments would disappear and cannot be restored exactly as they were previously (you can toggle it back on, but no guarantee the apps will be assigned exactly as they were before.) Saved or submitted reviews will stay assigned to the reviewer via manual assignment (converted into manual assignments). You can choose to go into an opportunity and manually unassign a saved/submitted review from a reviewer, although we will hold on to that review/score such that if you were to re-assign the review (manually or auto) the score/review would be restored.   

Do you need to delete the per reviewer limit before unclicking auto assign? Does order of operations matter here?
No, although if you toggle auto-assign back on it will restore the previous per-reviewer limit as it was when toggled off.