Automatic task assignment

Potato can automatically assign instances to annotators based on figurations.

  • on: whether do automatic task assignment for annotators, default False. If False, all the instances in your input data will be displayed to each participant.
  • sampling_strategy: how you want to assign the instances to each participant. If random, the instances will be randomly assigned. If set as ordered, the instances will be assigned following the order of your input data.
  • labels_per_instance: how many labels do you need for each instance, default 3
  • instance_per_annotator: how many instances do you want each participant to annotate, default 5
  • test_question_per_annotator: how many test instances do you want each annotator to see, default 0
"automatic_assignment": {
"on": True, #whether do automatic task assignment for annotators, default False.
"output_filename": 'task_assignment.json', #no need to change
"sampling_strategy": 'random', #currently we support random assignment or ordered assignment. Use 'random' for random assignment and 'ordered' for ordered assignment
"labels_per_instance": 3,  #the number of labels for each instance
"instance_per_annotator": 5, #the total amount of instances to be assigned to each annotator
"test_question_per_annotator": 0, # the number of attention test question to be inserted into the annotation queue. you must set up the test question in surveyflow to use this function
},

Setting up test questions

In some cases, you might need to insert some test questions into the annotation queue. These test questions are usually a small set of super easy instances with golden labels. To define test question instances, you can simply add _testing into the normal instance id. For example:

text,id
"This is test question 1", 0_testing
"This is test question 2", 1_testing
"This is test question 3", 2_testing
"Hello world", dkjfd
"Hello tomorrow", 998ds

You can easily define these test question instances in the same input data file. Potato will identify these questions and sample N of them into the annotation queue, where N = test_question_per_annotator