Tutorial: Assigning Coders

Background

Methods

Upload Fields - Record Codes (Additional Data)

Adding coders in the project

Assigning records to coders

• Option 1. Randomly assign individual records to a single coder

• Option 2. Assign records by category (grant number) to a single coder

• Option 3. Assign multiple coders per record in a single field (kappa test set)

• Option 4. Assign multiple coders per record in multiple fields

Start Coding

Bulk Coding

Sharing a unique, individualized (coding) link with a coder

Background

• iCode facilitates the manual curation of a portfolio, project, or RFI. Curation can be shared among multiple coders who review separate records or the same records. Currently, records cannot be assigned to coders directly within the tool. However, you can allocate records by creating one or more custom fields that identify the coding assignments and uploading them to the project. The custom fields will appear as filters on the Filters screen and can be used to identify the records assigned to each coder.
• In this tutorial, we’ll look at random and grouped allocation methods for assigning coders (Options 1–4 below). Other allocation methods can also be used. For example, you could assign the first 50 records to one coder. The process is the same as in the examples below; you would simply populate the Excel file differently. Any of these assignment methods can be created in Excel and applied using the upload process described below.

Methods

Upload Fields - Record Codes (Additional Data)

Upload additional tags to records in your project after adding records. 

You can use this method to add coder assignments or other tags to existing records. Using this method does not require you to upload all of your data again. You only need the System ID and the additional data fields you want to add to your existing records. Any other fields in your data file can be excluded from the upload.

Review the options below for assigning coders in Excel (or another tool), then use Upload Fields to add the assignments to your project.

Adding coders in the project

Once you have created your project, go to Add Collaborators and add the people who will be coding records, if known. If you are unsure who the coders will be, you can add or update collaborators later.

Assign these collaborators the Coder role. You may also assign them the Manager role if you want them to have additional permissions to edit fields, content, and user roles. See the Adding Collaborators article for more information about project roles and permissions.

You can also add coding fields at this time if you are ready to do so. Coding fields can also be added or updated later.

For coders to find the records assigned to them, you first need to decide how to make the assignments. For example, will you assign records randomly, by grant, or as part of a kappa (overlapping) set? Based on your assignment approach, choose the appropriate option below.

Assigning records to coders

Option 1: Randomly assign individual records to a single coder

1. In Excel, open your exported file. You only need the System ID column. Add a column for the Coder name.

   

2. Enter the names of your coders in the Coder column.

   

3. Highlight the coder names, then double-click the small green square in the bottom-right corner of the selected cells. This will copy the values down the list of records.

    

4. Delete any additional sheets from the workbook, if applicable, and save the spreadsheet.

5. Go to iCode and open your project. From the left panel, select one of the views (we’ll use Results Table), then click Upload Fields.

   

6. In the Upload Fields window, select the Record Codes (Additional Data) tab.

   

7. Click on the cloud icon to select and upload your file. 
8. Make sure the new Coder field is filterable by setting Filterable to Yes. Change the field type if necessary, although you will typically not need to change it. Then click Apply at the bottom.

9. Allow a few moments for the project to update. Close the notification window by clicking the X, then select Reset Search to view the newly added fields.

10. In the Filters view, scroll down to find the new Coder field. Select a coder’s name to filter the results and display only the records assigned to that coder.

Filtering on assignments

When you filter the records for an individual coder, a breadcrumb showing the applied filter will appear at the top of the screen.

Start Coding

If you have already created coding fields, click Start Coding to begin coding. As you code each record and click Next, you will move through the subset of records assigned to that coder.

Start coding full screen record view

Note: To go back to the main project and view all records, click on the project hyperlinked name in the navigation bar at the top.

For more information about coding, see the How to Code tutorial.

Bulk Coding

You can also add a Completed Coding? coding field that allows coders to filter for records they have not yet completed. Before coding begins, you will need to bulk code the Completed Coding? field as No for all records because Yes/No fields in iCode are blank by default.

Sharing a unique link with a coder

Once you have added the necessary filters (Coder and Completed Coding?), filter by a specific coder and set Completed Coding? to No. You can then send the unique URL to that coder. The coder can use this link to return to the filtered set of records that are assigned to them and still need to be coded.

Option 2: Assign records by category to a single coder

1. The process is similar to Option 1, but first you will assign coders based on a specific category. For example, you may want to group applications by grant number because the coding for related applications is likely to be similar.
2. Export the System IDs, grant numbers, and Appl IDs from your project.
3. In Excel, create a PivotTable to group the applications by grant number and see how many Appl IDs are associated with each grant number. Add the grant number to the rows and the count of Appl IDs to the values. You can then decide which grant numbers to assign to each coder. If you are unfamiliar with PivotTables in Excel, see tip #4 on the OPA Excel Tips page.

4. In the PivotTable, add a Coder column and enter the coder you selected for each grant number. Then return to the data sheet containing the data exported from iCode and add a Coder column. You will use an XLOOKUP formula to populate this column with the coder assignments from the PivotTable.

5. In the Coder column of the data sheet, enter an XLOOKUP formula that uses each record’s grant number to find the corresponding grant number in the PivotTable and return the assigned coder. Copy the formula down to populate the Coder field for all records. If you are unfamiliar with XLOOKUP, see OPA Excel tip #1. Before uploading the file to iCode, remove any columns you do not need to upload. Keep the System ID and Coder columns, and then save the file.

6. Follow Option 1 beginning with Step 5 to upload your file and update the project. Then use the new Coder field in Filters to filter the records by assigned coder and begin coding. The Coder filter will appear the same as in Option 1.

Option 3: Assign multiple coders per record in a single field (kappa test set)

1. In some instances, multiple coders may need to code the same record. It can be helpful to have all coders independently code the same small set of records before beginning the full coding exercise. This can help determine whether coders are applying the coding guidelines consistently and identify differences in interpretation. This set of overlapping records is referred to as the kappa (test) set.

   The appropriate number of records for the kappa set will depend on the size and complexity of the dataset. For example, a set of 20 or 50 records may be used to capture a range of content and potential interpretations.

2. Similar to Option 1, create one column containing the name of the first coder assigned to each record and a second column containing the name of the second coder. Then create a new column called Coder_Overlap. Use the CONCATENATE function in Excel to combine the two coder names in the Coder_Overlap column, separated by a semicolon (;), as shown below. If you are unfamiliar with CONCATENATE, see OPA Excel tip #5.

   Copy the formula down for all records you want to include in the kappa set. Then remove any columns you do not need to upload, keeping only the System ID and Coder_Overlap columns, and save the file.

3. Follow Option 1 beginning with Step 5 to upload the concatenated Coder_Overlap field. You can also upload the columns for individual coders if desired.

   iCode should recognize the semicolon-delimited column as a multi-valued field. If it does not, select Multi-Value in the upload options, as described in Step 8 of Option 1. Make sure the newly uploaded field is also configured as filterable.

   The newly uploaded field will appear as a new filter, as in the previous examples.

Option 4: Assign multiple coders per record in multiple fields

1. In Excel, create a separate column for each coder, using the coder’s name as the column header. Enter Yes to indicate that a coder is assigned to a record. For records not assigned to that coder, leave the cell blank or enter No. Assign records across the coder columns to create the desired combinations of coders.

2. Remove any columns you do not need to upload, keeping only the System ID and individual coder columns. Make sure the workbook contains only one worksheet, then save the file.

3. Follow Option 1 beginning with Step 5 to upload the file using Upload Fields. When selecting fields to upload, make sure to include the fields for all coders.

   After the upload is complete, each coder field will appear as a separate filter. 

Coders can use their individual filter and select Yes to view only the records assigned to them. In the example below, the records are filtered to Matt and Yes.

Further information

For more guidance on using iCode, visit the iCode User Guide.

Additional resources:

• Coding Fields and Vocabularies
• Upload Fields tutorial

Back to top