Tutorial: Assigning Coders – iCode Support

Methods

Method 1: Assign when adding content/making project

Method 2: Upload data - Record codes (Additional Data)

Option 1. Randomly assign individual records to a single coder
- Sharing a unique, individualized (coding) link with a coder
Option 2. Assign records by category (grant number) to a single coder
Option 3. Assign multiple coders per record in a single field (kappa test set)
Option 4. Assign multiple coders per record in multiple fields

Background

iCode facilitates the manual curation of a project/project/RFI. Curation can be shared amongst multiple coders reviewing separate records or multiple coders reviewing the same record. There is currently no way within the tool to assign records to coders. However, this is possible by deciding on curation allocation and uploading a custom field (or fields) to the project. This field will become a Filter in the Filters screen and can be used to select a result set for an individual to code.
In this tutorial, we’ll look at random allocation and grouped allocation. Other methods of allocation can be used, e.g. allocate first 50 records to the first coder etc. The process is the same as in the examples below you would just populate the Excel file in a different way.

Methods

Method 1: Assign when adding content/making project

Create your project in iCode and add the records you wish to code, along with additional fields.

Use this option to add all of your data to the project at once using Manage and Add Content. In this article, we will use those steps for the various methodologies.

Method 2: Upload data - Record codes (Additional Data)

Upload additional tags to records in your project at a later point.

You can use this option to add coders, or any other tag to a record. Using this feature does not require you to upload all of the data again.

Decide on the method to assign coders below, prep the coder field(s) in Excel and review the Upload data tutorial to learn more.

Regardless of the point in time which you add Coders, you have 4 methods to assign coders in the pre-uploaded spreadsheet.

Option 1. Randomly assign individual records to a single coder

Sharing a unique, individualized (coding) link with a coder

Option 2. Assign records by category (grant number) to a single coder

Option 3. Assign multiple coders per record in a single field (kappa test set)

Option 4. Assign multiple coders per record in multiple fields

Once you have created your project, go to Add Collaborators and add the people you want coding records if you know this information; if you’re unsure, this can be updated later. You will want your collaborators to be coders and potentially managers as well (the latter only if you want them to have permissions to edit fields, content and roles as well). More on project user functionality here: https://support.icode.opa-tools.od.nih.gov/hc/en-us/articles/51429225979419-Adding-Collaborators. You can also add coding fields at this time if you’re ready. These can also be added or updated later.

Next, use the export button to download all of the fields from the iCode project as an Excel file (if you don’t already have a file with all of your project data).

Option 1: Randomly assign individual records to single coder

In the download, add an additional column to contain the name of the ‘Coder’.
Next enter the names of your coders in this column.
Highlighting the names and double clicking on the small green square at the bottom right of the highlighted cells will copy the values down your list of records.
Now, delete any additional sheets from the workbook (if applicable) and save the spreadsheet. Keep all fields, as if you upload a spreadsheet using 'Add Content' with conflicting content, you may lost previous data.

THE NEXT STEPS ARE IF YOU ARE UPLOADING THIS DATA USING ADD CONTENT
Add Content. Go to iCode and ensure you are in your project. Click on Manage Content, then select Add Content and choose the Upload Data from spreadsheet option. Click Browse in the middle of the screen and find the spreadsheet you have just saved, click Open.

You will then be asked to select which field to use as the Identifier and which fields to upload. In the first Select ID Field dropdown, select the System ID if you have downloaded it (or the ApplID in Grants or PMID in publications). In the fields to include dropdown, make sure you select ‘Coders’, then click 'Add and Close’ at bottom right.

Click ‘Next’ to review the added spreadsheet and select the fields to include in the new upload (Coder in this case) to add to the project. Note: at current, you will need to select all fields as to not replace what is in the project when uploading a nearly identical spreadsheet. An append/replace option is planned to be added in the future.

On the next screen, ensure Coder is filterable and hit Save at the bottom.

Allow the project time to update. You will likely need to Refresh the screen to see the newly added fields.

Now, when you go to the Filters screen and scroll down, you should see the new ‘Coder’ field. Clicking on one of the values in the facet window will filter the results on that facet, so that you are only coding records assigned to that individual.

By filtering to an individual coder's records, a breadcrumb will populate at the top.

If you have created coding fields already, you can begin coding by clicking on ‘Start Coding’. As you code the records and click Next, it will move through the subset allocated to this coder.

Note: You can also add a ‘Completed coding?’ field so that coders can filter on this and always come to their unfinished records link. You will need to bulk code all of the ‘Completed coding?’ fields as No first, because Yes/No fields in iCode are blank by default.

Sharing a unique link with a coder

Once you have done this, you can filter on a particular coder and their corresponding ‘Completed Coding’=No filter, and send this unique url to the coder in order for them to always revisit their coding link and come to records that still need coding by them.

Add screenshot

Option 2: Assign records by category to a single coder

The process here is the same as for Option 1, but first you need to allocate coders in a more complex way. For example, you may want to group applications by grant number as curations are likely to be similar.
Export the system IDs from your project along with any additional fields required, e.g. grant number.
In Excel, create a PivotTable of your data with grant number as the rows and counts of ApplID in the cells. You can then decide which grant numbers you wish to assign to your coders. If you are unfamiliar with PivotTables in Excel, there is a tip sheet on the OPA Excel tips page, tip #4: https://dpcpsi.nih.gov/eo/intranet/opa/training/excel_tips

Now that you’ve now decided who will code each grant number, you need to assign the Appl IDs to coders so that they are ready to upload to iCode. You can do this using an XLOOKUP formula. Go to the data sheet (the data exported from iCode and add a new column header for ‘Coder’.
Enter a XLOOKUP, searching for the relevant grant number in the pivot table you created, and select the column you added to the pivot table including coder name. Copy the formula down by double clicking on the bottom right crosshair that populates when hovering on the first completed cell. If you are unfamiliar with XLOOKUPs, see OPA Excel tip #1: https://dpcpsi.nih.gov/sites/default/files/Training-Excel-Tip-XLookup-06-17-2022.pdf

You can now follow Option 1 from step 5 on to upload your file, update iCode views then filter your data to the assigned collaborators to begin curation. The new facet you can limit your records on will appear the same as before.

Option 3: Assign multiple coders per record in a single field (kappa test set)

In some instances, a curation exercise will require multiple people to code the same record. This is generally good practice to do for a small set of records prior to the full coding exercise. It ensures all potential coders are following the same coding guidelines/interpretations when coding; this is typically referred to as the kappa (test) set. The number necessary for this depends slightly on how large and difficult your dataset may be to code; 20 or 50 records may be necessary to capture the range of contents/potential interpretations.

Note: If all coders are coding all records, there is no need to allocate, however if you are dividing a project up amongst multiple coders, you may want e.g. two people to code every record. In this case, it is possible to use a single, multi-valued field, or a separate field for each coder. This option looks at the former, the latter is covered in option 4.

In a similar way to Option 1, create a column with the name of the first coder for each field, then create a second column containing the names of the second coder. These names can be combined into a single field using the CONCATENATE function in Excel, including a ‘;’ between names, as shown below. If you are unfamiliar with CONCATENATE, see OPA Excel tip #5: https://dpcpsi.nih.gov/eo/intranet/opa/training/excel_tips

Copy the formula to however many rows you wish to include as a kappa in your dataset, then save the file, ensuring you only save a single worksheet.

You can now follow Option 1 from step 5 on, uploading the concatenated ‘Coder_Overlap’ field, and if you wish, the columns for individual coders. iCode should recognize the semicolon delimited column as indicating multiple values (if it doesn’t, tell the system it is multi-valued in the Upload setup, which is viewable in Step 8 of Option 1 above) and assign the records accordingly. Make sure to make the newly uploaded data filterable too. Your newly uploaded data/field will appear as a new filter similarly as before.

Option 4: Assign multiple coders per record in multiple fields

This can be done in Excel by creating three (or more) columns with the coders’ names as column headers and a ‘Yes’ to indicate the coder has been assigned the record. We can leave records not assigned to the coder as blank or use No. Assign the records across coders to create the desired pairs, then copy to all rows. In this instance, we want to have full cross curation so we can compare each coder against the other two.
Save the spreadsheet, ensuring there is only a single worksheet.
Now, go to the Option 1 example above beginning at step 5, but ensure when selecting fields to upload, you select the fields for all coders. You will end up with a filter box for each coder, enabling them to select just the records they have been assigned (see the second image below where records have been limited to ‘Matt’ and ‘Yes’).