The SearchQA dataset preprocessing involved creating retrieval ground truth without test code and removing duplicated snippets and null values. However, these null values were hidden in numpy arrays wrapped in dictionaries within individual elements of the dataset, and sometimes there were exceptionally troublesome cases where they appeared not as null values but only as blank spaces such as ” or ’ ’.
Let’s always write code to check whether there are null values and whether the length is correct after each preprocessing step. As the dataset and codebase grew larger, it became increasingly difficult to verify everything by eye, and mistakes became more frequent.
Let’s overcome this with comments and test code~~ If you don’t get the idea, take a look at my repository from time to time.