This is an article organizing the ideas I came to realize while thinking about how to write code that is Pythonic, readable, and efficient.

Before

result = make_passages.apply(self.__make_passages_and_retrieval_gt, axis=1)
        make_passages['passages'] = [passage[0] for passage in result]
        passages = [passage for lst_passage in make_passages['passages'] for passage in lst_passage]

        # Create retrieval_gt and retrieval_gt_order
        self.qa_data['retrieval_gt'] = [passage[1] for passage in result]
        self.qa_data['retrieval_gt_order'] = [passage[2] for passage in result]

After

I unpacked it with Zip.

result = make_passages.apply(self.__make_passages_and_retrieval_gt, axis=1)
make_passages['passages'], self.qa_data['retrieval_gt'], self.qa_data['retrieval_gt_order'] = zip(*result)

# Flatten the list of passages
passages = [passage for lst_passage in make_passages['passages'] for passage in lst_passage]