The question of reproducing the effect of mhr on the ni dataset

[TJL233]

Hello!

Thank you for your work!

I had the following problems while trying to reproduce:

SuperNI data set 1, the paper provided the experimental effect of the table is under on the https://github.com/allenai/natural-instructions?

2. I ran create_datasets.sh in datasets to get the original ni folder and the ni_data/processed/ dataset folder after create_ni_dataset. But when I set data_dir in pretrain.json to any of the seed-num_examples subfolders in ni_data/processed, ni_original_dataset.py won't load the data for me, Instead, using the original ni folder is fine

3. Regarding the experimental effect of SuperNI provided in the paper, were all their training, testing, and verification data obtained by randomly selecting 100 samples from 20 tasks in splits/default/test_tasks.txt?

4. Is the experimental effect of SuperNI provided in the paper on zero-shot or few shot?

Looking forward to your reply, thank you!

[TJL233]

1. Is the SuperNI dataset obtained from this website? (https://github.com/allenai/natural-instructions)

[pclucas14]

Hi,

Thanks for your interest in our work!

1. Yes the dataset is obtained from this repository (if you look in create_datasets.sh you will see the command git clone https://github.com/allenai/natural-instructions.git /tmp/ni)

2. I don't recall this config exactly, it's very possible that you may need to do a small fix and that you are correct

3. I think so. @sordonia can you confirm ?

4. SNI results are few-shot

Hope this helps,
Lucas