| Chime-SMS-2M: training data credits |
|
|
| Luke Steuber trained this model on English text from the five sources below. The text was |
| filtered, normalized and deduplicated, and the model holds weights and a vocabulary, not a copy |
| of the text. The source authors have not endorsed this model. The sources are provided as-is, |
| without warranties, under their own licenses. |
|
|
| 1. NUS SMS Corpus, English portion, release 2015.03.09. |
| Tao Chen and Min-Yen Kan, "Creating a Live, Public Short Message Service Corpus: The NUS SMS |
| Corpus", Language Resources and Evaluation 47(2), 2013. |
| https://doi.org/10.25540/WVM0-4RNX |
| Release mirror: kite1988/nus-sms-corpus, commit bf0a365170d324e61e4025edf363d400f1edd1a6. |
| English messages only. Placeholder tags from the corpus's own anonymization (such as <EMAIL>) |
| were dropped from the text. |
| License: Creative Commons Attribution 4.0 International (CC BY 4.0). |
| https://creativecommons.org/licenses/by/4.0/ |
|
|
| 2. Taskmaster-1 written self-dialogs, TM-1-2019/self-dialogs.json. |
| Byrne et al., "Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset", EMNLP 2019. |
| Copyright Google LLC. |
| https://github.com/google-research-datasets/Taskmaster |
| Commit d92cb6af3005f1dc09c39e75e7daf4a04905e00b. |
| User utterances only. |
| License: Creative Commons Attribution 4.0 International (CC BY 4.0). |
| https://creativecommons.org/licenses/by/4.0/ |
|
|
| 3. OpenAssistant Conversations 2 (OpenAssistant/oasst2). |
| Köpf et al., "OpenAssistant Conversations: Democratizing Large Language Model Alignment", |
| NeurIPS 2023 Datasets and Benchmarks, arXiv:2304.07327. |
| https://huggingface.co/datasets/OpenAssistant/oasst2 |
| Revision 179dd21fc55192153d94adb0e0ce8f69e222bf75. |
| English prompts written by people; assistant turns were not used. |
| License: Apache License 2.0. A copy is in Apache-2.0.txt. |
| https://www.apache.org/licenses/LICENSE-2.0 |
|
|
| 4. Taskmaster-3 (TicketTalk) self-dialogs, TM-3-2020/data. |
| Byrne et al., "TicketTalk: Toward human-level performance with end-to-end, transaction-based |
| dialog systems", ACL 2021. Copyright Google LLC. |
| https://github.com/google-research-datasets/Taskmaster/tree/master/TM-3-2020 |
| Commit d92cb6af3005f1dc09c39e75e7daf4a04905e00b. |
| Customer utterances only. |
| License: Creative Commons Attribution 4.0 International (CC BY 4.0). |
| https://creativecommons.org/licenses/by/4.0/ |
|
|
| 5. SODA (allenai/soda). |
| Kim et al., "SODA: Million-scale Dialogue Distillation with Social Commonsense |
| Contextualization", arXiv:2212.10465. |
| https://huggingface.co/datasets/allenai/soda |
| Revision fdc848ab0183208ea7808206c91c724414d0a071. |
| Both speakers' turns. These dialogues were generated by GPT-3.5 (InstructGPT). |
| License: Creative Commons Attribution 4.0 International (CC BY 4.0). |
| https://creativecommons.org/licenses/by/4.0/ |
|
|
| Vocabulary selection also used a word-frequency list derived from hermitdave/FrequencyWords |
| (OpenSubtitles 2018 English), https://github.com/hermitdave/FrequencyWords, licensed under |
| Creative Commons Attribution-ShareAlike 4.0 International. It shaped which words are in the |
| vocabulary; it is not training text. The emoji tokens come from a word-to-emoji list written |
| for LocalType. |
|
|
