For testing NLP and other text processing software, you need some kind of text corpus. If you want your results to be reproducible, the person reproducing them needs to have access to the same text corpus you were using. But corpora of a useful size are often awkward and/or illegal to distribute, and, because the source materials are often revised over time, they can be impossible to reproduce at any later time.
I wanted a corpus that was:
So, this is a 50-megabyte, 6-million-word English text corpus derived from a subset of the freely-redistributable IETF RFCs, which anyone should be able to reproduce and verify for some time in the future. These RFC files date from the years 01976 to 01997, they are not revised over time, their canonical version is the ASCII text, and for several decades they have also been freely available on the ARPANET and Internet from the Network Information Center, later the InterNIC.
Nowadays they are also available from the RFC editor function of the IETF; for example, RFC 822 is available from https://www.rfc-editor.org/rfc/rfc822.txt.
I got them from Debian.
Under copyright these documents are licensed for unlimited copying and redistribution, and they contain very little of the kind of political or religious content that often results in restrictions on redistribution.
I’ve checked this corpus into a Git repository; Git compresses it to only 13 megabytes. You can currently obtain a copy of it with Git as follows:
git clone http://canonical.org/~kragen/rfc-corpus.git
This takes about 8 seconds for me, downloading 13 megabytes of data, and uses 63 megabytes of disk space.
I’ve also prepared a .tar.gz file of the RFC corpus, which is also 13 megabytes.
These are the shell commands I used to get the RFCs:
$ mkdir rfc{7..16}xx
$ for century in {7..16}; do
cp /usr/share/doc/RFC/links/rfc"$century"??.txt* ./rfc"$century"xx/.;
done
$ gzip -d ./rfc*/*.gz
See below for summaries of the results; some RFCs are missing.
To make the .tar.gz file mentioned above, after creating some other files including this README, I used this command:
tar czvf rfc-corpus.tar.gz --exclude rfc-corpus/.git --exclude '*~' rfc-corpus
These files were produced from the Debian doc-rfc package, version 20230121-1 from Debian “bookworm”, version 12.7. It pulls in some other packages.
$ dpkg -l doc-rfc*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================-============-============-======================================
ii doc-rfc 20230121-1 all RFC documents metapackage
ii doc-rfc-experimental 20230121-1 all Experimental RFCs
ii doc-rfc-fyi-bcp 20230121-1 all FYI and BCP RFCs
ii doc-rfc-informational 20230121-1 all Informational RFCs
ii doc-rfc-misc 20230121-1 all Historic and draft RFCs
ii doc-rfc-old-std 20230121-1 all Old Standard RFCs
ii doc-rfc-others 20230121-1 all Old experimental and unclassified RFCs
ii doc-rfc-std 20230121-1 all Standard RFCs
ii doc-rfc-std-proposed 20230121-1 all Proposed Standard RFCs
These particular package versions have the following message digests:
$ apt-cache show doc-rfc'*' | egrep '^Package|^MD5sum|^SHA256'
Package: doc-rfc
MD5sum: d6f60f50212d2248d8984afd4df76915
SHA256: b5e7635fe1a4c3913af91753f3238bde9f3eede282e012866bc76a9d986e954e
Package: doc-rfc-experimental
MD5sum: 4602f40bd3493f9222e46e086e4ea2e8
SHA256: bff38cd2d63f0f85ed7eaf954845f34bbaba56dfe3111c9e3e07624296889817
Package: doc-rfc-fyi-bcp
MD5sum: d573de3eb911bf755bcec747d8fe45ca
SHA256: d1db7332103b14ac2ce4e9e09188e38b3ade332bcc791990c7eb8064333a2b9e
Package: doc-rfc-informational
MD5sum: c26945614e2c285eddd501f1b9eafada
SHA256: 143e78778eda9f187548b74028e98acf64ed42e2ea075af0774ca3bbcb0c6d01
Package: doc-rfc-misc
MD5sum: e03705787bc836ae473b400d6a9a2310
SHA256: 395ce1db9ecbd281e162984aa6416487702cfa79f5e9d508522db55645ab20dc
Package: doc-rfc-old-std
MD5sum: 7b48fa052a383daabcbe108a1f856a65
SHA256: 13aa366cbed958a3decc3a89e32c058746764906e1045b5cd00425b1be0ffc8a
Package: doc-rfc-others
MD5sum: 9f4efe611161653ec3354138cec198a1
SHA256: 2b36a22145dd3fcb380079706d1a39afa7714499018d0a607a32325ead292500
Package: doc-rfc-std
MD5sum: c90fbc9a5a0defcc3a33acf468907df1
SHA256: b9bfead1a230da198ab96ce70cc9fbb64dc600d914c1504eeb4c19e29f95734b
Package: doc-rfc-std-proposed
MD5sum: 3ecf4fe293fc44c16db6972ebd3fb31c
SHA256: b34f60b4bee98891524a2c2508f3d4dcb1e1468a2bd26e0a5f5aecf95806eea0
I haven’t checked these extensively against canonical versions to attempt to detect divergences, but I have verified that rfc822.txt has the same MD5 as the version from rfc-editor.org, 59975cc77db34020d4f62e69c6f53dcf.
A few of the RFCs, even from this fairly recent period, have been lost:
$ for century in {7..16}; do
for item in {00..99}; do
fn="rfc${century}xx/rfc$century${item}.txt" ;
if [ ! -e "$fn" ]; then echo "$fn is missing"; fi;
done;
done
rfc7xx/rfc709.txt is missing
rfc7xx/rfc710.txt is missing
rfc7xx/rfc711.txt is missing
rfc7xx/rfc715.txt is missing
rfc7xx/rfc723.txt is missing
rfc8xx/rfc853.txt is missing
rfc10xx/rfc1061.txt is missing
rfc11xx/rfc1182.txt is missing
rfc12xx/rfc1260.txt is missing
It is conceivable that at some future time some or all of those lost RFCs will be recovered from printouts; if so, they are likely to go through multiple revisions.
For people who might want to reproduce this corpus in the future, it would be useful to be able to verify that they had succeeded.
The total amount of data in each subdirectory is as follows. The numerical columns indicate the number of lines, the number of words, and the number of bytes in each subdirectory.
$ for dir in rfc?xx rfc??xx; do echo -n "$dir "; cat "$dir"/* | wc; done
rfc7xx 91074 373094 3006571
rfc8xx 107803 442433 3741634
rfc9xx 143962 614715 5092855
rfc10xx 155347 741426 5956406
rfc11xx 163782 756379 6396720
rfc12xx 138235 626978 5188846
rfc13xx 128191 643769 4953090
rfc14xx 121066 530155 4397532
rfc15xx 157784 769799 6217306
rfc16xx 129446 590242 4840144
The SHA-1, MD5, and SHA-256 checksums of this concatenated data are as follows. The Unix shell command is concatenating the files in each directory in numerical order.
$ for dir in rfc?xx rfc??xx; do echo -n "$dir "; cat "$dir"/* | sha1sum; done
rfc7xx d5bf39bf14df9f9f8b795922e4cd0ca9e8c1a54b -
rfc8xx d85c70a96bb8e2790c4ea8c0745e318c3a7fb62a -
rfc9xx 557c5b27ad4ac9f0f50ba9b15bc16d76f476341b -
rfc10xx 508b96f66402e248c57492fd21e5aafd1730f6a5 -
rfc11xx d0466f00e6eb89f7fcbfccddd30dbfb3a976776b -
rfc12xx 67db47bc3dd52c5ac7f932082dbbeef30834e655 -
rfc13xx 8ade631f95e50440f5235de3312348ae1e92ed1f -
rfc14xx 3fcb7c9654e4c1ea03502f1f8255fe30b2cf586f -
rfc15xx a0b0c4349470213503cedb728150b9eeba6b9ea0 -
rfc16xx 8fa6ec474f8cc46291e6752de599cfa599adedbb -
$ for dir in rfc?xx rfc??xx; do echo -n "$dir "; cat "$dir"/* | md5sum; done
rfc7xx 28f32042dd0162e5843aebf0426bff8b -
rfc8xx 636d977e9c6d92a956b841abe8e36c08 -
rfc9xx 74c53ef48d74e6e4784f6624c7c84c2e -
rfc10xx 57f55e663c074ae60c6d0dc3a68cfb1f -
rfc11xx a3a52539bf86ef7e5f013c0ef5efc65c -
rfc12xx 0b8b3752082c92a2b8eeb41d71a16952 -
rfc13xx 3a19cd55bc6b77dbc7a01dd43044de51 -
rfc14xx daa176c38998a68a3e47ddc8fb01f751 -
rfc15xx 148a385766ccf53667dbb02c17c25b49 -
rfc16xx 9f0c44f0d8c01cf0a5155a82c58bad86 -
$ for dir in rfc?xx rfc??xx; do echo -n "$dir "; cat "$dir"/* | sha256sum; done
rfc7xx f72ef015b8879cef50dcbb86a557d73c705dae12205452c787cbcc79b6fa8dc1 -
rfc8xx aa97678d69a617e59a47cf1794fc4c493f0fd1a93de4a290850f2c831d74bef5 -
rfc9xx c1fc6d1de64441b57af10d59a1743051f5c5123481250a8727ef45bca3d959cb -
rfc10xx 1f09aea4bf2f2f75756e79af41550d589843411efd34514ea9f7f2766c5b24e4 -
rfc11xx f461664f96d25c0c5fb0c77c73fb910beae9efde282c1adf7eeebdf1e40db57e -
rfc12xx ac835f2ebbb2d5e5145540b441b824d9e21fa90ed0dcca1c97b8131cda25caf3 -
rfc13xx b0afdc1bd1b944adba72123c792242746c9e09bd764660be32b80d688a1bb364 -
rfc14xx 22700adcb79a077164ce1a3b2d333552ff1cb5f38336d976615f9abb4b3c5895 -
rfc15xx 22e04bcfe7def40bbd9fd390035ba34ff0a2fc33f8665d1657a7acc8fe407bd4 -
rfc16xx 255d43b997b8f33bb554907d2bc172944f81760ca67b4fc31984ad3e3f0827ce -
File-by-file checksums are in files rfcs.md5s, rfcs.sha1s, and
rfcs.sha256s; I am not including them here in the interest of
keeping this description file small.