Also, re: "a tiny handful of incredible nerds" - page 20 of this [0] document lists the sales figures for Infocom titles from 1981 to 1986: it sums up to over 2 million shipped units.
Granted, that number does not equal the number "nerds" who played the games because the same player will probably have bought multiple titles if they enjoyed interactive fiction.
However, also keep in mind that some of the games in that table were only available after 1981, i.e., at a later point during the 1981-1986 time frame. Also, the 80s were a prime decade for pirating games, so more people will have played Infocom titles than the sales figures suggest - the document itself mentions this because they sold hint books for some titles separately.
Granted, that number does not equal the number "nerds" who played the games because the same player will probably have bought multiple titles if they enjoyed interactive fiction.
However, also keep in mind that some of the games in that table were only available after 1981, i.e., at a later point during the 1981-1986 time frame. Also, the 80s were a prime decade for pirating games, so more people will have played Infocom titles than the sales figures suggest - the document itself mentions this because they sold hint books for some titles separately.
[0] https://ia601302.us.archive.org/1/items/InfocomCabinetMiscSa...