nltk - missing stop words - english

Is this some kind of known bug or ... ?

In[7]: import nltk

In[8]: "shouldn't" in nltk.corpus.stopwords.words("english")
Out[8]: True

In[9]: "couldn't" in nltk.corpus.stopwords.words("english")
Out[9]: True

In[10]: "wouldn't" in nltk.corpus.stopwords.words("english")
Out[10]: True

In[11]: "should" in nltk.corpus.stopwords.words("english")
Out[11]: True

In[12]: "could" in nltk.corpus.stopwords.words("english")
Out[12]: False

In[13]: "would" in nltk.corpus.stopwords.words("english")
Out[13]: False

Suggestion : 2

I just downloaded the latest NLTK version anycodings_nltk with all its resources. ,I see could and would are not listed as stop anycodings_nltk words. But should is treated as a stop anycodings_nltk word. ,Joel Nothman has once looked through the anycodings_python-3.x stopwords and found some disturbing anycodings_python-3.x results. See anycodings_python-3.x https://aclweb.org/anthology/papers/W/W18/W18-2502/,Agree with Ethan McCue, that this is anycodings_python-3.x definitely something that should be anycodings_python-3.x patched/resolved by bringing it up on anycodings_python-3.x NLTK's github issue tracker.

Is this some kind of known bug or ... ?

In[7]: import nltk

In[8]: "shouldn't" in nltk.corpus.stopwords.words("english")
Out[8]: True

In[9]: "couldn't" in nltk.corpus.stopwords.words("english")
Out[9]: True

In[10]: "wouldn't" in nltk.corpus.stopwords.words("english")
Out[10]: True

In[11]: "should" in nltk.corpus.stopwords.words("english")
Out[11]: True

In[12]: "could" in nltk.corpus.stopwords.words("english")
Out[12]: False

In[13]: "would" in nltk.corpus.stopwords.words("english")
Out[13]: False

Suggestion : 3

In the code below, text.txt is the original input file in which stopwords are to be removed. filteredtext.txt is the output file. It can be done using following code: ,Writing code in comment? Please use ide.geeksforgeeks.org, generate link and share the link here.,The process of converting data to something a computer can understand is referred to as pre-processing. One of the major forms of pre-processing is to filter out useless data. In natural language processing, useless words (data), are referred to as stop words. ,To check the list of stopwords you can type the following commands in the python shell.

To check the list of stopwords you can type the following commands in the python shell.

import nltk
from nltk.corpus
import stopwords
print(stopwords.words('english'))

Output:

['This', 'is', 'a', 'sample', 'sentence', ',', 'showing',
   'off', 'the', 'stop', 'words', 'filtration', '.'
]
['This', 'sample', 'sentence', ',', 'showing', 'stop',
   'words', 'filtration', '.'
]

Suggestion : 4

A list of positive and negative opinion words or sentiment words for English.,A list of pros/cons sentences for determining context (aspect) dependent sentiment words, which are then applied to sentiment analysis of comparative sentences.,Most corpora consist of a set of files, each containing a document (or other pieces of text). A list of identifiers for these files is accessed via the fileids() method of the corpus reader:,The OpinionLexiconCorpusReader also provides shortcuts to retrieve positive/negative words:

>>> import nltk.corpus
>>> # The Brown corpus:
>>> print(str(nltk.corpus.brown).replace('\\\\','/'))
<CategorizedTaggedCorpusReader in '.../corpora/brown' ...>
   >>> # The Penn Treebank Corpus:
   >>> print(str(nltk.corpus.treebank).replace('\\\\','/'))
   <BracketParseCorpusReader in '.../corpora/treebank/combined' ...>
      >>> # The Name Genders Corpus:
      >>> print(str(nltk.corpus.names).replace('\\\\','/'))
      <WordListCorpusReader in '.../corpora/names' ...>
         >>> # The Inaugural Address Corpus:
         >>> print(str(nltk.corpus.inaugural).replace('\\\\','/'))
         <PlaintextCorpusReader in '.../corpora/inaugural' ...>

>>> nltk.corpus.treebank.fileids()['wsj_0001.mrg', 'wsj_0002.mrg', 'wsj_0003.mrg', 'wsj_0004.mrg', ...] >>>
   nltk.corpus.inaugural.fileids()['1789-Washington.txt', '1793-Washington.txt', '1797-Adams.txt', ...]

>>> from nltk.corpus
import inaugural
   >>>
   inaugural.raw('1789-Washington.txt')
'Fellow-Citizens of the Senate ...' >>>
inaugural.words('1789-Washington.txt')['Fellow', '-', 'Citizens', 'of', 'the', ...] >>>
   inaugural.sents('1789-Washington.txt')[['Fellow', '-', 'Citizens'...], ['Among', 'the', 'vicissitudes'...]...] >>>
   inaugural.paras('1789-Washington.txt')[[
         ['Fellow', '-', 'Citizens'...]
      ],
      [
         ['Among', 'the', 'vicissitudes'...],
         ['On', 'the', 'one', 'hand', ',', 'I'...]...
      ]...]

>>> l1 = len(inaugural.words('1789-Washington.txt')) >>>
   l2 = len(inaugural.words('1793-Washington.txt')) >>>
   l3 = len(inaugural.words(['1789-Washington.txt', '1793-Washington.txt'])) >>>
   print('%s+%s == %s' % (l1, l2, l3))
1538 + 147 == 1685

>>> len(inaugural.words())
152901

>>> inaugural.readme()[: 32]
'C-Span Inaugural Address Corpus\n'