Natural Language Processing (NLP)

NLP aims to make human language accessible to computers.

🛡 Pre-Processing

Turns unstructured chaos into something that can be used in NLP.

flowchart LR
    A[Raw Text] --> B["1\. Cleaning
(Whole String)"]
    B --> C["2\. Normalize
(Whole String)"]
  C --> D["3\. Tokenize
\(Split into tokens\)"]
  D --> E["4\. Optional: Remove Stop Words"]
  E --> F["5\. Optional: Lemmatize / Stem"]
  F --> G["6\. Parse / Tag (Task dependent)"]

1. Cleaning

Text cleaning removes unwanted elements ("noise") so the model focuses on useful signal.

For example:

• Removing HTML Tags & Formatting: Stripping out webpage code like <p> or \n.
• Removing Special Characters: Handling emojis, URLs, or punctuation if they do not add meaning to the specific task.
• Removing excessive characters: Repeated characters in words like "BESTTTTTTTT" and "luvvvv" are reduced to a standard form to improve readability and eliminate noise.
• Removing excessive punctuation: Multiple exclamation marks (!!!) are reduced to a single one.
• Handling hashtags: Hashtags like "#amazing" and "#awesome" are kept as they carry topic information, but are separated from the surrounding text for clarity.
• Whitespace cleanup: Removing extra spaces and line breaks.

2. Normalization

Text normalization transforms text into a standard, consistent format so the computer recognizes variations of the same word as identical.

• Case folding: Converting everything to one uniform case so that "Apple", "apple", and "APPLE" are treated as the exact same token.
• Expanding abbreviations: "omg" is expanded to "oh my god" to normalize the language.
• Standardizing expressions: Non-standard spellings like "luvvv" are corrected to "love" for consistency.
• Emoji / emoticon normalization: Emojis like "😊" can be mapped to text labels like smiling_face so they are machine-readable.
• Unicode normalization: Treating visually similar text forms consistently (for example, accented variants).
• Negation handling (important): Phrases like "not good" should preserve negation because it flips sentiment.

3. Tokenization

Tokenization is the process of ➀ breaking down a stream, ➁ phrase, ➂ sentence, ➃ paragraph, or ➄ an entire text document of text into smaller units, which are called tokens.
These tokens can be words, phrases, symbols, or other meaningful elements depending on the context and the task at hand in NLP. Words, numbers or punctuation marks can be tokens.

Tokenization is the foundation and first step when working on your NLP pipeline or tasks; Tokens can help us to understand the ➀ frequency of a particular word in the data and ➁ can be used directly as a vector representing the data. In this way, the string goes from being unstructured into a numerical data structure which helps NLP.

Types of Tokenization:

1. Sentence Tokenization:

In this type of tokenization, text is split into sentences based on punctuation marks such as periods, exclamation marks, and question marks.

2. Word Tokenization:

This is the most common form, where text is split into words based on whitespace and punctuation. In this example, word tokenization splits the text into individual words, respecting punctuation marks and retaining the structure of the text. Each word becomes a token that can be further analyzed or processed for tasks such as sentiment analysis, named entity recognition, or text classification.

Handle contractions and possessives appropriately (e.g., "don't" -> ["do", "n't"], "cat's" -> ["cat", "'s"]).

3. Subword Tokenization:

This splits words into smaller units, which is often useful for languages with complex word formations or for generating more robust word representations. Subword tokenization breaks words into smaller units, useful for languages with complex morphology or for generating robust word representations. In this example, "Unsupervised" is split into "Un" and "supervised," providing more granularity in representation compared to traditional word tokenization.

mindmap
  root((Tokenization))
    WhiteSpaceTokenization["White Space Tokenization"]
    
    SubwordTokenization["Subword Tokenization"]
      SubwordTokenization --> BytePairEncoding["Byte Pair Encoding"]
      SubwordTokenization --> WordPieceEncoding["Word Piece Encoding"]
      SubwordTokenization --> SentencePieceEncoding["Sentence Piece Encoding"]
      SubwordTokenization --> UnigramModel["Unigram language model"]
    
    RuleBasedTokenization["Rule based Tokenization"]
    
    DictionaryBasedTokenization["Dictionary based Tokenization"]
    
    PennTreeTokenization["Penn Tree Tokenization"]
    
    MosesTokenizer["Moses Tokenizer"]
    
    SpacyTokenizer["Spacy Tokenizer"]

NLTK provides these types of tokenizers:

1. word_tokenize(): This function splits text into individual words or tokens.
2. sent_tokenize(): This function splits text into sentences based on punctuation marks such as periods, exclamation marks, and question marks. This function ensures that each sentence is treated as a separate unit of analysis.
3. TweetTokenizer(): This function is designed specifically for processing social media texts, which often contain hashtags, mentions, and emoticons.
4. RegexpTokenizer(): This tokenization is based on regular expressions, allowing users to define custom patterns for token boundaries.
5. WhitespaceTokenizer(): This function is based on whitespace characters, useful for cases where text is already preprocessed or structured.

Refer to www.comet.com/site/blog/tokenization-techniques-in-nlp for more details on each

Code example:

4. Stopword Removal

Remove common stopwords, sparse terms, and particular words to focus on meaningful words. NLTK provides a list of stopwords.

⚠️ Important: stopword removal is optional. It can help classical ML pipelines, but it can hurt tasks like sentiment and question answering where words like "not", "no", and "never" are critical.

Code Example:

5. Adv. Normalization — Stemming vs Lemmatization

Stemming and lemmatization replace groups of words with their root forms

1. Stemming

Stemming is a text normalization technique used to reduce words to their base or root form.

• Simple and fast method, useful for tasks where word meaning is less important (e.g., information retrieval).
• Often useful for search / retrieval style baselines where speed matters more than linguistic precision.
• Might result in non-words and lacks context.
• There are two stemmers available in NLTK,
  • PorterStemmer()
  • SnowballStemmer(). The Snowball stemmer, which is also called Porter2, is an improvement on the original Porter stemmer.

Code Example:

2. Lemmatization

Lemmatization is like stemming, but the outputs are always real Words. Lemmatization converts words to their base or dictionary form.

• Considers the context and part of speech of a word before transforming it, resulting in valid words.
• Provides more meaningful base forms, making it suitable for tasks like text analysis and language understanding.

Example with POS awareness:

• "better" -> "good" (adjective context)
• "saw" -> "see" (verb context)

Code Example:

Comparison between Stemming and Lemmatization

• Stemming is a process that stems or removes last few characters from a word, often leading to incorrect meanings and spelling.
• Lemmatization considers the context and converts the word to its meaningful base form, which is called Lemma.

For instance, stemming the word 'amazed' can return 'amaz'. However, lemmatizing 'amazed' gives 'amaze'.

6. Text EDA Checkpoint (After Basic Pre-processing)

Exploratory Data Analysis (EDA) is typically done after basic cleaning/tokenization and before heavy feature engineering.

What to inspect:

• Word frequency analysis:
  • Analyze the frequency distribution of words in your text data to identify common and rare words.
  • Visualizations like line plot or word-cloud can help in understanding the word-distribution better.
• Sentence length distribution: Helps set truncation/padding limits.
• Class balance: Detects label imbalance early.
• Vocabulary size and OOV risk: Indicates whether subword tokenization is needed.

Common visuals:

• Frequency bar plots
• Word clouds (for quick intuition only)
• Histogram of token counts per sample

🛡 Text Parsing

Text parsing means analyzing grammatical structure after tokenization, not just counting words.

1. Part of Speech (POS) Tagging

POS tagging assigns grammatical labels to tokens (noun, verb, adjective, etc.).
It helps with syntax, information extraction, and downstream tasks like NER and dependency parsing.

Common approaches:

• Rule-based: uses hand-written grammar patterns.
• Statistical: HMM / CRF style sequence models.
• Neural: BiLSTM/Transformer models (current standard in production systems).

Code Example:

2. Syntactic Parsing

Syntactic parsing is an essential Natural Language Processing (NLP) technique that analyzes the grammatical structure of a sentence. By mapping words to their structural or relational roles (e.g., nouns, verbs, subjects, objects), it breaks down text to help machines decode grammar and extract meaning.

• Part of Speech tags Breaks a sentence into words and each word is associated with a part of speech tag such as nouns, verbs, adjectives, adverbs.
• Dependency Grammar: Determines the relationship among the words in a sentence (for example, nsubj, dobj, amod).

As a result of recursively parsing the observed relationship between the words is represented in a top-down manner and depicted as a tree, which is known as the dependency tree.
These grammar relations can be used as features for many NLP problems such as entity-wise sentiment analysis, entity identification, and text classification.

Code Example:

3. Chunking (Shallow Parsing)

Chunk extraction or partial parsing is a process of extracting short phrases from the sentence (tagged with Part-of-Speech) without building a full parse tree.
It is useful when you need light-weight structure with lower computational cost. It uses a special regexp syntax for rules that delimit the chunks. These rules must be converted to ‘regular’ regular expressions before a sentence can be chunked.

Code Example:

4. Named Entity Recognition (NER)

Named Entity Recognition identifies and classifies entities in text into categories such as person, organization, location, date, and money.

Example Text: "Amazon, the multinational technology company, announced plans to acquire Whole Foods Market for $13.7 billion."

Entity Extraction Output:

• Organization: "Amazon", "Whole Foods Market"
• Money: "$13.7 billion"

Code Example:

5. Relation Extraction

Relation extraction identifies semantic relationships between entities.

Example:

• Text: "Amazon acquired Whole Foods Market for $13.7 billion."
• Entities: Amazon (ORG), Whole Foods Market (ORG), $13.7 billion (MONEY)
• Relation triples:
  • (Amazon, acquired, Whole Foods Market)
  • (acquisition, value, $13.7 billion)

External References

• https://www.nltk.org/book/ch03.html