Usage Guide

Learn how to use the advanced features of the nlp_text_preprocessing package — from building custom pipelines to seamlessly integrating with Pandas DataFrames.

Quick Install

Haven't installed yet? Run pip install nlp-text-preprocessing then import with import nlp_text_preprocessing as tp.

1. Pipeline Automation Engine

The Pipeline class allows you to chain cleaning and transformation steps cleanly without messy nested function calls. Define your steps once and reuse them across your project.

import nlp_text_preprocessing as tp

# Build a custom step pipeline
pipe = tp.Pipeline([
    'to_lower_case',
    'expand_hashtags',
    'remove_urls',
    'remove_emails',
    'remove_extra_whitespace',
    'convert_to_base'
])

# Process a single text string
raw_text = "Check out #MachineLearning at https://example.com!"
clean_result = pipe.run(raw_text)

print(clean_result)
# Output: "check out machine learning at"

Default Pipeline

Just want standard cleaning without configuring individual steps? Use pipe = tp.Pipeline.default() — it includes the most common sanitization routines automatically.

2. Pandas DataFrame Integration

Because all functions have built-in NaN/None defensive guards, you can use them directly on DataFrame columns with .apply() — no lambda wrappers needed.

import pandas as pd
import nlp_text_preprocessing as tp

# Sample messy data with None and NaN rows
df = pd.DataFrame({
    'comments': [
        "Great product! http://buy.me",
        None,
        "Terrible service 😡",
        float('nan')
    ]
})

# Apply a single function safely — no try/except needed
df['clean_comments'] = df['comments'].apply(tp.remove_urls)

# Or apply an entire pipeline over a series
pipe = tp.Pipeline(['to_lower_case', 'emoji_to_text'])
df['processed'] = pipe.run_series(df['comments'])

print(df[['comments', 'processed']])

Null Safety in Action

The None and float('nan') rows in the example above are handled automatically — they return empty strings ("") instead of raising AttributeError or TypeError.

3. Batch Feature Extraction

Use text_stats_batch() to compute multiple features at once across an entire DataFrame column, returning a new DataFrame of features ready for ML pipelines.

import pandas as pd
import nlp_text_preprocessing as tp

df = pd.DataFrame({
    'reviews': [
        "The product quality is excellent and delivery was fast!",
        "Terrible experience, will never buy again.",
        "Average product, nothing special."
    ]
})

# One call — returns word_count, char_count, avg_word_len, sentence_count, etc.
features = tp.text_stats_batch(df, col='reviews')

print(features.head())
#    word_count  char_count  avg_word_len  sentence_count  stop_words_count
# 0           9          49          4.78               1                 4
# 1           7          41          4.86               1                 2
# 2           5          31          5.20               1                 1

Performance Tip

For very large DataFrames (> 1M rows), consider chunking your data or using pipe.run_series() with pandas.Series.swifter.apply() for parallel execution.