Skip to main content

TextPipe Batch Processing | DataMystic Developer

TextPipe Batch Processing via Command Line

Process hundreds or thousands of text files from the Windows command line using TextPipe. These examples demonstrate wildcard patterns, output folder redirection, multi-filter pipelines, and automated scheduled execution with Windows Task Scheduler.

Basic Batch Processing

Run TextPipe from the command line to apply a saved filter list to one or more input files. The /filter parameter specifies the transformation, /input the source file(s), and /output the destination. 

REM Basic TextPipe command-line invocation
REM Apply a filter to a single file

TextPipe.exe /filter:"C:\Filters\cleanup_csv.fll" /input:"C:\Data\raw_export.csv" /output:"C:\Data\cleaned_export.csv"

REM Process with silent mode (no GUI)
TextPipe.exe /filter:"C:\Filters\ebcdic_to_ascii.fll" /input:"C:\Data\mainframe_dump.dat" /output:"C:\Data\ascii_output.txt" /silent

REM Overwrite existing output files without prompting
TextPipe.exe /filter:"C:\Filters\normalize_dates.fll" /input:"C:\Data\transactions.csv" /output:"C:\Data\transactions_fixed.csv" /overwrite /silent

Wildcard Patterns

Use wildcard patterns to process multiple files matching a specific naming convention. TextPipe supports standard Windows wildcards (* and ?) and can recurse into subdirectories. 

REM Process all CSV files in a folder
TextPipe.exe /filter:"C:\Filters\standardize_headers.fll" /input:"C:\Data\Imports\*.csv" /output:"C:\Data\Processed\" /silent /overwrite

REM Process files matching a date pattern (e.g., report_20250101.txt)
TextPipe.exe /filter:"C:\Filters\strip_control_chars.fll" /input:"C:\Data\Reports\report_2025????.txt" /output:"C:\Data\Clean\" /silent /overwrite

REM Recurse into subdirectories for all .log files
TextPipe.exe /filter:"C:\Filters\extract_errors.fll" /input:"C:\Logs\**\*.log" /output:"C:\Data\ErrorSummary\" /silent /overwrite /recurse

REM Process multiple file types in one pass
TextPipe.exe /filter:"C:\Filters\fix_line_endings.fll" /input:"C:\Source\*.csv" /input:"C:\Source\*.tsv" /input:"C:\Source\*.txt" /output:"C:\Source\Fixed\" /silent /overwrite

Output Folder Redirection

Redirect output to a different folder structure while preserving original filenames. This is essential for production pipelines where source files must remain untouched. 

@echo off
REM Production batch processing script with folder redirection
REM Processes daily data feeds and archives originals

set FILTER_PATH=C:\Filters\daily_feed_cleanup.fll
set INPUT_DIR=C:\Data\Incoming
set OUTPUT_DIR=C:\Data\Processed
set ARCHIVE_DIR=C:\Data\Archive\%date:~-4,4%%date:~-7,2%%date:~-10,2%

REM Create output and archive directories if needed
if not exist "%OUTPUT_DIR%" mkdir "%OUTPUT_DIR%"
if not exist "%ARCHIVE_DIR%" mkdir "%ARCHIVE_DIR%"

REM Process all incoming CSV files
TextPipe.exe /filter:"%FILTER_PATH%" /input:"%INPUT_DIR%\*.csv" /output:"%OUTPUT_DIR%\" /silent /overwrite

REM Check exit code
if %ERRORLEVEL% NEQ 0 (
    echo ERROR: TextPipe processing failed with code %ERRORLEVEL%
    echo %date% %time% - FAILED - Exit code %ERRORLEVEL% >> C:\Logs\batch_processing.log
    exit /b 1
)

REM Archive original files after successful processing
move "%INPUT_DIR%\*.csv" "%ARCHIVE_DIR%\"

echo %date% %time% - SUCCESS - Files processed and archived >> C:\Logs\batch_processing.log

Multi-Filter Pipeline

Chain multiple TextPipe filters in sequence to build a data transformation pipeline. Each stage processes the output of the previous stage. 

@echo off
REM Multi-stage data transformation pipeline
REM Stage 1: Character encoding conversion (EBCDIC to ASCII)
REM Stage 2: Fixed-width to delimited conversion
REM Stage 3: Data validation and cleansing

set SOURCE=C:\Data\Mainframe\daily_extract.dat
set STAGE1_OUT=C:\Data\Pipeline\stage1_ascii.txt
set STAGE2_OUT=C:\Data\Pipeline\stage2_delimited.csv
set FINAL_OUT=C:\Data\Pipeline\final_clean.csv

echo [%time%] Starting pipeline...

REM Stage 1: EBCDIC to ASCII
TextPipe.exe /filter:"C:\Filters\ebcdic_to_ascii.fll" /input:"%SOURCE%" /output:"%STAGE1_OUT%" /silent /overwrite
if %ERRORLEVEL% NEQ 0 (echo Stage 1 FAILED & exit /b 1)
echo [%time%] Stage 1 complete: Encoding conversion

REM Stage 2: Fixed-width to CSV
TextPipe.exe /filter:"C:\Filters\fixed_to_csv.fll" /input:"%STAGE1_OUT%" /output:"%STAGE2_OUT%" /silent /overwrite
if %ERRORLEVEL% NEQ 0 (echo Stage 2 FAILED & exit /b 1)
echo [%time%] Stage 2 complete: Format conversion

REM Stage 3: Data cleansing
TextPipe.exe /filter:"C:\Filters\validate_and_clean.fll" /input:"%STAGE2_OUT%" /output:"%FINAL_OUT%" /silent /overwrite
if %ERRORLEVEL% NEQ 0 (echo Stage 3 FAILED & exit /b 1)
echo [%time%] Stage 3 complete: Validation and cleansing

REM Cleanup intermediate files
del "%STAGE1_OUT%" "%STAGE2_OUT%"

echo [%time%] Pipeline complete: %FINAL_OUT%

Scheduled Execution

Automate TextPipe batch processing by scheduling execution with Windows Task Scheduler. Configure daily, hourly, or event-triggered runs for unattended data processing.

Using schtasks.exe (Command Line)

REM Create a scheduled task to run TextPipe batch processing daily at 2:00 AM
schtasks /create /tn "DataMystic\TextPipe Daily Processing" /tr "\"C:\Scripts\textpipe_daily_batch.bat\"" /sc daily /st 02:00 /ru SYSTEM /rl HIGHEST

REM Run every 15 minutes during business hours (Mon-Fri, 8am-6pm)
schtasks /create /tn "DataMystic\TextPipe Feed Monitor" /tr "\"C:\Scripts\textpipe_feed_process.bat\"" /sc minute /mo 15 /st 08:00 /et 18:00 /d MON,TUE,WED,THU,FRI /ru SYSTEM

REM Trigger on a specific Windows Event (e.g., file drop via event log)
schtasks /create /tn "DataMystic\TextPipe Event Trigger" /tr "\"C:\Scripts\textpipe_on_event.bat\"" /sc onevent /ec Application /mo "*[System[EventID=1000]]" /ru SYSTEM /rl HIGHEST

REM View existing scheduled tasks
schtasks /query /tn "DataMystic\TextPipe Daily Processing" /v /fo LIST

REM Delete a scheduled task
schtasks /delete /tn "DataMystic\TextPipe Daily Processing" /f

Using PowerShell (Task Scheduler Cmdlets)

# Create a scheduled task for TextPipe batch processing using PowerShell
# Runs nightly at 2:00 AM with retry on failure

$action = New-ScheduledTaskAction `
    -Execute "C:\Program Files\DataMystic\TextPipe\TextPipe.exe" `
    -Argument '/filter:"C:\Filters\nightly_etl.fll" /input:"C:\Data\Incoming\*.csv" /output:"C:\Data\Processed\" /silent /overwrite' `
    -WorkingDirectory "C:\Data"

$trigger = New-ScheduledTaskTrigger -Daily -At "02:00"

$settings = New-ScheduledTaskSettingsSet `
    -StartWhenAvailable `
    -RestartCount 3 `
    -RestartInterval (New-TimeSpan -Minutes 5) `
    -ExecutionTimeLimit (New-TimeSpan -Hours 2) `
    -AllowStartIfOnBatteries `
    -DontStopIfGoingOnBatteries

$principal = New-ScheduledTaskPrincipal -UserId "SYSTEM" -RunLevel Highest

Register-ScheduledTask `
    -TaskName "DataMystic\TextPipe Nightly ETL" `
    -Action $action `
    -Trigger $trigger `
    -Settings $settings `
    -Principal $principal `
    -Description "Nightly TextPipe batch processing of incoming CSV data feeds"

# Verify the task was created
Get-ScheduledTask -TaskName "TextPipe Nightly ETL" -TaskPath "\DataMystic\" | Format-List

Next Steps