DataMystic

TextPipe is a market leading self-service data preparation solution for data extraction. Extract, transform and load (ETL) data from anywhere - giving super powers to your data-wranglers such as ETL developers, programmers, data scientists and data analytics professionals. Connect to multiple data s...

Thanks David - this has been fixed for 11.8

Hi there, TextPipe isn't designed to solve the problem this way. In principle, consider that TextPipe processes your entire file with the first filter, before passing it into the next filter. So the cause-effect you're after here will only work for the last occurrence. Also, regex matches are built ...

Thanks David, I suspect we will have to redo these using UTF-8 as the input

Thanks for the suggestion David!

Hi David, please try reinstalling. This fixed the issue for me.

Hi David,

This can be easily achieved by putting the Add file header filter AFTER a merge filter.

Does PCRE need to care if a character is two bytes or not? Can't it just match the sequence? Isn't it an "all or nothing" literal comparison?

I am not suggested everything gets squished to 2 bytes.

Please see sample filter attached.

Let's take Unicode to UTF-8 Byte Codes. If we assume the Unicode is in UTF-8 form already (if not, just use a filter to convert it), then all you need to do is 1. Restrict to the text (a line or column) 2. Hex dump 3. Use a pattern match to match 2 hex characters at a time, and add \x{ } around it. ...

Hi David, Can't this already be done? Just use a restriction to match each line in turn, or the replace column: For the Unicode to Perl Escape Codes 1. Hex dump 2. Add left margin of \x{ 3. Add right margin of } For the Unicode to UTF-8 Byte Codes 1. Hex dump 2. Search for 2 hex bytes, and add a \x ...

You're welcome David - and Happy New Year too!

Hi David, this sounds like a filter list that could be prepared, rather than an internal filter.

If you design one we're happy to include it in the TextPipe distribution.

The help notes refer to:

- Updated "Convert HTML/XML Entities to Text" to fully support HTML5, with
~2200 entities now supported.

Thanks David, I am happy to reconsider if there is a broader base of use-cases.