Save as txt : How to detect if Unicode encoding is required?

macropod · #1 11-20-2018, 01:56 PM

VBorNotVB: When posting code, please use the code tags, indicated by the # button on the posting menu. Without them, your code loses much of whatever structure it had. See your edited post.

Looping through every character in a document is slow and unnecessary - excruciatingly so when every character is unnecessarily selected. In any event, testing whether characters fall in ASCII range 0-127 says nothing about whether they're Unicode; they might also be characters that fall in ASCII range 128-255!

Instead, try something based on:

Code:

Sub Demo()
Application.ScreenUpdating = False
Dim i As Long
With ActiveDocument
  With .Range.Find
    .ClearFormatting
    .Replacement.ClearFormatting
    .Text = "[^1-^255]"
    .Replacement.Text = ""
    .Forward = True
    .Wrap = wdFindContinue
    .Format = False
    .MatchWildcards = True
    .Execute Replace:=wdReplaceAll
  End With
  i = Len(.Range.Text) - .InlineShapes.Count
  .Undo
End With
If i > 1 Then
  MsgBox "Document contains Unicode characters."
Else
  MsgBox "Document contains only ASCII characters."
End If
Application.ScreenUpdating = True
End Sub

VBorNotVB · #2 11-21-2018, 07:32 AM

Paul,

I created a blank document, added the paragraph "This is a test.", and ran your routine. It reported it as having Unicode characters!

I know my routine is slow, worse yet using wdDialogInsertSymbol makes it even slower!
But I believe it's the only surefire way to catch Unicode characters. Using "(normal text)" as the litmus test forces Word to do the work for us, otherwise special characters like Wingdings can slip through if their character code point falls below 255.

As for testing for ASCII characters 0-127, the first 128 characters of Unicode are ACSII, but represented in long. Thus the use of AscW() which tests for wide characters and guards against reading character code points as negative values!

macropod · #3 11-21-2018, 03:14 PM

Quote:

Originally Posted by VBorNotVB

I created a blank document, added the paragraph "This is a test.", and ran your routine. It reported it as having Unicode characters!

I forgot that a paragraph break would remain in the document. The 'If i > 0 Then' should have been 'If i > 1 Then'. Fixed.

Quote:

Originally Posted by VBorNotVB

I know my routine is slow, worse yet using wdDialogInsertSymbol makes it even slower!
But I believe it's the only surefire way to catch Unicode characters.

My approach clearly demonstrates that is not so.

Quote:

Originally Posted by VBorNotVB

As for testing for ASCII characters 0-127, the first 128 characters of Unicode are ACSII, but represented in long.

You still seem to be missing the point that ASCII characters 128-255 are no different in that regard.

VBorNotVB · #4 11-22-2018, 11:38 AM

Your code works really well and it's fast too. Well done!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Macro help - converting proprietary encoding to unicode, keep formatting	kmawhood	Word VBA	3	04-29-2016 04:06 PM
Does 'Save as Unicode text' equal UTF-8	Chayes	Excel	1	07-20-2012 03:07 AM
Custom Dictionary & Unicode encoding?	markus staubmann	Word	3	03-28-2012 05:23 AM
Save Word doc in unicode html (utf-8)	rybrns	Word	5	09-26-2011 02:18 PM
Unicode Encoding Type	Rose44	Excel	2	08-09-2009 09:05 PM