NUL character

[Nemo271]

When I open file *.csv (in Notepad3 v5.19.630.2381) between each character I have NUL character.

On version Notepad3 5.19.108.1602 (and old) was correct

[RaiKoHoff]

@Nemo271 Please attach an example file (.zip packed) to this issue.

[Nemo271]

This is example file (.zip packed).
test.zip

[hpwamr]

Hello @RaiKoHoff , here the result of a first analyze:

"test.cvs' (the file is "UCS-2 LE")

• Notepad3 v5.19.719.2399 BETA "Out-of-the-Box" --> showed in statusbar: UTF-8 👎
• Notepad3 v5.19.501.1685 RC "Out-of-the-Box" --> showed in statusbar: UTF-8 👎
• Notepad3 v5.19.108.1602 "Out-of-the-Box" --> showed in statusbar: UTF-16 LE 👍
• Notepad++ 7.7.1 "Out-of-the-Box" --> showed in statusbar: UCS-2 Little Endian 👍

Details:

• Notepad3 v5.19.719.2399 BETA "OTB", UCHARDET in Debug Mode: 1
  UCD=ASCII 100% (reliable), OS (CP-1252), showed in statusbar: ANSI (CP-1252) 👎
• Notepad3 v5.19.501.1685 RC "OTB", UCHARDET in Debug Mode: 1
  UCD=ASCII 100% (reliable), CED=UTF-16LE, Conf= 20% (NOT reliable),
  OS (CP-1252) showed in statusbar: ANSI (CP-1252) 👎

My conclusion:

• The file is coded in an older obsolete "UCS-2 LE" (Unicode UTF-16 LE without BOM).
• Our current "UCHARDET fails to detect "UCS-2 LE" 🤔

From Unicode Technical Site:

• Glossary: UCS-2. ISO/IEC 10646 encoding form: Universal Character Set coded in 2 octets, limited to the Basic Multilingual Plane. (See Appendix C, Relationship to ISO/IEC 10646.)
• UTF-16 FAQ: Q: What is the difference between UCS-2 and UTF-16?

[RaiKoHoff]

Oh, the first case, where deprecated CED has been better 🤔.
Ed.: The issue has been discussed here:

• Issue: UTF-16 without BOM detection chardet/chardet#105
• PR: UTF detection when missing Byte Order Mark chardet/chardet#109
  ( unfortunately this is the Python version of (U)CHARDET )

[hpwamr]

Hello @Nemo271 ,

Feel free to test the Beta version "Notepad3Portable_5.19.721.2402_BETA" or higher.
See "Notepad3 beta-channel access #1129" or download it from Notepad3Portable_5.19.721.2402_BETA.

[Nemo271]

Hello!
On this beta version all is OK , thx.
Now I will wait for setup version for replacing the default Windows Notepad.

[hpwamr]

Hello @Nemo271 , Just for your info:
You can use the installer version Notepad3_5.19.630.2381_Setup version to configure the default Windows Notepad override.
Then, following the instructions of #1105, you will be able to pass to the last version "BETA". 🤔

[hpwamr]

On this beta version all is OK , thx.

As far as I am concerned, this issue may be closed....

[Nemo271]

OK.
Thx for instructions of #1105

[hpwamr]

Hello @Nemo271 ,
As far as I'm concerned, I think you (as requester) can close this issue...

[Nemo271]

Is ok now.