# utfconv

The following routines allow simple conversion between UTF-8, UTF-16, and UTF-32.
They are intended for use more at the line/string level rather than whole file.
Note that no special handling of BOM (Byte Order Mark, aka "ZERO WIDTH NO-BREAK SPACE" prefix characters) is performed by any of these routines.
My recommendation is that such should be handled at the read/write file level.
For ease of reference, the following byte order marks are in common use (copy and paste these definitions as needed):

    constant
        UTF8    = "\#EF\#BB\#BF",
        UTF16BE = "\#FE\#FF",
        UTF16LE = "\#FF\#FE",
        UTF32BE = "\#00\#00\#FE\#FF",
        UTF32LE = "\#FF\#FE\#00\#00",

I would hesitantly suggest the following, more legacy byte order marks be detected and an error message shown, as opposed to blindly rushing in to support them before finding out at some later date that some minor error in their handling has quietly led to severe data corruption. Besides, the Phix distribution does not contain any routines for converting any of these:

        UTF7      = "\#2B\#2F\#76", -- (38|39|2B|2F|38 2D)
        UTF1      = "\#F7\#64\#4C",
        UTFEBCDIC = "\#DD\#73\#66\#73",
        SCSU      = "\#0E\#FE\#FF",
        BOCU1     = "\#FB\#EE\#28",
        GB18030   = "\#84\#31\#95\#33"

For a complete listing of pretty much every character format ever invented, see http://www.iana.org/assignments/character-sets/character-sets.xhtml
Also note these routines have no comprehension of any difference between LE and BE encodings: that is down to how the calling application reads/writes or peeks/pokes.
Quite clearly by the time you pass a value/character/unicode point to these routines, it should be, well, a value, in the proper and expected endian-ness of the machine the program is running on, rather than (say) a sequence of (optionally) byte-swapped elements, which could only ever serve to make everything far harder than it needs to be.
The implementation of these routines can be found in builtins\utfconv.e (an autoinclude) and test\t62utf.exw has several tests, which should of course be extended if any glitches are found, and is obviously run as part of 'p -test'.

utf8_to_utf32 | - | convert a UTF-8 string to a UTF-32 sequence
utf32_to_utf8 | - | convert a UTF-32 sequence to a UTF-8 string
utf16_to_utf32 | - | convert a UTF-16 sequence to a UTF-32 sequence
utf32_to_utf16 | - | convert a UTF-32 sequence to a UTF-16 sequence

The routines utf16_to_utf8() and utf8_to_utf16() are simple nested wrappers of the above routines.
See also demo\rosetta\VisualiseTree.exw for the kind of recommended shenanigans needed to output Windows-1252 to a Windows console but proper UTF-8 to a Linux terminal or Browser tab.
The above routines are not compatible with Euphoria.
Under pwa/p2js the routines utf8_to_utf32() and utf32_to_utf8() are actually null-ops since JavaScript handles utf8 natively, and all the utf16 routines are unsupported. Note that in some cases (/specific browsers) JavaScript may yield surrogate pairs, and you may have to deal with that within an "if platform()=JS then" guard, but I am prepared to investigate further and improve matters should a genuine need arise.
You may of course pass basic latin ascii (#00..#7F) strings as if they were utf8, however if you have found encodings in ISO-8859, Windows-1252, or some other code page, for characters with cedilla, umlaut, etc, (ie any single-byte characters in the range #80..#FF), they will almost certainly cause problems. Use an editor which is utf8 compliant (such as Edita), rather than one that uses some legacy code page, and you should avoid such difficulties, or at least immediately see what needs to be changed. My commiserations if you have some database or similar chock full of such non-latin ascii characters from a legacy application, these routines are unlikely to help you. In such cases I would suggest performing the appropriate substitutions to the correct unicode points directly, there will be at most 128 of them, but it is up to you to find out what they are, and perhaps running that back through utf32_to_utf8. Alternatively, you can try using the standard iconv utility (which can be installed on Windows as part of MinGW), eg "iconv -f WINDOWS-1252 -t UTF-8 filename.txt"