Unicode Cheat Sheet

Posted : admin On 1/2/2022
  • Jun 02, 2017 EMu Unicode Cheat Sheet Author: Gerard Wood Created Date: 6/2/2017 11:15:15 AM.
  • This cheat sheet lists a series of XSS attacks that can be used to bypass certain XSS defensive filters. Please note that input filtering is an incomplete defense for XSS which these tests can be used to illustrate. Basic XSS Test Without Filter Evasion.

Contents

Initial Steps
I/O, Database
Stream I/O
BOM Values
Constants & Globals
Data Types
Platform String API
TCHAR String API
References

Here is a cheat sheet for converting Microsoft C or C++ source code to support Unicode.It does not attempt to explain much and presumes you are generally familiar with Microsoft'sapproach to Unicode. The goal is just to have a single place to look for names, correct spellings, etc.of relevant data types, functions, etc.
Also see: Cheat Sheet: Unicode-enabling Microsoft C/C++ Source Code in Chinese

Ionicons v2.0.0 Cheatsheet, 733 icons: Ionicons Homepage. 12 14 16 18 21 24 32 48 64 96. Classname: Selectable: Escaped HTML: CSS Content.

Initial Steps for Unicode-enabling Microsoft C/C++ Source

  • Define _UNICODE, undefine _MBCS if defined.
  • Convert literal strings to use L or _T
  • Convert string functions to use Wide or TCHAR versions.
  • Clarify string lengths in API as byte or character counts.For character-based display or printing (as opposed to GUI which is pixel-based)use column counts, not byte or character.
  • Replace character pointer arithmetic with GetNext style, ascharacters may consist of more than one Unicode code unit.
  • Watch buffer size and buffer overflows- changing encodingsmay require either larger buffers or limiting string lengths. Ifcharacter size changes from 1 byte to as many as 4 bytes, andstring length was formerly 20 characters and 20 bytes, eitherexpand the string buffer(s) from 20 to 80 bytes or limit thestring to 5 characters (and therefore 20 bytes).Note maximum buffer expansion may be constrained (for example to 65 KB).Reducing string length to a fixed number of characters may break existing applications.Limiting strings to a fixed byte length is dangerous. For example, allowing anystring that fits into 20 bytes. Simple operations such as uppercasinga string may cause it to grow and exceed the byte length.
  • Replace functions that accept or return arguments of a single character, with functions that use strings instead.(International) Operations on a single character may result in more than one code point being returned. For example,upper('ß')returns 'SS'.
  • Use wmain instead of main.The environment variable is then _wenviron instead of _environ.
    wmain( int argc, wchar_t *argv[ ], wchar_t *envp[ ] ).
  • MFC Unicode applications use wWinMain as the entry point.
    In the Output page of the Linker folder in the project's Property Pages dialog box,set the Entry Point symbol to wWinMainCRTStartup.
  • Consider fonts. Identify the fonts that will render each language or script used.

File I/O, Database, Transfer Protocol Considerations

  • Consider whether to read/write UTF-8 or UTF-16 in files, databases, and for data exchange.
  • Consider Endian-ness in UTF-16 files.
    Read/Write Big-Endian on networks. Use Big-Endian if you don't produce a BOM.
    Endian-ness of files will depend on the file format and/or the architectureof the source or target machine.
    When reading files encoded in UTF-16 or UTF-32,be prepared to swap-bytes to convert endian-ness.
    Also consider streams and transfer protocols and the encoding used in each.
  • Label files or protocols for data exchange with the correct character encoding.E.g. set HTTP, HTML, XML to UTF-8 or UTF-16.
  • Consider Unicode BOM (Byte Order Marker) and whetherit should be written with data. Remove it when reading data.
  • Consider encoding conversion of legacy data and files, import and export, transfer protocols.(MultiByteToWideChar,WideCharToMultiByte,mbtowc,wctomb,wctombs,mbstowcs )
  • Consider writing to the Clipboard-
    use CF_TEXT format and write native character encoding (ANSI) text, and
    use CF_UNICODETEXT format and write Unicode text.
  • Database applications should consider Data Type(NCHAR, NVARCHAR)and Schema Changes, Triggers, Stored Procedures, and Queries. Data Storagegrowth, Indexes and Performance.
    Note that the Unicode schema changes will have different impacts and concerns on different vendors' databases.If database portability is a requirement, the features and behaviors of each database need to be taken into account.
    (I know this item is seriously understated. To be expanded sometime in the future.)

Stream I/O

Streams are difficult in Microsoft C++. You may run into 3 types of problems:

  1. Unicode filenames are not supported.The workaround is to useFILE * _wfopen and if needed, use the FILE handle in subsequent stream I/O.
  2. Stream I/O will convert Unicode data from/to native (ANSI) code page on read/write,not UTF-8 or UTF-16.However the stream class can be modified to read/write UTF-8.You can implement a facet to convert between Unicode and UTF-8.
    codecvt <wchar_t, char_traits <wchar_t> >
  3. To read/write UTF-16 with stream I/O, use binary opens and binary I/O. To set binary I/O:
    Also see the Microsoft run-time library reference: 'Unicode Stream I/O in Text and Binary Modes'.

Note: There aren't TCHAR equivalents forcout/wcout, cin/wcin, etc.You may want to make your own preprocessor definition for 'tout',if you are compiling code both ways.

Internationalization, Advanced Unicode, Platform and Other Considerations

  • Consider using locale-based routines and further internationalization.
  • For Windows 95, 98 and ME, consider using theMicrosoft MSLU (Microsoft Layer for Unicode)
  • Consider string compares and sorting, Unicode Collation Algorithm
  • Consider Unicode Normalization
  • Consider Character Folding
  • Reconsider doing this on your own. Bring in an experiencedUnicode consultant,and deploy your existing resources on the tasks they do best.(Hey, an I18nGuy's gotta earn a living...)

Unicode BOM Encoding Values

Encoding FormBOM Encoding
UTF-8EF BB BF
UTF-16
(big-endian)
FE FF
UTF-16
(little-endian)
FF FE
UTF-16BE, UTF-32BE
(big-endian)
No BOM!
UTF-16LE, UTF-32LE
(little-endian)
No BOM!
UTF-32
(big-endian)
00 00 FE FF
UTF-32
(little-endian)
FF FE 00 00
SCSU
(compression)
0E FE FF

The Byte Order Marker (BOM) is Unicode character U+FEFF.(It can also represent a Zero Width No-break Space.)The code point U+FFFE is illegal in Unicode, and shouldnever appear in a Unicode character stream. Therefore the BOM can be used in the first characterof a file (or more generally a string), as an indicator of endian-ness. With UTF-16,if the first character is read as bytes FE FF then the texthas the same endian-ness as the machine reading it. If the character is read as bytesFF FE, thenthe endian-ness is reversed and all 16-bit words should be byte-swapped as they are read-in.In the same way, the BOM indicates the endian-ness of text encoded with UTF-32.

Note that not all files start with a BOM however. In fact, the Unicode Standard says that text thatdoes not begin with a BOM MUST be interpreted in big-endian form.

The character U+FEFF also serves as an encoding signaturefor the Unicode Encoding Forms.The tableshows the encoding of U+FEFF in each of the Unicode encoding forms.Note that by definition, text labeled as UTF-16BE, UTF-32BE, UTF-32LE or UTF-16LE shouldnot have a BOM. The endian-ness isindicated in the label.

For text that is compressed with the SCSU (Standard Compression Scheme for Unicode) algorithm,there is also a recommended signature.

Constant and Global Variables

ANSIWideTCHAR
EOFWEOF_TEOF
_environ_wenviron_tenviron
_pgmptr_wpgmptr_tpgmptr

Data Types

ANSIWideTCHAR
charwchar_t_TCHAR
_finddata_t_wfinddata_t_tfinddata_t
__finddata64_t__wfinddata64_t_tfinddata64_t
_finddatai64_t_wfinddatai64_t_tfinddatai64_t
intwint_t_TINT
signed charwchar_t_TSCHAR
unsigned charwchar_t_TUCHAR
charwchar_t_TXCHAR
L_T or _TEXT
LPSTR
(char *)
LPWSTR
(wchar_t *)
LPTSTR
(_TCHAR *)
LPCSTR
(const char *)
LPCWSTR
(const wchar_t *)
LPCTSTR
(const _TCHAR *)
LPOLESTR
(For OLE)
LPWSTRLPTSTR

Platform SDK String Functions

There are many Windows API that compile into ANSI or Wide forms,depending on whether the symbol UNICODE isdefined. Modules that operate on both ANSI and Wide characters, need tobe aware of this. Otherwise, using the Character Data Type-independent namerequires no changes, just compile with the symbol UNICODEdefined.

The following list is by no means all of the Character Data Type-dependent API, justsome character and string related ones. Look in WinNLS.h for some code pageand locale related API.

ANSIWideCharacter Data Type-
Independent Name
CharLowerACharLowerWCharLower
CharLowerBuffACharLowerBuffWCharLowerBuff
CharNextACharNextWCharNext
CharNextExACharNextExWCharNextEx
CharPrevACharPrevWCharPrev
CharPrevExACharPrevExWCharPrevEx
CharToOemACharToOemWCharToOem
CharToOemBuffACharToOemBuffWCharToOemBuff
CharUpperACharUpperWCharUpper
CharUpperBuffACharUpperBuffWCharUpperBuff
CompareStringACompareStringWCompareString
FoldStringAFoldStringWFoldString
GetStringTypeAGetStringTypeWGetStringType
GetStringTypeExAGetStringTypeExWGetStringTypeEx
IsCharAlphaAIsCharAlphaWIsCharAlpha
IsCharAlphaNumericAIsCharAlphaNumericWIsCharAlphaNumeric
IsCharLowerAIsCharLowerWIsCharLower
IsCharUpperAIsCharUpperWIsCharUpper
LoadStringALoadStringWLoadString
lstrcatAlstrcatWlstrcat
lstrcmpAlstrcmpWlstrcmp
lstrcmpiAlstrcmpiWlstrcmpi
lstrcpyAlstrcpyWlstrcpy
lstrcpynAlstrcpynWlstrcpyn
lstrlenAlstrlenWlstrlen
OemToCharAOemToCharWOemToChar
OemToCharBuffAOemToCharBuffWOemToCharBuff
wsprintfAwsprintfWwsprintf
wvsprintfAwvsprintfWwvsprintf

TCHAR String Functions

Functions sorted by ANSI name, for ease of converting to Unicode.

ANSIWideTCHAR
_access_waccess_taccess
_atoi64_wtoi64_tstoi64
_atoi64_wtoi64_ttoi64
_cgets_cgetwscgetts
_chdir_wchdir_tchdir
_chmod_wchmod_tchmod
_cprintf_cwprintf_tcprintf
_cputs_cputws_cputts
_creat_wcreat_tcreat
_cscanf_cwscanf_tcscanf
_ctime64_wctime64_tctime64
_execl_wexecl_texecl
_execle_wexecle_texecle
_execlp_wexeclp_texeclp
_execlpe_wexeclpe_texeclpe
_execv_wexecv_texecv
_execve_wexecve_texecve
_execvp_wexecvp_texecvp
_execvpe_wexecvpe_texecvpe
_fdopen_wfdopen_tfdopen
_fgetchar_fgetwchar_fgettchar
_findfirst_wfindfirst_tfindfirst
_findnext64_wfindnext64_tfindnext64
_findnext_wfindnext_tfindnext
_findnexti64_wfindnexti64_tfindnexti64
_fputchar_fputwchar_fputtchar
_fsopen_wfsopen_tfsopen
_fullpath_wfullpath_tfullpath
_getch_getwch_gettch
_getche_getwche_gettche
_getcwd_wgetcwd_tgetcwd
_getdcwd_wgetdcwd_tgetdcwd
_ltoa_ltow_ltot
_makepath_wmakepath_tmakepath
_mkdir_wmkdir_tmkdir
_mktemp_wmktemp_tmktemp
_open_wopen_topen
_popen_wpopen_tpopen
_putch_putwch_puttch
_putenv_wputenv_tputenv
_rmdir_wrmdir_trmdir
_scprintf_scwprintf_sctprintf
_searchenv_wsearchenv_tsearchenv
_snprintf_snwprintf_sntprintf
_snscanf_snwscanf_sntscanf
_sopen_wsopen_tsopen
_spawnl_wspawnl_tspawnl
_spawnle_wspawnle_tspawnle
_spawnlp_wspawnlp_tspawnlp
_spawnlpe_wspawnlpe_tspawnlpe
_spawnv_wspawnv_tspawnv
_spawnve_wspawnve_tspawnve
_spawnvp_wspawnvp_tspawnvp
_spawnvpe_wspawnvpe_tspawnvpe
_splitpath_wsplitpath_tsplitpath
_stat64_wstat64_tstat64
_stat_wstat_tstat
_stati64_wstati64_tstati64
_strdate_wstrdate_tstrdate
_strdec_wcsdec_tcsdec
_strdup_wcsdup_tcsdup
_stricmp_wcsicmp_tcsicmp
_stricoll_wcsicoll_tcsicoll
_strinc_wcsinc_tcsinc
_strlwr_wcslwr_tcslwr
_strncnt_wcsncnt_tcsnbcnt
_strncnt_wcsncnt_tcsnccnt
_strncnt_wcsncnt_tcsnccnt
_strncoll_wcsncoll_tcsnccoll
_strnextc_wcsnextc_tcsnextc
_strnicmp_wcsnicmp_tcsncicmp
_strnicmp_wcsnicmp_tcsnicmp
_strnicoll_wcsnicoll_tcsncicoll
_strnicoll_wcsnicoll_tcsnicoll
_strninc_wcsninc_tcsninc
_strnset_wcsnset_tcsncset
_strnset_wcsnset_tcsnset
_strrev_wcsrev_tcsrev
_strset_wcsset_tcsset
_strspnp_wcsspnp_tcsspnp
_strtime_wstrtime_tstrtime
_strtoi64_wcstoi64_tcstoi64
_strtoui64_wcstoui64_tcstoui64
_strupr_wcsupr_tcsupr
_tempnam_wtempnam_ttempnam
_ui64toa_ui64tow_ui64tot
_ultoa_ultow_ultot
_ungetch_ungetwch_ungettch
_unlink_wunlink_tunlink
_utime64_wutime64_tutime64
_utime_wutime_tutime
_vscprintf_vscwprintf_vsctprintf
_vsnprintf_vsnwprintf_vsntprintf
asctime_wasctime_tasctime
atof_wtof_tstof
atoi_wtoi_tstoi
atoi_wtoi_ttoi
atol_wtol_tstol
atol_wtol_ttol
character compareMaps to macro or inline function_tccmp
character copyMaps to macro or inline function_tccpy
character lengthMaps to macro or inline function_tclen
ctime_wctime_tctime
fgetcfgetwc_fgettc
fgetsfgetws_fgetts
fopen_wfopen_tfopen
fprintffwprintf_ftprintf
fputcfputwc_fputtc
fputsfputws_fputts
freopen_wfreopen_tfreopen
fscanffwscanf_ftscanf
getcgetwc_gettc
getchargetwchar_gettchar
getenv_wgetenv_tgetenv
getsgetws_getts
isalnumiswalnum_istalnum
isalphaiswalpha_istalpha
isasciiiswascii_istascii
iscntrliswcntrl_istcntrl
isdigitiswdigit_istdigit
isgraphiswgraph_istgraph
islead (Always FALSE)(Always FALSE)_istlead
isleadbyte (Always FALSE)isleadbyte (Always FALSE)_istleadbyte
islegal (Always TRUE)(Always TRUE)_istlegal
isloweriswlower_istlower
isprintiswprint_istprint
ispunctiswpunct_istpunct
isspaceiswspace_istspace
isupperiswupper_istupper
isxdigitiswxdigit_istxdigit
mainwmain_tmain
perror_wperror_tperror
printfwprintf_tprintf
putcputwc_puttc
putcharputwchar_puttchar
puts_putws_putts
remove_wremove_tremove
rename_wrename_trename
scanfwscanf_tscanf
setlocale_wsetlocale_tsetlocale
sprintfswprintf_stprintf
sscanfswscanf_stscanf
strcatwcscat_tcscat
strchrwcschr_tcschr
strcmpwcscmp_tcscmp
strcollwcscoll_tcscoll
strcpywcscpy_tcscpy
strcspnwcscspn_tcscspn
strerror_wcserror_tcserror
strftimewcsftime_tcsftime
strlenwcslen_tcsclen
strlenwcslen_tcslen
strncatwcsncat_tcsncat
strncatwcsncat_tcsnccat
strncmpwcsncmp_tcsnccmp
strncmpwcsncmp_tcsncmp
strncpywcsncpy_tcsnccpy
strncpywcsncpy_tcsncpy
strpbrkwcspbrk_tcspbrk
strrchrwcsrchr_tcsrchr
strspnwcsspn_tcsspn
strstrwcsstr_tcsstr
strtodwcstod_tcstod
strtokwcstok_tcstok
strtolwcstol_tcstol
strtoulwcstoul_tcstoul
strxfrmwcsxfrm_tcsxfrm
system_wsystem_tsystem
tmpnam_wtmpnam_ttmpnam
tolowertowlower_totlower
touppertowupper_totupper
ungetcungetwc_ungettc
vfprintfvfwprintf_vftprintf
vprintfvwprintf_vtprintf
vsprintfvswprintf_vstprintf
WinMainwWinMain_tWinMain

References

  • 'MSLU: Develop Unicode Applications for Windows 9x Platforms with the Microsoft Layer for Unicode'Michael Kaplan, Cathy Wissink
  • 'Design a Single Unicode App that Runs on Both Windows 98 and Windows 2000'F. Avery Bishop, April 1999.
  • 'Supporting Multilanguage Text Layout and Complex Scripts with Windows NT 5.0'F. Avery Bishop, David C Brown and Davis M Meltzer, November 1998.
  • Newsgroup: MSDN MSLU
  • Newsgroup: MSDN Internationalization

Copyright © 2003-2010 Tex Texin. All rights reserved.
Top of page

Unicode (UTF-8) with PHP 7.4, MySQL 5.7 and HTML5 Cheat Sheet


1. Conversion

1.1. How to transform file encoding

Example with PHP files on Linux:


1.2. How to transform character encoding in MySQL databases

Procedure [1] (use the INFORMATION_SCHEMA database to build a script automatically):

  • create a temporary, identical structure in a new database,
  • copy all data to that structure,
  • drop the initial structure and
  • recreate it with the new character encoding:
  • Check the maximum length of columns and index keys [2]
  • Copy all data from the temporary structure to the new structure, converting all texts the new encoding, and finally
  • Drop the temporary structure.

2. Configuration

2.1. HTTP and HTML

In php.ini [1]:


or in httpd.conf or .htaccess [5]:


or in the PHP code [5]:


Additionally, put this in you HTML <head> block:


2.2. PHP

In php.ini [1.1]:


or in httpd.conf or .htaccess:


or in the PHP code [3] [1]:


2.3. Verifications [4]

Run this small PHP script:


3. MySQL Code

3.1. MySQL

Right after each connection, call [5] [2]:


3.2. Ordering in MySQL

Ordering in MySQL depends on the collation you choose. Detailed information about this subject may be found in the documentation on MySQL.com [2]. Look especially at the Unicode Character Sets section: https://dev.mysql.com/doc/refm[...]et-unicode-sets.html.

3.3. Stored Procedures and Functions

Old function:


New function:


4. PHP Code

4.1. Multibyte string functions [1]


Replace With Notes
str_split()mb_str_split()
strlen()mb_strlen()
strlen()
mb_strwidth()
// How many characters
// How many bytes
// Monotype characters
substr()mb_substr()
strstr()
stristr()
mb_strstr()
mb_stristr()
strrchr()mb_strrchr()[6]
strpos()
stripos()
strrpos()
strripos()
mb_strpos()
mb_stripos()
mb_strrpos()
mb_strripos()
strtolower()
strtoupper()
mb_strtolower()
mb_strtoupper()
substr_count()mb_substr_count()[7]
ord()mb_ord()

4.2. String access by character [1]

Search all use of curly or square brackets to extract single characters of strings:


Regular expressions to find them:


Replace


with


Replace


with

4.3. UTF-8-safe functions [8]


addslashes()
bin2hex()
explode() [4]
implode()
nl2br()
stripslashes()
strip_tags()
str_repeat()
str_replace() [4]

4.4. Escapement functions

The functions htmlentities()[9] and htmlspecialchars() both have a third parameter which corresponds to the character set used during conversion. Unlike with multibyte functions( mb_*() ), this 3rd parameter is mandatory if not 'UTF-8', no matter what the internal encoding is!


The functions urlencode() and rawurlencode() do not have any character encoding parameter. The safest solution is to put your UTF-8 strings in session variables instead of URL arguments.

4.5. Comparing strings and sorting arrays

Use the Collator class:
https://www.php.net/manual/en/class.collator.php

Sheet

4.6. SimpleXML

SimpleXML uses UTF-8 internally and converts all XML content to UTF-8 [1.2], so usually nothing needs to be done.

4.7. PRCE functions [1]

Search all PRCE function calls ( preg_* ) and append the /u pattern modifier [10]

4.8. Storable representation of variables

The serialize() and unserialize() functions can be used transparently. However, be careful when reading or writing serialized UTF-8 strings with other languages than PHP [4].


4.9. String functions that are problematic and for which there is no built-in replacement function


Replace With a function from the PHP UTF8 Library [11] [3] Comment
count_chars($string, $mode)This function doesn't work if primary parameter is UTF-8. Write your own implementation.
ord($chr)[12]utf8_ord($chr)mb_ord() available with PHP 7.2
sprintf()The x and X type specifiers could be an issue, according to [4].
str_ireplace($search, $replace,$subject [, &$count])utf8_ireplace($search, $replace,$subject [, &$count])Alternatively, write your own implementation using preg_replace().
str_pad($input, $length, $padStr, $type)utf8_str_pad($input, $length, $padStr, $type)
str_split($str, $split_len)utf8_str_split($str, $split_len)mb_str_split() available with PHP 7.4
strcasecmp($str1, $str2) Write your own implementation using collator_compare() and mb_strtolower()
strncmp($str1, $str2, $len) Cut the two strings at the specified length, and use collator_compare()
strncasecmp($str1, $str2, $len) Write your own implementation using your replacement of strncmp() and mb_strtolower()
strspn($str1, $str2[, $start[, $len]])
strcspn($str1, $str2[, $start[, $len]])
utf8_strspn($str1, $str2[, $start[, $len]])
utf8_strcspn($str1, $str2[, $start[, $len]])
strrev($string)utf8_strrev($string)
strtr()This function doesn't work if any parameter is UTF-8. Write your own implementation.
substr_replace()utf8_substr_replace()
trim($str, $charlist)
ltrim($str, $charlist)
rtrim($str, $charlist)
utf8_trim($str, $charlist)
utf8_ltrim($str, $charlist)
utf8_rtrim($str, $charlist)
The original functions trim(), ltrim() and rtrim() are UTF-8-safe as long as the 2nd parameter is not used[4].
ucfirst($str)
ucwords($str)
utf8_ucfirst($str)
utf8_ucwords($str)
wordwrap()Write your own implementation

5. Sources

[1] PHP.net documentation
[1.1] PHP.net, Multibyte String Runtime Configuration, https://www.php.net/manual/en/[...]ng.configuration.php
[1.2] A comment about SimpleXML on PHP.net: https://www.php.net/manual/en/ref.simplexml.php#79258
[2] MySQL.com documentation
[3] Harry Fuecks, PHP UTF-8 library, https://sourceforge.net/projects/phputf8
[4] Web Application Component Toolkit, Handling UTF-8 with PHP, http://www.phpwact.org/php/i18n/utf-8
[5] W3C, Setting the HTTP charset parameter, https://www.w3.org/International/O-HTTP-charset.php

Unicode Cheat Sheet Pdf


Unicode Emoji Cheat Sheet

Footnotes:

[1]
Trying to change the character encoding of TEXT, CHAR and VARCHAR fields directly with an ALTER TABLE will corrupt existing data.
[2]
https://dev.mysql.com/doc/refm[...]code-conversion.html
[3]
Some php.ini directives cannot be modified in the PHP code.
[4]
Source: utf8.php in PHP UTF-8 library [3]
[5]
Once this is done, PHP sees MySQL databases as if each TEXT, CHAR or VARCHAR field were encoded in UTF-8, no matter what the actual encoding is. Thus, there is no need to prepare the encoding of query parameters or to convert the results in PHP.
[6]
Note that the mb_strrchr() functions has one additional argument, that may be ignored since we just want to adapt existing function calls. Note also that there is a mb_strrichr() function, which has no equivalent in standard PHP functions.
[7]
Be careful because the 3rd and 4th arguments of substr_count() no longer exist with mb_substr_count(). You can use mb_substr() to circumvent this limitation.
[8]
The strcmp() function is UTF-8 safe as well. However, to perform a locale-aware comparison, use Collator::compare instead: https://www.php.net/manual/en/collator.compare.php
[9]
The function htmlentities() converts latins characters only [todo source]. Moreover, according to Handling UTF-8 with PHP [4]: “Using [htmlentities] on a UTF-8 string with the wrong charset would, very likely, result in corruption / junk output”, and “when using UTF-8, you don’t need entities”.
[10]
However, there may still have problems, as explained in Handling UTF-8 with PHP [4].
[11]
Version 0.5
[12]
Underlined parameters fail if they are UTF-8-encoded

Second Life Unicode Cheat Sheet