CharBusters

10 Unicode myths

������ ����, @���

CharBusters

10 Unicode myths

Tomasz Nurkiewicz, @tnurkiewicz

काचं शक्नोम्यत्तुम् । नोपहिनस्ति माम् ॥
Μπορῶ νὰ φάω σπασμένα γυαλιὰ χωρὶς νὰ πάθω τίποτα 🇬🇷
ᛁᚳ᛫ᛗᚨᚷ᛫ᚷᛚᚨᛋ᛫ᛖᚩᛏᚪᚾ᛫ᚩᚾᛞ᛫ᚻᛁᛏ᛫ᚾᛖ᛫ᚻᛖᚪᚱᛗᛁᚪᚧ᛫ᛗᛖ᛬
⠊⠀⠉⠁⠝⠀⠑⠁⠞⠀⠛⠇⠁⠎⠎⠀⠁⠝⠙⠀⠊⠞⠀⠙⠕⠑⠎⠝⠞⠀⠓⠥⠗⠞⠀⠍⠑
Я можу їсти скло, і воно мені не зашкодить. 🇺🇦
მინას ვჭამ და არა მტკივა. 🇬🇪
Կրնամ ապակի ուտել և ինծի անհանգիստ չըներ։ 🇦🇲
నేను గాజు తినగలను మరియు అలా చేసినా నాకు ఏమి ఇబ్బంది లేదు 🇮🇳
איך קען עסן גלאָז און עס טוט מיר נישט װײ
ᐊᓕᒍᖅ ᓂᕆᔭᕌᖓᒃᑯ ᓱᕋᙱᑦᑐᓐᓇᖅᑐᖓ
ᠪᠢ ᠰᠢᠯᠢ ᠢᠳᠡᠶᠦ ᠴᠢᠳᠠᠨᠠ ᠂ ᠨᠠᠳᠤᠷ ᠬᠣᠤᠷᠠᠳᠠᠢ ᠪᠢᠰᠢ 🇲🇳
我能吞下玻璃而不伤身体。🇨🇳
أنا قادر على أكل الزجاج و هذا لا يؤلمني.

0. I don’t need to worry about Unicode
1. 1 character = 1 byte
2. 1 character = 1 char
3. Java is UTF-16
4. Unicode is unambiguous
5. 1 character ≤ 1 int
6. 1 character ≤ 2 ints
7. String.length() is useful
8. Whitespace is straightforward
9. Upper case is simple
10. Unicode is harmless

I don’t need to worry about Unicode

Myth 0

History of IT

(or how we estimate badly)

Parkinson's Law

Work expands so as to fill the time available for its completion

Hofstadter's Law

It always takes longer than you expect, even when you take into account Hofstadter's Law

IPv4

1978

The number of connected devices in 2021 is set to hit 46 billion

640 KiB

Bill Gates IBM, 1981

Y2K38

512K problem

GPS 2019

ASCII

1963

1 character = 1 byte

Myth 1

Let’s talk about "ą"

ą

🇵🇱

🇨🇿

🇫🇷

🇷🇺

🇩🇪 🇬🇷 🇪🇸

Unicode 1.0

modern [characters], whose number is undoubtedly far below 2¹⁴ = 16 384

1988

Unicode code points

a = U+0061 =    97
å = U+00E5 =   229
ą = U+0105 =   261
鑫= U+946B = 37995

1 character = 1 char

Myth 2

[...] undoubtedly far below 2¹⁴ = 16 384

1988

🇨🇳

讓我來! 让我来!

CJK

Almost 93 thousand characters in Unicode 14.0

𝄞

U+1D11E (119 070)

String

					"𝄞".codePointAt(0)
					//119070
				

Unicode 2.0

1996

Correct Java type for one character is...?

1. byte
2. char
3. int
4. String

String

					codePointAt(int)         : int
					codePoints()             : IntStream
					codePointCount(int, int) : int
				

Unicode

vs

UTF-*

a

U+0061

ą

U+0105

鑫

U+946B

𝄞

U+1D11E

Surrogate pairs

𝄞

• 1 code point
• 2 code units (a pair)

🤔

"𝄞".length() == 2

C#/Java/JS:

					"(𝄞)".Substring(0, 2)
					
					(?
				

var life = "🏭" + "🏖";

					StringBuilder rev = new StringBuilder();
					for (int i = life.length() - 1; i >= 0; i--)
						rev.append(life.charAt(i));
				

var life = "🏭🏖";

"?🏭?"

C#

					char[] array = "🇬🇧".ToCharArray() ;
					Array.Reverse(array) ;
					Console.WriteLine(new string(array));
				

					????
				

Java is UTF-16

Myth 3

Java 8

					private final char value[];
				

Java 9+

					private final byte[] value;
				

Java 9+

					public int indexOf(int ch, int fromIndex) {
						return isLatin1() 
						  ? StringLatin1.indexOf(value, ch, fromIndex) 
						  : StringUTF16.indexOf(value, ch, fromIndex);
					}
				

String.getBytes()

UTF-8 by default

Unicode is unambiguous

Myth 4

"ą".equals("ą")

Little tail

Normalizer.normalize("ą", Form.NFKC)
				

1 character ≤ 1 int

Myth 5

Let’s talk about emoji

https://www.imdb.com/title/tt4877122/

https://www.dailymail.co.uk/femail/article-4794964/World-s-emoji-translator-ridiculed-Twitter.html

https://twitter.com/sundarpichai/status/924487551372615680

http://curlicuecal.tumblr.com/post/175362924100/an-entomologist-rates-ant-emojis

🇵🇱

🇵+🇱

🇵🇱

• 2 code points
• 4 code units

Ruby

					$ irb

					2.7.2 :001 > "abc".reverse!
					 => "cba"
				

					2.7.2 :002 > "🇬🇧".reverse!
					 => "🇧🇬"
				

🇬 🇧 vs. 🇧 🇬

C#

					char[] array = "🇬🇧".ToCharArray() ;
					Array.Reverse(array) ;
					Console.WriteLine(new string(array));
				

					????
				

👧🏽

🏻 🏼 🏽 🏾 🏿

1 character ≤ 2 ints

Myth 6

👩🏾‍🚀

👨‍👩‍👧‍👦

String.length() is useful

Myth 7

public int length()

Returns the length [...] equal to the number of Unicode code units in the string.

Code unit 😱

The minimal bit combination that can represent a unit of encoded text.

"a".length()	  // 1
"ą".length()	  // 1
"ą".length()	  // 2
"𝄞".length()	  // 2
				

"👰".length()    // 2
"🇵🇱".length()    // 4
"👩🏾‍🚀".length()    // 7
"👨‍👩‍👧‍👦".length()  	// 11
"🏴󠁧󠁢󠁳󠁣󠁴󠁿".length()    // 14
				

"T̢̗̮͉͈̠̣͆͆̎͐̌͒͢ȍ̵͑̾͒͂͛̄̔͢҉̡̦͙͎̱̹͍͎͖̪̮̙̪͔̺͕̞̰̤̯m̍ͩ̓͋ͫ̑҉̵̷͓̦̩̭̗̩̫̺e̵̦̫̭̫̬͉̞̪̹̓̆̈́͊̂̃̀͡ǩ̸̴̢̛̫̦̬̪̘̱̖̼̺͕͇͕̞͓̮̭̯ͣ̌̂̏ͨͤͬ͛̏̋̉̀".length()

119

What is character?

• Code point
• Code unit
• Grapheme cluster
• Glyph

UTF

UTF is an [...] mapping from every Unicode code point [...] to a unique byte sequence

Whitespace is straightforward

Myth 8

How many different types of whitespaces there are?

All of them:

Space, tab, enter...

					IntStream.rangeClosed(0, 0x10FFFF)
							 .filter(Character::isDefined)
							 .count();
				

					IntStream.rangeClosed(0, 0x10FFFF)
							 .filter(Character::isWhitespace)
							 .count();
				

Character.isWhitespace()

25 characters

String.trim()

32 characters

Pattern.compile("\\s")

6 characters

Upper case is simple

Myth 9

🇹🇷

🇹+🇷

"i".toUpperCase(tr_TR)

i → İ

1. jira.atlassian.com/browse/CONFSERVER-7167 (Confluence)
2. jira.atlassian.com/browse/CONFSERVER-7168 (Confluence)
3. blogs.msdn.microsoft.com/anutthara/2005/12/05/avoiding-the-turkish-i-issue/ (.NET)
4. bz.apache.org/bugzilla/show_bug.cgi?id=38787 (BCEL)
5. bugzilla.redhat.com/show_bug.cgi?id=1408950 (Fedora)

1. JDK-6220064 - REGRESSION: SSL connections fail with Turkish input locale
2. JDK-6240755 - Swing rendering error for Turkish locale on XP look and feel
3. JDK-6240963 - XSLT transforms broken in Turkish locale in JDK 1.5.0+
4. JDK-6341798 - XMLDecoder fails when using Turkish Locale
5. JDK-8023943 - Method description fix for String.toLower/UpperCase() methods
6. JDK-4624534 - JarEntry.getCertificates() returns null on Turkish locale (tr_TR)
7. JDK-6867345 - Turkish regional options cause NPE in sun.security.x509.AlgorithmId.algOID
8. JDK-6972386 - issues with String.toLowerCase/toUpperCase
9. JDK-6353473 - REGRESSION: NameGenerator.captialize doesn't work properly with Turkish locale
10. JDK-8020037 - String.toLowerCase incorrectly increases length, if string contains \u0130 char

Unicode is harmless

Myth 10

https://twitter.com/JenMsft/status/1012586276678086656

• 'Trojan Source' attack method can hide bugs into open-source code
• Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

effective. Power لُلُصّبُلُلصّبُررً ॣ ॣh ॣ ॣ 冗

జ్ఞా

<⚫️>👈🏻

‏‎

Google search crashes when you ask "How many emojis on Apple"

sıkışınca ≠ sikişince

Zaten sen sıkışınca konuyu değiştiriyorsun.

Zaten sen sikişınce konuyu değiştiriyorsun.

sıkışınca ≠ sikişince

Anyhow, whenever you can't answer an argument, you change the subject.

Anyhow, whenever they are f***ing you, you change the subject.

Romanization

Zażółć gęślą jaźń

👇

Zazolc gesla jazn

Pangram

The quick brown fox jumps over the lazy dog

Jeżu klątw, spłódź Finom część gry hańb

Conclusions

Ińtërnâtiônàlizætión☃⛄️

Which encoding is the best?

UTF-8 is used by 97.7% of all the websites whose character encoding we know

References

Thank you!

nurkiewicz.com/slides/charbusters