The History and Future of Character Encoding

字符编码的历史与未来

Every piece of text you have ever read on a screen is, somewhere, a sequence of numbers. The mapping from abstract symbols to those numbers is what we call a character encoding. For decades this mapping was a mess: dozens of incompatible standards, mysterious question marks when files crossed borders, and entire families of bugs known as mojibake. Today, one encoding — UTF-8 — carries more than 98 percent of the web, and the story of how we got there is one of the most quietly influential in computing.

你在屏幕上读过的每一段文字，归根结底都是一串数字。把抽象符号映射到这些数字的规则，就是我们所说的"字符编码"。过去几十年里，这种映射极其混乱：互不兼容的标准、文件跨边界时莫名其妙的问号、以及被称为"乱码"（mojibake）的一大类 Bug。如今，一种编码 —— UTF-8 —— 承载了 Web 上超过 98% 的文本。我们是如何走到这一步的，是计算史上影响最深却又最低调的故事之一。

ASCII and the seven-bit bargain

ASCII 与七位约定

In 1963, the American Standards Association published ASCII, the American Standard Code for Information Interchange. It mapped 128 characters into the values 0 through 127, fitting comfortably in seven bits. The first 32 codes were control characters: line feed, carriage return, tab, bell. The rest covered the English alphabet in upper and lower case, digits, common punctuation, and a space.

1963 年，美国标准协会发布了 ASCII（美国信息交换标准代码）。它把 128 个字符映射到 0 到 127 之间的数值，恰好能用 7 个二进制位表示。前 32 个码位是控制字符：换行、回车、制表符、响铃等等；其余部分覆盖了英文大小写字母、数字、常用标点以及一个空格。

Why seven bits and not eight? The honest answer is cost. In 1963 memory was measured in bytes and the eighth bit was often a parity bit for telegraph error detection. The early hardware simply could not afford to waste that bit. The result was an English-centric alphabet that could not represent accented characters at all. The eighth bit would later be reclaimed, but the legacy of fitting 95 printable symbols into 128 slots is still with us.

为什么是 7 位而不是 8 位？真实的答案是成本。1963 年时内存以字节计费，而第 8 位常常被用作电报的奇偶校验。当时的硬件负担不起"浪费"这一位。早期的结果就是一套以英文为中心的字母表，完全无法表示带音调的字符。后来第 8 位被重新使用，但"把 95 个可打印符号塞进 128 个槽位"这一历史遗产至今仍影响着我们。

Latin-1, Windows-1252, and the 256-slot problem

Latin-1、Windows-1252 和 256 槽位难题

The obvious next step was to use the eighth bit and double the range to 256 characters. ISO-8859-1, also known as Latin-1, did exactly that in 1987, filling the upper half with characters needed for Western European languages: é, ñ, ü, ç, and friends. Microsoft, however, took a slightly different route: Windows-1252 reassigned a handful of the rarely-used control slots in the 0x80–0x9F range to printable characters like the euro sign and curly quotes.

顺理成章的下一步是利用第 8 位，把范围扩到 256 个字符。1987 年发布的 ISO-8859-1（也叫 Latin-1）就这么做了，把高 128 位填入西欧语言所需的字符：é、ñ、ü、ç 等。但微软走了略有不同的路线：Windows-1252 重新分配了 0x80–0x9F 中一些几乎没人使用的控制字符槽位，改为欧元符号和弯引号等可打印字符。

This sounds reasonable until you realize that 256 slots cannot hold the Chinese, Japanese, Korean, Arabic, Hebrew, Devanagari, Cyrillic, Greek, and Thai writing systems all at once. Vendors responded with a Babel of regional encodings: GB2312 and GBK for simplified Chinese, Big5 for traditional Chinese, Shift-JIS and EUC-JP for Japanese, KOI8-R for Russian. A text file from Moscow opened in Tokyo would render as noise. This is the era mojibake was born in.

听起来很合理，但只要意识到 256 个槽位根本塞不下中日韩、阿拉伯、希伯来、天城文、西里尔、希腊、泰文等等所有文字，问题就暴露了。各厂商于是搞出了一套"巴别塔"式的地区编码：简体用 GB2312 和 GBK，繁体用 Big5，日文用 Shift-JIS 和 EUC-JP，俄文用 KOI8-R。来自莫斯科的文本文件，在东京打开就是一堆乱码。"乱码"这个词，就诞生在这个时代。

Unicode's mission and the code-point abstraction

Unicode 的使命与码点抽象

In the late 1980s, engineers from Xerox and Apple — including Joe Becker, Lee Collins, and Mark Davis — began work on a universal character set. Unicode 1.0 shipped in October 1991 with 7,161 characters. Today, Unicode 16.0 (released in 2024) defines more than 155,000 characters across 168 scripts, plus symbols, emoji, and historic scripts. The single, central idea is the code point: a unique integer from 0 to 1,114,111 assigned to every abstract character. "A" is U+0041, "中" is U+4E2D, "🦄" is U+1F984.

1980 年代末，来自施乐和苹果的工程师 —— 包括 Joe Becker、Lee Collins、Mark Davis 等人 —— 开始着手打造一套通用字符集。Unicode 1.0 于 1991 年 10 月发布，包含 7,161 个字符。如今的 Unicode 16.0（2024 年发布）已定义超过 155,000 个字符，涵盖 168 种文字，加上大量符号、表情和古文字。其中最核心的单一概念就是"码点"：一个从 0 到 1,114,111 的唯一整数，分配给每一个抽象字符。"A" 是 U+0041，"中" 是 U+4E2D，"🦄" 是 U+1F984。

Crucially, Unicode separates the question of "what is a character" from "how is it stored on disk or in memory". The first question is the Unicode Standard. The second is the realm of encodings, of which the most important are the UTF family: UTF-8, UTF-16, and UTF-32. This separation is what allowed Unicode to scale while keeping file formats sane.

关键在于，Unicode 把"什么是字符"与"字符如何存到磁盘或内存里"这两件事解耦了。前者由 Unicode 标准定义；后者由编码方式决定，其中最重要的是 UTF 家族：UTF-8、UTF-16、UTF-32。正是这种解耦，让 Unicode 在保持文件格式简洁的同时不断扩展。

UTF-8: variable width, elegant backward compatibility

UTF-8：变长编码与优雅的向后兼容

UTF-8 was invented by Ken Thompson and Rob Pike at Bell Labs in 1992, on a placemat in a New Jersey diner, according to folklore. The design is brutally simple and brilliant. Every ASCII character (code points 0–127) is encoded as a single byte, identical to ASCII. Code points 128 and above use 2, 3, or 4 bytes, with each byte in the 0x80–0xFF range clearly marked by its leading bits so a decoder can never lose its place in a stream.

UTF-8 是 Ken Thompson 和 Rob Pike 在 1992 年于贝尔实验室发明的，据说起草于新泽西一家餐厅的餐垫纸上。它的设计既简洁又精妙：所有 ASCII 字符（码点 0–127）都只用一个字节表示，与 ASCII 完全相同；128 及以上的码点用 2、3 或 4 个字节表示，并且每个位于 0x80–0xFF 范围的字节都通过起始位清晰标记，解析器永远能知道自己在字节流中的位置。

The bit patterns are:

具体的位模式如下：

• 0xxxxxxx — 1 byte, code points 0–127 (ASCII).
• 0xxxxxxx —— 1 字节，码点 0–127（ASCII）。
• 110xxxxx 10xxxxxx — 2 bytes, code points 128–2047 (most Latin, Greek, Cyrillic, Arabic, Hebrew).
• 110xxxxx 10xxxxxx —— 2 字节，码点 128–2047（大多数拉丁文、希腊文、西里尔文、阿拉伯文、希伯来文）。
• 1110xxxx 10xxxxxx 10xxxxxx — 3 bytes, code points 2048–65535 (BMP: Chinese, Japanese, Korean, most modern scripts).
• 1110xxxx 10xxxxxx 10xxxxxx —— 3 字节，码点 2048–65535（BMP：中、日、韩以及大多数现代文字）。
• 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx — 4 bytes, code points 65536 and beyond (emoji, historic scripts, mathematical symbols).
• 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx —— 4 字节，码点 65536 及以上（表情符号、古文字、数学符号）。

A consequence: a file written in pure English is exactly the same size in UTF-8 as in ASCII. A file in Chinese is about 1.5 times the size of UTF-16 but the same size as legacy GBK. A file with many emoji is bigger than UTF-16, but emoji-heavy text is the only case where that matters in practice. Because of this balanced trade-off, UTF-8 quietly won: HTML5 mandates it, JSON requires it, the Linux kernel uses it, and HTTP/2 specifies it.

这带来的一个结果是：纯英文文件在 UTF-8 下与 ASCII 大小完全一致；中文文件比 UTF-16 大约 1.5 倍，但与传统 GBK 相同；含有大量表情的文件比 UTF-16 大，不过在实践中这是唯一真正受影响的场景。正是这种均衡的取舍，让 UTF-8 悄悄胜出：HTML5 强制要求它、JSON 标准要求它、Linux 内核使用它、HTTP/2 协议也指定了它。

How emoji broke the surrogate-pair assumption

表情符号如何打破了代理对的假设

For most of UTF-16's life, the assumption was that every important character lives in the Basic Multilingual Plane, code points 0–65535, and can be encoded as a single 16-bit unit. The 2,048 code points from U+D800 to U+DFFF were deliberately left empty to serve as a "surrogate" mechanism for code points above U+FFFF. UTF-16 uses pairs of these surrogates to represent higher code points, meaning a single abstract character like "🎉" is two UTF-16 code units.

在 UTF-16 问世后的很长时间里，行业默认的假设是：所有重要字符都位于"基本多语言平面"（BMP），即码点 0–65535，可以用单个 16 位单元表示。U+D800 到 U+DFFF 这 2,048 个码点被故意留空，用作高码点的"代理对"机制：UTF-16 用一对代理单元表示 BMP 之外的码点，因此像 "🎉" 这样的单个字符在 UTF-16 中是两个码元。

This worked until emoji became mainstream. A "👨‍👩‍👧‍👦" family emoji is a sequence of four emoji joined by zero-width joiners and skin tone modifiers. In UTF-16 it can be seventeen code units long. Algorithms that counted "characters" by counting UTF-16 units, as much JavaScript code did in the 2000s, gave nonsensical results. This is why modern APIs report code points, code units, and grapheme clusters as three different concepts, and why the String iterator in Java, JavaScript, and Swift walks code points rather than code units.

在表情符号走入主流之前，这套机制一直运行良好。但一个"👨‍👩‍👧‍👦"家庭表情其实是由四个表情通过零宽连接符和肤色修饰符拼接而成。在 UTF-16 中，它最多可能长达 17 个码元。许多 2000 年代的 JavaScript 代码把"UTF-16 单元数"等同于"字符数"，结果完全错位。这正是为什么现代 API 把"码点""码元""字素簇"作为三个不同概念来报告，Java、JavaScript、Swift 等语言中的字符串迭代器也都按码点而不是码元来遍历。

UTF-8 vs UTF-16 vs UTF-32 — the real trade-offs

UTF-8 vs UTF-16 vs UTF-32 —— 真实的权衡

UTF-32 is the simplest: every code point is exactly four bytes, fixed width. It is the fastest encoding to index by code point and the easiest for compilers to deal with, which is why some virtual machines and font shapers use it internally. It is, however, the largest on disk and on the wire, and almost no serious file format or protocol uses it as a wire format.

UTF-32 最简单：每个码点恰好 4 字节，宽度固定。它是按码点索引最快的编码，也是编译器最容易处理的形式，因此一些虚拟机和字体引擎内部会使用它。但它在磁盘和网络上的体积最大，几乎没有任何主流文件格式或协议把它作为线上编码。

UTF-16 is a compromise: most characters are two bytes, but supplementary characters are four. It is the native string type of Java, C# (historically), Windows APIs, and JavaScript. For predominantly BMP text, it is more compact than UTF-8. For text with significant emoji or historic scripts, it can be larger.

UTF-16 是一种折中：大多数字符占 2 字节，但补充字符占 4 字节。它是 Java、C#（历史上）、Windows API 以及 JavaScript 的原生字符串类型。对于以 BMP 字符为主的文本，它比 UTF-8 更紧凑；而对于包含大量表情或古文字的文本，它反而会更大。

UTF-8 wins in practice because of three properties: backward compatibility with the gigantic existing ASCII ecosystem, dominance on the web, and excellent behavior in network protocols where resynchronization after errors matters. The only operational pain is that you cannot jump to "the 1,000th character" without scanning from the start — a problem solved by algorithms like UTF-8 binary search in constant time on a 64-bit word.

UTF-8 之所以在实践中胜出，源于三个特性：与庞大既有 ASCII 生态的向后兼容、在 Web 上的统治地位，以及在网络协议中错误后能快速重新同步的能力。它唯一的运行痛点是：无法直接跳到"第 1000 个字符"，必须从头扫描。不过这一问题已通过"按 64 位字二进制搜索"等算法得到常数时间解决。

The BOM, mojibake, and double encoding

BOM、乱码与双重编码

The byte order mark, or BOM, is a single character (U+FEFF) placed at the start of a text stream to indicate the encoding and byte order. In UTF-8 it is the bytes EF BB BF, in UTF-16 little-endian FF FE, in UTF-16 big-endian FE FF. The UTF-8 BOM is not required and is in fact discouraged on the web, but many Windows tools still emit one, which can break shell scripts and JSON parsers if not stripped.

字节顺序标记（BOM）是放在文本流开头的一个特殊字符（U+FEFF），用来指示编码和字节序。UTF-8 下的 BOM 是 EF BB BF 三个字节，UTF-16 小端序是 FF FE，UTF-16 大端序是 FE FF。UTF-8 的 BOM 并非必需，Web 上甚至建议省略，但很多 Windows 工具仍然会输出它，如果不预先剥离，常常会让 shell 脚本和 JSON 解析器出错。

Mojibake is the famous "é" instead of "é" syndrome. It happens when bytes valid in one encoding are interpreted as another. The most common case is UTF-8 bytes read as Latin-1: the two-byte sequence for "é" (0xC3 0xA9) is displayed as "é". Double encoding compounds the disaster: a string is encoded to UTF-8, then that result is encoded again as Latin-1 or Windows-1252, producing "é" — two layers of garbling. The fix is to ensure every step of your pipeline reads and writes in the same declared encoding. The single most useful habit is to declare charset=utf-8 in HTTP headers and meta tags and to never, ever open a file without knowing its encoding.

乱码最经典的表现就是 "é" 代替 "é"。这通常发生在"以编码 A 写入的字节被当成编码 B 读取"时。最常见的情况是 UTF-8 字节被当作 Latin-1 解读：表示 "é" 的双字节序列（0xC3 0xA9）被显示为 "é"。双重编码会让问题更糟：先以 UTF-8 编码一次，再把结果当成 Latin-1 或 Windows-1252 编码一次，最终结果是 "é" —— 两层乱码叠加。修复方法是确保管线中每一步都用同一声明的编码读写。最有用的习惯是在 HTTP 头和 meta 标签中声明 charset=utf-8，并且任何时候打开文件都要先弄清楚它的编码。

What comes next?

接下来会是什么？

Unicode is unlikely to be displaced. Its role has expanded beyond text to identifiers in programming languages, security profiles, and machine-readable rules for normalization, casing, and collation. The remaining challenges are operational: legacy encodings will linger in Asian mainframes and in decades-old file archives; mixed-script attacks (using visually identical characters from different scripts) drive the need for stricter normalization in identifiers. UTF-8's reign is safe for the foreseeable future, but the standard itself will keep growing — version 17 is already in the pipeline, adding new characters including additional script support for underrepresented languages.

Unicode 短期内不太可能被取代。它的角色早已超出"文本"本身，渗透到编程语言的标识符、安全配置、以及规范化、大小写、排序等机器可读规则中。剩下的挑战更多是工程层面的：旧编码还会在亚洲大型机和几十年历史的归档文件中长期存在；混写脚本攻击（用视觉上几乎相同的不同文字字符）也推动着对标识符的更严格规范化。UTF-8 的统治地位在可预见的未来是稳固的，而标准本身会继续扩展 —— 17 版已经在筹备中，将增加新的字符，包括对一些代表性不足语言的支持。

The lesson from the encoding wars is that simplicity and compatibility beat cleverness. ASCII had 128 slots and worked for everyone who only spoke English. UTF-8 has a clever bit-pattern table and works for nearly everyone on Earth. That is the future of character encoding: not a new format, but the quiet universal adoption of one that finally worked.

编码之争的教训是：简洁与兼容胜过精巧。ASCII 只有 128 个槽位，但让所有只说英语的人都能使用；UTF-8 用一套精巧的位模式表，服务了地球上几乎所有人。这正是字符编码的未来 —— 不再是出现新格式，而是让那套"终于奏效"的方案被全人类静默地、彻底地采纳。

Try the tools

试试这些工具

Experiment with byte-level encoding using the Base64 encoder/decoder and the URL encoder/decoder. Both run entirely in your browser, so you can paste text and see exactly how bytes map between encodings without sending anything to a server.

用 Base64 编码/解码 和 URL 编码/解码 工具，在字节层面亲自体验编码过程。两者都完全在你的浏览器中运行，粘贴文本就能看到字节在不同编码之间如何映射，无需上传任何数据。