Every software developer, regardless of their experience level, should have a basic understanding of Unicode and character sets. Surprisingly, many developers are not well-versed in these topics. This article aims to provide a concise overview, ensuring that every developer has the foundational knowledge they need.

The Mystery of Character Sets: Have you ever wondered about the ‘Content-Type’ tag in HTML? Or received an email with indecipherable characters in the subject line? Many developers are unfamiliar with the intricacies of character sets, encodings, and Unicode. However, in today’s interconnected world, understanding these concepts is crucial.

A Brief History: To grasp the evolution of character sets, it’s helpful to look back in time. While ancient character sets like EBCDIC might seem irrelevant, understanding the progression from ASCII to modern-day encodings is essential. Initially, ASCII was sufficient for English speakers, but as computing became global, the limitations became evident. Different regions began using the 128-255 byte range for their characters, leading to confusion and miscommunication. This chaos eventually led to the establishment of the ANSI standard and the concept of code pages.

The Advent of Unicode: Unicode emerged as a solution to the growing complexity of character sets. Contrary to popular belief, Unicode isn’t just a 16-bit code. Instead, it assigns a unique number, or ‘code point’, to every character from every writing system. This system allows for a vast number of characters, far exceeding the 65,536 that two bytes can represent.

Encoding Matters: Knowing the Unicode code point of a character is only part of the equation. How these code points are stored or transmitted is determined by their encoding. The initial idea was to use two bytes for each character, but this approach had its challenges, leading to the development of various encoding methods like UTF-8, UTF-16, and more.

The Importance of Specifying Encodings: One crucial takeaway is that a string’s meaning is lost without knowledge of its encoding. Strings without specified encodings can lead to display issues or data corruption. For web content, specifying the encoding using the ‘Content-Type’ header or meta tag is essential. Browsers might attempt to guess the encoding, but this method is unreliable.

Top 5 Most Used Unicode Charsets

1. UTF-8: The Dominant Charset

• Overview: UTF-8 (8-bit Unicode Transformation Format) is a variable-width character encoding that can represent every character in the Unicode standard. It has become the dominant charset for the web and many other applications.
• Advantages:
  • Backward Compatibility: UTF-8 is compatible with ASCII, meaning that any ASCII text is also a valid UTF-8 text.
  • Flexibility: It can encode any character in the Unicode standard yet remains compact for ASCII characters.
  • Widespread Adoption: From websites to databases, UTF-8 is the preferred encoding due to its versatility.

2. UTF-16: Bridging the Gap

• Overview: UTF-16 (16-bit Unicode Transformation Format) uses 16 bits as its basic unit. It can represent the most commonly used characters using a single 16-bit code unit and others using a pair of 16-bit code units.
• Usage: It’s commonly used in many programming environments, like Java and Windows, and is suitable for texts with a mix of common and uncommon characters.
• Considerations: While it offers a balance between size and range, it’s not as compact as UTF-8 for ASCII text.

3. UTF-32: Direct Mapping to Unicode Code Points

• Overview: UTF-32 uses a single 32-bit code unit to represent each character, providing a direct mapping to Unicode code points.
• Advantages:
  • Simplicity: Each character corresponds to 4 bytes, making certain operations simpler.
  • Full Coverage: It can represent every Unicode character without the need for multi-unit sequences.
• Drawbacks: Its fixed-width nature means it’s not as space-efficient as UTF-8 or UTF-16 for texts primarily in ASCII or BMP (Basic Multilingual Plane).

4. UCS-2: The Precursor to UTF-16

• Overview: UCS-2 (2-byte Universal Character Set) was a precursor to UTF-16 and uses two bytes for each character. However, it can only represent characters in the BMP.
• Limitations: UCS-2 cannot represent characters outside the BMP, making it less versatile than UTF-16. As a result, its usage has declined in favor of UTF-16.

5. GB18030: Bridging Unicode and China

• Overview: GB18030 is a Chinese government standard that covers all Unicode characters, ensuring compatibility between China’s character sets and Unicode.
• Significance: It’s mandatory for all software products in China, making it crucial for international software developers to support it.

How to specify Unicode Charsets in htaccess, html pages, WordPress and more?

The correct specification of character sets is crucial for ensuring that web content is displayed as intended. Here’s how you can specify Unicode charsets in various platforms and technologies:

1. .htaccess (for Apache servers):

To set the character set for your entire website using the .htaccess file, you can use the AddDefaultCharset directive:

AddDefaultCharset UTF-8

2. HTML Pages:

In HTML documents, you can specify the character set using the <meta> tag within the <head> section:

<meta charset="UTF-8">

For older versions of HTML (HTML 4), the specification looks a bit different:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

3. WordPress:

In WordPress, the charset is typically specified in the theme’s header file (header.php). However, WordPress automatically sets the charset to UTF-8. If you need to check or modify it:

1. Navigate to your theme’s directory and open header.php.
2. Look for the <meta charset> tag. It should look like this:

<meta charset="<?php bloginfo( 'charset' ); ?>">

This PHP function will output the charset set in WordPress settings, which is UTF-8 by default.

4. PHP:

To send a raw HTTP header specifying the charset in a PHP script, use the header() function:

header('Content-Type: text/html; charset=UTF-8');

5. MySQL:

When creating or modifying a database or table in MySQL, you can specify the character set:

CREATE DATABASE mydatabase DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Or for a table:

CREATE TABLE mytable (
...
) DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

6. CSS:

Though not common, you can specify a charset in a CSS file:

@charset "UTF-8";

This should be the very first line of your CSS file.

7. JavaScript:

For external JavaScript files, you can specify the charset when you link to them in your HTML:

<script src="myscript.js" charset="UTF-8"></script>

8. XML:

In XML documents, the charset can be specified in the XML declaration:

<?xml version="1.0" encoding="UTF-8"?>

mysqldump -u username -p database_name > backup.sql

2. Check Current Character Set and Collation:

First, check the current character set and collation of your database and its tables.

SHOW CREATE DATABASE database_name;
SHOW CREATE TABLE table_name;

3. Alter Database and Tables:

If the database or tables are not using the desired character set (e.g., utf8mb4) and collation (e.g., utf8mb4_unicode_ci), you can alter them.

Alter Database:

ALTER DATABASE database_name CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Alter Tables and Columns:

For each table:

ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

4. Fixing Data:

If you have wrongly encoded data in the database, you might need to convert it. One common scenario is double-encoded UTF-8 data. Here’s how you can fix it:

UPDATE table_name SET column_name = CONVERT(CAST(CONVERT(column_name USING latin1) AS BINARY) USING utf8mb4) WHERE some_condition;

Replace table_name, column_name, and some_condition with appropriate values. This code snippet assumes that the data was originally in UTF-8, then mistakenly treated as Latin1 and stored as UTF-8 again.

5. Update MySQL Client:

Ensure your MySQL client connection uses the correct character set:

SET NAMES 'utf8mb4';

Or, if you’re using a configuration file (like my.cnf or my.ini), you can add:

[client]
default-character-set = utf8mb4

6. Check Application Code:

Ensure that your application’s database connection string specifies the correct character set. For example, in PHP with PDO:

Conclusion:

Fixing Unicode issues in MySQL involves both adjusting the database schema and ensuring that data is correctly encoded. Always make sure to test changes in a safe environment before applying them to a production database.