大网数据广告活动
网站运营与故障问答

Same Arabic phrase encoding into two different urls, why? How to handle?

· 52CE 我爱测

I have a multi-lingual site which two arabic phrases which to my eye (and to the Arabic speakers I've asked) looks the same, but they encode differently. إعادةصياغةالنصتلقائياباللغةالعربيةمجانا becomes %D8%A7%D9%95%D8%B9%D8%A7%…

问题原文

原帖链接 · Kevin Danikowski · CC BY-SA 4.0

I have a multi-lingual site which two arabic phrases which to my eye (and to the Arabic speakers I've asked) looks the same, but they encode differently.

إعادةصياغةالنصتلقائياباللغةالعربيةمجانا becomes %D8%A7%D9%95%D8%B9%D8%A7%D8%AF%D8%A9%D8%B5%D9%8A%D8%A7%D8%BA%D8%A9%D8%A7%D9%84%D9%86%D8%B5%D8%AA%D9%84%D9%82%D8%A7%D9%8A%D9%94%D9%8A%D8%A7%D8%A8%D8%A7%D9%84%D9%84%D8%BA%D8%A9%D8%A7%D9%84%D8%B9%D8%B1%D8%A8%D9%8A%D8%A9%D9%85%D8%AC%D8%A7%D9%86%D8%A7

إعادةصياغةالنصتلقائياباللغةالعربيةمجانا becomes %D8%A5%D8%B9%D8%A7%D8%AF%D8%A9%D8%B5%D9%8A%D8%A7%D8%BA%D8%A9%D8%A7%D9%84%D9%86%D8%B5%D8%AA%D9%84%D9%82%D8%A7%D8%A6%D9%8A%D8%A7%D8%A8%D8%A7%D9%84%D9%84%D8%BA%D8%A9%D8%A7%D9%84%D8%B9%D8%B1%D8%A8%D9%8A%D8%A9%D9%85%D8%AC%D8%A7%D9%86%D8%A7

The difference is minor, but how can I control these examples? Also, does anyone have any insights into what is happening above?

已采纳解答

原帖链接 · Patrick Mevzek · CC BY-SA 4.0

The difference comes from normalization.

I know nothing about Arabic language but if I use both strings in a Unicode viewer, the difference starts right at the left part of the string.

First string is composed of these Unicode characters:

 ‎0627 ARABIC LETTER ALEF
 ‎0655 ARABIC HAMZA BELOW

when for the second string it is:

0625 ARABIC LETTER ALEF WITH HAMZA BELOW

The graphical display is the same, because the above are just two representations, one decomposed and one composed, of the same glyph. You can normalize strings in one direction, decomposing all characters, or in the other, composing them back.

But that explains what you have in string as UTF8 since percent encoded:

U+0627 is %D8%A7 
U+0655 is %D9%95

where

U+0625 is %D8%A5

Both versions are technically correct and the fact they exist are mostly because of how Unicode was created and the need to try keeping some compatibility with earlier situations (that were not able to handle "decomposed" characters).

However "internalized" characters in URLs are governed by a standard called "IRI", like an extension of URI. This is RFC3987 "Internationalized Resource Identifiers (IRIs)"

It explains how to go from an arbitrary Unicode character to an URL/URI/IRI.

Inside "5.3.2.2. Character Normalization" there is this:

To avoid false negatives and problems with transcoding, IRIs SHOULD be created by using NFC.

As such the NFC form of your string is really the following list of characters:

 ‎0625 ARABIC LETTER ALEF WITH HAMZA BELOW
 ‎0639 ARABIC LETTER AIN
 ‎0627 ARABIC LETTER ALEF
 ‎062F ARABIC LETTER DAL
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0635 ARABIC LETTER SAD
 ‎064A ARABIC LETTER YEH
 ‎0627 ARABIC LETTER ALEF
 ‎063A ARABIC LETTER GHAIN
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0627 ARABIC LETTER ALEF
 ‎0644 ARABIC LETTER LAM
 ‎0646 ARABIC LETTER NOON
 ‎0635 ARABIC LETTER SAD
 ‎062A ARABIC LETTER TEH
 ‎0644 ARABIC LETTER LAM
 ‎0642 ARABIC LETTER QAF
 ‎0627 ARABIC LETTER ALEF
 ‎0626 ARABIC LETTER YEH WITH HAMZA ABOVE
 ‎064A ARABIC LETTER YEH
 ‎0627 ARABIC LETTER ALEF
 ‎0628 ARABIC LETTER BEH
 ‎0627 ARABIC LETTER ALEF
 ‎0644 ARABIC LETTER LAM
 ‎0644 ARABIC LETTER LAM
 ‎063A ARABIC LETTER GHAIN
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0627 ARABIC LETTER ALEF
 ‎0644 ARABIC LETTER LAM
 ‎0639 ARABIC LETTER AIN
 ‎0631 ARABIC LETTER REH
 ‎0628 ARABIC LETTER BEH
 ‎064A ARABIC LETTER YEH
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0645 ARABIC LETTER MEEM
 ‎062C ARABIC LETTER JEEM
 ‎0627 ARABIC LETTER ALEF
 ‎0646 ARABIC LETTER NOON
 ‎0627 ARABIC LETTER ALEF

which gives this URL encoding:

%D8%A5%D8%B9%D8%A7%D8%AF%D8%A9%D8%B5%D9%8A%D8%A7%D8%BA%D8%A9%D8%A7%D9%84%D9%86%D8%B5%D8%AA%D9%84%D9%82%D8%A7%D8%A6%D9%8A%D8%A7%D8%A8%D8%A7%D9%84%D9%84%D8%BA%D8%A9%D8%A7%D9%84%D8%B9%D8%B1%D8%A8%D9%8A%D8%A9%D9%85%D8%AC%D8%A7%D9%86%D8%A7

(you can get it from https://r12a.github.io/app-conversion/index.html for example)

So in theory that should be the only form used, especially if you are the producer of those URLs.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2022-11-14)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。