> > how could Planet RDF (Resource Description Framework) do things better?
>
Danny, you can see the problem above.
Is the Planet RDF code available? I would gladly provide a patch.
The problem is not in your input feed. You contain the following code: ’. Planet RDF converts that into binary, which I will express in hex: xC3A2C280C299. The correct formulation would be xE28099. I've got a good idea about what is going on under the covers as
> * â => xC3A2
* € => xC280
* ™ => xC299
In other words, for some reason, RDF Planet is effectively doing a iso-8859-1 to utf-8 conversion, on utf-8 data.
“Sam Ruby [1]”
It's a mess.
Part of the problem I'm sure is that not many programmers are used to i18n (Internationalization). And part of it is not many programmers are aware of what data is in what format at what part of the processing. And that is related to variable types in programming.
The first problem is i18n. I won't say it's trivial, but it does take some forethought to handle it. Thirty years ago it was simple—you use the character set of whatever country you were in and that was that. Since computers work with numbers, a “character set” is just a mapping of values (say, 1) to a visual representation of a character, a “glyph” (say, “A”). Here in the US (United States), we used ASCII (American Standard Code for Information Interchange), where in this case, the character “A” is internally represented with the value 65 (and the lower case “a” by a different value, 97 since technically, it is a different glyph). The character “A” in say, Freedonia might be stored as the value 193.
No problem. Well, unless your software was sold internationally, then you had a problem of handing different character sets. And there are a lot of them.
So a solution (and it's a pretty good solution) is to use some internal representation of every possible character set as the programming is running (say, Unicode [2], which was defined to handle pretty much any written language) and do any conversions on input and output to the locally defined character set (I've found the GNU iconv library [3] to be pretty easy to use and it can handle a ton of different character sets). Yes, there are some really obscure corner cases doing this, but for the most part, it'll handle perhaps 95% of all i18n issues.
But web based documents—or rather, documents based upon HTML (HyperText Markup Language)/XML (eXtensible Markup Language)—present a wrinkle. HTML/XML use the “<” character (value of 60) to start the beginning of tag, like <P> (in this case, this particular tag denotes the start of a paragraph). But what if the content (that is, non-tag portion) actually requires a literal “<”? What then? Well, HTML/XM use another character, “&”, to encode characters that you would otherwise be unable to use. So, to get a “<” you would write it (or “encode it”) as <, which means that to use a “&” in your document, you need to write it as &.
But you can use this form, called “entities” (or HTML entities) to include other characters that might otherwise not be part of your local character set, such as the much nicer typographical quotes “” (compare to the computerish ""), or even the star in the middle of Wal★Mart.
Now, most of these entities, like & are named, such as “ for the nice looking opening double quotes, or → for a right arrow. But a lot don't have names, like the star in the middle of Wal★Mart. To get those, you need to use a numeric value, like ★.
But, where do you get these values?
Well, they're Unicode values.
So, even if you're writing your page in a Slavic langauge (using for instance, the character set defined as ISO (International Standards Orgranization)-8859-5 [4]), the “á” can still be encoded as á, using the Unicode value for “á” (since ISO-8859-5 doesn't contain that glyph). To further confound things (and seeing how for this example, we're using ISO-8859-5) the character “Љ” can be inserted as a character with value 169 (whereas if viewed as an ISO-8859-1 character, it would be “©”) or as Љ (which would work reguardless of the character set you are using for your document).
Got it?
If not, you're in good company with lots of other programmers out there. And that's even before we get to the issue of submitting text to a CGI (Common Gateway Interface) script (I'll get to that later).
And that leads to the second problem, where programmers don't know what's what where in their program. And that's related to variable types. Or rather, the lack of variable types. Or rather, the rather lackadaisical approach to typing most modern computer languages have today (Python? Perl? I'm looking at you).
In today's modern languages, unlike slightly older languages like C or Pascal, you don't really need to declare your variables before you use them— you can just use them. Sure, you can predeclare them (and if you are smart, you do) but you don't have to, and even when you do, you just declare that foobar is a variable, not what it can hold (well, granted, in Perl, you have to tell it if the variable can contain a single value, or a list of values, or a table of values). As Lisp programmers are wont to say: “a variable doesn't have a type, values do.”
Now, why is that such a problem?
Well, other than muddle headed thinking [5], there's overhead at runtime in determining if the operation is permitted on the type of the value held in a variable and if not, what can we do with the value in the variable to make it the right type for the operation in question. So, in the following Perl code:
>
```
$result = $a + $b;
```
at run time the computer has to determine the types of the values of the variables $a and $b—if numeric it can just add them; if strings, it has to convert the strings to a numeric value and then add them. And what if $a contains the string “one” and $b contains the string “two”? $result will have the numeric value of 0 because “one” and “two” do not convert to numeric values (the ol “Garbage in, garbage out” saying in Computer Science; oh, you were expecting maybe “onetwo”? That's a different operator altogether).
But even if we just stick with strings (and HTML processing is a lot of string handling) we're not really out of the woods. Even if we were to use a computer language with stonger typing (say, Ada) that wouldn't even help us because the type systems of any language today just aren't up to it.
For instance, given a simple HTML form that accepts input, we can feed it the following:
“This—is–a★test”
Now, when the data is submitted, the browser will encode the data in yet another encoding scheme (application/x-www-form-urlencoded [6] to be precise), which can contain HTML encoded entities (see above) in any number of character sets (although, unless otherwise stated, in the same character set the browser thinks the page that has the form uses). Using the latest version of Firefox (1.5 as of this writing), if instructed to send the data using charset US-ASCII the data I get is:
>
```
%26%238220%3BThis%26%238212%3Bis%26%238211%3Ba%26%239733%3Btest%26%238221%3B
```
Which, when decoded, is:
>
```
“This—is–a★test”
```
But, if the data is sent as charset ISO-8859-1:
>
```
%93This%97is%96a%26%239733%3Btest%94
```
Which, when decoded, is:
>
```
“This—is–a★test”
```
The quotes and dashes (the “mdash” and “ndash” respectively) are single characters but the star is still sent as an HTML entity. Yet if the browser is told to send the data as UTF (Unicode Transformation Format)-8:
>
```
%E2%80%9CThis%E2%80%94is%E2%80%93a%E2%98%85test%E2%80%9D
```
Which, when decoded, is:
>
```
“This—is–a★test”
```
So, what does all this have to do with variable types and the weakness of typechecking even in a language like Ada?
Well, if we could declare our variables with not only the type (in this case, “string”) but with additional clarifications on said type (say, “charset ISO-8859- 5 with HTML entities” or “charset UTF-8 encoded as application/x-www-form-urlencoded”) it would clarify the data flow through a lot of web based software and prevent stupid mistakes [7] (and yes, it would piss off a lot of muddle-headed programmers but this would at least force them to think for once).
All this talk about encoding issues, and I still blew it and had to manually fix some encoding issues on this page.
Sigh.
[1] http://www.intertwingly.net/blog/2004/10/20/Attractive-
[2] http://en.wikipedia.org/wiki/Unicode
[3] http://www.gnu.org/software/libiconv/
[4] http://en.wikipedia.org/wiki/Iso-8859
[6] http://www.w3.org/TR/html4/interact/forms.html#form-content-