Het Bijstere Spoor

Over the last little while I've come across quite a few XML feed generators written in PHP, with varying degrees of 'correctness'. Even though generating XML should be very simple, there's still quite a bit of pitfalls I feel every PHP or (insert your language)-developer should know about.

1. You are better off using an XML library

This is the first and foremost rule. Most people end up generating their xml using simple string concatenation, while there are many dedicated tools out there that really help you generate your own XML.

In PHP land the best example is XMLWriter. It is actually quite easy to use:

<?php
 
$xmlWriter = new XMLWriter();
$xmlWriter->openMemory();
$xmlWriter->startDocument('1.0','UTF-8');
$xmlWriter->startElement('root');
$xmlWriter->text('Contents of the root tag');
$xmlWriter->endElement(); // root
$xmlWriter->endDocument();
echo $xmlWriter->outputMemory();
 
?>

Granted, XMLWriter is verbose, but you have to worry a lot less about escaping and validating your xml documents.

2. Understand Unicode

Do you know the difference between a byte, a character and a codepoint? If you don't, I'd probably think twice about hiring you. It's absolutely shocking how many programmers are out there that don't understand the basics of unicode, UTF-8 and how it relates to the web.

An often-heard excuse for not having to care for non-ascii characters, such as people in English speaking countries. However, if you need to use the euro-sign (€) or if you deal with people copy-pasting from word documents, you most definitely will come across problems.

A simple call to utf8_encode is not actually enough. If some of your source-data was already encoded as UTF-8 you will end up losing data. Only use utf8_encode if you know your source-data is encoded as ISO-8859-1.

The one true way to go about it, is to make sure that every step of the way in your web application is UTF-8. Including your HTTP/HTML contenttype, MySQL database and anything that basically ingests data for your application (email, csv importers, xml readers, web services). Once you are absolutely sure every part in your application is UTF-8, and converted any old data things will start to behave correctly.

3. CDATA is never a solution

It might be tempting to solve any encoding issues by simply surrounding it with <![CDATA[ and ]]>. This might make sure that XML parsers don't throw an error when reading, but they still have 'incorrect' characters. If your XML document has CDATA tags, or you think you need CDATA, you are probably wrong.

More often than not using CDATA actually stems from encoding problems (see section 2). CDATA is not a method to encode binary characters, xml parsers will still throw errors if they come across certain byte sequences. If you do really need to encode binary data in XML, the best way is to use something like base64_encode instead.

If your XML feed uses CDATA because of encoding issues you actually defer your problem to the consumer of your XML feed. So instead of seeing 'weird characters' on your side, the person that reads your xml feed now has no good way to detect which encoding was actually used. If it's for example an RSS feed you're generating, this can result in RSS readers throwing errors, or characters showing up incorrectly.

4. Be liberal with whitespace

An error like "unexpected character at line 1, column 176456" is much harder to debug than "line 5078, column 24". Whitespace between xml tags does usually not have any significance, so you can add as much indentation and linebreaks (\n) as you want. Note that tools such as XMLWriter will indent for you automatically.

5. Be verbose

Even though you might easily figure out that <ORD_NR> means 'order number', there's no reason why you shouldn't actually state it as <order-number>. Note that the following rules appear to fall in favor for most people:

• Use lowercase for tags and attribute names.
• Use dashes (-) to separate words, not underscores (_).
• Minimize the use of attributes, nested tags allow more flexibility.

6. Be careful with entities

The only valid entities in XML are &lt; (<), &gt; (>) &amp; (&) and &quot; ("), so any other entity will simply not work and throw errors.

HTML DTD's add many entities, so if you're mostly used to using HTML you might expect other entities to work. If your source-data already has entities, you might have to get rid of these first.

In PHP it means you should use htmlspecialchars, instead of htmlentities.

Feel free to discuss, disagree, or add on to this list in the comments, I'm happy to hear your experiences.

I've started working on a CalDAV implementation, which also requires analysis of ICalendar (rfc 2554) objects.

ICalendar objects have properties, components (such as VEVENT, VTODO) and attributes. This is awfully familiar to XML. So instead of trying to come up with a complicated parser and object structure, I decided to just convert it to XML and use PHP's simplexml.

This is my current script:

<?php
 
function iCalendarToXML($icalendarData) {
 
    // Detecting line endings
    if (strpos($icalendarData,"\r\n")) $lb = "\r\n";
    elseif (strpos($icalendarData,"\n")) $lb = "\n";
    else $lb = "\r\n";
 
    // Splitting up items per line
    $lines = explode($lb,$icalendarData);
 
    // Properties can be folded over 2 lines. In this case the second
    // line will be preceeded by a space or tab.
    $lines2 = array();
    foreach($lines as $line) {
 
        if ($line[0]==" " || $line[0]=="\t") {
            $lines2[count($lines2)-1].=substr($line,1);
            continue;
        }
 
        $lines2[]=$line;
 
    }
 
    $xml = '<?xml version="1.0"?>' . "\n";
 
    $spaces = 0;
    foreach($lines2 as $line) {
 
        $matches = array();
        // This matches PROPERTYNAME;ATTRIBUTES:VALUE
        if (preg_match('/^([^:^;]*)(?:;([^:]*))?:(.*)$/',$line,$matches)) {
            $propertyName = strtoupper($matches[1]);
            $attributes = $matches[2];
            $value = $matches[3];
 
            // If the line was in the format BEGIN:COMPONENT or END:COMPONENT, we need to special case it.
            if ($propertyName == 'BEGIN') {
                $xml.=str_repeat(" ",$spaces);
                $xml.='<' . strtoupper($value) . ">\n";
                $spaces+=2;
                continue;
            } elseif ($propertyName == 'END') {
                $spaces-=2;
                $xml.=str_repeat(" ",$spaces);
                $xml.='</' . strtoupper($value) . ">\n";
                continue;
            }
 
            $xml.=str_repeat(" ",$spaces);
            $xml.='<' . $propertyName;
            if ($attributes) {
                // There can be multiple attributes
                $attributes = explode(';',$attributes);
                foreach($attributes as $att) {
 
                    list($attName,$attValue) = explode('=',$att,2);
                    $xml.=' ' . $attName . '="' . htmlspecialchars($attValue) . '"';
 
                }
            }
 
            $xml.='>'. htmlspecialchars($value) . '</' . $propertyName . ">\n";
 
        }
 
    }
 
    return $xml;
 
}
 
?>

This will convert:

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Example Corp.//CalDAV Client//EN
BEGIN:VTIMEZONE
LAST-MODIFIED:20040110T032845Z
TZID:US/Eastern
BEGIN:DAYLIGHT
DTSTART:20000404T020000
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=4
TZNAME:EDT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
END:DAYLIGHT
BEGIN:STANDARD
DTSTART:20001026T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
TZNAME:EST
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DESCRIPTION:Hello Im evert
 Next line also
  Blabla
ATTENDEE;PARTSTAT=ACCEPTED;ROLE=CHAIR:mailto:cyrus@example.com
ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:lisa@example.com
DTSTAMP:20060206T001220Z
DTSTART;TZID=US/Eastern:20060104T100000
DURATION:PT1H
LAST-MODIFIED:20060206T001330Z
ORGANIZER:mailto:cyrus@example.com
SEQUENCE:1
STATUS:TENTATIVE
SUMMARY:Event #3
UID:DC6C50A017428C5216A2F1CD@example.com
X-ABC-GUID:E1CX5Dr-0007ym-Hz@example.com
END:VEVENT
END:VCALENDAR

<?xml version="1.0"?>
<VCALENDAR>
  <VERSION>2.0</VERSION>
  <PRODID>-//Example Corp.//CalDAV Client//EN</PRODID>
  <VTIMEZONE>
    <LAST-MODIFIED>20040110T032845Z</LAST-MODIFIED>
    <TZID>US/Eastern</TZID>
    <DAYLIGHT>
      <DTSTART>20000404T020000</DTSTART>
      <RRULE>FREQ=YEARLY;BYDAY=1SU;BYMONTH=4</RRULE>
      <TZNAME>EDT</TZNAME>
      <TZOFFSETFROM>-0500</TZOFFSETFROM>
      <TZOFFSETTO>-0400</TZOFFSETTO>
    </DAYLIGHT>
    <STANDARD>
      <DTSTART>20001026T020000</DTSTART>
      <RRULE>FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10</RRULE>
      <TZNAME>EST</TZNAME>
      <TZOFFSETFROM>-0400</TZOFFSETFROM>
      <TZOFFSETTO>-0500</TZOFFSETTO>
    </STANDARD>
  </VTIMEZONE>
  <VEVENT>
    <DESCRIPTION>Hello Im evertNext line also Blabla</DESCRIPTION>
    <ATTENDEE PARTSTAT="ACCEPTED" ROLE="CHAIR">mailto:cyrus@example.com</ATTENDEE>
    <ATTENDEE PARTSTAT="NEEDS-ACTION">mailto:lisa@example.com</ATTENDEE>
    <DTSTAMP>20060206T001220Z</DTSTAMP>
    <DTSTART TZID="US/Eastern">20060104T100000</DTSTART>
    <DURATION>PT1H</DURATION>
    <LAST-MODIFIED>20060206T001330Z</LAST-MODIFIED>
    <ORGANIZER>mailto:cyrus@example.com</ORGANIZER>
    <SEQUENCE>1</SEQUENCE>
    <STATUS>TENTATIVE</STATUS>
    <SUMMARY>Event #3</SUMMARY>
    <UID>DC6C50A017428C5216A2F1CD@example.com</UID>
    <X-ABC-GUID>E1CX5Dr-0007ym-Hz@example.com</X-ABC-GUID>
  </VEVENT>
</VCALENDAR>

I hope this is useful to anyone else.

While working on a simple feed parser, I hit upon some wordpress feeds.

I noticed that wordpress feeds make heavy usage of CDATA to encode content. I always figured this was a bad idea if you cannot control what ends up in the xml feed. (Example here.).

Doing some googling to see if I'm not just kicking dust brought me to an xml.com article titled 'Escaped Markup Considered Harmful, which seems to agree with my standpoint for the following reason:

Escaping markup, particularly with CDATA sections, just doesn't work. There are other things that might be wrong that would make the documents not well formed. There are Unicode characters that are forbidden, there are encoding issues for the characters that are allowed, and there are sequences of characters that must be avoided. (e.g., "]]>"). Not to mention the fact that CDATA sections don't nest.

CDATA can't be used to just dump in any type of content that won't work in normal XML sections.. You're still obligated to make your data valid unicode. In fact, it's the opposite; There's no way you could ever escape the ]]> character sequence.