File types revisited
File typing has been a hot topic in Mac OS X ever since file name extensions became mandatory in Apple's interface guidelines in 2001. Philosophical issues aside, there are real, practical problems with file name extensions as a means to store and identify file type information. Let's even ignore storage issue entirely and focus just on identifying a file's type.
The first problem with file name extensions as a means to identify a file's type is that they are not unique. File name extensions are used on many operating systems, and there is no universal arbiter for which extensions indicate which file types.
Maybe "universal" uniqueness is too high a standard. But even within a single operating system, many file name extensions are overloaded. Even "
.doc", which most people know as the file name extension for Microsoft Word documents, is used by at least four other applications: FrameMaker, WordStar, WordPerfect, and DisplayWrite. Clearly, a file's type cannot be conclusively and uniquely identified based on its file name extension.
This problem alone is sufficient to disqualify file name extensions as a "correct" (let alone "good") system for file type identification. But ignoring even this failure, there is another inadequacy. File name extensions have historically been opaque; the entire extension is a single string with no predefined internal structure. This precludes any sort of namespace or inheritance system.
Apple has been more expressive than most vendors with its file name extensions. Smart folders have a "
.savedSearch" extension; text clippings have a "
.textClipping" extension; frameworks have a "
.framework" extension. In the Windows world, these extensions would probably be something like "
.txc," and "
.fwk," respectively. (Despite the long-dead three-character limit on file name extensions in Windows, Microsoft continues to favor "compatible" file name extensions—and file names, for that matter.) But either way, extensions remain imprecise and inflexible.
There have been many other file typing systems over the years. Classic Mac OS used a combination of two 32-bit integers, usually expressed as four-character strings, to identify file types. Apple maintained a registry of type and creator codes. Since no other platform used them, they were guaranteed to be unique among all respectable Mac applications. But type and creator codes are even less expressive than file name extensions, especially the verbose ones used in Mac OS X. (The sane storage location for type and creator codes more than made up for this, however.)
More recently, MIME, a data typing system originally designed for use with e-mail, has spread to other domains, most notably the web. MIME types are hierarchical text strings with well defined rules controlling their format. While a JPEG image file might have a "
.jpg" or "
.jpeg" file name extension and a "
JPEG" classic Mac OS type code, the MIME type would be "
image/jpeg" which indicates that a JPEG is a subtype of "image." There are only a few official primary types (image, video, audio, text, application, etc.) but there are several mechanisms for vendor-specific ("
application/vnd.ms-excel") and ad-hoc ("
BeOS, pioneer of all things metadata related that it was, chose to use MIME as its native file typing system. It was superior to both the classic Mac OS and Windows approaches, storing file type metadata in its own dedicated location like classic Mac OS, but also supporting a file type hierarchy and expansion mechanism.
BeOS didn't live long enough to find this out the hard way, but MIME has its weaknesses as a file typing system as well. MIME's confinement to a two-level hierarchy is very limiting. Sure, all image MIME types are under "
image/...", but what about the vendor who creates a new image format? New subtypes under "
image/..." or any other top-level type must be "registered with, and approved by IANA" according to the RFC. The only other option is to use the ad hoc "
x-" prefix which offers no guarantees of uniqueness.
Now imagine that a vendor invents what can only be reasonably considered a new top-level type. Actually, it doesn't take much imagination because there are existing file formats that are already beyond the set of top-level MIME types. MIME defines top-level types for video, audio, and images, but nothing that completely covers something like, say, a QuickTime movie file which can contain video, audio, and images all in the same file.
MIME does offer the same "
x-" prefix for ad hoc top-level types, but again there is no guarantee of uniqueness. There's also the "
application/vnd.*" subtype, as used in "
application/vnd.ms-excel", for example. But even that lacks a suitable system for uniqueness. (Microsoft gets
then? Can Apple use
vnd.apple-*? How about
vnd.apple.*? Not reassuring...)
Finally, all of these work-arounds bypass the intended hierarchy of MIME. Anything under an "
x-" prefix or buried in "
application/vnd.*" will not be caught by a MIME wildcard like "
image/*", which is meant to catch "all images." Such a match will miss any image that is vendor-specific or otherwise not part of the (small) "officially approved" list of image MIME types maintained by IANA.
So, great, we've just eliminated file name extensions, classic Mac OS type and creator codes, and MIME types as suitable file typing systems. What's an OS vendor to do?
As part of its ongoing (I hope) file metadata penance, Apple has taken another look at file typing in Mac OS X. The situation in Panther and earlier versions of OS X was pretty grim. All of the different file typing systems described above were used in one place or another. But if forced to name a single, "official" file typing system for Mac OS X, it'd have to be filename extensions.
Filename extensions were the only file typing system all OS X applications were required (by the Apple human interface guidelines) to use. Applications could decide to write type and creator codes, and many did, at least initially. As time wore on, fewer and fewer applications bothered.
MIME types were mapped to file name extensions, and sometimes type and creator codes, by applications that needed to deal with MIME-encapsulated or tagged data (e.g., Safari and Mail). But without support for extended attributes, there was no place to actually store MIME types along with the actual files.
There were APIs for dealing with all three kinds of file type data. Despite the de-facto standardization on file name extensions as the one, true (crappy) file typing system in Panther and earlier, different APIs required different file type arguments. Especially during the early days of Mac OS X, the Carbon APIs tended to work with type and creator codes, while the Cocoa APIs dealt mostly with file name extensions. By the time Panther rolled around, most APIs had learned the tricks of their siblings.
Panther actually introduced a fourth, more generic "data typing" system for use with "pasteboard" ("clipboard" to long-time Mac users) data. There was little fanfare and even less documentation. Only developers who actually had to use the APIs were likely to take notice, and to them it probably didn't look much like it had anything to do with file typing.
Well, in Tiger, that oddball pasteboard data typing system has flowered into something much more important: the new One, True File Typing System for Mac OS X. And unlike file name extensions, it doesn't suck. In fact, it's superior to all of the file typing systems described above. Finally, a solution befitting "the world's most advanced operating system."
Apple's new system uses what are called "Uniform Type Identifiers" (with the medically unfortunate abbreviation "UTI"), and here's how they work.
The first thing to understand about UTIs is that, in Apple's words, they "uniquely identify a class of entities considered to have a 'type.'" That's about as broad a description as you can imagine, but in plain English it means that UTIs describe a lot more than just "file formats." They can also describe data (e.g., in the pasteboard or in a stream), structured groups of files and folders (e.g., Mac OS X bundles), symbolic links, hard links, even entire disk volumes. It really is as Apple describes: if it exists as an identifiably typed entity on the system, it's fair game for a UTI.
UTIs set out to solve all of the problems of file name extensions, type and creator codes, and MIME types.
- Uniqueness - As a brand new system for data typing, UTIs are unencumbered by past mistakes. They are defined as being unique, and there is a system for ensuring that they remain so.
- Expressiveness - UTIs have no restrictions on length and have associated human-readable (and optionally localized) descriptions.
- Extensibility - UTIs support a well-defined system for adding new types easily and safely, both with and without the blessing of the organizing body (Apple, in this case).
- Taxonomy - Selecting a subset of data types (e.g., "all images") is straightforward and does not exclude any vendor-specific or ad-hoc UTIs.
Mmmm, I love the smell of ambition in the morning. Smells like... hierarchy.
UTIs exist in a tree-like structure where each node is said to "conform to" its parent nodes. All JPEGs are image files, all images files are data files, and so on. MIME tried to create a similar sort of hierarchy, with all images under the "
image/..." prefix. But as covered earlier, this breaks down quite badly when vendor-specific file formats are thrown into the mix, and the number and scope of top-level types isn't very expressive either.
From my perspective, the primary breakthrough—the "big idea," if you will—of UTIs is that they separate the hierarchy within each individual UTI from the hierarchy of the UTIs with respect to each other. Here are the basics.
UTIs are text strings that can contain numbers, letters, hyphens, and periods, plus any Unicode characters above the ASCII range. Standard types have a "
public." prefix. These are also called "public identifiers," and only Apple can define them. UTIs created by anyone other than Apple must use (surprise) the reverse-DNS naming style. (e.g.,
Each UTI can "inherit from" or "conform" to any other UTI. Below is a sample Uniform Type Identifier hierarchy.
Here we can see that HTML files (
public.html) and plain text files (
public.plain-text) are both kinds of text files (
public.text). In turn, all text files are data files (
public.data), and so on. Again, note that the hierarchy within the UTI names has absolutely nothing to do with where they are in the tree. The UTI
com.mycorp.myspecialtext is a kind of plain text file, but the UTI name itself doesn't begin with "
public.", let alone
Also keep in mind that
public.data is not the top of the UTI tree by any means, nor is there one single root. Any UTI that does not inherit from another UTI can be considered a root.
UTIs can inherit from multiple parents as well, as in the case of application packages (the specially structured folders that appear as single application icons in Mac OS X). Application packages are both "bundles" and "packages."
Applications declare UTIs in their
Info.plist (XML property list) files inside their application bundles. There are two kinds of declarations, "exported" and "imported."
Exported UTIs are made available to the entire system. An application that has its own file format should have an exported UTI declaration describing that format.
Imported UTIs declare UTIs that are not "owned" by the application, but that need to be known to the system in order for the application to do its work. For example, imagine an application that does some sort of post-processing on Adobe Photoshop files. Such an application should continue to work even if Photoshop is not installed on the system.
In order for Mac OS X to, say, handle drag and drop properly for this application, it must understand that the file being dragged is an Adobe Photoshop file. But if Photoshop (which has an exported UTI declaration for the Photoshop file format,
com.adobe.photoshop) is not installed, how will the system know what a Photoshop file is? To solve this problem, the Photoshop file processor application includes an imported UTI declaration for
com.adobe.photoshop, essentially "pre-declaring" the existence and nature of the Photoshop file UTI, even if Photoshop isn't installed.
In the case where an exported UTI declaration conflicts with an imported UTI declaration, the exported declaration takes precedence. This makes sense, since the exported declaration is made by the application that "owns" the UTI. Once it is installed, the imported UTI declaration is no longer needed anyway.
A UTI declaration includes the following information.
- The UTI string itself.
- A list of UTIs that it inherits from.
- A list of alternate type identifiers that are equivalent to this UTI (file name extensions, MIME types, etc.)
- The path to an icon file in the application bundle used to display files of this type.
- An optionally localized human-readable description.
- The URL of a document describing the UTI.
This list of "equivalent" types is used to provide conversion APIs from one type to another. As you might imagine, it's entirely possible that any given file name extension, classic Mac OS type code, or MIME type will resolve to more than one possible UTI. The conversion API accepts "hints" to help pick one (e.g., only consider subtypes of
public.data because I got this file name extension from a file, not a folder or other type of entity). Without any hints, the API prefers public identifiers over others.
In the case where there is no suitable UTI for an item, a dynamically generated UTI is used. These begin with a "
dyn." prefix and exist to ensure that everything on the system has some sort of unique UTI, even if it is only transiently meaningful.
Developers are encouraged to inherit from public identifiers unless the proprietary UTI they want to inherit from is defined in the same application (or, presumably, in another application made by the same company).
Apple has published a list of almost 100 predefined UTIs that cover a huge range of entities. Here are just a few examples.
||-||Base type for all general content.|
||-||Base type for all document content.|
||Base type for mixed content. For example, a PDF file contains
both text and special formatting data.
||Base type for all data.|
||Base type for all text, including text with markup information (HTML,
RTF, and so on).
||'RTF ' .rtf, text/rtf||Rich Text.|
|MPEG-1 or MPEG-2 content.|
||Base type for directories.|
||A file package.|
||.rtfd||Rich Text Format Directory. That is, Rich Text with content
embedding, on-disk format.
As you can see, nothing is safe from UTIs. Apple also includes imported UTI declaration for common file formats such as Adobe PDF, Microsoft Word and Excel, etc.
In what must be the ultimate act of dominance, UTI declarations actually use UTIs to describe the other data typing systems.
||Classic Mac OS type code.|
As a man in an orange Bandicoot suit once said, "Booya, Grandma! Booya!"
It's simple. When it comes to uniquely identifying data types, MIME types, type and creator codes, and file name extensions all suck, and UTIs rule. While MIME types and type and creator codes may be better data typing systems than file name extensions (especially when it comes to storage location), they suffer from many of the same problems. Just look at some of the examples above and note how many different type codes and MIME types mean the same thing: '
MPG ', '
video/x-mpeg. It's not just file name extensions that suffer from uniqueness problems.
UTIs wipe the slate clean and bring many new abilities to the table. The only obvious way that UTIs could be improved is for them to become an open standard rather than a system administered entirely by Apple and used only in Mac OS X. But really, those things would just be icing on the cake. So long as values from any other data typing system can be mapped to UTIs, which typing system other OSes use is of less concern. One would hope that they'll catch up eventually, but in the meantime Mac OS X Tiger has a powerful, extensible data typing system that's well beyond its predecessors.
A great standard is nothing if it is not used, of course. I expect all Mac OS X APIs to slowly migrate towards using UTIs as the preferred means of data typing. In fact, remember when I mentioned that Spotlight uses one metadata importer plug-in for each file type? Well, the "file types" Spotlight uses to match each file to a particular metadata importer plug-in are actually UTIs. Each importer must indicate a single UTI that it is able to index. The UTI hierarchy comes into play when Spotlight chooses the "most specific" metadata importer for each file based on how closely the file's UTI is matched by each of the (possibly several) applicable importers.
Tiger already includes APIs for converting older data types to and from UTIs. Developers can begin changing their application to use UTIs exclusively when making internal decisions about file types. Apple appears to be encouraging this.
Unfortunately, UTIs exist only as a concept internal to applications. Apple has chosen not to take the obvious next step and move the recommended external representation of file type away from file name extensions. File name extensions are, by far, the worst of the file typing systems, but they're also the most commonly used. I've never objected to (and have actually supported) the use of and support for file name extensions in Mac OS X. The terrible mistake Apple made was to bless file name extensions as the "standard" file typing system in Mac OS X.
With UTIs, Apple has effectively reversed at least half of that decision. Internally (in applications and the OS itself), UTIs are now the recommended standard, not just file typing, but for all "typing" in general.
But externally, file name extensions are still the only mandatory file typing system according to Apple's interface guidelines. That's a shame because, with the advent of extended attributes, Tiger has the ability to store UTIs along with every file as the "official" file type identifier. Apple has simply chosen not to recommend that practice.
(Interestingly, type and creator codes are already (redundantly?) stored this way in Tiger. Assigning them to a file results in a new extended attribute key,
com.apple.FinderInfo, with a value that is the concatenation of the type and creator codes. This is done transparently whenever a type or creator code is assigned to a file.)
With the addition of the conversion APIs, all the pieces are already in place for UTIs to be officially blessed as the recommended external file typing system in Mac OS X. File name extensions could then safely be relegated to "optional, but on by default" status. Finally, the file name would be entirely back under the user's control, as it should have been from the start.
All of that remains a dream, or at least my dream. In the meantime, it's nice to finally see some sort of movement on this front in Tiger. It was long overdue. What has been done in Tiger has been done well. Although it's less than I wanted, it's better than I expected. I only hope the trend continues.