iPhone, iPad, Security, and Privacy. Oh my!

Intro

Plenty of people are debating the significance of the iPad. Opinions range from those who believe it represents the future of computing, to those who think it is just a toy or a diversion. There are lots of aspects to these new types of devices, and any discussion about their impact that would necessarily look at a lot of different issues.

Personally, I find the iPad very interesting for many reasons, though most of them are probably different than what excites the average consumer. One of the aspects I find most exciting is that the iPad (and iPhone) represent a new platform that has been designed from the ground up in such a way that they can avoid a number of the security problems that have plagued computers in past, problems that cannot be fixed because legacy operatings need to support legacy applications that cannot be made to work securely. This is a key advantage of these new platforms, but it is also one that could easily evaporate if Apple is not careful as it designs and implements new APIs.

In this post I am going to walk through a brief history of computer security, an explanation of why iPhone OS can be more secure than Mac OS X or Windows, describe some API flaws in iPhone OS 2.x/3.x that reduce user security, and explain potential exploit vector in iPhone OS, and ways they can be fixed. Finally, I will take a quick look at some of the features announced at the iPhone OS 4.0 event, and their potential security and privacy implications, though I will not discuss their actual APIs or any specific analysis I have done.

It should be noted, that I do not, and have never, worked professionally in software security. Having said that, everyone who writes software needs to be concerned with these sorts of issues and have some expertise with them, especially people who deal with any potentially sensitive data. It should probably also be noted that most of the issues I am about to raise are not security issues in the traditional sense (they are technically privacy issues), but to the end user they are the same thing. It doesn't matter whether a nefarious application gets personal information by exploiting a bug, or because the system was designed to let applications get that data.

In the beginning

When the first computers were built, security was a non-issue. Early computers took teams of people to operate, could only input and output through switches and lights (and slightly later terminals), could only run a single program at a time, and that program was generally written by the people operating the machine.

Over time the machines were enhanced, but for decades they were very expensive. As any buisness person will tell you, if you have an expensive fixed asset sitting idle you are wasting money, so people needed to find ways to keep those machines utilized. The first common way to do this was through batch processing. With batch processing you would write a program, and send it to the computer. When the computer finished one program the operators would take the next program from the queue and run it immediately. This had the benefit of keeping the computer busy, but it had some pretty substantial downsides as well. The most obvious one (in hindsight) was that it was impossible to deal with the computer interactively. If your program had a bug, you couldn't just fix your program while it was on the computer and rerun it, you would wait until you got the batch results, fix the bug, and put it back in the queue, and wait another day until it was scheduled to run again.

The benefits of interactive computing were pretty obvious once it was technologically feasible, but that still didn't solve the cost issues. A new way of sharing computers had to be invented: Time-sharing. In a time-sharing system, the computer runs multiple programs at once, and keeps switching between them. If a program isn't using its timeslice (because it is waiting for the user to type something, for instance), it can yield its time to other programs. One of the first such systems was MIT CTSS. CTSS was arguably the first OS to resemble what we now call a modern OS, and among other significant achievements, it hosted the first known electronic mail implementation, the first interactive shell, was the first computer to run background only daemon processes, and the first system to use a virtual machine to support legacy applications. In fact, you can track iPhone OS's lineage back to it: iPhone OS -> Mac OS X -> BSD Unix -> AT&T Unix -> Multics -> CTSS.

One of the other things that became obvious once time-sharing came into being was that it changed fundamental assumptions about computers, and the programs they ran, worked. In order to even demo the first time sharing code the IBM 704 at MIT required hardware changed to support interrupts. Before CTSS was implemented several other modifications were made to allow for memory relocation (early virtual memory), and memory protection. This changed what had previous been invariant assumptions about the environment in which a program ran, and resulted in CTSS having a slightly more dubious distinction. CTSS wass the first operating system to have a known software security issue.

Enter the sandbox

Until that point, all software had been written assuming it was the only software running, and programmers had never had to consider the issues involved when multiple programs ran. It was clear that some sort of mechanism to isolate users would be advantaguous, but since the segmentation and protection mechanisms were retrofit into an existing piece of hardware they were not necessarily as flexible or well thought out as one would hope.

The relocation and protection features IBM added to the 704 allowed CTSS to implement the first ever sandbox. A sandbox is a small virtual world within a computer. The important thing to understand about the a sandbox is that once something is in a sandbox, it cannot get out of the sandbox (absent a bug or design flaw in the OS). Unfortunately, since those features were retrofitted into an existing (deplyed!) piece of hardware they were not as flexible or well thought out as they might have been, which meant there were ways in which to breach the sandbox. Despite those design flaws, the security issue that occured was not due to breaching a sandbox. It was instead due to a race condition that existed because of the assumption that only a single program would run at the same time, and an administrative decision allowing two users to play in the same sandbox.

What happened was that there was a single system account that multiple people needed to login to in order to do system maintenance. When two admins both logged in, they were both inside the system sandbox. In addition to that, the editor software assumed only one copy of itself could be running at a time, so it stored its temporary data fixed location. One day two people were logged in doing two unrelated system tasks, one was editting the message of the day, the other was modifying user account information by editing the system password file. Since both of their editors were using the same temp file, the system system password file ended up being written out as the message of the day, and everyone who logged in could see everyone else's passwords. This incident also resulted in the idea of using hashed passwords.

Evolution

After CTSS, a number of time-sharing systems came into existence, but given iPhone OS's lineage, it is best to look at how Unix handled this situation. Unix was an OS inspired by Multics. By the time it came into existence hardware had evolved enough to support isolating user processes in a fairly robust manner. The world was still pretty different back then. Computers will still expensive, they still tended to have dedicated administrators, and the amount of software available for them was still pretty small compared to today.

While the notion of a hostile user certainly existed, most of the software you were running was likely to have been provided as part of the AT&T Unix distribution (and later the BSD Unix distribution), or written in house. Networking still wasn't prevelent. As a consequence, when Unix was designed the goal was to sandbox off the users from each other, but no effort was made to isolate any of a user's data from a program they themselves ran. This is a completely different than the environment of the average user today, where most systems only have one user accessing them at a time, and are running lots of third party code of unknown provenance.

Mac OS X is a modern version of Unix, and as such it inherits Unix's basic security design. The result is we have a system which has great support for isolating things into seperate sandboxes, but for the most part everything runs in a single sandbox (the user's account). The unfortunate thing is that most programs need to run in the same sandbox, because the way the Cocoa API's are designed they won't function properly if they are not. In many cases the necessary API changes to secure things are difficult and incompatible with existing applications. Since Apple has no way to analyze all existing software, force developers to update their apps, or provide compatibility without opening up the exact same holes they would want to close, they can't fix them with long difficult transitions.

The situation on the iPhone is far better, but there are still some very serious issues. iPhone OS shares much of OS X's code, but it is a new platform that does not need to maintain compatibility with all the software on Mac OS X that depends on now invalid security assumptions. iPhone OS can fix them without the pain of breaking old software, but if Apple is not careful it can just as easily make the same mistakes again.

The CoreLocation and Address Book APIs

So lets look at two APIs available on iPhone, the CoreLocation API, and the Address Book API, as an example of where iPhone has screwed up its security, and an example of where iPhone has gotten it right.

CoreLocation

There are obvious privacy concerns with applications being able to determine a user's location. Apple handles this from a user perspective by asking the user if an application should be allowed to access their current location. What happens under the hood was that if CoreLocation access is approved by the user, CoreLocation signals over to a background service (running its own sandbox) asking for the location information. That service would check the system preference, confirm the app asking for information was allowed to have it, and send it back.

That is not to say that CoreLocation was perfect in iPhone OS 2.0. While the basic design was right, it turned out that Apple actually cached location data inside the application's sandbox in a way that allowed it be accessed using private APIs. As a result, it was possible to get a (sometimes stale) location without the user knowing. Apple stopped caching the data, and now all CoreLocation access is gated messaging the background service. Fixing this particular bug did not break any apps that were using the CoreLocation API, since the API has always been able to return an error saying there was no access, so all apps using CoreLocation have always had to deal with the fact that sometimes the user does not give them access. Obviously the apps using the private APIs broke, but that is a separate issue.

In iPhone OS 4.0 this is being greatly improved, so much so that the improvements were demoed in the keynote. In 4.0 there will be a status bar indicating that an app has recently used CoreLocation, it will be possible to look at the list apps to see what has used your location in the las day, and turn on and off their access. All of this works without changes to the existing APIs, so all existing apps the CoreLocation will be effected, resulting in much better security of the user's location information, and the ability to notice and identify if something is using the data without your consent. For more details people can watch the event stream.

AddressBook

Now lets look at the address book. The AddressBook API is basically the opposite of CoreLocation, in that Apple completely botched the security of the API, and the implementation, on multiple levels. The mistakes are so deeply ingrained that they cannot be fixed in any meaningful way without break a ton of apps. For that reason, the only viable options are to leave it broken, or fix it incrementally over several releases.

The most obvious failure is that there are no controls on what can access your address book. Any app can read the address book, and there is no visible indication to the user, no log of the events, and no way to turn it off. In addition to that, all software currently using the address book has an implicit presumption it can read and write to the address book. This has actually caused fairly public incidents, such as the original Aurora Feint incident. In that case it was noticed by people monitoring network traffic of the app after it had been approved for the store, and the app wasn't actually trying to do anything malicious. An app actually trying to steal that data could encrypt its transmissions, and hide what it is doing in such a way that Apple would not notice it during the approval process. Fixing the API is not simple, because if the existing API were changed such that it could return an error most apps would be unable to cope with it, and either crash or behave unexpectedly. An option might be fore the OS to return an empty address book to such apps, but that could cause some serious problems for users as well.

Even ignoring the API related issues, the infrastructure for restricting access to the address book doesn't exist. Rather than keeping address book database in its own sandbox with a daemon to arbitrate access, Apple exposes the address book database (/var/mobile/Library/AddressBook/AddressBook.sqlitedb) into all sandboxes and the API they provide directly accesses it. That sounds similiar to the above story about CTSS and the motd, though we have 40 years more experience and address book handles concurrent access to the database just fine. In an environment with no malicious apps that would be fine, but that is not the environment we live in. By exposing the address book database in this way not only has Apple introduced a way for applications to read all of the address book data without going through the AddressBook API, they have also exposed an attack vector for applications to insert poisoned address data that can be synced back to other device, or even attack other applications inside their own sandboxes.

By carefully constructing a malformed address book database file it may be possible to exploit bugs in either sqlite3, the AddressBook API, or the code of a specific application stack in order to corrupt their stack. At that that point it is possible to run arbitrary code using a return-to-libc payload, despite countermeasures like an non-executable stack. All the app has to do is overwrite the users address book database with the new one, then the next the user runs a targetted app that accesses the database the exploit can occur. To be fair, sqlite3 is heavily vetted against malformed files, but that is no guarantee that this is not possible, and by directly accessing the database Apple has added an potential exploit vector that just doesn't need to be there.

What I want to see in the future

Well, the first thing I want is for Apple to expose the same sort of UI for managing AddressBook as exists for CoreLocation. I want notification when things are accessing it (especially when apps are writing to it). Furthur more, I want it to be possible to turn off an app's access to the address book, despite the potential for current apps to break, since I would rather risk the app crashing then it stealing information or inserting information. I want the API expanded to support read only and read/write access. As a compatibility measure Apple should default to permissive access to the address book for 4.0 using the existing APIs (that way no apps would break unless the user explicitly denied access). In addition, Apple should start testing all 4.0 apps submitted to make sure they operate correctly when address book access is denied. That way, when 5.0 comes out all apps submitted in the previous year will work correctly if they change the default to asking the user for access and the user tends to deny access.

I also think there is room for some significant improvements to the UI features added for CoreLocation privacy in 4.0. Aside for the basic UI for notifying users about access, the system should log accesses somewhere that is accessible via iPhone Configuration Utility. Also, if Apple adds the ability to control address book read and write access there may be a need to rethink how to layout the UI in the Settings app, as having seperate table views for Push Notifications, Location, Address Book (read), and Address Book (read/write) is excessive and unintuitive. It would probably better to have a single app security table where each app should a synopsis of what it was trying to use, and could be expanded in order to inspect them more closely and edit them. Ultimately I trust Apple's abilities to create a UI that can handle presenting the necessary information to the user, that is the sort of thing they excel at.

While I am at it, I also want applications to state up front what they want access to, rather than asking me as they access them. This could be handled by embedding it as metadata in the application's plist. That way the first time an application attempts to access anything that requires permission the OS could bring interface for approving everything at once (instead of multiple dialogs as accesses to different services happen).

Why did I decide to write this blog post

I decided to write this blog post because I think iPhone OS is at a crossroads, where it is either going to end up with up as a much more secure platform then we currently have, or it could end up throwing away all of that progress. This is especially important if the iPhone OS ends up being the basis for most of my computer usage through products like the iPad. Its fundamental sandboxing design allows it to be much more secure, but in order for that to remain the case Apple has to carefully vet every API that allows access to user data, and provide the user ways to control that access. At this point there are a fairly limited amount of data exposed to applications:

• Location
  • Well secured
• Address Book
  • Insecure
  • No worse than on Windows or Mac OS X, but it could be better
• iTunes database metadata
  • Insecure
  • Not particularly sensitive
• Camera access
  • Currently imited to Apple provided UI (see below)
  • Can be globally turned off via iPCU).

The thing is, Steve announced some very exciting features for 4.0 that have potential privacy concerns. As you can see from my artistic rendition below, there are 3 features I want to call out. They are:

• Calender
  • If knowing where I am is a privacy concern, then knowing where I am going to be certainly is)
• Photo Library Access
  • Apps stealing your photos and uploading them to a server is a very real privacy issue, just wait until apps can read all the photos and movies you have taken without going through the OS's UI.
• Raw access to the camera data
  • This a particularly nasty one, since iPhones do not include visible indicators that the camera is turned on. A similiar issue has been the subject of a recent lawsuit in the Lower Merion School District. In that sort of case the situation is even worse, because companies and schools can create their own applications under the iPhone Enterprise Program, that don't have conform the public API or be vetted by Apple, eliminating any safety derived from Apple's approval process.

It is important that all of these APIs are designed in such a way that they can expose restricted access to apps, since if apps can simply pull all of this data with no restrictions then we have effectively given up a lot of the value of the sandboxes. After all, who cares that apps can't get out of their sandbox when all the user's personal data is right there for them to play with. Apple needs to use their sandbox not just as way to protect the integrity of the OS on the phone, but also to protect the privacy of the user's data.

(For any Apple readers, please check out these 6 radars).

Its all about the framework…

Section 3.3.1

There are a lot of posts right now about Flash and iPhone. It is pretty heated, a lot of people have very real amounts of time and money tied up in this. I am an iPhone developer, so I am not going to reveal anything covered under NDA, but the text of the relevant license changes has been widely disseminated. Based on my reading of the aforementioned text, I have a few opinions.

In my opinion this is not purely aimed at Flash, but it is certainly precipitated by Flash CS5. I can't imagine Apple is happy about environments like MonoTouch, Unity3D, PhoneGap, Appcelerator, or Corona, but I am doubtful they would have changed the license in this way just to stop developers using those environments. Some of the people using those environments actually claim they approve of Apple blocking Flash, and are complaining that their tool of choice is merely collateral damage. I think that is very naive, Apple probably doesn't want these environments, they just weren't motivated to stop them until it was clear Adobe was going to pursue Flash apps on the iPhone despite Apple's wishes. They are not collateral damage, they are opportunistic targets that Apple decided to kill once they decided they would have to take action to stop Flash as an iPhone runtime environment anyway.

The more interesting thing from my standpoint is that this makes it a license violation to include a language interpreter inside a game. If you aren't a game developer you might not be familiar with how large games are structured, but most games consist of a game engine, which is high performance code for doing things like rendering graphics, and an interpreter which runs the game logic (determining how sprites move, determining when to pop up in game text boxes, etc). This is how practically every commercial RPG works, as well as many (most?) other types of games. This affects major app store publishers, like EA, Gameloft, Tapulous, and ngmoco:). Looking at the top ten lists on the app store right now I see several titles that I know have embedded Lua interpreters. In this case I think these apps are genuine collateral damage, though I honestly doubt Apple would attempt to enforce the clause against them. In fact, using an interpretted language for game logic is already technically in violation of section 3.3.2 in the current agreement, though many developers may not realize it because under the original agreement it was okay, and the change that made it verboten was very subtle (changing an "and" to an "or"). I am actually not sure exactly when that changed, and only noticed it myself while I was researching this blog post.

Why doesn't Apple like 3rd party runtime environments?

People have speculated about why Apple doesn't like these 3rd party runtimes. The general conclusion is that Apple craves control over the user experience. While they do want control over the user experience, that is generally not a reason to object to 3rd party tools. After all, the majority of developers using these tools are using them to write games, many of which present their own user experience written completely in OpenGL even when using Apple's tools.

What Apple does care about is their ability to control their own development cycles. iPhoneOS runs on extremely tight schedules, with a very high degree of secrecy, and at a pace completely controlled by Apple. I know it is popular to claim that maintaining binary compatibility is easy, that is the argument du jour made by people claiming Apple should just support developers using private APIs. Well, they are just wrong. Ask anyone who has been involved with a couple of releases of Mac OS or Windows about the amount of effort involved in keeping old apps working, especially those using private APIs. There is a reason why the majority of current and former framework engineers who comment on the issue come out really strongly against any use of private APIs. To really delve into what it takes would be an entire blog post.

So, if you will indulge my claim that backwards compatibility is hard (even absent the private API issue) it is pretty easy to see why supporting other runtimes is ceding a lot of control to a 3rd party. Imagine if 10% of the apps on iPhone came from Flash. If that was the case, then ensuring Flash didn’t break release to release would be a big deal, much bigger than any other compatibility issues. Since Apple doesn’t have access to Flash CS5’s runtime library code or compiler frontend, they might be put in a position where they would need to coordinate with Adobe to resolve those issues. Shipping a new release where Apple breaks any specific application, even a top seller, is not an issue if the release is compelling, most apps work, and Apple has the option of working with the vendor to help them fix their app. Shipping a release where they break a large percentage of apps is not generally an option. Letting any of these secondary runtimes develop a significant base of applications in the store risks putting Apple in a position where the company that controls that runtime can cause delays in Apple’s release schedule, or worse, demand specific engineering decisions from Apple, under the threat of withholding the information necessary to keep their runtime working.

This isn’t some perceived risk, I can think of incidents where Apple reverted OS changes, dumped new APIs, or was forced to committing massive engineering resources to something it did not want to do because a Must Not Break™ app vendor told them to. Apple does not want to give anyone that sort of influence over them. So ultimately, preventing Flash on the platform is about control, but is not control over the user experience of the Flash applications, or even the languages used. It is about the runtimes they bring on to the system, and Apple's control over future releases of iPhone OS.

So who is to blame?

It is easy to blame all this on Apple, and you will find no end of blogs screaming about their monopolistic power-hungry tendencies. I certainly agree that Apple should probably be more open, and that they are the party with the power to resolve this. If people want to complain about that, you will not hear me defending Apple. The developers using Flash, Unity3D, and MonoTouch have my sympathy, and I understand their anger with Apple. The Adobe evangelists writing screeds get none though.

The reason is that I think Adobe holds much more of the blame. Adobe is a large company with a significant, and complicated, relationship with Apple. They have frequent high level contacts and meetings. Adobe has known for quite some time about Apple's desire not to have Flash on the iPhone. There is no doubt in my mind that if they asked Apple to bless this they were rebuffed, and if they didn't ask the only reason they didn't was because they knew Apple would say no. In either event, they announced the product to their customers and sold them on an idea they were not in a position to deliver, hoping Apple would be unwilling to piss off developers by not fulfilling Adobe's promises. They tried to force Apple's hand by putting Apple in a position where in order stop the Flash they would have to do it publicly in front of Adobe's users. That was a bad call on Adobe's part.

Personally, in this whole thing the most distasteful part is that Adobe used its userbase and their livelihood as a bargaining chip. These kinds of high stakes negotiations have happened in the past many times. They are much more common than people think, and until the last few years Apple was more likely to be on the weaker side of the negotiation. The story of MacBasic is a classic example, but I can think of other (not publicly disclosed) incidents involving Adobe and Macromedia (which was acquired by Adobe, and is where the Flash team comes from) applying extreme pressure to Apple. This is the only case where I feel an active user community was publicly jerked around like this in order for one side to try to gain leverage over the other. That is saying a lot, because I am not pleased with Apple's actions either, but Adobe put Apple in a position where either Adobe got its way or Apple screwed developers.

What Adobe can do

If Adobe actually wants to persuade Apple to support Flash on iPhone (either as a plugin or compiled to native apps), I know how they can do it. They can get an awesome, high performance, Flash environment working on Android, and get a bunch of great Flash apps running on Android phones. As much as Apple wants to control iPhone, I am willing to bet they want to beat Android more. Currently Adobe is asking Apple to put Adobe in a position where they wield influence over Apple, in exchange for the promise of apps in the future, apps that Apple thinks will be low quality. That is a bad deal. On the other hand, if Adobe proved the apps were high quality by deploying them on competitor's platform, and was offering a library of existing high quality apps that neutralized another competitor's advantage, then there is enough value that Apple probably could be influenced.

I hear that Flash Player 10.1 is the real deal, and it is going to work fairly well on the HTC EVO 4G. This is an oppurtunity window that is quickly closing. If Adobe had delivered a decent solution two years ago it would have carried a lot more weight, but at this point Apple has managed to cultivate its own 3rd party apps, and convince a lot of major sites to support html5. Every day Adobe does not have a widely deployed mobile Flash, is a day they are not having Flash based mobile apps developed, and is a day the odds of mobile Flash being successful goes down a little bit.

If Android was making significant gains against Apple, and all of its best apps were Flash based, then Adobe could offer Apple access to all of Android's best apps, which would give them a lot of power. The fact is that there have yet to be any widely deployed Android phones that support Flash. That's right, Adobe has been making the case for Flash on iPhone for 3 years, but still hasn't deployed a non-lite version of Flash on any phones, even when Apple is not obstructing them.

What I want from Apple

I am comfortable with what Apple has done with Flash, it doesn't change my view on the platform. It sucks for some developers, but there are not a large number of deployed apps out there. If I were Apple, at this point I would not want to allow Adobe’s tools explicitly because of how they have acted. If this is the kind of crap they try to pull when they don't have influence over Apple through being a major toolchain vendor, imagine the kind of crap they could pull if a substantial portion of the apps on the store were dependent on them. Note that this has nothing to do with the performance or features of the apps themselves.

I would like Apple to find a way to work with the existing vendors who have (for the most part) been good citizens, and find some way to let them continue. As I said above, I suspect Apple doesn't like their runtimes on the iPhone, but there is still some value in those platforms and I would like Apple to find some way to support them. I think Apple can probably achieve the control it wants over the runtime, and allow for more language flexibility than current license allows.

Apple should also find some way to loosen the license to officially allow games to use included interpreters for their internal logic. I don't know how to do that without opening up loopholes, but that is what lawyers are for. If they don't do that then effectively they are just using their discretion on a case by case basis, which is completely legitimate, but causes significant uncertainty for developers. Ideally Apple should have some provisions for adjunct or secondary runtimes that let developers use internal scripting. Failing that, Apple should include some language interpreters on the system that games can use for their app logic. While not as flexible as letting games include their own interpreters, if Apple exposed JavascriptCore outside of UIWebView, and included a Lua interpreter in the base system it would solve this as practical a issue for many games.

Oh, and while I am at it, I want them to do a rapid development tool aimed at light apps. Having a rapid development tool that could be used by non-programmers would probably quench some of the desire for Flash, which is largely driven by the ability of designers to actually create functional apps without significant programming resources. While we are at it, it should run right on the iPad. In other words, I want "HyperCard Touch." And a pony.

New theme

I have changed the this blogs theme since a number complained that it was hard to read light text on a dark background. I am still fiddling around with them template, so if something is broken, that is probably why.

The loss of ZFS

Well, in case you haven't read any of the myriad stories about it, it appears that Apple has decided not to use ZFS on Mac OS X. Gruber has sources that say it was primarily licensing concerns, which is consistent with what people have implied to me, both recently, and around WWDC (although at that time I think there was probably still hope of resolving the issues).

Now, some people jump may comment that it couldn't be licensing issues, since ZFS is opensource (under the CDDL), and that Apple already uses CDDL software (DTrace). That may be true, but often in deals that involve large companies there is more to it than that. Apple may have wanted guarantees of indemnification in the NetApp lawsuit. Maybe it wanted guarantees that certain modifications it wanted to make would be accepted upstream, or even to get Sun to make certain changes. It also might have wanted additional distribution rights that were not granted under the CDDL. It is typical for companies to negotiate custom agreements in such cases (and for some money to change hands), so the idea that licensing issues are why it fell through is entirely reasonable, even though it is an opensource product. Obviously Sun's steady decline in the market place, and the uncertainty caused by the Oracle acquisition may have greatly complicated any such negotiations.

Why not do a new filesystem?

Apple has a lot of talented filesystem engineers. They are certainly capable of doing something comparable to ZFS, at least for their target market. The problem with developing a new modern filesystem is that it generally takes longer than a single OS release cycle. Most companies are really bad at having large teams focused on projects that will not ship in the next version of the project they are working on.

This is a particularly acute problem at Apple, which traditionally has done things with very few engineers. I don't want to get into exact numbers, but I recall having a discussion with the head of a university FS team who was discussing the FS he was working on. He was pitching it to a group of Apple engineers. It was some interesting work, but there were some unsolved problems. When he was asked about them he commented that they didn't have enough people to deal with them, but he had some ideas and it shouldn't be an issue for a company with a real FS team. It turned out his research team had about the same number of people working on their FS as Apple had working on HFS, HFS+, UFS, NFS, WebDAV, FAT, and NTFS combined. I think people don't appreciate how productive Apple is on a per-engineer basis. The downside of that is that sometimes it is hard to find the resources to do something large and time consuming, particularly when it is not something that most users will notice in a direct sense. That is especially true if senior management is not excited about the idea.

Because of that, I was fairly convinced ZFS was a credible future primary FS for Apple. Not because it was an optimal design for them (it isn't), but because it was a lot less work than doing a new design from scratch. The fact its fundamental architecture is 20 years newer than HFS meant it would still be better than HFS+ in almost all respects even if it was not designed for Apple's exact needs. Clearly I was wrong, since Apple has stopped the ZFS project.

What changed?

Well, a couple of things have happened. The first is that Mac OS X has gotten more mature. They no longer need to port all of those FSes, they already have them working, and in most cases they work fairly well. That frees up some engineers. Apple has also greatly expanded the number of people working on their kernel since it is amortized over many different products (Mac OS X, iPhone, AppleTV, etc).

Suddenly the notion of doing a new filesystem seems doable, so long as it is a real priority and the FS team doesn't get pulled to keep adding features or doing major work to legacy FSes. That is still a lot of work when Apple had ZFS approaching production quality on OS X.

Apple can do better than ZFS

Sun calls ZFS "The Last Word in Filesystems", but that is hyperbole. ZFS is one of the first widely deployed copy on write FSes. That certainly makes it a tremendous improvement over existing FSes, but pioneers are the ones with arrows in their back. By looking at ZFS's development it is certainly possible to identify mistakes that they made, and ways to do things better if one were to start from scratch. From where I sit, there are 3 obvious ways doing a new FS will be better for Apple than ZFS:

1. There have been new fundamental research since ZFS was designed that simplifies many of the issues involved with it. In particular the "B-trees, Shadowing, and Clones" (PDF). That paper is the basis for the design of BtrFS, which has a very similar feature set to ZFS, but internally is entirely different. LWN has an article about BtrFS that explains the significance in some detail (it is written Valerie Aurora, who worked on ZFS at Sun).

2. ZFS was designed for the storage interfaces available a decade ago. Spinning disks are going to be with us for a long time, especially for bulk storage in data centers and on backup devices. The future is all about solid state. Flash SSDs have significantly different performance characteristics than spinning media, and there may be FS design decisions one could make that would benefit from that. Now, any FS Apple designs will have to work acceptably on traditional drives, but if they are designing for the future then flash is what to target.

   ZFS has had some optimization work for flash, but it is all in terms of using flash as part of a storage hierarchy. That makes complete sense, since ZFS's primary deployment targets are high-end systems and data center storage. Those systems have multiple drives, so the idea of separate flash drives for a ZIL and L2ARC are completely reasonable. Most consumers have one drive in their system, and maybe an external drive for bulk data, data exchange, and backup.

3. That brings up the last point. ZFS is designed for big systems. It works on small systems, but most of the tradeoffs favor very large computers, with lots of drives. This shows up in a number of ways. The first is that ZFS is not currently capable of adding single drives to an existing vdev or migrating vdevs between various types (mirror, raidz, raidz2). This is a major feature for smaller users who might want to add a single drive, but is a non-issue for data center users who tend to add large number of drives all at once, since they will add whole vdevs. Another issue is that ZFS assumes you have a lot of ram. NEC has been doing a port of OpenSolaris to ARM, and they determined they could not get ZFS to use less than 8 megabytes of ram without making incompatible format changes (Compacted ZFS). With those changes they could squeeze it into a more reasonable 2 megabytes. On a desktop that doesn't seem like a big deal, but on an iPhone 3G or a Time Capsule 8MB of wired memory is an enormous issue.

The only major downside is that if Apple is just starting on a next generation FS now it could be a long time before we get our hands on it.

But now we are going to have another incompatible next generation filesystem

Wolf brought this point up during some of the ZFS talk on twitter yesterday. My general opinion is that it doesn't matter. People use drives for two largely unrelated tasks. One is running their computers. This is fixed storage. The other is for data exchange. In the old days people used floppies for their sneakernet media, which made the situation much simpler to understand. In recent years the market realities have caused people to move to using SD cards, thumbdrives, and hard drives as the exchange medium of sneakernet.

The important point is that understand is that while the physical devices may be the same, the use model is different, just as the using a floppy disk and an internal hard drive were different. Nobody would balk at the notion that floppies should use different FSes than internal drives. Likewise, most people shouldn't care that their external drives are formatted differently than their internal drives.

There are complicated features you want for your boot drives and system disks. Ideally you could have them on your interchange disks, but there are other features that are more important, particularly interoperability, and simplicity. ZFS didn't bring either of those. There might have been a few people who were psyched to be able to use ZFS to share disks between a Mac and a Solaris or FreeBSD box, but honestly those people are few and far between. Whether Apple used ZFS or something else it is just as interoperable with Linux and Windows (which is to say, not at all). So that fact that Apple looks to be doing a new FS does not impact interoperability in any real sense.

The other feature you really want for an interchange FS is simplicity. There are a lot of devices out there that use an FS to communicate with a computer. The simplest example is a digital camera via its media cards, but there are many others. Something like ZFS is way too complex for those devices, and honestly most of the features of ZFS like multiple drive support and snapshots are useless since the devices don't have the physical interconnects or user interfaces to expose those features. There is certainly an argument to be made that we could use something a bit better than FAT32 or exFAT as that format, but ZFS was not the right solution for that.

In other words, for that disk you want to use as an external drive to drag between computers you don't want something like ZFS, you want something that is simple enough that a firmware engineer can write a read-only implementation from the specs in less than a week. For the disk embedded in your computer (operationally or literally) you want something like ZFS, but it doesn't matter if it is interoperable with anything else because you won't be moving it between systems.

This is basically how Windows works. Microsoft generally uses NTFS for internal drives, but FAT for external drives. Ultimately somebody should design a filesystem explicitly for use as an interchange format and license it for free, then everyone can deal with their internal FSes and do what makes the most sense for their OSes and markets.

Flash on the iPhone

Update: One of the developers of Trading Stuff posted a comment below, and has a great blog post about his experience getting it running on the iPhone.

There has been a lot of discussion about running Flash apps on the iPhone over
the last few days. It was precipitated by Adobe's announcement that Flash
Professional CS5 would have support for publishing apps as iPhone native
executables. They went into a little more detail, saying that they were going to
use an Ahead of Time (AOT) compiler backend based on LLVM, and that
there are already several apps on the store using it. This generated a large
number of responses from various people, some knee-jerk, some well
reasoned out. Of course, the fact that there are samples we can dissect means
that it is possible to make some informed analysis about them.

Personal views

Before we get into the technical details of this, let me go into my background a
bit, lest I be accused of being biased or having an agenda. I have around 10
years of Objective C development experience, and almost no experience writing
anything substantial with Flash or ActionScript. I am also primarily a Macintosh
user, where the Flash experience (even in the browser) is often less than ideal.
I am told it is a better experience on Windows. I do use the Hulu Desktop app, which is written in Flash, and think it is pretty nice (though
they should make cut and paste work in their text fields, grrr). Of course,
something like Hulu is an immersive app that has no need to integrate with the
native OS experience, but neither do most games.

There are a lot of neat little web games and what not that are written in Flash,
that I would like to run. If that was the tool the author felt it best to
express their ideas and it worked for them then great. In particular, for
software that I don't feel needs interface with OS and which can use a
completely custom UI (in other words, games) I think there is no difference to
the user. So long as the environment generates good code that can run at full
frame rate without killing the battery I am in favor of getting Flash apps to
run on the iPhone.

Warm up

Okay, so first off lets look at what we know. Adobe is using the LLVM arm
backend to generate code. Right off the bat that gets me bit worried. Don't get
me wrong, I love LLVM. Hell, I was one of the people who wrote
the LLVM ppc backend. Having said that, I wouldn't use the LLVM arm backend at
this time. The reason I wouldn't is because every time I have asked the
people who are actively working on it they tell me it is not ready for prime
time. That is the reason why LLVM-gcc and clang are not supported compiler
targets for iPhone, despite them being supported (and encouraged) for development
on Snow Leopard. Apple has a lot of compiler engineers and has basically stated
that LLVM is their future compiler direction, so if those guys are telling us to
hold back, then how is it good enough for AOT Flash?

Now, Adobe could potentially have another arm backend they developed, or maybe
they branched off a particular build and have fixed whatever bugs would impact
them while ignoring stuff they didn't need. It is entirely possible that they
could have reasonable code coming out of this thing, and I don't have access to
the toolchain itself to inspect it, but it definitely gets me nervous. This may
also be one of the reasons why they are not ready to widely release the tools
yet.

Lets get to it

Okay, so I downloaded one of the games that was available, Trading
Stuff, decompressed its IPA and had a look inside. At first
glance it looked like a pretty normal iPhone app. Then I noticed there were no
resources besides a basic MainWindow.nib. No images, no sounds, no
localizations. The next thing I noticed was that the binary was ~13 megabytes,
or approximately ~95% the size of the entire app. That is enormous for a binary.
For reference, compare that to a normal iPhone game, like The Oregon
Trail, which is ~106 megabyte game has ~1 megabyte executable, or
about 1% the size of the app.

What is going on is that the Flash build environment is not using any of the
standard Mac OS X/iPhone OS bundling or localization mechanisms. Instead they
are transforming all their assets into embeddable objects and shoving them
directly into their application's TEXT section. At first glance that might not
seem so bad, but it has a bunch of consequences. It defeats almost any sort of
caching or prefetch logic the OS has for specific data types (like images), and
instead places all of the pressure directly on the VM and paging subsystems.

To be clear this is no way violation of the SDK agreement, and embedding objects
into an app in this way is occasionally appropriate, but the degree to which it
is happening with these cross compiled apps is different, and likely will have a
number of significant (negative) performance implications.

Now, moving on from the obvious, its time to actually start poking at the
generated code. If we just take a cursory look at the linkage, we can see some
bad stuff going on here. Why are they calling dlopen, dlclose, and dlsym? The
only reason to use those is load in frameworks and resolve symbols after launch,
something that is strictly verboten. In the best case that might be some dead
code they use from debugging that should have been stripped that should have
out. In the worst case, they depend on them to get access to symbols they are
not supposed to use. I want to be clear about this: There is no legitimate
reason why any app that follows the terms of the SDK agreement should use these
functions, and I find it shocking that Apple lets apps that link against them
on the store at all. I should also note this is not exhaustive, those are just the most obvious things, but in my cursory inspection I saw a dozen or so objective C selectors that I believe are private.

Actually that brings up another point. Despite the app developer doing nothing
wrong, one of their toolchain or middleware vendors is doing something that could
be an issue. When I write apps for the store I might choose to play fast and
loose with something if there is a compelling reason, but if I am providing a
library for someone else I never do. Using private APIs is putting your
customer's apps at risk. Not only do I find that unacceptable,
but I think any vendor who does that is generally irresponsible and makes it me
hesitate to use any of their other products because I feel it shows a certain
casualness about how you treat their customers. If there is a legitimate reason
you need to do something that risks your customer's products, then as a company
you need to disclose it so your customers can make an informed decision.

I should note this is not an intrinsic issue with Flash, I know for a fact
certain major vendors ship iPhone libraries that call APIs that can get your app
rejected from the store without informing developers. For instance, various analytics companies
really shouldn't be poking at private APIs to try to find cached location
framework data. It isn't just a privacy breach, it places your clients apps at risk.

So what. How does it run?

On my iPhone 3G it runs really choppy, on my 3GS it runs acceptably, but it
still isn't smooth. Given the OpenGL performance people have seen on the 3GS
that is still pretty bad. I have not done any invasive tests by instrumenting
the binary, that is just what I can get via basic usage. The sad thing is that
there is no reason it has to have performance like this. This is not an inherent
issue with the ActionScript used in this app (though that may have issues), it
is that what is coming out of the toolchain is a huge, monstrous binary that stresses
the runtime and has performance characteristics completely different than
anything the iPhone is currently setup for.

Also, remember, the slower the frame rate the more work the phone is doing per frame, and the
more battery the app is using. When you see an app that can do 120 FPS in its
demo loop, that means that when it is running at 30 FPS it is using ~25% of the
CPU/GPU assets. When you see one that can only get 20 FPS that means it cannot
hit 30 FPS to clamp at despite maxing some or the costly (in terms of battery)
system assets.

Punchline

Technically speaking, these do appear to basically be within letter of the SDK
agreement, modulo the fact that Adobe appears to making private API calls. They
should be able to do what they need to without making those calls, so ultimately
that should be a non-issue.

Now, the notion that what this thing emits is indistinguishable from
something Xcode emits is laughable. They are very different, and not in a good
way. While the apps may get acceptable frame rates on an iPhone 3GS, they don't
on earlier hardware, and they almost certainly use substantially more
battery power than native games.

I want to be excited about this thing, both because it is a seriously cool
piece of tech, and because there are Flash games I would like to run on my
phone, but looking at what this thing is spitting out I think the apps it will
generate will perpetuate the stereotypes about Flash (especially on cell
phones), and give Objective C programmers a (somewhat misplaced) sense of
vindication about their views on Flash.

This is all still in beta, it could end up a lot better than it currently is. It
could be something that can make some great games available on the iPhone.
Unfortunately looking it right now I am very skeptical, and I think that is the
right position to have given Flash's performance elsewhere. Yes, this is
entirely new technology, but it comes from the same company with the same
priorities. Given the product they have delivered to me on my desktop for the
last 5 years they don't get benefit the doubt, they have to pull themselves out
of the doghouse as far as I am concerned. Come on Adobe, prove me wrong!