Category Archives: Mozilla

Why Ubuntu is not using the Firefox ESR

Recently, a thread has appeared on the ubuntu-desktop mailing list asking why Ubuntu (or specifically, Ubuntu 12.04 LTS) is not using the Extended Support Release of Firefox by default. I’ve also been asked this a few times on IRC over the last few weeks (from people inside and outside of Canonical), so I just wanted to clarify the reasoning for this, and why I think that our choice will offer the best experience for Ubuntu users.

Arguments against the ESR

Aside from the fact that offering the Firefox ESR by default to Ubuntu users would make our good friends at Mozilla unhappy (and from my perspective, we have a pretty good relationship with them. I certainly don’t want this to change), here are some reasons why I think that shipping the Firefox ESR by default would be bad for Ubuntu users. Some of these are points from the ESR proposal, and some of them are my own thoughts.

Over time, Firefox ESR will become less secure than Firefox

The ESR proposal states that:

Mozilla will backport security bugs qualified as “Critical” and “High” to the ESR where feasible (there may be cases where a backport cannot be applied with reasonable effort, and those cases are expected to be exceptional)

This means that people running the Firefox ESR may have to wait for the next major release to receive fixes for security bugs with sg:moderate, sg:low or sg:dos severity ratings. What is worse is that ESR users may also miss out on some sg:critical or sg:high rated bugs, where it is just not feasible to backport the fix to the ESR branch.

In addition to this, Firefox ESR users will have to wait longer than regular Firefox users for proactive security improvements such as support for the iframe sandbox attribute, CA pinning, the Mixed Content Blocker or any other new security features / improvements.

Because of this, the Firefox ESR will always become less secure than the regular Firefox releases.

Ubuntu 10.04 LTS users have already had to wait longer than Firefox users on other platforms for new security and privacy features such as Content Security Policy, HTTP Strict Transport Security and Do Not Track. We don’t want Ubuntu to lag behind other platforms in the future.

The risk of introducing bugs is greater with Firefox ESR

There is a common misconception that when a piece of software receives only reactive security fixes, it is the safest option for users and that the risk of breakage is minimal with this approach. In reality, this isn’t exactly true. There is always risk associated with backporting any form of code change from one branch to an older branch, and this risk increases as:

  • The 2 branches diverge further apart, making the backport less straightforward
  • The amount of testing exposure decreases

Clearly, the Firefox ESR will be affected by both of these factors.

The regular Firefox releases pass through the beta channel for 6 weeks before release, where they are exposed to a large community of users who are using the beta as their day-to-day browser. And, whilst the Linux testing community is relatively small (I would like to grow this though), we mustn’t take for granted the positive effects on quality that Ubuntu users get from Firefox beta testers across all platforms. The Firefox ESR will not benefit from this type of large scale pre-release exposure.

In my opinion, bug 667087 is a perfect example of how the ESR approach can lead to the introduction of bugs (this was a regression which only affected the 3.6 branch).

It is only supported for 1-year (well, 54 weeks, to be more exact)

The Firefox ESR is supported for 54 weeks (9 regular Firefox release cycles), with a 12-week (2-cycle) overlap. This means that it would be inevitable that we would have to upgrade users to a new version of Firefox ESR every year, if we provided this by default. Instead of small incremental changes every 6 weeks, users would be faced with much larger and more obvious changes every year. We believe this is generally worse for most users.

We have been following the new Firefox release process since Ubuntu 11.04, and users seem to have adapted to it quite well. Having just upgraded Ubuntu 10.04 LTS users from Firefox 3.6 to Firefox 10, we know that this scale of update is much more painful – for users and for us.

The web is not static

All of the major browser vendors are pushing new technologies on the web, and existing standards are constantly evolving. It is important that we provide Ubuntu users with a browser which keeps up with this, as users coming from competing platforms expect. In the time since Firefox 4 (which isn’t that dissimilar to the time between 2 ESR versions), Mozilla has added support for things such as the <bdi> element, page visibility API, Mozilla’s full screen API, CSS 3D transforms, CSS font-stretch property, cross-domain textures in WebGL, Web Timing, CSS hyphenation in languages other than English, HTML5 context menus and added a bunch of other improvements to IndexedDB, WebSockets, and canvas. Web developers and Firefox users on other platforms already have access to these features.

We don’t want Ubuntu users to regularly be the last people to have access to evolving technologies on the web, and I don’t think it’s great to say to them “if you want access to the latest web technologies like users on other platforms have, you need to upgrade your entire OS in 6 months or use this unsupported PPA instead”. It wouldn’t be good for Ubuntu if the function or appearance of our users favourite websites ends up being degraded by default on our flagship product, as the web evolves faster than the browser that we are shipping.

Until recently, Ubuntu 10.04 LTS users have already been missing out on major technologies such as CSS transitions (which is used in some places in Google+), WebGL and WebM (available on YouTube) whilst we have been shipping Firefox 3.6 to them. In addition to this, Google have already effectively dropped support for Firefox 3.6, as have Flickr, and we have had reports from other people saying that some online banking sites have already bumped their minimum browser requirements beyond Firefox 3.6.  As the web evolves faster, this type of thing may occur more frequently in the future (the alternatives to this are that web developers give themselves a hard time when trying to adopt new technologies by having to support fallbacks for older browsers, or innovation on the web just stagnates as developers are reluctant to adopt new features).

Over time, Firefox ESR will become slower than Firefox

In the same way that Firefox ESR will become less secure than Firefox, it will also become slower and less resource efficient than regular Firefox due to initiatives such as MemShrink, Snappy and continual work on improving performance in the JS engine.

Performance and memory consumption really matter to users, and these things can affect people’s perception of Ubuntu when they compare browser performance with browsers that are shipped on other platforms.

In addition to this, we offer the latest version of Chromium alongside Firefox in the Ubuntu archive. It would be bad for Mozilla for us to offer an outdated Firefox ESR against the very latest version of Chromium, as the difference in performance between the 2 can significantly influence our users perception of the quality of Mozilla’s product. I’m not the only person who thinks this:

I think it would hurt us competitively if Fedora or Ubuntu shipped ESR, because users or journalists would compare ESR with up-to-date Chrome

Arguments in favour of the ESR

Of course, other people have some good points about why the ESR might be a positive thing. I’ll list some of the more frequent points I hear, and explain why I disagree with them.

The Ubuntu LTS is for enterprise users

This isn’t true. Whilst it is true that enterprise users tend to stick with the LTS release for the longer period of support and less frequent upgrades between OS versions, the LTS is targeted and used by all types of users.

Users who stick with the LTS want stability. Users who want the latest-and-greatest should upgrade between the regular 6-month releases

There are several things wrong with this argument:

  • It assumes that because a user doesn’t want to upgrade their entire OS every 6 months and because they want 3-5 years of support, that they don’t want the latest applications.  I don’t want to upgrade my cell phone more than once every 2 years because it is a pain to adapt to a new device, but I certainly do want to be offered the latest apps on it for the time that it is supported.
  • It assumes that LTS users choose to stay on the LTS.  In fact, when somebody installs the LTS, we will only offer LTS – LTS upgrades for them unless they change a setting in the Updates tab of the Software Sources settings.
  • It assumes that the Firefox ESR provides more stability than the regular Firefox release, and that we won’t get stability from shipping regular Firefox releases.  I’ve already explained above why I don’t think this is the case.
  • It assumes that stability is all that is required to satisfy LTS users.  The reality is a lot more complicated than this.

LTS users actually do seek out the latest software.  As the maintainer of the Firefox Beta PPA and the (now retired) Firefox Stable PPA, I have some interesting download statistics for these PPA’s:

  • Ubuntu 10.04 LTS users are consistently the second highest consumer of the Firefox Beta PPA.  In fact, the number of downloads from Ubuntu 10.04 LTS users is around the same as (and sometimes exceeds) the number of downloads from Ubuntu 10.10 and Ubuntu 11.04 combined.  Note that the highest consumer is always the most recent supported release.
  • The last upload to the Firefox Stable PPA (9.0.1, which was also uploaded to lucid-proposed) was downloaded by 3-times as many users on Ubuntu 10.04 LTS as it was from users on Ubuntu 10.10.

Also, I accidentally introduced a packaging bug in to our daily builds last week which temporarily broke upgrades for daily build users on Ubuntu 10.04 LTS and Ubuntu 11.04.  To my surprise, we got a bug report from a 10.04 user within minutes of the broken packages being published. We then got a fairly steady stream of bug reports from 10.04 users until the packages were fixed.  In total, we had 7 bug reports from Ubuntu 10.04 LTS users, and 1 bug report from an Ubuntu 11.04 user.  Prior to this, I had always made the assumption that Ubuntu 10.04 LTS users would be the smallest consumer of Firefox daily builds, but I may have to reevaluate this view now.

Ok, it’s difficult to read too much in to this relatively small amount of data and I’m not sure how much it really proves.  In any case, I think you’ll find that the LTS users aren’t quite as conservative as some people make them out to be.

The LTS should be stable, secure, supported, predictable

The regular Firefox releases are more secure than the ESR, will be just as stable (with the significantly larger audience of beta testers) and are better supported. The 6 weekly releases are also predictable.

Of course our flagship product needs to be stable, secure and supported. But, it needs to be much more than this too.

Addons break between releases

Whilst this was problematic in the early stages of the rapid release process, this isn’t as much of a problem now. Starting in Firefox 10, most addons are compatible by default (the exceptions are themes and addons with binary components). Prior to this, addon compatibility has been regularly exceeding 95% before each new Firefox release (for the top 95% of addons which were compatible with the previous Firefox version).


I hope this answers some of your questions about why Ubuntu is not shipping the Firefox ESR by default. Of course, I’m more than happy to listen to peoples concerns.

It is entirely possible that we might provide a Firefox ESR build for people who are managing large deployments of Ubuntu, although the details of this aren’t decided yet. However, this isn’t going to be the default browser for our LTS. If it existed, it would be shipped in a PPA (much like we have been doing for the Firefox Stable PPA), and we would have to be clear to users that it wouldn’t receive the same level of support we give to the regular Firefox versions.

Thank you for reading :-)

Retiring the Firefox Stable PPA

With the announcement that users of Ubuntu 10.04 LTS and Ubuntu 10.10 will be upgraded to the latest version of Firefox, we are going to be retiring the Firefox Stable PPA.

Since the release of Firefox 4 last year, we have been providing this PPA so that Ubuntu 10.04 LTS and Ubuntu 10.10 users didn’t have to miss out on the new features, performance improvements, security enhancements and improved compatibility with newer web technologies provided by the latest releases of Firefox.

The next version of Firefox (Firefox 10, to be released on Tuesday of this week) will be delivered as a security update to all Ubuntu releases – hopefully by the end of the week. This means that users will receive the update without having to do anything.

I have no plans to upload the next release to the Firefox Stable PPA.

If you’re aware of any online resource (eg, on the Ubuntu Wiki, Ubuntu Forums, Ask Ubuntu or any other site providing help to Ubuntu users) which recommends to use this PPA to receive the latest version of Firefox, I would appreciate you taking the time to either correct this information (or removing it, if more appropriate).

Ubuntu, Firefox updates, beta and why we need you

Everyone reading this is probably already well aware of Mozilla’s (now not-so) new 6-week release cycle for Firefox, and that we (Ubuntu) have been delivering these new releases to Ubuntu users since 11.04.  We’re going to continue to do this for the next LTS, Ubuntu 12.04, and with the big drive for increased quality for this cycle, we need to ensure that this quality is maintained for the 5 years that the LTS is supported.  The Firefox release process obviously brings some additional challenges in this area that we don’t have for other applications in Ubuntu.

Stable release updates (SRUs)

Traditionally for other applications, all individual changes landing in to a stable release are reviewed for quality and suitability.  All changes go through the “proposed” pocket where they can be tested by a limited audience for 7 days before being released to the general user base.  All individual changes generally have a test case that users of the proposed pocket can follow.  This process is documented here

Firefox updates

It isn’t realistic for us to review every individual change in each new Firefox release, nor is it realistic for us to write a test case for each individual change and ask a small number of users to verify them.  It is also not normally appropriate to delay security updates for 7 days whilst we wait for feedback from proposed testers.  Because of this, we haven’t used the traditional SRU method to deliver Firefox updates in the past.

So, what’s the problem now?

With the new 6-week releases introducing more than just security and stability fixes, we need to rethink how we test browser updates.  The SRU process doesn’t scale well to browser updates for a few reasons:

  • We can’t wait 7 days after the official Mozilla release to ship a security update
  • The SRU process is great for testing self-contained, specific and isolated fixes with a well defined test-case, but I would question the value of dropping a browser update in to the proposed pocket every 6 weeks, asking users to run it to “see if it works ok” and then waiting for one user to wave their hand on a bug report to say that they started it and it seems to run (eg, this comment on Firefox 5 tracking bug)
  • A few days testing every 6 weeks isn’t long enough. In fact, 7 days is nowhere near long enough in my opinion.

Well, how do Mozilla handle this?

Each Firefox release has 6 weeks of development, followed by 12 weeks of stabilization whereby the release is tested by a community of enthusiastic testers.  This process can be visualized by the following graphic, borrowed from here.

You can see that there are 4 channels running in parallel (these are called nightly, aurora, beta and release), and each Firefox version spends 6 weeks in each channel before moving on to the next one in the release train.  Features land on the nightly channel, whereas aurora and beta are (meant to be) focused solely on stabilizing these new features.  Aurora is less stable than beta, and is intended for users who don’t mind sacrificing a little bit of stability for the newest innovations.  The beta channel is intended for people who want to try out new features in the next version of Firefox, but don’t want to sacrifice the stability that they are used to.

What pops out at the end of the 6 week beta cycle is the next Firefox release.

Ok, so what’s the point in this blog post?

With each Firefox release spending 6 weeks in the beta channel, there is no point in wasting all of this time by using only 7 days of it for testing in Ubuntu.  To make the best use of this 6 weeks we provide our own beta PPA for Ubuntu users, which tracks the upstream beta channel.  Builds are provided for all Ubuntu releases.  This PPA is the Firefox equivalent of the proposed pocket for other Ubuntu packages, with the following extra advantages:

  • It provides 6 weeks of testing as opposed to 1 week
  • Users can opt in to it and use it continuously as their default browser.  After 6 weeks, they will just get the next beta and the whole cycle starts again.

Some additional information about this PPA:

  • It is still possible to send crash reports directly to upstream, just like you can with the version in the official archive.
  • It is possible to report bugs to Launchpad in the usual way (using Apport).
  • We provide all associated supporting packages, to make the user experience as smooth as possible.
  • It’s supported by the Ubuntu developers who provide Firefox for you.

Don’t be afraid of the “beta” name – these builds are much closer to RC quality than beta, and experience so far has shown this to be the case even at the start of each 6 week beta cycle.

So, what can I do to help?

Well, first of all, start running the Firefox beta if you aren’t already.  If you’re on the development release of Ubuntu, then you are already running it.  If you aren’t, then just run:

sudo add-apt-repository ppa:mozillateam/firefox-next && sudo apt-get update && sudo apt-get upgrade

That’s all there is to it! Now you can start enjoying the next Firefox release :-)

You can also help by blogging and tweeting about this, and telling your friends and family about it.

B…b…b…but, I prefer to stick to official packages rather than use PPA’s, and I can still report bugs and help you if I stick to the release build

Yes, of course you can. But it’s way too late to be finding bugs and regressions in the official release build – by this point, we’ve already shipped it to our users as a security update. It’s much more helpful for people to find and report regressions before it ends up in the archive as the next security update.

I generally believe that anybody involved with Ubuntu Development in any capacity who runs Firefox should at least be using the beta releases all of the time.