Monthly Archives: March 2014

Customizing the user agent string in Oxide

The Ubuntu webbrowser sets its own default user agent string, and also provides a list of overrides which define different user agent strings for specific sites that do broken UA sniffing. This is a concept that we’ve borrowed from Firefox OS.

With QtWebkit, the overriding is performed via the same mechanism that is used for setting the default user agent string – by setting the WebViewExperimental.userAgent property. However, this has a major shortcoming – it provides no means to override the user agent string for specific frames. Say, for example, that you need to provide an override for www.twitter.com, but not for www.example.org. If a page from www.example.org embeds www.twitter.com in an iframe, there isn’t a mechanism to change the user agent string for the child frame.

When overriding the user agent string for a frame, there are 2 places that this needs to happen:

  • navigator.userAgent
  • The User-Agent header in outgoing HTTP requests.

Fortunately, Chromium provides a mechanism by which we can implement the second one – net::NetworkDelegate::OnBeforeSendHeaders(). Unfortunately, this is called on Chromium’s IO thread which makes it difficult to expose as a signal in QML (a QML engine can only execute code on a single thread). The notification is asynchronous, which means we could dispatch an event to the UI thread, emit a signal in QML and then dispatch an event back to Chromium’s IO thread. However, this has shortcomings of its own:

  • A typical page load can generate more than 100 network requests. For example, loading BBC News generated 165 requests when I tested it here. Assuming we have multiple notifications per request, processing all of these on the UI thread in QML could have a noticeable performance impact.
  • Not all notifications we get from net::NetworkDelegate are asynchronous. For example, cookie access permission requests aren’t.

Enter WebContextDelegateWorker

For handling events that happen on Chromium’s IO thread, we have a new class – WebContextDelegateWorker. This is very similar to QML’s WorkerScript – it runs script on a worker thread and provides sendMessage/onMessage API’s for exchanging data with QML on the UI thread. However, it provides a mechanism for scripts to expose entry points that can be called from Chromium’s IO thread (actually, they’re not called from Chromium’s IO thread – they’re called on the worker thread by blocking the IO thread. But, you don’t need to care about that).

To overridethe User-Agent header for specific requests in Oxide, you can do something like the following:

In QML:

WebView {
  ...
  context: WebContext {
    ...
    networkRequestDelegate: WebContextDelegateWorker {
      source: Qt.resolvedUrl("worker.js")
      onMessage: {
        console.log("We set User-Agent on " + message.url);
      }
    }
  }
}

In worker.js:

// Unfortunately the properties of QUrl don't appear to be available in QML,
// so provide a simple regex to extract the host
var re=/^[^:]+:\/\/([^\/]*)\/.*/;
exports.onBeforeSendHeaders = function(event) {
  if (re.exec(event.url)[1] == "www.twitter.com") {
    event.setHeader("User-Agent", "Foo");
    oxide.sendMessage({url: event.url});
  }
}

This doesn’t affect navigator.userAgent though. In order to provide an override mechanism for this, we had to apply a small patch to Chromium to give embedders the opportunity to provide an override value when accessed. We did consider exposing a signal in QML for overriding this, based on the assumption that this would run much less frequently than network notifications. However, in the end we decided to reuse WebContextDelegateWorker, based on the assumption that developers who want to override navigator.userAgent will probably also want to use the same overrides for the User-Agent header. Doing this means that you only need the override list in one script context.

To override navigator.userAgent, the above code becomes:

In QML:

WebView {
  ...
  context: WebContext {
    ...
    networkRequestDelegate: WebContextDelegateWorker {
      source: Qt.resolvedUrl("worker.js")
      onMessage: {
        console.log("We set User-Agent on " + message.url);
      }
    }
    userAgentOverrideDelegate: networkRequestDelegate
  }
}

In worker.js:

// Unfortunately the properties of QUrl don't appear to be available in QML,
// so provide a simple regex to extract the host
var re=/^[^:]+:\/\/([^\/]*)\/.*/;

exports.onBeforeSendHeaders = function(event) {
  if (re.exec(event.url)[1] == "www.twitter.com") {
    event.setHeader("User-Agent", "Foo");
    oxide.sendMessage({url: event.url});
  }
}

exports.onGetUserAgentOverride = function(data) {
  if (re.exec(data.url)[1] == "www.twitter.com") {
    data.userAgentOverride = "Foo";
  }
}

Setting a default string

Oxide doesn’t have a per-WebView property that’s equivalent to WebViewExperimental.userAgent in QtWebkit, as we decided that this would be redundant with the above override mechanism. Instead, the default user agent string is configured by setting the WebContext.userAgent property.

Other uses for WebContextDelegateWorker

WebContextDelegateWorker is also used for overriding the default storage access policy via WebContext.storageAccessPermissionDelegate (in QML) and exports.onStoragePermissionRequest (in the worker). This is currently only used for cookies, but is due to be expanded to local storage, app cache, indexedDB and WebDB. The default storage access policy is controlled by WebContext.cookiePolicy

Oxide status

It’s been a while since I wrote my last blog post – Introducing Oxide. I’ve been quite busy since, so I thought that I should take this opportunity to provide a quick update on progress.

Our overall aim is to have the webapps container using Oxide in Ubuntu 14.04, and we’re also hoping to be able to transition the Ubuntu browser quite soon as well.

Summary of progress

A lot has changed since my last post:

It….

  • runs independently of X11
  • supports accelerated compositing on both Mir and X11
  • has been running on several devices with Ubuntu Touch – Nexus 4 (mako), Nexus 7 (grouper) and Galaxy Nexus (maguro)
  • has a navigation history API (WebView.navigationHistory)
  • has a WebPreferences API (WebView.preferences)
  • is DPI aware
  • has input method support, and the onscreen keyboard works on Ubuntu Touch
  • has support for touch events and gestures (pinch-to-zoom and fling works)
  • supports third-party cookie blocking
  • has a mechanism for intercepting and modifying outgoing HTTP requests, which can be used for redirecting URL’s or modifying headers (such as User-Agent)
  • has a mechanism for intercepting accesses to navigator.userAgent which allows applications to customize this on a per-frame and per-domain basis (more on this and the above point in another blog post)

In addition to this, I also have the following features in my review queue:

  • Support for JS dialogs (alert, confirm etc)
  • File picker support
  • Cursor change support
  • API for exposing favicons
  • API for delivering console messages to embedders

Oxide also runs WebGL and plays video quite well on both the desktop and the device.

 

Next steps

Here is a list of things we need to tackle quite urgently:

And then, there are some other major tasks which are still important but not complete blockers at the moment:

  • Geolocation support (requires permission request API and an API key for Google’s geolocation service. Eventually, this should use the Ubuntu location service)
  • Device access permission request API (for microphone and webcam)
  • Support window.open() and HTML anchor ‘target=”_blank”‘
  • Add a fullscreen API – the mobile version of Youtube requires this to play videos
  • We need to join Chromium’s release train with the creation of beta and release branches

I also need to clear my review queue :)

For a more complete list of remaining work, please see the list of open bugs.

Testing Oxide

Builds of Oxide are currently uploaded fairly regularly to the Phablet Team PPA. Feel free to download it, play around with the API and report bugs. Sorry, there’s no documentation yet, so you will need to look at the source code or ask me questions about how it works for now. It is quite easy to use though.

The PPA also contains a build of webbrowser-app that uses Oxide instead of QtWebkit. You can try this out, but if you do, you should be aware of the following caveats:

  • It does replace the current webbrowser-app, which uses QtWebkit
  • Whilst it’s ok for basic browsing, testing and bug reporting, there is still major functionality missing that means you probably don’t want to use it for day-to-day browsing just yet.

Getting involved

If you want to help make Oxide (and our web browser) awesome, please take a look at the list of currently open bugs. If you see anything that you’d like to work on, feel free to come and talk to me on IRC (I’m “chrisccoulson” on irc.freenode.net), and I’ll be glad to help out.

Thanks…

For all the hard work from everyone who has contributed or contributing to Oxide so far: Olivier Tilloy, Alexandre Abreu, Jamie Strandboge, Justin McPherson and Maxim Ermilov.

Also, thanks to Bill Filler, David Barth and Pat McGowan for keeping everybody focused.

And to Parameswaran Sivatharman for his tireless efforts on Continuous Integration.

I’m really excited about this :)