Automated testing

Selenium -> browser Mobile -> ??

Canvas; screen grab w/ control stream

Mozilla eideticker (monitors output of device)


Remote developer console on devices

Wienre / WebDriver? (Wire)

What to do with permission dialogs / extra-DOM stuff

JavaScript? unit frameworks

 * qunit
 * tape

Instrumetnation in the wild (w/ YouTube?)

 * each browser vendor has suite of browser tests

Orchestrated multiple browser w/ mult. participants very hard to test (WebRTC)

KitKat? will solve a lot of problems (runs on older, weaker hardware)

Success Stories?

 * Xiph: Testing for Opus
  * 160 core cluster w/ Jenkins
  * Each commit 1000's of hours of audio processed
  * Performance testing is harder
   * Multi-platform support (ARM, etc)
   * Lots of different chips
   * Consistent test results: hard
  * What test hooks can be added to codec implementations?
   * Support extensions for test hooks

Rdio: Capture HDMI output and compare to expected decoded frames

 * not generalizable 

Expose device hardware capabilities via API?

Performance tests / certification