Quality of Experience metics

Application of data towards improving the user experience.

QoE? Standards Proposal: https://docs.google.com/document/d/1hPYayD4uyV-7LuRMlM5JP1pvWL5V_y44kF_7DusqPOU/edit?usp=sharing

  • Provide a common language
  • Same APIs? in the browser to get this data.
  • How to use the metrics ?
  • Measurement and application of QoE? on the player side.
  • Optimize on a different metric then raw bitrate or raw size
  • Reporting unity against device and application contexts.
  • QoS? and QoE?, use machine learning as a tool to understand that data.
  • How to get rid of bots noise.
  • Browser app suspension
  • Bad content, very scenarios.
  • offline apis
  • bringing the data back into the players adaptive streaming decisions
  • analytics toward error reporting and discovery.
  • relating QoE? to conversion goals.
  • What is the pain threshold level, or drop-off-level.
  • Are we asking the right questions / capturing the right data ?
  • Aggregate views of quality data
  • Cleaner mapping for what bitrate is being consumed.
  • Accuracy and standardization, what do you do with the data.
  • Actionability of QoE? data; looking at control of cost.
  • How it affects the streaming server.
  • How can we be more proactive with the data.
  • What people want from the browser for these QoE? goals.

How people are using metrics today?

  • CDN selection
  • cost
  • applying to conversions
  • AB testing
  • Alerting operational concerns.
  • Deal with anomaly detection for engineering team, at the same time feeds up to business, so need "Trustworthiness".

"Watchtime" a metric to optimize to.

  • the sum of time spent consuming; time spent consuming the actual content. Amount of time the user intended.
  • watchtime a useful heuristic towards drilling into more information.
 * content and technology contribute to watchtime. 
 * subscription vs ads considerations for watchtime. 
 * how much watchtime to retain ? 
 * too much watchtime is potentially problematic.
 * how do you track watchtime? 
 appending to every event? a heartbeat ? granularity of what your trying to answer. 
 * lead metrics: 

video start time

  • beacons API, lets you send an event as the page session ends. ( sendbeacon )

Standards specifics:

  • Session level metrics ( not aggregate metrics )
  • WatchTime?, video start time,
  • Aggregation metrics.

Standard ways to collect the numbers; Vimeo uses Percentiles. 50 70 80 95 99

  • Client side timestamps sometimes problematic.
  • "Throw out the bad data"
  • end up storing both client and server data.

Youtube has average 700ms startup time; Customers want 700ms for 90 percentile. Very big difference across platforms; very hard to compare against population bias. At the codec level significant constraints.

Much higher tolerance for startup time on different platforms. Say Roku may be hard to compare to web.

Advice the publishers on how to make the video start faster.

  • Gather what your doing, what "they" are doing.
  • Compromise and start at the high profile. Startup time is a good metric, but needs to be combined with other data.

Looking at just starttime by itself is a little dangerous.

  • Gather information about the page itself; actionable and advise customers.

Quality how to track that well?

  • 720P over 1080P; per content encoding; Machine learning for circumstantial or conditional ABR.
  • Maybe you don't ABR.

Could we have a standard QoS? feed going into ABR systems ?

Steve's Summary Notes

- People started by giving feedback around how they are using QoE? metrics today

  - Discussion around watch time as a success metric
    - How people use it if they do
    - What the right way to track watch time is? ""
    - Trailing metrics vs. a leading metric
  - Questions on how best to use QoE? feed into the adaptive algorithm, and CDN switching

- A lot of people interested in the general standardization metrics,

  - both naming and how to track accurately
  - Anecdotes on being compared to youtube (there's a blog post), what does that mean, and when does that not even make sense to do

- Good feedback on metrics that I presented earlier, conversations around other ones

  - Percentiles are valuable
  - Not predefining the buckets of a histogram, or percentiles, but suggesting the values as a starting point.
    - People will have different prefs, change by device, by region
  - Tracking quality, no conclusions except maybe it's hard

- Browser-specific notes

  - Consistent support for the sendBeacon API
  - Report QoE? details from lower in the stack of the browser, video player runs in separate thread from javascript
    - Performance timing API
  1. Topics we didn't get to

- Dealing with Issues of QoE?

  - Dealing with Bots
  - Browser App Suspension
  - Offline QoE?

- Server Side ** (live streaming) - Machine Learning (Feed aggregate into algorithms) ** Applying it for better quality

  - Potentially standardizing how QoE? platforms feed data back into adaptive logic

- Reporting QoE? Lower in the stack