MSE

Areas of Interest:

 - MSE v2
  -- What is it? More importantly, what will be in it?
  -- Promise based
  -- Worker compatible
  -- Gather usecases/bugs for implementers
  -- Harmonize implementations across implementations
 - Insight into the decoder
 - DASH implementations across devices/browsers
  -- Trickplay
 - More robust set of code examples around MSE
  -- It's hard for newcomers to approach the API or understand how it works
  -- Focus on practical/realistic examples
  -- Focus on errors & how to solve them
  --- MDN has a lot of good articles, but do we need more articles out of the reference?
 - Requests for MSEv2?

Current MSE Issues:

 - Rough transitions when switching sourcebuffers
  -- Needs to be frame accurate
 - Decouple MSE from video element
 - Usecase: background loading of sourcebuffers; video element as "view" into sourcebuffer
 - Codec switching would be useful but against the spec, but should "similar" ones be allowed?
  -- Firefox thinks its a good idea & easy to implement
    --- Very valuable to be able to switch between codecs within the same sourcebuffer
    --- Much easier to do multiple codecs than multiple SBS
  -- Chrome: don't want multiple codecs in SB, but support multiple SBs? and seamless transitions between
 - Can't put data in a SB without tearing down existing one
  -- Not against the spec but implementers have not done it
 - Segment mode problems:
    -- Hard to sync a/v with offsets
 - More info out of MSE on what actually happened with the data we appended
  -- Did it actually append?
  -- What timeranges did it append to?
  -- Chrome does not report bufferedranges when it doesn't have enough data
   --- Perhaps this would be valuable to surface
 - Cross browser issues
  -- May be solved with better feedback from MSE, such that we could tell when browsers are doing something differently
 - Performance hit on main thread
 - Callback interface isn't ideal
 - Not on iOS
 - Hard to implement
 - Hard tech, but also lack of examples
 - Knowledge from browser devs on how to optimize MSE implementations & avoid performance pitfalls

What's currently on the MSEv2? roadmap (according to Chrome):

 - Worker context
 - Cross Codec
 - Deprecate support (john: follow up on what)
 - Check MSE on github

What we want in MSEv2?:

 - API for seamlessly switching sourcebuffers
  -- Hard to do in JS, but can we make an API to signal to the browser when a SB should be switched, in order to ensure a smooth transition?
  -- Ideas: 
    --- Multiple sourcebuffers
      ---- Hard to sync the timeline
      ---- Unanimous good idea, can solve a lot of issues
    --- Multiple codec SBs?
 - Decouple SB from video tag
 - More data out of MSE
  -- Expose append state so that we know what we can/cannot do with the MSE API
   --- i.e. cannot change timestamp in certain append states, will throw error if you try
  -- Expose data on what we actually appended
  -- What is the value of exposing additional information?
    --- Not having to parse the MP4? to figure out data
    --- FF: Sometimes hard to get data out of buffer right after append
  -- Get the bitrate at the playhead time
 - Last years FOMS: appending decorated/generic data
  -- May be a problem because of data around edges of SB being inconsistent
 - Appending metadata track to SB at append time so that we can now, at any time in the SB, we can know what has been appended (Metadata track interface)
  -- Apple: sounds useful, and we already do it in Safari in a similar way
  -- Lots of usecases
  -- Would unify MSE implementations
  -- Challenges: how to match bytestream to metadata
  -- FF: sounds hard to do
 - Workers
  -- Current discussion on github [insert link]
  -- Problem to be solved: MSE hurts main thread performance
  -- Workers are hard, and the implementation is currently uncertain
  -- Need to solve MSE out of DOM issue
  -- Decouple MSE from DOM
  -- Need multiple SBs?/seamless transition first
  -- Browser: move more media stuff ourselves off of main thread
  -- FF: fetch is already on a worker and it saves a lot of time b/c you don't need to wait for layout
  -- Challenges: moving blobs between main threads so that we can create MSE off thread & append it back to main, performantly - MSE on a separate thread (done by the browser)
  -- Problem to be solved: lots of things happening, especially on startup, which delays video playback
  -- Applications don't necessarily know about this
 - AppendStream? API, esp. in combination with streaming body fetch response
 - Tag MSE git issue with v2
  -- Anything not v1-fix is v2
 - Start forum on MSEv2? in order to prioritize requests
 - Promisfy events coming from MSE/SB

Action items:

    - Create usecases for implementers so that we know what's important
    - Enhance discussion around MSEv2?
    - Chrome: still wrapping up compatibility, will start MSEv2? in 2018
    - github.com/w3c/media-source, v-next milestone
    - If you want iOS MSE, submit bugs, tell your friends

DASH:

    - Does anyone use trickplay?
    - More/enhanced trickplay support coming
    - https://en.wikipedia.org/wiki/Trick_mode