I'm working with the HDS streaming technologies, within the Adobe
Flash world, and I'm trying to compile a set if 'FAQs' about each main
browser available to consumers and how it will react to the 'chunks'
being delivered via the HTTP stack.

My current understanding on Firefox is that, although the Adobe Flash
plugin handles presentation of the video, the chunks are requested via
the host browser's network stack. Resulting in an HTTP request for a
chunk (aka 'fragment').

A couple of things have struck me about this:

1/ What is the default setting in Firefox for the Cache? Is it RAM
then Disk. Is this flow described anywhere?

2/ How is garbage collection instigated - the latest Firefox has an
option suggesting that, by default, there is an automatic garbage
collection which must use some rules... perhaps about disk space? For
example: in Google Chrome, the default cache can take up huge amounts
of disk space growing without some sort of size limit it seems, plus
you can only change the cache settings on the CLI of Chrome (no
'consumer' level configuration in the 'preferences' dialogs).

3/ What are the methods employed when multiple chunks are requested -
as with loading a web page, there is some 'pipeline' going on in the
network stack. Perhaps limiting the total number of working threads
for 'same domain' requests - this could impact on HDS chunks. Consider
where more than one HDS stream is present in a users browser (e.g.
multiple tabs/windows with an HDS stream) - if chunks are served from
the same FQDN, would there be a 'fight' due to slots within the worker
threads in the network stack?

Although I mention HDS here, this would likely go for any other
chunked streaming methods (MPEG-DASH, HLS) which may have a client
implemented inside a browser host.

I hope that someone in your team could assist me with these queries.

Alan
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

I just realised I posted this from my personal Google Account.

I should be clear - I am a member of staff at the BBC () and I work on the Streaming Technologies we use as a broadcaster. This should, perhaps, identify the scale of content that may start breaking Chrome/Chromium browsers.

Any help on this would be much appreciated.

Alan
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

Maybe steve can give a little insight here. He's been working on a DASH
implementation.

original post:
https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.tech.network/wIom4Q5Tvr4

On Thu, 2012-05-17 at 03:16 -0700, AlanO wrote:
> I just realised I posted this from my personal Google Account.
>
> I should be clear - I am a member of staff at the BBC () and I work on the Streaming Technologies we use as a broadcaster. This should, perhaps, identify the scale of content that may start breaking Chrome/Chromium browsers.
>
> Any help on this would be much appreciated.
>
> Alan
> _______________________________________________
> dev-tech-network mailing list
> dev-tech-
> https://lists.mozilla.org/listinfo/dev-tech-network


_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

Hi Alan,

(I'm cross posting to dev-media to get some insight on the media cache - media folks, this email is with respect to Flash/HDS, and I don't think it uses the media cache - please confirm.

And Patrick, I'm bouncing it back to you for the resource-sharing question about parallel chunk requests :) I'm not 100% on the details, so I'm going to spend some time on that, but in the meantime, you might be able to give Alan a quicker response than me.

And Nick and Michal, please take a look over the disk cache question).

On May 17, 2012, at 6:31 AM, Patrick McManus wrote:

> Maybe steve can give a little insight here. He's been working on a DASH
> implementation.


As Patrick wrote, I'm working on DASH/WebM for Firefox, so I'll comment as best I can from that context, but there may be differences with how Flash works. Also, for some of these things I'm learning as well, so forgive me for forwarding you to other folks for specific details.

On May 13, 2012, at 5:20 PM, AlanO wrote:

> I'm working with the HDS streaming technologies, within the Adobe
> Flash world, and I'm trying to compile a set if 'FAQs' about each main
> browser available to consumers and how it will react to the 'chunks'
> being delivered via the HTTP stack.
>
> My current understanding on Firefox is that, although the Adobe Flash
> plugin handles presentation of the video, the chunks are requested via
> the host browser's network stack. Resulting in an HTTP request for a
> chunk (aka 'fragment').
>
> A couple of things have struck me about this:
>
> 1/ What is the default setting in Firefox for the Cache? Is it RAM
> then Disk. Is this flow described anywhere?

One of our cache experts, Nick, has told me that the resources are cached in both RAM and disk by default.

Nick, Michal, is there a flow described somewhere for the cache that Alan could look at? Do you know how plugins like Flash use the cache? Is there a difference?
>
> 2/ How is garbage collection instigated - the latest Firefox has an
> option suggesting that, by default, there is an automatic garbage
> collection which must use some rules... perhaps about disk space? For
> example: in Google Chrome, the default cache can take up huge amounts
> of disk space growing without some sort of size limit it seems, plus
> you can only change the cache settings on the CLI of Chrome (no
> 'consumer' level configuration in the 'preferences' dialogs).

I'll let Nick and Michal comment on garbage collection for regular cache objects. We also have a media cache, which the guys on dev-media have a better understanding since they wrote it, but I don't think it's used for Flash, only for built-in decoders.
>
> 3/ What are the methods employed when multiple chunks are requested -
> as with loading a web page, there is some 'pipeline' going on in the
> network stack. Perhaps limiting the total number of working threads
> for 'same domain' requests - this could impact on HDS chunks. Consider
> where more than one HDS stream is present in a users browser (e.g.
> multiple tabs/windows with an HDS stream) - if chunks are served from
> the same FQDN, would there be a 'fight' due to slots within the worker
> threads in the network stack?

Chunks as a concept is something on my todo-list-to-think-about for DASH, albeit to examine the effect of different chunk sizes on TCP window size and thus bandwidth usage, which is, of course, a different issue.

So, to make sure I understand your question, you're asking if there could be a fight for resources if there were two or more media presentations grabbing files/chunks from the same FQDN. Assuming that's right, I'm going to look into it and get back to you, because I'm not 100% sure. I have a notion that it depends on what protocol you're using (HTTP, maybe with pipelining, or SPDY) and the number of parallel TCP connections available. As far as the number of threads or other resources we have in Firefox, I'm not sure of those details.

In the meantime, Patrick may actually have a better idea about the inner workings - sorry to bounce it back, Patrick :)
>
> Although I mention HDS here, this would likely go for any other
> chunked streaming methods (MPEG-DASH, HLS) which may have a client
> implemented inside a browser host.

For sure, so it's useful discussion to have all round.
>
> I hope that someone in your team could assist me with these queries.
>
> Alan
> _______________________________________________
> dev-tech-network mailing list
> dev-tech-
> https://lists.mozilla.org/listinfo/dev-tech-network


_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

Just realised that it would be useful if I supplied you with a real-world example:

http://www.bbc.co.uk/bbcone/watchlive/

This is 'UK Only', restricted based on the requesting IP address.
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

(Note: lists.mozilla.org seems to be having some issues this week according to a few colleagues, so these posts might not all appear. Feel free to email directly if you're concerned).

On May 18, 2012, at 4:17 AM, Alan Ogilvie wrote:

> Thank you all for your response so far - clearly this is going to be a
> somewhat complicated answer (I didn't believe that it wouldn't) ;-)

:) Yup, but we can work it out and de-complexify (new word maybe) it all.
>
> 3/ - yes, my (and somewhat 'our') hypothesis is that because chunk requests
> are made through the host browser, they will be restricted as any other
> internet content is - for example, requesting a web page, then all the
> assets from the HTML response, is clearly managed by limiting the number of
> concurrent 'threads' (I may obviously be using the wrong words here) to
> some sort of rule for 'same domain'. If we had more than one player in the
> browser instance (maybe on multiple tabs/windows) all requesting chunks
> from the same domain, but different 'channels' (ie. different streams),
> then they will likely 'fight' each other for spare slots in the concurrent
> 'threads'. We're not sure how much of an 'edge case' that is - though I can
> believe some of our viewers/listeners having a couple of windows open
> during big events (like the Olympics in London coming soon) with HDS
> playback in Flash.

That's a good example.

So, I've done so more poking around today, and this is what I can see in our networking code as far as resources for connections are concerned.

At a high level, we have 4 maximums for the number of connections (you can look at these in "about:config" - if you're unfamiliar with that, just type about:config in the address bar and tell Firefox that you'll be careful):

network.http.max-connections = 256
network.http.max-connections-per-server = 15
network.http.max-persistent-connections-per-proxy = 8
network.http.max-persistent-connections-per-server = 6

I'm guessing that proxy will matter for those folk watching the games from their workplaces, but there may be other scenarios. And of course, we're not limited to the Olympic games here.

These values are configurable, but for the vast majority of users, they won't touch them.

Once we get an HTTP request, we create a transaction and determine what to do next based on the following logic in nsHttpConnectionMgr::ProcessNewTransaction (currently found at http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpConnectionMgr.cpp#1641). Note that the transaction is created by the content thread and then dispatched to the socket thread for the following:

// 0 - If this should use spdy then dispatch it post haste.
// 1 - If there is connection pressure then see if we can pipeline this on
// a connection of a matching type instead of using a new conn
// 2 - If there is an idle connection, use it!
// 3 - if class == reval or script and there is an open conn of that type
// then pipeline onto shortest pipeline of that class if limits allow
// 4 - If we aren't up against our connection limit,
// then open a new one
// 5 - Try a pipeline if we haven't already - this will be unusual because
// it implies a low connection pressure situation where
// MakeNewConnection() failed.. that is possible, but unlikely, due to
// global limits
// 6 - no connection is available - queue it

So, deciding what to do with the transaction depends on what protocol it's using (SPDY, HTTP pipelining or non-pipelining), how many connections are free and the type of connection. If the connection is SPDY, the transaction is added as a stream if there is room (max of 100 streams per connection I think). Otherwise the transaction is queued until a slot opens up in the SPDY stream. So there are multiple layers in the decision-making to determine what is a relevant resource (num of sockets, num of streams, etc).

Some points:
"under pressure" means most of the allowed parallel connections are already used up.
"idle connections" are persistent connections, i.e. sockets that are kept open thereby avoiding the DNS and TCP RTT for subsequent requests.

Once we figure out what type of connection, the transaction is dispatched.

Some things I'm still exploring:
- We have one socket thread on which the previous decision about how to add a transaction is processed. I'm not sure if we have more socket threads for reading and writing or how we multiplex all of that - I'll find that out.
- There may be a max for the number of requests which can be pipelined.

>
> I have supplied an example of HDS through Flash which is live on our site
> at the moment. This may help you see how it's working in the real world.
> I've already poked around to see what happens to the chunks.

What have you observed if anything?

Steve.

_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

> I'll let Nick and Michal comment on garbage collection for regular

> > cache objects. We also have a media cache, which the guys on
> > dev-media have a better understanding since they wrote it, but I
> > don't think it's used for Flash, only for built-in decoders.
>

Correct. Our media cache does not affect Flash in any way.

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network

On 17/05/12 11:16, AlanO wrote:
> I should be clear - I am a member of staff at the BBC
> () and I work on the Streaming Technologies we
> use as a broadcaster. This should, perhaps, identify the scale of
> content that may start breaking Chrome/Chromium browsers.

Presumably you mean "Firefox"? Otherwise you are in the wrong place...

Gerv
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

Let's just change the words for 'Host Browser' - I'm also trying to get answers from Chromium team which explains my slip up (though none as yet, unlike your team). I shall go and take detention for using such words. Sorry ;-)

On Monday, 21 May 2012 10:18:47 UTC+1, Gervase Markham wrote:
> On 17/05/12 11:16, AlanO wrote:
> > I should be clear - I am a member of staff at the BBC
> > () and I work on the Streaming Technologies we
> > use as a broadcaster. This should, perhaps, identify the scale of
> > content that may start breaking Chrome/Chromium browsers.
>
> Presumably you mean "Firefox"? Otherwise you are in the wrong place...
>
> Gerv

_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)

On 05/13/2012 05:20 PM, AlanO wrote:
> I'm working with the HDS streaming technologies, within the Adobe
> Flash world, and I'm trying to compile a set if 'FAQs' about each main
> browser available to consumers and how it will react to the 'chunks'
> being delivered via the HTTP stack.

I can answer some of your questions, though I'm not sure how they
specifically relate to HDS streaming.

> 1/ What is the default setting in Firefox for the Cache? Is it RAM
> then Disk. Is this flow described anywhere?

We currently only use the RAM cache for for requests that can't be
stored on disk--no-store and certain SSL requests. So the flow as data
comes in is that data is delivered immediately to the program logic that
requested it, plus asynchronously scheduled to be written to the disk
cache.

For media requests (such as HDS), our media processing layer also has
its own, RAM-backed cache. I'm not sure what state the data is in
(decoded in any form yet) while it's in the media cache. But our media
cache is probably not getting used for Flash, so it's probably not
relevant for you. Our DASH implementation will use it, though.

Is there a specific issue related to caching and HDS that you're
concerned about?

> 2/ How is garbage collection instigated - the latest Firefox has an
> option suggesting that, by default, there is an automatic garbage
> collection which must use some rules... perhaps about disk space? For
> example: in Google Chrome, the default cache can take up huge amounts
> of disk space growing without some sort of size limit it seems, plus
> you can only change the cache settings on the CLI of Chrome (no
> 'consumer' level configuration in the 'preferences' dialogs).

I assume by "garbage collection" you're referring to evicting items from
the HTTP cache (we usually reserve "garbage collection" for other things
not related to the cache, like collecting reference counted objects in
memory). Both Chrome and Firefox use a LRU (least-recently-used)
algorithm to evict items from the disk cache, and we also both use an
algorithm that grabs up to a certain % of disk drive space (the % drops
as the disk size increases), up to 1 GB max. In Firefox you can
easily provide a user-defined cache size instead--the UI for it is in
Preferences | Advanced | Network | Override automatic cache
management. Or you can set it in about:config by setting
'browser.cache.disk.smart_size.enabled' to false, and
'browser.cache.disk.capacity' to the size you want.


> 3/ What are the methods employed when multiple chunks are requested -
> as with loading a web page, there is some 'pipeline' going on in the
> network stack. Perhaps limiting the total number of working threads
> for 'same domain' requests - this could impact on HDS chunks. Consider
> where more than one HDS stream is present in a users browser (e.g.
> multiple tabs/windows with an HDS stream) - if chunks are served from
> the same FQDN, would there be a 'fight' due to slots within the worker
> threads in the network stack?
>
> Although I mention HDS here, this would likely go for any other
> chunked streaming methods (MPEG-DASH, HLS) which may have a client
> implemented inside a browser host.
We do have a bunch of limits for the maximum number of TCP connections
we allow to a server (or a proxy server, if that's being used). Search
for "connections" in about:config and you'll see them. Generally the
limit is 6 per FQDN (for persistent connections, which are the most
common) or 15 for non-keep-alive connections. When using a proxy we're
limited to 8 connections total (across all domains, IIRC), so that's a
more constrained case. These limits are also lower on mobile
Firefox--don't recall them off the top of my head, can let you know if
you need them.

We do have a queuing system for when there are more HTTP requests for a
domain pending than we have free connections (I'm assuming that's what
you mean by 'pipeline' here? It doesn't sound like HTTP pipelining per
se as in RFC 2616, which is a particular way of handling queued
connections, in some sense, but it sounds like you're talking about the
general case of "more requests than connections"). So if you have more
HDS streams (and/or other requests to the same domain) pending than the
above connection limits, some of them will have to wait for the others
to complete before being satisfied. It's definitely not a good idea to
request 8 media chunks in the reverse order that they'll be needed, for
instance.

Does that answer any of your questions? I think it might help to have
more of an idea of what problems/issues you're anticipating, or specific
info you need for your guide.

cheers,

Jason
Mozilla
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)