All,
As part of our grand plans for the disk cache, the next big area we
are going to look at is crash recovery, specifically how to avoid
trashing the cache unless we're relatively certain that it's needed.
Right now, we have one dirty bit for the entire cache. This bit is set
when the disk cache is opened, and cleared when the disk cache is
closed. When we go to open the disk cache, if the bit is already set,
then we assume that the disk cache is corrupt, and trash the whole
thing and start from scratch. This approach is, to say the least,
probably a bit over cautious.
The plan is to get to the point where we can experience an unclean
shutdown and assume the disk cache is just fine until we have proof
otherwise. At the point when we determine something is amiss in the
cache, we would either delete the entire cache (as we do now), or (if
possible in whatever future scheme we come up with) delete just the
affected portion of the cache, leaving the rest intact.
So far, I've come up with 3 possible ways of having better crash
recovery. In no particular order, they are:
-Keep some sort of hash of contents per entry (similar to how Chrome
does things)
*This could conceivably allow us to only delete entries that are known corrupt
*This doesn't (necessarily) require any wild changes to the current
on-disk format (it could be kept as a metadata field on all entries)
*At first glance, the implementation seems relatively straightforward
(adding the standard caveats here about how nothing is ever nearly as
simple as one thinks, etc)
-Use some sort of journaling and/or append-only structure to store the
cache, with some markers for consistency per entry (or some other
subdivision of the data)
*The consistency markers are still undetermined
*There are a lot of open implementations of filesystems using these
kinds of structures, so lots of good reference material
*I have a sneaking suspicion that even with lots of good reference
material, this would not exactly be a simple implementation
-Use leveldb to store the cache
*We are still unsure if leveldb uses fsync to ensure consistency (this
could be a deal-breaker)
*Import of leveldb has already been started:
https://bugzilla.mozilla.org/show_bug.cgi?id=leveldb
*However, we're unsure of how much more effort would be required to
get leveldb fully imported before we could start using it
Of course, these are not the only pros/cons with any of these ideas,
just the main ones that come to mind as I'm writing this email.
What I would like everyone's input on is:
(1) More big pros/cons of the above ideas which may help swing the
decision one way or another
(2) More ideas on how to avoid trashing everything when we don't have
to (may be variations on the above, or other ideas that I haven't
thought of or have forgotten since the necko team discussed this in
person in February)
I also plan to point some known-smart people who may not subscribe to
this list at this post to try to get as many smart people thinking
about this as we can.
-Nick
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)
On 12-06-11 17:42 , Nick Hurley wrote:
> So far, I've come up with 3 possible ways of having better crash
> recovery. In no particular order, they are:
>
To me it seems like the simpler the approach, the better. I'm in favor
of option #1, unless somebody can show that it doesn't meet the
requirements.
Also, do you have somebody lined up to do this work already? I'm
interested in taking it on if you don't. As blassey pointed out, Fennec
would benefit a lot from having a more granular cache, since it gets
killed a lot by Android.
kats
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)
On 06/11/2012 11:42 PM, Nick Hurley wrote:
> -Keep some sort of hash of contents per entry (similar to how Chrome
> does things)
> *This could conceivably allow us to only delete entries that are known corrupt
> *This doesn't (necessarily) require any wild changes to the current
> on-disk format (it could be kept as a metadata field on all entries)
> *At first glance, the implementation seems relatively straightforward
> (adding the standard caveats here about how nothing is ever nearly as
> simple as one thinks, etc)
The main drawback of this approach is that it would probably affect the
performance. Without doing any major change in how we open/read/write
cache entries we would need to check the content when opening the entry.
This means, we would need to read and hash up to 50MB (this is the
default max_entry_size) even if we just want to check entry's metadata.
> -Use some sort of journaling and/or append-only structure to store the
> cache, with some markers for consistency per entry (or some other
> subdivision of the data)
> *The consistency markers are still undetermined
> *There are a lot of open implementations of filesystems using these
> kinds of structures, so lots of good reference material
> *I have a sneaking suspicion that even with lots of good reference
> material, this would not exactly be a simple implementation
I was looking into this option about a year ago and I couldn't come up
with any journaling solution (for our current data structures) that
wouldn't use fsync. And fsync is a no-go option...
Michal
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)
On Wed, Jun 13, 2012 at 9:01 AM, Michal Novotny
<> wrote:
> On 06/11/2012 11:42 PM, Nick Hurley wrote:
>>
>> -Keep some sort of hash of contents per entry (similar to how Chrome
>> does things)
>> *This could conceivably allow us to only delete entries that are known
>> corrupt
>> *This doesn't (necessarily) require any wild changes to the current
>> on-disk format (it could be kept as a metadata field on all entries)
>> *At first glance, the implementation seems relatively straightforward
>> (adding the standard caveats here about how nothing is ever nearly as
>> simple as one thinks, etc)
>
> The main drawback of this approach is that it would probably affect the
> performance. Without doing any major change in how we open/read/write cache
> entries we would need to check the content when opening the entry. This
> means, we would need to read and hash up to 50MB (this is the default
> max_entry_size) even if we just want to check entry's metadata.
Yes, I had thought of this when I was listing the options, not sure
why I didn't list it (I know I intended to).
>> -Use some sort of journaling and/or append-only structure to store the
>> cache, with some markers for consistency per entry (or some other
>> subdivision of the data)
>> *The consistency markers are still undetermined
>> *There are a lot of open implementations of filesystems using these
>> kinds of structures, so lots of good reference material
>> *I have a sneaking suspicion that even with lots of good reference
>> material, this would not exactly be a simple implementation
>
> I was looking into this option about a year ago and I couldn't come up with
> any journaling solution (for our current data structures) that wouldn't use
> fsync. And fsync is a no-go option...
You make a good point about fsyncing. However, I had envisioned this
as something that would involve changing our current data structures,
which then may or may not get us a no-fsync solution. You might have a
better idea as to whether or not that's possible, given that you've
already looked into journaling at least a little bit.
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)
On Wed, Jun 13, 2012 at 7:04 AM, Kartikaya Gupta <> wrote:
> On 12-06-11 17:42 , Nick Hurley wrote:
>>
>> So far, I've come up with 3 possible ways of having better crash
>> recovery. In no particular order, they are:
>>
>
> To me it seems like the simpler the approach, the better. I'm in favor of
> option #1, unless somebody can show that it doesn't meet the requirements.
See Michal's post :) The perf hit may or may not be a deal breaker (we
don't have any hard numbers to say one way or the other). Given that
Chromium uses a similar method, it might not be too bad, but with the
ever-increasing size of resources on the web, it may be a problem
sooner rather than later.
> Also, do you have somebody lined up to do this work already? I'm interested
> in taking it on if you don't. As blassey pointed out, Fennec would benefit a
> lot from having a more granular cache, since it gets killed a lot by
> Android.
We don't have anything set in stone yet, no. The general idea is that
Michal and/or myself would work on it, given that we're the most
familiar with the disk cache. However, since we don't even have a plan
yet, it's kind of hard to allocate resources :)
_______________________________________________
dev-tech-network mailing list
dev-tech-
https://lists.mozilla.org/listinfo/dev-tech-network
)
