All right, here's some prod data on the webvac stuff. (The TL;DR if you missed it earlier: FSE is using venti to serve its static content directly out of its backups. Open-source, here's the README:
https://git.freespeechextremist.com/gitweb/?p=webvac;a=blob_plain;f=README;hb=HEAD , here's the README passed through RDoc if you think HTML is a reasonable thing to do:
https://www.rubydoc.info/gems/webvac/0.1.5 ).
This data came from the nginx logs for the last 12 hours. They are plots of the time the backend took against the filesize, for GET requests that didn't 404. The filesize is capped at 4MB; anything bigger and we just serve it from the FS.
The first image is a scatterplot with a fitted line for all reqs. It's a little spiky, as you can see, but the overhead looks like O(log(n)), which is better than expected (I figured it'd be linear; probably because traffic patterns mean the same resource is usually requested in succession so the blocks are all cache hits for venti). It is spiky, some of the reqs take up to 10s; this is probably a combination of server load and a cache miss.
The second graph is the same data but it averages time for each individual resource and tosses out the ones that were only requested once. There are still a few outliers, but overall it's performing better than expected.
venti-data.pngventi-data-avg.png