Cache Stampede
So today on a beautiful Friday morning, while I was thinking about how to be a Mark Zugburg 2.0, my youtube start to recommended some football matches. Of course, I don’t have time for it since I still thinking about how to get a job now. But at that moment, suddenly I got a question: How can a betting company update the result of the match instantly and can I make something similar?
Application X
Let’s say I am making an application called X that can update the result with data latency 1 second. It means if there is a change in the match (someone score, someone dies, etc), the data have to be updated after 1 second. This means after X seconds, the app will pull data from the API to do it.
The API logic is simple: Filter matches by day and check for video highlight. So what is the challenge here? The request per second (RPS). What if we have 100000 RPS? How can I solve that?
After checking around then I found this Cache Aside Pattern.
“Load data on demand into a cache from a data store. This can improve performance and also helps to maintain consistency between data held in the cache and data in the underlying data store.”
So the idea is: First we try to query from the cache and if we have data, we will return it right away, if not, we will perform query data and put it in the cache.
def fetch_data(key):
try:
data = cache.get(key)
except Exception as e:
data = fetch_data_from_other_stuffs()
cache.put(key,data,TTL=10000)
return data
But now we have a new problem when we first start the application, all requests will be empty (cache missing — dogpile the database — time out).
I did a few searching and found out this is called: Cache Stampede.
Solutions
So, in order to build my billion (fake) dollars application. I need to find a way to solve this. Did a few more digging and I found 3 solutions.
Lock
We can use lock so we can avoid the repeating of cache misses.
- Request A gets the lock.
- Other requests that do not get the lock will stay and await.
- After request A finishes, it will write down into cache.
- Other requests read from the cache.
Cache warming
Setup a working so it can always update the cache when it about to expire (cronjob aside).
Predict cache expiration
We can try to predict at point X, a key will need to be updated base on other data that got expired before.
From my point of view, the 1 and 2 are easier to make, the 3 one is more likely to create a ML. Of course, if I can make a ML to predict it, I won’t make the application anymore, I will go to bet and be a billionaire for real. :)).
CloudFront
The final approach would be to make a content delivery network ( I just learned about it from one of my interviews). Basically, it will deliver the static content of the app so it can load faster for users.
After found out the solution, I decided to … not doing it, and go to learn other stuff. I mean, I don’t watch football anyway.
Budapest 05/02/2021
Rice
