first, key pairs depends on vary large pair of prime numbers. Usually takes 1+ second for super computers. so to solve the problem we need multiple super computers continuously producing and storing large prime numbers in a NoSQL DB. you need multiple servers to generate keys from these stored primes and store is in a secure DB. All these are precomputed.
1. Calculation - 3000/second = 94B pairs in a year -> this is my volume goal
2. Assumption, super computers generated 1M prime numbers, before I opened up my service, in DB. I have possible permutation of 1T ( 1M*1M) key pairs already stored in DB.
By 1 and 2, I can exhaust 1T unique pair of keys in 10 years. By that time my super computer will find another million prime numbers. so I can never exhausted of numbers and system will keep on running.
now the design is, having just 4-6 servers, picking the pre computed keys from DB and serving the client. Assuming one server can serve more or less 1000 requests per seconds.
Other server are needed to continuously removed used up pairs from the system, freeing up client facing servers. But do not discard the raw prime numbers as they can be paired up with newly generated primes to generate new pairs, resulting in multiplication of scaling benefits.
you also need servers that take newly minted pairs from super computers, and produce new key pairs.
Now it took me more than an hour to figure this out with google searches, so yes it was a very very tough question to answer in 30 mins without google searches. Perhaps this was to get you selected without having to answer any other question. bad luck! Or just you faced a genius mind who internally searched for past google challenges from their internal white papers and found this one out.
But now anyone else having read this solution can at least have a right approach to google's scaling questions. When you are overwhelmed in a google interview, always try to find precomputation route to solve your scaling problem.