About a quarter of South Koreans already pay for AI services, compared to just 2% of Americans
South Korea is preparing to offer free generative AI services to its entire population, using the technology to help people handle everyday tasks such as booking medical appointments, finding housing, and sorting out taxes.
The program is intended to expand access to homegrown AI systems while giving the government a larger role in how they are used for everyday services.
They still scale to some degree. There’s a certain amount of VRAM you need to even load a model (frontier models likely over a terabyte now), but once you’re there, adding extra users is less cost since you only need to accommodate their KV caches on top of the model of itself. You’ll need to add more GPUs once you’ve saturated what you have, but it’s nothing compared to the ~5 B300s for the first user (they actually come in servers containing 8 GPUs that cost about 500k EUR each, so realistically you’d buy one DGX B300 server to start with). Once you’ve loaded the model itself which has a fixed size, all the extra capacity you add is free to use for KV cache (which means more users).
It’s true, but the reason they depreciate so quickly is that there’s a new generation out every 2-3 years that does more using less energy. You could actually run old GPUs, it’s just cheaper to buy new ones if you’re a hyperscaler.
on the point of depreciation. would a hyperscaler be able to make a profitable return on a per GPU basis before they’ve maxed out its use?
genuine question not a rhetorical.