Maybe you should stop smelling text and try reading it instead. :P
Running an LLM in deployment can be done locally on one’s machine, on a single GPU, and in this case is like playing a video game for under a minute. OpenAI models are larger than by a factor of 10 or more, so it’s maybe like playing a video game for 15 minutes (obviously varies based on the response to the query.)
It makes sense to measure deployment usage marginally based on its queries for the same reason it makes sense to measure the environmental impact of a car in terms of hours or miles driven. There’s no natural way to do this for training though. You could divide training by the number of queries, to amortize it across its actual usage, which would make it seem significantly cheaper, but it comes with the unintuitive property that this amortization weight goes down as more queries are made, so it’s unclear exactly how much of the cost of training should be assigned to a given query. It might make more sense to talk in terms of expected number of total queries during the lifetime deployment of a model.
You’re way overcomplicating how it could be done. The argument is that training takes more energy:
Typically if you have a single cost associated with a service, then you amortize that cost over the life of the service: so you take the total energy consumption of training and divide it by the total number of user-hours spent doing inference, and compare that to the cost of a single user running inference for an hour (which they can estimate by the number of user-hours in an hour divided by their global inference energy consumption for that hour).
If these are “apples to orange” comparisons, then why do people defending AI usage (and you) keep making the comparison?
But even if it was true that training is significantly more expensive that inference, or that they’re inherently incomparable, that doesn’t actually change the underlying observation that inference is still quite energy intensive, and the implicit value statement that the energy spent isn’t worth the affect on society
That’s a good point. I rescind my argument that training is necessarily more expensive than sum-of-all-deployment.
I still think people overestimate the power draw of AI though, because they’re not dividing it by the overall usage of AI. If people started playing high-end video games at the same rate AI is being used, the power usage might be comparable, but it wouldn’t mean that an individual playing a video game is suddenly worse for the environment than it was before. However, it doesn’t really matter, since ultimately the environmental impact depends only on the total amount of power (and coolant) used, and where that power comes from (could be coal, could be nuclear, could be hydro).
You’re absolutely right that the environmental impact depends on the source of the energy, and less obviously, by the displaced demand that now has to seek energy from less clean sources. Ideally we should have lots of clean energy, but unfortunately we often don’t, and even when AI uses clean sources, they’re often just forcing preexisting load elsewhere. If we can start investing in power infrastructure projects at the national (or state/province level) then maybe it wouldn’t be so bad, but it never happens at a scale that we need.
I think the argument isn’t the environmental impact alone, it’s the judgement about the net benefit of both the environmental impact and the product produced. I think the statement is “we spent all this power, and for what? Some cats with tits and an absolutely destroyed labour market. Not worth the cost” Especially because it’s a cost that the users of AI are forcing everyone to pay. Privatize profits, socialize losses, and all that.
I think a different way to look at what you’ve brought up in the second paragraph is that people are angry and talking about the power usage because the dislike AI, not the other way around. It doesn’t really make sense for people to be angry about the power usage of AI if the power usage had no environmental impact.
I agree, although I think that phrasing it as “dislike” does a disservice to the legitimate grievances.
To give it a more nuanced spin, it’s not so much about disliking one because of the other, it’s about taking everything together. The power usage is just one more grievance, exacerbating opinions on AI.
I think the reason that power usage comes up a lot is because it’s easy to discuss, while talking about it through the lens of economics or communal good can easily get derailed.
Maybe you should stop smelling text and try reading it instead. :P
Running an LLM in deployment can be done locally on one’s machine, on a single GPU, and in this case is like playing a video game for under a minute. OpenAI models are larger than by a factor of 10 or more, so it’s maybe like playing a video game for 15 minutes (obviously varies based on the response to the query.)
It makes sense to measure deployment usage marginally based on its queries for the same reason it makes sense to measure the environmental impact of a car in terms of hours or miles driven. There’s no natural way to do this for training though. You could divide training by the number of queries, to amortize it across its actual usage, which would make it seem significantly cheaper, but it comes with the unintuitive property that this amortization weight goes down as more queries are made, so it’s unclear exactly how much of the cost of training should be assigned to a given query. It might make more sense to talk in terms of expected number of total queries during the lifetime deployment of a model.
You’re way overcomplicating how it could be done. The argument is that training takes more energy:
Typically if you have a single cost associated with a service, then you amortize that cost over the life of the service: so you take the total energy consumption of training and divide it by the total number of user-hours spent doing inference, and compare that to the cost of a single user running inference for an hour (which they can estimate by the number of user-hours in an hour divided by their global inference energy consumption for that hour).
If these are “apples to orange” comparisons, then why do people defending AI usage (and you) keep making the comparison?
But even if it was true that training is significantly more expensive that inference, or that they’re inherently incomparable, that doesn’t actually change the underlying observation that inference is still quite energy intensive, and the implicit value statement that the energy spent isn’t worth the affect on society
That’s a good point. I rescind my argument that training is necessarily more expensive than sum-of-all-deployment.
I still think people overestimate the power draw of AI though, because they’re not dividing it by the overall usage of AI. If people started playing high-end video games at the same rate AI is being used, the power usage might be comparable, but it wouldn’t mean that an individual playing a video game is suddenly worse for the environment than it was before. However, it doesn’t really matter, since ultimately the environmental impact depends only on the total amount of power (and coolant) used, and where that power comes from (could be coal, could be nuclear, could be hydro).
You’re absolutely right that the environmental impact depends on the source of the energy, and less obviously, by the displaced demand that now has to seek energy from less clean sources. Ideally we should have lots of clean energy, but unfortunately we often don’t, and even when AI uses clean sources, they’re often just forcing preexisting load elsewhere. If we can start investing in power infrastructure projects at the national (or state/province level) then maybe it wouldn’t be so bad, but it never happens at a scale that we need.
I think the argument isn’t the environmental impact alone, it’s the judgement about the net benefit of both the environmental impact and the product produced. I think the statement is “we spent all this power, and for what? Some cats with tits and an absolutely destroyed labour market. Not worth the cost”
Especially because it’s a cost that the users of AI are forcing everyone to pay. Privatize profits, socialize losses, and all that.
I think a different way to look at what you’ve brought up in the second paragraph is that people are angry and talking about the power usage because the dislike AI, not the other way around. It doesn’t really make sense for people to be angry about the power usage of AI if the power usage had no environmental impact.
I agree, although I think that phrasing it as “dislike” does a disservice to the legitimate grievances.
To give it a more nuanced spin, it’s not so much about disliking one because of the other, it’s about taking everything together. The power usage is just one more grievance, exacerbating opinions on AI.
I think the reason that power usage comes up a lot is because it’s easy to discuss, while talking about it through the lens of economics or communal good can easily get derailed.