Did The Shift Project really overestimate the carbon footprint of online video ?

An analysis published in March correctly pointed out an error by The Shift Project regarding the climate footprint of online video. This error appeared during an interview. It has no impact on the results published in our reports, which are not contested. Here is our detailed explanation. We call for further scientific discussion.

Background: The Shift Project wanted to comment on George Kamiya’s CarbonBrief article “Factcheck: What is the carbon footprint of streaming video on Netflix?” published on  February 28th 2020 (republished on the IEA,  website on  March 25^th 2020: “The carbon footprint of streaming video: fact-checking the headlines”). G. Kamiya’s article testifies to the progress of scientific discussion concerning the environmental impact of digital technology. This is a most welcome development that The Shift Project intends to support by discussing in detail G. Kamiya’s article.

The current debates on the environmental impact of digital technology have recently acquired a quantitative edge. We, The Shift Project, approve of this new stage, as we consider this to be the actual challenge to be met. Along with major historical players in this field, we have been advocating for this type of research since 2018.

The Shift Project is in contact with George Kamiya, and would like to follow his most recent research developments that could be integrated into our updated results, along the lines presented last January in our interim report.

At this point, we consider G. Kamiya’s paper to be the most comprehensive and transparent review of our 2019 results. It does point out to a figure that is undeniably wrong. However, this piece of data is not to be found in any of our reports, but has been provided by one of our collaborators, during an interview, and subsequently presented misleadingly in several media outlets.

This unfortunate event reveals the importance of considering system-wide dynamics as a whole. The media tend to systematically focus on Netflix in particular, showing how difficult it is to deal with the issue of digital practices. It must be noted that in the media debate, this issue always crystallises around blaming particular uses, thus mostly shunning a systemic reflection on the digital system itself. This is why (among other reasons), when producing figures we, at The Shift Project, always refrain from referring to a single player in particular.

The source of the figure for “30 minutes of Netflix” in CarbonBrief’s article, actually corresponds to a contraction of some data provided by the AFP, regarding an average consumption for a 30 minutes of online video. This figure was later taken up and broadcasted by several media, including France 24, the New York Post, CBC, Yahoo, DW, Gizmodo, Phys.org and BigThink. Another reason why The Shift Project figures never refer to a single player in particular is that individual actors rarely disclose enough data to allow for precise conclusions – though they have been increasingly transparent since the topic has come to the forefront in the last 18 months.

No matter how complex a task, it is crucial at this stage to investigate the “digital system” as such and to consider collective strategies to be implemented. G. Kamiya rightly does so in his article, particularly in the last part. Indeed, he addresses the issues that relate to the digital system as a whole, and highlights their central importance: dealing with the rebound effect issue, designing large-scale strategies by bringing all players together, etc. Considering the “digital system” from the point of view of uses means dealing with the uses as part of a larger system, which has its own logic, players, infrastructure, regulation, history, and so on. As far as the responsibility of individuals is concerned, as we wrote in 2019: “The broadcasting platforms (their design, the underlying economic model, audience metrics, etc.) play a central role in the form taken by uses and thus their environmental impact. So, uses are to a great extent the product of a system, and not the sole result of individual consumer behavior.”

We provide here the most recent update on our results, and we will pursue the scientific dialogue with experts on these issues, away from discussions held through the media. As G. Kamiya states in the conclusion of his article for CarbonBrief: “Instead of relying on misleading media coverage, [achieving the goals of the Paris Agreement] will require rigorous analysis, corporate leadership, sound policy and informed citizens.”

We discuss each part of G. Kamiya’s article in detail. We intended to make each comment as self-contained as possible (this accounts for the multiple repetitions). Below is a summary in 4 points:

A rightly pointed error, but without consequences for the results presented in The Shift Project‘s reports

1/ G. Kamiya quite rightly points out an error on the CO₂e emissions corresponding to watching 30 minutes of video (although the data in question are to be found neither in our 2018 report, nor in our 2019 report: the source is an unchecked oral interview). There was indeed a major error on the bitrate (3 MBps instead of 3 Mbps), which accounts for 90% of the discrepancies exposed in the CarbonBrief article (i.e. discrepancies between G. Kamiya’s approach and the misleading figure mentioned by the relevant interview):

• The gap between the emissions calculated by G. Kamiya for a 30-minute video, and the emissions’ figure from the interview with The Shift Project’s collaborator (including the bitrate error) ranges from a factor of 28 to 57;
• Once the bitrate error is corrected, the difference between these two calculations is down to a factor of 4 to 7.
• While the discrepancy does remain substantial after correction, it clearly appears that it is mainly due to one single error, which has no effect on the overall results published by The Shift Project reports (“Lean ICT – Towards Digital Sobriety” (2018) and “Climate crisis: the unsustainable use of online video” (2019)). The details and reasons for these discrepancies are discussed later in this paper.

Impact of the bitrate error correction (bit-to-byte conversion error) on the difference between The Shift Project’s and G. Kamiya’s models. The correction reduces the difference between the two approaches by 90%. For details and sources, please refer to page 17 of the full analysis.

Our bitrate error accounts for 90% of the observed discrepancies, but a scientific discussion the remaining 10% is needed

2/ Some 10% of the observed discrepancies remain (the difference between the two assessments after correction of the bitrate error, i.e. a gap of 0.143 to 0.172 kgCO₂e), which are mainly accounted for by the different assumptions regarding the power consumption of network infrastructures (kWh/byte of data transferred). These assumptions account for most of the remaining 10% of differences after correction of the bitrate.

• This means that the scientific discussion (i.e. the task of comparing and analysing research results and hypotheses – whereby we engage with the experts, away from discussions held through the media) primarily focuses on network infrastructures’ electricity consumption, and on the underlying assumptions. We also plan to update our scenarios’ results: this undertaking is part of a new research step on the digital system (see the interim report published on 16^th January 2020).
• The scientific discussion could proceed along several avenues. On the one hand, the accounting methods for power consumption should be compared. While our macro approach only allows for reasoning on average values (thus forbidding any calculation on a specific player), it allows us to compute both indirect and fixed consumptions from every component of the network infrastructure. On the other hand, there are major discrepancies between the various evaluations of global power consumption, e.g. the consumption of data centres: the IEA’s evaluation is significantly lower than those of The Shift Project, Greenpeace East-Asia, GreenIT and Borderstep Institute. If we are to make collective progress, we need to compare and confront both methodologies and results.
• Assumptions other than network infrastructure consumption have interesting, though much less decisive effects.

Comparisons of data centre consumption estimates in the world, China and Europe in 2018, according to several studies.
Sources: The Shift Project, IEA, GreenIT.fr, Greenpeace East Asia, Borderstep Institute.

The uncertainties of modelling

3/ After correction of the bitrate, our model yields 0.400 kgCO₂e per hour. This represents only 10% of the discrepancy noted by G. Kamiya, i.e. 4 to 7 times his own estimates. In order to properly assess these deviations, we must confront them with the uncertainties of his own modelling: his results vary by a factor of 2 between average estimates, and up to a factor of 32 between outlying values. One ought to bear in mind these intrinsic uncertainty ranges in order to correctly interpret the discrepancies observed between the figures produced by various sources (IEA, The Shift Project, Greenpeace East-Asia, GreenIT, etc.).

• The intrinsic sensitivity of G. Kamiya’s modelling leads to uncertainties and ranges of values that span factors of 2 to 32: there is a factor of 2 between the lowest figure in the average range given by G. Kamiya (0.12 kWh or 56 gCO2e per hour of video) and the highest figure in the average range (0.24 kWh or 114 gCO2e per hour); there is also a factor of 32 between the figure for the lowest consumption consuming extreme scenario modelled by G. Kamiya (SD video watched on smartphone via Wi-Fi – 0.037 kWh per hour) and his highest consumption extreme scenario (4K video watched on smartphone via 4G network – 1.199 kWh per hour).
• These uncertainties are fully consistent with those typically occurring when modelling the power consumption of digital systems, with the choice of assumptions and parameters having a very significant impact on the final result.
• After correction, our data still diverge significantly from G. Kamiya’s results (by a factor of 4 to 7), but they are very close to the figure from the first scientific paper cited in the CarbonBrief paper (Shehabi, A., et al. (2014). The energy and greenhouse-gas implications of internet video streaming in the United States). Three factors account for the discrepancy between G. Kamiya’s assessment and the results proposed by Shehabi A. et al., the latter converging with our corrected result:
• • There are significant year-to-year technology-induced energy efficiency gains in data transmission, particularly in network infrastructures and data centres;
  • Some IEA estimates of the global energy consumption of digital systems are lower than those of other studies (including data centre power consumption, as detailed later in this paper);
  • The improving energy efficiency of data transmission processes (i.e. the decreasing energy consumption associated with the transfer of one minute of video) tends to be offset by a competing dynamic of rapid increase in average bitrates and volumes of video content, with the spread of high- and very high-definition (which tends to increase the energy consumption of transferring one minute of video).

Despite the divergences between our modelling methods, The Shift Project agrees with G.Kamiya’s macro results and with the need to closely monitor the explosive growth of digital technologies and services

4/ In the last part of his article, G. Kamiya expresses concern about the dramatic increase in video usage as well as the soaring energy consumption it involves (he highlights the evolution of Netflix’s power consumption, which supposedly increased by 84% between 2018 and 2019). Despite the divergences between our modelling methods, we do agree with his macro results and with the need to closely control the dynamics of our digital system. The bulk of the issue remains indeed the yet unbridled pace of growth of the digital system, which absorbs all the efficiency gains due to technological progress. G.Kamiya concludes: “What is indisputable is the need to keep a close eye on the explosive growth of […] digital technologies and services to ensure society is receiving maximum benefits, while minimising the negative consequences – including on electricity use and carbon emissions.”

An indeed, by disregarding the constraints of our finite world as we design and choose our digital practices, we run the risk that these constraints will apply randomly: this was the conclusion of our 2019 report “Climate crisis: the unsustainable use of online video”. Exogenous crises, whether climate- or energy-induced, will challenge the resilience of all our systems, including the digital system – health crises too, as we have been experiencing globally the consequences of the COVID-19 for several months. In the 21^st century, it would therefore be a major mistake to disregard the need for an appropriate management of our digital system.

Advancing the scientific discussion on the energy consumption of network infrastructures: more transparency and more work are needed

We thank G. Kamiya for publishing his results, sources and assumptions at the end of this article on the CarbonBrief website, as well as for the transparency effort made by publishing the calculator in his article on the IEA website. Our own transparency efforts have enabled G. Kamiya to analyse our work. His transparency efforts have enabled us to analyse his work. These combined efforts help the advancement of scientific discussion on the energy consumption of network infrastructures.

…now that you’ve come so far, and if you are interested in the scientific discussion of the issue, you can also read our full analysis of G.Kamiya’s paper (29 pages): “Did The Shift Project really overestimate the carbon footprint of online video? Our analysis of the CarbonBrief and IEA articles” (The Shift Project, June 2020)

Effect of the bitrate error correction (bit-to-byte conversion error) on the difference between The Shift Project’s and G. Kamiya’s models. The correction reduced the difference between the two approaches by 90%. The remaining difference sheds light on the nature of the scientific discussion, which lies primarely in the energy consumption of network infrastructures, with the assumptions made about this consumption having a major influence on the outcome. For details and sources, please refer to page 18 and 21 of the full analysis.

• Jean-Noël Geist, jean-noel.geist@theshiftproject.org | 06 95 10 81 91