Shell sees AI as fuel for its sustainability goals

Energy giants are under significant pressure from governments and consumers to reduce carbon emissions. For multinational oil and gas company Shell, artificial intelligence could be a key catalyst in achieving this long-term goal.

The London-headquartered energy company’s ongoing digital transformation, powered by a hybrid cloud platform and Databricks’ data lake house, includes a mix of AI technologies aimed at optimizing business efficiencies, profits and over time time to reduce its carbon footprint.

“AI has become central to our entire digital transformation journey,” says Dan Jeavons, Shell’s chief AI guru in 2015. About 20 Databricks employees are associated with the Shell account.

Jeavons, who served just six months as Shell’s vice president of computational science and digital innovation, is Shell’s former general manager of data science and has been knee-deep in data science since 2015.

In his new role, reporting to Shell Group CIO Jay Crotts, Jeavons will be tasked with using AI and emerging technologies such as blockchain, IoT and edge computing to overhaul Shell’s future technology strategy and drive its commitment to… to reduce its carbon footprint to become a net-zero emissions energy company by 2050.

Gartner AI analyst Anthony Mullens says Shell’s AI implementations go beyond what most other companies do. “Shell is over the hill in terms of initial experimentation across the company,” says Mullens, citing Shell’s Center for Excellence and participation in OpenAI.

Jeavons’ group has several hundred data scientists who use AI — primarily on Databricks’ Spark-based platform — and write algorithms to perform tasks such as failing as well as improving offerings for customers.

“Faced with the threat of climate change, we need to transition to a lower-carbon energy system, and digitalization is a key part of that,” says Jeavons, noting that many of the CO2 monitoring data streams will flow through Databricks’ AI platform. “Digital technology is one of the key levers we can pull to significantly reduce the carbon footprint of the energy system.”

According to Jeavons, Shell’s use of digital technology reduced CO2 emissions from a liquefied natural gas (LNG) facility by up to 130 kilotons per year – the equivalent of taking 28,000 US vehicles off the road for a year.

“A lot of the people who work for us have a compelling goal of actually using AI to try to accelerate the energy transition,” he says. “But I’m not going to pretend it’s easy.”

Data is the basis

As part of its digital transformation, Shell relies on two public clouds, Microsoft Azure and AWS, and Docker and Kubernetes containerization technologies to run increasingly advanced workloads across various aspects of its $210 billion oil and gas business.

A key aspect of this strategy, according to Jeavons, is the company’s foundational data layer — a pool from which multiple tools and technologies can access data in a systematic way.

“A dual cloud strategy means you need some consistency in how you manage and integrate your data. Now, of course, not all data will be in one place. They have a variety of databases; everyone does that,” says Jeavons. “But from an analytical perspective, we are increasingly consolidating certain types of data into an integrated Lake House architecture based on Databricks.”

On the analytics side, integrating data into a common layer in Databricks’ Delta Lake and using Python in a common platform allows for simple queries and the integration of classic report queries with visualization tools like Power BI.

But on the AI ​​front, “it also allows you to run the machine learning workloads all on the same platform,” says Jeavons. “It was a big change for me.”

For example, Shell has integrated all of its global time-series data – information like temperature, pressure, a specific device – into a common cloud powered by Delta Lake, allowing the energy giant to keep its finger on the pulse of its assets, including data from refineries, plants, upstream facilities, wind farms and solar panels. “Today, 1.9 trillion data series are aggregated, which is a huge amount globally,” says Jeavons. “We measure everywhere.”

Shell’s AI efforts also include performing failure predictions and assessing the integrity of its power assets by using machine vision to detect corrosion. “We’re also using AI to develop technologies that allow the plants to be optimized and run more efficiently at scale and optimized based on historical performance,” says Jeavons, noting that much of Shell’s AI magic comes from the Implementation of its data is attributed to Lake, none of which could be accomplished without cloud advances.

“Really, the most important thing was cloud maturation and the ability to remove some extra layers that we had [in order] Take data directly from the systems and stream it to the cloud. That was helpful to drive both the data analysis and the AI ​​strategy forward,” he says.

Along the road

In total, Shell employs approximately 350 professional data scientists and approximately 4,000 professional software engineers working remotely and/or in one of Shell’s hubs in Bangalore, India; the United Kingdom; the Netherlands and Houston, Texas.

Aside from cloud and data lake house, Shell has also moved to advanced development tools like Microsoft Azure DevOps and integrates GitHub into the way its developers work. It’s also deploying more sophisticated code-screening tools for the cloud, running “proper” CI/CD workflows, and monitoring “north” of 10,000 devices worldwide using AI as part of its remote monitoring centers, Jeavons says.

But it’s the development of a common Lake House architecture that has made the biggest difference, giving Shell “an integrated data layer that exposes all of the data across our organization in a consistent way,” says Jeavons.

“We were very early adopters of Delta,” he says. “For a while it was in proof-of-concept mode rather than deployed at scaled load. In the last 18 months we’ve really seen a clear change and we’ve been running pretty hard.”

However, change management remains one of the company’s greatest challenges.

“How do you embed the technology into the business process and make it usable and part of what happens every day and how do you develop algorithms that work? I will not underestimate how difficult it is. It’s not trivial,” says Jeavons. “It’s harder to develop adoption [of AI] on a scale. It’s still a long journey and we’ve made some progress, but there’s still a lot more to do.”