The vast majority of data engineers are burnt out, a recent survey found. Experts say the healthcare sector is no exception.
The survey, which was commissioned by data.world and DataKitchen, reached 600 data engineers, 100 of whom are managers. Nearly all of them reported experiencing burnout in their daily roles, and most reported considering leaving the industry entirely or their current company in the next year.
The study defined a data engineer as someone who prepares data for use by data analysts or scientists, who analyze it. Christopher Bergh, CEO and co-founder of DataKitchen, told Fierce Healthcare that the aim of the study was to address the deteriorating mental health of data engineers and to focus the study on their emotional states as opposed to just on business values.
The survey identified the greatest sources of burnout to include too much time spent fixing errors, repetitive manual tasks related to data preparation and a constant stream of requests from colleagues, many of which are often unrealistic.
Other barriers identified by more than two-thirds of respondents included being scapegoated for problems, data governance policies that stymie transparency and productivity and disruptions to work-life balance from unexpected workloads. Most (78%) of respondents said they want a therapist to come with their job.
More than half felt their companies did not seriously address or test for data quality issues—which inevitably lead to errors and production failures—in a thorough way. In a summary report of the survey, the companies noted that managing constant outages while also trying to develop new projects “is like trying to play ‘whack a mole’ while simultaneously reading a book.”
Data professionals work together to help drive data-based decisions that can impact operations. In the context of healthcare, that could look like finding and testing compounds for new drugs, a meticulous process that requires formulating hypotheses and testing them out. Anything to speed up development cycle time would greatly benefit a business, Kurt Zimmer, former head of data engineering for data enablement at AstraZeneca, which is a DataKitchen client, told Fierce Healthcare.
Frustratingly, there is “too much data, too many things in process, too many people in the loop,” he said. He was brought on at AstraZeneca to speed up decision-making time and yet saw their budget diminish and, with it, their ability to hire the right people. Burnout is inevitably an “obvious” symptom of that type of frustration. Neither he nor Bergh were surprised by the findings in the DataKitchen survey, given their own experiences in the field.
In healthcare, there is a “promise of a data-driven future,” Bergh told Fierce Healthcare, such as precision medicine. Yet the work it takes to achieve it is exhausting and filled with too many daily tasks, he said. And while technology is rapidly evolving, the processes in place to use it are not. These antiquated methods drive a “fixing crap” culture instead of systematically addressing root problems and maximizing data engineer productivity, Zimmer said. The survey found that only about 22% of data engineers’ time is spent on innovation that drives value.
To combat the problem, Zimmer and Bergh commended a methodology known as DataOps, which DataKitchen offers. It automates mundane processes, ensures the quality of underlying data flows, improves collaboration between teams and ultimately enables data engineers to focus on tasks that drive higher value. “The system makes the team,” Bergh said.