With Generative AI, Data Analysis Will Never Be the Same
But humans will still be helpful.
Image: Depositphotos
Source: Dr. Tom Davenport
My first paying data analysis job was more than 50 years ago. And for the first 45 years or so of my career in this field, it was a pretty consistent set of tasks. You put your data into rows and columns, figured out what to do with missing data, specified what type of analysis you wanted on what variables, and fed it all into a statistical analysis program. You hoped you didn’t make any spelling missteaks (sic), because you’d have to do it all over again. Interpretation of the results, of course, was up to the analyst, as was the explanation of the implications for action for your decision-maker boss.
How GenAI Can Change Data Analysis
But generative AI is clearly changing the process of data analysis. I’ve been experimenting with quantitative data analysis using ChatGPT for several years now. I was certainly impressed by the LLM’s ability to generate Python code to analyze structured data and create machine learning models. With only a short, two-line prompt it could do the following:
Figure out (in a CSV file) which were the independent variables to analyze as predictors of the dependent variable; all it had to be told was the label of the dependent variable.
Determine how best to handle missing data
Do a bit of feature engineering to see if it could create a better fit for the model
Try several different algorithm types and run a little competition among them
Describe which features had the greatest importance in the best-fitting model.
Even more impressively, the LLM could provide a page or two of recommendations for a manager seeking to maximize or minimize the dependent variable (I often used an attrition dataset from the telecom industry). It might suggest, for example, what products a company should offer in what bundles in order to reduce attrition.
There was only one problem with this approach to data analysis. I noticed that minor differences in prompt language led to differences in the results. I’m not experienced enough in Python to determine why this happened, but it made me hesitate to recommend ChatGPT-based data analysis to others. Perhaps that’s why LLM-driven machine learning isn’t practiced all that much.
Enter Claude
A few months ago I tried uploading a CSV dataset to Claude for analysis, but it told me that it couldn’t accept uploaded CSV files. But over the December holidays I had some data to analyze for a survey on the economics of AI. My collaborator on the project, Laks Srinivasan, suggested that we see what Claude could do with the analysis. He said that Claude still had issues with uploading CSV files, but that it seemed to accept SPSS data files, which we had received from the survey provider. Laks fed Claude the SPSS files and gave it some prompts with context about our study and what we were expecting to address in the reports and articles emerging from it.
And in general, Claude did an amazing job. It quickly identified what it thought were the most interesting and important findings. It created an outline for the final report and would no doubt have written a draft of the report had we let it. It performed crosstabulations of several variables, and identified some subtle findings that I am not sure I would have noticed.
We all, including Claude, were heavily focused on one issue. Laks and I had previously identified the survey questions and variables involved in AI economic maturity. Claude made that the focus of its analysis. As Claude pointed out, “Every consulting firm has an AI maturity model. Most are useless….We took a different approach: economic value creation maturity.” (bold font supplied by Claude).
Claude also found it very important to point out that many companies in the survey were “stuck” in the middle of the economic maturity model, particularly on Stage 3 (in which economic value from AI is measured but not aggregated across the company). It suggested several times that the surveyed companies had been stuck there for a median of six years. It commented:
“Nearly one-third of organizations (30.1%) are stuck at Stage 3 for a median of 6 years. They achieve respectable value (44.2%) but can’t aggregate because each use case is measured differently. The escape requires standardization, not more measurement.”
I found that conclusion interesting, since we didn’t ask companies in the survey how long they had been at each stage. When Laks asked Claude how he came to that conclusion that many companies were “stuck at Stage 3,” it replied that 6 years was the median time Stage 3 companies had been using AI. Claude did not acknowledge that companies could have been at a higher stage and fallen back, or that they had perhaps been in Stage 3 for a short time.
Claude may have been wrong about this finding, but it was never in doubt and continued to push that idea. To its credit, however, when we asked for an explanation it said that these companies “have achieved a fair amount of value but can’t standardize value across use cases.” The evidence it cited included:
52.8% cite “lack of AI-ready data”
50.5% cite “technology difficult”
49.2% cite “lack of framework”
38.3% cite “leaders lack understanding”
That logic was moderately compelling, so we left the “stuck at 6 years” in the report.
It also turns out that Claude is also a big evangelist for agentic AI. Perhaps that’s not surprising, as it represents an important direction for the LLM’s future. Even though a relatively small fraction (about a third) of companies in our survey were using agentic AI, and those using that technology got the least amount of high value from it (2%) than any other technology, Claude continued to be a strong advocate for it as a “sophistication signal.” Claude commented:
While the first three AI types are table stakes (80%+ adoption), Agentic AI is the sophistication signal. Organizations using Agentic AI consistently outperform across multiple value dimensions—not because Agentic AI itself creates all the value, but because deploying Agentic AI requires organizational maturity (measurement frameworks, governance, risk management) that also drives value in other AI initiatives.
We weren’t sure that any of these conclusions were true, and they were certainly overstated. The high value of AI that agentic adopters are getting is almost certainly coming from other AI technologies. But its comments were interesting, and we did hint at the “sophistication signal” in the report.
A Smart but Pushy Intern
Others have drawn this analogy before, but on our project Claude was akin to a very smart and pushy intern who saved us considerable time. We were able to engage in a dialog with it—in many cases an effective one. The LLM sometimes went too far beyond the data, and rarely backed away from the ideas it advocated. Once we did point out that it was over-generalizing, and it apologized for this behavior. But despite a few glitches, our work was definitely better as a result of having Claude on the team.
Will Claude replace a lot of data scientists or data analysts? I think the most likely effect of these capabilities will be the democratization of data analysis and some data science. More people will be analyzing data, not less. Some only moderately qualified practitioners may need to move into other fields. However, highly skilled analysts and data scientists will still be necessary. It’s occasionally important for somebody to tell Claude and its LLM relatives what to do, to figure out whether it’s making sense, and to push back on this pushy intern when it ventures too far away from the data.
Tom Davenport, Ph.D.
Dr. Tom Davenport is a world-renowned thought leader and author, is the President’s Distinguished Professor of Information Technology and Management at Babson College, a Fellow of the MIT Center for Digital Business, and an independent senior advisor to Deloitte's Chief Data and Analytics Officer Program.
An author and co-author of 25 books and more than 300 articles, Tom helps organizations to transform their management practices in digital business domains such as artificial intelligence, analytics, information and knowledge management, process management, and enterprise systems.
He's been named:
- A "Top Ten Voice in Tech" on LinkedIn in 2018
- The #1 voice on LinkedIn among the "Top Ten Voices in Education 2016"
- One of the top 50 business school professors in the world in 2012 by Fortune magazine
- One of the 100 most influential people in the technology industry in 2007 by Ziff-Davis
- The third most important business/technology analyst in the world in 2005 by Optimize magazine
- One of the top 25 consultants in the world in 2003