Virtually Human: AI’s GPT-4 vs. Human Data Analysts – who won?

One of GPT-4’s strengths lies in its ability to comprehend data by making appropriate comparisons.

As large language models continue to showcase their impressive capabilities across numerous domains and tasks, including context comprehension, code generation, language generation, and data storytelling, among others, it’s understandable that many data analysts might express concerns about the potential impact of artificial intelligence on their profession. This controversial topic has garnered significant public attention.  

However, we find ourselves in a phase characterized by divergent opinions, without a definitive consensus having emerged. The question is: is GPT-4 an effective data analyst? 

Performance Test  

GPT-4 and human data analysts tackle tasks across various datasets including data extraction, data visualization, and drawing insights from the data.  

Out of the three levels of human data analysts examined, there’s a clear performance hierarchy. Overall, the senior analysts tend to outperform their junior counterparts, who in turn surpass the interns, particularly when it comes to analysis correctness and complexity.  

“With any new technology there’s going to be really positive use cases and there’s some things we need to really consider,” Deng said during a SXSW session on Ai. ”The way we can suss those things out, and what those challenges are, and how we solve them, is to release (technology) at a responsible rate.”

Now, how does GPT-4 stack up against these human analysts? Interestingly, GPT-4 outperforms both the junior analysts and interns on most metrics. However, there’s still a noticeable gap between GPT-4 and the senior analysts in areas like figure correctness, figure aesthetics, and analysis correctness. 

Apart from its comparable performance to human analysts, one standout aspect of GPT-4 is its efficiency. GPT-4 completed the tasks in significantly less time compared to even the most experienced human data analysts. So, while it may not quite match the senior experts in certain areas, GPT-4’s speed is quite remarkable. 

Overall comparison between several senior/junior/intern data analysts and GPT-4 on 100 random examples in total. Time spent is shown in seconds (s). 

GPT-4’s Strengths 

One of GPT-4’s strengths lies in its ability to comprehend data by making appropriate comparisons. For instance, it can identify the “most successful”, “less successful”, or highlight a “diverse range” of data points. Moreover, GPT-4 demonstrates an aptitude for extracting insights from datasets, such as recognizing patterns that “indicate dominance in the competition.”  

These skills encompass several of GPT-4’s core capabilities, including contextual understanding, code generation, and data storytelling. By seamlessly integrating these abilities, GPT-4 can effectively analyze information, generate code or queries as needed, and then communicate its findings in a clear and compelling narrative. 

Comparably Human 

So, whether it’s making sense of complex datasets, automating analytical processes through code, or conveying insights in an engaging manner, GPT-4 showcases a well-rounded proficiency that could potentially augment or complement the work of human data analysts. 

GPT-4 can outperform an intern data analyst or a junior data analyst and can achieve comparable performance to a senior data analyst, but further studies are needed before concluding that GPT-4 can replace data analysts. 

The flow of our proposed framework regarding GPT-4 as a data analyst.  

At the close of his SXSW presentation, Deng expressed that ChatGPT and other AI technologies serve as potent equalizers in the job market. He regards ChatGPT as a “thought partner,” capable of aiding humans in their tasks and professions. Nonetheless, he acknowledged that individuals leveraging AI are likely to excel in their careers compared to those who do not.

Source:  

DAMO Academy, Alibaba Group, Singapore; Hupan Lab, 310023; Hangzhou, China Nanyang Technological University, Singapore  

Jan Iverson is the Creative Director at FS Studio and an award-winning product leader with over 20-years of extensive experience in digital media and marketing, with a specialization in the design and development of AR, VR and 3D activations: mobile apps, games, LBE, sales tools, digital twins; with XR cross-platform content development, and a track record of success in leading award-winning digital creative teams. Virtually Human is her bi-weekly series.