Understanding User Engagement with Cross-Platform Social Media Content Created by Humans Versus AI: An Evaluation of ChatGPT in Content Marketing

Abstract

Even though generative artificial intelligence (GenAI) is increasingly integrated into user-facing technologies like social media, its impact on content marketing remains unverified. Early evidence suggests that language models (LLMs) can generate content that rivals human-created content (HCC) in terms of appeal. However, the question of adapting such content for various social media platforms remains unanswered. This study examines the effectiveness of an LLM, GPT-4, in customizing cross-platform content for Facebook, Instagram, and X. A total of 892 participants evaluated 30 pairs of AI-created content (ACC) and HCC. The findings reveal that ACC was preferred by users, delivered stronger calls to action, and elicited more user engagement than HCC, especially on Facebook, with a less pronounced effect for shorter posts on X and Instagram. We further generated six data-driven user personas of the 892 participants, illustrating the differences between those who preferred ACC or HCC on the three platforms. The results indicate that GPT-4 can adapt content to platform-specific requirements and maintain high perceived quality, making LLMs applicable for cross-platform content creation for user engagement. Findings contribute to understanding user engagement with AI-generated content across platforms. We also discuss the role of LLMs in content creation, including their ethical implications.