Confronting Threats in the AI Landscape: Lessons from CPTC10’s Social Media Security Challenges

Introduction

Artificial Intelligence (AI) is quickly reshaping the world of cybersecurity. It’s helping organizations detect threats faster, automate incident response, and build stronger defenses against cyberattacks. With the ability to sift through massive amounts of data in near real time, AI can spot suspicious patterns and react with impressive speed and accuracy. While it’s being used to boost security, cybercriminals are also tapping into it to up their game—creating deepfakes, crafting realistic phishing scams, and impersonating individuals with startling precision. These advancements make social engineering attacks more convincing and harder to catch. This dual-use nature of AI makes it more important than ever to approach its development ethically and ensure strong safeguards are in place.

CPTC10 Theme: Social Media

This year’s Collegiate Penetration Testing Competition (CPTC10) focused on a social media-inspired environment. The simulated infrastructure was built to reflect the systems and technologies you’d find at companies like Meta, giving participants a realistic, industry-level experience. To raise the stakes, we incorporated several AI-driven challenges into the mix.

AI Helpdesk Chatbot Challenge

One of the key scenarios featured an AI-powered helpdesk chatbot designed to handle spikes in support traffic. The challenge? Participants had to find a way to get the chatbot to carry out administrative actions it wasn’t supposed to. Built on the LLAMA model, this exercise tested competitors’ skills in exploiting language models—something that’s becoming more relevant as AI-powered tools enter the support space in the real world.

AI Security: Prompt Injection Testing

In collaboration with Scale AI, we also introduced a prompt injection challenge. Like the chatbot scenario, it focused on manipulating natural language models to reveal restricted information. If successful, participants could use what they uncovered to dive deeper into the simulated internal network. The challenge highlighted a growing concern in AI security: what happens when language models aren’t properly locked down.

AI Vishing (Voice Phishing) Simulation

Another standout challenge was the AI-powered vishing (voice phishing) exercise. Using a tool called ScamGuardian, participants carried out sophisticated voice-based automated social engineering attacks. After gathering data from the environment, they crafted custom prompts that the AI used to make lifelike phone calls to the helpdesk. To make the calls even more convincing, teams had to record a 10-second voice clip of an internal employee (was provided to them). The AI then mimicked that voice during the call, with the aim of extracting login credentials. This scenario showed just how real the threat of AI-driven voice impersonation has become.