How We Analyzed 18,250 Reddit Comments With Claude Code

In March 2026, we analyzed 18,250 Reddit comments to decide what an Instagram product for creators and social media teams should actually solve. Claude Code helped us build the pipeline in about two hours; the full analysis then ran for roughly three hours in the background.

The result was not “one giant prompt.” It was a repeatable system: collect public discussions, remove irrelevant text, score visible pain, cluster similar problems, and generate a report with links back to the evidence.

These figures come from our internal project run. They describe this one dataset and workflow, not a public benchmark of Reddit or Claude.

Sources 6 selected subreddits

Threads 1,670 public posts collected

Discussion 18,250 comments processed

Evidence 1,112 pain records retained

Structure 14 final pain clusters

Time 2h + 3h build plus background run

Why we did this

We were building a small product for creators and social media managers who work with Instagram. Before choosing features, we needed to know what people repeatedly struggled with—not what we personally assumed they needed.

Reddit was useful because creators and marketers often describe workarounds, failed tools, account problems, and buying questions in detail. We selected six relevant communities, including r/SocialMediaMarketing, r/content_marketing, and r/influencermarketing.

The aim was not to treat every comment as truth. It was to find repeated signals worth investigating through interviews, prototypes, and product tests.

The complete pipeline

01 Collect Posts, comments, source links, dates, and community

→

02 Clean Remove duplicates, noise, deleted text, and weak records

→

03 Filter Haiku classifies relevance and severity

→

04 Cluster Sonnet groups related pains and labels themes

→

05 Review Markdown report, source evidence, and HTML overview

Step 1: collect the discussions

The first component walked through a fixed list of communities and created structured records for public posts and their comments. Each record kept enough source information to return to the original discussion during review.

If we reproduced the project today, we would use authorized Reddit Data API access and follow its current rate limits and deletion requirements. Reddit requires OAuth, a unique user agent, and compliance with its current terms. Commercial use can require separate permission. Do not bypass technical limits or collect private content.

Step 2: remove obvious noise

Code handled the cheap, predictable work before any model call:

• empty, deleted, or extremely short records;
• duplicated text;
• obvious bot and moderator messages;
• fields that were not needed for analysis;
• malformed records.

This stage matters. Paying a model to classify duplicates increases cost and makes popular copied phrases look more important than they are.

Step 3: classify relevance and pain severity with Haiku

We used a smaller Claude model for the first judgment: does this text describe a relevant user problem, and how strong is the signal?

The prompt included concrete anchors:

## Severity Rubric (use these anchors consistently)

1 = Passing mention: "It would be nice if..." / "I wonder if there's a way to..."
2 = Has workaround: "I manually check each competitor..." / "I use spreadsheets to track..."
3 = Actively seeking solution: "Anyone know a tool for...?" / "What do you use to...?"

The rubric is intentionally small. A three-level scale is easier to apply consistently than a vague score from one to ten.

Step 4: cluster the retained pains with Sonnet

The smaller model filtered the large dataset. We then sent the retained records to Sonnet in manageable batches for the harder task: grouping related problems into broader themes.

Example themes included account restrictions and distrust of AI-generated content. The cluster name was only a navigation aid. The useful evidence remained the underlying records and links.

Step 5: calculate priority

For every cluster, we calculated a simple Priority Score:

How many people? Frequency

×

How painful? Severity

=

What to inspect first Priority Score

The score ranked research themes. It did not prove demand or replace customer interviews.

This was a prioritization shortcut, not market size. Frequency can be distorted by one active community, and severe stories can be overrepresented because people post when something goes wrong.

Step 6: build a report people could inspect

The final process generated a Markdown report with the top pain clusters, scores, representative evidence, and product questions. We discussed that report in a separate AI conversation that had the product context.

We also generated an interactive HTML overview. Its job was simple: show the relative size of clusters and let us move from a pattern to the underlying records. It did not need to look like a public dashboard.

What we learned from the dataset

One unusual story described a person managing more than 300 social accounts for a model agency. According to that respondent, high-value account bans could sometimes be resolved through a direct platform contact within hours, while smaller creators depended on the standard appeal process.

That is an anecdote from one discussion—not proof of an official Meta policy and not a finding we would generalize without more evidence. But it reveals a useful research question: how much does access to support change the real cost of an account restriction?

This is why the source trail matters. A surprising claim should send a researcher back to the conversation and then toward independent confirmation. It should not become a product decision just because a model placed it in a report.

Why the project was fast

Claude Code helped create the parser, transformation scripts, prompts, result schemas, report generator, and HTML view. The speed came from splitting the work correctly:

Deterministic code	Language models	Human review
Collection, cleanup, deduplication, batching	Relevance, severity, cluster labels, summaries	Rubric, source choice, cluster corrections, product decisions

One person could build and run a useful first version in hours because the model assisted with both code and language classification. Building, testing, and documenting the same system manually would likely have taken us at least several working days. That is our project estimate, not a universal delivery benchmark.

What we would improve next

1. Deduplicate by meaning, not only exact text. Similar reposts can inflate frequency.
2. Require structured JSON from every model step. Invalid records should fail validation instead of entering the report.
3. Review a random sample. Manually label a small set and compare it with the model before processing everything.
4. Track cluster stability. Run the clustering more than once and inspect themes that move or disappear.
5. Separate communities and time periods. A pain common in one subreddit may not represent the wider creator market.
6. Preserve deletion and source controls. Reddit’s current rules require attention to deleted content and current API terms.
7. Follow research with interviews. Comments reveal language and hypotheses; they do not confirm willingness to pay.

A safe way to reproduce the method

Start with one narrow question and two or three relevant communities. Collect only data you are authorized to use. Keep source IDs and links, minimize personal information, and document how deleted content will be removed.

Test the severity rubric on 100 records before scaling. Review false positives, adjust the anchors, and only then run the complete set. The business-data privacy guide explains how to choose the right model surface for sensitive research.

Bottom line

The important result was not that an AI could read 18,250 comments. It was that a small, inspectable pipeline turned unstructured conversation into product hypotheses in a few hours.

Code handled collection and consistency. Haiku removed noise. Sonnet organized the harder themes. A human defined the question, checked the evidence, and decided what deserved another test.

That combination is much more reliable than uploading a giant file and asking, “What do customers want?”