# How We Built a Data Pipeline: From Raw 2.8M Messy Tweets to Training Data

### The Problem

We wanted to build a customer support agent powered by real conversation data. We found a dataset of 2.8 million Twitter support conversations on Kaggle. Sounds perfect, right?

Not exactly.

### What's Actually in This Dataset?

The **Customer Support on Twitter** dataset is a collection of roughly 2.8 million real tweets — public support conversations between customers and brands. For example:

> **Customer → @Nike:** "My shoes haven't arrived yet. Where is my order?" **Nike → Customer:** "Sorry about that! Please DM us your order number." **Customer → Nike:** "Sure, I've sent it."

There are thousands of exchanges like this across many different brands.

### What "Messy" Actually Means Here

Since this data came straight from real Twitter activity, it isn't neatly organized. There's no clean "conversation" column — just individual tweets, each one only aware of the tweet it replied to.

And a single customer issue is rarely one tweet — it's usually a **chain**:

```plaintext
Customer tweet
      ↓
Brand reply
      ↓
Customer reply
      ↓
Brand reply
```

So the dataset is really ~2.8 million *individual* messages scattered across many different brand-customer exchanges — not 2.8 million conversations. Our actual job: turn this messy historical data into structured knowledge that a customer-support AI agent can learn from.

To train an agent that actually works, we needed clean, structured conversation threads — not a pile of individual tweets.

**So we built a pipeline.**

* * *

### Functional Requirements: What We Needed

**Input**

*   **2.8 million raw tweets** from Twitter support channels
    
*   Messy format: individual tweets, not grouped conversations
    
*   Missing metadata, duplicates, and noise
    

**Output**

*   **Clean conversation threads**
    
*   Properly grouped and deduplicated
    
*   Valid, structured data ready for training
    

* * *

### **Step 1: Download the dataset**

1.  Head to [kaggle.com](http://kaggle.com) and sign up (Google login works fine)
    
2.  Search for **"Customer Support on Twitter"** by thoughtvector
    
3.  Download this `twcs.csv` (516MB) from Kaggle to your machine.
    
4.  Put the downloaded `twcs.csv` into:
    
    ```plaintext
    customer-support-agent/data/raw/twcs.csv
    ```
    

**Understanding What's Inside the CSV**

Before writing a single line of code, it's worth actually looking at what this data contains.

At its core, the dataset is a giant table of tweets — roughly 2.8 million rows, each one a single message. Every row has seven columns, but only a handful matter for reconstructing conversations:

![](https://cdn.hashnode.com/uploads/covers/624226a5db84f8c50fa5b247/b1b0e991-51d8-49d1-9d99-8cbd634873ba.png align="center")

Those last two columns are the whole game. They're the only thing linking one tweet to another.

**Example from the dataset:**

```plaintext
tweet_id: 119237
author_id: 105837
inbound: true
text: "@AppleSupport hi #apple, I've a concern about the latest ios is too slow on #iphone6"
in_response_to_tweet_id: (none, first message)
```

```plaintext
tweet_id: 119248
author_id: AppleSupport
inbound: false
text: "@105837 We can help. Which version of iOS are you on? You can find that in Settings > General..."
in_response_to_tweet_id: 119237
```

![](https://cdn.hashnode.com/uploads/covers/624226a5db84f8c50fa5b247/be25c1cc-d176-41d5-831f-328d99c4bf3a.gif align="center")