Recipe: social listening pipeline

Goal: a daily job that finds new posts mentioning your brand, collects their comments, and hands the text to a sentiment model.

1. Find mentions

import httpx

BASE = "https://api.scrapersocial.com/v1"
H = {"Authorization": "Bearer sk_live_..."}
BRAND = "acme widgets"

hits = []
for path in ("tiktok/search", "instagram/reels-search"):
    r = httpx.get(f"{BASE}/{path}", params={"query": BRAND, "limit": 25},
                  headers=H, timeout=120).json()
    hits += [(path.split("/")[0], item) for item in r["data"]]

2. Pull comments on anything that spiked

def comments(platform: str, url: str, pages: int = 2):
    params = {"url": url, "limit": 50}
    for _ in range(pages):
        r = httpx.get(f"{BASE}/{platform}/comments", params=params,
                      headers=H, timeout=120).json()
        yield from r["data"]
        if not r["meta"]["has_more"]:
            return
        params["cursor"] = r["meta"]["cursor"]

mentions = []
for platform, item in hits:
    if item["metrics"]["views"] < 10_000:
        continue
    for c in comments(platform, item["url"]):
        mentions.append({"post": item["url"], "text": c["text"],
                         "likes": c["likes"], "at": c["posted_at"]})

3. Score and store

Feed mentions to your sentiment model of choice and upsert into your warehouse keyed on comment id — comment lists are age-aware cached, so re-polls are cheap on latency and idempotent in your store.

Cost model

25 + 25 search results (4 + 3 credits each) + ~10 posts × 100 comments (2–3 credits each) ≈ 2,400 credits/day worst case. Run it on X and LinkedIn too by adding their stats/comments endpoints — same request shape.

Related: Async jobs for bigger pulls · Social listening use case