MyNeedEcho
Home › How data is collected & verified

How data is collected & verified

Every need on MyNeedEcho traces to a real, linkable discussion. Here is exactly where the data comes from, how it is scored, and how we stop fabricated claims from ever reaching a page.

The 7-stage pipeline

Each daily report runs through these stages, in order, at 09:00 UTC:

1
Collection — Pull fresh discussions and posts from developer/founder communities and public tenders via their official APIs.
Hacker News, GitHub trending + issues, DEV.to, Product Hunt, Reddit, Stack Overflow, Bluesky; EU TED and US SAM.gov tenders.
2
Scoring — Every raw item gets a heat score from engagement signals (upvotes, comment counts, recency).
3
Clustering — Related items are grouped into trends by language model, producing 20 ranked themes with names in 5 languages.
4
Pain-point extraction + double verification — Pain points and exact quotes are extracted, then each quote is matched character-for-character against its source post and every link is checked to point to the real thread. Anything that cannot be verified is discarded.
5
Mention filtering — Trends are filtered by cross-source mention count against a MentionScore threshold, then padded to 20.
6
Rendering — A single-file HTML report is generated with a 5-language toggle (English / Chinese / Spanish / French / German). Quotes always stay in the original language.
7
History snapshot — The daily Top-20 is archived so paid subscribers can see which needs repeatedly appear over 7 days.

Data sources

SourceTypeStatus
Hacker NewsDeveloper community, hot posts + top comments✅ Live
GitHubTrending repos + high-engagement issues✅ Live
DEV.toDeveloper blog articles✅ Live
Product HuntProduct launches & maker discussions✅ Live
LobstersTech community hot posts⚠️ Intermittent
RedditFounder/developer subreddit discussions❌ Disabled (credentials pending)
TED (EU)Public procurement tenders across 27 EU states — real paying demand✅ Live (20 notices per run)
SAM.gov (US)US federal contract opportunities — real paying demand⏳ Code ready, API key pending

Tender sources feed a dedicated 💵 Paid demand section and never pollute community trends.

MentionScore — how a need proves it is real

To separate a real, repeated need from a one-off complaint, each trend is scored across independent sources:

MentionScore = Σsources (mentions per source × source weight)
Source weightRationale
3.0 — SAM.gov / TEDReal money: public tenders are already-budgeted demand
1.5 — RedditMass-market signal
1.0 — GitHubBuild-in-progress signal
0.8 — Hacker News / DEV.to / LobstersDeveloper attention

A trend passes the gate when MentionScore ≥ 5 and it appears in ≥ 2 independent sources. A tender hit in any source passes automatically. A trend is flagged 🔥 Surging when its mentions make up ≥ 50% of the last 2 days with ≥ 3 evidence posts.

Anti-fabrication — the two-stage check

You can audit any page yourself: open the Evidence or Pain points section, follow the source ↗ links, and read the quoted line in context.

Update cadence & what you get

FreePaid — $19.99/mo
Daily Top-20 report✅ (1-day window, 09:00 UTC)✅ Same
Paid-demand signals (EU/US tenders)✅✅ Same
Repeat trends — needs entering Top-20 on multiple of the past 7 days—✅ (Google login)
Dedicated need pages with evidence & exact quotes✅✅ Same

Free reports are regenerated every day at 09:00 UTC. No login required. Subscribe on Gumroad for the 7-day repeat-needs layer.