Civil: The Data Engine Behind a CRM for High-Impact Outreach
The campaign finance intelligence engine powering Civil (civilcrm.ai)—a unified API over tens of millions of FEC, IRS, and state records with donor network traversal and sub-100ms cached queries.
The Problem
Political contribution data is public but practically inaccessible. It fragments across federal FEC filings, IRS Form 8872 disclosures, and state-level databases—each with its own schema, portal, and quirks—so researching a single donor means hours of manual searches across systems that do not talk to each other. Understanding giving patterns or network connections across those silos is harder still.
Scale compounds the fragmentation: federal individual contributions alone run to tens of millions of records, and naive queries over them are far too slow for interactive research. Bulk state data access is its own project—Massachusetts OCPF, for instance, publishes records through an API that practical use requires paginating through in ten-thousand-record pages with careful rate limiting.
What We Built
We built a unified Go REST API consolidating federal individual contributions, committee-to-committee transfers, PAC summaries, IRS Form 8872 filings, and state-level expenditures behind one normalized interface. Dynamic SQL query building supports 15+ filter parameters—dates, amounts, contributor types, locations, employers, occupations—and a Redis caching layer with parameter-hashed keys serves repeated queries in under 100ms even over tens of millions of records. Entity profiles offer dual-mode lookup by ID or name with complete giving and receiving summaries, and related-transaction traversal maps money flows across the network.
Feeding the platform, a Python bulk collector retrieves complete campaign finance datasets from state APIs with adaptive pagination, randomized rate limiting, retry logic, and incremental JSONL persistence—unattended runs that lose nothing on failure and extend cleanly to new states. The whole system deploys serverlessly on AWS App Runner through Docker and GitHub Actions CI/CD, with dual Clerk JWT and API-key authentication.
One API over fragmented sources
FEC, IRS 8872, and state contribution data—previously three incompatible databases—query through a single normalized interface with 15+ filter dimensions.
Entity and network intelligence
Donor and committee profiles resolve by ID or name with full giving history, and related-transaction traversal surfaces network patterns like coordinated bundling that record-by-record review misses.
Sub-100ms cached analytics
Redis caching with consistent parameter-hashed keys keeps interactive research fast over tens of millions of records.
Unattended bulk state collection
The state scraper paginates entire categories—receipts, expenditures, subvendor payments, segregated funds—with rate limiting and crash-safe incremental persistence, structured to extend to all 50 states.
How It Works in Practice
A journalist investigating whether a major donor coordinated contributions across federal and state committees.
- 1
Resolve the entity
A name lookup returns the donor profile with complete giving history aggregated across FEC, IRS, and state records—research that formerly meant hours across three portals.
- 2
Filter the record set
Queries by date range, amount, employer, and location narrow tens of millions of records in one call, with cached responses returning in under 100ms.
- 3
Traverse the network
Related-transaction discovery follows the money to connected committees and co-contributors, surfacing a bundling pattern invisible in any single filing.
- 4
Verify the state layer
Freshly collected state data—pulled by the unattended bulk scraper—confirms the same pattern at the state level, completing the cross-jurisdiction picture.
Screenshots
Engineering Approach
Normalize the sources, keep the provenance
Heterogeneous filings unify into one queryable schema while retaining source attribution, so speed of research never costs verifiability.
Cache for the research workflow
Investigative research revisits the same entities repeatedly—parameter-hashed Redis caching turns that access pattern into sub-100ms responses instead of repeated heavy scans.
Collection that respects its sources
Randomized rate limiting, retry logic, and incremental persistence keep bulk public-data collection reliable and sustainable for the long term.
Have a similar problem?
Tell us what you're working on and we'll tell you honestly whether we can help.
