California Civic Data for IRS data

[hampelm]

At http://www.californiacivicdata.org/, they produce:

The complete package of 80 database tables released by California's Secretary of State, which we have cleaned up and converted into flat files with 40.7 million rows of comma-separated values*.

Along with related documentation.

• Detailed per-field documentation of the IRS exempt org extracts: https://www.irs.gov/uac/soi-tax-stats-charitable-and-exempt-organizations-statistics
• Scrape and host historical copies of the business master file: https://www.irs.gov/uac/soi-tax-stats-charitable-and-exempt-organizations-statistics
• Deduplicate and provide running historical data for the BMF

Work with the new IRS XML files at https://aws.amazon.com/public-data-sets/irs-990/

• Document differences between years and forms
• Produce standardized scrapers
• Make a web interface to make the index CSVs easily searchable so you can more easily pull collections of files by organization, year, state, etc.

[hampelm]

https://www.reddit.com/r/aws/comments/4p772f/how_the_heck_do_i_view_the_990_documents_on/

People wanted:

• Searchable index
• Flat files / easier way to pull the data ("I am working on a project right now with some people that iterates through the index, downloads the xml files and appends to a large flat file. Let me know if you are interested in the data once we are done. We've sampled around 75k docs and the beginning results look interesting and def useful.")
• EBS volume of all the files

[hampelm]

Turning this into a real plan here: https://docs.google.com/document/d/1Kf1en80g0w_ElDBlaiYpX99jUYjkG_u_BrIhN9Qm8Vw/edit