This repository has been archived by the owner on Jun 16, 2020. It is now read-only.
SQLite-ify #67
Open
SQLite-ify #67
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
So rather than try to merge many disparate flat file sources, lets just pretend that the 1970s actually happened and use a SQL engine. This commit adds a script to generate a SQLite database from the job postings. It leverages quite a few more dependencies on CPAN but all in all I think it's worth it in the long run for making this project a bit more developer friendly. Additionally we commit the first version of `Perl_Companies.db` the actual companies database. Next steps are to generate the various file formats FROM `Perl_Companies.db` and to look at ways to scrub the data in `Perl_Companies.db`.
This took a little re-jiggering (hence the big-ass commit). Basically I didn't want to lose any information that was in the existing YAML file so I wrote a script to import it into the SQLite DB. I then re-wrote the scripts to generate the YAML file as well as the markdown and CSV files from the SQLite DB. This should complete the *code* portion of this re-write.
|
Apologies for the delay in looking at this. I've reviewed the code and it seems logical but there are no README changes so I don't know how you intend them to be run/used. These, I assume, are a part of the aforementioned "consequences"? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a fairly large PR but it does kill #18 entirely.
** Please don't merge this without talking with me about the consequences. **
This PR replaces the source of truth from the YAML file (and/or the other flat files) with an SQLite database. This should make maintaining the data in the long-term easier. It makes everything else in the short-term harder.
There have been approx. 2,000 jobs posted to jobs.perl.org since this project was last updated. I've re-imported everything into the SQLite database and attempted to perform some basic scrubbing/data integrity as I've gone along. However I know there are still some issues outstanding which we'll have to approach.
The biggest public facing issue is there is currently no simple front end for adding a new company to the database. It won't take long to write one but I didn't want the perfect to yadda yadda yadda.
The docs need work, but I need direction on where you want to go with them and how you see this working as a project.