feat: add Astro DB #304

ViorelMocanu · 2024-08-12T00:38:52Z

📚 Description

This PR adds Astro DB as a data source, both for local development (via CSV seeder files) and remote Studio connection.

This is still a draft until I manage to make the resource page functional, plus all the taxonomy pages. The rest will have to be optimized in further PRs.

Known issues include:

Astro Studio (remote) is populating with the DB structure but not with the contents of the CSV files in seed.ts - maybe I should use fetching techniques instead of the file system
GitHub Actions for DB are failing
several TSC warnings left to fix
etc

I should fix them before merging this.

🔗 Linked issue(s)

Fixes #12 (Figure out initial database solution).
Fixes #73 (Refactor getStaticPaths).
Fixes #76 (Add all document links).
Fixes #100 (YouTube Embeds).
Fixes #137 (Refactor ResourceList and ResourceTOC).
Fixes #159 (Add Supabase).
Fixes #160 (Test Supabase as initial storage solution).
Fixes #170 (Redirect table in Supabase).
Fixes #284 (Test AstroDB as initial storage solution).

❓ Type of change

📄 Changelog

cleanup and editing of operational files
update all packages and add PNPM action to ease updating via pnpm up
add some additional packages and activate experimental features like ViewTransition and Astro Server Islands as a POC
scrape document to get links, scrape all links to get meta titles and descriptions where possible, generate taxonomy and resource CSVs with ChatGPT, clean them up, upload to Google Sheets as visual database, populate with missing info and download data as final CSV for seeding into database
insert Astro DB into configurations and use it to store data
steal good code from the Supabase branch and factor it into the current one

✅ Checklist

My code follows the style guidelines of this project
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings or errors
I have manually tested the newly added functionality with no discernable errors
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules

vercel · 2024-08-12T00:38:55Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
digital-resources	⬜️ Ignored (Inspect)	Visit Preview		Oct 7, 2024 10:48am

db/seed.ts

+				u.startsWith('https://vimeo.com/')
+			)
+				resourceType = 4;
+			else if (u.indexOf('udemy.com/') > -1 || u.indexOf('course') > -1) resourceType = 3;


db/seed.ts

+			)
+				resourceType = 4;
+			else if (u.indexOf('udemy.com/') > -1 || u.indexOf('course') > -1) resourceType = 3;
+			else if (u.indexOf('amazon.com/') > -1 || u.indexOf('pdf') > -1 || u.indexOf('book') > -1) resourceType = 2;


To fix the problem, we need to parse the URL and check its host against a whitelist of allowed hosts. This ensures that the host is exactly what we expect and not just a substring within a potentially malicious URL.

Use the url module to parse the URL and extract the host.

Define a whitelist of allowed hosts.

Check if the parsed host is in the whitelist before categorizing the resource.

db/seed.ts

+				resourceType = 4;
+			else if (u.indexOf('udemy.com/') > -1 || u.indexOf('course') > -1) resourceType = 3;
+			else if (u.indexOf('amazon.com/') > -1 || u.indexOf('pdf') > -1 || u.indexOf('book') > -1) resourceType = 2;
+			else if (u.indexOf('github.com/') > -1 || u.indexOf('gitlab.com/') > -1) resourceType = 5;


db/seed.ts

+				resourceType = 4;
+			else if (u.indexOf('udemy.com/') > -1 || u.indexOf('course') > -1) resourceType = 3;
+			else if (u.indexOf('amazon.com/') > -1 || u.indexOf('pdf') > -1 || u.indexOf('book') > -1) resourceType = 2;
+			else if (u.indexOf('github.com/') > -1 || u.indexOf('gitlab.com/') > -1) resourceType = 5;


db/seed.ts

+			else if (u.indexOf('udemy.com/') > -1 || u.indexOf('course') > -1) resourceType = 3;
+			else if (u.indexOf('amazon.com/') > -1 || u.indexOf('pdf') > -1 || u.indexOf('book') > -1) resourceType = 2;
+			else if (u.indexOf('github.com/') > -1 || u.indexOf('gitlab.com/') > -1) resourceType = 5;
+			else if (u.indexOf('medium.com/') > -1 || u.indexOf('dev.to/') > -1 || u.indexOf('blog') > -1 || t.indexOf('blog') > -1 || u.indexOf('.pdf') > -1) resourceType = 6;


To fix the problem, we need to parse the URL and check the host component explicitly against a whitelist of allowed hosts. This approach ensures that the check is accurate and not prone to substring matching issues.

Parse the URL using a URL parsing library to extract the host component.

Define a whitelist of allowed hosts.

Check if the parsed host is in the whitelist.

Update the resource type assignment logic to use this secure check.

github-actions · 2024-10-07T10:49:43Z

Your database schema is up-to-date.

ViorelMocanu added 5 commits August 4, 2024 03:12

feat: initial astrodb introduction alongside data

8b0cbec

chore: db connection improvement

c51e1f2

chore: update resources and fix slugification errors

6458365

chore: update db and steal cool features from other branch

15b5625

chore: linting and final steals

e566ed2

ViorelMocanu added this to the v0.5 milestone Aug 12, 2024

ViorelMocanu self-assigned this Aug 12, 2024

github-advanced-security bot found potential problems Aug 12, 2024

View reviewed changes

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

ViorelMocanu added 3 commits August 12, 2024 03:40

Merge branch 'main' into feat/add-astrodb

7023121

chore: several steps closer to launch

57fb235

chore: update packages

24b07ae

github-advanced-security bot found potential problems Aug 13, 2024

View reviewed changes

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

db/seed.ts Fixed Show fixed Hide fixed

chore: updates

680f72b

github-advanced-security bot found potential problems Oct 7, 2024

View reviewed changes

@@ -1,2 +1,3 @@
             import * as fs from 'fs';
+            import { URL } from 'url';
             import { Author, Like, LikeTest, NOW, Rating, RelationResourceTag, Resource, ResourceType, Tag, TagType, Taxonomy, TaxonomyType, User, Visits, db } from 'astro:db';
@@ -305,15 +306,25 @@
             			const u = r.url.toLowerCase();
-            			if (
-            				u.startsWith('https://youtube.com/') ||
-            				u.startsWith('http://youtube.com/') ||
-            				u.startsWith('https://www.youtube.com/') ||
-            				u.startsWith('http://www.youtube.com/') ||
-            				u.startsWith('https://youtu.be/') ||
-            				u.startsWith('https://vimeo.com/')
-            			)
-            				resourceType = 4;
-            			else if (u.indexOf('udemy.com/') > -1 || u.indexOf('course') > -1) resourceType = 3;
-            			else if (u.indexOf('amazon.com/') > -1 || u.indexOf('pdf') > -1 || u.indexOf('book') > -1) resourceType = 2;
-            			else if (u.indexOf('github.com/') > -1 || u.indexOf('gitlab.com/') > -1) resourceType = 5;
-            			else if (u.indexOf('medium.com/') > -1 || u.indexOf('dev.to/') > -1 || u.indexOf('blog') > -1 || t.indexOf('blog') > -1 || u.indexOf('.pdf') > -1) resourceType = 6;
+            			const parsedUrl = new URL(u);
+            			const host = parsedUrl.host;
+            			const allowedHosts = {
+            				'youtube.com': 4,
+            				'www.youtube.com': 4,
+            				'youtu.be': 4,
+            				'vimeo.com': 4,
+            				'udemy.com': 3,
+            				'amazon.com': 2,
+            				'github.com': 5,
+            				'gitlab.com': 5,
+            				'medium.com': 6,
+            				'dev.to': 6
+            			};
+            			if (allowedHosts[host]) {
+            				resourceType = allowedHosts[host];
+            			} else if (u.indexOf('course') > -1) {
+            				resourceType = 3;
+            			} else if (u.indexOf('pdf') > -1 || u.indexOf('book') > -1) {
+            				resourceType = 2;
+            			} else if (u.indexOf('blog') > -1 || t.indexOf('blog') > -1 || u.indexOf('.pdf') > -1) {
+            				resourceType = 6;
+            			}
             			r.resource_type_id = !!row.resource_type_id ? row.resource_type_id : resourceType;

@@ -304,16 +304,16 @@
             			const t = r.title.toLowerCase();
-            			const u = r.url.toLowerCase();
-            			if (
-            				u.startsWith('https://youtube.com/') ||
-            				u.startsWith('http://youtube.com/') ||
-            				u.startsWith('https://www.youtube.com/') ||
-            				u.startsWith('http://www.youtube.com/') ||
-            				u.startsWith('https://youtu.be/') ||
-            				u.startsWith('https://vimeo.com/')
-            			)
-            				resourceType = 4;
-            			else if (u.indexOf('udemy.com/') > -1 || u.indexOf('course') > -1) resourceType = 3;
-            			else if (u.indexOf('amazon.com/') > -1 || u.indexOf('pdf') > -1 || u.indexOf('book') > -1) resourceType = 2;
-            			else if (u.indexOf('github.com/') > -1 || u.indexOf('gitlab.com/') > -1) resourceType = 5;
-            			else if (u.indexOf('medium.com/') > -1 || u.indexOf('dev.to/') > -1 || u.indexOf('blog') > -1 || t.indexOf('blog') > -1 || u.indexOf('.pdf') > -1) resourceType = 6;
+            			const u = new URL(r.url.toLowerCase());
+            			const host = u.host;
+            			const allowedHosts = {
+: ['youtube.com', 'www.youtube.com', 'youtu.be', 'vimeo.com'],
+: ['udemy.com'],
+: ['amazon.com'],
+: ['github.com', 'gitlab.com'],
+: ['medium.com', 'dev.to']
+            			};
+            			if (allowedHosts[4].includes(host)) resourceType = 4;
+            			else if (allowedHosts[3].includes(host) || u.pathname.includes('course')) resourceType = 3;
+            			else if (allowedHosts[2].includes(host) || u.pathname.includes('pdf') || u.pathname.includes('book')) resourceType = 2;
+            			else if (allowedHosts[5].includes(host)) resourceType = 5;
+            			else if (allowedHosts[6].includes(host) || u.pathname.includes('blog') || t.includes('blog') || u.pathname.endsWith('.pdf')) resourceType = 6;
             			r.resource_type_id = !!row.resource_type_id ? row.resource_type_id : resourceType;

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Astro DB #304

feat: add Astro DB #304

ViorelMocanu commented Aug 12, 2024 •

edited

Loading

vercel bot commented Aug 12, 2024 •

edited

Loading

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

github-actions bot commented Oct 7, 2024

feat: add Astro DB #304

Are you sure you want to change the base?

feat: add Astro DB #304

Conversation

ViorelMocanu commented Aug 12, 2024 • edited Loading

📚 Description

🔗 Linked issue(s)

❓ Type of change

📄 Changelog

✅ Checklist

vercel bot commented Aug 12, 2024 • edited Loading

github-actions bot commented Oct 7, 2024

ViorelMocanu commented Aug 12, 2024 •

edited

Loading

vercel bot commented Aug 12, 2024 •

edited

Loading