Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fumadocs-mdx -> content-collections #3419

Merged
merged 19 commits into from
Nov 5, 2024
Merged

fumadocs-mdx -> content-collections #3419

merged 19 commits into from
Nov 5, 2024

Conversation

berekuk
Copy link
Collaborator

@berekuk berekuk commented Oct 26, 2024

Notes:

  • content-collections uses mdx-bundler, which is not very import-friendly, so I had to remove all imports in MDX files and handle them through <MDXContent components={{...}}> whitelist
  • everything else works as expected, for now

TODO:

  • figure out how to handle basicPrompt (it's broken again, because of imports)
  • provide a separate collection for API docs, get rid of MDX generation
  • move content to a separate package to reuse it in AI

@berekuk berekuk requested a review from OAGr as a code owner October 26, 2024 18:29
Copy link

changeset-bot bot commented Oct 26, 2024

⚠️ No Changeset found

Latest commit: 8ef5842

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link

vercel bot commented Oct 26, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
quri-hub ✅ Ready (Inspect) Visit Preview Oct 31, 2024 7:42pm
squiggle-website ✅ Ready (Inspect) Visit Preview Oct 31, 2024 7:42pm
2 Skipped Deployments
Name Status Preview Updated (UTC)
quri-ui ⬜️ Ignored (Inspect) Visit Preview Oct 31, 2024 7:42pm
squiggle-components ⬜️ Ignored (Inspect) Visit Preview Oct 31, 2024 7:42pm

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: empty file was ignored, so an empty frontmatter is necessary
(maybe I could put this in comment, not sure)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and all following API files are converted from templates.mts in website. I hope I haven't made any typos while I did that, the conversion process was LLM-assisted.

Danger.jsonString([pointSet, fn]),
]
result
`}/>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is a bit messy: section descriptions contain MDX, but in *.mdx files they're stored as frontmatter string fields. But it's still better than storing it in JS multiline strings. At least the main intro content is the real MDX.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the schema for content/api frontmatter is type-checked, though! See src/collections/api.ts configuration.


This is a style guide for how to write good Squiggle code. It's mostly designed to help LLMs write good Squiggle code, but could be useful for humans as well.

---
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prefix before --- here and in BasicPrompt.mdx will be cut off in LLM system prompt; see content/src/prompts.ts.

This is a bit of an action-at-distance that I dislike because it seems fragile, but at least we check that --- is present in prompts.ts, so there's some safety net.

@@ -15,9 +15,9 @@ This bot often produces poor results, or fails to properly call the server, so i

## Document with all documentation

[All Squiggle Documentation, in One Document](/llms/documentationBundle.markdown)
[All Squiggle Documentation, in One Document](/api/documentation-bundle)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated just-in-time now.

@@ -3,8 +3,6 @@ title: Basic Types
description: Basic types supported by Squiggle
---

import { SquiggleEditor } from "../../../components/SquiggleEditor";
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imports in MDX are not allowed anymore; instead, we pass the full list of components in website configuration.

This is one significant downside of this PR, but it's kind of unavoidable because of how MDX works.

throw new Error(`Invalid response: ${JSON.stringify(json, null, 2)}`);
}
return code;
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy-pasted from AI where similar code is used for fine-tuning.

In the future, we can move both to packages/hub-client library.

const docs = allDocs.filter((doc) => doc._meta.directory === directory);

// Sort docs by the order of the pages in the Guides metadata.
// This matches Fumadocs behavior, which we assume is the most natural ordering both for humans and LLMs.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how important this is, previously the pages weren't sorted, but it wasn't very hard to implement.

import { squiggleSystemPrompt } from "@quri/squiggle-ai/server";

export async function GET() {
return new Response(squiggleSystemPrompt);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://squigglehub.org/ai/api/system-prompt will be useful for checking that system prompt was generated correctly (many ways to make a mistake there, e.g. once I forgot an await, and documentation bundle was injected as [object Promise])

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://squiggle-language.com/api/documentation-bundle (after this PR is merged), linked from docs

extension: "json",
path: "Api/meta",
},
},
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also: the comment in content/src/collections/docs.ts, on how apiDocs is a separate collection

@OAGr
Copy link
Contributor

OAGr commented Nov 5, 2024

App looks fine now. Will review some parts of the code later, when I work in these areas, but I expect it to broadly be okay now.

@OAGr OAGr merged commit 642005e into main Nov 5, 2024
6 checks passed
@OAGr OAGr deleted the content-collections branch November 5, 2024 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants