feat(community): Adds an HTML loader for URLS #7184
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
dosubot
bot
added
the
auto:improvement
philnash
force-pushed
the
philnash-html_loader
branch
2 times, most recently
from
ce78d96
to
4572f1d
Compare
jacoblee93
changed the title
jacoblee93
previously approved these changes
Nov 12, 2024
jacoblee93
reviewed
Nov 12, 2024
|
Sorry, noticed one small thing. Can you also run |
jacoblee93
added
close
|
@jacoblee93 I moved the I then imported and re-exported the Hope that's all ok! |
I recently discovered the same issue that was reported in langchain-ai#2467. The documentation and examples suggest chaining the `CheerioWebBaseLoader` with `MozillaReadabilityTransformer` to load and transform HTML documents. However, the `CheerioWebBaseLoader` uses [Cheerio's `text` method to extract the text](https://github.com/langchain-ai/langchainjs/blob/05e5813715150cd69d9e384924818562e3b7c1fa/libs/langchain-community/src/document_loaders/web/cheerio.ts#L144C34-L144C40) from the HTML document, or provided selector. This is great if you want the text, but the MozillaReadabilityTransformer needs to act on HTML. I have added an `HTMLWebBaseLoader` that simply uses `fetch` to get an HTML document and returns the full HTML content. I've also updated the `MozillaReadabilityTransformer` example to use the `HTMLWebBaseLoader` instead of the `CheerioWebBaseLoader`.
I moved the WebBaseLoader and WebBaseLoaderParams to the HTML loader as they were relevant and it seems that loaders export their own types. I also added the `selector` property back to WebBaseLoaderParams and deprecated it from there. I then imported and re-exported the WebBaseLoaderParams from the Cheerio loader with deprecation, so that it doesn't break for anyone that was importing it from here.
philnash
force-pushed
the
philnash-html_loader
branch
from
67f69ab
to
5bfd07d
Compare
|
Force pushed to fix the merge conflict. |
|
@jacoblee93 can we trigger the jobs again to see if this is looking good? Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I recently discovered the same issue that was reported in #2467. The documentation and examples suggest chaining the
CheerioWebBaseLoaderwithMozillaReadabilityTransformerto load and transform HTML documents. However, theCheerioWebBaseLoaderuses Cheerio'stextmethod to extract the text from the HTML document, or provided selector. This is great if you want the text, but the MozillaReadabilityTransformer needs to act on HTML.I have added an
HTMLWebBaseLoaderthat simply usesfetchto get an HTML document and returns the full HTML content.I've also updated the
MozillaReadabilityTransformerexample to use theHTMLWebBaseLoaderinstead of theCheerioWebBaseLoader.Fixes #2467
I'm @philnash if you're still doing shout outs :)