Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Base models and notebooks for general IPU embeddings model #436

Merged
merged 31 commits into from
Jul 14, 2023

Conversation

arsalanu
Copy link
Contributor

@arsalanu arsalanu commented Jun 27, 2023

Support for multiple embeddings models (e5, mpnet, t5 (pending))
Idea is to create a general embeddings class which supports the pool and norm stage of embeddings model. These models use encoder stages from various base models (bert, t5, mpnet) which we don't support the direct use of with pipelining. I've added these in this folder for now, but they may need to be in the source lib if integrated.

Seemed like not a great idea to create new pipelined classes for all of these models, as they inherit from existing models we support and would just be an extra couple of lines which are common amongst them, so makes more sense to have a generic wrapper for embeddings, which instantiates the encoder stages through AutoModel as currently shown in HF.

The inference setup is manually outlined, but may be more user-friendly to create a pipeline for these eventually.

MPNet modeling file only includes the base model and maskedLM model, not tested for pretraining, only for embeddings.

notebook will likely be restructured

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@arsalanu
Copy link
Contributor Author

relies on #437 being merged first

@arsalanu arsalanu marked this pull request as ready for review July 4, 2023 16:11
@katalinic-gc katalinic-gc merged commit 4ecd177 into huggingface:main Jul 14, 2023
3 checks passed
@katalinic-gc katalinic-gc deleted the text_embeddings_models branch July 14, 2023 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants