Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fail to remove dataset feature from huggingface (unable to align with DatasetDict.map parameters remove_columns in transformers) #307

Open
EdwinWang37 opened this issue Oct 9, 2024 · 1 comment

Comments

@EdwinWang37
Copy link

Environment

mindspore2.3.1

Problems

def map(self, operations, input_columns=None, output_columns=None, column_order=None,

unable to align with DatasetDict.map parameters remove_columns in transformers

Experimental data screenshot

as follow, we can see the upper data with no "text", cause it is easily processed with DatasetDict.map(remove_columns = ''text''), but in the bottom data, i fail to process it in mindspore with similar easy method. And maybe it will increase programmers' development cost to
tackle with it.
image

Requirement

If there is similar function to do that, please tell me, i will appreciate it.
And if no, is it possible to add this parameters to easily finish it?

@wyq-Git
Copy link

wyq-Git commented Oct 23, 2024

please try mindnlp. https://github.com/mindspore-lab/mindnlp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants