feat(python): lance write huggingface dataset directly

[eddyxu]

Be able to directly write a huggingface dataset

[github-actions]

ACTION NEEDED

Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[wjones127]

Please make this import lazy. Right now it will import huggingface anytime you call this function. It should only import if the object is from the huggingface datasets module.

For how we've done this for Polars and other modules, see: https://github.com/lancedb/lance/blob/main/python/python/lance/dependencies.py

Suggested change

	try:
	# Huggingface datasets
	import datasets
	if isinstance(data_obj, datasets.Dataset):
	if schema is None:
	schema = data_obj.features.arrow_schema
	data_obj = data_obj.data.to_batches()
	except ImportError:
	pass
	if _check_for_huggingface(data_obj):
	# Huggingface datasets
	import datasets
	if isinstance(data_obj, datasets.Dataset):
	if schema is None:
	schema = data_obj.features.arrow_schema
	data_obj = data_obj.data.to_batches()

[eddyxu]

Done

[changhiskhan]

is there a generic safe_import utility function lance?

[changhiskhan]

if it is a HF dataset but datasets is not installed, then the user will likely get a very cryptic error message no?

[eddyxu]

changed to use lazy loading

[changhiskhan]

for the datasets that have embeddings, are they usually list or fsl? do we need to check/cast or anything like that?

[eddyxu]

It is too smart at the lance level tho.

[changhiskhan]

Lance <3 HuggingFace

The HuggingFace Hub has become the go to place for ML practitioners to find pre-trained models and useful datasets. HuggingFace datasets can be written directly into Lance format by using the lance.write_dataset method. You can write the entire dataset or a particular split. For example: