Onelake connector

[Delacrobix]

Hello!

Following the recommendations described in the article
How to create custom connectors for Elasticsearch, I am opening this issue.

I am working on the connector development in this fork.

I have some doubts about:

• I would like to confirm if dependencies must be allocated in the framework.txt
  I couldn't find any requirements.txt

• Regarding the tests, I found information in the article "How to Create Custom Connectors for Elasticsearch" about their location. It mentions they are in the path connectors/source/test (referenced here: https://www.elastic.co/search-labs/blog/how-to-create-customized-connectors-for-elasticsearch#acceptance-criteria-for-contributing-connectors). However, that path does not exist. I assume this has changed, and they are now located in the test/sources folder. I bring this up because in the following document: https://github.com/elastic/connectors/blob/main/docs/CONTRIBUTING.md#acceptance-criteria. It is stated that any modification outside the connectors/sources folder will be rejected.

• I am using the library azure-storage-file-datalake==12.17.0, which has compatibility issues with azure-storage-blob==12.19.1, used in other connectors. What is a better option? : Should I update azure-storage-blob? Or should I downgrade azure-storage-file-datalake?
  For now, I downgraded azure-storage-file-datalake used by Onelake connector to 12.14.0 version.

I would also appreciate some feedback on:

Tests

• Link: https://github.com/Delacrobix/connectors/blob/onelake-connector/tests/sources/test_onelake.py

Connector implementation

• Link: https://github.com/Delacrobix/connectors/blob/onelake-connector/connectors/sources/onelake.py

Please let me know if you find anything else that can be improved.

[seanstory]

I would like to confirm the dependencies must be allocated in the framework.txt file, I don’t find any file named requirements.txt

Yep! The requirements/ directory, as opposed to a single file, allows us to separate requirements by use case and platform. Most of the time, new requirements go into framework.txt.

Regarding the tests [...]

You're correct. The source code lives in the connectors/ dir, and the tests for that code lives in the tests/ dir. We will not auto-reject PRs that have tests :) I'll make a note to revisit the CONTRIBUTING.md guide, to make sure it's more clear.

I am using the library azure-storage-file-datalake==12.17.0, which has compatibility issues with azure-storage-blob==12.19.1, used in other connectors. What should I do in this case?

I'm not sure. If you can update azure-storage-blob to a newer version without breaking things, go for it. If it breaks minor things that you can address in the same PR, go for it. If it's a significantly problematic update, then yes, might be better to start with azure-storage-file-datalake at a lower version.

I would also appreciate some feedback

Best way to get that would be to open a Draft PR.
I would ask that you provide some more information about your use case though. What it is that you have in Onelake, why you want a connector for it, how you intend to search over it, etc. This will help our product team evaluate if this would be a good fit to add to our catalog. CC @danajuratoni