Handle "native" dependencies

[damon-kwok]

This was edited by @SeanTAllen:

This started as an idea to use vcpkg from @damon-kwok.

https://github.com/Microsoft/vcpkg
Microsoft's vcpkg provides cross-platform c/c++ library installation. It works well with pony ffi.

It has since expanded into a conversation for handling native dependencies.

[damon-kwok]

I think corral's support for vcpkg only needs to include its lib file directory in the search path, so that users do not need to fill in the full path.

[SeanTAllen]

Corral doesn't have anything to do with c/c++ library installation. Rather than have an issue for this, I think it is important to first decide if and how we want to approach non-pony dependencies with corral.

@kulibali have you given any thought to non-pony dependencies and corral?

[chalcolith]

I haven't played with vcpkg any. I'm a bit hesitant to introduce an external dependency to corral. My tentative plan, which I swear I'm going to get around to any time now, is to introduce the ability to download binaries and put them in the PONY_PATH. The use case is merely to get the crypto and ssl libraries working on windows so that I can get ponyup working.

[SeanTAllen]

@kulibali any thoughts on how to do in a distro/OS dependent way?

[chalcolith]

The idea would be to have a section in corral.json like "libs": { "windows": { "bin": "foo.dll", "url": "https:foo.zip" }, "linux": {...} } and then corral would uncompress the archive, find the required bin file, and add the directory it's in to PONY_PATH.

[damon-kwok]

There are many compilers and distro/OS, and the burden of precompilation can be heavy. Provide the source address, using CORRAL to call VC GCC clang in local compilation, it can minimize the burden, maximum compatibility. There are many libraries that offer a number of options that users may want to tap into the compilation process.

[SeanTAllen]

@kulibali I think "linux" isn't expansive enough given the desire folks might have to use a package manager etc.

[SeanTAllen]

There are many compilers and distro/OS, and the burden of precompilation can be heavy. Provide the source address, using CORRAL to call VC GCC clang in local compilation, it can minimize the burden, maximum compatibility. There are many libraries that offer a number of options that users may want to tap into the compilation process.

It's unclear to me what you are suggesting.

[SeanTAllen]

Can anyone comment on what Rust's cargo does for this?

[damon-kwok]

Simply put, I don't think Corral needs to provide precompiled files for users to download. We can learn from the treatment of VCPKG. It simply provides the source code address and the script for the compilation process. Even so, this is a lot of work, so at this stage we can choose to use VCPKG directly. When I ran "VCPKG install openssl", I found the experience great and it worked well on non-Windows systems

[SeanTAllen]

What does " Corral needs to provide precompiled files for users to download" mean to you? I don't think anyone has suggested that Corral provides the libraries. Rather it can allow for them to be installed. Is that what you mean?

[damon-kwok]

I saw that Cargo didn't do a good job of that. It just used the library that came with the system. But I have noticed that the Rust ecosystem thrives, for example if you want to use MongoDB, the MongoDB team provides the Rust driver, so FFI is always an alternative to the Rust. So FFI is more important for Pony.

[SeanTAllen]

I saw that Cargo didn't do a good job of that.

Didn't do a good job of what @damon-kwok ?

[damon-kwok]

Corral needs to provide precompiled files for users to download"

I mean there's a lot of work involved in providing pre-compiled binaries.

[damon-kwok]

I saw that Cargo didn't do a good job of that.

Cargo depends on the system library version.

[SeanTAllen]

I saw that Cargo didn't do a good job of that.

Cargo has provided no additional support for this

What is "this"?

[damon-kwok]

Caogo hasn't done anything, but can we do better

[SeanTAllen]

Corral needs to provide precompiled files for users to download"

I mean there's a lot of work involved in providing pre-compiled binaries.

You keep saying "provided". I don't know what "provided" means to you in this context. To me it suggests that you think that we the pony team would compile binaries of libraries and store them somewhere for people to download using corral.

No one has suggested that we provide any precompiled binaries of the sort I mention above. Rather there is a question of "should we make it possible for people to use corral to install precompiled binaries".

[SeanTAllen]

We discussed this during the sync call today (July 7, 2020 if you want to refer to it once uploaded).

General thoughts:

After Theo mentioned how cargo handles (he will be detailing later), we think "that's eh" but we don't have better ideas.

We think that allowing the usage of tools like brew, apt etc to install packages would be very difficult. Many package managers don't allow you to select a version and for a specific release of code that uses a foreign library, that would be problematic. Supporting installation via package manager seems very unlikely.

Allowing for corral in some fashion to download and build libraries from source and then install them in a pony specific fashion however, seems doable and something we want to look into more as a first step.

What that means as a first step needs to be determined further.

[damon-kwok]

Yes, I may have misunderstood the sentence:

The idea would be to have a section in corral.json like "libs": { "windows": { "bin": "foo.dll", "url": "https:foo.zip" }, "linux": {...} } and then corral would uncompress the archive, find the required bin file, and add the directory it's in to PONY_PATH.

[damon-kwok]

Allowing for corral in some fashion to download and build libraries from source and then install them in a pony specific fashion however, seems doable and something we want to look into more as a first step.

This is the right approach. the first thing we need to do is to determine the format of the description file, which contains the platform-wide compilation process script.

[chalcolith]

It occurs to me that since we already have the pattern of including a Makefile and make.ps1 in library project to do things like testing, we could have Corral call that (perhaps with a special target thirdparty or something) if present, and then the Makefile would get and compile the external libs and put them in the directory that Corral is going to add to PONY_PATH.

[SeanTAllen]

I want to get away from using Makefiles. I'd like to roll all that functionality into corral.