Feature: distribution of the package is cumbersome

[130s]

Issue aimed to address

• Re-distribution of the package is cumbersome (i.e. distribute by an installer) #13
• Unhelpful error msg: "ERROR:root:These file(s) user inputs are not found: ['.']" #18

Review items

• "Functionalities" to be minimally available for the inaugural release
  • Package name docker-vcs-tools is kind of unintended. Also has dash in it (almost illegal in Python). Maybe rename with docker_vcs or docker_vc_lib.
  • No executable Feature: distribution of the package is cumbersome #17 (comment)
  • Even though the latest is 0.1.2 defined in setup.py, I still get 0.1.0 from testpypi (log in Feature: distribution of the package is cumbersome #17 (comment))
  • Module path not resolved Feature: distribution of the package is cumbersome #17 (comment)
  • No "project description" on testpypi page https://test.pypi.org/project/docker-vcs-tools/0.1.0/

    The author of this package has not provided a project description

  • (Optional) No code checking.
• Test
  • Dev tested by the author.
• Maturity status indicators
  • CI green.
  • Build status indicator?
• Review
  • Approved by maintainers --> Solely owned for now so skipped.

[130s]

Got installed, but not executable :/

# pip3 install -i https://test.pypi.org/simple/ docker-vcs-tools
Looking in indexes: https://test.pypi.org/simple/
Requirement already satisfied: docker-vcs-tools in /usr/local/lib/python3.9/site-packages (0.1.0)
Requirement already satisfied: docker>=5.0.3 in /usr/local/lib/python3.9/site-packages (from docker-vcs-tools) (6.1.3)
Requirement already satisfied: vcstool>=0.3.0 in /usr/local/lib/python3.9/site-packages (from docker-vcs-tools) (0.3.0)
Requirement already satisfied: packaging>=14.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs-tools) (23.1)
Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs-tools) (2.31.0)
Requirement already satisfied: urllib3>=1.26.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs-tools) (2.0.4)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs-tools) (1.6.3)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.9/site-packages (from vcstool>=0.3.0->docker-vcs-tools) (6.0.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from vcstool>=0.3.0->docker-vcs-tools) (68.2.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker>=5.0.3->docker-vcs-tools) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker>=5.0.3->docker-vcs-tools) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker>=5.0.3->docker-vcs-tools) (2023.7.22)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
# find / -iname docker-vcs-tools
# find / -iname docker_vcs_tools
/usr/local/lib/python3.9/site-packages/docker_vcs_tools
# ll /usr/local/lib/python3.9/site-packages/docker_vcs_tools
bash: ll: command not found
# ls -lath /usr/local/lib/python3.9/site-packages/docker_vcs_tools
total 44K
drwxr-xr-x 1 root root 4.0K Sep 14 20:53 ..
drwxr-xr-x 3 root root 4.0K Sep 13 14:53 .
-rw-r--r-- 1 root root    0 Sep 13 14:53 __init__.py
drwxr-xr-x 2 root root 4.0K Sep 13 14:53 __pycache__
-rw-r--r-- 1 root root  27K Sep 13 14:53 docker_builder_vcs.py

[130s]

Even though the latest is 0.1.2 defined in setup.py, I still get 0.1.0 from testpypi.

log

# pip install -i https://test.pypi.org/simple/ docker-vcs
Looking in indexes: https://test.pypi.org/simple/
Collecting docker-vcs
  Obtaining dependency information for docker-vcs from https://test-files.pythonhosted.org/packages/fa/62/eb3b526b651e653ce9059f0e88988e1f2b0cfe56209b52d9ace2d36c24f6/docker_vcs-0.1.0-py3-none-any.whl.metadata
  Using cached https://test-files.pythonhosted.org/packages/fa/62/eb3b526b651e653ce9059f0e88988e1f2b0cfe56209b52d9ace2d36c24f6/docker_vcs-0.1.0-py3-none-any.whl.metadata (727 bytes)
Requirement already satisfied: docker>=5.0.3 in /usr/local/lib/python3.9/site-packages (from docker-vcs) (6.1.3)
Requirement already satisfied: vcstool>=0.3.0 in /usr/local/lib/python3.9/site-packages (from docker-vcs) (0.3.0)
Requirement already satisfied: packaging>=14.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs) (23.1)
Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs) (2.31.0)
Requirement already satisfied: urllib3>=1.26.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs) (2.0.4)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python3.9/site-packages (from docker>=5.0.3->docker-vcs) (1.6.3)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.9/site-packages (from vcstool>=0.3.0->docker-vcs) (6.0.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from vcstool>=0.3.0->docker-vcs) (68.2.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker>=5.0.3->docker-vcs) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker>=5.0.3->docker-vcs) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker>=5.0.3->docker-vcs) (2023.7.22)
Using cached https://test-files.pythonhosted.org/packages/fa/62/eb3b526b651e653ce9059f0e88988e1f2b0cfe56209b52d9ace2d36c24f6/docker_vcs-0.1.0-py3-none-any.whl (16 kB)
Installing collected packages: docker-vcs
Successfully installed docker-vcs-0.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@130s-p16s:~# 

[130s]

Just for now, (manually) setting PYTHONPATH lets Python find our module.

export PYTHONPATH=/usr/local/lib/python3.9/site-packages

PYTHONPATH shouldn't be the solution though.

[130s]

Exec runtime error. Maybe main issue: DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')).

Log

# ipython3    
Python 3.9.2 (default, Feb 28 2021, 17:03:44)     
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.         
          
In [1]: from docker_vcs_lib.docker_vcs import main 
---------------------------------------------------------------------------
:             
In [2]: from docker_vcs_lib.docker_builder_vcs import DockerBuilderVCS
   ...:   
      
In [3]: dbv = DockerBuilderVCS()             
---------------------------------------------------------------------------
FileNotFoundError         Traceback (most recent call last)
/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    789             # Make the request on the HTTPConnection object          
--> 790             response = self._make_request(           
    791 conn,
             
/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    495         try:
--> 496             conn.request(
    497 method,     
      
/usr/local/lib/python3.9/site-packages/urllib3/connection.py in request(self, method, url, body, headers, chunked, preload_content, decode_content, enforce_content_length)
    394             self.putheader(header, value)            
--> 395         self.endheaders()     
    396         
      
/usr/lib/python3.9/http/client.py in endheaders(self, message_body, encode_chunked) 
   1249             raise CannotSendHeader() 
-> 1250         self._send_output(message_body, encode_chunked=encode_chunked)
   1251   
      
/usr/lib/python3.9/http/client.py in _send_output(self, message_body, encode_chunked)    
   1009         del self._buffer[:]   
-> 1010         self.send(msg)     
   1011               
             
/usr/lib/python3.9/http/client.py in send(self, data)        
    949             if self.auto_open:
--> 950 self.connect()       
    951             else: 
             
/usr/local/lib/python3.9/site-packages/docker/transport/unixconn.py in connect(self)     
     26         sock.settimeout(self.timeout)         
---> 27         sock.connect(self.unix_socket)               
     28         self.sock = sock
             
FileNotFoundError: [Errno 2] No such file or directory       
          
During handling of the above exception, another exception occurred:

ProtocolError             Traceback (most recent call last)             
/usr/local/lib/python3.9/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    485         try:
--> 486             resp = conn.urlopen(
    487 method=request.method,

/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    843 
--> 844             retries = retries.increment(
    845 method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]

/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    469             if read is False or method is None or not self._is_method_retryable(method):
--> 470 raise reraise(type(error), error, _stacktrace)
    471             elif read is not None:

/usr/local/lib/python3.9/site-packages/urllib3/util/util.py in reraise(tp, value, tb)
     37         if value.__traceback__ is not tb:
---> 38             raise value.with_traceback(tb)
     39         raise value

/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    789             # Make the request on the HTTPConnection object
--> 790             response = self._make_request(
    791 conn,

/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    495         try:
--> 496             conn.request(
    497 method,

/usr/local/lib/python3.9/site-packages/urllib3/connection.py in request(self, method, url, body, headers, chunked, preload_content, decode_content, enforce_content_length)
    394             self.putheader(header, value)
--> 395         self.endheaders()
    396 

/usr/lib/python3.9/http/client.py in endheaders(self, message_body, encode_chunked)
   1249             raise CannotSendHeader()
-> 1250         self._send_output(message_body, encode_chunked=encode_chunked)
   1251 

/usr/lib/python3.9/http/client.py in _send_output(self, message_body, encode_chunked)
   1009         del self._buffer[:]
-> 1010         self.send(msg)
   1011 

/usr/lib/python3.9/http/client.py in send(self, data)
    949             if self.auto_open:
--> 950 self.connect()
    951             else:

/usr/local/lib/python3.9/site-packages/docker/transport/unixconn.py in connect(self)
     26         sock.settimeout(self.timeout)
---> 27         sock.connect(self.unix_socket)
     28         self.sock = sock

ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

ConnectionError           Traceback (most recent call last)
/usr/local/lib/python3.9/site-packages/docker/api/client.py in _retrieve_server_version(self)
    213         try:
--> 214             return self.version(api_version=False)["ApiVersion"]
    215         except KeyError:

/usr/local/lib/python3.9/site-packages/docker/api/daemon.py in version(self, api_version)
    180         url = self._url("/version", versioned_api=api_version)
--> 181         return self._result(self._get(url), json=True)

/usr/local/lib/python3.9/site-packages/docker/utils/decorators.py in inner(self, *args, **kwargs)
     45 kwargs['headers'].update(self._general_configs['HttpHeaders'])
---> 46         return f(self, *args, **kwargs)
     47     return inner

/usr/local/lib/python3.9/site-packages/docker/api/client.py in _get(self, url, **kwargs)
    236     def _get(self, url, **kwargs):
--> 237         return self.get(url, **self._set_request_timeout(kwargs))
    238 

/usr/local/lib/python3.9/site-packages/requests/sessions.py in get(self, url, **kwargs)
    601         kwargs.setdefault("allow_redirects", True)
--> 602         return self.request("GET", url, **kwargs)
    603 

/usr/local/lib/python3.9/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    588         send_kwargs.update(settings)
--> 589         resp = self.send(prep, **send_kwargs)
    590 

/usr/local/lib/python3.9/site-packages/requests/sessions.py in send(self, request, **kwargs)
    702         # Send the request
--> 703         r = adapter.send(request, **kwargs)
    704 

/usr/local/lib/python3.9/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    500         except (ProtocolError, OSError) as err:
--> 501             raise ConnectionError(err, request=request)
    502 

ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

DockerException           Traceback (most recent call last)
<ipython-input-3-a6e2b2f53efb> in <module>
----> 1 dbv = DockerBuilderVCS()

/usr/local/lib/python3.9/site-packages/docker_vcs_lib/docker_builder_vcs.py in __init__(self)
     42     def __init__(self):
     43         try:
---> 44             self._docker_client = docker.from_env()
     45         except (docker.errors.APIError, docker.errors.TLSParameterError) as e:
     46             logging.error(str(e))

/usr/local/lib/python3.9/site-packages/docker/client.py in from_env(cls, **kwargs)
     94         version = kwargs.pop('version', None)
     95         use_ssh_client = kwargs.pop('use_ssh_client', False)
---> 96         return cls(
     97             timeout=timeout,
     98             max_pool_size=max_pool_size,

/usr/local/lib/python3.9/site-packages/docker/client.py in __init__(self, *args, **kwargs)
     43     """
     44     def __init__(self, *args, **kwargs):
---> 45         self.api = APIClient(*args, **kwargs)
     46 
     47     @classmethod

/usr/local/lib/python3.9/site-packages/docker/api/client.py in __init__(self, base_url, version, timeout, tls, user_agent, num_pools, credstore_env, use_ssh_client, max_pool_size)
    195 str
    196 ) and version.lower() == 'auto'):
--> 197             self._version = self._retrieve_server_version()
    198         else:
    199             self._version = version
/usr/local/lib/python3.9/site-packages/docker/api/client.py in _retrieve_server_version(self)
    219             )
    220         except Exception as e:
--> 221             raise DockerException(
    222 f'Error while fetching server API version: {e}'
    223             )

DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

---

Ignoring this error for now.

That happened inside of ipython3 so maybe missing some initializing process.

---

That happened inside of ipython3 so maybe missing some initializing process.

Unlikely, it's probably due to the executable being executed inside of a Docker container (which is NOT an intended usecase).

Indeed, on a host this prints. Yay!

$ docker_vcs --help
usage: docker_vcs [-h] [--debug] [--docker_base_img DOCKER_BASE_IMG] [--dockerfile DOCKERFILE] [--entrypoint_exec ENTRYPOINT_EXEC]
                  [--log_file] [--network_mode NETWORK_MODE] [--docker_out_img DOCKER_OUT_IMG] [--push_cloud] [--rm_intermediate]
                  [--tmp_context_path TMP_CONTEXT_PATH] [--workspace_in_container WORKSPACE_IN_CONTAINER]
                  [--workspace_on_host WORKSPACE_ON_HOST]
                  [--path_repos_file PATH_REPOS_FILE | --volume VOLUME | --volume_build VOLUME_BUILD]

'docker build' command with 3 additional features. 1) clone arbitrary set of repos utilizing "vcstool", 2) automatically resolve
dependency of the software (using "rosdep install", then 3) build and install (only catkin_tools is supported as of 2021/02). The software
to be cloned and built have to be Catkin/Colcon packages.

options:
  -h, --help            show this help message and exit
  --debug               Disabled by default.
  --docker_base_img DOCKER_BASE_IMG
                        Image Dockerfile begins with.
  --dockerfile DOCKERFILE
                        Dockerfile path to be used to build the Docker image with. This can be remote. Default is './Dockerfile'.
  --entrypoint_exec ENTRYPOINT_EXEC
                        Docker's entrypoint that will be passed to Dockerfile.
  --log_file            If defined, std{out, err} will be saved in a file. If not passed output will be streamed.
  --network_mode NETWORK_MODE
                        Same options are available for networking mode for the 'docker run' command
  --docker_out_img DOCKER_OUT_IMG
                        Entire path of the Docker image ot be built.
  --push_cloud          If defined, not pushing the resulted Docker image to the cloud.
  --rm_intermediate     If False, the intermediate Docker images are not removed.
  --tmp_context_path TMP_CONTEXT_PATH
                        Absolute path for the temporary context path docker_vcs creates (TBD we need a specific name for that). This
                        option can save exec time, and is primarily helpful in docker_vcs' subsequent runs after the 1st run where you
                        don't want to keep generating the temp folder. Double-quote the value from bash console.
  --workspace_in_container WORKSPACE_IN_CONTAINER
                        Workspace where the software obtained from vcs will be copied into. Also by default install space will be under
                        this dir.
  --workspace_on_host WORKSPACE_ON_HOST
                        Current dir, as Docker's context shouldn't change in order for Dockerfile to be accessible.
  --path_repos_file PATH_REPOS_FILE
                        Path to .repos file to clone and build in side the container/image.
  --volume VOLUME       Not implemented yet. Bind volume mount. Unlike '--volume_build' option, this doesn't do anything but mounting.
                        Anything you want to happen can be defined in your 'Dockerfile'. For now this cannot be defined multiple times, as
                        opposed to 'docker run' where multiple '-v's can be passed to.Therefore the value passed to this needs to be the
                        ***top directory*** of all source folders to be built, e.g. 'src'.
  --volume_build VOLUME_BUILD
                        The path to be bound as volume mount. Sources in this path will be targeted to build into Docker container. For
                        now this cannot be defined multiple times, as opposed to 'docker run' where multiple '-v's can be passed
                        to.Therefore the value passed to this needs to be the ***top directory*** of all source folders to be built, e.g.
                        'src'. There may be usecases to use both 'volume_build' and 'path_repos', which is not yet covered as the need is
                        unclear.

[130s]

Wait, docker_vcs is not what I wrote, it might be auto-generated.

Log

# pip3 install -v -i https://test.pypi.org/simple docker_vcs==0.1.11
Using pip 23.2.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
Looking in indexes: https://test.pypi.org/simple
ERROR: Could not find a version that satisfies the requirement docker_vcs==0.1.11 (from versions: 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.10)
ERROR: No matching distribution found for docker_vcs==0.1.11

# pip3 install -v -i https://test.pypi.org/simple docker_vcs
Using pip 23.2.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
Looking in indexes: https://test.pypi.org/simple
Collecting docker_vcs
  Obtaining dependency information for docker_vcs from https://test-files.pythonhosted.org/packages/ed/86/0153d89814a6b7139c80d02b17edbb8a8cf1f07ada0e7afee1198a7bcc74/docker_vcs-0.1.11-py3-none-any.whl.metadata
  Downloading https://test-files.pythonhosted.org/packages/ed/86/0153d89814a6b7139c80d02b17edbb8a8cf1f07ada0e7afee1198a7bcc74/docker_vcs-0.1.11-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: docker in /usr/local/lib/python3.9/site-packages (from docker_vcs) (6.1.3)
Requirement already satisfied: sh in /usr/local/lib/python3.9/site-packages (from docker_vcs) (2.0.6)
Requirement already satisfied: vcstool in /usr/local/lib/python3.9/site-packages (from docker_vcs) (0.3.0)
Requirement already satisfied: packaging>=14.0 in /usr/local/lib/python3.9/site-packages (from docker->docker_vcs) (23.1)
Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.9/site-packages (from docker->docker_vcs) (2.31.0)
Requirement already satisfied: urllib3>=1.26.0 in /usr/local/lib/python3.9/site-packages (from docker->docker_vcs) (2.0.4)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python3.9/site-packages (from docker->docker_vcs) (1.6.3)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.9/site-packages (from vcstool->docker_vcs) (6.0.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from vcstool->docker_vcs) (68.2.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker->docker_vcs) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker->docker_vcs) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests>=2.26.0->docker->docker_vcs) (2023.7.22)
Downloading https://test-files.pythonhosted.org/packages/ed/86/0153d89814a6b7139c80d02b17edbb8a8cf1f07ada0e7afee1198a7bcc74/docker_vcs-0.1.11-py3-none-any.whl (18 kB)
Installing collected packages: docker_vcs
  changing mode of /usr/local/bin/docker_vcs to 755
Successfully installed docker_vcs-0.1.11
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

# docker_vcs --help
Traceback (most recent call last):
  File "/usr/local/bin/docker_vcs", line 5, in <module>
    from docker_vcs_lib.docker_vcs import main
ImportError: cannot import name 'main' from 'docker_vcs_lib.docker_vcs' (/usr/local/lib/python3.9/site-packages/docker_vcs_lib/docker_vcs.py)

# more /usr/local/bin/docker_vcs
#!/usr/local/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from docker_vcs_lib.docker_vcs import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

---

Managed to resolve by https://stackoverflow.com/questions/27784271/how-can-i-use-setuptools-to-generate-a-console-scripts-entry-point-which-calls