Problem using git@ uri: "SSH agent requested but SSH_AUTH_SOCK not-specified"

[leberknecht]

Hi,
im getting the above error when trying to add a pipeline with "git@...". Im running the gaiapipeline/gaia:latest container, i configured /root/.ssh/id_rsa and pub key, so if i exec the git-clone inside of the container, it works fine. Any hints? Google mostly points me to something with windows and putty, which isnt the case for me (running on k8s cluster on gce). The PipelineGitLSRemote seems not to be very verbose, so i guess the error is thrown further down, is there anything i can do to get more insight?

[Skarlso]

@leberknecht Hi. Looks almost like the SSH agent is not running. 🤔 Or the key is not loaded / appended to the call / git is not configured to use the proper keys?

[leberknecht]

Indeed, i was able to workaround this by setting up ssh env in the entrypoint:

cp /secrets/gitlab/gaiaGitlabKey ~/.ssh/id_rsa
touch ~/.ssh/known_hosts
chmod 600 ~/.ssh/*

eval $(ssh-agent -s)
ssh-add <(cat ~/.ssh/id_rsa)
ssh [email protected] -T
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config

export SSH_KNOWN_HOSTS=~/.ssh/known_hosts
/app/gaia-linux-amd64 -jwtPrivateKeyPath /secrets/gaia/jwtPrivateKey.pem

The container is pretty straight forward:

FROM gaiapipeline/gaia:latest

RUN pip install google-cloud-pubsub psq

RUN mkdir /root/.ssh
RUN echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSsvxw4RzrDP2pukaRXtggoTOKAHpOPeengukcRcMwL9AE6GCcy9PzO729SiTkont+1vuudZENGt4pKx2HJf+JZaX44ii694y3XhEZSf3Ert8jTCQim5FE33epK5jfu/CIrWOB2V7dvHxqzn+AQXPpmUpLyJO219atoZCX7FyVteUSqmat+X+Seh9Xac+mJTasBWVpLCr7gzbVpOdq950UEhKbU+1D4y2dOEMiza2NqVPSqbVQW2NX88nvkXNBXtv2TcQkZXQI5/XnratdOT7kKVLqcDkHzwny1Jg/bA8Ik6Oya4onkPgRR5DuTUYpyK4tFiBgXzN6yRFp0mSgQJ7D gaia@staging' > /root/.ssh/id_ras.pub

COPY pubsub-listener.py /app/
COPY entrypoint.sh /app/

RUN chmod +x /app/entrypoint.sh

ENTRYPOINT ["/bin/sh", "-c", "/app/entrypoint.sh"]

Is gaiapipeline/gaia:latest supposed to invoke the ssh-agent?

[Skarlso]

No, not really. The container isn't running an ssh-agent.

What you could do is forward your local socket to your container like this:

docker run -d -p 8080:8080 -v $PWD:/data gaiapipeline/gaia:latest /
-v $(dirname $SSH_AUTH_SOCK):$(dirname $SSH_AUTH_SOCK) /
-e SSH_AUTH_SOCK=$SSH_AUTH_SOCK gaia

You could try this..

[leberknecht]

Hm, wont solve the problem on gce :D But im ok with the workaround, will close this one. But maybe this should be mentioned on the docs :)

[Skarlso]

Generally ssh-agents aren't running in containers. This isn't really Gaia specific but rather a Docker specific thing. Unless you know otherwise and I'm missing something.... But that's how it used to be as far as I remember, and I'll be honest, I didn't do research on this ATM. :)

[michelvocks]

@leberknecht Could you elaborate a bit on your use case? Would be awesome to hear a bit about the background. 😄

[leberknecht]

Sure :)
Actually the use case not very special i think: if i fire

docker run -d -p 8080:8080 -v $PWD:/data gaiapipeline/gaia:latest

on my host, the container is launched successfully, i can log in on http://localhost:8080
Then i want to create a pipeline and enter a repo url like [email protected]:gaia-pipeline/gaia.git
And then the error shows up :)
(Just double checked, this happens on the k8s pod as well as on my local host)

Does this work for you (that would be confusing somehow)?

On a broader scope: We are looking into using Gaia for our ETL pipeline (yes, im aware that it is alpha state, just looking :) ), the pipeline will be triggered via an API call, the pipeline repo serves the python-example from https://github.com/gaia-pipeline/python-example

[Skarlso]

@leberknecht FYI: #150
This is happening. So there should be a much easier way after this is implement to trigger a pipeline.

[michelvocks]

@leberknecht I was able to reproduce your problem and that is definitively not intended and therefore a bug. Thanks for reporting! I will reopen your issue as long as this is not fixed.

[Skarlso]

Ops. My bad for missing it then. I apologise.

[michelvocks]

@leberknecht #172 should fix your problem. It's important to know that you included your SSH username in your git url. I've added a small validation step which will indicate that.