Updating go.sum artifact on Bitbucket Cloud

[caphrim007]

Which Renovate are you using?

Renovate Open Source CLI

Which platform are you using?

Bitbucket Cloud

Have you checked the logs? Don't forget to include them if relevant

I'm not sure if the logs are relevant or not. If they are, please advise.

What would you like to do?

I have a BitBucket cloud repo with a go.mod file in it which resembles the following,

module bitbucket.org/myorg/service

go 1.13

require (
	bitbucket.org/myorg/services-sdk v0.0.0-20200304002603-d4886093cfe2
	github.com/aws/aws-sdk-go v1.30.23
	github.com/go-chi/chi v4.0.3+incompatible
	github.com/go-chi/cors v1.0.0
)

The relevant part of this question concerns the bitbucket.org require line. This is a private Bitbucket-cloud repository.

Renovate successfully finds and creates an issue with an update to the github.com requirements. I have a excludePackagePatterns package rule defined for the bitbucket.org due to missing support for such things in Renovate.

After Renovate creates the PR, it attempts to update the go.sum file. It fails though and adds a comment to the PR with the following in it

Command failed: go get -d ./...
go: bitbucket.org/myorg/[email protected]: reading https://api.bitbucket.org/2.0/repositories/myorg/services-sdk?fields=scm: 403 Forbidden
    server response: Access denied. You must have write or admin access.

For the life of me, I can't see to figure out what I need to configure to give it the necessary access. The user that renovate runs as has an app token and is a member of a group in Bitbucket which has write access to all repos.

I was hoping there is an obvious config or environment variable I can configure on the renovate side (not the repo side) to give it the necessary info (like a host rule or something).

I see mention of Github specific config concerning preCommands here

https://github.com/renovatebot/renovate/blob/master/lib/manager/gomod/artifacts.ts#L13

though its unclear to me if applying the same for Bitbucket would make the above go command work as this appears to be specifically raised by go.

Any help is appreciated.

[rarkins]

You must have write or admin access. is strange. Perhaps it's an incorrect log message and only read is necessary.

We will need an enhancement to our go artifacts code here:

renovate/lib/manager/gomod/artifacts.ts

Lines 13 to 29 in 6ec460b

	function getPreCommands(): string[] | null {
	const credentials = find({
	hostType: PLATFORM_TYPE_GITHUB,
	url: 'https://api.github.com/',
	});
	let preCommands = null;
	if (credentials && credentials.token) {
	let token = global.appMode
	? `x-access-token:${credentials.token}`
	: credentials.token;
	token = quote(token);
	preCommands = [
	`git config --global url.\"https://${token}@github.com/\".insteadOf \"https://github.com/\"`, // eslint-disable-line no-useless-escape
	];
	}
	return preCommands;
	}

[caphrim007]

@rarkins in much the same spirit as was done with python hash support, I'm happy to test anything that may be proposed. I will try duplicating the mentioned code and adjusting for Bitbucket to see if that results in any progress. Will provide feedback here for you.

One question, are host-rules capable of being specified in the Renovate configuration that runs as a cronjob? or must it be specified in the renovate.json within the repository under management.

[rarkins]

hostRules can be defined in the bot config too

[taraspos]

Hey, I'm having a similar issue.

I'm using renovate cli from docker image on bitbucket pipelines like:

    renovate-bot:
      - step:
          image: renovate/renovate:23.89 
          name: Renovate Bot
          clone:
            enabled: false
          script:
            - export GOPRIVATE=bitbucket.org/my_org/*
            - export BB_USER=renovateuser
            - git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org"
            - renovate  --username=${BB_USER} --log-level debug --platform=bitbucket $BITBUCKET_REPO_FULL_NAME

however, getting the following error:

DEBUG: Failed to look up dependency bitbucket.org/my_org/go-lib

Question:

1. git config --global I provided before, seems to be ignored. Is it because renovatebot overwrites it during run?
2. Can I solve the problem by providing custom hostRules or it will not work until #3496 is implemented? If adding hostRules can help, what is the correct format for the CLI parameter? Is it --host-rules hostName:bitbucket.org,username:${BB_USER},password:${RENOVATE_PASSWORD}?

[viceice]

the parameter value need to be an stringified json array. so better add a config.js and use that to configure hostrules with passwords from process.env.XXX_PASSWORD.

Renovate doesn't use your gitconfig for lookup updates, it needs a hostRule for auth.

[taraspos]

@viceice thanks for your reply!

Can you share the example of how to use process.env.XXX_PASSWORD in config? I was trying to find something in the docs, but no luck for now.

Would it be something like:

    "hostRules": [
        {
            "domainName": "bitbucket.org",
            "username": process.env.BB_USER,
            "password": process.env.RENOVATE_PASSWORD
        }
    ]

and that is it?

[viceice]

yes:

module.exports = {
  "hostRules": [
        {
            "domainName": "bitbucket.org",
            "username": process.env.BB_USER,
            "password": process.env.RENOVATE_PASSWORD
        }
    ]
}

[taraspos]

I had no luck with using this, it still fails.
From what I see, hostRules are applied for the renovate app itself, but not the external tools it calls (like go get).

Code here has hardcoded github, so no matter what is set in the hostRules it is not being applied:

renovate/lib/manager/gomod/artifacts.ts

Lines 12 to 25 in a1cddea

	function getPreCommands(): string[] | null {
	const credentials = find({
	hostType: PLATFORM_TYPE_GITHUB,
	url: 'https://api.github.com/',
	});
	let preCommands = null;
	if (credentials?.token) {
	const token = quote(credentials.token);
	preCommands = [
	`git config --global url.\"https://${token}@github.com/\".insteadOf \"https://github.com/\"`, // eslint-disable-line no-useless-escape
	];
	}
	return preCommands;
	}

So looks like there is no way to use it until https://github.com/renovatebot/renovate/issues/3496 is implemented.
Interesting that setting git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org" makes go get work for me, but when the renovatebot runs it doesn't work :(

[rarkins]

Where/how do you run the bot? e.g. are you using our official image, or installing the CLI via npm into your own container, etc?

[taraspos]

@rarkins right now I'm using the official image locally:

$ docker run -it --rm renovate/renovate -v
23.86.0

Also,

I'm using renovate cli from docker image on bitbucket pipelines like:

    renovate-bot:
      - step:
          image: renovate/renovate:23.89 
          name: Renovate Bot
          clone:
            enabled: false
          script:
            - export GOPRIVATE=bitbucket.org/my_org/*
            - export BB_USER=renovateuser
            - git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org"
            - renovate  --username=${BB_USER} --log-level debug --platform=bitbucket $BITBUCKET_REPO_FULL_NAME

[taraspos]

Also, interesting that I do not see any specific error explaining what happened:

DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep1 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep1)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep2 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep2)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep3 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep3)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep4 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep4)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep5 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep5)

no details at all

[rarkins]

You would need a host rule for Renovate, and the git config rule for go. Did you have both? That lookup failure would most likely be a host rule missing.

[taraspos]

I do have git config rule for go:
git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org"

And it works for me when I run go get.

About the host rule, I tried in many different ways and none of them worked for me, can you share an example of how should it look like?
In some cases adding host config, the rule didn't change anything, in some I got permission denied, and also I got Too many redirects error.

Also, in the trace logs, I was able to spot:

{"name":"renovate","hostname":"e19b786e7ef3","pid":1682,"level":10,"logContext":"bHvOPOxOl","repository":"my_org/my_service","url":"https://bitbucket.org/my_org/go-dep1?go-get=1","options":{"method":"get","hostType":"go","hooks":{"beforeRedirect":[null]},"headers":{"accept-encoding":"gzip, deflate","user-agent":"https://github.com/renovatebot/renovate"},"timeout":60000},"msg":"got request","time":"2020-12-04T12:44:34.068Z","v":0}

And when I tried to fetch the URL https://bitbucket.org/my_org/go-dep1?go-get=1 from Postman I also get Too Many Redirects URL. Valid API URL would be the https://api.bitbucket.org/2.0/repositories/my_org/go-dep1?go-get=1

[viceice]

yes:

module.exports = {
  "hostRules": [
        {
            "domainName": "bitbucket.org",
            "username": process.env.BB_USER,
            "password": process.env.RENOVATE_PASSWORD
        }
    ]
}

Didn't this work in combination with your global git config?

[taraspos]

Nope, let me try again, so I can share what was the outcome.

[viceice]

yes, some more debug log lines would be helpful

[viceice]

@trane9991 Can you create a small public github repo with a go dependency pointing to a public bitbucket repo. If this works, we know it's and auth problem, otherwise it's a general renovate go datasource issue

[taraspos]

ok, let me try that

[viceice]

maybe we need to add some special handling for bitbucket.org here:

renovate/lib/datasource/go/index.ts

Line 31 in 2932e88

if (goModule.startsWith('github.com/')) {

so like we replace github.com with api.github.com

[taraspos]

Here, it failed with the public dependency as well:
https://bitbucket.org/Trane9991/renovatebot-issue/addon/pipelines/home#!/results/1

DEBUG: Failed to look up dependency bitbucket.org/AiSee/common-go-lib (repository=Trane9991/renovatebot-issue, packageFile=go.mod, dependency=bitbucket.org/AiSee/common-go-lib)

You probably right about this:

maybe we need to add some special handling for bitbucket.org here:

renovate/lib/datasource/go/index.ts

Line 31 in 2932e88

if (goModule.startsWith('github.com/')) {

so like we replace github.com with api.github.com

For example, this handling of bitbucket.org in the go code:

https://github.com/golang/go/blob/ac38af2f3db7b16067bd8983d1f5278c5c9ef706/src/cmd/go/internal/vcs/vcs.go#L1188-L1193

[viceice]

and finally here is the query against api.bitbucket.org:
https://github.com/golang/go/blob/ac38af2f3db7b16067bd8983d1f5278c5c9ef706/src/cmd/go/internal/vcs/vcs.go#L1275_L1280

[viceice]

I'll check the code, maybe i can send a fast feature to support this

[viceice]

nope, needs a new bitbucket-tags datasource

renovate/lib/datasource/go/index.ts

Line 116 in 2932e88

if (source?.datasource !== github.id && source?.datasource !== gitlab.id) {

[taraspos]

@viceice @rarkins I made an attempt to resolve this in PR #7892.
Since I'm not very good with TS and not familiar with the codebase, would be glad to receive early feedback.

I want to try to run it on the repo I prepared before to make sure that it works, just need to figure out how to run it locally :)

[taraspos]

Ok, was able to run it locally. Getting tags seems to work fine, but that repo has no Tags so it fails.

DEBUG: Found no results from datasource that look like a version (repository=trane9991/renovatebot-issue, dependency=bitbucket.org/AiSee/common-go-lib)
       "result": {"sourceUrl": "https://api.bitbucket.org/AiSee/common-go-lib", "releases": []}
DEBUG: Could not getDigest (repository=trane9991/renovatebot-issue)
       "newValue": "v0.0.0-20191112142628-3dc422aaa32a"

Looks like I need to implement getDigest as well.

[hannibal-jsmd]

module.exports = {
  "hostRules": [
        {
            "domainName": "bitbucket.org",
            "username": process.env.BB_USER,
            "password": process.env.RENOVATE_PASSWORD
        }
    ]
}

Is the above config , the correct way to get the private go packages if they are in the bitbucket server.
@viceice

[viceice]

It should probably work 🤷‍♂️ I do not use go nor bitbucket.org, so you need to validate yourself.

Please postback any issues here. If it works we can close this issue.

[IngCr3at1on]

I hate to piggy back on another issue but I appear to be running into this exact issue trying to run renovate on a kubernetes cluster.

the logs show:

"artifactErrors": [
         {
           "lockFile": "go.sum",
           "stderr": "Command failed: go get -d -t ./...\ngo: bitbucket.org/<org>/<repo>@<tag>: reading https://api.bitbucket.org/2.0/repositories/<org>/<repo>?fields=scm: 403 Forbidden\n\tserver response: Access denied. You must have write or admin access.\n"
         }
       ]

for all of our private repos.

I can confirm from other experience that this error message is generic and shows up in places where only read access is needed.

Just like the OP my pull requests are opened but a note is left with an error for each failed go.sum update.

my abridged (removed the repos list) config is

{
    "platform": "bitbucket",
    "username": "<user>",
    "password": "RENOVATE_APP_PASSWORD",
    "hostRules": [
        {
            "domainName": "bitbucket.org",
            "username": "<user>",
            "password": "RENOVATE_APP_PASSWORD"
        }
    ],
    "postUpdateOptions": [
        "gomodTidy"
    ]
}

My most recent run from today was using renovate/renovate:31.66.3 but I can happily test a dev build if someone has one.

[IngCr3at1on]

So I couldn't really leave this alone lol...

I took another look at the renovate config docs as I knew I had seen a mention of GOPRIVATE support (as this is what we normally do in our Go environments to be able to access these repos, normally we also set a URL in our gitconfig to force using git over ssh vs HTTPS (I don't think such a thing is needed for a bitbucket bot user token?)) but I'm unclear reading the documentation how this value is meant to be set from within the config.

So I took a look at the code and found https://github.com/renovatebot/renovate/blob/main/lib/datasource/go/releases-goproxy.ts#L175 so I was hoping that simply setting the GOPRIVATE value within my kubernetes environment variables would work for this (I editted my yaml on my k8s workload directly to test this) but it seems that I'm still getting the same error in my previous post.

Happy to try to debug this (even by taking a stab at it myself, though my TS is blah at best) but I think I need at least some direction to attempt that.

[github-actions[bot]]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team