Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add agent sidecar flame for implementation #404

Merged
merged 4 commits into from
May 26, 2020

Conversation

kpango
Copy link
Collaborator

@kpango kpango commented May 25, 2020

Signed-off-by: kpango [email protected]

Description:

We decided to implement blob storage backup functionality.
so, I added a prototype of agent sidecar flame, @rinx please implement backup logic after this PR merged.

Related Issue:

How Has This Been Tested?:

Environment:

  • Golang Version: 1.14.3
  • Docker Version: 19.03.8
  • Kubernetes Version: 1.18.2
  • NGT Version: 1.11.5

Types of changes:

  • Bug fix [type/bug]
  • New feature [type/feature]
  • Add tests [type/test]
  • Security related changes [type/security]
  • Add documents [type/documentation]
  • Refactoring [type/refactoring]
  • Update dependencies [type/dependency]
  • Update benchmarks and performances [type/bench]
  • Update CI [type/ci]

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

Checklist:

  • I have read the CONTRIBUTING document.
  • I have checked open Pull Requests for the similar feature or fixes?
  • I have added tests and benchmarks to cover my changes.
  • I have ensured all new and existing tests passed.
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly.

Signed-off-by: kpango <[email protected]>
@pull-assistant
Copy link

pull-assistant bot commented May 25, 2020

Score: 0.96

Best reviewed: commit by commit


Optimal code review plan

     add agent sidecar

     fix

     fix

     fix

Powered by Pull Assistant. Last update 376857c ... 06897f5. Read the comment docs.

@vdaas-ci
Copy link
Collaborator

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

@codecov
Copy link

codecov bot commented May 25, 2020

Codecov Report

Merging #404 into master will decrease coverage by 0.17%.
The diff coverage is 0.80%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #404      +/-   ##
=========================================
- Coverage    7.88%   7.71%   -0.18%     
=========================================
  Files         356     372      +16     
  Lines       18380   18850     +470     
=========================================
+ Hits         1450    1454       +4     
- Misses      16721   17183     +462     
- Partials      209     213       +4     
Impacted Files Coverage Δ
cmd/agent/core/ngt/main.go 0.00% <ø> (ø)
cmd/agent/sidecar/main.go 0.00% <0.00%> (ø)
...k/benchmark/internal/starter/agent/core/ngt/ngt.go 0.00% <ø> (ø)
...enchmark/internal/starter/agent/core/ngt/option.go 5.26% <ø> (ø)
hack/tools/config/agent/core/ngt/main.go 0.00% <ø> (ø)
internal/client/agent/grpc/client.go 0.00% <ø> (ø)
internal/config/ngt.go 0.00% <0.00%> (ø)
internal/config/sidecar.go 0.00% <0.00%> (ø)
internal/file/watch/option.go 0.00% <0.00%> (ø)
internal/file/watch/watch.go 0.00% <0.00%> (ø)
... and 48 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb392fc...06897f5. Read the comment docs.

Copy link
Contributor

@rinx rinx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good. Please revise several lines i pointed out. 👍

internal/file/watch/option.go Outdated Show resolved Hide resolved
internal/file/watch/watch.go Show resolved Hide resolved
err = w.onChmod(ctx, event.Name)
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

badge

Suggested change
}
case err, ok = <-w.w.Errors:
}

and remove the goroutine started from L145.
it's better to reduce the number of useless goroutines.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fsnotify recommend to watch event and error in a different goroutine.
howeyc/fsnotify#7

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read it but i couldn't found a recommendation about that.
i think it's just about "not to create a fsnotify Watcher and monitor channels in a same goroutine".

I'm considering about the race condition about init() func when the w.w.Errors and w.w.Events closed simultaneously.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry they recommend this thing in README.md

Do I have to watch the Error and Event channels in a separate goroutine?

As of now, yes. Looking into making this single-thread friendly (see howeyc #7)

and I agree with init race condition, I'll fix race condition problem about it but I think it's okay to run in different goroutine that mentioned in howeyc/fsnotify#7

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as described in README.md of fsnotify, it is only needed "a separate goroutine".
we don't have to have two goroutines here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a separate goroutine doesn't mean using different goroutine?

internal/config/sidecar.go Outdated Show resolved Hide resolved
pkg/agent/sidecar/service/option.go Outdated Show resolved Hide resolved
@vdaas-ci
Copy link
Collaborator

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

return w, nil
}

func (w *watch) Start(ctx context.Context) (<-chan error, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
Function 'Start' is too long (81 > 60) (funlen)

so service.StorageObserver
}

func New(cfg *config.Data) (r runner.Runner, err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
Function 'New' is too long (76 > 60) (funlen)

"reflect"
"testing"

"github.com/vdaas/vald/internal/errors"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
File is not goimports-ed (goimports)

}

func (r *run) Start(ctx context.Context) (<-chan error, error) {
ech := make(chan error, 5)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
mnd: Magic number: 5, in detected (gomnd)

return nil
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
File is not goimports-ed (goimports)

"reflect"
"testing"

"github.com/vdaas/vald/internal/errors"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
File is not goimports-ed (goimports)

@kpango kpango force-pushed the feature/agent-sidecar/add-s3-sidecar branch from 2c18b64 to 6787380 Compare May 26, 2020 01:40
@vdaas-ci
Copy link
Collaborator

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

}

func (w *watch) Start(ctx context.Context) (<-chan error, error) {
ech := make(chan error, 10)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
mnd: Magic number: 10, in detected (gomnd)

handleErr(ctx, err)
}
}
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
unreachable: unreachable code (govet)

}
}
}
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
unreachable: unreachable code (govet)

"reflect"
"testing"

"github.com/vdaas/vald/internal/errors"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
File is not goimports-ed (goimports)

}
d, err := timeutil.Parse(dur)
if err != nil {
d = time.Minute * 5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
mnd: Magic number: 5, in detected (gomnd)

fix
Signed-off-by: kpango <[email protected]>
@kpango kpango force-pushed the feature/agent-sidecar/add-s3-sidecar branch from 6787380 to d3bcfb8 Compare May 26, 2020 01:51
@vdaas-ci
Copy link
Collaborator

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

func TestWithOnChmod(t *testing.T) {
type T = interface{}
type args struct {
f func(ctx context.Context, name string) error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field f is unused (unused)

func TestWithOnError(t *testing.T) {
type T = interface{}
type args struct {
f func(ctx context.Context, err error) error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field f is unused (unused)

func TestWithOnWrite(t *testing.T) {
type T = interface{}
type args struct {
f func(ctx context.Context, name string) error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field f is unused (unused)

func TestWithDirs(t *testing.T) {
type T = interface{}
type args struct {
dirs []string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field dirs is unused (unused)

func TestWithErrGroup(t *testing.T) {
type T = interface{}
type args struct {
eg errgroup.Group
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field eg is unused (unused)

func TestWithAccessKey(t *testing.T) {
type T = interface{}
type args struct {
ak string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field ak is unused (unused)

func TestWithSecretAccessKey(t *testing.T) {
type T = interface{}
type args struct {
sak string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field sak is unused (unused)

func TestWithTimeout(t *testing.T) {
type T = interface{}
type args struct {
timeout string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field timeout is unused (unused)

func TestWithHandler(t *testing.T) {
type T = interface{}
type args struct {
h rest.Handler
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field h is unused (unused)

}

func (o *observer) backup(ctx context.Context) (err error) {
ctx, span := trace.StartSpan(ctx, "vald/agent-sidecar/service/StorageObserver.backup")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
ineffectual assignment to ctx (ineffassign)

fix
Signed-off-by: kpango <[email protected]>
@vdaas-ci
Copy link
Collaborator

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

rinx
rinx previously approved these changes May 26, 2020
Copy link
Contributor

@rinx rinx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 Thanks.

fix
Signed-off-by: kpango <[email protected]>
@vdaas-ci
Copy link
Collaborator

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

golangci

pkg/agent/sidecar/usecase/sidecard.go|137 col 2| only one cuddle assignment allowed before if statement (wsl)
pkg/agent/sidecar/usecase/sidecard.go|140 col 2| if statements should only be cuddled with assignments (wsl)
pkg/agent/sidecar/usecase/sidecard.go|147 col 2| assignments should only be cuddled with other assignments (wsl)
pkg/agent/sidecar/usecase/sidecard.go|167 col 2| return statements should not be cuddled if block has more than two lines (wsl)
pkg/agent/sidecar/usecase/sidecard.go|178 col 2| return statements should not be cuddled if block has more than two lines (wsl)
pkg/agent/sidecar/usecase/sidecard_test.go|38 col 2| declarations should never be cuddled (wsl)
pkg/agent/sidecar/usecase/sidecard_test.go|54 col 3| if statements should only be cuddled with assignments (wsl)
pkg/agent/sidecar/usecase/sidecard_test.go|237 col 3| if statements should only be cuddled with assignments (wsl)
pkg/agent/sidecar/usecase/sidecard.go|17 col 1| ST1000: at least one file in a package should have a package comment (stylecheck)
internal/file/watch/watch.go|94 col 1| cognitive complexity 54 of func (*watch).Start is high (> 30) (gocognit)

func TestWithBackupDuration(t *testing.T) {
type T = interface{}
type args struct {
dur string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field dur is unused (unused)

type test struct {
name string
args args
want want
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field want is unused (unused)

type args struct {
dur string
}
type want struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
type want is unused (unused)

func TestWithDirs(t *testing.T) {
type T = interface{}
type args struct {
dirs []string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field dirs is unused (unused)

func TestWithBackupDurationLimit(t *testing.T) {
type T = interface{}
type args struct {
dur string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
field dur is unused (unused)

service.WithBackupDurationLimit(cfg.AgentSidecar.AutoBackupDurationLimit),
service.WithDirs(cfg.AgentSidecar.WatchPaths...),
)
if err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
only one cuddle assignment allowed before if statement (wsl)

if err != nil {
return nil, err
}
g := handler.New(handler.WithStorageObserver(so))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
assignments should only be cuddled with other assignments (wsl)

if r.observability != nil {
return r.observability.PreStart(ctx)
}
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
return statements should not be cuddled if block has more than two lines (wsl)


func (r *run) Start(ctx context.Context) (<-chan error, error) {
ech := make(chan error, 5)
var soech, sech, oech <-chan error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
declarations should never be cuddled (wsl)

func (r *run) Start(ctx context.Context) (<-chan error, error) {
ech := make(chan error, 5)
var soech, sech, oech <-chan error
var err error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[golangci] reported by reviewdog 🐶
declarations should never be cuddled (wsl)

@rinx rinx merged commit 7d5c1d1 into master May 26, 2020
@rinx rinx deleted the feature/agent-sidecar/add-s3-sidecar branch May 26, 2020 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants