Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename components directory to resources under .dapr directory #1149

Draft
wants to merge 38 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
bd3657f
set dapr run defaults precedence
pravinpushkar Dec 9, 2022
4e6d8b5
dapr init create resources dir and moves existing components content …
pravinpushkar Dec 9, 2022
8358a21
modify tests
pravinpushkar Dec 9, 2022
c3f41e9
fix test
pravinpushkar Dec 9, 2022
6affa75
fix tests
pravinpushkar Dec 12, 2022
1b421b0
add tests
pravinpushkar Dec 12, 2022
0363365
trigger pr checks
pravinpushkar Dec 12, 2022
d9caa15
Update pkg/standalone/common.go
pravinpushkar Dec 13, 2022
2007ed5
Apply suggestions from code review
pravinpushkar Dec 13, 2022
bf0e7ff
review comments
pravinpushkar Dec 13, 2022
d8422a7
review comments
pravinpushkar Dec 13, 2022
e73ec7d
fix error handle on path not present
pravinpushkar Dec 13, 2022
093239b
fix tests
pravinpushkar Dec 13, 2022
5df0795
trigger pr checks
pravinpushkar Dec 13, 2022
125ebc8
readme changes and some more messages today
pravinpushkar Dec 14, 2022
3b55cd5
trigger pr checks
pravinpushkar Dec 14, 2022
3c8020b
Update README.md
pravinpushkar Dec 14, 2022
ff6ce5d
update readme
pravinpushkar Dec 14, 2022
5334d15
Typo
shubham1172 Dec 14, 2022
57cfa8a
Type (2)
shubham1172 Dec 14, 2022
15a894c
Merge branch 'master' into feat/resources_dir
mukundansundar Dec 15, 2022
a4c5f81
few more refactoring
pravinpushkar Dec 15, 2022
03d190a
Merge branch 'master' into feat/resources_dir
mukundansundar Dec 19, 2022
5a624af
fix review comment and failing merge
pravinpushkar Dec 20, 2022
3989aad
make symlink
pravinpushkar Dec 24, 2022
0749c5c
Merge branch 'master' into feat/resources_dir
pravinpushkar Dec 26, 2022
4a421d5
fix tests
pravinpushkar Dec 26, 2022
f73af11
review comments
pravinpushkar Dec 29, 2022
e7ccf04
fix tests
pravinpushkar Dec 29, 2022
771b256
some more refactor
pravinpushkar Dec 30, 2022
c5646d4
merge master and fix conflicts
pravinpushkar Jan 3, 2023
968b5ca
fix tests
pravinpushkar Jan 3, 2023
ed1169c
change few uninstall to uninstallAll and add cleanup in upgrade test
pravinpushkar Jan 4, 2023
5871ad4
Merge branch 'master' into feat/resources_dir
mukundansundar Jan 12, 2023
771d423
Merge branch 'master' into feat/resources_dir
shubham1172 Jan 16, 2023
e471c28
fix conflicts & merge master
pravinpushkar Feb 28, 2023
e2794b0
Merge branch 'master' into feat/resources_dir
pravinpushkar Feb 28, 2023
d605587
lint fix
pravinpushkar Feb 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ Output should look like so:

```
⌛ Making the jump to hyperspace...
✅ Downloaded binaries and completed components set up.
✅ Downloaded binaries and completed resources set up.
ℹ️ daprd binary has been installed to $HOME/.dapr/bin.
ℹ️ dapr_placement container is running.
ℹ️ dapr_redis container is running.
Expand All @@ -99,10 +99,12 @@ Output should look like so:

This step creates the following defaults:

1. components folder which is later used during `dapr run` unless the `--resources-path` (`--components-path` is deprecated and will be removed in future releases) option is provided. For Linux/MacOS, the default components folder path is `$HOME/.dapr/components` and for Windows it is `%USERPROFILE%\.dapr\components`.
2. component files in the components folder called `pubsub.yaml` and `statestore.yaml`.
1. resources folder which is later used during `dapr run` unless the `--resources-path` (`--components-path` is deprecated and will be removed in future releases) option is provided. For Linux/MacOS, the default resources folder path is `$HOME/.dapr/resources` and for Windows it is `%USERPROFILE%\.dapr\resources`.
2. resoources files in the resources folder called `pubsub.yaml` and `statestore.yaml`.
3. default config file `$HOME/.dapr/config.yaml` for Linux/MacOS or for Windows at `%USERPROFILE%\.dapr\config.yaml` to enable tracing on `dapr init` call. Can be overridden with the `--config` flag on `dapr run`.

> Note: If there is a `components` directory present in the default dapr installation path `$HOME/.dapr/resources` (for Linux/MacOS) or `%USERPROFILE%\.dapr\resources` (for windows) then `dapr init` will copy the `resources` from the `components` to `resources` directory. This is to ensure safe migration from using `components` to `resources` directory.

#### Slim Init

Alternatively to the above, to have the CLI not install any default configuration files or run Docker containers, use the `--slim` flag with the init command. Only Dapr binaries will be installed.
Expand All @@ -115,13 +117,15 @@ Output should look like so:

```bash
⌛ Making the jump to hyperspace...
✅ Downloaded binaries and completed components set up.
✅ Downloaded binaries and completed resources set up.
ℹ️ daprd binary has been installed to $HOME/.dapr/bin.
ℹ️ placement binary has been installed.
✅ Success! Dapr is up and running. To get started, go here: https://aka.ms/dapr-getting-started
```

>Note: When initializing Dapr with the `--slim` flag only the Dapr runtime binary and the placement service binary are installed. An empty default components folder is created with no default configuration files. During `dapr run` user should use `--resources-path` (`--components-path` is deprecated and will be removed in future releases) to point to a components directory with custom configurations files or alternatively place these files in the default directory. For Linux/MacOS, the default components directory path is `$HOME/.dapr/components` and for Windows it is `%USERPROFILE%\.dapr\components`.
>Note: When initializing Dapr with the `--slim` flag only the Dapr runtime binary and the placement service binary are installed. An empty default `resources` folder is created with no default configuration files. During `dapr run` user should use `--resources-path` (`--components-path` is deprecated and will be removed in future releases) to point to a `resources` directory with custom configurations files or alternatively place these files in the default directory. For Linux/MacOS, the default `resources` directory path is `$HOME/.dapr/resources` and for Windows it is `%USERPROFILE%\.dapr\resources`.

> Note: If there is a `components` directory present in the default dapr installation path `$HOME/.dapr/resources` (for Linux/MacOS) or `%USERPROFILE%\.dapr\resources` (for windows) then `dapr init` will copy the `resources` from the `components` to `resources` directory. This is to ensure safe migration from using `components` to `resources` directory.

#### Install a specific runtime version

Expand Down Expand Up @@ -325,7 +329,7 @@ To remove all Dapr Custom Resource Definitions:
dapr uninstall -k --all
```

*Warning: this will remove any components, subscriptions or configurations that are applied in the cluster at the time of deletion.*
*Warning: this will remove any resources(and components also), subscriptions or configurations that are applied in the cluster at the time of deletion.*
pravinpushkar marked this conversation as resolved.
Show resolved Hide resolved

### Upgrade Dapr on Kubernetes

Expand Down Expand Up @@ -444,7 +448,7 @@ Launch Dapr and your app:
dapr run --app-id nodeapp --app-port 3000 node app.js
```

Note: To choose a non-default components folder, use the --resources-path(--components-path is deprecated) option.
Note: To choose a non-default resources folder, use the --resources-path(--components-path is deprecated) option.
pravinpushkar marked this conversation as resolved.
Show resolved Hide resolved

Invoke your app:

Expand Down
2 changes: 1 addition & 1 deletion cmd/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -377,7 +377,7 @@ func init() {
RunCmd.Flags().StringVarP(&logLevel, "log-level", "", "info", "The log verbosity. Valid values are: debug, info, warn, error, fatal, or panic")
RunCmd.Flags().IntVarP(&maxConcurrency, "app-max-concurrency", "", -1, "The concurrency level of the application, otherwise is unlimited")
RunCmd.Flags().StringVarP(&protocol, "app-protocol", "P", "http", "The protocol (gRPC or HTTP) Dapr uses to talk to the application")
RunCmd.Flags().StringVarP(&componentsPath, "components-path", "d", standalone.DefaultComponentsDirPath(), "The path for components directory")
RunCmd.Flags().StringVarP(&componentsPath, "components-path", "d", standalone.DefaultResourcesDirPrecedence(), "The path for components directory")
RunCmd.Flags().StringVarP(&resourcesPath, "resources-path", "", "", "The path for resources directory")
pravinpushkar marked this conversation as resolved.
Show resolved Hide resolved
// TODO: Remove below line once the flag is removed in the future releases.
// By marking this as deprecated, the flag will be hidden from the help menu, but will continue to work. It will show a warning message when used.
Expand Down
67 changes: 62 additions & 5 deletions pkg/standalone/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,15 @@ import (
"os"
path_filepath "path/filepath"
"runtime"

"github.com/dapr/cli/pkg/print"
"github.com/dapr/cli/utils"
)

const (
defaultDaprDirName = ".dapr"
defaultDaprBinDirName = "bin"
defaultComponentsDirName = "components"
defaultConfigFileName = "config.yaml"
defaultDaprDirName = ".dapr"
defaultDaprBinDirName = "bin"
defaultConfigFileName = "config.yaml"
)

func defaultDaprDirPath() string {
Expand All @@ -44,9 +46,64 @@ func binaryFilePath(binaryDir string, binaryFilePrefix string) string {
}

func DefaultComponentsDirPath() string {
return path_filepath.Join(defaultDaprDirPath(), defaultComponentsDirName)
return path_filepath.Join(defaultDaprDirPath(), utils.DefaultComponentsDirName)
}

func DefaultResourcesDirPath() string {
return path_filepath.Join(defaultDaprDirPath(), utils.DefaultResourcesDirName)
}

// DefaultResourcesDirPrecedence returns the path to the resources directory if it exists, or otherwise the components directory.
// TODO: Remove this function and use `DefaultResourcesDirPath` when `--components-path` flag is removed.
func DefaultResourcesDirPrecedence() string {
shubham1172 marked this conversation as resolved.
Show resolved Hide resolved
defaultResourcesDirPath := DefaultResourcesDirPath()
if _, err := os.Stat(defaultResourcesDirPath); os.IsNotExist(err) {
return DefaultComponentsDirPath()
}
return defaultResourcesDirPath
}

func DefaultConfigFilePath() string {
return path_filepath.Join(defaultDaprDirPath(), defaultConfigFileName)
}

// emptyAndCopyFiles copies files from src to dest. It deletes the existing files in dest before copying from src.
// TODO: Remove this function when `--components-path` flag is removed.
func emptyAndCopyFiles(src, dest string) error {
pravinpushkar marked this conversation as resolved.
Show resolved Hide resolved
if _, err := os.Stat(src); err != nil {
// if the src directory does not exist, return nil, because there is nothing to copy from.
if os.IsNotExist(err) {
return nil
}
return err
}
files, err := os.ReadDir(dest)
if err != nil {
return err
}
for _, file := range files {
err = os.Remove(dest + "/" + file.Name())
pravinpushkar marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return err
}
}
files, err = os.ReadDir(src)
if err != nil {
return err
}
if len(files) > 0 {
print.InfoStatusEvent(os.Stdout, "Moving files from %q to %q", src, dest)
for _, file := range files {
content, err := os.ReadFile(src + "/" + file.Name())
pravinpushkar marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return err
}
// #nosec G306
err = os.WriteFile(dest+"/"+file.Name(), content, 0o644)
if err != nil {
return err
}
}
}
return nil
}
8 changes: 4 additions & 4 deletions pkg/standalone/run_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ func assertArgumentNotEqual(t *testing.T, key string, expectedValue string, args
}

func setupRun(t *testing.T) {
componentsDir := DefaultComponentsDirPath()
componentsDir := DefaultResourcesDirPath()
configFile := DefaultConfigFilePath()
err := os.MkdirAll(componentsDir, 0o700)
assert.Equal(t, nil, err, "Unable to setup components dir before running test")
Expand All @@ -66,7 +66,7 @@ func setupRun(t *testing.T) {
}

func tearDownRun(t *testing.T) {
err := os.RemoveAll(DefaultComponentsDirPath())
err := os.RemoveAll(DefaultResourcesDirPath())
assert.Equal(t, nil, err, "Unable to delete default components dir after running test")
err = os.Remove(DefaultConfigFilePath())
assert.Equal(t, nil, err, "Unable to delete default config file after running test")
Expand All @@ -87,7 +87,7 @@ func assertCommonArgs(t *testing.T, basicConfig *RunConfig, output *RunOutput) {
assertArgumentEqual(t, "app-max-concurrency", "-1", output.DaprCMD.Args)
assertArgumentEqual(t, "app-protocol", "http", output.DaprCMD.Args)
assertArgumentEqual(t, "app-port", "3000", output.DaprCMD.Args)
assertArgumentEqual(t, "components-path", DefaultComponentsDirPath(), output.DaprCMD.Args)
assertArgumentEqual(t, "components-path", DefaultResourcesDirPrecedence(), output.DaprCMD.Args)
assertArgumentEqual(t, "app-ssl", "", output.DaprCMD.Args)
assertArgumentEqual(t, "metrics-port", "9001", output.DaprCMD.Args)
assertArgumentEqual(t, "dapr-http-max-request-size", "-1", output.DaprCMD.Args)
Expand Down Expand Up @@ -148,7 +148,7 @@ func TestRun(t *testing.T) {
EnableProfiling: false,
ProfilePort: 9090,
Protocol: "http",
ComponentsPath: DefaultComponentsDirPath(),
ComponentsPath: DefaultResourcesDirPrecedence(),
AppSSL: true,
MetricsPort: 9001,
MaxRequestBodySize: -1,
Expand Down
34 changes: 20 additions & 14 deletions pkg/standalone/standalone.go
Original file line number Diff line number Diff line change
Expand Up @@ -243,8 +243,8 @@ func Init(runtimeVersion, dashboardVersion string, dockerNetwork string, slimMod
stopSpinning := print.Spinner(os.Stdout, msg)
defer stopSpinning(print.Failure)

// Make default components directory.
err = makeDefaultComponentsDir()
// Make default resources directory.
err = makeDefaultResourcesDir()
if err != nil {
return err
}
Expand Down Expand Up @@ -279,9 +279,9 @@ func Init(runtimeVersion, dashboardVersion string, dockerNetwork string, slimMod

stopSpinning(print.Success)

msg = "Downloaded binaries and completed components set up."
msg = "Downloaded binaries and completed resources set up."
if isAirGapInit {
msg = "Extracted binaries and completed components set up."
msg = "Extracted binaries and completed resources set up."
}
print.SuccessStatusEvent(os.Stdout, msg)
print.InfoStatusEvent(os.Stdout, "%s binary has been installed to %s.", daprRuntimeFilePrefix, daprBinDir)
Expand All @@ -295,9 +295,10 @@ func Init(runtimeVersion, dashboardVersion string, dockerNetwork string, slimMod
if isAirGapInit {
dockerContainerNames = []string{DaprPlacementContainerName}
}
var ok bool
for _, container := range dockerContainerNames {
containerName := utils.CreateContainerName(container, dockerNetwork)
ok, err := confirmContainerIsRunningOrExists(containerName, true, runtimeCmd)
ok, err = confirmContainerIsRunningOrExists(containerName, true, runtimeCmd)
if err != nil {
return err
}
Expand All @@ -307,6 +308,11 @@ func Init(runtimeVersion, dashboardVersion string, dockerNetwork string, slimMod
}
print.InfoStatusEvent(os.Stdout, "Use `%s ps` to check running containers.", runtimeCmd)
}
// TODO: remove below method when components-path flag is removed.
err = emptyAndCopyFiles(DefaultComponentsDirPath(), DefaultResourcesDirPath())
pravinpushkar marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return err
}
return nil
}

Expand Down Expand Up @@ -659,14 +665,14 @@ func createComponentsAndConfiguration(wg *sync.WaitGroup, errorChan chan<- error
var err error

// Make default components directory.
componentsDir := DefaultComponentsDirPath()
resourcesDir := DefaultResourcesDirPath()

err = createRedisPubSub(redisHost, componentsDir)
err = createRedisPubSub(redisHost, resourcesDir)
if err != nil {
errorChan <- fmt.Errorf("error creating redis pubsub component file: %w", err)
return
}
err = createRedisStateStore(redisHost, componentsDir)
err = createRedisStateStore(redisHost, resourcesDir)
if err != nil {
errorChan <- fmt.Errorf("error creating redis statestore component file: %w", err)
return
Expand All @@ -693,19 +699,19 @@ func createSlimConfiguration(wg *sync.WaitGroup, errorChan chan<- error, info in
}
}

func makeDefaultComponentsDir() error {
func makeDefaultResourcesDir() error {
// Make default components directory.
componentsDir := DefaultComponentsDirPath()
resourcesDir := DefaultResourcesDirPath()
//nolint
_, err := os.Stat(componentsDir)
_, err := os.Stat(resourcesDir)
if os.IsNotExist(err) {
errDir := os.MkdirAll(componentsDir, 0o755)
errDir := os.MkdirAll(resourcesDir, 0o755)
if errDir != nil {
return fmt.Errorf("error creating default components folder: %w", errDir)
return fmt.Errorf("error creating default resources folder: %w", errDir)
}
}

os.Chmod(componentsDir, 0o777)
os.Chmod(resourcesDir, 0o777)
return nil
}

Expand Down
8 changes: 4 additions & 4 deletions tests/e2e/standalone/init_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ func verifyBinaries(t *testing.T, daprPath, runtimeVersion, dashboardVersion str
}
}

// verifyConfigs ensures that the Dapr configuration and component YAMLs
// verifyConfigs ensures that the Dapr configuration and resources YAMLs
// are present in the correct path and have the correct values.
func verifyConfigs(t *testing.T, daprPath string) {
configSpec := map[interface{}]interface{}{}
Expand All @@ -218,9 +218,9 @@ func verifyConfigs(t *testing.T, daprPath string) {
},
}

// The default components are not installed in slim mode.
// The default resources are not installed in slim mode.
if !isSlimMode() {
configs[filepath.Join("components", "statestore.yaml")] = map[string]interface{}{
configs[filepath.Join("resources", "statestore.yaml")] = map[string]interface{}{
"apiVersion": "dapr.io/v1alpha1",
"kind": "Component",
"metadata": map[interface{}]interface{}{
Expand All @@ -245,7 +245,7 @@ func verifyConfigs(t *testing.T, daprPath string) {
},
},
}
configs[filepath.Join("components", "pubsub.yaml")] = map[string]interface{}{
configs[filepath.Join("resources", "pubsub.yaml")] = map[string]interface{}{
"apiVersion": "dapr.io/v1alpha1",
"kind": "Component",
"metadata": map[interface{}]interface{}{
Expand Down
15 changes: 15 additions & 0 deletions tests/e2e/standalone/run_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,21 @@ func TestStandaloneRun(t *testing.T) {
assert.Contains(t, output, "Exited Dapr successfully")
})

// TODO: Remove this test when the deprecated --components-path flag is removed.
t.Run(fmt.Sprintf("check run with components-path flag"), func(t *testing.T) {
args := []string{
"--app-id", "testapp",
"--components-path", "../testdata/resources",
"--", "bash", "-c", "echo 'test'",
}
output, err := cmdRun("", args...)
t.Log(output)
require.NoError(t, err, "run failed")
assert.Contains(t, output, "component loaded. name: test-statestore, type: state.in-memory/v1")
assert.Contains(t, output, "Exited App successfully")
assert.Contains(t, output, "Exited Dapr successfully")
})

t.Run("run with unknown flags", func(t *testing.T) {
output, err := cmdRun("", "--flag")
require.Error(t, err, "expected error on run unknown flag")
Expand Down
Loading