You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What is the bug?
It is forbidden to replicate indices starting from . in OpenSearch, but Dashboards indices start from . and are not system. So, there is no ability to run successful (without failed_indices) autofollow for all indices (*) with installed dashboards.
How can one reproduce the bug?
Steps to reproduce the behavior:
Install Dashboards along with OpenSearch on primary and secondary clusters.
Run autofollow replication on the secondary cluster
Login to dashboards on primary cluster with admin username and make some actions: create own dashboard, make a backup, etc.
Check indices on primary cluster (GET _cat/indices?v):
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open test 1BUqHYGmRPCW0AO-ix8tuA 1 1 2 0 7.4kb 3.7kb
green open .kibana_92668751_admin_1 I7fIY7F7StmRgh9ZaQhREw 1 1 8 1 60.8kb 30.4kb
green open .kibana_1 mvJuKuevQViCoZS33_r2yg 1 1 0 0 416b 208b
green open .opendistro_security qg36BdInQQS5ACC4qu3ROQ 1 2 10 3 203.9kb 67.9kb
green open .opensearch-notifications-config HHDun-gLSNWY3UyyPTZytw 1 2 0 0 624b 208b
Check autofollow statistics on secondary cluster (GET _plugins/_replication/autofollow_stats):
[2023-05-19T15:44:28,651][INFO ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Auto follow starting replication from leader-cluster:.opensearch-notifications-config -> .opensearch-notifications-config
[2023-05-19T15:44:28,651][WARN ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Failed to start replication for leader-cluster:.opensearch-notifications-config -> .opensearch-notifications-config.
org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: Value .opensearch-notifications-config must not start with '.';2: Value .opensearch-notifications-config must not start with '.';
at org.opensearch.replication.action.index.ReplicateIndexRequest.validate(ReplicateIndexRequest.kt:94) ~[?:?]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:177) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invokeSuspend(Coroutines.kt:89) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlinx.coroutines.BuildersKt.withContext(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at org.opensearch.replication.util.CoroutinesKt.suspendExecute(Coroutines.kt:87) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.util.CoroutinesKt.suspendExecute$default(Coroutines.kt:84) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplication(AutoFollowTask.kt:202) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplicationJobs(AutoFollowTask.kt:175) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask.access$startReplicationJobs(AutoFollowTask.kt:49) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask$startReplicationJobs$1.invokeSuspend(AutoFollowTask.kt) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:147) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.4.1.jar:2.4.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-19T15:44:58,661][INFO ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Auto follow starting replication from leader-cluster:.kibana_92668751_admin_1 -> .kibana_92668751_admin_1
[2023-05-19T15:44:58,661][WARN ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Failed to start replication for leader-cluster:.kibana_92668751_admin_1 -> .kibana_92668751_admin_1.
org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: Value .kibana_92668751_admin_1 must not start with '.';2: Value .kibana_92668751_admin_1 must not start with '.';
at org.opensearch.replication.action.index.ReplicateIndexRequest.validate(ReplicateIndexRequest.kt:94) ~[?:?]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:177) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invokeSuspend(Coroutines.kt:89) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlinx.coroutines.BuildersKt.withContext(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at org.opensearch.replication.util.CoroutinesKt.suspendExecute(Coroutines.kt:87) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.util.CoroutinesKt.suspendExecute$default(Coroutines.kt:84) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplication(AutoFollowTask.kt:202) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplicationJobs(AutoFollowTask.kt:175) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask.access$startReplicationJobs(AutoFollowTask.kt:49) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at org.opensearch.replication.task.autofollow.AutoFollowTask$startReplicationJobs$1.invokeSuspend(AutoFollowTask.kt) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:147) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.4.1.jar:2.4.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
What is the expected behavior?
Skip indices starting from . symbol, because they are already forbidden.
What is your host/environment?
OpenSearch Version - 2.4.1
Kubernetes Version - 1.23.6
The text was updated successfully, but these errors were encountered:
OpenSearch has a lot of indices that start from .. As I understand, they are all system and do not cause any problems with replication. All Dashboards indices I've seen are prefixed with .. Does it mean they are system? If they aren't, why do they start with .?
Also, is it okay to replicate Dashboards indices? Could the Dashboards start with replicated indices successfully?
For now we don't need to replicate Dashboards indices, but it would be good to have solution according to OpenSearch conception.
What is the bug?
It is forbidden to replicate indices starting from
.
in OpenSearch, butDashboards
indices start from.
and are not system. So, there is no ability to run successful (withoutfailed_indices
) autofollow for all indices (*
) with installeddashboards
.How can one reproduce the bug?
Steps to reproduce the behavior:
Dashboards
along with OpenSearch on primary and secondary clusters.autofollow
replication on the secondary clusterdashboards
on primary cluster withadmin
username and make some actions: create own dashboard, make a backup, etc.autofollow
statistics on secondary cluster (GET _plugins/_replication/autofollow_stats):What is the expected behavior?
Skip indices starting from
.
symbol, because they are already forbidden.What is your host/environment?
The text was updated successfully, but these errors were encountered: