Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenSearch / OpenSearch Dashboards - DEB distribution (X64) #28

Closed
1 task
bbarani opened this issue May 12, 2021 · 66 comments
Closed
1 task

OpenSearch / OpenSearch Dashboards - DEB distribution (X64) #28

bbarani opened this issue May 12, 2021 · 66 comments
Assignees
Labels
deb distributions enhancement New Enhancement roadmap triaged This issue has been reviewed by the triage team v1.3.9

Comments

@bbarani
Copy link
Member

bbarani commented May 12, 2021

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Provide signed (X64)Debian artifacts for OpenSearch / OpenSearch Dashboards

Describe alternatives you've considered
Generating the artifacts using the source code

Additional context
Currently there are no DEB artifacts for OpenSearch / OpenSearch Dashboards

  • Refresh Button

@bbarani bbarani added the enhancement New Enhancement label May 12, 2021
@bbarani bbarani removed the v1.0.0 label Jun 16, 2021
@bbarani bbarani added v1.1.0 and removed v1.1.0 labels Aug 19, 2021
@peternied peternied added v1.2.0 OpenSearch 1.2 version triaged This issue has been reviewed by the triage team labels Sep 14, 2021
@jcgraybill
Copy link
Contributor

It looks like Elasticsearch's DEB packages set up SysV or systemd hooks for running Elasticsearch. In the spirit of allowing OpenSearch to be a drop-in migration, let's be sure its DEB packages are set up the same way. https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html#start-deb

@mbreevoort
Copy link

Using opendistro and waiting until the debian packages are available, does it make v1 2.0 this time?

@peternied peternied removed the v1.2.0 OpenSearch 1.2 version label Nov 16, 2021
@gaiksaya
Copy link
Member

[Triage] We are working on the design documentation at this point. Please see #956

@deknos
Copy link

deknos commented Mar 24, 2022

Hello, please consider to generate not only debian/rpm binary packages, but also source packages.

This will for one ensure that your packaging quality will be better, it also would enhance the possibilities that your software would be included in standard repositories of systems like redhat, debian, ubuntu, suse as they will only include packages where the have the source version of.

Please consider this. This would dramatically enhance your position as a project in the opensource community.

@dblock
Copy link
Member

dblock commented Mar 24, 2022

@deknos Do you have some more specifics about this for those like me not familiar with the different kinds of packages?

@deknos
Copy link

deknos commented Mar 24, 2022

Sure!
For Debian at least here: https://wiki.debian.org/Packaging/SourcePackage

What you do in general is, that you have an original software (like your source) and you build with debian tooling a debian source package which are the original tar.gz (just an example) and a dsc-file. with these you can build a debian binary package with dpkg-buildpackage.

for the question how to do this, there's a good tutorial by a former debian project leader:
https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf

the only drawback is: debian in general wants that you also package your dependency properly, i do not know if there's some kind of workaround if you do not want to to this.

But also: when you did this for debian (source package, from which a debian binary package is built easily), building an rpm package is very easy because they adhere to similar principles.

if you want, you can contact me ([email protected]) and i could try to help with debian packaging, i packaged software for a company a few years back :)

@dblock
Copy link
Member

dblock commented Mar 24, 2022

Thanks @deknos. This is similar to how FreeBSD folks do it, or how OpenSearch was added to Homebrew, i.e. build from source using tooling on the host on which the software is ultimately going to reside. Generally I see two drawbacks to that approach: it would produce a different set of binaries depending on the environment, it doesn't produce signed binaries that users can rely on and verify the origin of (or we'd have to use the rpm process to build the binaries, then release the signed bits, but rpm doesn't exist on all platforms, and in this issue we chose to go directly to a binary package that includes signed bits built from source).

I think we'd spend our energy building support for things like MacOS or Windows before we build a second or a third way of distributing for Linux. I think though we would definitely welcome a community project just like what was done for FreeBSD and Homebrew and we could integrate it similarly to those (some source code is in opensearch-project, such as docs, and some code is not, such as the FreeBSD ports themselves).

Thanks for the offer to do the debian packaging work though, don't let me stop you from doing that! We'd gladly accept PRs here if they move us forward into the right direction. You could absolutely start contributing those things here, or build a personal repo, up to you and let us know how we can help.

@theOperand
Copy link

I think we'd spend our energy building support for things like MacOS or Windows before we build a second or a third way of distributing for Linux.

Does this mean that this issue is postponed indefinitely? A .deb package would be very useful, as many organizations (such as mine) that use Ubuntu servers but no Docker are still effectively stuck on the last non-SSPL Elasticsearch version.

@dblock
Copy link
Member

dblock commented May 17, 2022

It just means nobody is working on it. @peterzhuamazon care to update this ticket with a list of things you think one would need to do to make a .deb package so that someone can contribute?

@bbarani
Copy link
Member Author

bbarani commented May 18, 2022

We are currently working on putting together a high level design and plan for both Windows and Deb distribution. We will publish the initial version of the plan very soon on this issue. Having said that, we don't have the release timeline for additional Linux distributions yet.

@gruselglatz
Copy link

How is the current State? Do you need some Help? There is already a Debian Build in the project iirc in the build.gradle?

@marcinwisniowski
Copy link

@bbarani Can you tell us any ETA of DEBs for OpenSearch? It's over a year that Ubuntu users are waiting to adopt OpenSearch in place of OpenDistro :)

@bertvandepoel
Copy link

I see this has moved to In-progress. Is there anything we can do to help with development or testing? We're upgrading from 20.04 to 22.04 and are concerned that our old pre-SSPL packages will at some point fail. So we have a lot of motivation to help out 😅

@peterzhuamazon
Copy link
Member

Fixed OSD size:

318M	opensearch-dashboards-2.4.0-linux-x64.deb

@peterzhuamazon
Copy link
Member

Need proper maintainers scripts just like in RPM to make DEB run correctly.

@peterzhuamazon
Copy link
Member

After a lot of research we will not sign DEB packages but sign APT repo InRelease files.
As for the deb packages directly available for download, not through apt, we will provide detached .sig file just like rpm.

@peterzhuamazon
Copy link
Member

OS deb now:

$ dpkg-deb -I opensearch-2.4.1-linux-x64.deb
 new Debian package, version 2.0.
 size 554486428 bytes: control archive=25738 bytes.
    1687 bytes,    32 lines      conffiles
     418 bytes,    12 lines      control
   94409 bytes,   935 lines      md5sums
    2239 bytes,    63 lines   *  postinst             #!/bin/sh
    1040 bytes,    31 lines   *  preinst              #!/bin/sh
     774 bytes,    26 lines   *  prerm                #!/bin/sh
      32 bytes,     1 lines      shlibs
      73 bytes,     2 lines      triggers
 Package: opensearch
 Version: 2.4.1
 Architecture: amd64
 Maintainer: OpenSearch Team <[email protected]>
 Installed-Size: 759872
 Section: web
 Priority: optional
 Homepage: https://opensearch.org/
 Description: An open source distributed and RESTful search engine
     OpenSearch makes it easy to ingest, search, visualize, and analyze your data
     For more information, see: https://opensearch.org/
 License: Apache-2.0

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 8, 2023

Since Ubuntu will point the default /bin/sh to dash instead of the normal bourne shell, I will have to force everything on bash so some of the scripts like the install demo config will not fail due to it is written in bash build-ins.

$ ll /usr/bin/sh
lrwxrwxrwx 1 root root 4 Dec 12 21:39 /usr/bin/sh -> dash*
$ ll /bin/sh
lrwxrwxrwx 1 root root 4 Dec 12 21:39 /bin/sh -> dash*
$ sh /usr/share/opensearch/plugins/opensearch-security/tools/install_demo_configuration.sh -y -i -s
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
/usr/share/opensearch/plugins/opensearch-security/tools/install_demo_configuration.sh: 9: Bad substitution

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 8, 2023

OS online:

$ sudo dpkg -i opensearch-2.4.1-linux-x64.deb
Selecting previously unselected package opensearch.
(Reading database ... 146329 files and directories currently installed.)
Preparing to unpack opensearch-2.4.1-linux-x64.deb ...
Running preinst script
Unpacking opensearch (2.4.1) ...
Setting up opensearch (2.4.1) ...
Running postinst script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for systemd (245.4-4ubuntu3.19) ...

$ cat /var/log/opensearch/install_demo_configuration.log
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
This script maybe require your root password for 'sudo' privileges
OpenSearch install type: rpm/deb on DISTRIB_ID=Ubuntu
OpenSearch config dir: /etc/opensearch
OpenSearch config file: /etc/opensearch/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.4.1
Detected OpenSearch Security Version: 2.4.1.0

### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../../../config/opensearch-security and execute:
sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/etc/opensearch/opensearch-security" -icl -key "/etc/opensearch/kirk-key.pem" -cert "/etc/opensearch/kirk.pem" -cacert "/etc/opensearch/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)

$ sudo systemctl start opensearch.service
$ sudo systemctl status opensearch.service
● opensearch.service - OpenSearch
     Loaded: loaded (/lib/systemd/system/opensearch.service; disabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-01-08 06:04:54 UTC; 20s ago
       Docs: https://opensearch.org/
   Main PID: 19546 (java)
      Tasks: 71 (limit: 18830)
     Memory: 1.2G
     CGroup: /system.slice/opensearch.service
             └─19546 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMe>

Jan 08 06:04:40 ip systemd[1]: Starting OpenSearch...
Jan 08 06:04:42 ip systemd-entrypoint[19546]: WARNING: A terminally deprecated method in java.lang.System has been called
Jan 08 06:04:42 ip systemd-entrypoint[19546]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.4.1.jar)
Jan 08 06:04:42 ip systemd-entrypoint[19546]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jan 08 06:04:42 ip systemd-entrypoint[19546]: WARNING: System::setSecurityManager will be removed in a future release
Jan 08 06:04:43 ip systemd-entrypoint[19546]: WARNING: A terminally deprecated method in java.lang.System has been called
Jan 08 06:04:43 ip systemd-entrypoint[19546]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.4.1.jar)
Jan 08 06:04:43 ip systemd-entrypoint[19546]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jan 08 06:04:43 ip systemd-entrypoint[19546]: WARNING: System::setSecurityManager will be removed in a future release
Jan 08 06:04:54 ip systemd[1]: Started OpenSearch.

$ curl https://localhost:9200 -u admin:admin --insecure
{
  "name" : "ip",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "M2nPp2cyS9edB6bpRlQYxg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.4.1",
    "build_type" : "deb",
    "build_hash" : "f2f809ea280ffba217451da894a5899f1cec02ab",
    "build_date" : "2023-01-06T21:08:43.134090306Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
$ curl https://localhost:9200/_cat/plugins -u admin:admin --insecure
ip-172-31-11-175 opensearch-alerting                  2.4.1.0
ip-172-31-11-175 opensearch-anomaly-detection         2.4.1.0
ip-172-31-11-175 opensearch-asynchronous-search       2.4.1.0
ip-172-31-11-175 opensearch-cross-cluster-replication 2.4.1.0
ip-172-31-11-175 opensearch-geospatial                2.4.1.0
ip-172-31-11-175 opensearch-index-management          2.4.1.0
ip-172-31-11-175 opensearch-job-scheduler             2.4.1.0
ip-172-31-11-175 opensearch-knn                       2.4.1.0
ip-172-31-11-175 opensearch-ml                        2.4.1.0
ip-172-31-11-175 opensearch-neural-search             2.4.1.0
ip-172-31-11-175 opensearch-notifications             2.4.1.0
ip-172-31-11-175 opensearch-notifications-core        2.4.1.0
ip-172-31-11-175 opensearch-observability             2.4.1.0
ip-172-31-11-175 opensearch-performance-analyzer      2.4.1.0
ip-172-31-11-175 opensearch-reports-scheduler         2.4.1.0
ip-172-31-11-175 opensearch-security                  2.4.1.0
ip-172-31-11-175 opensearch-security-analytics        2.4.1.0
ip-172-31-11-175 opensearch-sql                       2.4.1.0
$ curl https://localhost:9200/_cluster/health?pretty -u admin:admin --insecure
{
  "cluster_name" : "opensearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

@peterzhuamazon
Copy link
Member

PA test, also test post script to stop OS and PA services before uninstall:

$ sudo systemctl status opensearch-performance-analyzer.service
● opensearch-performance-analyzer.service - OpenSearch Performance Analyzer
     Loaded: loaded (/lib/systemd/system/opensearch-performance-analyzer.service; disabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-01-08 06:11:12 UTC; 6s ago
   Main PID: 20300 (java)
      Tasks: 24 (limit: 18830)
     Memory: 108.0M
     CGroup: /system.slice/opensearch-performance-analyzer.service
             └─20300 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Xms4m -Xmx64m -XX:+UseSerialGC -Dlog4j.configurationFile=/etc/opensearch/opensearch-performance-analyzer/log4j2.xml -Xms64M -Xmx64M -XX:+UseSerialGC -XX:CICompilerCount=1 -XX:-TieredCompilation -XX:InitialCodeCacheSize=4096 -XX:MaxRAM=400m -Dope>

Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: Jan 08, 2023 6:11:17 AM org.jooq.tools.JooqLogger info
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: Jan 08, 2023 6:11:17 AM org.jooq.tools.JooqLogger info
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: Jan 08, 2023 6:11:17 AM org.jooq.tools.JooqLogger info
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: Jan 08, 2023 6:11:17 AM org.jooq.tools.JooqLogger info
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: Jan 08, 2023 6:11:17 AM org.jooq.tools.JooqLogger info
Jan 08 06:11:17 ip performance-analyzer-agent-cli[20300]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
$ sudo dpkg -r opensearch
(Reading database ... 147663 files and directories currently installed.)
Removing opensearch (2.4.1) ...
Running prerm script
Stop existing opensearch.service
Stop existing opensearch-performance-analyzer.service
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 8, 2023

OSD online:

$ sudo dpkg -I opensearch-dashboards-2.4.1-linux-x64.deb
 new Debian package, version 2.0.
 size 333229730 bytes: control archive=2200231 bytes.
     162 bytes,     4 lines      conffiles
     423 bytes,    12 lines      control
 12169994 bytes, 87732 lines      md5sums
    1484 bytes,    46 lines   *  postinst             #!/bin/bash
     943 bytes,    27 lines   *  preinst              #!/bin/bash
     605 bytes,    22 lines   *  prerm                #!/bin/bash
      45 bytes,     1 lines      shlibs
      73 bytes,     2 lines      triggers
 Package: opensearch-dashboards
 Version: 2.4.1
 Architecture: amd64
 Maintainer: OpenSearch Team <[email protected]>
 Installed-Size: 1179197
 Section: web
 Priority: optional
 Homepage: https://opensearch.org/
 Description: Open source visualization dashboards for OpenSearch
     OpenSearch Dashboards is the visualization tool for data in OpenSearch
     For more information, see: https://opensearch.org/
 License: Apache-2.0

$ sudo dpkg -i opensearch-dashboards-2.4.1-linux-x64.deb
Selecting previously unselected package opensearch-dashboards.
(Reading database ... 147998 files and directories currently installed.)
Preparing to unpack opensearch-dashboards-2.4.1-linux-x64.deb ...
Running OpenSearch-Dashboards Pre-Installation Script
Unpacking opensearch-dashboards (2.4.1) ...
Setting up opensearch-dashboards (2.4.1) ...
Running OpenSearch-Dashboards Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch-dashboards service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch-dashboards.service
### You can start opensearch-dashboards service by executing
 sudo systemctl start opensearch-dashboards.service
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for systemd (245.4-4ubuntu3.19) ...


$ sudo systemctl restart opensearch-dashboards

$ curl localhost:5601/api/status -u admin:admin
{"name":"ip","uuid":"ab449934-7f03-4093-a23f-9a26f97419d2","version":{"number":"2.4.1","build_hash":"ea36827cdedf1e726e7cb8315ffc49f73f9b4eb7","build_number":1,"build_snapshot":false},"status":{"overall":{"since":"2023-01-08T23:01:14.854Z","state":"green","title":"Green","nickname":"Looking good","icon":"success","uiColor":"secondary"},"statuses":[{"id":"core:[email protected]","message":"OpenSearch is available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"core:[email protected]","message":"SavedObjects service has completed migrations and is available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-01-08T23:01:14.854Z","state":"green","icon":"success","uiColor":"secondary"}]},"metrics":{"last_updated":"2023-01-08T23:01:11.200Z","collection_interval_in_millis":5000,"os":{"platform":"linux","platformRelease":"linux-5.15.0-1026-aws","load":{"1m":0.34,"5m":0.16,"15m":0.06},"memory":{"total_in_bytes":16468430848,"free_in_bytes":8015761408,"used_in_bytes":8452669440},"uptime_in_millis":178125610,"distro":"Ubuntu","distroRelease":"Ubuntu-20.04","cpuacct":{"control_group":"/system.slice/opensearch-dashboards.service","usage_nanos":6621138114},"cpu":{"control_group":"/system.slice/opensearch-dashboards.service","cfs_period_micros":100000,"cfs_quota_micros":-1,"stat":{"number_of_elapsed_periods":0,"number_of_times_throttled":0,"time_throttled_nanos":0}}},"process":{"memory":{"heap":{"total_in_bytes":107233280,"used_in_bytes":99110840,"size_limit":2197815296},"resident_set_size_in_bytes":162652160},"pid":25723,"event_loop_delay":0.6528150141239166,"uptime_in_millis":28527.517938999998},"response_times":{"avg_in_millis":0,"max_in_millis":0},"concurrent_connections":0,"requests":{"disconnects":0,"total":0,"statusCodes":{},"status_codes":{}}}}

@peterzhuamazon
Copy link
Member

More issues:

@BlackMetalz
Copy link

2.5.0, lets goooooooo

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 19, 2023

APT/DEB will release with the 2.5.0 version.
#2908 (comment)

https://opensearch.org/downloads.html

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 24, 2023

We would need to add debian apt automation:
* #3126

@bbarani
Copy link
Member Author

bbarani commented Jan 24, 2023

Added Debian support in 2.5.0 release . We will keep this issue open until we add Debian support for 1.x version.

@bbarani bbarani added the v1.3.9 label Jan 26, 2023
@bbarani bbarani removed the v2.5.0 'Issues and PRs related to version v2.5.0' label Feb 7, 2023
@BlackMetalz
Copy link

BlackMetalz commented Mar 1, 2023

I'm trying to write automation for setup Opensearch cluster.
Opensearch version: 2.6.0
OS: Ubuntu 20.04
The first-time install is successful, after that, I tried to uninstall it. It still works but the second time attempted to install it. The error raised

apt install opensearch=2.6.0
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  opensearch
0 upgraded, 1 newly installed, 0 to remove and 391 not upgraded.
Need to get 0 B/744 MB of archives.
After this operation, 971 MB of additional disk space will be used.
Selecting previously unselected package opensearch.
(Reading database ... 153685 files and directories currently installed.)
Preparing to unpack .../opensearch_2.6.0_amd64.deb ...
Running OpenSearch Pre-Installation Script##############........................................................................................................................................................] 
Unpacking opensearch (2.6.0) ...
Setting up opensearch (2.6.0) ...#############################################################..................................................................................................................] 
Running OpenSearch Post-Installation Script#########################################################################################............................................................................] 
dpkg: error processing package opensearch (--configure):
 installed opensearch package post-installation script subprocess returned error exit status 255
Processing triggers for systemd (245.4-4ubuntu3.3) ...
Processing triggers for libc-bin (2.31-0ubuntu9) ...
Errors were encountered while processing:
 opensearch
E: Sub-process /usr/bin/dpkg returned an error code (1)

@etgraylog
Copy link
Contributor

etgraylog commented Mar 1, 2023

Hi @BlackMetalz! It appears execution is failing on line 24 of the Post-Installation Script. Reproducing issue after adding set -x to /var/lib/dpkg/info/opensearch.postinit (on line 4?) may help you confirm this (Note: 1st completely uninstall opensearch before modification & subsequent execution). The contents of file /var/log/opensearch/install_demo_configuration.log might contain hints on what is causing error as well. To track investigation of this problem, please create Github issue. Thank you!

@BlackMetalz
Copy link

Hi, after your comment, I took a look deeper. Problem resolve with:

  1. Purge the package, not remove
  2. Delete all folders to make we left nothing behind:
product_dir=/usr/share/opensearch
config_dir=/etc/opensearch
data_dir=/var/lib/opensearch
log_dir=/var/log/opensearch
pid_dir=/var/run/opensearch

After that, I can run state ( Saltstack ) for install / uninstall multiple times for testing

@peterzhuamazon
Copy link
Member

Hi,

Me and @zelinh starting to check the 1.x branch support on DEB.

We will test the following scenarios on 1.3.8 version:

x64: OS, OSD
arm64: OS OSD

Thanks,
Peter

@zelinh
Copy link
Member

zelinh commented Mar 6, 2023

Current building/assembling status for 1.3.8:
for x64: OS & OSD are building & assembling successfully.
for arm64: OS & OSD are building & assembling successfully.

@zelinh
Copy link
Member

zelinh commented Mar 7, 2023

@peterzhuamazon and I will verify the installations/executions of above debian packages tomorrow.

@peterzhuamazon
Copy link
Member

Thanks @zelinh.

We will verify the installation, api calling, even native plugins installation tomorrow to make sure the 1.3.8 packages runs, before trying the process on upcoming 1.3.9 version.

Thanks.

@peterzhuamazon
Copy link
Member

Test OS:

{
  "name" : "ip-172-31-11-119",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "SDEDaK2vQNuVSduQ_rYRdw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.3.8",
    "build_type" : "deb",
    "build_hash" : "2ac1c13e3877692bc9cbbab4c796d7244fe8486a",
    "build_date" : "2023-03-06T22:22:50.498038Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

OSD:


ubuntu@ip-172-31-11-119:~$ curl localhost:5601/api/status -u admin:admin --insecure
{"name":"ip-172-31-11-119","uuid":"a1b51042-002b-4c97-a3ce-92dcf330b49c","version":{"number":"1.3.8","build_hash":"55783c976d503fc1de2611d1c56b967d453bbf27","build_number":1,"build_snapshot":false},"status":{"overall":{"since":"2023-03-07T23:40:08.911Z","state":"green","title":"Green","nickname":"Looking good","icon":"success","uiColor":"secondary"},"statuses":[{"id":"core:[email protected]","message":"OpenSearch is available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"core:[email protected]","message":"SavedObjects service has completed migrations and is available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2023-03-07T23:40:08.911Z","state":"green","icon":"success","uiColor":"secondary"}]},"metrics":{"last_updated":"2023-03-07T23:40:04.173Z","collection_interval_in_millis":5000,"os":{"platform":"linux","platformRelease":"linux-5.15.0-1028-aws","load":{"1m":0.40478515625,"5m":0.1201171875,"15m":0.041015625},"memory":{"total_in_bytes":16377524224,"free_in_bytes":11480748032,"used_in_bytes":4896776192},"uptime_in_millis":12018000,"distro":"Ubuntu","distroRelease":"Ubuntu-20.04","cpuacct":{"control_group":"/system.slice/opensearch-dashboards.service","usage_nanos":4358022928},"cpu":{"control_group":"/system.slice/opensearch-dashboards.service","cfs_period_micros":100000,"cfs_quota_micros":-1,"stat":{"number_of_elapsed_periods":0,"number_of_times_throttled":0,"time_throttled_nanos":0}}},"process":{"memory":{"heap":{"total_in_bytes":124121088,"used_in_bytes":96783264,"size_limit":1526909922},"resident_set_size_in_bytes":165933056},"pid":5864,"event_loop_delay":0.3935230001807213,"uptime_in_millis":17644},"response_times":{"avg_in_millis":0,"max_in_millis":0},"concurrent_connections":0,"requests":{"disconnects":0,"total":0,"statusCodes":{},"status_codes":{}}}}

@bbarani
Copy link
Member Author

bbarani commented Mar 23, 2023

Closing this issue as we have added support in both 2.5.0 and 1.3.9 versions

@bbarani bbarani closed this as completed Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deb distributions enhancement New Enhancement roadmap triaged This issue has been reviewed by the triage team v1.3.9
Projects
Development

No branches or pull requests