diff --git a/config/redirects.json b/config/redirects.json index 25be181ec..c62e9c428 100644 --- a/config/redirects.json +++ b/config/redirects.json @@ -498,5 +498,15 @@ "source": "/docs/learn/contributing/contributing_docs", "destination": "/docs/learn/resources/contributing_docs", "permanent": true + }, + { + "source": "/docs/guides/database/meilisync_mysql", + "destination": "https://github.com/long2ice/meilisync", + "permanent": true + }, + { + "source": "/docs/guides/database/meilisync_postgresql", + "destination": "https://github.com/long2ice/meilisync", + "permanent": true } ] diff --git a/config/sidebar-guides.json b/config/sidebar-guides.json index 8ad5c8a3e..b9458d33f 100644 --- a/config/sidebar-guides.json +++ b/config/sidebar-guides.json @@ -124,22 +124,6 @@ } ] }, - { - "title": "Database", - "slug": "database", - "routes": [ - { - "source": "guides/database/meilisync_mysql.mdx", - "label": "Syncing databases with MySQL and meilisync", - "slug": "meilisync_mysql" - }, - { - "source": "guides/database/meilisync_postgresql.mdx", - "label": "Syncing databases with PostgreSQL and meilisync", - "slug": "meilisync_postgresql" - } - ] - }, { "title": "Deployment", "slug": "deployment", diff --git a/guides/database/meilisync_mysql.mdx b/guides/database/meilisync_mysql.mdx deleted file mode 100644 index 6708810e0..000000000 --- a/guides/database/meilisync_mysql.mdx +++ /dev/null @@ -1,200 +0,0 @@ ---- -title: Syncing databases with MySQL and meilisync — Meilisearch documentation -description: Learn how to keep Meilisearch up to date with your MySQL database. ---- - -# Syncing databases with MySQL and meilisync - -Though Meilisearch is a database, it is not recommended you use it as your primary data store. Instead, you should use an external database to store your data and periodically synchronize it with Meilisearch. - -This guide teaches you to use [`meilisync`](https://github.com/long2ice/meilisync) to keep Meilisearch up to date with a MySQL database. **Please be advised that meilisync is in its early developmental phase and is NOT PRODUCTION READY yet. You may encounter issues while interacting with it.** - -## Requirements - -- A command-line console, such as macOS's terminal or Window's Cygwin -- A MySQL database populated with data -- A Meilisearch instance, either managed via [Meilisearch Cloud](https://meilisearch.com/cloud?utm_campaign=oss&utm_source=docs&utm_medium=meilisync) or self-hosted - -## Install `meilisync` - -Install `meilisync` to the same location as your primary database. You can do this manually or with Docker: - - - - -### Docker - -First, create a `docker-compose.yml` file in your Docker project directory. Open it with a text editor and add the following content: - -```yaml -services: - meilisync: - platform: linux/x86_64 - image: long2ice/meilisync - volumes: - - ./config.yml:/meilisync/config.yml -``` - -If your project already has a `docker-compose.yml`, add the `meilisync` settings to the existing `services` field. - -When you are done, open your console, navigate to your Docker project directory, and run the following command: - -```sh -docker compose pull -``` - -After a few seconds, Docker will inform you it has successfully pulled `meilisync`'s image. - - - - - -### Manual install - -First, make sure you have Python 3.9 and `pip` installed. - -Then, open your command-line console and run the following command to install `meilisync`: - -```sh -pip install meilisync -``` - -`pip` will install `meilisync`. Once it is done, run the following command to verify your installation: - -```sh -meilisync --help -``` - -If the install process was successful, you should see a brief description of `meilisync`'s options. - - - - -## Connect `meilisync` to Meilisearch - -With `meilisync` installed, you now need to connect it to your Meilisearch instance. - -Create a `config.yml` file in your project's directory. Open it with a text editor and add the following configuration options depending on whether you are using Docker or not: - - - - -```yaml -meilisearch: - api_url: http://host.docker.internal:7700/ - api_key: 'MEILISEARCH_API_KEY' - insert_size: 1000 - insert_interval: 10 -``` - - - - - -```yaml -meilisearch: - api_url: MEILISEARCH_API_URL - api_key: 'MEILISEARCH_API_KEY' - insert_size: 1000 - insert_interval: 10 -``` - - - - -Replace `MEILISEARCH_API_KEY` with an API key able to create, update, and delete documents and indexes. Replace `MEILISEARCH_API_URL` with your instance's API URL. If you are using [Meilisearch Cloud](https://cloud.meilisearch.com?utm_campaign=oss&utm_source=docs&utm_medium=meilisync), this URL should look similar to this: `https://ms-4d85L33tC0d3-5041.fra.meilisearch.io`. - -## Configure MySQL - -After connecting `meilisync` to Meilisearch, you need to connect it to your MySQL database. - -### Binary log format - -To use `meilisync` with MySQL, you must set your binary log format to `ROW`. - -Launch your MySQL server using the `--binlog-format` command-line option: - -```sh -mysql --binlog-format=ROW -``` - -Alternatively, set the binary log format during runtime by running the following command in your MySQL server console: - -```sql -SET GLOBAL binlog_format = 'ROW'; -``` - -### Connect `meilisync` and MySQL - -Open `meilisync`'s `config.yml` once again and append the following settings to the bottom of the file: - -```yaml -source: - type: mysql - host: 127.0.0.1 # assuming your MySQL server is running on the same machine as `meilisync` - port: 3306 - database: MYSQL_DATABASE_NAME - user: MYSQL_USERNAME - password: MYSQL_PASSWORD -``` - -Replace MYSQL_DATABASE_NAME, MYSQL_USERNAME, and MYSQL_PASSWORD with the name of your database, a user with read access to that database, and its credentials. - -## Configure `meilisync` - -The last configuration step is specifying which tables in your database should be synced to which Meilisearch indexes. - -Open `config.yml` and append the following settings to the bottom of the file: - -```yaml -sync: - - table: MYSQL_TABLE_NAME_1 - index: MEILISEARCH_INDEX_NAME_1 - pk: id # Read the "Primary key inference" section below - full: true - - table: MYSQL_TABLE_NAME_2 - index: MEILISEARCH_INDEX_NAME_2 - full: true -``` - -Replace `MYSQL_TABLE_NAME_1` and `MEILISEARCH_INDEX_NAME_1` with the names of your tables and indexes. As the above example demonstrates, you may sync multiple tables. - -### Primary key inference - -If your table contains multiple columns with `id` in their name, `meilisync` will not be able to infer which column is the primary key. In this case, you must specify the primary key manually by adding a `pk` field to the table's configuration. - - -This tutorial describes a fairly basic `meilisync` setup. For more customization options, consult [`meilisync`'s documentation](https://github.com/long2ice/meilisync#configuration). - - -## Run `meilisync` - -With the configuration done, open your command-line prompt once again and run `meilisync` with Docker or your manual install: - - - - -```sh -docker compose up -``` - - - - - -```sh -meilisync start -``` - - - - -`meilisync` should immediately start syncing your primary MySQL database with your Meilisearch instance. - -Once `meilisync` is done, use Meilisearch's search preview to perform a few test searches and confirm data has been successfully synced between your primary database and Meilisearch. - -## Conclusion - -Congratulations—you have successfully set up `meilisync`. - -You may leave the process running permanently on the background so all changes to your primary database are automatically synced, or you may interrupt the process and only run it when you update your database again. diff --git a/guides/database/meilisync_postgresql.mdx b/guides/database/meilisync_postgresql.mdx deleted file mode 100644 index dc66d5cc9..000000000 --- a/guides/database/meilisync_postgresql.mdx +++ /dev/null @@ -1,208 +0,0 @@ ---- -title: Syncing databases with PostgreSQL and meilisync — Meilisearch documentation -description: Learn how to keep Meilisearch up to date with your PostgreSQL database. ---- - -# Syncing databases with PostgreSQL and meilisync - -Though Meilisearch is a database, it is not recommended you use it as your primary data store. Instead, you should use an external database to store your data and periodically synchronize it with Meilisearch. - -This guide teaches you to use [`meilisync`](https://github.com/long2ice/meilisync) to keep Meilisearch up to date with a PostgreSQL database. **Please be advised that meilisync is in its early developmental phase and is NOT PRODUCTION READY yet. You may encounter issues while interacting with it.** - -## Requirements - -- A command-line console, such as macOS's terminal or Window's Cygwin -- A PostgreSQL database populated with data -- A Meilisearch instance, either self-hosted or via [Meilisearch Cloud](https://meilisearch.com/cloud?utm_campaign=oss&utm_source=docs&utm_medium=meilisync) - -## Install `meilisync` - -Install `meilisync` to the same location as your primary database. You can do this manually or with Docker: - - - - -### Docker - -First, create a `docker-compose.yml` file in your Docker project directory. Open it with a text editor and add the following content: - -```yaml -services: - meilisync: - platform: linux/x86_64 - image: long2ice/meilisync - volumes: - - ./config.yml:/meilisync/config.yml -``` - -If your project already has a `docker-compose.yml`, add the `meilisync` settings to the existing `services` field. - -When you are done, open your console, navigate to your Docker project directory, and run the following command: - -```sh -docker compose pull -``` - -After a few seconds, Docker will inform you it has successfully pulled `meilisync`'s image. - - - - -### Manual install - -First, make sure you have Python 3.9 and `pip` installed. - -Then, open your command-line console and run the following command to install `meilisync`: - -```sh -pip install meilisync -``` - -`pip` will install `meilisync`. Once it is done, run the following command to verify your installation: - -```sh -meilisync --help -``` - -If the install process was successful, you should see a brief description of `meilisync`'s options. - - - -## Connect `meilisync` to Meilisearch - -With `meilisync` installed, you now need to connect it to your Meilisearch instance. - -Create a `config.yml` file in your project's directory. Open it with a text editor and add the following configuration options depending on whether you are using Docker or not: - - - - -```yaml -meilisearch: - api_url: http://host.docker.internal:7700/ - api_key: 'MEILISEARCH_API_KEY' - insert_size: 1000 - insert_interval: 10 -``` - - - - - -```yaml -meilisearch: - api_url: MEILISEARCH_API_URL - api_key: 'MEILISEARCH_API_KEY' - insert_size: 1000 - insert_interval: 10 -``` - - - - -Replace `MEILISEARCH_API_KEY` with an API key able to create, update, and delete documents and indexes. Replace `MEILISEARCH_API_URL` with your instance's API URL. If you are using [Meilisearch Cloud](https://cloud.meilisearch.com?utm_campaign=oss&utm_source=docs&utm_medium=meilisync), this URL should look similar to this: `https://ms-4d85L33tC0d3-5041.fra.meilisearch.io`. - -## Configure PostgreSQL - -After connecting `meilisync` to Meilisearch, you need to connect it to your PostgreSQL database. - -### Enable logical replication - -To use `meilisync` with PostgreSQL, you need to enable the [`wal2json`](https://github.com/eulerto/wal2json) extension. - -The `wal2json` extension requires the [Write-Ahead Log settings](https://www.postgresql.org/docs/current/runtime-config-wal.html) to be set to `logical`. - -This can be done by adding the following line to your PostgreSQL server's `postgresql.conf` file: - -```conf -wal_level = logical -# -# these parameters only need to set in versions 9.4, 9.5 and 9.6 -# default values are ok in version 10 or later -# -max_replication_slots = 10 -max_wal_senders = 10 -``` - -Alternatively, set the binary log format during runtime by running the following command in your PostgreSQL server console: - -```sql -ALTER SYSTEM SET wal_level = logical; -``` - -This will require a server configuration reload to take effect. Read more about this in the [`ALTER SYSTEM` documentation](https://www.postgresql.org/docs/current/sql-altersystem.html). - -### Connect `meilisync` and PostgreSQL - -Open `meilisync`'s `config.yml` once again and append the following settings to the bottom of the file: - -```yaml -source: - type: postgres - host: 127.0.0.1 # assuming your PostgreSQL server is running on the same machine as `meilisync` - port: 5432 - database: POSTGRESQL_DATABASE_NAME - user: POSTGRESQL_USERNAME - password: POSTGRESQL_PASSWORD -``` - -Replace POSTGRESQL_DATABASE_NAME, POSTGRESQL_USERNAME, and POSTGRESQL_PASSWORD with the name of your database, a user with read access to that database, and its credentials. - -## Configure `meilisync` - -The last configuration step is specifying which tables in your database should be synced to which Meilisearch indexes. - -Open `config.yml` and append the following settings to the bottom of the file: - -```yaml -sync: - - table: POSTGRES_TABLE_NAME_1 - index: MEILISEARCH_INDEX_NAME_1 - pk: id # Read the "Primary key inference" section below - full: true - - table: POSTGRES_TABLE_NAME_2 - index: MEILISEARCH_INDEX_NAME_2 - full: true -``` - -Replace `POSTGRES_TABLE_NAME_1` and `MEILISEARCH_INDEX_NAME_1` with the names of your tables and indexes. As the above example demonstrates, you may sync multiple tables. - -### Primary key inference - -If your table contains multiple columns with `id` in their name, `meilisync` will not be able to infer which column is the primary key. In this case, you must specify the primary key manually by adding a `pk` field to the table's configuration. - - -This tutorial describes a fairly basic `meilisync` setup. For more customization options, consult [`meilisync`'s documentation](https://github.com/long2ice/meilisync#configuration). - - -## Run `meilisync` - -With the configuration done, open your command-line prompt once again and run `meilisync` with Docker or your manual install: - - - - -```sh -docker compose up -``` - - - - - -```sh -meilisync start -``` - - - - -`meilisync` should immediately start syncing your primary PostgreSQL database with your Meilisearch instance. - -Once `meilisync` is done, use Meilisearch's search preview to perform a few test searches and confirm data has been successfully synced between your primary database and Meilisearch. - -## Conclusion - -Congratulations—you have successfully set up `meilisync`. - -You may leave the process running permanently on the background so all changes to your primary database are automatically synced, or you may interrupt the process and only run it when you update your database again.