Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Dify compatible with MySQL database #8364

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

powerfooI
Copy link
Contributor

@powerfooI powerfooI commented Sep 13, 2024

Checklist:

Important

Please review the checklist below before submitting your pull request.

  • Please open an issue before creating a PR or link to an existing issue
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

As issue #8343 said, some organizations are expecting to take mysql as database of dify, while there are some postgresql native usages in dify's definition of models that make seamless switch to mysql difficult.

What this PR does are mainly as following,

  1. Replace some server side postgresql statements like 'some words'::text with dialect irrelevant ones. Concretely, I did the following replacements.
server_default=db.text("uuid_generate_v4()") --> default=lambda: uuid.uuid4()
server_default=db.text("true") --> default=True
server_default=db.text("false") --> default=False
server_default=db.text\("'(.*)'::character varying"\) --> default="$1"
server_default=db.text\("'(.*)'::text"\) --> server_default=db.text("$1")
server_default=db.text("CURRENT_TIMESTAMP(0)") --> server_default=func.current_timestamp()
db.Column(db.Text, ..., server_default="...") --> db.Column(db.Text, ..., default="...")
  1. Ignore direct JSON index: MySQL does not support indexing JSON column directly. (Reference)

  2. Add env vars and docker-compose files for MySQL.

  3. Create a separate migrations folder for mysql database and update upgrade-db command.

This PR resolves #8343 and is related to #415, #3813, #2371.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update, included: Dify Document
  • Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
  • Dependency upgrade

Testing Instructions

Assume a main workflow: User opens the dify webpage, signs up the first account, logs in to the webpage, creates a dataset, uploads several papers, dify embeds the documents, the user does hit tests, creates workflow to retrieve the document segment, dify generates answer for user's question with help of LLM.

I have tested the changes with the main workflow on the following deployment x database matrix and all tests passed.

Deployment Database Test Result
Docker-compose PostgreSQL 15.8 (default)
Docker-compose MySQL 5.7.44
Docker-compose MySQL 8.0.39
Middleware PostgreSQL 15.8 (default)
Middleware MySQL 5.7.44
Middleware MySQL 8.0.39
Middleware OceanBase 4.2.1.7

Migrations Compatibility

This PR takes a snapshot of the old migrations' head and create a new migrations folder to make dify compatible with both postgresql and mysql. For users using dify of previous version, they should do flask db upgrade and flask db upgrade -d migrations_new. For users using dify with mysql database, they only need to do flask db upgrade -d migrations_new. In other word, users using pg and users using mysql will get the identical database schema at the snapshot point.

image

@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. 💪 enhancement New feature or request labels Sep 13, 2024
@laipz8200
Copy link
Collaborator

I think we're not quite ready to maintain two sets of migrations yet.

@powerfooI
Copy link
Contributor Author

Actually we just need to archive the old migrations and create future revisions in new migrations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💪 enhancement New feature or request size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can dify use MySQL as backend database for storing data?
2 participants