Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Databases without any existing migrations will fail when updating to v7.9.7 #5306

Open
grantfitzsimmons opened this issue Oct 3, 2024 · 0 comments
Assignees
Labels
1 - Bug Incorrect behavior of the product pri:high Must be resolved before releasing regression This is behavior that once worked that has broken. Must be resolved before the next release.
Milestone

Comments

@grantfitzsimmons
Copy link
Member

Describe the bug
When attempting to run all of the migrations upon first run, all databases receive this error:

`MySQLdb.OperationalError: (1054, "Unknown column 'collection.CollectionObjectTypeID' in 'field list'")`

This is due to the 0004_intialize_defaults.py migration failing. If you remove new fields that are unexpected, it succeeds, and you can add it back and compose up.

I removed this line from the models file and it succeeded:

collectionobjecttype = models.ForeignKey('CollectionObjectType', db_column='CollectionObjectTypeID', related_name='collections', null=True, on_delete=models.SET_NULL)

We need to fix incorrect assumptions made by the migration that lead to this update being non-feasible.

To Reproduce
Steps to reproduce the behavior:

  1. Create a new database using the Specify Wizard in 6
  2. Configure your .env file to point to this database
  3. Run docker compose up --build after pulling the latest version of production
  4. See failure

Expected behavior

On a brand new Specify 6 database, the migration fails every time. If you remove all migrations, it will fail as well.

MySQLdb.OperationalError: (1054, "Unknown column 'collection.CollectionObjectTypeID' in 'field list'")

WARN[0009] Found orphan containers ([specify7-mariadb-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 8/8
 ✔ Container specify7-report-runner-1    Created                                                                                        0.0s 
 ✔ Container specify7-redis-1            Created                                                                                        0.0s 
 ✔ Container specify7-specify6-1         Created                                                                                        0.0s 
 ✔ Container specify7-asset-server-1     Created                                                                                        0.0s 
 ✔ Container specify7-webpack-1          Recreated                                                                                      0.1s 
 ✔ Container specify7-nginx-1            Created                                                                                        0.0s 
 ✔ Container specify7-specify7-worker-1  Recreated                                                                                      0.1s 
 ✔ Container specify7-specify7-1         Recreated                                                                                      0.1s 
Attaching to asset-server-1, nginx-1, redis-1, report-runner-1, specify6-1, specify7-1, specify7-worker-1, webpack-1
report-runner-1    | rosetta error: Rosetta is only intended to run on Apple Silicon with a macOS host using Virtualization.framework with Rosetta mode enabled
redis-1            | 1:C 03 Oct 2024 14:10:24.787 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-1            | 1:C 03 Oct 2024 14:10:24.787 * Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis-1            | 1:C 03 Oct 2024 14:10:24.787 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis-1            | 1:M 03 Oct 2024 14:10:24.787 * monotonic clock: POSIX clock_gettime
redis-1            | 1:M 03 Oct 2024 14:10:24.788 * Running mode=standalone, port=6379.
redis-1            | 1:M 03 Oct 2024 14:10:24.788 * Server initialized
redis-1            | 1:M 03 Oct 2024 14:10:24.789 * Loading RDB produced by version 7.4.0
redis-1            | 1:M 03 Oct 2024 14:10:24.789 * RDB age 2952 seconds
redis-1            | 1:M 03 Oct 2024 14:10:24.789 * RDB memory usage when created 1.40 Mb
redis-1            | 1:M 03 Oct 2024 14:10:24.789 * Done loading RDB, keys loaded: 6, keys expired: 0.
redis-1            | 1:M 03 Oct 2024 14:10:24.789 * DB loaded from disk: 0.001 seconds
redis-1            | 1:M 03 Oct 2024 14:10:24.789 * Ready to accept connections tcp
report-runner-1    |  
nginx-1            | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx-1            | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx-1            | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx-1            | 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
nginx-1            | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
nginx-1            | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-1            | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx-1            | /docker-entrypoint.sh: Configuration complete; ready for start up
asset-server-1     | rosetta error: Rosetta is only intended to run on Apple Silicon with a macOS host using Virtualization.framework with Rosetta mode enabled
asset-server-1     |  
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: using the "epoll" event method
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: nginx/1.27.1
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: built by gcc 13.2.1 20240309 (Alpine 13.2.1_git20240309) 
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: OS: Linux 6.6.26-linuxkit
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker processes
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 21
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 22
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 23
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 24
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 25
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 26
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 27
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 28
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 29
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 30
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 31
nginx-1            | 2024/10/03 14:10:24 [notice] 1#1: start worker process 32
specify6-1         | Successfully updated Specify 6 files\n
specify6-1         | This container is only needed during start up to give Specify 7 information about the data model\n
specify6-1         | It will now exit
report-runner-1 exited with code 133
asset-server-1 exited with code 133
specify6-1 exited with code 0
specify7-1         | Updating static files in /volumes/static-files/.
specify7-1         | Applying Django migrations.
webpack-1          | Browserslist: caniuse-lite is outdated. Please run:
webpack-1          |   npx update-browserslist-db@latest
webpack-1          |   Why you should do it regularly: https://github.com/browserslist/update-db#readme
nginx-1            | 2024/10/03 14:10:26 [error] 22#22: *2 asset-server could not be resolved (3: Host not found), client: 192.168.65.1, server: localhost, request: "GET /web_asset_store.xml HTTP/1.1", host: "host.docker.internal"
nginx-1            | 2024/10/03 14:10:26 [error] 21#21: *1 asset-server could not be resolved (3: Host not found), client: 192.168.65.1, server: localhost, request: "GET /web_asset_store.xml HTTP/1.1", host: "host.docker.internal"
nginx-1            | 192.168.65.1 - - [03/Oct/2024:14:10:26 +0000] "GET /web_asset_store.xml HTTP/1.1" 502 157 "-" "python-requests/2.32.2" "-"
nginx-1            | 192.168.65.1 - - [03/Oct/2024:14:10:26 +0000] "GET /web_asset_store.xml HTTP/1.1" 502 157 "-" "python-requests/2.32.2" "-"
specify7-1         | [03/Oct/2024 14:10:26] [ERROR] [specifyweb.attachment_gw.views:220] Failed fetching asset server configuration
specify7-worker-1  | [03/Oct/2024 14:10:26] [ERROR] [specifyweb.attachment_gw.views:220] Failed fetching asset server configuration
specify7-worker-1  |  
specify7-worker-1  |  -------------- celery@853599ed2a94 v5.2.7 (dawn-chorus)
specify7-worker-1  | --- ***** ----- 
specify7-worker-1  | -- ******* ---- Linux-6.6.26-linuxkit-aarch64-with-glibc2.27 2024-10-03 14:10:26
specify7-worker-1  | - *** --- * --- 
specify7-worker-1  | - ** ---------- [config]
specify7-worker-1  | - ** ---------- .> app:         specify7:0xffffb469e430
specify7-worker-1  | - ** ---------- .> transport:   redis://redis:6379/0
specify7-worker-1  | - ** ---------- .> results:     redis://redis/1
specify7-worker-1  | - *** --- * --- .> concurrency: 1 (prefork)
specify7-worker-1  | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
specify7-worker-1  | --- ***** ----- 
specify7-worker-1  |  -------------- [queues]
specify7-worker-1  |                 .> geospecify       exchange=geospecify(direct) key=geospecify
specify7-worker-1  |                 
specify7-worker-1  | 
specify7-worker-1  | [tasks]
specify7-worker-1  |   . specifyweb.celery_tasks.debug_task
specify7-worker-1  |   . specifyweb.specify.record_merging.record_merge_task
specify7-worker-1  |   . specifyweb.specify.update_locality.parse_locality_task
specify7-worker-1  |   . specifyweb.specify.update_locality.update_locality_task
specify7-worker-1  |   . specifyweb.workbench.tasks.unupload
specify7-worker-1  |   . specifyweb.workbench.tasks.upload
specify7-worker-1  | 
specify7-worker-1  | [2024-10-03 14:10:26,709: INFO/MainProcess] Connected to redis://redis:6379/0
nginx-1            | 2024/10/03 14:10:27 [error] 23#23: *5 asset-server could not be resolved (3: Host not found), client: 192.168.65.1, server: localhost, request: "GET /web_asset_store.xml HTTP/1.1", host: "host.docker.internal"
nginx-1            | 192.168.65.1 - - [03/Oct/2024:14:10:27 +0000] "GET /web_asset_store.xml HTTP/1.1" 502 157 "-" "python-requests/2.32.2" "-"
specify7-1         | [03/Oct/2024 14:10:27] [ERROR] [specifyweb.attachment_gw.views:220] Failed fetching asset server configuration
specify7-1         | Operations to perform:
specify7-1         |   Apply all migrations: accounts, attachment_gw, auth, businessrules, contenttypes, notifications, permissions, sessions, specify, workbench
specify7-1         | Running migrations:
specify7-1         |   Applying accounts.0001_initial... OK
specify7-1         |   Applying accounts.0002_auto_20211223_1206... OK
specify7-1         |   Applying accounts.0003_auto_20220621_1541... OK
specify7-1         |   Applying attachment_gw.0001_initial... OK
specify7-1         |   Applying contenttypes.0001_initial... OK
specify7-1         |   Applying contenttypes.0002_remove_content_type_name... OK
specify7-1         |   Applying auth.0001_initial... OK
specify7-1         |   Applying auth.0002_alter_permission_name_max_length... OK
specify7-1         |   Applying auth.0003_alter_user_email_max_length... OK
specify7-1         |   Applying auth.0004_alter_user_username_opts... OK
specify7-1         |   Applying auth.0005_alter_user_last_login_null... OK
specify7-1         |   Applying auth.0006_require_contenttypes_0002... OK
specify7-1         |   Applying auth.0007_alter_validators_add_error_messages... OK
specify7-1         |   Applying auth.0008_alter_user_username_max_length... OK
specify7-1         |   Applying auth.0009_alter_user_last_name_max_length... OK
specify7-1         |   Applying auth.0010_alter_group_name_max_length... OK
specify7-1         |   Applying auth.0011_update_proxy_permissions... OK
specify7-1         |   Applying auth.0012_alter_user_first_name_max_length... OK
specify7-worker-1  | [2024-10-03 14:10:31,718: INFO/MainProcess] mingle: searching for neighbors
specify7-1         |   Applying businessrules.0001_initial... OK
specify7-worker-1  | [2024-10-03 14:10:32,722: INFO/MainProcess] mingle: all alone
specify7-worker-1  | [2024-10-03 14:10:32,728: WARNING/MainProcess] /opt/specify7/ve/lib/python3.8/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
specify7-worker-1  |             leak, never use this setting in production environments!
specify7-worker-1  |   warnings.warn('''Using settings.DEBUG leads to a memory
specify7-worker-1  | 
specify7-worker-1  | [2024-10-03 14:10:32,728: INFO/MainProcess] celery@853599ed2a94 ready.
specify7-1         |   Applying businessrules.0002_default_unique_rules... OK
specify7-1         |   Applying permissions.0001_initial... OK
specify7-1         |   Applying permissions.0002_role_rolepolicy_userrole... OK
specify7-1         |   Applying permissions.0003_auto_20220321_1445... OK
specify7-1         | Traceback (most recent call last):
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
specify7-1         |     return self.cursor.execute(sql, params)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
specify7-1         |     return self.cursor.execute(query, args)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
specify7-1         |     res = self._query(query)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
specify7-1         |     db.query(q)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
specify7-1         |     _mysql.connection.query(self, query)
specify7-1         | MySQLdb.OperationalError: (1054, "Unknown column 'collection.CollectionObjectTypeID' in 'field list'")
specify7-1         | 
specify7-1         | The above exception was the direct cause of the following exception:
specify7-1         | 
specify7-1         | Traceback (most recent call last):
specify7-1         |   File "manage.py", line 25, in <module>
specify7-1         |     execute_from_command_line(sys.argv)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
specify7-1         |     utility.execute()
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
specify7-1         |     self.fetch_command(subcommand).run_from_argv(self.argv)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
specify7-1         |     self.execute(*args, **cmd_options)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
specify7-1         |     output = self.handle(*args, **options)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
specify7-1         |     res = handle_func(*args, **kwargs)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
specify7-1         |     post_migrate_state = executor.migrate(
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
specify7-1         |     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
specify7-1         |     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
specify7-1         |     state = migration.apply(state, schema_editor)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/migrations/migration.py", line 123, in apply
specify7-1         |     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
specify7-1         |     self.code(from_state.apps, schema_editor)
specify7-1         |   File "/opt/specify7/specifyweb/permissions/migrations/0004_intialize_defaults.py", line 7, in initialize
specify7-1         |     init(wipe=True)
specify7-1         |   File "/opt/specify7/specifyweb/permissions/initialize.py", line 24, in initialize
specify7-1         |     assign_users_to_roles()
specify7-1         |   File "/opt/specify7/specifyweb/permissions/initialize.py", line 46, in assign_users_to_roles
specify7-1         |     for collection in Collection.objects.all():
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/models/query.py", line 280, in __iter__
specify7-1         |     self._fetch_all()
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
specify7-1         |     self._result_cache = list(self._iterable_class(self))
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
specify7-1         |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
specify7-1         |     cursor.execute(sql, params)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
specify7-1         |     return super().execute(sql, params)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
specify7-1         |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
specify7-1         |     return executor(sql, params, many, context)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
specify7-1         |     return self.cursor.execute(sql, params)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
specify7-1         |     raise dj_exc_value.with_traceback(traceback) from exc_value
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
specify7-1         |     return self.cursor.execute(sql, params)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
specify7-1         |     return self.cursor.execute(query, args)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
specify7-1         |     res = self._query(query)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
specify7-1         |     db.query(q)
specify7-1         |   File "/opt/specify7/ve/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
specify7-1         |     _mysql.connection.query(self, query)
specify7-1         | django.db.utils.OperationalError: (1054, "Unknown column 'collection.CollectionObjectTypeID' in 'field list'")
specify7-1         | Watching for file changes with StatReloader
nginx-1            | 192.168.65.1 - - [03/Oct/2024:14:10:39 +0000] "GET /web_asset_store.xml HTTP/1.1" 502 157 "-" "python-requests/2.32.2" "-"
nginx-1            | 2024/10/03 14:10:39 [error] 24#24: *7 asset-server could not be resolved (3: Host not found), client: 192.168.65.1, server: localhost, request: "GET /web_asset_store.xml HTTP/1.1", host: "host.docker.internal"
specify7-1         | [03/Oct/2024 14:10:39] [ERROR] [specifyweb.attachment_gw.views:220] Failed fetching asset server configuration
webpack-1          | assets by status 53.3 MiB [cached] 89 assets
webpack-1          | asset manifest.json 15.7 KiB [compared for emit]
webpack-1          | orphan modules 77.6 KiB [orphan] 106 modules
webpack-1          | runtime modules 17.5 KiB 23 modules
webpack-1          | modules by path ./node_modules/ 12.1 MiB 1810 modules
webpack-1          | modules by path ./lib/ 3.36 MiB 638 modules
webpack-1          | modules by path ./css/*.css 149 KiB 4 modules
webpack-1          | modules by mime type image/svg+xml 1.02 KiB
webpack-1          |   data:image/svg+xml,%3csvg xmlns=%27.. 284 bytes [built] [code generated]
webpack-1          |   + 3 modules
webpack-1          | modules by mime type image/png 1.08 KiB
webpack-1          |   data:image/png;base64,iVBORw0KGgoAAAAN.. 329 bytes [built] [code generated]
webpack-1          |   + 3 modules
webpack-1          | fs (ignored) 15 bytes [built] [code generated]
webpack-1          | crypto (ignored) 15 bytes [optional] [built] [code generated]
webpack-1          | stream (ignored) 15 bytes [built] [code generated]
webpack-1          | webpack 5.73.0 compiled successfully in 13389 ms

Crash Report

Here's a sample database that you can download and import into MariaDB:
geospecify_2024_10_03.sql.zip

It has 3 disciplines and collections, and has never been used in Specify 7.

@grantfitzsimmons grantfitzsimmons added pri:high Must be resolved before releasing 1 - Bug Incorrect behavior of the product regression This is behavior that once worked that has broken. Must be resolved before the next release. labels Oct 3, 2024
@grantfitzsimmons grantfitzsimmons added this to the 7.9.8 milestone Oct 4, 2024
@CarolineDenis CarolineDenis modified the milestones: 7.9.8, 7.9.11 Oct 17, 2024
@grantfitzsimmons grantfitzsimmons changed the title Databases without any existing migrations will fail when updating to the latest version of Specify 7 Databases without any existing migrations will fail when updating to v7.9.7 version of Specify 7 Oct 17, 2024
@grantfitzsimmons grantfitzsimmons changed the title Databases without any existing migrations will fail when updating to v7.9.7 version of Specify 7 Databases without any existing migrations will fail when updating to v7.9.7 version of Specify 7 Oct 17, 2024
@grantfitzsimmons grantfitzsimmons changed the title Databases without any existing migrations will fail when updating to v7.9.7 version of Specify 7 Databases without any existing migrations will fail when updating to v7.9.7 Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1 - Bug Incorrect behavior of the product pri:high Must be resolved before releasing regression This is behavior that once worked that has broken. Must be resolved before the next release.
Projects
None yet
Development

No branches or pull requests

3 participants