Update Schema Config to include new geo fields and COG picklists #5257

sharadsw · 2024-09-03T19:13:05Z

Fixes #5215
Fixes #5289
Fixes #5292
Fixes #5293

NOTE: COG -> cojo has been removed and COG -> childCojos is renamed to COG -> children

This migration updates the data model for COG and Schema Config entries for pre-geo tables and creates picklists for COGTypes.

Data model changes:
Removed COG -> cojo
Added COG -> children

Schema Config changes:
Added StorageTreeDef -> institution
Added COG -> children
Removed CollectionObject -> collectionObjectType (duplicate)
Removed COG -> cojo

Creates a picklist for COGType -> type and updates an existing incorrect picklist for COG -> COGType

Checklist

Self-review the PR after opening it to make sure the changes look good
and self-explanatory (or properly documented)
Add automated tests
Add relevant issue to release milestone

Testing instructions

Test new geo fields

StorageTreeDef -> institution
CollectionObjectGroup -> children

Verify above fields exist in Schema Config

Test COG picklists in Workbench

Create a workbench dataset with base table CollectionObjectGroup
Map a column to CogType -> name and to CogType -> type
Click save
Verify CogType -> name consists of all CogTypes added in the db as table records
Verify CogType -> type consists of values: Discrete, Consolidated, Drill Core

Test COG picklists in Data Entry

Here are some viewdefs that can be used to test the picklists in data entry

CollectionObjectGroup

<viewdef name="CollectionObjectGroup" class="edu.ku.brc.specify.datamodel.CollectionObjectGroup" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroup Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="cogtype" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

CollectionObjectGroupType

<viewdef name="CollectionObjectGroupType" class="edu.ku.brc.specify.datamodel.CollectionObjectGroupType" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroupType Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="type" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

Go to Data Entry > CollectionObjectGroup
Verify CogType is a picklist consisting of all CogTypes added in the db
Go to Data Entry > CollectionObjectGroupType
Verify type is a picklist consisting of values: Discrete, Consolidated, Drill Core
Verify a COGType and a COG can be saved
Verify that 'Default Collection Object Group Types' pick list is incorrectly defined #5289 is fixed
Verify that Cannot create COG Type via UI #5292 is fixed (i.e: A new CollectionObjectGroupType can be created)
Verify that Creating a COG without a type causes a crashes #5293 is fixed (i.e: There's a save blocker when COGType has not been selected)

sharadsw · 2024-09-09T15:01:18Z

~~TODO: Add Sp7 tables~~
The tables all look like they're only internally used. I don't think we need to add these to Schema Config

Missing Sp tables in Schema Config:
specifyuser_spprincipal
~~splibraryrole~~ (added in geo migration)
~~splibraryrolepolicy~~ (added in geo migration)
spprincipal_sppermission
~~sprole~~ (added in geo migration)
~~sprolepolicy~~ (added in geo migration)
spstynthy
~~spuserpolicy~~ (added in geo migration)
~~spuserrole~~ (added in geo migration)
sp_schema_mapping

NOTE: Spattachmentdataset already exists in Schema config but the table it refers to in the db is called attachmentdataset

ALL missing tables in Schema Config - Most of these seem to be only be internally used:
~~attachmentdataset~~ (exists as Spattachmentdataset)
auth_group
auth_group_permissions
auth_permission
autonumsch_coll
autonumsch_div
autonumsch_dsp
collectionobjecttype_collectionobjecttypeid
collectionobjecttype_CollectionObjectTypeID
continentcodes
countryinfo
django_content_type
django_migrations
django_session
dwc_ento
ento_web_portal
exup
geoname
hibernate_unique_key
ios_colobjagents
ios_colobjbio
ios_colobjchron
ios_colobjcnts
ios_colobjgeo
ios_colobjlitho
ios_geogeo_cnt
ios_geogeo_cty
ios_geoloc
ios_geoloc_cnt
ios_geoloc_cty
ios_taxon_pid
localityupdate
localityupdaterowresult
notifications_message
project_colobj
sgrbatchmatchresultitem
sgrbatchmatchresultset
sgrmatchconfiguration
specifyuser_spprincipal
splibraryrole
splibraryrolepolicy
spprincipal_sppermission
sprole
sprolepolicy
spstynthy
spuserpolicy
spuserrole
sp_schema_mapping
uniquenessrule_fields

sharadsw · 2024-09-30T20:13:46Z

TODO: ~~Verify picklist params, check all usages of the default cogtype picklist~~

specifyweb/specify/migrations/0004_schema_config_update.py

specifyweb/specify/migrations/0002_geo.py

sharadsw · 2024-10-22T16:32:30Z

@emenslin Thank you! Turns out some of those fields were already added in a previous migration. I have removed the duplicate fields and changed the testing instructions. Also, the formatter being invalid is expected. The formatter will be added in #5258

emenslin

Testing instructions

Test new geo fields

StorageTreeDef -> institution
CollectionObjectGroup -> children

Verify above fields exist in Schema Config

Test COG picklists in Workbench

Create a workbench dataset with base table CollectionObjectGroup
Map a column to CogType -> name and to CogType -> type
Click save
Verify CogType -> name consists of all CogTypes added in the db as table records
Verify CogType -> type consists of values: Discrete, Consolidated, Drill Core

Test COG picklists in Data Entry

Here are some viewdefs that can be used to test the picklists in data entry

CollectionObjectGroup

<viewdef name="CollectionObjectGroup" class="edu.ku.brc.specify.datamodel.CollectionObjectGroup" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroup Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="cogtype" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

CollectionObjectGroupType

<viewdef name="CollectionObjectGroupType" class="edu.ku.brc.specify.datamodel.CollectionObjectGroupType" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroupType Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="type" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

Go to Data Entry > CollectionObjectGroup
Verify CogType is a picklist consisting of all CogTypes added in the db
Go to Data Entry > CollectionObjectGroupType
Verify type is a picklist consisting of values: Discrete, Consolidated, Drill Core
Verify a COGType and a COG can be saved
Verify that 'Default Collection Object Group Types' pick list is incorrectly defined #5289 is fixed
Verify that Cannot create COG Type via UI #5292 is fixed (i.e: A new CollectionObjectGroupType can be created)
Verify that Creating a COG without a type causes a crashes #5293 is fixed (i.e: There's a save blocker when COGType has not been selected)

CollectionObjectGroup -> children does not show up in the schema config but it is present in the database schema. Everything else looks good though!

sharadsw · 2024-10-22T19:23:44Z

@emenslin Looks like the migration was either skipped due to it being run already previously or incomplete on certain databases. I have manually reverted the migration on kufish. Should hopefully work if you create a new instance of this branch on the test panel. Can you try testing it with another db as well just in case?

pashiav

Test new geo fields

StorageTreeDef -> institution
CollectionObjectGroup -> children

Verify above fields exist in Schema Config

Test COG picklists in Workbench

Create a workbench dataset with base table CollectionObjectGroup
Map a column to CogType -> name and to CogType -> type
Click save
Verify CogType -> name consists of all CogTypes added in the db as table records
Verify CogType -> type consists of values: Discrete, Consolidated, Drill Core

Test COG picklists in Data Entry

Here are some viewdefs that can be used to test the picklists in data entry

CollectionObjectGroup

<viewdef name="CollectionObjectGroup" class="edu.ku.brc.specify.datamodel.CollectionObjectGroup" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroup Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="cogtype" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

CollectionObjectGroupType

<viewdef name="CollectionObjectGroupType" class="edu.ku.brc.specify.datamodel.CollectionObjectGroupType" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroupType Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="type" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

Go to Data Entry > CollectionObjectGroup
Verify CogType is a picklist consisting of all CogTypes added in the db
Go to Data Entry > CollectionObjectGroupType
Verify type is a picklist consisting of values: Discrete, Consolidated, Drill Core
Verify a COGType and a COG can be saved
Verify that 'Default Collection Object Group Types' pick list is incorrectly defined #5289 is fixed
Verify that Cannot create COG Type via UI #5292 is fixed (i.e: A new CollectionObjectGroupType can be created)
Verify that Creating a COG without a type causes a crashes #5293 is fixed (i.e: There's a save blocker when COGType has not been selected)

Great work! I didn't find any issues that were a part of this pr.

Comments:
For future @specify/ux-testing:

Add the form definition for COG in App Resources, or else the name text field will be defaulted into a picklist because of the workbench picklist.

@sharadsw: "The field defaults to the workbench picklist created in this PR if the form definition does not explicitly set it as a text field."

"Formatter will be invalid until we implement defaults for geo":
Having multiple COGs with the same name is not reflected in the workbench picklist. WorkBench picklist only displays unique COGtype names, ignoring duplicates #5346

grantfitzsimmons

Test new geo fields
StorageTreeDef -> institution
CollectionObjectGroup -> children
Verify above fields exist in Schema Config

Test COG picklists in Workbench

Create a workbench dataset with base table CollectionObjectGroup

Map a column to CogType -> name and to CogType -> type

Click save

Verify CogType -> name consists of all CogTypes added in the db as table records

Verify CogType -> type consists of values: Discrete, Consolidated, Drill Core

This would have saved me a lot of time before the demo this morning. Nice work!

Test COG picklists in Data Entry

Here are some viewdefs that can be used to test the picklists in data entry

CollectionObjectGroup

<viewdef name="CollectionObjectGroup" class="edu.ku.brc.specify.datamodel.CollectionObjectGroup" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroup Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="cogtype" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

CollectionObjectGroupType

<viewdef name="CollectionObjectGroupType" class="edu.ku.brc.specify.datamodel.CollectionObjectGroupType" type="form" gettable="edu.ku.brc.af.ui.forms.DataGetterForObj" settable="edu.ku.brc.af.ui.forms.DataSetterForObj">
	<desc>The CollectionObjectGroupType Table</desc>
	<enableRules/>
	<columnDef>p,2px,p,2px,p,2px,p,2px,p,2px,p,p:g</columnDef>
	<rowDef auto="true" cell="p" sep="2px"/>
	<rows>
		<row>
			<cell type="label" labelfor="1"/>
			<cell type="field" id="1" name="name" uitype="text"/>
			<cell type="label" labelfor="2"/>
			<cell type="field" id="2" name="type" uitype="combobox"/>
		</row>
	</rows>
</viewdef>

Go to Data Entry > CollectionObjectGroup
Verify CogType is a picklist consisting of all CogTypes added in the db
Go to Data Entry > CollectionObjectGroupType
Verify type is a picklist consisting of values: Discrete, Consolidated, Drill Core
Verify a COGType and a COG can be saved
Verify that 'Default Collection Object Group Types' pick list is incorrectly defined #5289 is fixed

A little different, but functional. That was the intention of that issue.

Verify that Cannot create COG Type via UI #5292 is fixed (i.e: A new CollectionObjectGroupType can be created)

Verify that Creating a COG without a type causes a crashes #5293 is fixed (i.e: There's a save blocker when COGType has not been selected)

Yes, as long as the field is on the form, which is not unlike existing issues elsewhere (e.g. this error happens when you remove AccessionNumber (or any other required field) from the form and try to save)

grantfitzsimmons · 2024-10-23T19:32:46Z

specify@91f75de1bd1b:/opt/specify7$ ve/bin/python manage.py migrate specify 0006_fix_tectonic_tree_fields

Operations to perform:
  Target specific migration: 0006_fix_tectonic_tree_fields, from specify
Running migrations:
  Rendering model states... DONE
  Unapplying specify.0007_schema_config_update... OK

It's even reversible. 💯

fixed

sharadsw added 2 commits September 3, 2024 15:11

Add migration for geo fields

c77a61c

Update Schema Config README

942b8a4

sharadsw added this to the 7.9.8 milestone Sep 3, 2024

sharadsw changed the base branch from production to issue-5114 September 3, 2024 19:13

sharadsw changed the base branch from issue-5114 to production September 3, 2024 19:14

sharadsw changed the title ~~Update Schema Config to include new fields in existing tables~~ Update Schema Config to include new geo fields in existing tables Sep 3, 2024

CarolineDenis added 2 commits September 4, 2024 11:42

Add COG picklist migration

7ff1982

Add missing file

c20eeb2

sharadsw changed the title ~~Update Schema Config to include new geo fields in existing tables~~ Update Schema Config to include SP tables, new geo fields, and COG picklists Sep 12, 2024

sharadsw marked this pull request as ready for review September 12, 2024 16:27

sharadsw changed the title ~~Update Schema Config to include SP tables, new geo fields, and COG picklists~~ Update Schema Config to include new geo fields and COG picklists Sep 12, 2024

sharadsw requested review from melton-jason, acwhite211 and CarolineDenis September 12, 2024 17:04

CarolineDenis approved these changes Sep 12, 2024

View reviewed changes

sharadsw mentioned this pull request Sep 18, 2024

Add new geo view def + missing default #5258

Open

sharadsw mentioned this pull request Sep 30, 2024

'Default Collection Object Group Types' pick list is incorrectly defined #5289

Closed

grantfitzsimmons linked an issue Sep 30, 2024 that may be closed by this pull request

'Default Collection Object Group Types' pick list is incorrectly defined #5289

Closed

sharadsw mentioned this pull request Sep 30, 2024

Creating a COG without a type causes a crashes #5293

Closed

Use correct picklist in cogtype business rule

87db686

sharadsw mentioned this pull request Oct 1, 2024

Cannot create COG Type via UI #5292

Closed

melton-jason previously requested changes Oct 1, 2024

View reviewed changes

specifyweb/specify/migrations/0004_schema_config_update.py Outdated Show resolved Hide resolved

specifyweb/specify/migrations/0004_schema_config_update.py Outdated Show resolved Hide resolved

specifyweb/specify/migrations/0002_geo.py Outdated Show resolved Hide resolved

sharadsw added 5 commits October 4, 2024 11:37

Merge remote-tracking branch 'origin/production' into issue-5215

6a4171d

Update discipline relations and change dependency

bee072c

Update cogtype -> type picklist

e7f82b4

Merge remote-tracking branch 'origin/production' into issue-5215

57e2489

Rename migration

0a5547d

sharadsw requested a review from melton-jason October 8, 2024 18:57

sharadsw added 4 commits October 18, 2024 11:33

Fix duplicate schema config entries

dac4da4

temp change revert

1fcefb5

Revert age migration

b132374

Fix duplicate fields being added in migration

723e536

sharadsw requested review from a team and removed request for a team October 21, 2024 20:35

Base automatically changed from issue-5317 to production October 21, 2024 21:11

Add changes for COG -> children

58ef9de

sharadsw requested review from emenslin and a team October 22, 2024 16:27

Change formatter name to table name

9aeb7d3

emenslin previously requested changes Oct 22, 2024

View reviewed changes

Merge branch 'production' into issue-5215

d8800df

sharadsw requested review from emenslin, grantfitzsimmons and a team October 22, 2024 19:23

pashiav approved these changes Oct 23, 2024

View reviewed changes

pashiav requested a review from a team October 23, 2024 15:29

acwhite211 approved these changes Oct 23, 2024

View reviewed changes

grantfitzsimmons approved these changes Oct 23, 2024

View reviewed changes

sharadsw merged commit d437dcb into production Oct 23, 2024
12 checks passed

sharadsw deleted the issue-5215 branch October 23, 2024 19:42

sharadsw mentioned this pull request Oct 24, 2024

Schema records for new Geo fields are not present #5339

Closed

This was referenced Oct 24, 2024

WorkBench picklist only displays unique COGtype names, ignoring duplicates #5346

Open

Batch Edit, Workbench, Query Builder backend improvements #4929

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Schema Config to include new geo fields and COG picklists #5257

Update Schema Config to include new geo fields and COG picklists #5257

sharadsw commented Sep 3, 2024 •

edited

Loading

sharadsw commented Sep 9, 2024 •

edited

Loading

sharadsw commented Sep 30, 2024 •

edited

Loading

sharadsw commented Oct 22, 2024

emenslin left a comment

sharadsw commented Oct 22, 2024

pashiav left a comment •

edited

Loading

grantfitzsimmons left a comment

grantfitzsimmons commented Oct 23, 2024

Update Schema Config to include new geo fields and COG picklists #5257

Update Schema Config to include new geo fields and COG picklists #5257

Conversation

sharadsw commented Sep 3, 2024 • edited Loading

Checklist

Testing instructions

Test new geo fields

Test COG picklists in Workbench

Test COG picklists in Data Entry

sharadsw commented Sep 9, 2024 • edited Loading

sharadsw commented Sep 30, 2024 • edited Loading

sharadsw commented Oct 22, 2024

emenslin left a comment

Choose a reason for hiding this comment

Testing instructions

Test new geo fields

Test COG picklists in Workbench

Test COG picklists in Data Entry

sharadsw commented Oct 22, 2024

pashiav left a comment • edited Loading

Choose a reason for hiding this comment

Test new geo fields

Test COG picklists in Workbench

Test COG picklists in Data Entry

grantfitzsimmons left a comment

Choose a reason for hiding this comment

Test new geo fields

Test COG picklists in Workbench

Test COG picklists in Data Entry

grantfitzsimmons commented Oct 23, 2024

sharadsw commented Sep 3, 2024 •

edited

Loading

sharadsw commented Sep 9, 2024 •

edited

Loading

sharadsw commented Sep 30, 2024 •

edited

Loading

pashiav left a comment •

edited

Loading