Major work around DbDataSource management, enum handling and plugins #3167

roji · 2024-05-12T22:08:24Z

Added user-facing enum mapping API on NpgsqlDbContextOptionsBuilder, and removed hacky reflection-based pulling of enum type mappings from the Npgsql ADO.NET layer. This is #3063.
Thanks to the above, NpgsqlTypeMappingSource (a singleton service) no longer needs the NpgsqlDataSource (to pull enum mappings from). Changed DbDataSource to be scoped (when configured explicitly by the user via UseNpgsql()), allowing it to change across invocations. This fixes #2891, #3086.
When the new EF MapEnum() API is used, we now create a data source internally, configuring it with the enum definition as well, so users no longer need to configure both EF and ADO.NET for the enum. This is #1026.
- The internally created data source is (obviously) a singleton, and if the user passes different connection strings to UseNpgsql(), ~~we throw~~ we create new data sources, cached by the connection string. ~~This means it's no longer possible to do both enums and switch connection strings (breaking change).~~
Similarly to enums, added infrastructure to allow EF plugins to participate in the NpgsqlDataSourceBuilder configuration. This is used by the NTS and NodaTime plugins to add the corresponding ADO.NET-level plugins.
Removed all enum- and plugin-related global configuration in the tests, switching to the above new data source APIs.
- Now that the plugins just configure the internal data source and the obsolete global type mapper is no longer used, merged the NodaTime test project into the general test project (NodaTime previously had its own project to avoid interference via the global type mapper).
Added a convention to add enums to the model based on the new, explicit user API above. This means that the single EF MapEnum() in UseNpgsql() is now sufficient to configure EF's type mapping, the ADO.NET type mapping, and cause the enum to be created in the database. This completes #1026.

Future task: add NpgsqlDataSourceBuilder to NpgsqlDbContextOptionsBuilder, to allow easy configuration of Npgsql via the EF API (#2542). Accessing the DataSourceBuilder would also count as "requires internal data source", just like enums/plugins.

Fixes #2891
Fixes #3063
Fixes #1026

See comments below for last remaining tasks.

/cc @NinoFloris @ajcvickers

src/EFCore.PG/Design/Internal/NpgsqlCSharpRuntimeAnnotationCodeGenerator.cs

src/EFCore.PG/Metadata/Conventions/NpgsqlPostgresModelFinalizingConvention.cs

Fixes npgsql#2891 Fixes npgsql#3063 Fixes npgsql#1026 Co-authored-by: Nino Floris <[email protected]>

NinoFloris

Looks like a great upgrade! My comments mostly consist of nits and questions.

src/EFCore.PG/Query/ExpressionTranslators/Internal/NpgsqlStringMethodTranslator.cs

src/EFCore.PG/Storage/Internal/Mapping/NpgsqlArrayTypeMapping.cs

src/EFCore.PG/Storage/Internal/NpgsqlDataSourceManager.cs

NinoFloris · 2024-05-14T12:19:41Z

src/EFCore.PG/Storage/Internal/NpgsqlDataSourceManager.cs

+
+            // If the user hasn't configured anything in UseNpgsql (no data source, no connection, no connection string), check the
+            // application service provider to see if a data source is registered there, and return that.
+            { ConnectionString: null } when applicationServiceProvider?.GetService<NpgsqlDataSource>() is DbDataSource dataSource


Follow-up to this approach could be the user providing a serviceKey to some new option on UseNpgsql.

Nice idea, can absolutely see that happening.

src/EFCore.PG/Storage/Internal/NpgsqlDataSourceManager.cs

src/EFCore.PG/Storage/Internal/NpgsqlTypeMappingSource.cs

test/EFCore.PG.FunctionalTests/Query/NodaTimeQueryNpgsqlTest.cs

test/EFCore.PG.FunctionalTests/TestUtilities/NpgsqlTestStore.cs

NinoFloris · 2024-05-14T12:34:25Z

src/EFCore.PG/Storage/Internal/NpgsqlTypeMappingSource.cs

+            _sqlGenerationHelper.DelimitIdentifier(name, schema),
+            schema is null ? name : schema + "." + name,
+            enumDefinition.ClrType,
+            enumDefinition.Labels);


What was the conclusion around whether to sort these alphabetically?

Yeah, this is done in the convention which "copies" the definitions from the context options to the model (code), because that's the part that ends up forming the migration. In the type mapping source I don't think there should be any significance to the ordering, we only use the labels when we need to stick a label constant in the SQL.

Also adding sorting in NpgsqlMigrationsSqlGenerator, when detecting new labels (this is an improvement regardless of the other changes in this PR).

NinoFloris · 2024-05-14T12:36:03Z

src/EFCore.PG/Storage/Internal/NpgsqlTypeMappingSource.cs

+        }
+        else
+        {
+            // TODO: Not sure what to do about quoting. Is the user expected to configure properties


Do you think the current code as-is is good enough for merging for the moment (e.g. until we do npgsql/npgsql#5710)?

Yeah as long as users can pass quoted names - as I understand they can - they should always be able to make EF map things correctly when mappingInfo.StoreTypeName is quoted.

… internally

roji · 2024-05-17T18:20:39Z

@NinoFloris can I get another quick review from you on the added changes etc.? Especially on the new logic adding the ConcurrentDictionary for multiple data sources in NpgsqlDataSourceManager, and the concurrency handling around that?

NinoFloris · 2024-05-18T16:17:36Z

src/EFCore.PG/Storage/Internal/NpgsqlDataSourceManager.cs

-            throw new InvalidOperationException(NpgsqlStrings.DataSourceWithMultipleConnectionStrings(dataSourceFeature));
+            newDataSource.Dispose();
+        }
+        else if (_isDisposed == 1)


Not sure what the infra looks like - in terms of full disposal blocking this method from being called later on - but you might want this check to be done before as well. Just so you don't keep adding more items to the dictionary when things are already disposed.

Not sure what the infra looks like

Me neither... My logic here is that it's OK if a data source gets added to the dictionary during/after Dispose (after all Dispose doesn't clear the dictionary, we don't really care) - the important bit is that the data sources always get disposed, which I think this achieves...

Will go ahead and merge, we can always revisit if needed (unlikely).

NinoFloris

LGTM

Following npgsql/efcore.pg#3167, which removes EF's logic for extracting enum mappings from Npgsql.

roji commented May 12, 2024

View reviewed changes

src/EFCore.PG/Design/Internal/NpgsqlCSharpRuntimeAnnotationCodeGenerator.cs Outdated Show resolved Hide resolved

src/EFCore.PG/Metadata/Conventions/NpgsqlPostgresModelFinalizingConvention.cs Outdated Show resolved Hide resolved

roji force-pushed the ScopedDataSource branch 2 times, most recently from d8db176 to d4ae264 Compare May 13, 2024 13:13

Major work around DbDataSource management, enum handling and plugins

18b0a4c

Fixes npgsql#2891 Fixes npgsql#3063 Fixes npgsql#1026 Co-authored-by: Nino Floris <[email protected]>

NinoFloris reviewed May 14, 2024

View reviewed changes

roji added 2 commits May 17, 2024 20:16

Address review comments

983ab15

Support multiple connection strings by managing multiple data sources…

79f8abe

… internally

roji force-pushed the ScopedDataSource branch from d4ae264 to 79f8abe Compare May 17, 2024 18:19

roji marked this pull request as ready for review May 17, 2024 18:19

roji mentioned this pull request May 17, 2024

Add ConfigureDataSource() under UseNpgsql() #2542

Closed

NinoFloris reviewed May 18, 2024

View reviewed changes

NinoFloris approved these changes May 18, 2024

View reviewed changes

roji merged commit 6b3ca68 into npgsql:main May 18, 2024
15 checks passed

roji deleted the ScopedDataSource branch May 18, 2024 17:42

roji added a commit to roji/Npgsql that referenced this pull request May 18, 2024

Remove hacky enum type mapping introspection

995c5d0

Following npgsql/efcore.pg#3167, which removes EF's logic for extracting enum mappings from Npgsql.

This was referenced May 18, 2024

Remove hacky enum type mapping introspection npgsql/npgsql#5711

Merged

Using NpgsqlDataSource in DependencyInjection pollutes other containers #2891

Closed

Brar mentioned this pull request May 22, 2024

Make the enumName parameter in NpgsqlDbContextOptionsBuilder.MapEnum<T>() optional. #3178

Open

NinoFloris pushed a commit to roji/Npgsql that referenced this pull request May 30, 2024

Remove hacky enum type mapping introspection

e6000a9

Following npgsql/efcore.pg#3167, which removes EF's logic for extracting enum mappings from Npgsql.

roji added a commit to npgsql/npgsql that referenced this pull request May 30, 2024

Remove hacky enum type mapping introspection (#5711)

d1c62a1

Following npgsql/efcore.pg#3167, which removes EF's logic for extracting enum mappings from Npgsql.

This was referenced May 31, 2024

NpgsqlDataSourceBuilder MapEnum doesn't work, Unable to cast object of type 'System.Int32' to type 'System.Enum' #2853

Closed

Question: Correct way to rotate database credentials when using a datasource #3193

Closed

roji linked an issue Jun 27, 2024 that may be closed by this pull request

Make NpgsqlDataSource a scoped service instead of singleton #3086

Closed

This comment was marked as resolved.

Sign in to view

roji mentioned this pull request Sep 13, 2024

Type mapping plugins and NpgsqlDataSource #2518

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Major work around DbDataSource management, enum handling and plugins #3167

Major work around DbDataSource management, enum handling and plugins #3167

roji commented May 12, 2024 •

edited

Loading

NinoFloris left a comment

NinoFloris May 14, 2024

roji May 17, 2024

NinoFloris May 14, 2024

roji May 17, 2024 •

edited

Loading

roji May 17, 2024

NinoFloris May 14, 2024

roji May 17, 2024

NinoFloris May 18, 2024

roji commented May 17, 2024

NinoFloris May 18, 2024

roji May 18, 2024

NinoFloris left a comment

This comment was marked as resolved.

This comment was marked as resolved.

Major work around DbDataSource management, enum handling and plugins #3167

Major work around DbDataSource management, enum handling and plugins #3167

Conversation

roji commented May 12, 2024 • edited Loading

NinoFloris left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roji May 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roji commented May 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NinoFloris left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

roji commented May 12, 2024 •

edited

Loading

roji May 17, 2024 •

edited

Loading