PGVector problem

[MBrekhof]

I have an Blazor Web app that uses EF Core 7 with Postgresql 15 and the PGVector extension installed (https://github.com/pgvector/pgvector). This all works great using ' options.UseNpgsql(connectionString, o => o.UseVector()).UseLowerCaseNamingConvention();' to configure my DBContext.

However as soon as I add Hangfire using:
services.AddHangfire(config => config.UsePostgreSqlStorage(hfconnectionString));
services.AddHangfireServer();

I get this error:
23:04:17:442 SELECT c.codeobjectid, c.codeobjectcategoryid, c.codeobjectcontent, c.subject, c.tokens, c.vectordatastring, c0.codeobjectcategoryid, c0.category
23:04:17:442 FROM codeobject AS c
23:04:17:442 INNER JOIN codeobjectcategory AS c0 ON c.codeobjectcategoryid = c0.codeobjectcategoryid
23:04:17:694 Exception thrown: 'System.InvalidCastException' in Npgsql.dll
23:04:17:694 Exception thrown: 'System.InvalidCastException' in Npgsql.dll
23:04:17:694 Microsoft.EntityFrameworkCore.Query: Error: An exception occurred while iterating over the results of a query for context type 'DocGPT.Module.BusinessObjects.DocGPTEFCoreDbContext'.
23:04:17:694 System.InvalidCastException: Can't cast database type . to Vector
23:04:17:694 at Npgsql.Internal.TypeHandling.NpgsqlTypeHandler.ReadCustom[TAny](NpgsqlReadBuffer buf, Int32 len, Boolean async, FieldDescription fieldDescription)
23:04:17:694 at Npgsql.NpgsqlDataReader.GetFieldValue[T](Int32 ordinal)
23:04:17:694 at lambda_method628(Closure, QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator)
23:04:17:694 at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable1.Enumerator.MoveNext() 23:04:17:694 23:04:17:694 System.InvalidCastException: Can't cast database type .<unknown> to Vector 23:04:17:694 at Npgsql.Internal.TypeHandling.NpgsqlTypeHandler.ReadCustom[TAny](NpgsqlReadBuffer buf, Int32 len, Boolean async, FieldDescription fieldDescription) 23:04:17:694 at Npgsql.NpgsqlDataReader.GetFieldValue[T](Int32 ordinal) 23:04:17:694 at lambda_method628(Closure, QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator) 23:04:17:694 at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable1.Enumerator.MoveNext()
23:04:17:945 Exception thrown: 'System.InvalidCastException' in Microsoft.EntityFrameworkCore.Relational.dll
23:04:17:945 Microsoft.AspNetCore.Components.Server.Circuits.RemoteRenderer: Warning: Unhandled exception rendering component: Can't cast database type . to Vector
23:04:17:945
23:04:17:945 System.InvalidCastException: Can't cast database type . to Vector
23:04:17:945 at Npgsql.Internal.TypeHandling.NpgsqlTypeHandler.ReadCustom[TAny](NpgsqlReadBuffer buf, Int32 len, Boolean async, FieldDescription fieldDescription)
23:04:17:945 at Npgsql.NpgsqlDataReader.GetFieldValue[T](Int32 ordinal)
23:04:17:945 at lambda_method628(Closure, QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator)
23:04:17:945 at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable1.Enumerator.MoveNext() 23:04:17:945 at DevExpress.EntityFrameworkCore.Security.Query.EnumerableDecorator1.EnumeratorDecorator.MoveNext()
23:04:17:945 at System.Linq.Enumerable.OfTypeIterator[TResult](IEnumerable source)+MoveNext()
23:04:17:945 at System.Linq.Enumerable.WhereEnumerableIterator`1.ToList()
23:04:17:945 at DevExpress.ExpressApp.EFCore.EFCoreCollection.Init()
23:04:17:945 at DevExpress.ExpressApp.EFCore.EFCoreCollection.get_InnerList()
23:04:17:945 at DevExpress.ExpressApp.EFCore.EFCoreCollection.get_Count()
23:04:17:945 at DevExpress.ExpressApp.ProxyCollection.get_Count()
23:04:17:945 at DevExpress.Data.DataControllerBase.ResetValidateBindingListChangedConsistency(Boolean isSubscribe)

Same query (totaly unrelated to Hangfire) works great when I disable Hangfire and the Hangfire.Postgresql.

Any ideas how to try and solve this?

[azygis]

Nothing in the stack trace indicates anything Hangfire-related which makes me think it's XY problem, or that maybe you're using EF drop create database migrations, which in turn doesn't work out too well since the database can get created by this provider first before EF migrations.

Can you provide

• startup logs;
• runnable reproduction?

[MBrekhof]

hangfire_startup.txt
This is the startup log, not using migrations. As you can see it runs until I navigate to an entity that has a vector datatype in the table definition. I also included my startup.cs just in case.
Startup - Copy.cs.txt

[dmitry-vychikov]

@MBrekhof

I looked through Hangfire.Postgres source code, but I couldn't find any places that could be potentially causing this issue.

PgVector uses Npgsql extensions to add new handlers which allows it to support deserealization of their customVector type. Extension registration is done via UseVector extension method (https://github.com/pgvector/pgvector-dotnet/blob/master/src/Pgvector/Npgsql/VectorExtensions.cs#L7-L11).

Since all Npgsql extensions are registered on a global static object, any other library could potentially break this by registering something on top of it or even removing older registrations. But I do not see hangfire even touching that NpgsqlConnection.GlobalTypeMapper object.

@MBrekhof can you try the following?

• Change registration order for Hangfire and DbContext services. Try putting Hangfire first.
  • If Hangfire somehow removes Vector type mappings, then it can probably be solved if you add them after hanfire has been initialized
• Save NpgsqlConnection.GlobalTypeMapper to a local variable at the end of your Startup file. Run it with a debugger and inspect what Extensions are registered there. Look for Vector extension. We need to make sure it is registered correctly.

[MBrekhof]

Hi,
Thanks for taking time to look at this. I am going on a short holiday so I will not be able to look at this problem for some time. However I decided to try and solve this problem by giving Hangfire its own webservice, this should make this problem go away without having to go to deep into the problem of the npgsql connection.
regards,
Martin

[azygis]

Closing the issue as the type mappings are not explicitly changed anywhere in Hangfire.PostgreSql. If trying the suggested changes and/or moving to a separate service doesn't help, you can reopen it.

[msdbcardoso]

It appears that I encountered a similar issue. When I transitioned from using Hangfire's InMemoryStorage to PostgreSQL, I encountered an "invalid cast exception" related to PgVector vector properties.
The culprit is for sure Hangfire.PostgreSql.

I also attempted @MBrekhof suggestion
"Change registration order for Hangfire and DbContext services. Try putting Hangfire first.
If Hangfire somehow removes Vector type mappings, then it can probably be solved if you add them after hanfire has been initialized"
but didnt work.

It seems there is really some conflicting behavior going on

CC: @azygis

Here is the full stack trace

[project-x-api_626157d2-e]: [20:22:55 ERR] An exception occurred in the database while saving changes for context type 'Infrastructure.Data.ApplicationDbContext'.
[project-x-api_626157d2-e]: Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
[project-x-api_626157d2-e]: ---> System.InvalidCastException: Writing values of 'Pgvector.Vector' is not supported for parameters having no NpgsqlDbType or DataTypeName. Try setting one of these values to the expected database type..
[project-x-api_626157d2-e]: at Npgsql.Internal.AdoSerializerHelpers.g__ThrowWritingNotSupported|1_0(Type type, PgSerializerOptions options, Nullable1 pgTypeId, Nullable1 npgsqlDbType, Exception inner)
[project-x-api_626157d2-e]: at Npgsql.Internal.AdoSerializerHelpers.GetTypeInfoForWriting(Type type, Nullable1 pgTypeId, PgSerializerOptions options, Nullable1 npgsqlDbType)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlParameterCollection.ProcessParameters(PgSerializerOptions options, Boolean validateValues, CommandType commandType)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: --- End of inner exception stack trace ---
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) [project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) [project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
[project-x-api_626157d2-e]: ---> System.InvalidCastException: Writing values of 'Pgvector.Vector' is not supported for parameters having no NpgsqlDbType or DataTypeName. Try setting one of these values to the expected database type..
[project-x-api_626157d2-e]: at Npgsql.Internal.AdoSerializerHelpers.g__ThrowWritingNotSupported|1_0(Type type, PgSerializerOptions options, Nullable1 pgTypeId, Nullable1 npgsqlDbType, Exception inner)
[project-x-api_626157d2-e]: at Npgsql.Internal.AdoSerializerHelpers.GetTypeInfoForWriting(Type type, Nullable1 pgTypeId, PgSerializerOptions options, Nullable1 npgsqlDbType)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlParameterCollection.ProcessParameters(PgSerializerOptions options, Boolean validateValues, CommandType commandType)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: --- End of inner exception stack trace ---
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) [project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) [project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken)
[project-x-api_626157d2-e]: at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)

[azygis]

The culprit is for sure Hangfire.PostgreSql.

Can't see how it's for sure, sorry. This package depends on both Npgsql and Dapper which have their own intristicts about mapping.

We do nothing with geometry or special type mappings, at all. It's all happening on Npgsql side. There is a discussion in a linked issue #322 which shows how we can workaround the problem in the last messages, but that has not been done yet and I have no time for it currently.

There's also another workaround mentioned, maybe you can try that as well to some extent.