Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

.NET 8 and Native AOT approach #2146

Open
3 of 8 tasks
AlexanderPopovSLIC opened this issue Nov 30, 2023 · 4 comments · May be fixed by #2176
Open
3 of 8 tasks

.NET 8 and Native AOT approach #2146

AlexanderPopovSLIC opened this issue Nov 30, 2023 · 4 comments · May be fixed by #2176

Comments

@AlexanderPopovSLIC
Copy link

Description

If I publish my code in Native AOT then all parts of code which are related to Kafka usage (such as: Producers and Consumers) stopped working. I have got the next message in Console after running published to native AOT app

Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
   at System.Linq.ThrowHelper.ThrowNoMatchException() + 0x2b
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1, Func`2) + 0x27
   at Confluent.Kafka.Impl.Librdkafka.SetDelegates(Type) + 0xdb
   at Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List`1) + 0x16
   at Confluent.Kafka.Impl.Librdkafka.LoadNetStandardDelegates(String) + 0xb2
   at Confluent.Kafka.Impl.Librdkafka.Initialize(String) + 0xd0
   at Confluent.Kafka.Producer`2..ctor(ProducerBuilder`2) + 0x19d
   at BLL.Core.KafkaService..ctor(IServiceProvider serviceProvider) + 0x1f5
   at WebApplication1!<BaseAddress>+0xcf395d
   at System.Reflection.DynamicInvokeInfo.InvokeWithFewArguments(IntPtr, Byte&, Byte&, Object[], BinderBundle, Boolean) + 0x95
   at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0x11d
   at Internal.Reflection.Execution.MethodInvokers.InstanceMethodInvoker.CreateInstance(Object[], BinderBundle, Boolean) + 0x44
   at Internal.Reflection.Core.Execution.MethodBaseInvoker.CreateInstance(Object[], Binder, BindingFlags, CultureInfo) + 0x3b
   at System.Reflection.Runtime.MethodInfos.RuntimePlainConstructorInfo`1.Invoke(BindingFlags, Binder, Object[], CultureInfo) + 0x55

I'm using a very simple code which works in regular published app:

IDictionary<string, string> config = new Dictionary<string, string>()
{
    { "bootstrap.servers", BootstrapServers },
    { "socket.max.fails", "0" },
    { "message.send.max.retries", "10" },
    { "batch.num.messages", "1000000" },
    { "queue.buffering.max.ms", "1" },
    { "request.timeout.ms", "2000" },
    { "message.timeout.ms", "5000" },
    { "client.id", Dns.GetHostName() }
};

//     -->  This line of code throws an exception  <--
_producer = new ProducerBuilder<Null, byte[]>(new ProducerConfig(config)).Build();

OS -> Windows 11

Nuget package is the latest one on the current moment
Confluent.Kafka, Version=2.3.0.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e

How to reproduce

  1. Create a new ConsolApp project (.NET 8) using Visual Studio, VS Code, or just using terminal
  2. Install Nuget package (Confluent.Kafks 2.3.0.0)
  3. Add the next lines into .csproj file
<PublishAot>true</PublishAot>
<PublishReadyToRun>true</PublishReadyToRun>
<PublishTrimmed>true</PublishTrimmed>
  1. Use my code I provided above to try to create Producer (there are required minimum changes)
  2. Publish App using the next command (terminal should be open from the path where .csproj file is located)
    dotnet publish -c Release -o "otput_path_where_app_will_be_published"
  3. Run published app and check the error

Checklist

Please provide the following information:

  • A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
  • Confluent.Kafka nuget version.
  • Apache Kafka version.
  • Client configuration.
  • Operating system.
  • Provide logs (with "debug" : "..." as necessary in configuration).
  • Provide broker log excerpts.
  • Critical issue.
@latop2604
Copy link

Hello, not sure if it will help but I managed to make it work on windows aot by changing only one line

        private static void LoadNetStandardDelegates(string userSpecifiedPath)
        {
            if (userSpecifiedPath != null)
            {
                if (WindowsNative.LoadLibraryEx(userSpecifiedPath, IntPtr.Zero, WindowsNative.LoadLibraryFlags.LOAD_WITH_ALTERED_SEARCH_PATH) == IntPtr.Zero)
                {
                    // TODO: The Win32Exception class is not available in .NET Standard, which is the easy way to get the message string corresponding to
                    // a win32 error. FormatMessage is not straightforward to p/invoke, so leaving this as a job for another day.
                    throw new InvalidOperationException($"Failed to load librdkafka at location '{userSpecifiedPath}'. Win32 error: {Marshal.GetLastWin32Error()}");
                }
            }

-            TrySetDelegates(new List<Type> { typeof(NativeMethods.NativeMethods) });
+            SetDelegates(typeof(NativeMethods.NativeMethods));
        }

@latop2604 latop2604 linked a pull request Jan 21, 2024 that will close this issue
Open
@BoutemineOualid
Copy link

Having the same issue here (Publishing as trimmed, no AOT).

@Im5tu
Copy link

Im5tu commented Dec 11, 2024

Any updates from the team on this?

@egrangel
Copy link

Any updates on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support AOT for basic consumer app latop2604/confluent-kafka-dotnet
5 participants
@BoutemineOualid @latop2604 @egrangel @Im5tu @AlexanderPopovSLIC