Subscribe topic with regex , When kafka added new topic, Consume will not receive new topic message

[chunln]

Description

Subscribe topic with regex , When kafka added new topic, Consume will not receive new topic message, Unless resubscribed.

How to reproduce

nuget:1.3.0
kafka:2.4.0
env:windows

Code:

           using (var consumer = new ConsumerBuilder<Ignore, string>(conf).Build())
           {
               consumer.Subscribe("^project.*");

               CancellationTokenSource cts = new CancellationTokenSource();
               Console.CancelKeyPress += (_, e) =>
               {
                   e.Cancel = true; 。
                   cts.Cancel();
               };

               try
               {
                   while (true)
                   {
                       try
                       {
                           var cr = consumer.Consume(cts.Token);
                       }
                       catch (ConsumeException e)
                       {
                       }
                   }
               }
               catch (OperationCanceledException e)
               {
                   consumer.Close();
               }
           }

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version.
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with "debug" : "..." as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[edenhill]

The consumer will see the new topics on its next periodic metadata refresh which is controlled by topic.metadata.refresh.interval.ms.
You can also trigger a refresh manually by calling Metadata()

[thangld322]

Hi. I tried this and did not work.
Any update for 2025 please? Thanks.

UPDATE: I tried metadata.max.age.ms and it did work. But it waits very long to consume from newly created topic even when I set metadata.max.age.ms to 1000. How can I make it scan for new topic faster? Here is my current config:

// Kafka configuration
	config := &kafka.ConfigMap{
		"bootstrap.servers":   bootstrapServers,
		"group.id":            "webhook-notifier",
		"auto.offset.reset":   "earliest",
		"metadata.max.age.ms": 1000,
	}