Event does not get tagged after updating to 1.5.51.1

[Haukinger]

Version Information
all current (1.5.51.1 / 1.5.52)

Describe the bug
When updating Akka.Persistence.Sql.Hosting from 1.5.51 to 1.5.51.1, some tests fail. One of them relies on a message getting tagged and showing up in a stream. It does before the update but not after.

To Reproduce

1. send a SubscribeCommand to the SubscriptionActor
2. that produces and persists a SubscribedEvent
3. the event gets tagged with "Subscription"
4. SubscriptionListActor maintains a stream looking for that tag and the event should show up there

Expected behavior
in 1.5.51 the event is in the stream

Actual behavior
in 1.5.51.1 it's not

Environment
Windows, VS 2022, Resharper's Nunit Runner, Sqlite

Additional context
the tagger is configured here:

.WithSqlPersistence( connectionString,
                     providerName,
                     useWriterUuidColumn: true,
                     journalBuilder: journalBuilder =>
                                     {
                                         journalBuilder.AddWriteEventAdapter<MessageTagger>( "status-tagger",
                                         [
                                             typeof(ISubscriptionEvent)
                                         ] );
                                     } )

the SubscriptionActor lives in a shard region

.WithShardRegion<SubscriptionActor>( "subscriptions",
                                     ( _, _, dependencyResolver ) => entityId => dependencyResolver.Props<SubscriptionActor>( entityId ),
                                     new MessageExtractor( 10 ),
                                     new()
                                     {
                                         RememberEntities = true,
                                         Role = "core",
                                         JournalOptions = shardingJournalOptions,
                                         SnapshotOptions = shardingSnapshotOptions,
                                         RememberEntitiesStore = RememberEntitiesStore.Eventsourced,
                                         StateStoreMode = StateStoreMode.DData,
                                     } )

this is the tagger

internal sealed class MessageTagger : IWriteEventAdapter
{
    public const string SUBSCRIPTION_TAG = "Subscription";

    #region IWriteEventAdapter
    public string Manifest(object evt) => string.Empty;

    public object ToJournal(object @event)
    { // <- breakpoint here is never hit in 1.5.51.1
        switch (@event)
        {
            case ISubscriptionEvent subscriptionEvent:
                return subscriptionEvent.HasStreamer() && subscriptionEvent.HasSubscriber() ? new Tagged( @event, new[] { SUBSCRIPTION_TAG } ) : @event;
            default:
                return @event;
        }
    }
    #endregion
}

[Aaronontheweb]

I am almost certain this is related to our changes in #549 - by standardizing / cleaning up the mess inside Akka.Persistence.Sql.Hosting we almost certainly caused a breaking configuration change from earlier versions of the software.

[Aaronontheweb]

Configuration Regression Testing Results

I've created comprehensive snapshot tests comparing v1.5.51 and v1.5.51.1 configuration (PR #553).

✅ Finding: No Configuration Regression at Global Level

All 15 tests PASSED - the HOCON configuration is identical between versions, including:

• Event adapter registration
• Event adapter bindings
• Query configuration (tag-read-mode, etc.)
• All persistence settings

However, we're not testing your exact scenario.

❓ Critical Missing Information

Your sharded actors use custom JournalOptions:

.WithShardRegion<SubscriptionActor>( "subscriptions",
    ...,
    new()
    {
        JournalOptions = shardingJournalOptions,  // <-- What is this?
        ...
    } )

Questions:

1. What does shardingJournalOptions contain?
2. Is it:
   • new JournalOptions() (empty/default)?
   • A copy of the global SQL journal options?
   • Something else?
3. Does it specify a custom PluginId or use the default "akka.persistence.journal.sql"?

Theory

If shardingJournalOptions creates a separate journal configuration without inheriting event adapters from the global config, that would explain why tagging stopped working in your sharded actors.

The refactoring in PR #549 changed how configuration merging works. If sharded actors with custom JournalOptions don't properly inherit the global event adapters, that's the bug.

Please share what shardingJournalOptions contains so we can create a test that reproduces this exact scenario.

[Aaronontheweb]

Regression Confirmed

Successfully reproduced the bug:

• v1.5.51: Event adapters work correctly
• v1.5.51.1: Event adapters fail (timeout on EventsByTag queries)

Root Cause

PR #549 changed how event adapters are registered. The v1.5.51 approach invoked journalBuilder early and stored adapters in journalOpt.Adapters before serialization. The v1.5.51.1 approach delegates to Akka.Persistence.Hosting's API, passing the journalBuilder callback.

When WithSqlPersistence() is called multiple times (first with adapters, then for sharding config), something in the second call breaks the adapter registration from the first call.

Key Finding

Configuration snapshot tests show HOCON is identical between versions - adapters ARE in the config. But runtime tests fail. This means the issue is adapter registration/application, not configuration generation.

The Akka.Persistence.Hosting API must be storing registration state beyond HOCON that gets overwritten by subsequent calls.

Test Added

Created RuntimeEventAdapterSpec.cs in src/Akka.Persistence.Sql.Hosting.Tests that reproduces the exact scenario from the user's repro code.

[Aaronontheweb]

Root Cause Identified and Fix Submitted

The issue has been traced to a bug in Akka.Persistence.Hosting where event adapters configured via the NEW callback API were not being invoked at runtime.

Root Cause: AkkaPersistenceJournalBuilder.Build() was creating a Config with .WithFallback(Persistence.DefaultConfig()) which interfered with HOCON merging, causing adapters to be present in configuration but not invoked during persistence operations.

Fix PR: akkadotnet/Akka.Hosting#674

The fix removes the unnecessary fallback and includes comprehensive regression tests. All tests pass including verification with Akka.Persistence.Sql scenarios.

This fix is especially critical since the old Adapters property API has been deprecated in akkadotnet/Akka.Hosting#665, making the callback API (which was broken) the only option going forward.

Once the Akka.Hosting fix is released, this issue will be resolved.