Config file same key overriding order/precedence bug

[blackr1234]

Describe the bug

Library versions

• Spring Boot 2.7.1
• Spring Cloud 2021.0.3
• Spring Cloud Config 3.1.3 (managed)

---

Issue

I have a minimal Spring Cloud Config setup with Git backend using a GitHub public repo.

I deliberately put properties of the same keys in multiple config files to test the behavior of Spring Cloud Config in resolving config values.

These are the types of my config files:

1. application.yaml
2. {spring.application.name}.yaml
3. application-{profile}.yaml
4. {spring.application.name}-{profile}.yaml

My expectation is, values in profile-specific files should override values in files of the default profile.
The overriding order should be "similar" to the order of the above ordered list, so if the exact same property key exists in all 4 files, the final value should come from the last one (the 4th list item).

However, the actual result is as follows:

1. application-{profile}.yaml
2. {spring.application.name}-{profile}.yaml
3. application.yaml
4. {spring.application.name}.yaml

From the Spring Cloud Config official doc, it says:

If there are profile-specific YAML (or properties) files, these are also applied with higher precedence than the defaults. Higher precedence translates to a PropertySource listed earlier in the Environment. (These same rules apply in a standalone Spring Boot application.)

Reference: https://docs.spring.io/spring-cloud-config/docs/3.1.3/reference/html/

As it says the behavior should be the same as standalone Spring Boot, the Spring Boot official doc says:

Config data files are considered in the following order:

1. Application properties packaged inside your jar (application.properties and YAML variants).
2. Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).
3. Application properties outside of your packaged jar (application.properties and YAML variants).
4. Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).

Reference: https://docs.spring.io/spring-boot/docs/2.7.1/reference/htmlsingle/#features.external-config

From my testing with config files within the Spring Cloud Config Client application (under src/main/resources), it works fine and this is the overriding order:

1. application.yaml
2. application-{profile}.yaml

The value comes from the 2nd file.

---

Sample

Project source code and config files can be found in my public Git repos:

• Config server - spring-cloud-config-server-demo-git-public
• Config client - spring-cloud-config-client-demo-public
• Config files - spring-cloud-config-demo-public

Test command and environment

The command for starting the 2 projects in local:

mvn spring-boot:run

Results of Maven version command (mvn -v):

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: C:\maven
Java version: 11.0.17, vendor: Azul Systems, Inc., runtime: C:\zulu11
Default locale: en_US, platform encoding: MS950
OS name: "windows 11", version: "10.0", arch: "amd64", family: "windows"

Explanation of the code

It is a minimal Spring Cloud Config setup with Git backend using a GitHub public repo. I have 4 config files that are named with the aforementioned naming conventions.

The active Spring profile is dev only.

Test config properties:

• my.prop is for manual testing, by removing the greatest value one by one, I am able to find out the order of config overriding in Spring Cloud Config. Current output value should be 4 which comes from the application config file of the default profile (spring-cloud-config-client-demo.yaml).
• my.prop2 is for proving that the overriding order in config files within the project code are working fine. Current output value should be 2.
• order-test.* is for proving this bug. There are 4 properties in total. Values are simply the name of the config files and these files have 1~4 properties, so we can tell which files "win" in overriding. If you run the config client application, you will see from the log that all 4 properties have unique values and thus the aforementioned order is the only possible order. The class ConfigFileLoadOrderTestConfig.java contains Hiberate Validator annotations to validate the config values automatically. Error will be thrown if values do not match and the application will terminate.

Log

(I added newlines below for readability)

order-test object: ConfigFileLoadOrderTestConfig(
loadOrder01=application-dev.yaml,
loadOrder02=spring-cloud-config-client-demo-dev.yaml,
loadOrder03=application.yaml,
loadOrder04=spring-cloud-config-client-demo.yaml
)

my.prop: 4
my.prop2: 2

---

My thought

The official documentation isn't very clear as it lacks examples. Perhaps my understanding of "higher precedence" is wrong but apparently the behavior in Spring Cloud Config (at least the Git implementation) does not match the behavior of reading config files within src/main/resources in the project.

If A has a "higher precedence" than B, then I will expect that A gets applied but not B. In other words, A "wins".

Last but not least, if this is a known issue in some old versions of Spring, please provide the corresponding issue details and let me know the status of fix.

If you are also facing this issue or think that my issue is valid, I'd appreciate if you could provide quick feedback with👍 or ❤️ on this issue post.

Thanks a lot everyone and have a wonderful year ahead!

[ryanjbaxter]

Your sample client app has some issues, you are using bootstrap to load properties as well as spring.config.import. You should choose one or the other and not use both.

When I hit the config server it returns the properties sources in the right order

{
   "name":"spring-cloud-config-client-demo",
   "profiles":[
      "dev"
   ],
   "label":"master",
   "version":"38c671c28d05a41d775365acf199c67515f6a56f",
   "state":null,
   "propertySources":[
      {
         "name":"https://github.com/blackr1234/spring-cloud-config-demo-public.git/spring-cloud-config-client-demo-dev.yaml",
         "source":{
            "my.prop":2,
            "order-test.load-order-02":"spring-cloud-config-client-demo-dev.yaml",
            "order-test.load-order-03":"spring-cloud-config-client-demo-dev.yaml",
            "order-test.load-order-04":"spring-cloud-config-client-demo-dev.yaml"
         }
      },
      {
         "name":"https://github.com/blackr1234/spring-cloud-config-demo-public.git/application-dev.yaml",
         "source":{
            "my.prop":1,
            "order-test.load-order-01":"application-dev.yaml",
            "order-test.load-order-02":"application-dev.yaml",
            "order-test.load-order-03":"application-dev.yaml",
            "order-test.load-order-04":"application-dev.yaml"
         }
      },
      {
         "name":"https://github.com/blackr1234/spring-cloud-config-demo-public.git/spring-cloud-config-client-demo.yaml",
         "source":{
            "my.prop":4,
            "order-test.load-order-04":"spring-cloud-config-client-demo.yaml"
         }
      },
      {
         "name":"https://github.com/blackr1234/spring-cloud-config-demo-public.git/application.yaml",
         "source":{
            "my.prop":3,
            "order-test.load-order-03":"application.yaml",
            "order-test.load-order-04":"application.yaml"
         }
      }
   ]
}

Once you fix the client I think that will clear things up

[blackr1234]

Thanks @ryanjbaxter for the resolution. It works by removing the spring.config.import property in the config client! 👍❤️

I have created a new Git branch, namely fixed-1, in all my repos with the fix and this branch passes my testing.

The overriding order now becomes what I stated at the very beginning of my issue post, i.e.:

1. application.yaml
2. {spring.application.name}.yaml
3. application-{profile}.yaml
4. {spring.application.name}-{profile}.yaml

The results look good to me until I switch to the JDBC backend. The same test doesn't pass. 😓❌

So I have converted the YAML config properties into SQL statements and let Spring automatically initialize everything in an in-memory H2 database (src/main/resources/data-h2.sql).

By calling the /{spring.application.name}/{profile} endpoint of the config server like what you did in your reply, I find that the order of the property sources does not look right.

Below is what I get from /spring-cloud-config-client-demo/dev:

{
  "name": "spring-cloud-config-client-demo",
  "profiles": [
    "dev"
  ],
  "label": null,
  "version": null,
  "state": null,
  "propertySources": [
    {
      "name": "spring-cloud-config-client-demo-dev",
      "source": {
        "my.prop": "4",
        "order-test.load-order-04": "spring-cloud-config-client-demo-dev.yaml"
      }
    },
    {
      "name": "spring-cloud-config-client-demo-default",
      "source": {
        "my.prop": "2",
        "order-test.load-order-02": "spring-cloud-config-client-demo.yaml",
        "order-test.load-order-03": "spring-cloud-config-client-demo.yaml",
        "order-test.load-order-04": "spring-cloud-config-client-demo.yaml"
      }
    },
    {
      "name": "application-dev",
      "source": {
        "my.prop": "3",
        "order-test.load-order-03": "application-dev.yaml",
        "order-test.load-order-04": "application-dev.yaml"
      }
    },
    {
      "name": "application-default",
      "source": {
        "my.prop": "1",
        "order-test.load-order-01": "application.yaml",
        "order-test.load-order-02": "application.yaml",
        "order-test.load-order-03": "application.yaml",
        "order-test.load-order-04": "application.yaml"
      }
    }
  ]
}

When I run the config client application, it terminates with below error:

Binding to target org.springframework.boot.context.properties.bind.BindException: Failed to bind properties under 'order-test' to code.ConfigFileLoadOrderTestConfig$$EnhancerBySpringCGLIB$$26519f5a failed:

    Property: order-test.loadOrder03
    Value: spring-cloud-config-client-demo.yaml
    Origin: "order-test.load-order-03" from property source "bootstrapProperties-spring-cloud-config-client-demo-default"
    Reason: must match "^application-dev.yaml$"

It's because the overriding order becomes like this with JDBC backend:

1. application.yaml
2. application-{profile}.yaml
3. {spring.application.name}.yaml
4. {spring.application.name}-{profile}.yaml

---

Here is all the source code in my public GitHub repos:

• Config server (JDBC) - spring-cloud-config-server-demo-jdbc-public
• Config client - spring-cloud-config-client-demo-public (fixed-1)
• Config files - spring-cloud-config-demo-public (fixed-1)
• Config server (Git) - spring-cloud-config-server-demo-git-public (fixed-1)

[blackr1234]

Sorry I misclicked the "Close with comment" button.
Please let me know if I should raise another bug ticket as the issue is now related to the JDBC backend implementation.
Thanks a lot.

[blackr1234]

The involved code of the Spring Cloud Config JDBC implementation:
https://github.com/spring-cloud/spring-cloud-config/blob/v3.1.3/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/JdbcEnvironmentRepository.java#L112

Perhaps the 2 for loops should be swapped.

[woshikid]

@blackr1234 you are right about it. I just checked JdbcEnvironmentRepositoryTests.java and the test code shows as you described.
@ryanjbaxter it looks like the incorrect config order has existed for years. Changes will break existing projects. Which branch should we commit to?

[ryanjbaxter]

Its a bug so yes the changes will break people but its fixing a bug that shouldn't have been there in the first place. The fix should go against the 3.1.x branch.

Can you point me at the test you were looking at specifically?

[woshikid]

spring-cloud-config/spring-cloud-config-server/src/test/java/org/springframework/cloud/config/server/environment/JdbcEnvironmentRepositoryTests.java

Line 64 in c43fdc7

assertThat(env.getPropertySources().get(0).getName()).isEqualTo("foo-bar");

the basicProperties test and other tests all have similar asserts:

assertThat(env.getPropertySources().get(0).getName()).isEqualTo("foo-bar"); // {spring.application.name}-{profile}
assertThat(env.getPropertySources().get(0).getSource().get("a.b.c")).isEqualTo("foo-bar");
assertThat(env.getPropertySources().get(1).getName()).isEqualTo("foo"); // {spring.application.name}
assertThat(env.getPropertySources().get(1).getSource().get("a.b.c")).isEqualTo("foo-null");
assertThat(env.getPropertySources().get(2).getName()).isEqualTo("application-bar"); // application-{profile}
assertThat(env.getPropertySources().get(2).getSource().get("a.b.c")).isEqualTo("application-bar");
assertThat(env.getPropertySources().get(3).getName()).isEqualTo("application"); // application
assertThat(env.getPropertySources().get(3).getSource().get("a.b.c")).isEqualTo("application-null");

the getPropertySources() returns a List and these tests are expecting config order as follow:

1. {spring.application.name}-{profile}
2. {spring.application.name}
3. application-{profile}
4. application

this order does differ from the Git backend.