Wrong test name in XML report when running Scenario Outline

[frederico-betting]

👓 What did you see?

Scenario Outline name is not properly stored in the xml file when using variables.

Example:

Scenario Outline: Validate TLS ciphers for <service>
    Given I have the  domain "<domain>"
    Then I check that the following list of ciphers are configured:
      | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 |
      | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 |

   Examples:
      | service       | domain        |
      | qa-service    |  example.com  |

The output is:

<?xml version='1.0' encoding='UTF-8'?>
<testsuite name="Cucumber" time="25.413" tests="1" skipped="0" failures="0" errors="0">
<testcase classname="TLS Ciphers in public services" name="Validate TLS ciphers for &lt;service> - Example #1.1" time="24.496">
(...)

OR

Validate TLS ciphers for <service> - Example #1.1

✅ What did you expect to see?

I was expecting the following output instead:

The output is:

<?xml version='1.0' encoding='UTF-8'?>
<testsuite name="Cucumber" time="25.413" tests="1" skipped="0" failures="0" errors="0">
<testcase classname="TLS Ciphers in public services" name="Validate TLS ciphers for qa-service" time="24.496">
(...)

Validate TLS ciphers for qa-service

The variable <service> changed to the value it was defined in the example and (maybe) no reference the example ( - Example #1.1)

📦 Which tool/library version are you using?

I am using cucumber 7.14.0 and testng 7.5.1.

🔬 How could we reproduce it?

• You can have the latest version of cucumber and testng in a Java project.
• Create a Scenario Outline with variables in the name.
• Enable plugin to report in junit format.

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[shivki564]

I see this issue from cucumber 7.11.0 where Junit-xml-formatter version was updated from 0.1 -> 0.2.1.

Is this expected behavior with new version 0.2.x ??

[mpkorstanje]

This was indeed introduced in Cucumber JVM 7.11 with cucumber/cucumber-jvm#2638.

The change in naming is somewhat expected. There are unfortunately multiple ways to name the examples in scenario outlines in an unambiguous manner.

Using the path to an example is more likely to be unique as parametrized scenario outline names aren't a well known feature and do not work well when multiple parameters are involved. They don't always fit in the scenario outline name without making it unreadable. This makes the path to the example a better "default" (the current lack of alternatives not withstanding).

One solution would be to extract the naming strategy and then add a different strategy that uses the example name instead of the scenario outline name I.e.

public class MessagesToJunitXmlWriter {
...
    public void setNamingStrategy(NamingStrategy s) {
    }
}

If someone wants to write a pull request that it would be most welcome.

Do note that Cucumber JVM doesn't currently have the ability to pass options to plugins so that would be another hurdle to take.

For a more immediate term solution you could also take the code of the original plugin (see cucumber/cucumber-jvm#2638) and use that in your own project. But eventually that will break once the old plugin system is replaced with messages.

[shivki564]

Thanks for the response. We can look for a workaround to add a post processing step to parse the generated Cucumber XML report, modify the scenario names as needed, and save the modified XML before publishing Junit results in CI CD.

[mpkorstanje]

I see. What is the impact of the changing names for you? Are you using them as identifiers somewhere else?

[frederico-betting]

In my case I create a report with results of all tests I execute in the pipeline. With the current approach, it's too difficult to understand what's exactly the scenario failing (from the report only). I need to open the feature file and count the examples to know what are the values that made my test fails.

Since I add references of my examples in the title of the scenario, it's easier to know what's failing based on the name.
As in my example:

Currently this is what I get:
Validate TLS ciphers for <service> - Example #1.1

But it's easier when I get a list of names like this:
Validate TLS ciphers for qa-service

Considering I have a huge list of examples, it's easier to track when having the values available.

Right now, I created a custom build of this plugin and pushed in my local nexus, just replacing the values from the example table in the name of scenario.

This is what I'm doing so far:

In XmlReportData.java, I created 2 helper methods:
1 . Identify if I have references to the headers of example table in the name/title of the scenario.
2. Update the name of the scenario with the proper values (best effort).

At the end, I change the name of the scenario this way:

private void scenario(Feature feature, Rule rule, Scenario scenario) {
(...)
                StringBuilder suffix = new StringBuilder(" - ");
                if (!currentExamples.getName().isEmpty()) {
                    suffix.append(currentExamples.getName()).append(" - ");
                }
                suffix.append("Example #").append(examplesIndex + 1).append(".").append(exampleIndex + 1);

                // Replace values from the example in the name of the scenario
                String scenarioName = updateScenarioNameExample(currentExamples.getTableHeader().get(),
                        currentExample, exampleHeadersScenarioName, scenario.getName());

                pickleAstNodeIdToLongName.put(currentExample.getId(), rulePrefix + scenarioName + suffix);
(...)
}

With this code, I'll have something more interesting in my report. E.g.
Validate TLS ciphers for qa-service - Example #1.1

If you think this works, I can open a PR with my solution...

[mpkorstanje]

In my case I create a report with results of all tests I execute in the pipeline. With the current approach, it's too difficult to understand what's exactly the scenario failing (from the report only). I need to open the feature file and count the examples to know what are the values that made my test fails.

Okay that is a good reason.

The Pickle message should have the interpolated scenario title already, I would prefer a solution that used that - less logic would be better, this formatter is already more complicated than I would like it to be. 😄

If you happen to be interested in providing a pull request that uses the pickle message, I would be happy to take it. If not, perhaps someone else will get to it.

[frederico-betting]

That's ok. I agree with you. Let me try to make it simple and try to get the value from Pickle. I could not find at the first attempt.

[mpkorstanje]

Much appreciated!

I had a moment to refresh my memory. Pickle messages are processed after the gherkinDocument message. So you'd have get the name of the pickle in the getPickleName method and insert it into the value retrieved from pickleAstNodeIdToLongName somehow.

To do that you could split the pickleAstNodeIdToLongName map into a prefix and suffix map. But I'll let you decide on the best strategy.

I'm also not sure if there is data in the test set with parametrized scenario names. We get all our examples from https://github.com/cucumber/compatibility-kit. So if something is missing we'd have to add it there.
For the short term, to keep going, you could also copy and edit examples-tables.feature.ndjson.

[mpkorstanje]

I forgot we had a similar item on the backlog for JUnit 5 for Cucumber JVM (cucumber/cucumber-jvm#274). And I think it would make sense to harmonize the output from JUnit 5 and the XML report.

So given the feature:

Feature: Examples Tables

  Scenario Outline: Eating <eat> of <start> cucumbers 
    Given there are <start> cucumbers
    When I eat <eat> cucumbers
    Then I should have <left> cucumbers

    Examples: These are passing
      | start | eat | left |
      |    12 |   5 |    7 |
      |    20 |   5 |   15 |

    Examples: These are failing
      | start | eat | left |
      |    12 |  20 |    0 |
      |     0 |   1 |    0 |

When the example-number naming strategy is used this should be rendered as:

 * Example Tables - Eating <eat> of <start> cucumbers  - These are passing - Example #1.1
 * Example Tables - Eating <eat> of <start> cucumbers  - These are passing - Example #1.2
 * Example Tables - Eating <eat> of <start> cucumbers  - These are failing - Example #1.1
 * Example Tables - Eating <eat> of <start> cucumbers  - These are failing - Example #1.2

When the pickle-name strategy is used this should be rendered as:

 * Example Tables - Eating <eat> of <start> cucumbers - These are passing - Eating 5 of 12 cucumbers 
 * Example Tables - Eating <eat> of <start> cucumbers - These are passing - Eating 5 of 20 cucumbers 
 * Example Tables - Eating <eat> of <start> cucumbers - These are failing - Eating 20 of 12 cucumbers 
 * Example Tables - Eating <eat> of <start> cucumbers - These are failing - Eating 1 of 0 cucumbers 

[mpkorstanje]

@frederico-betting should you still be interested in this, I've merged some refactoring that should make it much easier to make this change. If you pass the pickle to the getPickleName method, you can easily replace the example number with a pickle name.

https://github.com/cucumber/cucumber-junit-xml-formatter/blob/a73086b5a47eb2da5382ff0158a67d80a6b1f88a/java/src/main/java/io/cucumber/junitxmlformatter/XmlReportData.java#L78-L100