Guide
The Cucumber JSON report: what is in the file, and which tools write it
By Miftode Vasilica, maker of Gherkin Studio ·
Draft: fields in [brackets] are filled in before the app is published.
A Cucumber JSON report is one JSON array of features. Each feature has elements, which are the scenarios. Each scenario has steps, and each step has a result with a status. An app that shows test results on a Jira story reads four things from it: the tags, the file path, the scenario names and the step statuses. This page shows the file, how each test tool writes it, and how Gherkin Studio turns it into pass or fail.
Valid JSON, zero results
A developer adds the send step to the pipeline, pushes, and reads the log:
{"error":"Report is not valid JSON"}
The file was real, and it came from Cucumber. But it had been written by the message formatter, which puts one JSON object on each line. Every line was valid JSON. The file as a whole was not. "A report from Cucumber" and "a report in the Cucumber JSON format" are two different things. Ten minutes with the file open would have shown it. This page is those ten minutes.
The file
One feature, one scenario, three steps, the last one failing:
[
{
"uri": "features/PROJ-12.feature",
"name": "Password reset",
"tags": [{ "name": "@PROJ-12" }],
"elements": [
{
"type": "scenario",
"name": "Link expires after one hour",
"line": 5,
"tags": [{ "name": "@PROJ-12" }],
"steps": [
{
"keyword": "Given ",
"name": "I requested a reset link 61 minutes ago",
"result": { "status": "passed" }
},
{
"keyword": "When ",
"name": "I open the link",
"result": { "status": "passed" }
},
{
"keyword": "Then ",
"name": "I see \"This link has expired\"",
"result": {
"status": "failed",
"error_message": "expected 'This link has expired' but saw 'Welcome back'"
}
}
]
}
]
}
]
Real files carry more: durations, step definition locations, hooks. The fields above are the ones that decide what a story shows.
Which tools write it
| Tool | How to get the file | Worth knowing |
|---|---|---|
| cucumber-js | --format json:cucumber.json |
Built in |
| Cucumber-JVM | plugin json:target/cucumber.json |
Built in |
| behave | behave -f json -o cucumber.json |
Tags are plain text without @. The path is in location, not uri |
| pytest-bdd | pytest --cucumberjson=cucumber.json |
Tags come without @. Each row of a Scenario Outline is its own entry |
| Reqnroll | No built-in formatter for this format | Its built-in formatters write HTML and Cucumber Messages. You need a converter, and we have not tested one |
The differences in the third column are why a report that works with one tool shows nothing with another. Gherkin Studio reads all of these forms: a tag as { "name": "@PROJ-12" } or as plain "PROJ-12", with or without the @, and the path from uri or from location.
How a status is decided
For one scenario:
| What the steps say | The scenario is |
|---|---|
Any step or hook is failed or ambiguous |
Failed |
Every step is passed |
Passed |
Anything else: skipped, pending, undefined, or no steps |
Not run |
Two rows deserve a second look.
- A failing hook fails the scenario. If a
beforeorafterhook fails, the scenario is failed even when every step passed. undefinedis not a failure. A scenario whose steps have no code behind them shows as not run. That is deliberate: a scenario nobody automated yet has not failed. It also means an untested story never goes red by itself. Watch the "not run" count.
For the story:
- Failing if any of its scenarios failed. The label reads, for example,
1/3 failing. - Passing if all of them passed:
3/3 passing. - If some passed and the rest did not run, the label still says
2/3 passing, in grey instead of green. - Not run otherwise.
On the story you also see each scenario with its status, the step it failed at, and the start of the error message, up to 500 characters.
How a scenario finds its story
In this order: an issue-key tag on the scenario, an issue-key tag on the feature, then a key in the file name, such as PROJ-12.feature. The forms that work and the ones that do not are in the guide to linking scenarios with tags.
What the endpoint answers
| Code | Body | Meaning |
|---|---|---|
| 200 | updated, unknownIssues, unmappedScenarios, totalScenarios, skippedOverLimit |
Read. The lists say what landed where |
| 400 | Report is not valid JSON |
The file is not JSON. A line-per-message output or an XML file ends here |
| 400 | Expected a Cucumber JSON report: a JSON array of features |
It is JSON, but not this format |
| 400 | Report is larger than 5 MB |
Split the run, or send one report per suite |
| 401 | Missing or wrong token |
The Authorization: Bearer header is absent or the token was replaced |
| 403 | A message that the subscription is not active | The app's license on this Jira site has lapsed. Reports are refused until it is renewed |
| 405 | Use POST |
The request was a GET |
The limits behind two of those rows:
- 5 MB per report.
- 500 stories per report. If a report mentions more, the rest are counted in
skippedOverLimit. Send the suites separately. - 200 scenarios listed per story. The counts stay exact above that. Only the list is cut.
Is this format on its way out?
The cucumber-js documentation calls it the legacy JSON format, says its JSON formatter is in maintenance mode, and recommends the newer message formatter. That one writes Cucumber Messages, one JSON object per line.
The first four tools in the table above still write the JSON format today. Gherkin Studio reads this format, and does not read Cucumber Messages yet.
Short answers
Can I send JUnit XML instead?
No. JUnit XML has no tags, so nothing in it says which story a test belongs to.
What if one story's scenarios run in two parallel jobs?
Merge the files first. Each report replaces the result of the stories it mentions, so the second job's report would replace the first job's. The format is an array, so merging is joining arrays: jq -s 'add' a.json b.json > cucumber.json.
Does a Background count as a scenario?
No. A Background entry in the report is skipped.
Why does my story say "Not run" when the pipeline is green?
Its scenarios were in the report without a passed or failed status: skipped, pending or undefined. Open the story to see the status of each one.
Where Gherkin Studio fits
Gherkin Studio is a BDD app for Jira Cloud, built on Atlassian Forge. It is on the Atlassian Marketplace, with a 30-day free trial. See how to send the report from GitHub Actions and from GitLab CI, Bitbucket Pipelines and Jenkins.