Guide
Send Cucumber results to Jira from GitLab CI, Bitbucket Pipelines and Jenkins
By Miftode Vasilica, maker of Gherkin Studio ·
Draft: fields in [brackets] are filled in before the app is published.
In every CI system the recipe has the same three parts: run the tests, keep going when they fail, and send the Cucumber JSON file with one curl call. What changes is how each system spells "keep going when they fail". This page has an example for GitLab CI, Bitbucket Pipelines and Jenkins, each checked against that system's own documentation. GitHub Actions is in the first guide.
Red in the pipeline, green in Jira
Tuesday, 10:02. A push breaks the password reset. The pipeline goes red.
At 10:30 the product owner opens the story in Jira. It says 3/3 passing. She tells the customer the fix is ready.
The story was showing Monday's result. The step that sends the report came after the tests, the tests failed, and the CI stopped right there, as it should. The report only ever travelled on good days.
Before you start
A Jira admin opens Apps → Gherkin Studio and creates a CI token. The page shows the address to send to. The token is shown once, so copy it then. Save both as secrets in your CI, under these names:
GHERKIN_STUDIO_URLGHERKIN_STUDIO_TOKEN
The examples use cucumber-js. For other tools, only the line that runs the tests changes. See which tools write Cucumber JSON.
GitLab CI
cucumber:
script:
- npx cucumber-js --format json:cucumber.json
after_script:
- >
curl -sS -X POST "$GHERKIN_STUDIO_URL?build=$CI_PIPELINE_URL"
-H "Authorization: Bearer $GHERKIN_STUDIO_TOKEN"
-H "Content-Type: application/json"
--data-binary @cucumber.json
The part that matters is after_script. GitLab runs it when the script section fails, which is exactly the day you need it.
Good to know:
- It runs in a new shell. Variables you exported in
scriptare gone. Files are still there, socucumber.jsonis found. - It cannot change the job's result. If the tests passed and the
curlcall fails, the job is still green. So a broken send never blocks a merge, and never announces itself either. More on that below. - It does not run when the job times out, unless you configure the runner's timeouts for it.
- Secrets go in Settings → CI/CD → Variables. Mark the token as masked.
CI_PIPELINE_URLis GitLab's own variable with the address of the pipeline. On the story it becomes the open build link.
Bitbucket Pipelines
pipelines:
default:
- step:
name: Cucumber
script:
- npx cucumber-js --format json:cucumber.json
after-script:
- >
curl -sS -X POST "$GHERKIN_STUDIO_URL?build=https://bitbucket.org/$BITBUCKET_REPO_FULL_NAME/pipelines/results/$BITBUCKET_BUILD_NUMBER"
-H "Authorization: Bearer $GHERKIN_STUDIO_TOKEN"
-H "Content-Type: application/json"
--data-binary @cucumber.json
Note the hyphen: Bitbucket writes after-script, GitLab writes after_script.
Good to know:
- It runs whether the script succeeded or failed. If you need to know which, Bitbucket sets
BITBUCKET_EXIT_CODEto 0 or 1 insideafter-script. - A failing command there does not change the step's status. Same as GitLab.
- There is no variable with the address of the run, so the example builds it from two that exist: the repository's full name and the build number.
- Secrets go in Repository settings → Pipelines → Repository variables. Tick Secured for the token.
Jenkins
pipeline {
agent any
environment {
GHERKIN_STUDIO_URL = credentials('gherkin-studio-url')
GHERKIN_STUDIO_TOKEN = credentials('gherkin-studio-token')
}
stages {
stage('Cucumber') {
steps {
sh 'npx cucumber-js --format json:cucumber.json'
}
}
}
post {
always {
sh '''
curl -sS -X POST "$GHERKIN_STUDIO_URL?build=$BUILD_URL" \
-H "Authorization: Bearer $GHERKIN_STUDIO_TOKEN" \
-H "Content-Type: application/json" \
--data-binary @cucumber.json
'''
}
}
}
Good to know:
post { always { … } }runs whatever the result of the build was.- Keep the single quotes around the shell script. With double quotes, Groovy puts the token into the command before the shell sees it. The Jenkins documentation has a section on how secrets leak that way.
- Secrets are two credentials of the kind "Secret text", with the IDs used above.
BUILD_URLon a Jenkins without HTTPS. Gherkin Studio keeps a build link only when it starts withhttps://. If your Jenkins is served over plainhttp://, the results still arrive. The story just has no open build link.- Unlike the other two, a failing
curlhere does fail the build. If you do not want a network hiccup to turn a green build red, add|| trueafter the last line of thecurlcommand.
The failure that makes no noise
curl reports success when the server answered, whatever the answer was. A wrong token gets a 401, and curl still exits with 0. Nothing turns red. In GitLab and Bitbucket nothing could turn red anyway.
So for the first few runs, open the job log and read what came back. The examples use -sS so that the answer is printed and the progress bar is not.
| You see | It means |
|---|---|
{"updated":["PROJ-12"], …} |
It worked. These stories received results |
"unknownIssues":["PROJ-99"] |
A tag names a story that does not exist in Jira |
"unmappedScenarios":4 |
Four scenarios have no issue key anywhere |
{"error":"Missing or wrong token"} |
The token secret is wrong, or a new token was created since |
{"error":"Report is not valid JSON"} |
The file is not Cucumber JSON. Check the --format line |
An error from curl itself about reading a file, such as curl: option --data-binary: error encountered when reading a file |
cucumber.json does not exist. The tests crashed before writing it |
In the last case nothing was sent, so Jira keeps showing the previous result. It is the same trap as in the story above, through a different door: a run that dies before it writes a report looks, from Jira, like a day when nothing happened.
Short answers
Should a failed send fail my build?
No. The tests decide the build. The send is a notification.
Do I need a token per pipeline?
No, one per Jira site. Creating a new token stops the old one at once, so every pipeline that used the old one needs the new one.
My CI is not on this page. Will it work?
If it can run curl, yes. Look in its documentation for the setting that runs a step even after a failure, and put the call there.
Can I send from several branches?
Yes, but every report replaces the result of the stories it mentions. If feature branches send too, the story shows whichever ran last. Sending from the main branch only keeps the status on the story meaningful.
Where Gherkin Studio fits
Gherkin Studio is a BDD app for Jira Cloud, built on Atlassian Forge. It is on the Atlassian Marketplace, with a 30-day free trial. See what the report file contains and how to tag scenarios with issue keys.