Articles

To demonstrate how to develop, deploy, and manage applications using Jira Software and various connected tools, our team created ImageLabeller, a simple demo application built on AWS that uses machine learning to apply labels to images.

This page covers how to deploy ImageLabeller with GitLab. Before you begin, we recommend reading the ImageLabeller architecture and AWS SageMaker setup pages for context.

Prerequisites

If you don’t already have a GitLab group, follow the steps in this GitLab guide to create one from scratch.

Public facing GitHub repositories with ImageLabeller code

https://github.com/AtlassianOpenDevOpsGuides

Jira GitLab integration demo video

Watch video

Integrate Jira and GitLab

From Jira click Board, then Apps, then GitLab.

Screenshot of dropdown menu in Jira Software to navigate to GitLab

Click Get it now.

Click Apps, then Manage your apps.

Gitlab app modal in Jira software with dropdown menu

Expand GitLab for Jira.

Expand Gitlab when on the Manage Apps screen in Jira Software

Click Add namespace.

Screen to add a namespace to your Gitlab Jira software configuration

Select your existing namespace, and click Link. This guide assumes you already have an existing GitLab account, and GitLab group.

Linking a Gitlab namespace in Jira Software

Add SSH Key to GitLab

Click on your profile icon in the top right corner, and click preferences.

Navigating to preferences using dropdown menu in Gitlab

Click SSH Keys and follow the instructions to generate a new SSH key or use an existing SSH key.

Create a repository for AWS S3 infrastructure

A standard developer loop typically has a developer picking up a task from Jira, moving it to work in progress, and then doing the development work. The Jira issue ID is the key which ties the development work to the Jira issue. It is the core integration component between the two systems.

Go to Jira, and create a new issue for adding an AWS S3 infrastructure repository to GitLab. Make note of the issue ID. IM-5 in this example.

Creating a new issue for your board in Jira software

Go to GitLab and click New project.

Navigating to create "New project" in GitLab

Click Create blank project.

Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

Creating a new project - detailed screen in Gitlab

In your terminal go to your s3_infra repository, and run the following to push your AWS CloudFormation template.yml file to GitLab.

git add --all
git commit -m "IM-5 add s3_infra repository to gitlab"
git remote add origin git@gitlab.com:pmmquickstartguides/s3_infra.git
git branch -m mainline
git push -u origin mainline

Add AWS access key

Click Settings, then CI/CD. Scroll down, and expand Variables. Click Add variable.

Create two variables. One for your AWS access key ID, and one for your AWS secret access key.

"Add variable" modal to add your AWS keys in GitLab

Protect the variables so they are only used by pipelines running on protected branches, and tags. Give the IAM user associated with the AWS access key AdministratorAccess. You can also opt to use more fine grained access control by choosing individual AWS access policies.

AWS keys listed under "Variables" section in the CI/CD settings page in GitLab

Configure protected branches for access to protected variables

Click Settings, then Repository. Scroll down, and expand Protected branches.

Enter your Jira issue ID prefix and a *.

The Jira issue IDs are like IM-5, and IM-6 in this example; The prefix is IM-.

Enter IM-*, and click Protect.

Configuring protected branches in GitLab

You will see mainline, and IM-* as protected branches.

Setup deployment environments

Click Deployments, then Environments. Click New environment to add new environments. There are Test environments in US-WEST-1, and US-EAST-2, and Production environments in US-WEST-2, US-EAST-1, and CA-CENTRAL-1 in this example.

Setting up deployment environments in GitLab

.gitlab-ci.yml for deploying to AWS

Go to your s3_infra repository in your terminal, and create a branch named after your Jira issue ID.

git checkout -b IM-5

Create a .gitlab-ci.yml file with the following yaml. This defines a deployment workflow for your Test, Staging, and Production environments.

stages:
  - merge-request
  - test-us-west-1
  - test-us-east-2
  - production-us-west-2
  - production-us-east-1
  - production-ca-central-1

merge-request-pipeline-job:
  stage: merge-request
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - echo "This pipeline always succeeds and enables merges during merge requests"
    - echo true

deploy-test-us-west-1:
  stage: test-us-west-1
  environment: test-us-west-1
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - aws cloudformation deploy --region us-west-1 --template-file template.yml --stack-name OpenDevOpsS3Infra

deploy-test-us-east-2:
  stage: test-us-east-2
  environment: test-us-east-2
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - aws cloudformation deploy --region us-east-2 --template-file template.yml --stack-name OpenDevOpsS3Infra

deploy-production-us-west-2:
  stage: production-us-west-2
  environment: production-us-west-2
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - aws cloudformation deploy --region us-west-2 --template-file template.yml --stack-name OpenDevOpsS3Infra

deploy-production-us-east-1:
  stage: production-us-east-1
  environment: production-us-east-1
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - aws cloudformation deploy --region us-east-1 --template-file template.yml --stack-name OpenDevOpsS3Infra

deploy-production-ca-central-1:
  stage: production-ca-central-1
  environment: production-ca-central-1
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - aws cloudformation deploy --region ca-central-1 --template-file template.yml --stack-name OpenDevOpsS3Infra

Understanding a .gitlab-ci.yml file

Stages

Add a stages block to define the execution order of your GitLab pipeline. Stages are executed in the order they are defined in the stages block. Jobs associated with a stage are executed in parallel.

stages:
  - merge-request
  - test-us-west-1
  - test-us-east-2
  - production-us-west-2
  - production-us-east-1
  - production-ca-central-1

Jobs

Jobs are associated with a stage, and can be associated with an environment. Rules control whether, or not a particular job will execute. The rule in this example checks if the pipeline branch is not the default branch, and if the pipeline is not being run automatically as part of a merge request.

You can specify a different image for each job. You can setup images with tools necessary for your job scripts. The script section defines the set of steps that run when the job is executed.

deploy-test-us-west-1:
  stage: test-us-west-1
  environment: test-us-west-1
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - aws cloudformation deploy --region us-west-1 --template-file template.yml --stack-name OpenDevOpsS3Infra

Merge request pipeline

A pipeline is run automatically by GitLab when a merge request is approved. You can create a job for this pipeline by adding a rule. The job always succeeds in this example.

merge-request-pipeline-job:
  stage: merge-request
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - echo "This pipeline always succeeds and enables merges during merge requests"
    - echo true

Pushing to a feature branch

Run the following from the command line to push your changes to the IM-5 branch of your s3_infra repository. Include the Jira issue ID in commit messages, and branch names to enable the Jira GitLab integration to keep track of what is happening in your project.

git add --all
git commit -m "IM-5 add .gitlab-ci.yml to s3_infra"
git push -u origin IM-5

Click CI/CD, then Pipelines to see the pipeline run.

Click the pipeline ID of the running pipeline.

Pipeline ID for the running pipeline in GitLab

Click on a job to see more details.

Detailed job screen for running pipeline in GitLab

Create a merge request

To create a merge request click Merge requests, then Create merge request.

Choose your feature branch as the source branch then, click Compare branches and continue.

Comparing source branch and target branch in GitLab

Choose an Assignee, and a Reviewer.

Choosing a reviewer for your merge request in GitLab

Click Create merge request.

Selecting "Create merge request" button in GitLab

Review the code changes, then click Approve.

Merge request detailed screen where you can review changes in GitLab

Click CI/CI, then click Pipelines to see the merge request pipeline run.

Navigating to the "pipelines" screen in Gitlab to see run merge requests

Click the pipeline ID. Notice that the merge-request-pipeline-job is the only job that ran.

"Pipeline" detailed page showing that only merge-request-pipeline-job ran in GitLab

Go back to the merge request by clicking Merge requests, then clicking the active merge request, and click Merge. This kicks off another pipeline.

Merging the active merge request in GitLab

Click CI/CD, then click Pipelines. Click the pipeline ID.

Pipeline detailed page in GitLab displaying "Merge branch 'IM-5' into 'mainline'"

Create a repository for SystemTests

Go to Jira and create a Jira issue for adding a SystemTests repository to GitLab. Take note of the Jira issue ID. In this example it is IM-7.

Create an issue in Jira Software to "add GitLab repo for SubmitImage AWS Lambda"

Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

Filling in project details when creating a new project in GitLab

In your terminal go to your SystemTests repository, and run the following to push your code to GitLab.

git add --all
git commit -m "IM-7 add SystemTests repository to gitlab"
git remote add origin git@gitlab.com:pmmquickstartguides/systemtests.git
git branch -m mainline
git push -u origin mainline

The SystemTests repository doesn’t need a .gitlab-ci.yml file. It has no pipeline of its own since it provides tests for other pipelines to run. Take note of your SystemTests' remote url. SubmitImage, GetImageLabel, and InvokeLabeller CI/CD pipelines will clone the SystemTests repository during testing steps. You will need to update the gitlab-ci.yml of later repositories with the correct url.

Add a Deploy token

You need to add a deploy token to clone this repository during the execution of other pipelines. Click Settings, then Repository. Scroll down, and expand Deploy tokens.

Entering in an example name 'cloneMe' under "Deploy tokens" in GitLab

Enter a name, check read_repository, and click Create deploy token.

Selecting the "read_repository" checkbox on the "Deploy tokens" settings page in GitLab

The deploy token username is auto-generated. The deploy token password is provided once at creation. Add it to a secret management tool so it can be referenced later. Later in this guide the deploy token username is referenced as gitlab_deploy_token, and the deploy token password is referenced as gitlab_deploy_password.

Deploy tokens screen in Gitlab, displaying Deploy Token username and password

Create a repository for SubmitImage AWS Lambda

Go to Jira, and create a new issue for adding a SubmitImage AWS Lambda repository to GitLab. Make note of the issue ID. IM-8 in this example.

Imagelabeller board in Jira software - highlighting issue "IM-8 add GitLab repo for SubmitImage AWS Lambda"

Go to GitLab and click New project, then Create blank project. Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

screenshot of creating new project "submitimage" in gitlab

In your terminal go to your SubmitImage repository, and run the following to push your code to GitLab.

git add --all
git commit -m "IM-8 add SubmitImage to gitlab"
git remote add origin git@gitlab.com:pmmquickstartguides/submitimage.git
git branch -m mainline
git push -u origin mainline

You must add AWS access keys, configure protected branches, and setup deployment environments.

Then add the deploy keys from your SystemTests repository to enable the SubmitImage GitLab pipeline to download, and run the SystemTests.

Finally, add your AWS account ID as a CI/CD Variable.

screenshot of variables screen in gitlab

.gitlab-ci.yml for deploying to AWS

Go to your SubmitImage repository in your terminal, and create a branch named after your Jira issue ID.

git checkout -b IM-8

Create a .gitlab-ci.yml file with the following yaml. This defines a deployment workflow for your Test, Staging, and Production environments. You must update the git clone line for SystemTests to be your SystemTests repository.

stages:
  - merge-request
  - run-unit-tests
#US-WEST-1
  - deploy-us-west-1
  - test-us-west-1
#US-EAST-2
  - deploy-us-east-2
  - test-us-east-2
#US-WEST-2
  - deploy-us-west-2
  - test-us-west-2
#US-EAST-1
  - deploy-us-east-1
  - test-us-east-1
#CA-CENTRAL-1
  - deploy-ca-central-1
  - test-ca-central-1

merge-request-pipeline-job:
  stage: merge-request
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - echo "This pipeline always succeeds and enables merge"
    - echo true

run-unit-tests:
  stage: run-unit-tests
  image: golang:buster
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  script:
    - cd submitImage
    - go test ./opendevopslambda/...

#US-WEST-1
deploy-us-west-1:
  stage: deploy-us-west-1
  environment: test-us-west-1
  image: python:latest
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
    - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
    - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz
    - export PATH=$PATH:/usr/local/go/bin
    - go version
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --region us-west-1
    - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage  --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-1 --no-fail-on-empty-changeset

      #test-us-west-1:
      #  stage: test-us-west-1
      #  environment: test-us-west-1
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-west-1

#US-EAST-2
deploy-us-east-2:
  stage: deploy-us-east-2
  environment: test-us-east-2
  image: python:latest
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
    - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
    - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz
    - export PATH=$PATH:/usr/local/go/bin
    - go version
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --region us-east-2
    - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage  --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-2 --no-fail-on-empty-changeset

      #test-us-east-2:
      #  stage: test-us-east-2
      #  environment: test-us-east-2
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-east-2

#US-WEST-2
deploy-us-west-2:
  stage: deploy-us-west-2
  environment: production-us-west-2
  image: python:latest
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
    - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
    - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz
    - export PATH=$PATH:/usr/local/go/bin
    - go version
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --region us-west-2
    - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage  --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-2 --no-fail-on-empty-changeset

      #test-us-west-2:
      #  stage: test-us-west-2
      #  environment: production-us-west-2
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-west-2

#US-EAST-1
deploy-us-east-1:
  stage: deploy-us-east-1
  environment: production-us-east-1
  image: python:latest
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
    - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
    - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz
    - export PATH=$PATH:/usr/local/go/bin
    - go version
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --region us-east-1
    - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage  --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-1 --no-fail-on-empty-changeset

      #test-us-east-1:
      #  stage: test-us-east-1
      #  environment: production-us-east-1
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-east-1

#CA-CENTRAL-1
deploy-central-1:
  stage: deploy-ca-central-1
  environment: production-ca-central-1
  image: python:latest
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
    - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
    - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz
    - export PATH=$PATH:/usr/local/go/bin
    - go version
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --region ca-central-1
    - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage  --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region ca-central-1 --no-fail-on-empty-changeset

      #test-central-1:
      #  stage: test-ca-central-1
      #  environment: production-ca-central-1
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=ca-central-1

The execution of the integration tests is commented out for now. The system tests will only pass when the entire application is deployed. Uncomment the integration test steps in your repository, and do another push to run the deployment pipeline after all components of ImageLabeller are deployed. You must update the git clone line for SystemTests to be your SystemTests repository.

Understanding a .gitlab-ci.yml file

This step executes unit tests that are a part of the SubmitImage repository.

unit-test-us-west-1:
  stage: unit-test-us-west-1
  environment: test-us-west-1
  image: golang:buster
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  script:
    - cd submitImage
    - go test ./opendevopslambda/...

This step deploys the SubmitImage AWS Lambda using AWS SAM. Notice the before_script section. This step runs before the script section, and can be used to install dependencies, and setup various tools.

deploy-us-west-1:
  stage: deploy-us-west-1
  environment: test-us-west-1
  image: python:latest
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
    - wget https://golang.org/dl/go1.16.6.linux-amd64.tar.gz
    - rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.6.linux-amd64.tar.gz
    - export PATH=$PATH:/usr/local/go/bin
    - go version
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file submit-image-packaged.yaml --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --region us-west-1
    - sam deploy --template-file submit-image-packaged.yaml --stack-name OpenDevOpsSubmitImage  --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-1 --no-fail-on-empty-changeset

This step downloads, and runs the integration tests in the SystemTests repository. You must update the git clone line for SystemTests to be your SystemTests repository.

test-us-west-1:
  stage: test-us-west-1
  environment: test-us-west-1
  image: golang:buster
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
    - cd systemtests
    - go test -v ./... -aws_region=us-west-1

The deploy token created earlier is referenced in the git clone line.

- git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git

Pushing to a feature branch

Run the following from the command line to push your changes to the IM-8 branch of your SubmitImage repository. Include the Jira issue ID in commit messages, and branch names to enable the Jira GitLab integration to keep track of what is happening in your project.

git add --all
git commit -m "IM-8 add .gitlab-ci.yml to SubmitImage"
git push -u origin IM-8

Click CI/CD, then Pipelines to see the pipeline run.

Create a merge request

Create a merge request to deploy to your production environments after GitLab deploys to your test environments. Choose the IM-8 branch.

Click CI/CD, then Pipelines to see the running merge request pipeline.

screenshot of running merge request oin gitlab

Merge the changes to mainline after the merge request pipeline completes. Click CI/CD, then Pipelines to see the running Production pipeline.

screenshot of running production pipeline in gitlab

Create a repository for InvokeLabeller AWS Lambda

Go to Jira, and create a new issue for adding a InvokeLabeller AWS Lambda repository to GitLab. Make note of the issue ID. IM-10 in this example.

screenshot of jira issue create repo "invokelabeller" in gitlab

Go to GitLab and click New project, then Create blank project. Add a Project name, and choose the appropriate group in Project URL. Click Create project to proceed.

screenshot of creating new project "invokelabeller" in gitlab

In your terminal go to your InvokeLabeller repository, and run the following to push your code to GitLab.

git add --all
git commit -m "IM-10 add InvokeLabeller to gitlab"
git remote add origin git@gitlab.com:pmmquickstartguides/invokelabeller.git
git branch -m mainline
git push -u origin mainline

You must add AWS access keys, configure protected branches, and setup deployment environments.

Then add the deploy keys from your SystemTests repository to enable the InvokeLabeller GitLab pipeline to download, and run the SystemTests.

Finally, add your AWS account ID as a CI/CD Variable.

.gitlab-ci.yml for deploying to AWS

Go to your InvokeLabeller repository in your terminal, and create a branch named after your Jira issue ID.

git checkout -b IM-10

stages:
  - merge-request
  - run-unit-tests
#US-WEST-1
  - deploy-us-west-1
  - test-us-west-1
#US-EAST-2
  - deploy-us-east-2
  - test-us-east-2
#US-WEST-2
  - deploy-us-west-2
  - test-us-west-2
#US-EAST-1
  - deploy-us-east-1
  - test-us-east-1
#CA-CENTRAL-1
  - deploy-ca-central-1
  - test-ca-central-1

merge-request-pipeline-job:
  stage: merge-request
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - echo "This pipeline always succeeds and enables merge"
    - echo true

run-unit-tests:
  stage: run-unit-tests
  image: python:3.8-buster
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install pytest
    - pip3 install moto
    - pip3 install -r tst/requirements.txt --user
  script:
    - python3 -m pytest -v tst/unit --junitxml=test-reports/report.xml

#US-WEST-1
deploy-us-west-1:
  stage: deploy-us-west-1
  environment: test-us-west-1
  image: python:3.8-buster
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --region us-west-1
    - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller  --s3-bucket open-devops-code-us-west-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-1 --no-fail-on-empty-changeset

      #test-us-west-1:
      #  stage: test-us-west-1
      #  environment: test-us-west-1
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-west-1

#US-EAST-2
deploy-us-east-2:
  stage: deploy-us-east-2
  environment: test-us-east-2
  image: python:3.8-buster
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --region us-east-2
    - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller  --s3-bucket open-devops-code-us-east-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-2 --no-fail-on-empty-changeset

      #test-us-east-2:
      #  stage: test-us-east-2
      #  environment: test-us-east-2
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-east-2

#US-WEST-2
deploy-us-west-2:
  stage: deploy-us-west-2
  environment: production-us-west-2
  image: python:3.8-buster
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --region us-west-2
    - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller  --s3-bucket open-devops-code-us-west-2-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-west-2 --no-fail-on-empty-changeset

      #test-us-west-2:
      #  stage: test-us-west-2
      #  environment: production-us-west-2
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-west-2

#US-EAST-1
deploy-us-east-1:
  stage: deploy-us-east-1
  environment: production-us-east-1
  image: python:3.8-buster
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --region us-east-1
    - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller  --s3-bucket open-devops-code-us-east-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region us-east-1 --no-fail-on-empty-changeset

      #test-us-east-1:
      #  stage: test-us-east-1
      #  environment: production-us-east-1
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=us-east-1

#CA-CENTRAL-1
deploy-central-1:
  stage: deploy-ca-central-1
  environment: production-ca-central-1
  image: python:3.8-buster
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
  script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - sam build
    - sam package --output-template-file invoke-labeller-packaged.yaml --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --region ca-central-1
    - sam deploy --template-file invoke-labeller-packaged.yaml --stack-name OpenDevOpsInvokeLabeller  --s3-bucket open-devops-code-ca-central-1-$AWS_ACCOUNT_ID --capabilities CAPABILITY_IAM --region ca-central-1 --no-fail-on-empty-changeset

      #test-central-1:
      #  stage: test-ca-central-1
      #  environment: production-ca-central-1
      #  image: golang:buster
      #  rules:
      #    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
      #  script:
      #    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      #    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      #    - git clone https://${gitlab_deploy_token}:${gitlab_deploy_password}@gitlab.com/pmmquickstartguides/systemtests.git
      #    - cd systemtests
      #    - go test -v ./... -aws_region=ca-central-1

Update src/app.py with AWS SageMaker endpoint

Open InvokeLabeller’s src/app.py file and look for query_endpoint. Change the endpoint_name, and client region_name to match your AWS SageMaker notebook.

def query_endpoint(img):
  endpoint_name = 'jumpstart-dft-image-labeller-endpoint'
  client = boto3.client(service_name='runtime.sagemaker', region_name='us-west-1')
  response = client.invoke_endpoint(EndpointName=endpoint_name, ContentType='application/x-image', Body=img)
  model_predictions = json.loads(response['Body'].read())['predictions'][0]
  return model_predictions

Pushing to a feature branch

Run the following from the command line to push your changes to the IM-10 branch of your InvokeLabeller repository. Include the Jira issue ID in commit messages, and branch names to enable the Jira GitLab integration to keep track of what is happening in your project.

git add --all
git commit -m "IM-10 add .gitlab-ci.yml to InvokeLabeller"
git push -u origin IM-10

Click CI/CD, then Pipelines to see the pipeline run.

screenshot of pipeline running in gitlab

Create a merge request

Create a merge request to deploy to your production environments after GitLab deploys to your test environments. Choose the IM-10 branch.

screenshot of creating merge request in gitlab

Merge the changes to mainline after the merge request pipeline completes. Click CI/CD, then Pipelines to see the running Production pipeline.

If you’ve made this far, congratulations! You just deployed ImageLabeller. The next step is to set up monitoring ImageLabeller with Opsgenie.

Warren Marusiak

Warren is a Canadian developer from Vancouver, BC with over 10 years of experience. He came to Atlassian from AWS in January of 2021.

Share this article

Next Topic

Monitor with Opsgenie

Featured

Jira

Confluence

Jira Service Management

Trello

Rovo NEW

Jira Product Discovery NEW

Compass NEW

Guard NEW

Loom NEW

Developers

Jira

Bitbucket

Compass NEW

Product Managers

Jira

Confluence

Jira Product Discovery NEW

IT professionals

Jira Service Management

Guard NEW

Business Teams

Jira

Confluence

Trello

Loom NEW

Leadership Teams

Jira

Confluence

Loom NEW

Jira Align

Teams

Software

Marketing

IT

solution

By team size

By industry

Why Atlassian

Integrations

Customers

FedRAMP

Resilience

Platform

Trust center

Resources

Customer Support

Find Partners

Migration Program

University

support

learn

Articles

Tutorials

Interactive Guides

Deploy ImageLabeller with GitLab

Warren Marusiak

Prerequisites

Public facing GitHub repositories with ImageLabeller code

Jira GitLab integration demo video

Integrate Jira and GitLab

Add SSH Key to GitLab

Create a repository for AWS S3 infrastructure

Add AWS access key

Configure protected branches for access to protected variables

Setup deployment environments

.gitlab-ci.yml for deploying to AWS

Understanding a .gitlab-ci.yml file

Stages

Jobs

Merge request pipeline

Pushing to a feature branch

Create a merge request

Create a repository for SystemTests

Add a Deploy token

Create a repository for SubmitImage AWS Lambda

.gitlab-ci.yml for deploying to AWS

Understanding a .gitlab-ci.yml file

Pushing to a feature branch

Create a merge request