[Testing] Automation flow by Robot Framework, GitHub Actions and Postman Monitor

   Robot Framework   GitHub Actions   Postman Monitor 

▌Introduction

---

 

Due to COVID-19, many people work from home and need to clock in or out online on working days.

Since I sometimes forget to clock in or out online and I am so lazy to open the HR website and enter user ID/PWD each time on it, I decided to make the automation clock in(out) flow by E2E testing stack.

 

I use the stack as following:

ü  Robot Framework

ü  Azure Container Registry

ü  GitHub Actions

ü  Postman Monitor

 

 

   

 

 

 

▌Steps

---

 

▋Record and prepare the E2E automation test

 

I use Katalon Recorder and Robot Framework to have both automation test as clock_in.robot and clock_out.robot.

 

I would like to run the tests in Docker environment, so I made a dockefile and could be built and published the Docker Image to a private Docker registry (e.q. Azure Container Registry).

 

See my previous article, [Robot Framework] Run E2E test by Chrome and SeleniumLibrary in Docker, for the implementation details.

 

Now, I have to find a way to pull the Docker Image and then run the tests automatically.

 

 

▋GitHub Actions

 

GitHub Actions supports amazing and easy way to automate the workflow.

This article, [GitHub] Github Actions - Workflow dependencies, shows how to pull and run a Docker Image by GitHub Actions workflow.

 

Here is the YAML file of workflow that has workflow dispatch event. I will talk about why using workflow dispatch instead of schedule event later.

 

 

▋docker_clock_in.yml

---

name: Docker Clock In

on:

  # schedule:

  #   - cron: '10 0 * * 1-5'

  workflow_dispatch:

    inputs:

      trigger:

        description: "The trigger"

        required: false

        default: "Postman Monitor"

      trigger_datetime:

        description: "The datetime for triggering the workflow"

        required: true  

jobs:

  clock_out:

    name: Clock In

    runs-on: ubuntu-18.04

    steps:

      - name: Print Environment Variables

        shell: bash

        run: |

          echo "Arguments are ${{ github.event.inputs.trigger }}, ${{ github.event.inputs.trigger_datetime }}"

      - name: Docker Run

        uses: addnab/docker-run-action@v3

        with:

          username: ${{ secrets.ACR_USERNAME }}

          password: ${{ secrets.ACR_PASSWORD }}

          registry: ${{ secrets.ACR_REGISTRY }}

          image: ${{ secrets.ACR_REGISTRY }}/robot-bank:latest

          options: --rm --shm-size=2gb

          run: |

            robot /usr/src/rf/tests/ClockInOut/clock_in.robot

 

And my GitHub Actions secrets:

 

 

After pushing the workflow YAML file to main/master branch, I can trigger the workflow (open the headless Chrome, go to HR website and then clock in) in three ways.
(Reference: Manually running a workflow)

1. GitHub CLI: gh workflow run <workflow>

2. Run the workflow in GitHub Actions UI

 

3. Run the workflow by sending the HttpPost request to

https://api.github.com/repos/<user_id>/<reponsitory_name>/actions/workflows/<workflow>.yml/dispatches

Notice that we have to set a Personal access token for Authorization header. 

(See GitHub Docs: Create a workflow dispatch event)

 

 

 

▋Why not use schedule event but workflow_dispatch?

 

I commented out the schedule event in docker_clock_in.yml, and use workflow_dispatch event instead. The reason is that schedule event has a uncertain delay time and won’t be trigger on the schedule time precisely. If your job can tolerates a delay time, schedule event might be a good way to trigger it. However, since I need to clock in before my office hours so I use Postman Monitor to automatically send the request for running the workflow.

 

 

 

▋Postman Monitor

 

Postman Monitor is the collection-based monitor that we can execute a collection of requests and collect the response time and result by scheduled time. Furthermore, it runs on Postman's cloud infrastructure, which is hosted by AWS. In other words, we do not need a live machine to execute the target request(s) by Monitor.

 

The following is my monitoring setup for requesting the clock-in workflow_dispatch API.

After dispatched the workflow, the workflow will pull and run the Docker Image that has automation tests for my clock-in or clock-out.

 

 

▌Summarize

---

 

There are many ways to accomplish this automation job.

I chose the stack/tools is because I don’t want to have extra cost for this routine job and I do want to save my time from it.

 

 

 

▌Reference

---

 ▋GitHub Docs: Events that trigger workflows

▋GitHub Docs: Create a workflow dispatch event

▋GitHub Docs: Manually running a workflow

▋Postman: Monitoring your APIs

 

 

 

[Notes] 2021 DevOpsDays Taipei

 DevOps   iThome  

下半年因為轉職，工作上忙碌了不少，還好當天可以排開其他會議，順利地參加今年的DevOpsDays。

TSMC DevOps Journey

台積電也在做數位轉型；可以想像在這種大型跟根深蒂固的文化下，要推廣DevOps文化一定是相對更困難的，這邊有提到一個重點是TSMC聘了一位矽谷回來的CIO，是個很正確及聰明的作法。從上做改革及轉型，會比從下來的更有力和快速。

 

談假設思維之下的開發者體驗

• DX: Developer Experience
• 新人犯錯，誰的責任？ 答案是整個團隊。
• 良好的DX，是透過團隊持續的協作、回饋和經驗傳承。
• 進行任務，需要退一步去看見全貌。
• 假設思維
• 頻繁低去做假設，並驗證它。
• 聆聽在該項目有經驗的人的意見，無論他是資深或資淺。
• 做出最接近目標的假設。
• 任務一定要訂出Milestone，而Milestone是建立在假設之上。
• 任何事情在完成到一定程度之後，會遇到"效率的邊界"（也就是即使在施力，它根本不會動），這時候可以再透過假設，去找到更適合施力的點。

雖然投影片內容好像跳來跳去，Ruddy老師其實在傳授心法。 在管理或是帶更大的團隊時，讓團隊有正確的思維和方向，會比鉅細靡遺的去管理任何小地方，來的更有效率。

 

 

Running and Operating Large Scale Real-time Data system on Cloud

 

• 如何開始建構大型Streaming的系統
• 先 POC 100K
• 迭代改進
• 可視性 => 可觀察整個系統的效能和瓶頸
• End to End
• Bottom-up thinking
• 建構團隊
• 初期要先以多職能團隊為主，除非緊急任務，否則不應該只讓特定人執行特定工作。
• 若是緊急的任務，再安排固定資源。
• 角色應該rotate。
• 初期應以讓團隊了解產品或系統的整個脈絡及流程，但是到了更大（如100人）團隊的時候，再來分職責及角色。
• 可視性及觀察性
• 大型分散式系統已不可能再一台一台機器看LOG。
• 講者有展示他們Monitor的工具及如何從中抓出一個高風險的BUG。

如何實踐品質內建之企業級DevOps流程架構

 

這一場主要是玉山銀行提出他們如何在內部整合包含派工、程式弱掃、容器安全性檢查、CI/CD及Monitor的流程。 印象比較深刻的是，他們大量使用JIRA作為整合的工具，這點價值非常高，因為等於IT和USER都可以在同一平台作業。

 

 

 

大型團隊落實 CI/CD 的挑戰

 

我每次都會參加安德魯大大的場子，因為分享內容除了技術，更著重於在處理人和事的看法和經驗上。 詳細的內容可參考 HackMD。

 

 

後記

這次的贈品都滿不錯的，我拿到了隨身碟、一條毛巾、旅行包和T-Shirt，也參加了一場Hands-on lab 來彌補幾個月鮮少寫code的時光  XDDD

[GitHub] Github Actions - Publish Docker images

  Github Action   Docker Image  

 

 

▌Introduction

---

 

This tutorial shows how to publish your docker images to Docker Hub or Azure Container Registry thru GithHub Actions.

 

 

 

▌Implement

---

 

▋Register Docker Hub/Create Azure Container Registry

 

1.  Sign up Docker Hub here and create an Access Token.

2.  Create Azure Container Registry by “How to create ACR by Azure CLI”.

 

 

Notice that we will store the credentials on GitHub later to push Docker Images.

 

Docker Registry	Credentials to store in GitHub
Docker Hub	Docker ID and Token (or your Docker Hub password, but not recommended)
Azure Container Registry	Username and password(or password2) of Access Keys. See Figure 1.

 

 

(Figure 1. Access Keys in ACR)

 

 

▋ GitHub: Setup Action secrets

 

Go to your GitHub repository: [Settings] -> [Secrets] and add the following New repository secret:

Repository secret	Value	Note
DOCKER_IMAGE_NAME	The name of your image	This is optional.
DOCKERHUB_USERNAME	Docker Hub ID
DOCKERHUB_PASSWORD	Docker Hub Access Token
ACR_REGISTRY	The ACR registry server	e.q. xxxx.azurecr.io Since we don’t want to expose our private Docker registry.
ACR_USERNAME	ACR username of Access keys
ACR_PASSWORD	ACR password of Access keys

 

The secrets will be used in our GitHub flow.

 

 

 

▋ GitHub Actions: Setup GitHub flow

 

Under your git repository, create the directory: .github/workflows/ and yaml file(s), each yaml file will be a work flow.

 

For example, I put 2 yaml files as following that will be 2 workflows as figure 2.

├── .github

|  ├── workflows

|  |  ├── push_docker_image_acr.yml

|  |  └── push_docker_image_dockerhub.yml

 

(Figure 2.)

 

 

PS. Of course you can merge the 2 workflows to one yaml file.

 

 

▋push_docker_image_dockerhub.yml

---

name: Publish Docker images to Docker Hub

on:

  push:

      branches: [ master ] # Set which branch to trigger this flow, e.q. [ master, develop ]

jobs:

  push_images_to_dockerhub:

    name: Push images to Docker Hub

    runs-on: ubuntu-18.04

    steps:

      - name: Checkout repo

        uses: actions/checkout@v2

      - name: Login to DockerHub

        uses: docker/login-action@v1

        with:

          username: ${{ secrets.DOCKERHUB_USERNAME }}

          password: ${{ secrets.DOCKERHUB_PASSWORD }}

      - name: Build and push image

        id: build_publish_image

        uses: docker/build-push-action@v2

        with:

          context: .

          file: ./dockerfile

          push: true

          tags: ${{ secrets.DOCKERHUB_USERNAME}}/${{ secrets.DOCKER_IMAGE_NAME }}:latest

There are 3 steps in the workflow:

1.  Checkout repository by actions/checkout.

2.  Login to Docker Hub by docker/login-action.

3.  Build Dockerfile and push image to Docker Hub by docker/build-push-action.

 

 

 

▋push_docker_image_acr.yml

---

name: Publish Docker images to ACR

on:

  push:

      branches: [ master ] # Set which branch to trigger this flow, e.q. [ master, develop ]

jobs:

  push_images_to_acr:

    name: Push images to ACR

    runs-on: ubuntu-18.04

    steps:

      - name: Checkout repo

        uses: actions/checkout@v2

      - name: Login to ACR

        uses: azure/docker-login@v1

        with:

          login-server: ${{ secrets.ACR_REGISTRY }}

          username: ${{ secrets.ACR_USERNAME }}

          password: ${{ secrets.ACR_PASSWORD }}

      - name: Build and push image

        id: build_publish_image

        uses: docker/build-push-action@v2

        with:

          context: .

          file: ./dockerfile

          push: true

          tags: ${{ secrets.ACR_REGISTRY }}//${{ secrets.DOCKER_IMAGE_NAME }}:latest

 

 

There are 3 steps on the workflow:

1.  Checkout repository by actions/checkout.

2.  Login to ACR by Docker Hub by azure/docker-login.

3.  Build Dockerfile and push image to Docker Hub by docker/build-push-action.

 

 

Furthermore, we can write commands to run more complex scenario:

 

steps

  - name: Pull and push openldap 

    id: publish_openldap

    run: |

      docker pull osixia/openldap:stable

      docker tag osixia/openldap:stable ${{ secrets.ACR_REGISTRY }}/openldap:stable

      docker push ${{ secrets.ACR_REGISTRY}}/openldap:stable

 

▋ See result and log on GitHub

 

Push the yaml files to GitHub and find the workflow result on Actions tab.

 

 

If the workflow fails, we will receive the email notification.

 

 

 

 

▌Reference

---

▋Publishing Docker images - GitHub Docs

▋Configure a GitHub action to create a container instance