[GitHub] Github Actions - Workflow dependencies

 Github Action   Workflow   Dependency 

 

▌Introduction

---

 

The jobs in a GitHub Actions: workflow by default run in parallel at the same time.

To run a job only when another job has completed, we can use needs keyword as following,

 

(The sample manifest comes from GitHub Docs: Creating dependent jobs)

jobs:

  setup:

    runs-on: ubuntu-latest

    steps:

      - run: ./setup_server.sh

  build:

    needs: setup

    runs-on: ubuntu-latest

    steps:

      - run: ./build_server.sh

  test:

    needs: build

    runs-on: ubuntu-latest

    steps:

      - run: ./test_server.sh

 

However, it needs some tricks on workflows dependencies, let us see the following example.

 

 

▌Implement

---

▋ Scenario

I am going to create 2 workflows:

1.  Publish Docker Image

2.  Run Docker Container (to execute some commands)

 

The second workflow depends on the result of the first one.

We will run the container after the Docker image publish successfully. In other words, if publishing fails, we won’t run the container.

 

▋ Constraints

 

1.  Currently (2021-11) GitHub Actions only supports putting the second workflow (that will be triggered by first one) on the “default branch”.

2.  To get the previous workflow’s state, use the value of github.event.workflow_run.conclusion.

 

 

▋ Workflow: Publish Docker Image

 

This is the first workflow and it is not what we will focus in the article, see [GitHub] Github Actions - Publish Docker images for more details.

 

▋publish_docker_image.yml

---

name: Publish Docker Image

on:

  push:

    branches: [ master ]

jobs:

  push_images_to_acr:

    name: Push images to ACR

    runs-on: ubuntu-18.04

    steps:

      - name: Checkout repo

        uses: actions/checkout@v2

      - name: Login to ACR

        uses: azure/docker-login@v1

        with:

          login-server: ${{ secrets.ACR_REGISTRY }}

          username: ${{ secrets.ACR_USERNAME }}

          password: ${{ secrets.ACR_PASSWORD }}

      - name: Build and push

        id: build_publish

        uses: docker/build-push-action@v2

        with:

          # context: .

          file: ./docker/dockerfile

          push: true

          tags: ${{ secrets.ACR_REGISTRY }}/my-demo:latest

 

 

 

▋ Workflow: Run Docker Container

The second workflow will need to watch the if the dependent workflow has been closed by

on:

  workflow_run:

    workflows:

      - The dependent workflow name

    types:

      - completed

 

▋run_docker_constainer.yml

---

name: Run Docker Container

on:

  workflow_run:

    workflows:

      - Publish Docker Image

    types:

      - completed

jobs:

  run_docker_container:

    name: Run

    if: ${{ github.event.workflow_run.conclusion == 'success' }}

    runs-on: ubuntu-18.04

    steps:

      - name: Run

        uses: addnab/docker-run-action@v3

        with:

          username: ${{ secrets.ACR_USERNAME }}

          password: ${{ secrets.ACR_PASSWORD }}

          registry: ${{ secrets.ACR_REGISTRY }}

          image: ${{ secrets.ACR_REGISTRY }}/my-demo:latest

          # options:

          run: |

            echo "The container is running."

 

Like we mentioned in Constraints, we have to enable the workflow (put the manifest file) in the default branch, e.q. master or main.

Notice that the above manifest used addnab/docker-run-action as the “Docker Run Action” Or we can use the default Action like this,

---

name: Run Docker Container

on:

  workflow_run:

    workflows:

      - Publish Docker Image

    types:

      - completed

jobs:

  run_docker_container:

    name: Run

    if: ${{ github.event.workflow_run.conclusion == 'success' }}

    runs-on: ubuntu-18.04

    container: karatejb/my-demo:latest

    steps:

      - name: Dcoker run

        run: |

          echo "The container is running."

 

 

▌Reference

---

▋GitHub Docs: workflow_run

▋addnab/docker-run-action

▋Using Docker Containers In Jobs - GitHub Actions

▋Workflow_run not working as expected

▋How to use the GitHub Actions `workflow_run` event?

▋actions/runner: Secrets cannot be used to condition job runs #520

▋Pass Github secrets to a docker github action

▋Workflow_run completed event triggered by failed workflow

 

 

 

 

 

 

 

 

[Notes] 2021 DevOpsDays Taipei

 DevOps   iThome  

下半年因為轉職，工作上忙碌了不少，還好當天可以排開其他會議，順利地參加今年的DevOpsDays。

TSMC DevOps Journey

台積電也在做數位轉型；可以想像在這種大型跟根深蒂固的文化下，要推廣DevOps文化一定是相對更困難的，這邊有提到一個重點是TSMC聘了一位矽谷回來的CIO，是個很正確及聰明的作法。從上做改革及轉型，會比從下來的更有力和快速。

 

談假設思維之下的開發者體驗

• DX: Developer Experience
• 新人犯錯，誰的責任？ 答案是整個團隊。
• 良好的DX，是透過團隊持續的協作、回饋和經驗傳承。
• 進行任務，需要退一步去看見全貌。
• 假設思維
• 頻繁低去做假設，並驗證它。
• 聆聽在該項目有經驗的人的意見，無論他是資深或資淺。
• 做出最接近目標的假設。
• 任務一定要訂出Milestone，而Milestone是建立在假設之上。
• 任何事情在完成到一定程度之後，會遇到"效率的邊界"（也就是即使在施力，它根本不會動），這時候可以再透過假設，去找到更適合施力的點。

雖然投影片內容好像跳來跳去，Ruddy老師其實在傳授心法。 在管理或是帶更大的團隊時，讓團隊有正確的思維和方向，會比鉅細靡遺的去管理任何小地方，來的更有效率。

 

 

Running and Operating Large Scale Real-time Data system on Cloud

 

• 如何開始建構大型Streaming的系統
• 先 POC 100K
• 迭代改進
• 可視性 => 可觀察整個系統的效能和瓶頸
• End to End
• Bottom-up thinking
• 建構團隊
• 初期要先以多職能團隊為主，除非緊急任務，否則不應該只讓特定人執行特定工作。
• 若是緊急的任務，再安排固定資源。
• 角色應該rotate。
• 初期應以讓團隊了解產品或系統的整個脈絡及流程，但是到了更大（如100人）團隊的時候，再來分職責及角色。
• 可視性及觀察性
• 大型分散式系統已不可能再一台一台機器看LOG。
• 講者有展示他們Monitor的工具及如何從中抓出一個高風險的BUG。

如何實踐品質內建之企業級DevOps流程架構

 

這一場主要是玉山銀行提出他們如何在內部整合包含派工、程式弱掃、容器安全性檢查、CI/CD及Monitor的流程。 印象比較深刻的是，他們大量使用JIRA作為整合的工具，這點價值非常高，因為等於IT和USER都可以在同一平台作業。

 

 

 

大型團隊落實 CI/CD 的挑戰

 

我每次都會參加安德魯大大的場子，因為分享內容除了技術，更著重於在處理人和事的看法和經驗上。 詳細的內容可參考 HackMD。

 

 

後記

這次的贈品都滿不錯的，我拿到了隨身碟、一條毛巾、旅行包和T-Shirt，也參加了一場Hands-on lab 來彌補幾個月鮮少寫code的時光  XDDD

[Robot Framework] Run E2E test by Chrome and SeleniumLibrary in Docker

  Robot Framework   SeleniumLibrary   Chrome   Docker  

 

 

▌Introduction

---

 

We will use Robot Framework to run E2E test in a Docker container.

Here are the frameworks and toolkit that we will use.

 

·     Robot Framework: An open source automation framework

·     SeleniumLibrary: Web testing library for Robot Framework.

·     ChromeDriver: WebDriver for Chrome

·     XVFB: It performs all graphical operations in virtual memory without showing any screen output.

·     Katalon Recorder: Web extension for automating actions and automated testing on the browser

 

 

 

▌Environment

---

▋Robot Framework 4.1.2

▋ChromeDriver 95.0.4638.17

▋Google Chrome 95.0.4638.54

▋SeleniumLibrary 5.1.3

 

 

 

 

 

▌Implement

---

 

▋Dockerfile

 

Here is a sample dockerfile that installs Chrome, ChromeDriver and XVFB on the fly.

FROM python:3.7

COPY src/env/drivers/Linux/chromedriver /usr/local/bin

RUN chmod +x /usr/local/bin/

# Install python packages

RUN /usr/local/bin/python -m pip install --upgrade pip

RUN pip install -r requirements.txt

# Install dependencies of Chrome driver and chrome

# Notice that xvfb is an in-memory display server for Linux

RUN apt-get update && \

    apt-get install -y libnss3 libdbus-1-dev && \

    apt-get install -y xdg-utils libgbm1 libasound2 fonts-liberation xvfb

# Install google-chrome

RUN wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb \

    && dpkg -i google-chrome*.deb \

    && rm google-chrome*.deb

# (Optional) Start dbus in WSL, see https://github.com/microsoft/WSL/issues/376#issuecomment-295933508

RUN /etc/init.d/dbus start

ENV PATH=/usr/local/bin:$PATH

 

▋Requirement: Python packages

 

Here are the python packages we need to run recorded E2E test by Chrome with Robot Framework.

certifi==2020.4.5.1

chardet==3.0.4

idna==2.9

requests==2.23.0

robotframework==4.1.2

robotframework-requests==0.7.0

urllib3==1.25.9

robotframework-seleniumlibrary==5.1.3

robotframework-xvfb==1.2.2

 

▋Export Test Case to Robot Framework from Katalon Recoder

 

One of the fast ways to have an E2E test case is recording the end-user’s steps by Selenium IDE that can be exported or playback. Take Katalon Recorder for example, we can export a test case as Robot Framework format (.robot file) like following.

 

 

However, if we want to run the test in a docker environment, we will need to modify some code of the .robot file.

 

 

▋Update Test Case

 

The original test case exported from Katalon Recorder is as following,

 

*** Settings ***

Library  SeleniumLibrary

*** Variables ***

${BROWSER}   chrome

${SELSPEED}  0.0s

*** Test Cases ***

My Test

    [Setup]  Run Keywords  Open Browser  https://172.19.160.1:5001/OpenId/Login  ${BROWSER}

    ...              AND   Set Selenium Speed  ${SELSPEED}

    # ... skip

    [Teardown]  Close Browser

*** Keywords ***

# ... skip

 

 

 

To run Chrome in a docker container (Like Debian in this sample), we need to run it with headless and no-sandbox arguments.   

·        --headless: Run Chrome without GUI.

·        --no-sandbox: Sandbox removes unnecessary privileges from the processes that don't need them in Chrome. Disable Sandboxing will run google chrome as a root user.

 

Let’s create a module that can create and return a new WebDriver instance with the above arguments.

 

▋ChromeConfiguration.py

from selenium.webdriver.chrome.options import Options

def config():

    options = Options()

    options.add_argument('--headless')

    options.add_argument("--no-sandbox")

    options.add_argument('--disable-gpu')

    options.add_argument('--disable-dev-shm-usage')

    options.add_argument("--window-size=1920,1080")

    options.add_argument('--log-level=ALL')

    return options

def serviceargs():

    return ["--verbose", "--log-path=/var/log/chromedriver.log"]

 

 

Then we can use it in our test case.

 

▋MyTest.robot

 

*** Settings ***

Library  SeleniumLibrary

Library       ChromeConfiguration.py

Library       XvfbRobot

*** Variables ***

${BROWSER}   chrome

${SELSPEED}  0.0s

*** Test Cases ***

My Test

    ${chrome_options}    ChromeConfiguration.Config

    ${args}    ChromeConfiguration.Serviceargs

    Start Virtual Display    1920    1080

    Create WebDriver    Chrome    chrome_options=${chrome_options}    service_args=${args}

    Go To    https://172.19.160.1:5001/OpenId/Login

    # ... skip

    [Teardown]  Close Browser

*** Keywords ***

# ... skip

Notice that after the WebDriver created, we must use Go To keyword to open the URL with the WebDriver instance.

Create WebDriver    Chrome    chrome_options=${chrome_options}    service_args=${args}

Go To    https://172.19.160.1:5001/OpenId/Login  

 

Or we can replace the above 2 line with Open Browser keyword.

Open Browser    https://172.19.160.1:5001/OpenId/Login    ${BROWSER}    options=${chrome_options}

 

 

If you encounter any problem, you can open /var/log/chromedriver.log to see what happened.

And please make sure the Chrome and ChromeDriver are the same version.

For example, I am having Chrome 95 for my testing.

 

 

 

 

▌Reference

---

 

▋Robot Framework

▋Robot Framework: SeleniumLibrary

▋Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed

▋WSL: dbus doesn't seem to work