[GitHub] Github Actions - Workflow dependencies

 Github Action   Workflow   Dependency 

 

▌Introduction

---

 

The jobs in a GitHub Actions: workflow by default run in parallel at the same time.

To run a job only when another job has completed, we can use needs keyword as following,

 

(The sample manifest comes from GitHub Docs: Creating dependent jobs)

jobs:

  setup:

    runs-on: ubuntu-latest

    steps:

      - run: ./setup_server.sh

  build:

    needs: setup

    runs-on: ubuntu-latest

    steps:

      - run: ./build_server.sh

  test:

    needs: build

    runs-on: ubuntu-latest

    steps:

      - run: ./test_server.sh

 

However, it needs some tricks on workflows dependencies, let us see the following example.

 

 

▌Implement

---

▋ Scenario

I am going to create 2 workflows:

1.  Publish Docker Image

2.  Run Docker Container (to execute some commands)

 

The second workflow depends on the result of the first one.

We will run the container after the Docker image publish successfully. In other words, if publishing fails, we won’t run the container.

 

▋ Constraints

 

1.  Currently (2021-11) GitHub Actions only supports putting the second workflow (that will be triggered by first one) on the “default branch”.

2.  To get the previous workflow’s state, use the value of github.event.workflow_run.conclusion.

 

 

▋ Workflow: Publish Docker Image

 

This is the first workflow and it is not what we will focus in the article, see [GitHub] Github Actions - Publish Docker images for more details.

 

▋publish_docker_image.yml

---

name: Publish Docker Image

on:

  push:

    branches: [ master ]

jobs:

  push_images_to_acr:

    name: Push images to ACR

    runs-on: ubuntu-18.04

    steps:

      - name: Checkout repo

        uses: actions/checkout@v2

      - name: Login to ACR

        uses: azure/docker-login@v1

        with:

          login-server: ${{ secrets.ACR_REGISTRY }}

          username: ${{ secrets.ACR_USERNAME }}

          password: ${{ secrets.ACR_PASSWORD }}

      - name: Build and push

        id: build_publish

        uses: docker/build-push-action@v2

        with:

          # context: .

          file: ./docker/dockerfile

          push: true

          tags: ${{ secrets.ACR_REGISTRY }}/my-demo:latest

 

 

 

▋ Workflow: Run Docker Container

The second workflow will need to watch the if the dependent workflow has been closed by

on:

  workflow_run:

    workflows:

      - The dependent workflow name

    types:

      - completed

 

▋run_docker_constainer.yml

---

name: Run Docker Container

on:

  workflow_run:

    workflows:

      - Publish Docker Image

    types:

      - completed

jobs:

  run_docker_container:

    name: Run

    if: ${{ github.event.workflow_run.conclusion == 'success' }}

    runs-on: ubuntu-18.04

    steps:

      - name: Run

        uses: addnab/docker-run-action@v3

        with:

          username: ${{ secrets.ACR_USERNAME }}

          password: ${{ secrets.ACR_PASSWORD }}

          registry: ${{ secrets.ACR_REGISTRY }}

          image: ${{ secrets.ACR_REGISTRY }}/my-demo:latest

          # options:

          run: |

            echo "The container is running."

 

Like we mentioned in Constraints, we have to enable the workflow (put the manifest file) in the default branch, e.q. master or main.

Notice that the above manifest used addnab/docker-run-action as the “Docker Run Action” Or we can use the default Action like this,

---

name: Run Docker Container

on:

  workflow_run:

    workflows:

      - Publish Docker Image

    types:

      - completed

jobs:

  run_docker_container:

    name: Run

    if: ${{ github.event.workflow_run.conclusion == 'success' }}

    runs-on: ubuntu-18.04

    container: karatejb/my-demo:latest

    steps:

      - name: Dcoker run

        run: |

          echo "The container is running."

 

 

▌Reference

---

▋GitHub Docs: workflow_run

▋addnab/docker-run-action

▋Using Docker Containers In Jobs - GitHub Actions

▋Workflow_run not working as expected

▋How to use the GitHub Actions `workflow_run` event?

▋actions/runner: Secrets cannot be used to condition job runs #520

▋Pass Github secrets to a docker github action

▋Workflow_run completed event triggered by failed workflow

 

 

 

 

 

 

 

 

[Robot Framework] Run E2E test by Chrome and SeleniumLibrary in Docker

  Robot Framework   SeleniumLibrary   Chrome   Docker  

 

 

▌Introduction

---

 

We will use Robot Framework to run E2E test in a Docker container.

Here are the frameworks and toolkit that we will use.

 

·     Robot Framework: An open source automation framework

·     SeleniumLibrary: Web testing library for Robot Framework.

·     ChromeDriver: WebDriver for Chrome

·     XVFB: It performs all graphical operations in virtual memory without showing any screen output.

·     Katalon Recorder: Web extension for automating actions and automated testing on the browser

 

 

 

▌Environment

---

▋Robot Framework 4.1.2

▋ChromeDriver 95.0.4638.17

▋Google Chrome 95.0.4638.54

▋SeleniumLibrary 5.1.3

 

 

 

 

 

▌Implement

---

 

▋Dockerfile

 

Here is a sample dockerfile that installs Chrome, ChromeDriver and XVFB on the fly.

FROM python:3.7

COPY src/env/drivers/Linux/chromedriver /usr/local/bin

RUN chmod +x /usr/local/bin/

# Install python packages

RUN /usr/local/bin/python -m pip install --upgrade pip

RUN pip install -r requirements.txt

# Install dependencies of Chrome driver and chrome

# Notice that xvfb is an in-memory display server for Linux

RUN apt-get update && \

    apt-get install -y libnss3 libdbus-1-dev && \

    apt-get install -y xdg-utils libgbm1 libasound2 fonts-liberation xvfb

# Install google-chrome

RUN wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb \

    && dpkg -i google-chrome*.deb \

    && rm google-chrome*.deb

# (Optional) Start dbus in WSL, see https://github.com/microsoft/WSL/issues/376#issuecomment-295933508

RUN /etc/init.d/dbus start

ENV PATH=/usr/local/bin:$PATH

 

▋Requirement: Python packages

 

Here are the python packages we need to run recorded E2E test by Chrome with Robot Framework.

certifi==2020.4.5.1

chardet==3.0.4

idna==2.9

requests==2.23.0

robotframework==4.1.2

robotframework-requests==0.7.0

urllib3==1.25.9

robotframework-seleniumlibrary==5.1.3

robotframework-xvfb==1.2.2

 

▋Export Test Case to Robot Framework from Katalon Recoder

 

One of the fast ways to have an E2E test case is recording the end-user’s steps by Selenium IDE that can be exported or playback. Take Katalon Recorder for example, we can export a test case as Robot Framework format (.robot file) like following.

 

 

However, if we want to run the test in a docker environment, we will need to modify some code of the .robot file.

 

 

▋Update Test Case

 

The original test case exported from Katalon Recorder is as following,

 

*** Settings ***

Library  SeleniumLibrary

*** Variables ***

${BROWSER}   chrome

${SELSPEED}  0.0s

*** Test Cases ***

My Test

    [Setup]  Run Keywords  Open Browser  https://172.19.160.1:5001/OpenId/Login  ${BROWSER}

    ...              AND   Set Selenium Speed  ${SELSPEED}

    # ... skip

    [Teardown]  Close Browser

*** Keywords ***

# ... skip

 

 

 

To run Chrome in a docker container (Like Debian in this sample), we need to run it with headless and no-sandbox arguments.   

·        --headless: Run Chrome without GUI.

·        --no-sandbox: Sandbox removes unnecessary privileges from the processes that don't need them in Chrome. Disable Sandboxing will run google chrome as a root user.

 

Let’s create a module that can create and return a new WebDriver instance with the above arguments.

 

▋ChromeConfiguration.py

from selenium.webdriver.chrome.options import Options

def config():

    options = Options()

    options.add_argument('--headless')

    options.add_argument("--no-sandbox")

    options.add_argument('--disable-gpu')

    options.add_argument('--disable-dev-shm-usage')

    options.add_argument("--window-size=1920,1080")

    options.add_argument('--log-level=ALL')

    return options

def serviceargs():

    return ["--verbose", "--log-path=/var/log/chromedriver.log"]

 

 

Then we can use it in our test case.

 

▋MyTest.robot

 

*** Settings ***

Library  SeleniumLibrary

Library       ChromeConfiguration.py

Library       XvfbRobot

*** Variables ***

${BROWSER}   chrome

${SELSPEED}  0.0s

*** Test Cases ***

My Test

    ${chrome_options}    ChromeConfiguration.Config

    ${args}    ChromeConfiguration.Serviceargs

    Start Virtual Display    1920    1080

    Create WebDriver    Chrome    chrome_options=${chrome_options}    service_args=${args}

    Go To    https://172.19.160.1:5001/OpenId/Login

    # ... skip

    [Teardown]  Close Browser

*** Keywords ***

# ... skip

Notice that after the WebDriver created, we must use Go To keyword to open the URL with the WebDriver instance.

Create WebDriver    Chrome    chrome_options=${chrome_options}    service_args=${args}

Go To    https://172.19.160.1:5001/OpenId/Login  

 

Or we can replace the above 2 line with Open Browser keyword.

Open Browser    https://172.19.160.1:5001/OpenId/Login    ${BROWSER}    options=${chrome_options}

 

 

If you encounter any problem, you can open /var/log/chromedriver.log to see what happened.

And please make sure the Chrome and ChromeDriver are the same version.

For example, I am having Chrome 95 for my testing.

 

 

 

 

▌Reference

---

 

▋Robot Framework

▋Robot Framework: SeleniumLibrary

▋Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed

▋WSL: dbus doesn't seem to work

 

 

 

 

 

 

 

 

 

 

 

[Dockerfile] Trouble shooting - cannot get environment variables and permission denied on Entrypoint

 

 Docker   Dockerfile   sudo  

 

 

▌Problem

---

 

We have a Dockerfile that will switch user to run a shell script on ENTRYPOINT, however we got the 2 problems,

 

1.  Cannot get the environment variables in the shell script.

2.  Get permission denied after we preserve the environment variables when sudo user.

 

Here are the sample files,

 

▋Dockerfile

FROM postgres:11

ENV POSTGRES_USER postgres

ENV POSTGRES_PASSWORD xxxxx

# Add user: postgres

RUN sudo adduser postgres sudo

EXPOSE 5432

ENTRYPOINT ["/bin/sh", "-c", "sudo -u postgres sh callback.sh"]

 

▋Shell script (callback.sh)

#!/bin/bash echo POSTGRES_USER=$POSTGRES_USER

echo POSTGRES_PASSWORD=$POSTGRES_PASSWORD

echo "some logs" >> ~/my_log

The container from the above Dockerfile could not get the environment variables $POSTGRES_USER and $POSTGRES_PASSWORD, and finally output,

 

POSTGRES_USER= POSTGRES_PASSWORD=

 

And the log file will be located at /var/lib/postgresql/my_log.

 

 

▌Environment

---

 

▋Docker desktop 2.3.0.3

 

 

 

▌Solution

---

 

▋Preserving environment variables

 

The problem was due to not preserving environment variables when switching to other user.

 

So we have to preserve the existing environment variables by adding the argument -U when sudo. (See sudo manual)

 

-E, --preserve-env: Indicates to the security policy that the user wishes to preserve their existing environment variables. The security policy may return an error if the user does not have permission to preserve the environment.

 

So we must update the following line of Dockerfile,

ENTRYPOINT ["/bin/sh", "-c", "sudo -E -u postgres sh callback.sh"]

 

But it caused the second problem: Permission denied when writing log to ~/my_log.

The output became as following,

 

POSTGRES_USER=postgres POSTGRES_PASSWORD=xxxxx callback.sh: 3: callback.sh: cannot create /root/my_log: Permission denied

 

 

As you can see the ~/ is changed to /root/ but not /var/lib/postgresql/, so that the user: postgres could not have the permission on ~/.

 

Finally I solved the problems by updating the Dockerfile as following,

 

 

▋Dockerfile

FROM postgres:11

ENV POSTGRES_USER postgres

ENV POSTGRES_PASSWORD xxxxx

 

# Add user: postgres

RUN sudo adduser postgres sudo

EXPOSE 5432

ENTRYPOINT ["/bin/sh", "-c", "sudo -E -u postgres sh callback.sh"]

# Or use this # ENTRYPOINT ["/bin/sh", "-c", "su --preserve-environment - postgres callback.sh'"]

 

▋Shell script (callback.sh)

#!/bin/bash echo POSTGRES_USER=$POSTGRES_USER echo POSTGRES_PASSWORD=$POSTGRES_PASSWORD echo "some logs" >> /var/lib/postgresql/my_log

 

[PostgreSQL] pgAgent - Scheduling agent

 PostgreSQL   pgAgent 

▌Introduction

---

pgAgent is a job scheduling agent for PostgreSQL, it can has multiple steps in a job, including SQL script,  batch or shell scripts.

This article will show how to install pgAgent inside a PostgreSQL Docker container.

▌Environment

---

▋Docker Engine Community 19.03.5

▋PostgreSQL Docker Official image 10/11

▌Implement

---

▋Install

First we have to install pgAgent in the container,

$ docker exec -it demo-postgres bash

$ apt-get update && apt-get install pgagent

Then install the pgAgent extension.

▋Install pgAgent extension

CREATE EXTENSION pgagent IF NOT EXISTS;

CREATE LANGUAGE plpgsql IF NOT EXISTS;

The 【pgAgent Jobs】 shows after installing the extension,

And the new schema named “pgagent” and tables were created.

▋(Optional) Grant permission

It is recommended to create a new role for using pgAgent.

For example,

GRANT USAGE ON SCHEMA pgagent TO pgagent_user;

GRANT ALL ON SCHEMA pgagent TO pgagent_user;

GRANT ALL ON all tables IN SCHEMA pgagent to pgagent_user;

GRANT ALL ON ALL SEQUENCES IN SCHEMA pgagent TO pgagent_user;

GRANT ALL ON ALL FUNCTIONS IN SCHEMA pgagent TO pgagent_user;

GRANT CONNECT ON DATABASE postgres TO pgagent_user;

▋Start pgAgent

Back to the container, use the following command to start the agent.

Note that using "127.0.0.1" instead of "localhost" to start pgagent.

$ cd /usr/bin

$ pgagent hostaddr=127.0.0.1 dbname=postgres user=<user_name> -s pgagent_log.log

We can check if the connection between pgAgent and database was established by,

SELECT * FROM pgagent.pga_jobagent;

 

▋Scheduling a job

Here is an example for scheduling a cleaning-table job.

1.Create a new pgAgent Job

2.【General】: Name the new job and set Job class, Host agent(Optional).

3.【Steps】: We can set multiple steps to run SQL scripts or batches here.

Notice that if we wanna run SQL script, we have to enter the database connection string, e.q.

host=localhost port=5432 dbname=postgres connect_timeout=10 user='xxxxx' password='my_pwd'

PS. We can use pgpass (The Password File) to skip storing the password in a pgAgent job. We will talk about it later in this article.

Navigate to Code tab, enter the SQL/Batch script.

4.【Schedules】: Set the job’s start/end time and how it repeats.

5.Save and get the new scheduling job.

For more details on creating a pgAgent job, see official document.

▋Logs

We can check the logs of a job from these tables:

SELECT * FROM pgagent.pga_joblog

SELECT * FROM pgagent.pga_jobsteplog

SELECT * FROM pgagent.pga_exception

▋Use Password File to protect the pwd

To not saving password in a pgAgent job, we will create .pgpass (The Password File) inside the PostgreSQL container.

$ sudo su postgres

$ echo localhost:5432:*:postgres:my_pwd >> /var/lib/postgresql/data/.pgpass

$ chmod 600 /var/lib/postgresql/data/.pgpass

$ chown postgres:postgres /var/lib/postgresql/data/.pgpass

$ export PGPASSFILE='/var/lib/postgresql/data/.pgpass'

 

Then we will be able to remove the password from the connection string of pgAgent job.

We can test if it works by connecting to database without entering password:

$ psql -h localhost -U postgres <db_name>

In Windows, you can use the Password File as following, $ cd %APPDATA% $ mkdir Postgres $ cd Postgres $ echo localhost:5432:*:postgres:my_pwd >> pgpass.conf $ SETX PGPASSFILE %APPDATA%\Postgres\pgpass.conf

▋Create/Drop a pgAgent job by SQL script

We can export the create sql script of a pgAgent job by pgAdmin as following,

Here are a pgAgent job’s create/drop sql script for reference.

/* Create pgAgent job */

DO $$

DECLARE

    jid integer;

    scid integer;

BEGIN

-- Creating a new job

INSERT INTO pgagent.pga_job(

    jobjclid, jobname, jobdesc, jobhostagent, jobenabled

) VALUES (

    1::integer, 'Routine Clean'::text, ''::text, ''::text, true

) RETURNING jobid INTO jid;

-- Steps

-- Inserting a step (jobid: NULL)

INSERT INTO pgagent.pga_jobstep (

    jstjobid, jstname, jstenabled, jstkind,

    jstconnstr, jstdbname, jstonerror,

    jstcode, jstdesc

) VALUES (

    jid, 'Clean_News'::text, true, 's'::character(1),

    'host=localhost port=5432 dbname=postgres connect_timeout=10 user=''postgres'''::text, ''::name, 'f'::character(1),

    'DELETE FROM public."News"'::text, ''::text

) ;

-- Schedules

-- Inserting a schedule

INSERT INTO pgagent.pga_schedule(

    jscjobid, jscname, jscdesc, jscenabled,

    jscstart, jscend,    jscminutes, jschours, jscweekdays, jscmonthdays, jscmonths

) VALUES (

    jid, 'Daily'::text, ''::text, true,

    '2020-04-24 06:14:44+00'::timestamp with time zone, '2020-04-30 05:51:17+00'::timestamp with time zone,

    -- Minutes

    ARRAY[false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,true,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false]::boolean[],

    -- Hours

    ARRAY[false,false,false,false,false,false,false,false,false,false,false,true,false,false,false,false,false,false,false,false,false,false,false,false]::boolean[],

    -- Week days

    ARRAY[false,false,false,false,false,false,false]::boolean[],

    -- Month days

    ARRAY[false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false]::boolean[],

    -- Months

    ARRAY[false,false,false,false,false,false,false,false,false,false,false,false]::boolean[]

) RETURNING jscid INTO scid;

END

$$;

 

/* Delete pgAgent job */

DO $$

DECLARE

    jname VARCHAR(50) :='Routine Clean';

    jid INTEGER;

BEGIN

SELECT "jobid" INTO jid from pgagent."pga_job"

WHERE "jobname"=jname;

DELETE FROM pgagent."pga_schedule"

WHERE "jscjobid"=jid;

DELETE FROM pgagent.pga_jobstep

WHERE "jstjobid"=jid;

DELETE FROM pgagent."pga_job"

WHERE "jobid"=jid;

END

$$;

▌Reference

---

▋pgadmin.org - Creating a pgAgent Job

▋Install pgAgent on Postgres 10 (Debian Linux)

▋pgAgent service for PostgreSQL does not run

▋SETTING UP PGAGENT AND DOING SCHEDULED BACKUPS