顯示具有 Architecture 標籤的文章。 顯示所有文章
顯示具有 Architecture 標籤的文章。 顯示所有文章

2022年6月4日 星期六

[Architect] 產品及專案架構設計思維

最近常常被問到,如何為一個產品或專案設計好的架構? 

我回答得很簡短:「首先要了解目的,然後因應當時環境時間資源,評估可以達成這個目的的架構和流程,就是當時適合的方式。」

然後就會得到



趁著最近比較有空,我把一些自己的想法整理在這篇文章。
但是在更往下之前,我想先談談自己對於 "架構" 這個詞的定義。


架構

我們知道產品和專案都有自己的生命周期。 而我自己心目中的架構是支撐它們可以在各自生命週期運行良好的軟硬體、流程和方法。  例如:

  • 功能開發完成後,團隊可以多久和以多少信心佈署到正式環境?
  • 新進員工加入多久,可以獨立開始作業?
  • 線上有問題時,流程是什麼? 
  • 不同團隊協作流程順暢嗎?
  • 軟硬體架構安全嗎? 有考慮到業務成長量的對應擴充方式嗎?
  • 開發團隊的價值有傳遞給客戶嗎? 我知道客戶在抱怨還是幫我們說話嗎?

所以如果你期望看到底下有軟硬體架構的解決方案,很抱歉並沒有 XDD
這篇文章講得是我個人的思維。


目的

講到"目的",通常我們想到的是:「如期如質上線,滿足老闆或客戶(或BU單位)的需求,讓公司營運正常並賺錢。」

這個大方向沒錯,但是我們再切成 "人"、"系統" 和 "業務" 來看的話,可能就沒有那麼單純。 以系統來看我們要達成的目的, 我們至少必須要知道

  • 分辨必要(Must have)或是次要(Nice to have)的功能。
  • Sizing,未來幾年的業務及系統成長量。
  • SLA, SLO。
  • 如果是專案,專案合約可能有明訂一些必要的規劃和使用的技術。

至於架構設計為何需要考慮到人(即利害關係人)? 原因是每一項任務的成因和老闆的期望,可能不是我們想的那麼單純... 

政治因素、外包廠商、專案價格... etc(還有不好說的那部分),都會影響到我們評估要花多少資源在整個架構及任務上。  或許老闆的期望是讓團隊花更多時間在研發上,那我們就可以花更多資源在設計解決方案上。

業務角度也是重點,產品的業務是以套裝解決方案、或客製化收入為主? 以 SaaS 還是導入客戶端來做為服務重點? 上線後的維運單位和成本? 這會影響到我們的架構設計和技術選型。  


當時的環境、時間和資源

試著回想起三年前或最近剛上線的專案,是不是覺得有更多改善的空間? 肯定是的。
但是當時的情況允許我們用目前更好的架構或方式來建置嗎? 不一定。

考慮環境,包含法規、公司及部門政策和團隊素質。  如果我們的解決方案和架構,和現行環境有衝突的話,就要回過頭去思考目的思考必要的 tradeoff`。 

至於時間和資源,就不用多說了,鐵定是要考量的因素。 

我自己衡量的標準是

  • 先理解相關利害關係人的期望,在可以完成任務及先滿足擁有權力和資源的關係人(通常是老闆和客戶)的情況下,去嘗試取得更多時間和資源完成團隊的期望。 
  • 透過數據、經驗和知識來下決策,有的時候我們必須要堅持對的事情,去和老闆或客戶持續溝通。
  • 團隊素質如果高的話,可以在過程中去做更多嘗試和想法,這有利於在架構設計上有更多的選擇和彈性。   

當然現實情況是殘忍的,經常不會有最佳的資源去達成我們想要的方式,仍是取捨問題。 


當時最適合

當時最適合的決策和設計,現在回顧可能是錯誤的;也表示我們現在已經成長了也學習到更多經驗和知識,來理解如何用更好的方式來處理它。 隨著資訊越來越進步、技術推陳出新,團隊掌握更多知識和經驗(不管從內部或外部),可以讓我們更快、更準確的於當下做更適合的決策。  我自己的想法是,勇於改變和挑戰,即使結果不如預期,勝於什麼都不想改變。




2022年5月15日 星期日

[Testing] Automation flow by Robot Framework, GitHub Actions and Postman Monitor

   Robot Framework   GitHub Actions   Postman Monitor 


Introduction


 

Due to COVID-19, many people work from home and need to clock in or out online on working days.

Since I sometimes forget to clock in or out online and I am so lazy to open the HR website and enter user ID/PWD each time on it, I decided to make the automation clock in(out) flow by E2E testing stack.

 

I use the stack as following:

ü  Robot Framework

ü  Azure Container Registry

ü  GitHub Actions

ü  Postman Monitor

 



 

   

 

 

 

Steps


 

Record and prepare the E2E automation test

 

I use Katalon Recorder and Robot Framework to have both automation test as clock_in.robot and clock_out.robot.


 

I would like to run the tests in Docker environment, so I made a dockefile and could be built and published the Docker Image to a private Docker registry (e.q. Azure Container Registry).

 

See my previous article, [Robot Framework] Run E2E test by Chrome and SeleniumLibrary in Docker, for the implementation details.

 

Now, I have to find a way to pull the Docker Image and then run the tests automatically.

 

 

GitHub Actions

 

GitHub Actions supports amazing and easy way to automate the workflow.

This article, [GitHub] Github Actions - Workflow dependencies, shows how to pull and run a Docker Image by GitHub Actions workflow.

 

Here is the YAML file of workflow that has workflow dispatch event. I will talk about why using workflow dispatch instead of schedule event later.

 

 

docker_clock_in.yml

---
name: Docker Clock In
on:
  # schedule:
  #   - cron: '10 0 * * 1-5'
  workflow_dispatch:
    inputs:
      trigger:
        description: "The trigger"
        required: false
        default: "Postman Monitor"
      trigger_datetime:
        description: "The datetime for triggering the workflow"
        required: true  
jobs:
  clock_out:
    name: Clock In
    runs-on: ubuntu-18.04
    steps:
      - name: Print Environment Variables
        shell: bash
        run: |
          echo "Arguments are ${{ github.event.inputs.trigger }}, ${{ github.event.inputs.trigger_datetime }}"
      - name: Docker Run
        uses: addnab/docker-run-action@v3
        with:
          username: ${{ secrets.ACR_USERNAME }}
          password: ${{ secrets.ACR_PASSWORD }}
          registry: ${{ secrets.ACR_REGISTRY }}
          image: ${{ secrets.ACR_REGISTRY }}/robot-bank:latest
          options: --rm --shm-size=2gb
          run: |
            robot /usr/src/rf/tests/ClockInOut/clock_in.robot


 

And my GitHub Actions secrets:


 

 

After pushing the workflow YAML file to main/master branch, I can trigger the workflow (open the headless Chrome, go to HR website and then clock in) in three ways.
(Reference:
Manually running a workflow)


1. GitHub CLI: gh workflow run <workflow>

2. Run the workflow in GitHub Actions UI


 


3. Run the workflow by sending the HttpPost request to

https://api.github.com/repos/<user_id>/<reponsitory_name>/actions/workflows/<workflow>.yml/dispatches


Notice that we have to set a Personal access token for Authorization header. 



 

 

 

Why not use schedule event but workflow_dispatch?

 

I commented out the schedule event in docker_clock_in.yml, and use workflow_dispatch event instead. The reason is that schedule event has a uncertain delay time and won’t be trigger on the schedule time precisely. If your job can tolerates a delay time, schedule event might be a good way to trigger it. However, since I need to clock in before my office hours so I use Postman Monitor to automatically send the request for running the workflow.

 

 

 

Postman Monitor

 

Postman Monitor is the collection-based monitor that we can execute a collection of requests and collect the response time and result by scheduled time. Furthermore, it runs on Postman's cloud infrastructure, which is hosted by AWS. In other words, we do not need a live machine to execute the target request(s) by Monitor.

 

The following is my monitoring setup for requesting the clock-in workflow_dispatch API.

After dispatched the workflow, the workflow will pull and run the Docker Image that has automation tests for my clock-in or clock-out.


 

 

Summarize


 

There are many ways to accomplish this automation job.

I chose the stack/tools is because I don’t want to have extra cost for this routine job and I do want to save my time from it.

 

 

 

Reference


 GitHub Docs: Events that trigger workflows

GitHub Docs: Create a workflow dispatch event

GitHub Docs: Manually running a workflow

Postman: Monitoring your APIs

 

 

 

2021年4月9日 星期五

[ASP.NET Core] Identity Server 4 – PKCE Authorization Code Flow (Javascript client)

 ASP.NET Core   Identity Server 4   Authorization Code    PKCE   JS  

 



 

 

 

Introduction


 

For how PKCE Authorization Code Flow works, you can have a look on my previous article:  [ASP.NET Core] Identity Server 4 – PKCE Authorization Code Flow.

The flow is as following, we will focus on how to create a JavaScript client that can authenticate user by Identity Server 4 and access the backend’s resource with a given Access Token.




 

 

Notice that I put the JavaScript client at the Backend Server to make the sample code simple (As the following figure).

In the real world, we have to separate the Javascript client (Client) and the Backend Server (Resource server), see reference: [ASP.NET Core] Identity Server 4 – Concepts.



 


Here is the final result’s demo.


 

 


 

Related articles

 

01.      [OpenLDAP] Create an OpenLDAP container

02.      [ASP.NET Core] Identity Server 4 – Concepts

03.      [ASP.NET Core] Identity Server 4 – LDAP authentication

04.      [ASP.NET Core] Identity Server 4 – Secure Web API

05.      [ASP.NET Core] Identity Server 4 – Custom Event Sink

06.      [ASP.NET Core] Identity Server 4 – Refresh Token

07.      [ASP.NET Core] Identity Server 4 – Role based authorization

08.      [ASP.NET Core] Identity Server 4 – Policy based authorization

09.      [ASP.NET Core] Identity Server 4 - Dockerize

10.      [ASP.NET Core] Identity Server 4 – Client Credential

11.      [ASP.NET Core] Identity Server 4 – Policy based authorization with custom Authorization Handler

12.      [ASP.NET Core] Identity Server 4 – Signing credential

13.      [ASP.NET Core] Identity Server 4 – Authenticate by multiple LDAP

14.      [ASP.NET Core] Identity Server 4 – Cache and refresh Discovery document

15.      [ASP.NET Core] Identity Server 4 – PKCE Authorization Code Flow

16.      [ASP.NET Core] Identity Server 4 – PKCE Authorization Code Flow (Javascript client)

 

 

 

 

Environment


 

Docker 18.05.0-ce

.NET Core SDK 3.1.201

IdentityServer4 3.1.2

oidc-client.js 1.11.5

 

 

 

Implement


 

The source code is on my Github.

 

PKCE Client configuration on Auth Server

 

Go to Idsrv4 project (Auth Server), and open InMemoryInitConfig.cs to set the below configuration on a new client:

 

 

InMemoryInitConfig.cs

 

Notice that there are some key settings:

 

Configuration

Description

Value

AllowedGrantTypes

Grant type

GrantTypes.Code

RequirePkce

Specifies whether a proof key is required for authorization code based token requests (defaults to false).

true

RequireClientSecret

If set to false, no client secret is needed to request tokens at the token endpoint.

false

RedirectUris

The allowed Uri(s) to return Authorization Code or Tokens.

 

PostLogoutRedirectUris

Specifies allowed URIs to redirect to after logout

 

AllowedCorsOrigins

Gets or sets the allowed CORS origins for JavaScript clients.

 

 

public static IEnumerable<Client> GetClients ()
{
     return new [] {
            new Client {
                ClientId = "PkceJS",
                    ClientName = "JavaScript Client",
                    AllowedGrantTypes = GrantTypes.Code,
                    RequirePkce = true,
                    RequireClientSecret = false,
                    RedirectUris = { "https://localhost:5001/OpenId/Login/JS" },
                    PostLogoutRedirectUris = { "https://localhost:5001/OpenId/Login/JS" },
                    AllowedCorsOrigins = { "https://localhost:5001" },
                    AllowedScopes = {
                        IdentityServerConstants.StandardScopes.OpenId,
                        IdentityServerConstants.StandardScopes.Profile,
                        ApiResources.MyBackendApi2
                    },
                    AllowOfflineAccess = true,
                    AccessTokenLifetime = 3600,
                    RefreshTokenUsage = TokenUsage.OneTimeOnly// Or ReUse
                    RefreshTokenExpiration = TokenExpiration.Sliding,
                    AbsoluteRefreshTokenLifetime = 360000,
                    SlidingRefreshTokenLifetime = 36000,
                    ClientClaimsPrefix = string.Empty,
            }
     }
}


 

We have to implement the AccountController in Identity Server, the code is as same as

[ASP.NET Core] Identity Server 4 – PKCE Authorization Code Flow

 

Or you can see the full code on Github.

 

 

Client side

 

Since we put the JavaScript client on Backend project, which is based on ASP.NET Core.

We have to do some initialization works to enable related JS files.

 

First, install oidc-client.js with NPM.

 

$ cd src/AspNetCore.IdentityServer4.WebApi
$ npm install oidc-client --save

 

 

Copy node_modules/oidc-client/dist/oidc-client.js (or oidc-client.min.js) to wwwroot/js/.

And create the following js files:


File name

Directory

Description

app.js

wwwroot/js

The main javascript to run thru the authentication flow

app-config.js

wwwroot/js

Set the client and identity server’s host URLs, that will be used for redirect URLs after logged in or out.

 

 

The js file structure is as following for reference.

├── node_modules
|  ├── oidc-client
|  |  ├── dist
|  |  |  ├── oidc-client.d.ts
|  |  |  ├── oidc-client.js
|  |  |  ├── oidc-client.min.js
|  |  |  ├── oidc-client.rsa256.slim.js
|  |  |  ├── oidc-client.rsa256.slim.min.js
|  |  |  ├── oidc-client.slim.js
|  |  |  └── oidc-client.slim.min.js
└── wwwroot
   ├── js
   |  ├── app-config.js
   |  ├── app.js
   |  └── oidc-client.js

 

Before we implement the client logic, we have to enable static file serving.

 

Startup.cs: Configure

 

public void Configure(IApplicationBuilder app)
{
        // Use static files
        app.UseStaticFiles();
}

 

 

 

Views/LoginByJs.cshtml

 

 

Let’s create a View as the main page, which will import the js files we had just created.

<h2 id="welcome_msg">
    Welcome!  <label id="uid"></label><button class="btn btn-warning" id="logout">Sign Out</button>
    <button class="btn btn-success" id="api">Test secured API</button>
</h2>
<h2 id="signin_msg">
    Welcome! Please <button class="btn btn-primary" id="login">Sign In</button> first.
</h2>
<table class="table">
    <thead class="thead-dark">
        <tr>
            <th>#</th>
            <th>Key</th>
            <th>Value</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <th scope="row">1</th>
            <td>id_token</td>
            <td><label id="id_token"></label></td>
        </tr>
        <tr>
            <th scope="row">2</th>
            <td>access_token</td>
            <td><label id="access_token"></label></td>
        </tr>
        <tr>
            <th scope="row">3</th>
            <td>refresh_token</td>
            <td><label id="refresh_token"></label></td>
        </tr>
        <tr>
            <th scope="row">4</th>
            <td>expires_at</td>
            <td><label id="expires_at"></label></td>
        </tr>
    </tbody>
</table>
<hr />
<h3>Result:</h3>
<pre id="results"></pre>

@section Scripts {
    <script src="~/js/oidc-client.js"></script>
    <script type="module" src="~/js/app-config.js"></script>
    <script type="module" src="~/js/app.js"></script>
}

 

 

The page will be like following, there are three buttons that we will implement their callbacks.

·       Sign In

·       Sign Out

·       Test secured API

 

(Before logged in)


 

 

(Logged in)


 

 

 

app-config.js

 

We will set the client and identity server’s host URLs that will be used for redirect URLs after logged in or out in app.js.


export const AUTH_HOST_URL = "https://localhost:6001";
export const CLIENT_HOST_URL = "https://localhost:5001"; 

 

 

 

app.js

 

Set the OIDC configuration and logging,

import * as constants from './app-config.js';

function log() {
    document.getElementById('results').innerText = '';

    Array.prototype.forEach.call(argumentsfunction (msg) {
        if (msg instanceof Error) {
            msg = "Error: " + msg.message;
        }
        else if (typeof msg !== 'string') {
            msg = JSON.stringify(msgnull2);
        }
        document.getElementById('results').innerHTML += msg + '\r\n';
    });
}

var config = {
    authority: constants.AUTH_HOST_URL,
    client_id: "PkceJS",
    redirect_uri: `${constants.CLIENT_HOST_URL}/OpenId/Login/JS`,
    response_type: "code",
    scope: "openid profile offline_access MyBackendApi2",
    post_logout_redirect_uri: `${constants.CLIENT_HOST_URL}/OpenId/Login/JS`
};
 

 

 

Implement the callbacks of Sign In, Sign Out and Test secured API.


document.getElementById("welcome_msg").hidden = true;
document.getElementById("login").addEventListener("click"loginfalse);
document.getElementById("api").addEventListener("click"apifalse);
document.getElementById("logout").addEventListener("click"logoutfalse);

var mgr = new Oidc.UserManager(config);

mgr.signinRedirectCallback().then(function (user) {
    if (user) {
        document.getElementById("signin_msg").hidden = true;
        document.getElementById("welcome_msg").hidden = false;
        document.getElementById("uid").innerText = user.profile.sub;
        document.getElementById("id_token").innerText = user.id_token;
        document.getElementById("access_token").innerText = user.access_token;
        document.getElementById("refresh_token").innerText = user.refresh_token;
        document.getElementById("expires_at").innerText = moment.unix(user.expires_at).utc();
        log("User logged in"user);
    }
    else {
        log("User not logged in");
    }
});

function login() {
    mgr.signinRedirect();
}

function logout() {
    // Signout
    mgr.signoutRedirect();
}

function api() {
    mgr.getUser().then(function (user) {
        var url = `${constants.CLIENT_HOST_URL}/api/DemoPolicyBased/Admin/Get`// You can change the test API 

        var xhr = new XMLHttpRequest();
        xhr.open("GET"url);
        xhr.onload = function () {
            log(xhr.statusxhr.responseText);
        }

        xhr.setRequestHeader("Authorization""Bearer " + user.access_token);
        xhr.send();
    });
}

 

The full code of app.js is here.

 

 

  

Source Code

Github: KarateJB/AspNetCore.IdentityServer4.Sample

 

 

 

Reference


 Authorization Code Flow with Proof Key for Code Exchange (PKCE)

IdentityModel/oidc-client-js

IdentityServer4: Adding a JavaScript client