Thursday, December 31, 2020

[ASP.NET Core] Feature Toggle

  ASP.NET Core   Feature toggle   Resource filter  


 

 

▌Introduction


 

While we have more and more features (APIs…) and environments (SIT, Staging…), we want to enable/disable certain features on specific environments by feature toggle.

 

This article shows how to do feature toggle by two ways

 

l   Implement IAsyncResourceFilter

l   Use official package: Microsoft.FeatureManagement.AspNetCore

 

The full sample code is in my Github: karatejb/AspNetCore.Filters.Sample

 

 

▋Related articles

1.  [MVC] Pass parameters to ActionFilterAttribute

2.  [ASP.Net Core] Action Filter with Parameter(s)

3.  [ASP.NET Core] Hybrid Filter

4.  [ASP.NET Core] Feature toggle

 

 

 

▌Environment


 

▋dotnet core sdk 3.1.300

 

 

 

▌Custom Resource Filter


 

The following example shows how to disable APIs that depends on the environment value of ASPNETCORE_ENVIRONMENT.

 

 

▋Implement IAsyncResourceFilter

 

Resource filters will be after Authorization filters and before Action filters and model binding, so we can implement IAsynResourceFilter to intercept the request and return 404(Not found) immediately.

 

 

▋DisableApiFilter.cs

public class DisableApiFilter : Attribute, IAsyncResourceFilter
{
    private readonly string env = string.Empty;

    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="hostEnvironment">IWebHostEnvironment</param>
    /// <param name="options">IOptions</param>
    /// <param name="onEnv">Regex for checking ASPNETCORE_ENVIRONMENT</param>
    public DisableApiFilter(
        IWebHostEnvironment hostEnvironment,
        IOptions<AppSettings> options,
        string onEnv = "")
    {
        this.env = hostEnvironment.EnvironmentName;
        this.OnEnv = string.IsNullOrEmpty(onEnv) ? options.Value?.EnvForDisableApiFilter : onEnv;
    }

    /// <summary>
    /// Regression expression of ASPNETCORE_ENVIRONMENT for disabling API
    /// </summary>
    public string OnEnv { get; set; }

    /// <summary>
    /// OnResourceExecutionAsync
    /// </summary>
    /// <param name="context">ResourceExecutingContext</param>
    /// <param name="next">ResourceExecutionDelegate</param>
    public async Task OnResourceExecutionAsync(ResourceExecutingContext context, ResourceExecutionDelegate next)
    {
        if (string.IsNullOrEmpty(this.OnEnv))
        {
            await next();
        }
        else
        {
            var regex = new Regex(this.OnEnv);

            if (regex.IsMatch(this.env))
            {
                context.Result = new EmptyResult();
                context.HttpContext.Response.StatusCode = StatusCodes.Status404NotFound;
            }
            else
            {
                await next();
            }
        }
    }
}

 

▋Use DisabelApiFilter by setting regex pattern

 

The variable: OnEnv, is the parameter of the filter that supports Regular Expression.

When environment variable: `ASPNETCORE_ENVIRONMENT` matches the OnEnv, the filter will force the API to return 404.

 

For example, the following API will response 404 when ASPNETCORE_ENVIRONMENT's value contains “Production” or “production”.

[TypeFilter(typeof(DisableApiFilter), Arguments = new object[] { "^(.*)[Pp]roduction(.*)$" })]
[HttpGet]
public async Task<IActionResult> Demo()
{}

 

 

▋Use DisableApiFilter by default regex pattern

 

We can set a default regex pattern to be checked in DisableApiFilter when parameter OnEnv is not specified.

 

▋appsettings.json

{
  "EnvForDisableApiFilter": "^(.*)[Pp]roduction(.*)$",
}

 

And we can disable API on demand environment(s) like this,

[TypeFilter(typeof(DisableApiFilter))]
[HttpGet]
public async Task<IActionResult> Demo()
{}

 

 

▌Microsoft.FeatureManagement.AspNetCore


 

Github: microsoft/FeatureManagement-Dotnet

 

This official package supports Feature toggle dynamically.

 

 

▋Install Nuget package

 

Microsoft.FeatureManagement.AspNetCore

 

 

▋Feature Management Configuration

 

Define the key: FeatureManagement in appsettings.json like following,

▋appsettings.json

"FeatureManagement": {
    "ApiDoc": false,
    "Demo": true,
    "DemoGlobalFilter": true,
    "Tests": true,
    "ServerEnvHeader": false
} 

 

 

We can create an enum for later usage.

public enum FeatureFlags
{
    ApiDoc = 1,
    Demo,
    DemoGlobalFilter,
    Tests,
    ServerEnvHeader
}

 

 

▋Enable Feature Management

 

▋Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    // Add feature mangement services 
    services .AddFeatureManagement();

    services .AddControllers() .AddNewtonsoftJson();
}

 

▋Feature toggle by Attribute

 

Now we can set feature toggle on API (Controller or Action) with FeatureGateAttribute.

 

For example, the API will return 404 if we set `false` on the mapping feature flag on appsettings.json.

[FeatureGate(FeatureFlags.Demo)]
[Route("api/[controller]")]
[ApiController]
public class DemoController : ControllerBase
{ }

 

 

▋ Feature toggle by checking flags from IFeatureManager

 

We can also use the injected Microsoft.FeatureManagement.IFeatureManager instance to get the value of feature flag.

[ApiController]
public class DemoController : ControllerBase
{
    private readonly IFeatureManager featureManager = null;

    public DemoController(IFeatureManager featureManager)
    {
        this.featureManager = featureManager;
    }

    [HttpGet]
    public async Task<IActionResult> TestDisableApiFilter()
    {
        if (await featureManager.IsEnabledAsync(nameof(FeatureFlags.Tests)))
        {
            return this.Ok();
        }
        else 
        {
            return StatusCode(StatusCodes.Status404NotFound);
        }
    }
}

 

This is useful when we can control enabling certain features on startup of application.

For example, we can toggle API Document, like Swagger, with an extension method,

 

▋IApplicationBuilderExtensions.cs

public static IApplicationBuilder UseToggleFeatures(this IApplicationBuilder app)
{
    var featureManager = app.ApplicationServices.GetService<Microsoft.FeatureManagement.IFeatureManager>();
    bool isEnableApiDoc = featureManager.IsEnabledAsync(nameof(FeatureFlags.ApiDoc)).Result;

    if (isEnableApiDoc)
    {
        // Enable something like Swagger, etc...
    }

    return app;
} 


And apply it in Startup.cs,

public class Startup
{
    public void Configure(IApplicationBuilder app)
    {
        // Use toggle features
        app.UseToggleFeatures();
    }
}

 

 

▋Enable/Disable global filter

 

We can also control the global filter(s) by Feature Management.

For example, there is a custom filter that will put the environment information to every HTTP response header,

 

▋CustomHeaderFilter.cs

public class CustomHeaderFilter : Attribute, IAsyncActionFilter
{
    private readonly string env;

    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="hostEnvironment">IWebHostEnvironment</param>
    public CustomHeaderFilter(IWebHostEnvironment hostEnvironment)
    {
        this.env = hostEnvironment.EnvironmentName;
    }

    /// <summary>
    /// OnActionExecutionAsyn
    /// </summary>
    /// <param name="context">ActionExecutingContext</param>
    /// <param name="next">ActionExecutionDelegate</param>
    public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        await next();

        string header = "Server-Env";
        context.HttpContext.Response.Headers.Remove(header);
        context.HttpContext.Response.Headers.Add(header, this.env);
    }

} 

 

 

We can enable/disable it as the global filter as following,

 

▋Startup.cs

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddFeatureManagement();

        // Add the global IAsyncActionFilter by feature toggle
        services .AddControllers(o => o.Filters.AddForFeature<CustomHeaderFilter>(nameof(FeatureFlags.ServerEnvHeader)));
    }
}

 

When we turn on the feature flag, we will see the custom header on response.


 

 

 

 

▋Feature Management Configuration from other service

 

We can implement IFeatureDefinitionProvider to get the Feature Management Configuration from other service's API.

 

We will use json-server to have an API running on http://localhost:3000/feature-configuration, which will return the json configuration as below,


 

 

Now lets see how to load the configuration by implementing the custom IFeatureDefinitionProvider.

 

 

▋Implement IFeatureDefinitionProvider

 

We can take a look at the source code of ConfigurationFeatureDefinitionProvider.cs and try to create our custom one.

The sample code also refer to the following article: [料理佳餚] 實作 IFeatureDefinitionProvider 從外部的服務載入 ASP.NET Core Feature Flags(Feature Toggle)的設定.

 

 

·         RemoteFeatureDefinitionProvider.cs

 

The feature management definitions will be load from remote API and cache for a period of time in memory cache.

public class RemoteFeatureDefinitionProvider : IFeatureDefinitionProvider
{
    private const string FEATURE_MANAGEMENT_SECTION = "FeatureManagement";
    private const string FFEATURE_FILTER_SECTION = "EnabledFor";
    private const int CACHE_TIME = 180; // Cache time in seconds
    private readonly ILogger<RemoteFeatureDefinitionProvider> logger;
    private readonly IHttpClientFactory httpClientFactory;
    private readonly IMemoryCache memoryCache;
    private ConcurrentDictionary<string, FeatureDefinition> definitions;
    private IConfiguration configuration;

    /// <summary>
    /// Custructor
    /// </summary>
    /// <param name="logger">Logger</param>
    /// <param name="httpClientFactory">HttpClientFactory</param>
    /// <param name="memoryCache">Memory cache</param>
    public RemoteFeatureDefinitionProvider(
        ILogger<RemoteFeatureDefinitionProvider> logger,
        IHttpClientFactory httpClientFactory,
        IMemoryCache memoryCache)
    {
        this.logger = logger;
        this.httpClientFactory = httpClientFactory;
        this.memoryCache = memoryCache;
        this.definitions = new ConcurrentDictionary<string, FeatureDefinition>();
    }

    /// <summary>
    /// Get all feature definitions
    /// </summary>
    /// <returns>Async enumerable of FeatrueDefinition</returns>
    public async IAsyncEnumerable<FeatureDefinition> GetAllFeatureDefinitionsAsync()
    {
        await this.reloadConfiguration();

        foreach (var featureSection in this.getFeatureDefinitionSections())
        {
            yield return this.definitions.GetOrAdd(featureSection.Key, _ => this.readFeatureDefinition(featureSection));
        }
    }

    /// <summary>
    /// Get single feature definition
    /// </summary>
    /// <param name="featureName">Feature name</param>
    /// <returns>FeatureDefinition</returns>
    public async Task<FeatureDefinition> GetFeatureDefinitionAsync(string featureName)
    {
        if (featureName == null)
        {
            throw new ArgumentNullException(nameof(featureName));
        }

        await this.reloadConfiguration();
        var definition = this.definitions.GetOrAdd(featureName, name => this.readFeatureDefinition(name));
        return definition;
    }

    private async Task reloadConfiguration()
    {
        string remoteApiUri = this.appSettings.FeatureManagementUri;
        if (!this.memoryCache.TryGetValue(CacheKeys.FeatureConfig, out IConfiguration featureConfig))
        {
            this.definitions.Clear();

            using var httpClient = this.httpClientFactory.CreateClient();
            var streamResponse = await httpClient.GetStreamAsync("http://localhost:3000/feature-management"); // Use json-server to run the json file at /AspNetCore.Filters.WebApi/Assets/feature_management.json
            this.configuration = new ConfigurationBuilder().AddJsonStream(streamResponse).Build();

            this.memoryCache.Set(CacheKeys.FeatureConfig, this.configuration, TimeSpan.FromSeconds(CACHE_TIME));
        }
        else
        {
            this.configuration = featureConfig;
        }
    }

    private FeatureDefinition readFeatureDefinition(string featureName)
    {
        var configurationSection = this.getFeatureDefinitionSections().FirstOrDefault(section => section.Key.Equals(featureName, StringComparison.OrdinalIgnoreCase));

        if (configurationSection == null)
        {
            return null;
        }

        return this.readFeatureDefinition(configurationSection);
    }

    private FeatureDefinition readFeatureDefinition(IConfigurationSection configurationSection)
    {
        var enabledFor = new List<FeatureFilterConfiguration>();

        var val = configurationSection.Value;

        if (string.IsNullOrEmpty(val))
        {
            val = configurationSection[FFEATURE_FILTER_SECTION];
        }

        if (!string.IsNullOrEmpty(val) && bool.TryParse(val, out bool result) && result)
        {
            enabledFor.Add(new FeatureFilterConfiguration { Name = "AlwaysOn" });
        }
        else
        {
            var filterSections = configurationSection.GetSection(FFEATURE_FILTER_SECTION).GetChildren();

            foreach (var filterSection in filterSections)
            {
                if (int.TryParse(filterSection.Key, out int i) && !string.IsNullOrEmpty(filterSection[nameof(FeatureFilterConfiguration.Name)]))
                {
                    enabledFor.Add(
                        new FeatureFilterConfiguration
                        {
                            Name = filterSection[nameof(FeatureFilterConfiguration.Name)],
                            Parameters = filterSection.GetSection(nameof(FeatureFilterConfiguration.Parameters))
                        });
                }
            }
        }

        return new FeatureDefinition { Name = configurationSection.Key, EnabledFor = enabledFor };
    }

    private IEnumerable<IConfigurationSection> getFeatureDefinitionSections()
    {
        bool isExist = this.configuration.GetChildren().Any(x => x.Key.Equals(FEATURE_MANAGEMENT_SECTION));
        if (isExist)
            return this.configuration.GetSection(FEATURE_MANAGEMENT_SECTION).GetChildren();
        else
            return this.configuration.GetChildren();
    }
}

 

 

▋Register the custom RemoteFeatureDefinitionProvider


Last step, we have to register the necessary services and RemoteFeatureDefinitionProvider on Starup.cs.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.Configure<AppSettings>(this.Configuration);
        services.AddHttpClient();
        services.AddMemoryCache();

        // Add feature mangement services 
        services
            .AddSingleton<IFeatureDefinitionProvider, RemoteFeatureProvider>()
            .AddFeatureManagement();

        services.AddControllers().AddNewtonsoftJson();
    }
}

 

And don't forget to remove the FeatureManagement settings from appsettings.json!


 

▌Reference


 

▋microsoft/FeatureManagement-Dotnet

▋[料理佳餚] 實作 IFeatureDefinitionProvider 從外部的服務載入 ASP.NET Core Feature Flags(Feature Toggle)的設定

 

 

 

 

Sunday, December 27, 2020

[Notes] NET Conf 2020 Taiwan

 

 .NET 5   NET Conf   2020   C# 9   EF Core 5   ASP.NET Core 

 



 

▌Notes


 

See my notes on my Github.

 

今年在疫情的影響下,台灣還能正常舉辦實體研討會是很難得跟珍貴的 :)

希望明年全球疫情能逐漸好轉。

 

明年計畫會逐漸將所有和技術有關的文章移至Github,畢竟用Markdown語法寫技術文章比用Blogger快太多了。

Happy New Years and goodbye, 2020!

 

 

 

 

 

 

 

Saturday, December 19, 2020

[PostgreSQL] Execute Dynamic SQL

 

 PostgreSQL   Dynamic SQL  






▌Introduction


 

 This is a sample of using EXECUTE to execute the dynamic SQL, which its parameters come from psql arguments.

 

 

▌Environment


 

▋PostgreSQL 11

 

 

▌Implement


 

▋The original SQL

 

Here is the original SQL we want to execute. The SQL will create a foreign data wrapper for dblink(cross database query) from database Demo to database postgres.

 

▋create_dblink_wrapper.sql

/* Create dblink wrapper to postgres */

DO $wrapper$
BEGIN

CREATE FOREIGN DATA WRAPPER pgagent_wrapper VALIDATOR postgresql_fdw_validator;
CREATE SERVER db_postgres FOREIGN DATA WRAPPER pgagent_wrapper OPTIONS (host 'localhost', dbname 'postgres' );
CREATE USER MAPPING FOR postgres SERVER db_postgres OPTIONS (user 'postgres', password 'xxxxx');

END
$wrapper$; 

 

▋ Goals

 

Since we put the user/pwd when creating the USER MAPPING and while executing the SQL on CI, it is insecure and cannot be modified depends on different environments.

 

We would like to decide the values of user/pwd by the environment variables, for example,

$ psql -h localhost -p 5432 -U postgres -d Demo -v postgres_user=$POSTGRESS_USER -v postgres_password=$POSTGRES_PASSWORD -f create_dblink_wrapper.sql

 

 

▋ This will NOT WORK!

 

Since we can get the argument values from psql command by :postgres_user. However, we cannot use the variable(s) inside the DO statement.

 


\echo USER/PWD=:postgres_user/:postgres_password -- This wiil successfully show USER/PWD=my_user/my_pwd

DO $wrapper$
CREATE USER MAPPING FOR postgres SERVER db_postgres OPTIONS (user :'postgres_user', password :'postgres_password'); --Error: syntax error on ":"
END
$wrapper$

 

▋Solution

 

We have to use the features of Dynamic SQL and use current_setting(get current value of setting) to make the SQL can read the value of arguments in DO statement.

 

▋create_dblink_wrapper.sql

 

See the SQL sample file on my Github.

/* Create dblink wrapper to postgres */

\echo POSTGRES_USER=:'postgres_user',POSTGRES_PASSWORD=:'postgres_password'

SET "pg.user" TO :'postgres_user'; --The variable must contains dot. If name it as "pg_user", it will cause error: "ERROR:  unrecognized configuration parameter "pg_user".
SET "pg.pwd" TO :'postgres_password';
SHOW "pg.user";
SHOW "pg.pwd";

DO $wrapper$
DECLARE
  usr text := current_setting('pg.user');
  pwd text := current_setting('pg.pwd');
BEGIN

CREATE FOREIGN DATA WRAPPER pgagent_wrapper VALIDATOR postgresql_fdw_validator;
CREATE SERVER db_postgres FOREIGN DATA WRAPPER pgagent_wrapper OPTIONS (host 'localhost', dbname 'postgres' );
--Execute Dynamic SQL
EXECUTE format('CREATE USER MAPPING FOR postgres SERVER db_postgres OPTIONS (user ''%I'', password ''%I'');', usr, pwd);

END
$wrapper$

 

Now we can execute the SQL as following,

$ psql -h localhost -p 5432 -U postgres -d Demo -v postgres_user=$POSTGRES_USER -v postgres_password=$POSTGRES_PASSWORD -f create_dblink_wrapper.sql

 

Output:

POSTGRES_USER='postgres',POSTGRES_PASSWORD='xxxxx'
SET
SET
 pg.user
----------
 postgres
(1 row)

  pg.pwd
-----------
 xxxxx
(1 row)

DO

 


▌Reference


▋ PostgreSQL Doc – psql

▋ PostgreSQL Doc - Dynamic SQL

▋ PSQL Command Line Arguments in DO script

▋ PostgreSQL: How to pass parameters from command line?