Ruby Zero

Tunnel SSH connection using cloudflared

Wed, 21 May 2025 00:16:00 +0000

I’ve been searching for a simple way to tunnel an SSH connection to my private server. There are many solutions to this problem, such as using a VPN service, a reverse proxy, etc. However, these often require installing additional packages and root permissions to manipulate network configurations.

Cloudflare offers a tunnel service that can be easily installed by downloading the cloudflared binary. In this blog post, I’ll explain the steps needed to set up an SSH tunnel using cloudflared.

1. Download and install cloudflared

Source code of cloudflared is on github. You can download a prebuilt binary from the release.

If your OS doesn’t have a prebuilt binary, you can also compile it using Go.

The cloudflared binary can be placed anywhere, but I recommend putting it somewhere in your $PATH.

2. Configure `cloudflared`

To use cloudflared for tunneling, you’ll need a Cloudflare account (a free plan works), and a domain pointed to Cloudflare’s nameservers.

Start by logging in to your Cloudflare account with the following command:

cloudflared tunnel login

This will open your browser and prompt you to log in to your Cloudflare account.

Next, create a new tunnel:

cloudflared tunnel create <tunnel-name>

# Example:
cloudflared tunnel create my-tunnel

You can view all your tunnels using:

cloudflared tunnel list

Each tunnel will have an ID and a name. You’ll use the tunnel ID in your configuration file.

Now, configure the tunnel by creating a file at ~/.cloudflared/config.yaml:

tunnel: <your-tunnel-id>
credentials-file: ~/.cloudflared/<your-tunnel-id>.json
ingress:
  - hostname: <your-subdomain>
    service: ssh://localhost:22
  - service: http_status:404

In the hostname field, specify a subdomain (from a domain managed by Cloudflare). For example: mytunnel.monotone.dev.

More information on the config.yaml format is available in the Cloudflare documentation.

3. Set Up the Route

Next, tell Cloudflare how to route traffic to your server by running:

cloudflared tunnel route dns <your-tunnel-name> <your-subdomain>

# Example:
cloudflared tunnel route dns my-tunnel mytunnel.monotone.dev

Then, start the tunnel:

cloudflared tunnel run my-tunnel

4. Connect to Your Private Server

Now, from another computer, you can connect to your private server. On that computer, you also need to install the cloudflared command and authorize it with your Cloudflare account.

Update your SSH config (~/.ssh/config) with a new entry:

Host mytunnel.monotone.dev
  ProxyCommand cloudflared access tcp --hostname %h

Now you can SSH into your private server using:

ssh mytunnel.monotone.dev

Getting Django to Work on a Cheap Shared Hosting Plan

Tue, 09 Jul 2024 00:16:00 +0000

I am renting a very cheap shared server with very generous specs (300 GB disk, TBs of bandwidth, etc.) for about $3/month. Though there are some limitations such as running on fairly old technology, basically a CPanel shared host, only supporting web server with Apache with PHP through CGI, no root access. However, they provide SSH access, so I can manage to run a modern framework with it.

Let’s first install python. I use pyenv to manage python versions:

pyenv install 3.10.2

The Apache webserver serves files under ~/www, so I will create a subfolder there and then make a virtual environment for Python:

mkdir ~/www/django && cd ~/www/django
python -mvenv py
./py/bin/pip install django

Now, let’s create a very simple Django application. I’ll call it index.cgi(also need to chmod u+x):

#!/home/myname/www/django/py/bin/python
#~/www/django/index.cgi
from wsgiref.handlers import CGIHandler
from django.conf import settings
from django.http import HttpResponse
from django.urls import path
from django.core.wsgi import get_wsgi_application

settings.configure(
        ROOT_URLCONF=__name__,
        DEBUG=True,
        SECRET_KEY="my-secret-key"
)

# path: / 
def index(request):
    return HttpResponse("index")

# path: /blog
def blog(request):
    return HttpResponse("blog")

urlpatterns = [
    path("", index),
    path("blog", blog)
]

if __name__ == "__main__":
    application = get_wsgi_application()
    CGIHandler().run(application)

With this file, the server can serve Django content so that when I access /django/index.cgi, it will return the index content, and /django/index.cgi/blog will return the blog content.

I will add a .htaccess file to rewrite URLs to a shorter form:

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.cgi/$1 [QSA,L]

With this .htaccess configuration, accessing /django will point to the index, and /django/blog will point to the blog.

Module in JavaScript

Thu, 20 Jun 2024 00:16:00 +0000

JavaScript modules can be a puzzling concept for many developers. In this post, I will try to gather and summarize how to use them effectively. I will split this post into two parts: covering modules in the NodeJS environment and those used in the browser environment.

Module in NodeJS

CommonJS

Traditionally, NodeJS relied on the CommonJS module system for managing code. This system allows developers to use the require statement to import functionalities from other files.

// a.js
const add = (a, b) => a + b;
exports.add = add;

// b.js: uses require to call a.js
const { add } = require('./a.js');
console.log(add(1, 2));

CommonJS is the default when NodeJS intepretes a .js file. We can explicit tell NodeJS that a file should be read as CommonJS by one of the following methods:

Uses .cjs extension; or
Add a "type": "commonjs" to top-level package.json

ESM

The ECMAScript Technical Committee (TC39), which defines the standard for JavaScript, later introduced another module system called ECMAScript Modules (ESM). Naturally, NodeJS adopted this standard and began supporting ESM starting with version 14.

ESM includes the keyword import/export. Here is an example:

// a.js
const add = (a, b) => a + b;
export { add }

// b.js
import { add } from './a.js';
console.log(add(1,2));

By default, NodeJS doesn’t implicitly consider a .js file ESM. Therefore, if you run this example, you’ll get the error SyntaxError: Cannot use import statement outside a module. We need to specify an ESM by:

Use .mjs file extension; or
Add a “type”: “module” to top-level package.json

Calling between CommonJS and ESM

We can call an ESM from a CommonJS module or vice versa by using the function async import(), for example:

// a.mjs
const add = (a, b) => a + b;
export { add }

// b.cjs
import('./a.mjs').then(m => {
    console.log(m.add(1, 2));
})

Current version of NodeJS also introduce a experiental flag --experimental-require-module that allows calling ESM by using require given that:

The module is fully synchronous (contains no top-level await); and
One of these conditions:
- The file has a .mjs extension.
- The file has a .js extension, and the closest package.json contains “type”: “module”
- The file has a .js extension, the closest package.json does not contain “type”: “commonjs”, and –experimental-detect-module is enabled.

So the advice here is if it’s your own code, try to be as explicit as possible such as using the .mjs and .cjs extension.

Module in browser

JavaScript originally lacked a proper module system. To include libraries like jQuery, developers added a script tag in their HTML file.

<script type="javascript" src="jquery.js"></script>

The browser’s JavaScript engine would then load and execute this external JavaScript file. Often, such libraries would define a global variable (e.g., window.jQuery = …) to make their functionality accessible throughout the page.

This approach served as the standard for a while. Then came build tools like Webpack, Gulp, and Parcel. These tools allow for better project structure by enabling you to organize your code into smaller modules. You can use export and import statements to manage dependencies between these modules, and the build tool will bundle them into a single file optimized for the browser.

Around 2017, the ECMAScript Modules (ESM) standard arrived, allowing developers to directly use import and export keywords within their client-side JavaScript code.

ESM in browser

We can declare our .js file an ESM module by set the type attribute of script tag to module, ie:

<script type="module" src="./main.js"></script>

<!-- or directly inside script -->
<script type="module">
  import { add } from './main.js';
  console.log(add(1,2));
</script>

<!-- without type=module, usaged of import/export will raise error -->
<script>
  import { add } from './main.js'; // will raise SyntaxError
</script>

then inside main.js we can use import/export statement.

// main.js
import { add } from './lib.js';
console.log(add(1,2));

＊ While the .mjs extension might be tempting for these types of files, browsers are quite strict about the MIME type for JavaScript files. Therefore, if your server is configured to return the correct MIME type (text/javascript) for .mjs files, you can use this extension. Otherwise, it’s best to stick with the .js extension on the browser side for better compatibility.

importmap

To make it easier to import module from an URL, we can use importmap.

<script type="importmap">
{
    "imports": {
        "react": "https://esm.sh/react@18.3.1",
        "react-dom": "https://esm.sh/react-dom@18.3.1"
    }
}
</script>

<!-- in our module we can import from React -->
<script type="module">
  // instead of import React from "https://esm.sh/react@18.2.0"
  // we can use:
  import React from "react";
  import { createRoot } from "react-dom";
</script>

＊ importmap is generally available since 2023 which is quite new at the time of this post.

ESM in the browser allows us a simple way to structure our project. However, if we need advanced features such as transpiling, TypeScript, JSX, Vue’s SFC, we still need a bundler.

Biểu thức chính quy và automata

Sat, 09 Sep 2023 00:16:00 +0000

Một số ghi chép về ngôn ngữ hình thức(formal language), ngữ pháp(grammar), regular language và automata.

Formal language & BNF grammar

Ngôn ngữ hình thức có thể hiểu như một mô hình để mô tả ngôn ngữ(ví dụ tiếng Anh, ngôn ngữ lập trình Python, …). Một ngôn ngữ hình thức(thường ký hiệu L) bao gồm các thành phần:

Bộ ký tự: tập hợp các ký tự trong mô hình, các ký tự sẽ được nhóm lại tạo thành từ(word) hoặc token. Tập các ký tự thường được ký hiệu bằng Σ.
Grammar: tập hợp các quy luật để quyết định xem một chuỗi các ký tự có thuộc mô hình hay không.

Ví dụ ①: tập ký tự Σ = { a } (chỉ bao gồm một ký tự a), và grammar: các chuỗi có chiều dài là một số chẵn. Như vậy ngôn ngữ của ta bao gồm các chuỗi L = { aa, aaaa, aaaaaa, ... }, dĩ nhiên chuỗi aaa, a ∉ L

Một ví dụ khác: tập ký tự của tiếng Việt Σ = { a, ă, â, b, c, d, đ, ... y }, L(tiếng Việt) = { tất cả các câu đúng ngữ pháp tiếng Việt}

BNF(Backus–Naur form): quy tắc ngữ pháp mà ghi bằng lời như ví dụ trên rất dài dòng và không chính xác. Vì vậy các quy tắc thường được diễn tả bằng BNF. Quy tắc ngữ pháp viết theo BNF như sau:

<symbol> ::= __expression__

Trong đó symbol là một non-terminal symbol(hay variable, ký hiệu chưa kết thúc) và expression là một hoặc nhiều terminal(ký hiệu kết thúc)/non-terminal symbol.

Ký hiệu ::= có nghĩa thay thế symbol bên vế trái bằng expression ở vế phải.

Một số khái niệm khác liên quan đến BNF:

ε: ký hiệu cho một chuỗi rỗng.
Start symbol

Ví dụ ②: quay lại với ví dụ ① ta có thể mô tả grammar bằng BNF như sau:

S ::= ε | aaS

Một cách viết khác:
S -> ε | aaS

Trong BNF, nếu có nhiều lựa chọn, ngăn cách các lựa chọn bằng ký hiệu |. Như ở ví dụ trên, S có thể được thay thế bằng một chuỗi rỗng(ε) hoặc một chuỗi bắt đầu bằng 2 ký tự aa.

Cũng ở ví dụ trên, S là một non-terminal symbol(có thể được thay thế bởi một biểu thức khác), aa là terminal symbol.

Bài toán parsing

Bài toán parsing được phát biểu như sau:

Cho tập ký tự Σ, L được mô tả bằng bộ ngữ pháp G, s là một chuỗi bất kỳ tạo thành bởi các ký tự thuộc Σ. Xác định s có thuộc L hay không.

Để giải bài toán parsing, trước tiên cần phân loại L dựa theo Chomsky hierarchy.

Chomsky hierarchy

Chomsky đã đề xuất phân chia ngôn ngữ thành các cấp bậc hay còn gọi là Chomsky Hierarchy. Có 4 cấp bậc trong Chomsky Hierarchy:

Regular language(ngôn ngữ chính quy): bộ ngữ pháp của ngôn ngữ chính quy có các ràng buộc sau:

A -> a (vế phải có 1 terminal symbol)
A -> aB (vế phải có tối đa 1 nonterminal symbol và symbol này phải nằm ngoài cùng, trái hoặc phải).

Trong đó:

A, B: nonterminal symbol
a: terminal symbol

Context-free language(ngôn ngữ phi ngữ cảnh): bộ ngữ pháp của ngôn ngữ phi ngữ cảnh(context-free grammar hay CFG) có các ràng buộc sau:

A -> α (vế trái chỉ có 1 non-terminal symbol)

Trong đó:

A: nonterminal symbol
α: một hoặc nhiều terminal/nonterminal symbol

Ví dụ ③:

<Câu> → <Chủ ngữ> <Động từ> <Vị ngữ> | <Chủ ngữ> <Động từ>
<Chủ ngữ> → <Danh từ>
<Danh từ> → tôi | bạn | anh | em ...

Context-sensitive: ngữ pháp có ràng buộc sau

αAβ -> αγβ

Trong đó:

A: nonterminal symbol
α, γ, β: một hoặc nhiều terminal, nonterminal symbol

Recursively enumerable: hầu như không có ràng buộc gì đối với grammar rule của ngôn ngữ này.

α -> γ

Trong đó:

α, γ: một hoặc nhiều terminal/nonterminal symbol. α ≠ ε

Trong Chomsky hierarchy, { Regular Language } ⊂ { Context-free } ⊂ { Context-sensitive } ⊂ { Recursively enumerable }

Noam Chomsky là một nhà ngôn ngữ học đã có những đóng góp quan trọng trong lý thuyết ngôn ngữ hình thức và được sử dụng rộng rãi trong bộ môn trình biên dịch(compiler) của khoa học máy tính.

Regular expression

Vẫn dùng tập ký tự Σ, các symbol sau là biểu thức chính quy:

∅: tập rỗng
ε: chuỗi rỗng, L(ε) = { ε }
a: một ký tự thuộc Σ, L(a) = { a }

Nếu R và S là 2 biểu thức chính quy, các phép toán sau cũng tạo ra biểu thức chính quy:

Phép nối (RS): ví dụ R = { a, aa }, S = { bb, bbbb }, (RS) = { abb, abbbb, aabb, aabbbb }. Có thể hiểu phép nối như tích Decartes giữa 2 tập R và S. (RS) = R x S.
Phép hợp(R | S): ví dụ với R, S như trên (R \| S) = {a, aa, aaa, bb, bbbb, ...}. Có thể hiểu (R | S) = R ∪ S.
Kleene star(R*): R* = {ε} ∪ R ∪ (RR) ∪ (RRR) ∪ (RRRR) ∪ ... ∪ (R^n)

Một số ví dụ:

ab: L = { ab }
(aa)* : L = { ε, aa, aaaa, aaaaaa, … }
a*(bb)*: L = { ε, a, aa, aaa, bb, bbbb, abb, aabb, …}

regex trong ngữ cảnh regular language rất khác với regex thực tế trong ngôn ngữ lập trình, một phần do regex trong ngôn ngữ lập trình đã được thêm nhiều extension/syntactic sugar, ví dụ: [a-z], a{n,m}, look-ahead, look-behind, backreferences, … Vì vậy, regex trong các ngôn ngữ lập trình được hiện thực phức tạp hơn regex trong regular language rất nhiều.

Finite automata

Phương pháp để mô hình hoá một quá trình tính toán. Finite automata bao gồm các thành phần:

Σ: tập ký tự, input sẽ là chuỗi chứa các ký tự thuộc tập này.
States: tập các trạng thái(hữu hạn), nên được gọi là Finite Automata.
δ: hàm nhận tham số là trạng thái hiện tại + ký tự input, sẽ cho ra trạng thái tiếp theo.
q: tập các trạng thái bắt đầu
F: tập các trạng thái kết thúc

Để cho dễ hiểu Finite automata thường được biểu diễn bằng dạng graph. Trở lại ví dụ 1, ta có

Σ = { a }
States = { Start, S1, S2 }
q = { Start }
F = { S2 }
δ(Start, a) = S1, δ(S1, a) = S2, δ(S2, a) = S1
$: ký tự đặc biệt đánh dấu kết thúc chuỗi

Có thể dùng bảng(transition table) để mô tả hàm δ như sau:

δ	a	$
Start	S1	End
S1	S2
S2	S1	End

stateDiagram-v2
  direction LR
  [*] --> S1: a
  [*] --> [*]: $
  S1 --> S2: a
  S2 --> S1: a
  S2 --> [*]: $

Automaton này sẽ chấp nhận các input chứa một số chẵn các ký tự a như aa, aaaa.

Ngược lại, chuỗi aaa sẽ không được chấp nhận vì khi xử lý đến ký tự a thứ ba, ta đến trạng thái S1, nhưng do S1 không phải là 1 trạng thái kết thúc cũng như không còn input nào để xử lý => chuỗi không được chấp nhận bởi automaton này.

Định nghĩa ở trên còn được gọi là Deterministic Finite Automata(DFA), ta còn có Nondeterministic Finite Automata(NFA). NFA tương tự như DFA nhưng có thêm một khái niệm ε-transition. Lấy một ví dụ như sau:

stateDiagram-v2
  direction LR
  [*] --> S1: a
  S1 --> S2: ε
  S1 --> S3: a
  S3 --> S1: a
  S2 --> [*]: a
  S3 --> [*]: $

Với ε-transition khi đến một trạng thái ta có thể đi đến trạng thái tiếp theo mà không cần nhận thêm input. Ở ví dụ trên, khi đến trạng thái S1 ta có thể đồng thời đi đến S2(thông qua ε-transition) hoặc S3(nếu input tiếp theo là a). Do ε-transition cho phép di chuyển đồng thời đến nhiều trạng thái, nên tại một thời điểm ta có thể ở nhiều trạng thái khác nhau(nondeterministic). Transition table cho ví dụ trên:

δ	a	$
Start	S1
S1	{ S3, End }
S2	End
S3	S1	End

Một định lý quan trọng của lý thuyết Automata là NFA và DFA tương đương nhau, theo đó ta có thể dùng các phép biến đổi để chuyển 1 NFA thành 1 DFA(và ngược lại). Thuật toán chuyển đổi qua lại giữa NFA và DFA được gọi là Powerset construction.

Khi chuyển đổi NFA -> DFA số lượng trạng thái có thể tăng lên theo cấp số mũ, nghĩa là nếu NFA có n trạng thái, khi chuyển thành DFA có tối đa 2^n trạng thái(đồng nghĩa sẽ tốn nhiều bộ nhớ hơn). Tuy nhiên về mặt hiện thực, DFA có thuật toán hiện thực hiệu quả hơn, NFA do có ε-transition nên cần dùng back-track. Tóm lại:

NFA: n states, O(mn) trong đó m: số states, n: chiều dài input
DFA: 2^n states, O(n)

Định lý Kleene

Định lý Kleene đã cho một kết quả đáng ngạc nhiên về mối liên hệ giữa ngôn ngữ chính quy, biểu thức chính quy và finite automata, theo đó các khái niệm này là tương đương với nhau. Nghĩa là một biểu thức chính quy sẽ tương đương với 1 finite automata, bộ ngữ pháp của ngôn ngữ chính quy tương đương với một biểu thức chính quy.

Nhờ định lý Kleene ta có thể dùng finite automata để hiện thực bài toán matching bằng regular expression.

Trở lại với ví dụ 1 ta có:

Grammar: S ::= ε | aaS

Regular expression: (aa)*

DFA:

stateDiagram-v2
  direction LR
  [*] --> Start
  Start --> S1: a
  Start --> [*]: $
  S1 --> S2: a
  S2 --> S1: a
  S2 --> [*]: $

Cả 3 đều mô ta một regular language trên tập Σ = { a } với các chuỗi có chiều dài là một số chẵn.

Từ định lý Kleene, ta có thể hiểu tại sao không thể dùng regular expression để mô tả L = { tập các cặp dấu (, ) lồng nhau }. Quy tắc ngữ pháp cho tập này S = () | (S) (không theo 2 ràng buộc của regular grammar) => L là context-free language.

Stephen Cole Kleene là một nhà toán học người Mỹ, học trò của Alonzo Church(tác giả lambda calculus). Kleene cùng với Rózsa Péter, Alan Turing, Emil Post có nhiều đóng góp cho lý thuyết tính toán(computation theory), đặt nền móng toán học cho khoa học máy tính.

Trở lại với bài toán parsing cho regular language, trước tiên ta có regular grammar hoặc regular expression, dùng phép biến đổi để xây dựng 1 NFA cho regular expression này, sau đó chuyển NFA sang DFA, cuối cùng dùng table để implement DFA.

Linux ODBC and Microsoft Access

Wed, 07 Jun 2023 00:16:00 +0000

The other day I needed to do some data analysis from a Microsoft Access file on my FreeBSD box. Here are the steps that I took.

Goal: load data from Access into Python to perform data analysis

Library/program used:

Python with PyODBC
UnixODBC
MDBTools

UnixODBC

UnixODBC is ODBC an implementation on Linux/Unix. Installing it is quite easy, we can simply download the source code, run ./configure, make and make install.

MDBTools

To actually connect to a database using ODBC, we’ll also need a driver for that database. The ODBC driver for Access provided by Microsoft doesn’t have a Linux installation, so we’ll need to find a third-party driver.

There are paid libraries, such as the one from Easysoft (closed source but may have better quality). Luckily, I found one open-source driver from MDBTools. MDBTools is a collection of CLI tools to interact with an Access database, and it also contains an ODBC driver. We’ll install this driver.

The source code is at https://github.com/mdbtools/mdbtools

Installation of MDBTools is quite easy, we just need to add the flag --with-unixodbc and set the path to our unixODBC installation. If we installed unixODBC in a non-standard path(e.g.,$HOME), we also need to set CPPFLAGS and LDFLAGS so that MDBTools knows where to find the library and header files.

Config unixODBC

Now we’ll need to config unixODBC so that it knows about our Access Driver, there are two files that we need:

odbcinst.ini (in etc folder): contains the configuration of ODBC drivers
odbc.ini (in etc folder): contains the configuration of Database

We’ll add Access driver to odbcinst.ini as follows:

[MDBTools]
Driver = <path to mdbtools installation>/lib/odbc/libmdbodbc.so

odbc.ini is where we configure our database. For example, if I have an Access file named test.accdb, I will configure it as follows:

[test]
Driver = MDBTools
Database = <path to our database file>/test.accdb

After this, we can use unixODBC to connect to access, for example using its isql command:

isql test
> select * from test_table

Connect from Python

I also wanted to load Access data into Python so I used pyodbc to do that. It’s a wrapper for unixODBC so we’ll need unixODBC installed before we can actually run pyodbc.

To check if pyodbc can actually load unixODBC, we need to import it:

# pip install pyodbc
import pyodbc

If there is an error such as ImportError: Shared object "libodbc.so.2" not found, required by "pyodbc.cpython-310.so", we need to check our unixODBC installation. If we installed it in a non-standard folder, we can set the LD_LIBRARY_PATH environment variable to our unixODBC’s lib folder.

After properly import pyodbc, we can read our Access database as follows:

import pyodbc
file_name = "<path to access file>"
con = pyodbc.connect(f'DRIVER=MDBTools;DBQ={file_name}')
cursor = con.cursor()
cursor.execute('SELECT * FROM test_table').fetchone()

# we can even load our Access data into a Pandas dataframe
import pandas as pd
df = pd.read_sql('SELECT * FROM test_table', con)

Conclusion

Although I can connect and load data from Access, sometimes I get some weird errors from pyodbc (or MDBTools). My suggestion is to save the Access data to other natively supported database systems on Linux, such as SQLite, as soon as we can load it. It’s easy to do that from pandas.

import pyodbc
import sqlite3
import pandas as pd

access_con = pyodbc.connect('DRIVER=MDBTools;DBQ=test.accdb')
sqlite_con = sqlite3.connect('test.sqlite3')

df = access_con.read_sql('SELECT * FROM test_table', access_con)
df.to_sql('test_table', sqlite_con)

AWS Cognito: client and server authentication

Wed, 15 Mar 2023 00:16:00 +0000

I have struggled for quite some time with setting up Cognito and integrating it into a web application as an authentication service. The documentation from AWS is not very clear on how to set it up and how the different pieces fit together. This post explains how I did it.

The setup

Client side

1. Create user pool

A user pool is like your users’ table. It will contain user information such as email, username, password, etc. The first step with Cognito is to create a user pool.

AWS has a step-by-step guide on setting up a user pool at Tutorial: create user pool

2. Create an app client

An app client specifies how your app will authenticate with your user pool. You can choose a client → Cognito authentication flow, client → server → Cognito flow, or even a custom flow.

For more detail, refer to this guide: Configuring a user pool app client

In this post, I’ve choosen client → cognito flow(no need to implement a server).

3. Create hosted UI

Hosted UI is AWS’s provided login page, if we don’t want to create a login page, we can use hosted UI. We can make some simple modification to the look of hosted UI by provide our own css file. Here is what we can custom the hosted UI: Customizing the built-in sign-in and sign-up webpages

4. Client’s code

4.1. Redirect to hosted UI

After creating a hosted UI, Cognito will provide an endpoint for us to redirect users. They will go to this endpoint, login(or register) and after a success login will be redirected back to our application. Our app will receive a code parameter that we’ll use to exchange for access token.

4.2. Callback endpoint

After user has successfully login, they will be redirected back to callback endpoint with code parameter. We will use this code to exchange for access_token.

var myHeaders = new Headers();
myHeaders.append("Content-Type", "application/x-www-form-urlencoded");

var urlencoded = new URLSearchParams();
urlencoded.append("grant_type", "authorization_code");
urlencoded.append("client_id", process.env.AWS_COGNITO_APP_CLIENT_ID);
urlencoded.append("code", code);
urlencoded.append("redirect_uri", "https://localhost:1234/callback");

var requestOptions = { method: 'POST', headers: myHeaders, body: urlencoded, redirect: 'follow' };

const response = await fetch("https://sakura-vinh.auth.ap-northeast-1.amazoncognito.com/oauth2/token", requestOptions)
const body = await response.json();

5. Server side

Now let’s say we have an API server and we need to protect it using Cogito user’s pool. I’ll use express and cognito-express package.

const cognitoExpress = new CognitoExpress({
  region: "ap-northeast-1",
  cognitoUserPoolId: process.env.AWS_COGNITO_USERPOOL_ID,
  tokenUse: "access",
  tokenExpiration: 3_600_000
});

API with authentication

We’ll add a middleware to extract access_token from Authorization header and check it against our user’s pool on Cognito

authenticatedRoute = express.Router();
app.use("/api", authenticatedRoute);
authenticatedRoute.use(function(req, res, next) {
  let accessTokenFromClient = extractToken(req.headers.authorization);
  if (!accessTokenFromClient) return res.status(401).send("Access Token missing from header");
  cognitoExpress.validate(accessTokenFromClient, function(err, response) {
    if (err) return res.status(401).send(err);
    res.locals.user = response;
    next();
  });
});

const extractToken = (authHeader) => {
  if (authHeader.startsWith("Bearer ")) {
    token = authHeader.substring(7, authHeader.length);
    return token;
  } else {
    throw Error("invalid header");
  }
}

after that we can put an API endpoint behind the middleware

authenticatedRoute.get("/hello", function(_req, res, _next) {
  res.send(`Hi ${res.locals.user.username}, your API call is authenticated!`);
});

6. Cognito’s identity pool

One interesting feature of Cognito is that we can allow our users to have temporary access to AWS. For example, an authenticated user can call a Lambda function from the front-end.

To do this, we need to create an identity pool (also known as Federated Identities). With an identity pool, we can use Cognito’s user pool as an identity source, or we can use other OAuth services such as Facebook, SAML, etc.

I’ll use my user pool as the identity source.

Create an identity pool.
Create a role with appropriate permissions.
Client code.

From 4.2 above we can get access_token, id_token and refresh_token from Cognito. We will use this id_token to exchange for an Identity ID using GetID command

import { CognitoIdentityClient, GetIdCommand, GetCredentialsForIdentityCommand } from "@aws-sdk/client-cognito-identity";
const client = new CognitoIdentityClient({ region: process.env.AWS_DEFAULT_REGION });
const logins = {
  [`cognito-idp.ap-northeast-1.amazonaws.com/${process.env.AWS_COGNITO_USERPOOL_ID}`]: info.id_token
}
const getIdParams = {
  IdentityPoolId: process.env.AWS_COGNITO_IDENTITY_POOL_ID,
  Logins: logins
}
const command = new GetIdCommand(getIdParams);
const getIdResponse = await client.send(command);

Then using Identity ID to exchange for AWS’s access_token by calling GetCredentialsForIdentity

const getCredsParams = {
  IdentityId: getIdResponse.IdentityId,
  Logins: logins
}
const credCommand = new GetCredentialsForIdentityCommand(getCredsParams);
const credsResp = await client.send(credCommand);
return credsResp.Credentials;

We can use this Credentials to call other AWS service, such as

const getSTSInfo = async (credential) => {
  const { AccessKeyId, Expiration, SecretKey, SessionToken } = credential;
  const credentials = {
    accessKeyId: AccessKeyId,
    expiration: Expiration,
    secretAccessKey: SecretKey,
    sessionToken: SessionToken
  }
  const client = new STSClient({ credentials, region: process.env.AWS_DEFAULT_REGION });
  const command = new GetCallerIdentityCommand({});
  try {
    const response = await client.send(command);
    console.log(response);
  } catch (error) {
    console.log(error);
  }
}

Basic usage of libuv

Sat, 11 Mar 2023 00:16:00 +0000

libuv and its relation with nodejs

When working with Javascript, there is the concept of event loop and the non-blocking model that we should aware of. Every Javascript runtime needs to have its own event loop implementation.

An event loop is like a job queue where events will be pushed into it and be processed in time. Those events can be an IO event, a user’s input, etc. Each event will be associated with a callback(or handler) that the event loop calls when it proccess the event.

NodeJS use libuv as its event loop library. In this post, I will make some simple program with libuv.

Programming model of livuv

Initialize the event loop
Create event handler
Run event loop

Initiate event loop

The center part of event loop’s initialization is uv_loop_t struct and uv_loop_init function.

uv_loop_init signature:

int uv_loop_init(uv_loop_t* loop)

Each event loop has its own data structure called uv_loop_t. To init a loop, we use uv_loop_init.

So our code could be like:

#include <uv.h>

int main() {
  uv_loop_t loop;
  uv_loop_init(&loop);
}

You may notice that all libuv public function and data structure have uv_ as its prefix.

Create event handler

libuv support various type of IO events, including:

File IO operations such as open/read/write file, pipe,…
Networking operations such as socket(TCP, UDP)

File operation is just the same as normal synchronous version of libc. We have 3 main functions for file operations:

uv_fs_open

This function is use to open a file, its signature:

int uv_fs_open(uv_loop_t* loop, 
               uv_fs_t* req, 
               const char* path, 
               int flags, 
               int mode, 
               uv_fs_cb cb)

loop is the one we’ve created at step 1.
req: everytime we create an event handler, we need to create its coresponding request. The request depends on IO operation. In file IO, request’s type is uv_fs_t. We will need to create and initialize this variable before passing it to uv_fs_open
path, flags, mode is the same as open(2)
that last parameter is our callback function. It needs to have a signature of void callback(uv_fs_t* req)

So to open a file, we’ll use it as:

void open_cb(uv_fs_t *req) {
  // a do nothing callback
}

uv_fs_t open_req;
uv_loop_t loop;
int main(){
  uv_loop_init(&loop);
  uv_fs_open(&loop, &open_req, filepath, O_RDONLY, NULL, open_cb);
}

The open_cb is our callback function which will be called after file-open operation finished. File descriptor will pass to req->result.

Notice that until we start the event loop, no operation will be run yet.

After open a file, we can read and write to it using uv_fs_read and uv_fs_write. Their signature are as follow:

int uv_fs_read(uv_loop_t* loop, 
               uv_fs_t* req, 
               uv_file file, 
               const uv_buf_t bufs[], 
               unsigned int nbufs, 
               int64_t offset, 
               uv_fs_cb cb)`

int uv_fs_write(uv_loop_t* loop, 
                uv_fs_t* req, 
                uv_file file, 
                const uv_buf_t bufs[], 
                unsigned int nbufs, 
                int64_t offset, 
                uv_fs_cb cb)`

loop, cb and req is the same as uv_fs_open
file is the file descriptor(result of open request, open_req.result)
bufs is an array of uv_buf_t each element represents a buffer that we’ll read from/write to. This allows us to read/write into multiple buffers at the same time.
nbufs is length of bufs
offset is the position to read from/write to. -1 for current file pointer.

Now let add a file-read operation. It will happen inside our open_cb.

char buffer[1024];

void open_cb(uv_fs_t *req) {
  // req->result is file descriptor
  iov = uv_buf_init(buffer, sizeof(buffer));
  uv_fs_t read_req;
  uv_fs_read(uv_default_loop(), &read_req, open_req.result,
    &iov, 1, -1, read_cb);
}

void read_cb(uv_fs_t *req) {
  // req->result is number of character read
  // let just write it back to stdout
  if (req->result > 0) {
    printf("%.*s", req->result, buffer);
    // continue reading till EOF
    iov = uv_buf_init(buffer, sizeof(buffer));
    uv_fs_t read_req;
    uv_fs_read(uv_default_loop(), &read_req, open_req.result,
      &iov, 1, -1, read_cb);
  } else if (req->result == 0) {
    // we've reached EOF
  }
}

Run event loop

To start event loop, we’ll use uv_run

uv_run(loop, UV_RUN_DEFAULT);
uv_loop_close(loop); // also clean up after event loop return

References

libuv documentation

Javascript’s event loop model

Writing a C Compiler

Wed, 12 Jan 2022 00:16:00 +0000

This year I’ll try learning about compiler by writing a toy C compiler. I’ll roughly follow chibicc but will use C++ and target x64 on OSX.

Repo is at mycc

Hopefully, I won’t drop it.

Ruby 3: Non-blocking Fiber

Fri, 25 Dec 2020 00:16:00 +0000

Ruby 3 was just released today(2020/12/25) with a lot of new features, among them are:

Static type
Ractor
Non-blocking fiber

I’ll focus on non-blocking fiber in this post.

I have written an introduction about Fiber some months ago. In case you haven’t heard about Fiber before, here it is: Event driven non blocking IO with Ruby’s fiber

Notable change to Fiber in Ruby 3 is the new boolean option blocking for every Fiber.new operation. So what does this keyword mean?

When blocking: true, fiber will act as it is before.

The story becomes interesting when blocking: false(this is the default). It will make this fiber to become a non-blocking fiber. Inside this fiber whenever there is a blocking operation such as IO, network, sleep, etc, it will be suspended(ie yield) and transfer control to other fiber.

Obviously, the next question is how can the former fiber continue its execution when says the IO operation finished? We will need a scheduler(ie an event-loop) to keep track of which fiber is in blocking state and when it’s posible to resume.

Ruby doesn’t provide a scheduler. It’s up to us to implement the scheduler whatever we like.

In the next part, we will implement a simple scheduler.

Fiber.scheduler

A fiber scheduler is like an event-loop. It keeps track of which fiber is in blocking state and its corresponding blocking operation. When blocking operation finished or unblocked, scheduler will resume its execution.

To implement a scheduler, we must implement the following hooks:

io_wait
process_wait
kernel_sleep
block and unblock

These hooks will be called when a non-blocking fiber call a blocking operation(eg: IO(io_wait), sleep(kernel_sleep))

A naive implementation of scheduler can be like

class Scheduler
  # trigger by events: IO#wait, IO#wait_readable, IO#wait_writeable
  def io_wait(io, events, timeout)
  end

  # trigger by events: Kernel#sleep, Mutex#sleep
  def kernel_sleep(duration = nil)
  end

  # trigger by events: Process::Status.wait
  def process_wait(pid, flags)
  end

  # trigger by events: Thread#join, Mutex
  def block(blocker, timeout = nil)
  end

  # trigger when a previous block called is unblock
  def unblock(blocker, fiber)
  end
  
  # Called when current thread exits
  def close
  end
end

With a scheduler, we can start using non-blocking Fiber by calling Fiber.set_scheduler

Now, let implement a simple kernel_sleep hook.

kernel_sleep

kernel_sleep hook will be called for events such as Kernel.sleep. Our scheduler will need to support kernel_sleep and close hooks

require 'fiber'

class SimpleScheduler
  def initialize
    # our internal Hash to track which Fiber is sleeping and for how long
    @waiting = {}
  end

  def run
    # we will loop until there is no more event
    # Our event loop for now will only check for sleeping fibers
    while @waiting.any?
      @waiting.keys.each do |fiber|
        # fiber needs to wake up
        if current_time > @waiting[fiber]
          @waiting.delete(fiber)
          fiber.resume
        end
      end
    end
  end

  def kernel_sleep(duration = nil)
    # this function(and other hooks) will run in context of the fiber calling sleep
    # hence Fiber.current will be our target fiber that need to be halted
    @waiting[Fiber.current] = current_time + duration
    # halt this fiber and transfer control to its parent
    Fiber.yield
    return true
  end

  def close
    run
  end

  private
  def current_time
    Process.clock_gettime(Process::CLOCK_MONOTONIC)
  end
end

To use our SimpleScheduler, we will need to call Fiber.set_scheduler

scheduler = SimpleScheduler.new
Fiber.set_scheduler(scheduler)

# now we're ready to call sleep
# let's create two fiber
Fiber.new do
  puts "Fiber 1: sleep for 2s"
  sleep(2)
  puts "Fiber 1: wake up"
end.resume

Fiber.new do
  puts "Fiber 2: sleep for 3s"
  sleep(3)
  puts "Fiber 2: wake up"
end.resume

which when running will result

Fiber 1: Sleep 2s
Fiber 2: Sleep 3s
Fiber 1: wakeup
Fiber 2: wakeup

We can see that while fiber 1 is sleeping, it is halted and fiber 2 gets its chance to run. Our scheduler keeps track of how long these fibers needs to sleep then resume when the time comes.

io_wait

We’ll use this simple program to test io_wait

require 'net/http'
Fiber.new do
  Net::HTTP.get(URI('https://www.google.com'))
end.resume

Fiber’s document of io_wait reads:

io_wait(io, events, timeout):

events is a bit mask of IO::READABLE, IO::WRITABLE, and IO::PRIORITY.

Expected to return the subset of events that are ready immediately.

so the stragegy here is to maintain two hash inside our scheduler, @readable and @writable. We’ll use IO#select to know when there is an io object’s ready.

class SimpleScheduler
  def initialize
    # our internal Hash to track which Fiber is waiting for a read event
    @readable = {}
    # our internal Hash to track which Fiber is waiting for a write event
    @writable = {}
    @waiting = {}
  end

  def io_wait(io, events, timeout)
    # events is a bit mask of IO::READABLE, IO::WRITABLE, and IO::PRIORITY.
    unless (events & IO::READABLE).zero?
      @readable[io] = Fiber.current
    end
  
    unless (events & IO::WRITABLE).zero?
      @writable[io] = Fiber.current
    end

    # small exercise: add IO::PRIORITY
  
    # halt fiber and transfer control to its parent
    Fiber.yield
    # when it's ready to read/write, resume fiber
    return events
  end
end

now, in our event-loop, we will need to handle @readable and @writable

class SimpleScheduler
  def run
    while @readable.any? or @writable.any? or @waiting.any?
      # select's signature:
      # select(read_array [, write_array [, error_array [, timeout]]]) → array or nil
      readable, writable = IO.select(@readable.keys, @writable.keys, [], 0)

      # when there's an io that's ready
      # we remove it from our @readable/@writable and resume its corresponding fiber
      readable&.each do |io|
        fiber = @readable.delete(io)
        fiber.resume
      end

      writable&.each do |io|
        fiber = @writable.delete(io)
        fiber.resume
      end

      @waiting.keys.each do |fiber|
        if current_time > @waiting[fiber]
          @waiting.delete(fiber)
          fiber.resume
        end
      end
    end
  end
end

However, when run this scheduler against our test program(the one making http request to google), we’ll get this exception:

undefined method ‘block’ for #<SimpleScheduler>

which implies that somewhere in Ruby’s http library, block/unblock hook is called.

Let’s add block/unblock

block/unblock

# our block's implementation is pretty simple
# we maintain @blocking variable, tracking how many fibers are blocked
# increase @blocking when there's block event
# decrease it when it's resumed
def block(blocker, timeout = nil)
  @blocking += 1
  begin
    Fiber.yield
  ensure
    @blocking -= 1
  end
end

# this is when previously block fiber is unblock
# we'll use an IO.pipe to notify event-loop that there's fiber unblocked
#  ie: @urgent = IO.pipe
# @ready is an array containing fiber that are unblocked
def unblock(blocker, fiber)
  @ready << fiber
  # we'll write to the pipe to notify event-loop
  io = @urgent.last
  io.write_nonblock('.')
end

# we also need to update our event-loop
def run
  # let's review what we're tracking:
  # @readable: for fibers that are waiting to read
  # @writable: for fibers that are waiting to write
  # @waiting: for fibers that are sleeping
  # @blocking: for fibers that are blocked
  # @ready: for fibers that are unblocked
  while @readable.any? or @writable.any? or @waiting.any? or @blocking.positive? or @ready.any?
    readable, writable = IO.select(@readable.keys + [@urgent.first], @writable.keys, [], 0)

    readable&.each do |io|
      if fiber = @readable.delete(io)
        fiber.resume
      end
    end

    writable&.each do |io|
      if fiber = @writable.delete(io)
        fiber.resume
      end
    end

    @waiting.keys.each do |fiber|
      if current_time > @waiting[fiber]
        @waiting.delete(fiber)
        fiber.resume
      end
    end

    # we simply resume all fibers in @ready
    ready, @ready = @ready, []
    ready.each do |fiber|
      fiber.resume
    end
  end
end

I’ve added the complete implemetation of simple_scheduler at the end of this post.

Conclusion

Non-blocking Fiber provides a simple interface to handle non-blocking IO. To us user, we probably don’t deal directly with fiber but through other wrapper such as Async.

In Ruby 3.0, non-blocking Fiber is still in development, there’re some IO event that aren’t Fiber-friendly. We’ll need to wait sometime for it to become mature.

Lastly, there’re some improvements that I’ll leave as an exercise

In our IO.select call, we return immediatelly if there is no event(by passing 0 as its last parameter). It would be more efficent if it can wait for a while. How can we achieve that?
Implement process_wait hook.
Our event-loop is while loop that will eat up CPU time is there is no event. Can we improve it?

Ruby 3: Non-blocking Fiber

Fri, 25 Dec 2020 00:16:00 +0000

Ruby 3がついにリリース！(2020/12/25)

Ruby 3が今日（2020/12/25）ついにリリースされた！
新機能がたくさん追加されてるけど、特に注目すべきなのは以下の3つ：

静的型（Static type）
Ractor（並行処理の新機能）
ノンブロッキングファイバー（Non-blocking Fiber）

今回はこの中でも ノンブロッキングファイバー について掘り下げてみるよ。

そもそもFiberって？

実は数ヶ月前にFiberの入門記事を書いたことがある。
もしFiberについて知らないなら、まずはこっちを読んでみて！

Rubyのファイバーを使ったイベント駆動型ノンブロッキングIO

Ruby 3のFiberに追加された`blocking`オプション

Ruby 3では、Fiber.newに blockingオプション が追加された。
これがどういう意味を持つのか見ていこう。

blocking: true → これは今までのFiberと同じ動作。
blocking: false（デフォルト）→ ノンブロッキングファイバー になる。

blocking: falseのファイバーは、IOやネットワーク通信、sleepなどのブロッキング操作を行うと、自動でyieldして制御を他のファイバーに渡す。
これによって、処理が止まらずスムーズに並行処理できるようになる！

どうやって処理を再開するの？

じゃあ、IO操作が終わった後、どうやって元のファイバーを再開させるの？
答えは スケジューラー（イベントループ） だ！

スケジューラーは「どのファイバーがブロック中なのか」を管理し、処理を再開できるタイミングでファイバーをresumeする。
Ruby自体にはスケジューラーが組み込まれてないので、自分で実装する必要がある。

次の章では、簡単なスケジューラーを作ってみよう！

Fiber.scheduler を作ってみる！

スケジューラーの役割はシンプル。

どのファイバーがブロック中かを管理する
ブロックが解除されたら、そのファイバーを再開する

スケジューラーを作るには、次のフックを実装する必要がある：

io_wait（IO待ち）
process_wait（プロセス待ち）
kernel_sleep（sleep処理）
block / unblock（ロック処理）

これらのフックは、ノンブロッキングファイバーがブロッキング操作を行ったときに呼び出される。

超シンプルなスケジューラーの実装

まずは最低限のスケジューラーを作ってみよう。

class Scheduler
  def io_wait(io, events, timeout)
  end

  def kernel_sleep(duration = nil)
  end

  def process_wait(pid, flags)
  end

  def block(blocker, timeout = nil)
  end

  def unblock(blocker, fiber)
  end
  
  def close
  end
end

これをFiber.set_schedulerで登録すれば、ノンブロッキングファイバーを動かせるようになる。

`kernel_sleep` フックを実装しよう

まずは sleep をノンブロッキングで動かせるようにしてみよう。

require 'fiber'

class SimpleScheduler
  def initialize
    @waiting = {} # どのファイバーがいつまでsleepするかを管理
  end

  def run
    while @waiting.any?
      @waiting.keys.each do |fiber|
        if current_time > @waiting[fiber]
          @waiting.delete(fiber)
          fiber.resume
        end
      end
    end
  end

  def kernel_sleep(duration = nil)
    @waiting[Fiber.current] = current_time + duration
    Fiber.yield
    return true
  end

  def close
    run
  end

  private
  def current_time
    Process.clock_gettime(Process::CLOCK_MONOTONIC)
  end
end

これで sleep がブロッキングしなくなる！

`io_wait` を実装する

次に、IOのノンブロッキングを実装しよう。

class SimpleScheduler
  def initialize
    @readable = {}
    @writable = {}
    @waiting = {}
  end

  def io_wait(io, events, timeout)
    unless (events & IO::READABLE).zero?
      @readable[io] = Fiber.current
    end
  
    unless (events & IO::WRITABLE).zero?
      @writable[io] = Fiber.current
    end

    Fiber.yield
    return events
  end
end

IOの状態を監視して、準備ができたらファイバーを再開させる感じだね。

`block/unblock` の実装

最後に、ブロッキング状態の管理を追加しよう。

def block(blocker, timeout = nil)
  @blocking += 1
  begin
    Fiber.yield
  ensure
    @blocking -= 1
  end
end

def unblock(blocker, fiber)
  @ready << fiber
  io = @urgent.last
  io.write_nonblock('.')
end

この機能を組み込めば、HTTPリクエストなどの非同期処理もスムーズに扱える！

まとめ

Ruby 3のノンブロッキングファイバーは、今まで以上に効率的な並行処理を実現できる！
ただし、Ruby自体にはスケジューラーが含まれていないので、自分で実装するか、Async のようなライブラリを使うのがベスト。

とはいえ、まだ開発中の機能なので、すべてのIO処理が完全にノンブロッキングになっているわけではない。
今後のアップデートでより使いやすくなるのを期待しよう！

改良ポイント

IO.select のタイムアウトを適切に設定すると、CPU負荷を減らせるかも？
process_wait の実装も試してみよう！
イベントループの効率化も考えてみよう！

Ruby 3、めちゃくちゃ進化してるね！
ノンブロッキングファイバーを活用して、もっと効率的なコードを書いていこう

Ruby Zero

Tunnel SSH connection using cloudflared

1. Download and install cloudflared

2. Configure cloudflared

3. Set Up the Route

4. Connect to Your Private Server

Getting Django to Work on a Cheap Shared Hosting Plan

Module in JavaScript

Module in NodeJS

CommonJS

ESM

Calling between CommonJS and ESM

Module in browser

ESM in browser

importmap

Biểu thức chính quy và automata

Formal language & BNF grammar

Bài toán parsing

Chomsky hierarchy

Regular expression

Finite automata

Định lý Kleene

Linux ODBC and Microsoft Access

UnixODBC

MDBTools

Config unixODBC

Connect from Python

Conclusion

AWS Cognito: client and server authentication

The setup

Client side

1. Create user pool

2. Create an app client

3. Create hosted UI

4. Client’s code

4.1. Redirect to hosted UI

4.2. Callback endpoint

5. Server side

6. Cognito’s identity pool

Basic usage of libuv

libuv and its relation with nodejs

Programming model of livuv

Initiate event loop

Create event handler

Run event loop

References

Writing a C Compiler

Ruby 3: Non-blocking Fiber

Fiber.scheduler

kernel_sleep

io_wait

block/unblock

Conclusion

Ruby 3: Non-blocking Fiber

Ruby 3がついにリリース！(2020/12/25)

そもそもFiberって？

Ruby 3のFiberに追加されたblockingオプション

どうやって処理を再開するの？

Fiber.scheduler を作ってみる！

超シンプルなスケジューラーの実装

kernel_sleep フックを実装しよう

io_wait を実装する

block/unblock の実装

まとめ

改良ポイント

2. Configure `cloudflared`

Ruby 3のFiberに追加された`blocking`オプション

`kernel_sleep` フックを実装しよう

`io_wait` を実装する

`block/unblock` の実装