Tag: Python

v2ray-configurator

Go, GORM, and gRPC

v2ray-core is written in Go.

So I tried GORM as yet another alternative to SQLAlchemy and Django. GORM can manage the database by migration just as Django does, and this part works fluently.

As to the config side, it feels uncomfortably hard to implement a model-based configurator in Go. Despite I can get all the struct directly from v2ray-core, they come with very strict type definitions. Implementing another model feels unnecessary, and is troublesome too - no type union, no constructor, no default value for functions. Indeed, Go is not an object-oriented language.

For scenarios where performance is not a problem, Python will do so I am going to save the trouble of reimplementing the whole project yet again in Go.

However, I do still need Go for gRPC for some features of v2ray to work. I can either use a Python package for that, or I can use the GORM model defined earlier to write gRPC directly in Go. The first approach is considered better in the long run because I can save the trouble of aligning the Python and Go ORM model.

v2ray-configurator

ordered list in a database

Previous development reveals that relation-based databases cannot properly handle arrays that need insert() method.

A hack is to use a string-based index that allows gaps between indexes so that insertion can be realized without reordering the rest of the entries. 1

This is feasible on the SQL level but the availability under ORM tools still needs investigation.

ORM: SQLAlchamy vs Django

On the ORM side, I also tried Django as an alternative to SQLAlchamy.

Some major difference includes:

  • Django builds the database according to the model, while SQLAlchamy requires manual mapping. This may be a desired feature for fast development but comes at a cost of less extensibility as an ORM
  • Django model object links to each other by a query manager while SQLAlchamy object links to each other directly. The Django way will cause a problem in type hint that would require further hacks like django_hint and manual type decalration everytime a query is made. I consider this as a feature missing bug of Django
  • Django comes with a lot of other features since it's developed as more of a website backend than a simple ORM

  1. Django: define a name for reverse ForeignKey - Stack Overflow ↩

v2ray-configurator

motivation

I spent 2 days redeveloping a v2ray config generator to fulfill my increasing demand for proxy server configuration.

Currently, I have more than 20 different simple configurations (i.e. one-level proxy) for more than 10 servers. Despite a lot of them being just servers of different domains to balance the load across CDN, managing all of them with complex proxy chains is a headache. A typical proxy chain configuration can reach 500+ lines of JSON, which is quite some work to write even one. Moreover, several of my friends are using the services too, which makes the configuration even harder.

I wrote a super long python script to work with a sqlite3 database one year ago trying to automate the process. It worked well until the demand for proxy chains arises. Due to the lack of knowledge and foresight, I use raw SQL to extract information into python dictionaries. This works at a simple level, but the model information is lost during the process. The work became exponentially more difficult as the requirement became complex.

With all the knowledge acquired last year, I re-implement the program with python and sqlite3, again. But this time I use sqlalchamy and jsonpickle so that I can model the configuration instead of using just dictionaries. It turned out that such a method facilitates my development to a great extent.

I also rebuild the database model. The complete database structure is shown below.

service.chain and virtual_server.chain actually depends on address_group, server, reverse_server, and virtual_server. However, these information are stored in JSON so can not be seen from the database structure.

At this stage, I only implement the config class of my requirement so a lot of functions of v2ray are neglected, thus not ready to be published as a public library.

Project: Terrain Modelling

This is a sub project of the ongoing project: Hengjin Villiage Plan.

Terrain Model Report

DEM data source

GoogleEarth - RenderDoc - Blender

earth.google.com not seamless, unable to use

SketchUp - Digital Globe

low resolution

picture 1

Dataset: ASTGTMV003_N29E121

DOI 10.5067/ASTER/ASTGTM.003 earthdata 2000-03-01 00:00:00 - 2013-11-30 23:59:59 resolution of 1 arc second (approximately 30 meter horizontal posting at the equator)

picture 2

Dataset: ALPSRP133840570-RTC_HI_RES

alaska.edu&resultsLoaded=true&granule=ALPSRP133840570-RTC_LOW_RES&dataset=ALOS&productTypes=RTC_LOW_RES,RTC_HI_RES) earthdata 2008-07-29 14:21:47 - 2008-07-29 14:21:55 resolution of 12m

picture 3

Satalite: TerraSAR-X

esa.int region not found

image data source

SketchUp - Digital Globe

Photoshop: remove excessive water print, and additional drawing Illustrator: vectorization caffe: scale up resolution

Satalite: Worldview-3

esa.int resolution of 0.3m proposal required, need waiting within 14 days owned by digital globe so potentially we may have already get it

Satalite: 高分二号

resdc.cn resolution of 0.8m 要钱 分辨率不如Worldview-3

Continue reading Project: Terrain Modelling

Project: create_m3u_from_NeteaseCloudMusic

This project is completed reworked for much better performance and .m3u8 compliance. See release/project-netease-cloudmusic-playlist-exporter/

Netease is working hard to prevent users from listening to local music, especially those VIP exclusive ones. Today (20200531) they even replaced the UWP app with the win32 version in Microsoft Store to force users to accept their new rules: No more VIP music unless you paid, even local ones are not permitted, which is totally ridiculous.

The original UWP app, despite deprecated quite long ago and a little bit ugly, was quite useful on windows tablets as UWP apps are the only choice to play music with the screen turned off.

The project has 2 parts: to find a UWP music player, and exporting the Netease Cloudmusic playlist to it. After some searching, the foobar2000 mobile (not foobar2000) seems to be the only choice.

The exporting process uses Netease Cloudmusic local SQLite database to retrieve the playlist information. Introducing project create_m3u_from_NeteaseCloudMusic: Command Line Tool to Extract ".m3u8" Playlist out of Netease Cloudmusic Library (forked from xyqyear/create_m3u_from_NeteaseCloudMusic)

Usage

Exporter.exe

(wich is equivalent to 'python Exporter.py')

Show help

Exporter.exe -h

Specify playlist to export, or export all

Exporter.exe [-p PLAYLISTNAME]

With this updated version of the command-line tool, an automatic update can be easily done with a '.bat'. Example:

@echo off
REM declare using UTF-8 encoding
chcp 65001
%UserProfile%\AppData\Local\Programs\Python\Python38\python.exe %UserProfile%\repo\create_m3u_from_NeteaseCloudMusic\Exporter.py --playlist 我喜欢的音乐

The way Foobar2000 mobile works will always ensure the updated '.m3u8' playlist be used.