Raspberry Pi libcamera low-latency http streaming

  • model: Raspberry Pi 4B
  • system: Ubuntu 22.04 aarch64

Through resources of Pi http streamings are widely available over the Internet, few address the libcamera library which is the only option under aarch64, and few address the low-latency part of the streaming.

I managed to achieve the above with the following:

This process has a very low latency due to the rendering workload being undertaken by the client, while Pi just hardware encoding the video into h264 and serving the binary directly.

pipeline

libcamera-vid
libcamera-vid
scene
scene
websockify
websockify
tcp+h264
tcp+h264
nginx
nginx
websocket+h264
websocket+h264
nginx
nginx
html
html
jmuxer
jmuxer
html
h264
html...
html
video
html...
Text is not SVG - cannot display

install

  • compile libcamera following the guide
  • compile libcamera-apps following the guide
  • install websockify following the guide
  • launch the following, or optionally create systemd services of these commands
    libcamera-vid -t 0 --width 1920 --height 1080 --inline --listen -o tcp://0.0.0.0:8000
    websockify 0.0.0.0:8001 0.0.0.0:8000
  • create static nginx html server with /rpicam/index.html as following

    <head>
    <meta name="color-scheme" content="dark">
    <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/[email protected]/dist/jmuxer.min.js"></script>
    <style>
        body{
            margin: 0;
        }
    </style>
    </head>
    <body>
    <div id="container" style="width: 100%; margin: 0 auto;">
        <div class="vsc-controller"></div>
        <video width="100%" autoplay muted id="player"></video>
    </div>
    <script>
        function parse(data) {
            var input = new Uint8Array(data),
            video = input;
    
            return {
                video: video,
            };
        }
    
        window.onload = function () {
            var socketURL = document.location.href.replace('http', 'ws')+'ws/';
            var jmuxer = new JMuxer({
                node: 'player',
                mode: 'video',
                flushingTime: 0,
                fps: 30,
                debug: false
            });
    
            var ws = new WebSocket(socketURL);
            ws.binaryType = 'arraybuffer';
            ws.addEventListener('message', function (event) {
                var data = parse(event.data);
                jmuxer.feed(data);
            });
        }
    </script>
    </body>
  • configure nginx server to serve /rpicam/ to static file directory, and /rpicam/ws/ to 0.0.0.0:8001. Typical configuration of the later is as following.
    location /rpicam/ws/ {
    proxy_redirect off;
    proxy_pass http://0.0.0.0:8001;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    }

known issues and solutions

  • current configuration can support only one client at a time
    • a dedicated backend to stream tcp/websocket data to multiple clients would be necessary
  • libcamera-vid will exit each time websocket is closed
    • auto restart is possible using systemd
    • if one have a dedicated backend this can be managed
  • no audio is served. jmuxer supports audio, to combine audio one will have to either
    • write a dedicated backend to repackage the binary data by adding audio and duration information
    • or serve audio through a seperate websocket, but could lead to audio/video being not synchronized

For now I am happy with the result, but a dedicated backend would be optimal/necessary to solve these issues.

Virtualize Dual-boot Linux

Comparing with WSL2

  • Pro
    • Allow maximum performance when using physical machine
    • Have Native GUI which can be accessed using VirtualBox
    • Native disk operation performance under virtual machine
  • Con
    • No app optimization and support in Windows, e.g. vscode, cuda (but vscode has ssh feature, and you have native linux cuda under physical machine)
    • Mapped file in RAM is not managed by Windows, naturally, thus requiring a larger RAM (Your RAM stores mapped files from both host Windows and VM Linux now)
  • Even
    • Other performance under windows, e.g disk operation across systems (using SMB)

Chrome OS on Surface Pro 7

20210113

Working

except # Not Working

Not Working

  • Multi-touch Finger Input (Single-touch works)
  • Stylus Input
  • Camera
  • Surprise, dark mode hasn't been implemented in Chrome OS 87!

Have Problem Working

Speaker will make hissing noises with headphone injected

Mechanism

Chrome OS cannot be installed directly to PC due to hardware compatibility. Some hardware, e.g. the Intel Precise Touch Screen of Surface Pro, even requires special driver. Moreover, Chrome OS, or Android, are built to install on the entire disk (not a partition).

'brunch' is a framework project featuring these problems by including support for PC hardwares and installing Chrome OS on a .img disk mirror file.

Repo

sebanc/brunch: Boot ChromeOS on x86_64 PC (supports most Intel CPU/GPU or AMD Stoney Ridge)

Rammus recovery bin from CrOS Updates Serving

Bottle neck upstream repo

linux-surface/iptsd: Userspace daemon for Intel Precise Touch & Stylus

Install script

I use WSL to pack up the '.img' file. Due to the warning given by brunch readme, I made a special partition, a 32 GB NTFS 'G:\', i.e. '/mnt/g/' in wsl for safety concern. The img size is thus set at 31 (GB).

The reason to use NTFS is to facilitate disk operation in Windows. An EXT4 file system will work too, but not FAT32 due to lack of support of large file.

sudo apt-get install pv
sudo apt-get install cgpt
sudo bash chromeos-install.sh -src chromeos_13505.73.0_rammus_recovery_stable-channel_mp-v2.bin -dst /mnt/g/chromeos.img -s 31;

GRUB2

Need to disable Secure Boot and Bitlocker first.

Turning Secure Boot back on is an easy method to disable GRUB - thus booting directly into Windows.

I use grub2win for multi-boot. Add the boot code generated by brunch (next to the generated '.img' file) to grub.cfg to add the Chrome OS boot entry, and specify options=ipts in kernel parameters to enable touchscreen input.

Update

Follow instructions of BiteDasher/brcr-update: Script to update Chrome OS installed using the brunch framework

n2n

n2n is a solution for Virtual LAN.

With VLAN many things can be quite easy, e.g. SMB sharing across NAT.