subsyt

description

subsyt is a wrapper around yt-dlp1 to download youtube channels based on your subscriptions. Downloads land in an isolated staging directory before being organised into {show}/{season} folders under your configured media library. During the organising step the tool generates nfo files, extracts thumbnails, downloads posters, banners, and fanart so the media should plug into media libraries well-enough, e.g. Jellyfin and Kodi.

A quick rundown on how to use it:

• download subsyt or build it into a binary yourself2
• install yt-dlp3
• patch it with POT support (POT optional -- yet recommended) 4
• setup a config file5
• run subsyt6

Subscriptions are managed via the subsytctl CLI or HTTP API and stored in a local SQLite database.

install

go install git.meatbag.se/varl/subsyt@latest

yt-dlp

Install uv on your system.

curl -LsSf https://astral.sh/uv/install.sh | sh

Then install yt-dlp:

uv tool install yt-dlp

Plugins (e.g. yt-dlp-ejs, curl-cffi) are configured via the plugins field in config.json and installed automatically on each sync cycle.

running

Configuration is read from, in order: the --config flag, the CONFIG environment variable, and otherwise $XDG_CONFIG_HOME/subsyt/config.json (~/.config/subsyt/config.json).

CONFIG="/path/to/config.json" ./subsyt

./subsyt --config="/path/to/config.json"

./subsyt    # reads ~/.config/subsyt/config.json

A ./config.json beside the binary is still picked up when one exists — that was the old default — but it is no longer where an unconfigured subsyt looks.

build

We want a statically linked binary so disable CGO.

CGO_ENABLED=0 go build

Or use the justfile to build both subsyt and subsytctl:

just build            # outputs to build/
just install          # installs subsytctl to ~/.local/bin ($XDG_BIN_HOME)

config

Full config.json:

{
    "daemon": true,
    "dry_run": true,
    "schedule": {
        "check_window": "22:00-01:00",
        "download_window": "01:00-07:00"
    },
    "log_file": "/data/log/subsyt.log",
    "db_file": "/data/db/subsyt.db",
    "download_dir": "./vids/_staging",
    "media_dir": "./vids",
    "http_api": {
        "enable": true,
        "listen": "0.0.0.0:6901",
        "queue_file": "./api-queue.json"
    },
    "provider": {
        "youtube": {
            "verbose": false,
            "cmd": "./yt-dlp",
            "plugins": ["yt-dlp-ejs", "curl-cffi", "bgutil-ytdlp-pot-provider"],
            "format": "best",
            "format_sort": "res:1080",
            "output_path_template": "s%(upload_date>%Y)s/%(channel)s.s%(upload_date>%Y)Se%(upload_date>%m%d)S.%(title)s.%(id)s.%(ext)s",
            "url": "https://www.youtube.com",
            "video_url_template": "",
            "match_filters": "!is_live & duration>?60",
            "throttle": 5,
            "limit_rate": "4M",
            "cookies_file": "",
            "subscription_file": "",
            "po_token": "",
            "bgutil_server": "http://127.0.0.1:4416",
            "player_client": ""
        }
    }
}

limit_rate caps each download's speed (yt-dlp's --limit-rate, e.g. 4M for 4 MB/s); empty means unlimited. A download that saturates the line on every video is one of the clearer bot signals. Downloads also wait a second between yt-dlp's requests and three seconds before subtitles.

Minimal config.json:

{
    "download_dir": "./vids/_staging",
    "media_dir": "./vids",
    "provider": {
        "youtube": {
            "cmd": "./yt-dlp",
            "url": "https://www.youtube.com",
            "throttle": 5
        }
    }
}

migration

Existing deployments that used to read media directly from download_dir should be migrated manually:

1. Stop the daemon or API workers so new downloads pause.
2. Back up your current media_dir and staging tree.
3. Move the contents of the legacy download_dir/shows and download_dir/episodes directories into the new media_dir, sorted by show and season as desired.
4. Optionally re-run subsyt with dry_run=true to confirm the new layout before enabling writes again.
5. Once satisfied, delete the obsolete per-show/per-episode staging directories under download_dir.

The application now keeps raw downloads inside download_dir and writes the organised library exclusively to media_dir.

subsytctl

subsytctl is the CLI for managing the server. It requires a running subsyt daemon with the HTTP API enabled.

export SUBSYT_HOST=subsyt.home.arpa:6901
export SUBSYT_KEY=<auth-token>

Commands:

subsytctl status                        show server status
subsytctl sync-discover                 trigger discovery of new uploads
subsytctl sync-dl                       trigger download of scheduled episodes
subsytctl list [id]                     list subscriptions, or episodes for a channel
subsytctl subscribe [--provider NAME] [--title TITLE] <channel>
                                        add a subscription (default provider: youtube);
                                        for youtube, <channel> is a UC channel ID, an
                                        @handle or a channel URL
subsytctl unsubscribe <id|channel>      remove a subscription by ID, or by the
                                        same channel references subscribe takes
subsytctl add <url> [out_dir]           enqueue a video for download
subsytctl scheduled                     list episodes scheduled for download
subsytctl download <video_id>           download a scheduled or failed episode now,
                                        outside the night window
subsytctl backfill [--depth N] [--dry-run] [channel]
                                        find uploads further back than the nightly
                                        check sees (default: newest 100 per channel)
                                        and schedule them; all channels when none given
subsytctl skip <video_id>               skip a scheduled episode
subsytctl scrub [--dry-run]             remove watched, non-favourite episodes
subsytctl scrub --older-than 90d [--watched] [--dry-run]
                                        remove non-favourite episodes that aired over 90d ago
subsytctl rename <id> <new-title>       rename a channel's media directory
subsytctl redownload <video_id>         re-download a video by ID
subsytctl token add [--scope all|delete] <name>
                                        create a named API token, shown once
subsytctl token list                    list named tokens
subsytctl token rm <name>               revoke a named token
subsytctl rotate-key                    rotate the API auth token

cookies

Warning

Your account MAY be banned when using cookies ! Consider using a throw-away account.

Install an extension that can download cookies per site, e.g. for firefox: https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/

The steps for the browser is:

1. install cookie export extension, allow in private mode
2. open a private browsing session (e.g. incognito)
3. go to youtube.com and login using a (throw-away) account
4. export the cookies using extension, save to disk
5. close private browsing session
6. point cookies_file in config.json to the cookies-file

Cookies may need to be refreshed if/when they expire, if so, repeat steps 2-5.

You can also yt-dlp to do it for you, though that exports all the cookies in the browser:

yt-dlp --cookies-from-browser {browser} --cookies cookies.txt

pot

Youtube has started requiring proof-of-origin tokens for some players, and it may help not getting hit with the "sign in to confirm you are not a bot" together with cookies.

Either add a manually generated POT to the config: "po_token": "{TOKEN}" or, set up bgutils8 with the youtube extractor to do POT generation automatically, in which case, leave the po_token as an empty string ("").

Add bgutil-ytdlp-pot-provider to the plugins list in your config.json (it is included in the full config example above).

On the same machine, run the bgutils http server, e.g. with compose:

 bgutil:
      image: brainicism/bgutil-ytdlp-pot-provider
      container_name: bgutil
      restart: unless-stopped
      ports:
        - 4416:4416

If using default ports and it's available on localhost, yt-dlp will pick up the plugin automatically and can be verified in the logs.

scheduling

When daemon is true, subsyt does its work at night, so new videos are ready in the morning and don't trickle in during the day. Each night it checks every channel for new uploads across schedule.check_window (default 22:00-01:00), then downloads what it found across schedule.download_window (default 01:00-07:00). Both are local time (set TZ); a window whose end comes before its start wraps past midnight.

Within a window, channels and downloads are spread evenly over the time left, in random order and with jittered gaps, rather than fired back to back. Downloads stay at least three minutes apart; whatever doesn't fit before the window closes, and any stream that hasn't started yet, waits for the next night.

To watch something sooner, subsytctl download <video_id> downloads one scheduled (or failed) episode straight away, ignoring the window and any cooldown; it waits for a download already in progress. subsytctl scheduled lists the IDs.

The nightly check only looks at each channel's 15 newest videos. To pull in older ones it never saw, such as a new subscription's back catalogue, subsytctl backfill --dry-run lists what's missing further back and subsytctl backfill schedules it for the nightly downloads. Episodes subsyt already knows, including skipped and deleted ones, are left alone.

subsytctl sync-discover and subsytctl sync-dl run a full pass immediately, at the short gaps of throttle rather than spread over a window.

systemd (non-daemon)

For one-shot mode (daemon is false), a systemd timer can drive the schedule:

~/.config/systemd/user/subsyt.service

[Unit]
Description=subsyt archival

[Service]
Type=oneshot
ExecStart=/home/varl/yt/subsyt --config=/home/varl/yt/config.json
WorkingDirectory=/home/varl/yt

~/.config/systemd/user/subsyt.timer

[Unit]
Description=subsyt archival on boot and daily

[Timer]
OnCalendar=*-*-* 4:00:00
Persistent=true
AccuracySec=1us
RandomizedDelaySec=30

[Install]
WantedBy=timers.target

container

podman run --rm \
    --volume=path/to/data:/data \
    registry.meatbag.se/varl/subsyt

compose

Runs in daemon mode (checks and downloads overnight), with automatic POT generation via bgutil.

services:
  subsyt:
      image: registry.meatbag.se/varl/subsyt:latest
      container_name: subsyt
      user: 1000:1000
      ports:
        - 6901:6901
      volumes:
        - /opt/subsyt/data:/data

  bgutil:
      image: brainicism/bgutil-ytdlp-pot-provider
      container_name: bgutil
      restart: unless-stopped
      ports:
        - 4416:4416

http api

Enable the built-in HTTP API to manage subscriptions and queue ad-hoc downloads. An admin token is generated on first run and printed to the log once; it isn't printed again. subsytctl rotate-key replaces it. If it's lost, set SUBSYT_ROTATE_TOKEN=1 in the environment and restart: subsyt generates a new one and prints it (unset it again afterwards, or it rotates on every start).

Other clients get their own named tokens, so each can be revoked without touching the rest. Managing them takes the admin token. A token's scope is all (the API, but not tokens or the admin key) or delete (only DELETE /v1/episodes, which is all compost needs):

subsytctl token add --scope delete compost

{
    "http_api": {
        "enable": true,
        "listen": "0.0.0.0:6901",
        "queue_file": "./api-queue.json"
    }
}

Submit new downloads with bearer authentication:

curl \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data '{"url":"https://youtu.be/VIDEO","out_dir":"Channel"}' \
  http://127.0.0.1:6901/v1/videos

Delete one episode from the library (its files) and mark it deleted, so it is never downloaded again; add &dry_run=true to see what would go:

curl -X DELETE -H "Authorization: Bearer <token>" \
  "http://127.0.0.1:6901/v1/episodes?video_id=VIDEO"

Check server status (sync timing, subscription/episode counts, queue):

curl -H "Authorization: Bearer <token>" http://127.0.0.1:6901/status

cooldowns lists providers whose traffic is paused because they pushed back (rate limiting, bot checks, or three unexplained failures in a row). Discovery and downloads skip a provider until its cooldown ends; each cooldown before a successful run is twice as long as the last, from 2h up to 48h.

Example response:

{
  "discover_sync": {
    "running": false,
    "next": "2025-03-12T04:15:30Z",
    "last_started": "2025-03-11T14:32:15Z",
    "last_ended": "2025-03-11T14:48:32Z",
    "last_duration_seconds": 977
  },
  "download_sync": {
    "running": false,
    "next": "2025-03-12T07:22:10Z",
    "last_started": "2025-03-11T15:01:00Z",
    "last_ended": "2025-03-11T15:12:45Z",
    "last_duration_seconds": 705,
    "last_downloaded": 3,
    "last_failed": 0
  },
  "cooldowns": { "youtube": "2025-03-11T17:12:45Z" },
  "subscriptions": 12,
  "episodes": 487,
  "scheduled": 5,
  "queue": { "count": 0, "items": [] }
}

result

.
├── Technology Connextras
│   ├── archive.txt
│   ├── fanart.jpg
│   ├── poster.jpg
│   ├── s2024
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.nfo
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.webm
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.nfo
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.webm
│   │   ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.nfo
│   │   └── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.webm
│   ├── s2025
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.nfo
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.webm
│   │   ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.nfo
│   │   └── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.webm
│   └── tvshow.nfo