varl/subsyt
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
YouTube OPML subscription archiver
73 commits 1 branch 0 tags 334 KiB
Go 96.3%
Dockerfile 3.5%
Nix 0.2%
Find a file
Viktor Varland 09134c46c4
Some checks are pending
build / build (push) Waiting to run
Details
refactor: split logic into distinct parts
2025-10-01 21:21:59 +02:00
.forgejo/workflows ci: master => main 2025-03-28 15:44:26 +01:00
internal refactor: split logic into distinct parts 2025-10-01 21:21:59 +02:00
.gitignore feat: add more config options 2025-03-28 12:44:38 +01:00
AGENTS.md chore: change host port 2025-09-30 12:24:51 +02:00
Containerfile refactor: split logic into distinct parts 2025-10-01 21:21:59 +02:00
go.mod refactor: use json for config to reach zero deps 2025-04-15 08:27:02 +02:00
go.sum refactor: use json for config to reach zero deps 2025-04-15 08:27:02 +02:00
main.go refactor: split logic into distinct parts 2025-10-01 21:21:59 +02:00
README.md refactor: split logic into distinct parts 2025-10-01 21:21:59 +02:00
shell.nix feat: support opml from youtube 2025-04-08 12:32:12 +02:00

README.md

subsyt

description

subsyt is a wrapper around yt-dlp1 to download youtube channels based on a OPML file containing all your subscriptions. Downloads land in an isolated staging directory before being organised into {show}/{season} folders under your configured media library. During the organising step the tool generates nfo files, extracts thumbnails, downloads posters, banners, and fanart so the media should plug into media libraries well-enough, e.g. Jellyfin and Kodi.

A quick rundown on how to use it:

  • download subsyt or build it into a binary yourself2
  • install yt-dlp3
  • patch it with POT support (POT optional -- yet recommended) 4
  • generate and download a OPML file5
  • setup a config file6
  • run subsyt7

install

go install git.meatbag.se/varl/subsyt@latest

yt-dlp

Install pipx on your system.

sudo apt install pipx           # debian
sudo pacman -Syu python-pipx    # archlinux

pipx install yt-dlp

running

Configuration can be loaded from a file specified either by the env variable CONFIG or --config flag.

The --config flag has priority over CONFIG environment variable.

CONFIG="/path/to/config.json" ./subsyt

./subsyt --config="/patch/to/config"

./subsyt    # assumes "./config.json"

build

We want a statically linked binary so disable CGO.

CGO_ENABLED=0 go build

config

Full config.json:

{
    "daemon": true,
    "dry_run": true,
    "download_dir": "./vids/_staging",
    "media_dir": "./vids",
    "provider": {
        "youtube": {
            "verbose": false,
            "cmd": "./yt-dlp",
            "format": "best",
            "format_sort": "res:1080",
            "output_path_template": "s%(upload_date>%Y)s/%(channel)s.s%(upload_date>%Y)Se%(upload_date>%m%d)S.%(title)s.%(id)s.%(ext)s",
            "url": "https://www.youtube.com",
            "throttle": 5,
            "cookies_file": "",
            "opml_file": "./youtube_subs.opml",
            "po_token": "",
            "schedule": "",
            "bgutil_server": "http://127.0.0.1:4416"
        }
    }
}

Minimal config.json:

{
    "download_dir": "./vids/_staging",
    "media_dir": "./vids",
    "provider": {
        "youtube": {
            "cmd": "./yt-dlp",
            "throttle": 5,
            "opml_file": "./youtube_subs.opml"
        }
    }
}

migration

Existing deployments that used to read media directly from download_dir should be migrated manually:

  1. Stop the daemon or API workers so new downloads pause.
  2. Back up your current media_dir and staging tree.
  3. Move the contents of the legacy download_dir/shows and download_dir/episodes directories into the new media_dir, sorted by show and season as desired.
  4. Optionally re-run subsyt with dry_run=true to confirm the new layout before enabling writes again.
  5. Once satisfied, delete the obsolete per-show/per-episode staging directories under download_dir.

The application now keeps raw downloads inside download_dir and writes the organised library exclusively to media_dir.

generate opml

Use this javascript snippet: https://github.com/jeb5/YouTube-Subscriptions-RSS to generate a file that has the format:

<?xml version="1.0"?>
<opml version="1.1">
    <body>
        <outline ...>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
        </outline>
        <outline ...>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
        </outline>
    </body>
</opml>

cookies

Warning

Your account MAY be banned when using cookies ! Consider using a throw-away account.

Install an extension that can download cookies per site, e.g. for firefox: https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/

The steps for the browser is:

  1. install cookie export extension, allow in private mode
  2. open a private browsing session (e.g. incognito)
  3. go to youtube.com and login using a (throw-away) account
  4. export the cookies using extension, save to disk
  5. close private browsing session
  6. point cookies_file in config.json to the cookies-file

Cookies may need to be refreshed if/when they expire, if so, repeat steps 2-5.

You can also yt-dlp to do it for you, though that exports all the cookies in the browser:

yt-dlp --cookies-from-browser {browser} --cookies cookies.txt

pot

Youtube has started requiring proof-of-origin tokens for some players, and it may help not getting hit with the "sign in to confirm you are not a bot" together with cookies.

Either add a manually generated POT to the config: po_token = "{TOKEN}" or, set up bgutils8 with the youtube extractor to do POT generation automatically, in which case, leave the po_token as an empty string ("").

# assumes pipx was used to install yt-dlp
pipx inject yt-dlp bgutil-ytdlp-pot-provider

Then change the provider.youtube option for cmd to the yt-dlp binary in the modified venv, e.g. /home/varl/.local/bin/yt-dlp.

On the same machine, run the bgutils http server, e.g. with compose:

 bgutil:
      image: brainicism/bgutil-ytdlp-pot-provider
      container_name: bgutil
      restart: unless-stopped
      ports:
        - 4416:4416

If using default ports and it's available on localhost, yt-dlp will pick up the plugin automatically and can be verified in the logs.

scheduling

systemd

Tip

Remember to change the ExecStart path to the venv'ed yt-dlp binary if using it.

~/.config/systemd/user/subsyt-archival.service

[Unit]
Description=subsyt archival of yt subscribtions

[Service]
Type=oneshot
ExecStart=/home/varl/yt/yt-dlp -U
ExecStart=/home/varl/yt/subsyt
WorkingDirectory=/home/varl/yt

~/.config/systemd/user/subsyt-archival.timer

[Unit]
Description=subsyt archival on boot and daily

[Timer]
OnCalendar=*-*-* 4:00:00
Persistent=true
AccuracySec=1us
RandomizedDelaySec=30

[Install]
WantedBy=timers.target

container

podman run --rm \
    --volume=path/to/opml:/data/opml.xml \
    --volume=path/to/vids:/data/vids \
    registry.meatbag.se/varl/subsyt

compose

Runs in scheduled mode (0400 hours daily), with automatic POT generation and the bgutil-ytdlp-pot-provider server.

services:
  subsyt:
      image: registry.meatbag.se/varl/subsyt:latest
      container_name: subsyt
      user: 1000:1000
      volumes:
        - /opt/subsyt/youtube_subs.opml:/data/opml.xml
        - /media/videos/youtube:/data/vids

  bgutil:
      image: brainicism/bgutil-ytdlp-pot-provider
      container_name: bgutil
      restart: unless-stopped
      ports:
        - 4416:4416

http api

Enable the built-in intake server to queue ad-hoc videos without editing the OPML export.

{
    "http_api": {
        "enable": true,
        "listen": "0.0.0.0:6901",
        "auth_token_file": "/run/secrets/subsyt-token",
        "queue_file": "./tmp/api-queue.json"
    }
}

Submit new downloads with bearer authentication:

curl \
  -H "Authorization: Bearer $(cat /run/secrets/subsyt-token)" \
  -H "Content-Type: application/json" \
  --data '{"url":"https://youtu.be/VIDEO","out_dir":"Channel"}' \
  http://127.0.0.1:6901/v1/videos

Requests reuse the configured yt-dlp binary, honor existing throttling, and persist through restarts when queue_file is provided.

Check the pending queue for debugging:

curl -H "Authorization: Bearer super-secret" http://127.0.0.1:6901/status

result

.
├── Technology Connextras
│   ├── archive.txt
│   ├── fanart.jpg
│   ├── poster.jpg
│   ├── s2024
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.nfo
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.webm
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.nfo
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.webm
│   │   ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.nfo
│   │   └── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.webm
│   ├── s2025
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.nfo
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.webm
│   │   ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.nfo
│   │   └── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.webm
│   └── tvshow.nfo