varl/subsyt
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
YouTube OPML subscription archiver
46 commits 1 branch 0 tags 334 KiB
Go 96.3%
Dockerfile 3.5%
Nix 0.2%
Find a file
Viktor Varland efc49fa4a2
All checks were successful
build / build (push) Successful in 1m34s
Details
docs: update readme with compose
2025-04-14 21:38:55 +02:00
.forgejo/workflows ci: master => main 2025-03-28 15:44:26 +01:00
internal feat: support configuring the bgutil server 2025-04-14 13:06:07 +02:00
.gitignore feat: add more config options 2025-03-28 12:44:38 +01:00
Containerfile ci: run as user 1000:1000 2025-04-14 21:35:58 +02:00
go.mod feat: youtube channel archiver 2025-03-27 15:54:31 +01:00
go.sum feat: youtube channel archiver 2025-03-27 15:54:31 +01:00
main.go feat: use daemon flag to toggle scheduler/standalone 2025-04-14 12:50:26 +02:00
README.md docs: update readme with compose 2025-04-14 21:38:55 +02:00
shell.nix feat: support opml from youtube 2025-04-08 12:32:12 +02:00

README.md

subsyt

description

subsyt is a wrapper around yt-dlp1 to download youtube channels based on a OPML file containing all your subscriptions, sorting the channels into {show}/{season} folders, and generates nfo files, extracts thumbnails, downloads posters, banners, and fanart so the media should plug into media libraries well-enough, e.g. Jellyfin and Kodi.

A quick rundown on how to use it:

  • download subsyt or build it into a binary yourself2
  • install yt-dlp3
  • patch it with POT support (POT optional -- yet recommended) 4
  • generate and download a OPML file5
  • setup a config file6
  • run subsyt7

install

go install git.meatbag.se/varl/subsyt@latest

yt-dlp

Install pipx on your system.

sudo apt install pipx           # debian
sudo pacman -Syu python-pipx    # archlinux

pipx install yt-dlp

running

Configuration can be loaded from a file specified either by the env variable CONFIG or --config flag.

The --config flag has priority over CONFIG environment variable.

CONFIG="/path/to/config.toml" ./subsyt

./subsyt --config="/patch/to/config"

./subsyt    # assumes "./config.toml"

build

We want a statically linked binary so disable CGO.

CGO_ENABLED=0 go build

config

Full config.toml:

dry_run = true                          # set to `false` for real run
out_dir = "./vids"                      # path to archive vids
daemon = true                           # true to run scheduler, false to run once

[provider]
[provider.youtube]
cmd = "./yt-dlp"                        # path to yt-dlp binary
quality = "res:1080"                    # set the preferred quality, blank for 1080
output_path_template = "s%(upload_date>%Y)s/%(channel)s.s%(upload_date>%Y)Se%(upload_date>%m%d)S.%(title)s.%(id)s-1080p.%(ext)s"    # yt-dlp output template
url = "https://www.youtube.com"         # full yt url
throttle = 5                            # throttle yt request, 5s works well
range = "1:5:1"                         # downloads last 5 videos: [START][:STOP][:STEP]
after_date = "20250326"                 # only download videos after date
cookies_file = ""                       # pass user cookies to yt, blank to disable
opml_file = "./opml.xml"                # the opml file to use
po_token = ""                           # manually pass a proof-of-origin token, blank to disable
verbose = true                          # debug info for provider

generate opml

Use this javascript snippet: https://github.com/jeb5/YouTube-Subscriptions-RSS to generate a file that has the format:

<?xml version="1.0"?>
<opml version="1.1">
    <body>
        <outline ...>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
        </outline>
        <outline ...>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
            <outline text="" title="" xmlUrl="" .../>
        </outline>
    </body>
</opml>

cookies

Warning

Your account MAY be banned when using cookies ! Consider using a throw-away account.

Install an extension that can download cookies per site, e.g. for firefox: https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/

The steps for the browser is:

  1. install cookie export extension, allow in private mode
  2. open a private browsing session (e.g. incognito)
  3. go to youtube.com and login using a (throw-away) account
  4. export the cookies using extension, save to disk
  5. close private browsing session
  6. point cookies_file in config.toml to the cookies-file

Cookies may need to be refreshed if/when they expire, if so, repeat steps 2-5.

You can also yt-dlp to do it for you, though that exports all the cookies in the browser:

yt-dlp --cookies-from-browser {browser} --cookies cookies.txt

pot

Youtube has started requiring proof-of-origin tokens for some players, and it may help not getting hit with the "sign in to confirm you are not a bot" together with cookies.

Either add a manually generated POT to the config: po_token = "{TOKEN}" or, set up bgutils8 with the youtube extractor to do POT generation automatically, in which case, leave the po_token as an empty string ("").

# assumes pipx was used to install yt-dlp
pipx inject yt-dlp yt-dlp-get-pot
pipx inject yt-dlp bgutil-ytdlp-pot-provider

Then change the provider.youtube option for cmd to the yt-dlp binary in the modified venv, e.g. /home/varl/.local/bin/yt-dlp.

On the same machine, run the bgutils http server, e.g. with compose:

 bgutil:
      image: brainicism/bgutil-ytdlp-pot-provider
      container_name: bgutil
      restart: unless-stopped
      ports:
        - 4416:4416

If using default ports and it's available on localhost, yt-dlp will pick up the plugin automatically and can be verified in the logs.

scheduling

systemd

Tip

Remember to change the ExecStart path to the venv'ed yt-dlp binary if using it.

~/.config/systemd/user/subsyt-archival.service

[Unit]
Description=subsyt archival of yt subscribtions

[Service]
Type=oneshot
ExecStart=/home/varl/yt/yt-dlp -U
ExecStart=/home/varl/yt/subsyt
WorkingDirectory=/home/varl/yt

~/.config/systemd/user/subsyt-archival.timer

[Unit]
Description=subsyt archival on boot and daily

[Timer]
OnCalendar=*-*-* 4:00:00
Persistent=true
AccuracySec=1us
RandomizedDelaySec=30

[Install]
WantedBy=timers.target

container

podman run --rm \
    --volume=path/to/opml:/data/opml.xml \
    --volume=path/to/vids:/data/vids \
    registry.meatbag.se/varl/subsyt

compose

Runs in scheduled mode (0400 hours daily), with automatic POT generation and the bgutil-ytdlp-pot-provider server.

services:
  subsyt:
      image: registry.meatbag.se/varl/subsyt:latest
      container_name: subsyt
      user: 1000:1000
      volumes:
        - /opt/subsyt/youtube_subs.opml:/data/opml.xml
        - /media/videos/youtube:/data/vids

  bgutil:
      image: brainicism/bgutil-ytdlp-pot-provider
      container_name: bgutil
      restart: unless-stopped
      ports:
        - 4416:4416

result

.
├── Technology Connextras
│   ├── archive.txt
│   ├── fanart.jpg
│   ├── poster.jpg
│   ├── s2024
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.nfo
│   │   ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.webm
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.nfo
│   │   ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.webm
│   │   ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.nfo
│   │   └── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.webm
│   ├── s2025
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.nfo
│   │   ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.webm
│   │   ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p-thumb.jpg
│   │   ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.nfo
│   │   └── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.webm
│   └── tvshow.nfo