252 lines
8.1 KiB
Markdown
252 lines
8.1 KiB
Markdown
# subsyt
|
|
|
|
## description
|
|
|
|
`subsyt` is a wrapper around `yt-dlp`[1] to download youtube channels
|
|
based on a OPML file containing all your subscriptions, sorting the
|
|
channels into `{show}/{season}` folders, and generates `nfo` files,
|
|
extracts thumbnails, downloads posters, banners, and fanart so the media
|
|
should plug into media libraries well-enough, e.g. Jellyfin and Kodi.
|
|
|
|
A quick rundown on how to use it:
|
|
|
|
- download `subsyt` or build it into a binary yourself[2]
|
|
- install `yt-dlp`[3]
|
|
- patch it with POT support (POT optional -- yet recommended)[4]
|
|
- generate and download a OPML file[5]
|
|
- setup a config file[6]
|
|
- run `subsyt`[7]
|
|
|
|
[1]: https://github.com/yt-dlp/yt-dlp
|
|
[2]: [install](#install)
|
|
[3]: [ytdlp](#yt-dlp)
|
|
[4]: [pot](#pot)
|
|
[5]: [opml](#generate opml)
|
|
[6]: [config](#config)
|
|
[7]: [run](#running)
|
|
|
|
## install
|
|
|
|
```
|
|
go install git.meatbag.se/varl/subsyt@latest
|
|
```
|
|
|
|
## yt-dlp
|
|
|
|
Install `pipx` on your system.
|
|
|
|
```
|
|
sudo apt install pipx # debian
|
|
sudo pacman -Syu python-pipx # archlinux
|
|
|
|
pipx install yt-dlp
|
|
```
|
|
|
|
## running
|
|
|
|
Configuration can be loaded from a file specified either by the env
|
|
variable `CONFIG` or `--config` flag.
|
|
|
|
The `--config` flag has priority over `CONFIG` environment variable.
|
|
|
|
```
|
|
CONFIG="/path/to/config.toml" ./subsyt
|
|
|
|
./subsyt --config="/patch/to/config"
|
|
|
|
./subsyt # assumes "./config.toml"
|
|
```
|
|
|
|
## build
|
|
|
|
We want a statically linked binary so disable CGO.
|
|
|
|
```sh
|
|
CGO_ENABLED=0 go build
|
|
```
|
|
|
|
## config
|
|
|
|
`config.toml`:
|
|
|
|
```toml
|
|
dry_run = true # set to `false` for real run
|
|
out_dir = "./vids" # path to archive vids
|
|
|
|
[provider]
|
|
[provider.youtube]
|
|
cmd = "./yt-dlp" # path to yt-dlp binary
|
|
quality = "res:1080" # set the preferred quality, blank for 1080
|
|
output_path_template = "s%(upload_date>%Y)s/%(channel)s.s%(upload_date>%Y)Se%(upload_date>%m%d)S.%(title)s.%(id)s-1080p.%(ext)s" # yt-dlp output template
|
|
url = "https://www.youtube.com" # full yt url
|
|
throttle = 5 # throttle yt request, 5s works well
|
|
range = "1:5:1" # downloads last 5 videos: [START][:STOP][:STEP]
|
|
after_date = "20250326" # only download videos after date
|
|
cookies_file = "" # pass user cookies to yt, blank to disable
|
|
opml_file = "./opml.xml" # the opml file to use
|
|
po_token = "" # manually pass a proof-of-origin token, blank to disable
|
|
verbose = true # debug info for provider
|
|
```
|
|
|
|
## generate opml
|
|
|
|
Use this javascript snippet:
|
|
https://github.com/jeb5/YouTube-Subscriptions-RSS to generate a file
|
|
that has the format:
|
|
|
|
```xml
|
|
<?xml version="1.0"?>
|
|
<opml version="1.1">
|
|
<body>
|
|
<outline ...>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
</outline>
|
|
<outline ...>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
</outline>
|
|
</body>
|
|
</opml>
|
|
```
|
|
|
|
## cookies
|
|
|
|
> [!WARNING]
|
|
> Your account **MAY** be banned when using cookies ! Consider using a
|
|
> throw-away account.
|
|
|
|
Install an extension that can download cookies per site, e.g. for
|
|
firefox: https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/
|
|
|
|
The steps for the browser is:
|
|
|
|
1. install cookie export extension, allow in private mode
|
|
1. open a private browsing session (e.g. incognito)
|
|
1. go to youtube.com and login using a (throw-away) account
|
|
1. export the cookies using extension, save to disk
|
|
1. close private browsing session
|
|
1. point `cookies_file` in `config.toml` to the cookies-file
|
|
|
|
Cookies may need to be refreshed if/when they expire, if so, repeat
|
|
steps 2-5.
|
|
|
|
You can also `yt-dlp` to do it for you, though that exports all the
|
|
cookies in the browser:
|
|
|
|
```
|
|
yt-dlp --cookies-from-browser {browser} --cookies cookies.txt
|
|
```
|
|
|
|
## pot
|
|
|
|
Youtube has started requiring proof-of-origin tokens for some players,
|
|
and it may help not getting hit with the "sign in to confirm you are not
|
|
a bot" together with cookies.
|
|
|
|
Either add a manually generated POT to the config: `po_token =
|
|
"{TOKEN}"` or, set up bgutils[2] with the youtube extractor to do POT
|
|
generation automatically.
|
|
|
|
```
|
|
# assumes pipx was used to install yt-dlp
|
|
pipx inject yt-dlp yt-dlp-get-pot
|
|
pipx inject yt-dlp bgutil-ytdlp-pot-provider
|
|
```
|
|
|
|
Then change the `provider.youtube` option for `cmd` to the `yt-dlp`
|
|
binary in the modified venv, e.g. `/home/varl/.local/bin/yt-dlp`.
|
|
|
|
On the same machine, run the bgutils http server, e.g. with compose:
|
|
|
|
```
|
|
bgutil:
|
|
image: brainicism/bgutil-ytdlp-pot-provider
|
|
container_name: bgutil
|
|
restart: unless-stopped
|
|
ports:
|
|
- 4416:4416
|
|
```
|
|
|
|
If using default ports and it's available on localhost, `yt-dlp` will
|
|
pick up the plugin automatically and can be verified in the logs.
|
|
|
|
[2]: https://github.com/Brainicism/bgutil-ytdlp-pot-provider
|
|
|
|
## scheduling
|
|
|
|
### systemd
|
|
|
|
> [!TIP]
|
|
> Remember to change the `ExecStart` path to the venv'ed `yt-dlp`
|
|
> binary if using it.
|
|
|
|
`~/.config/systemd/user/subsyt-archival.service`
|
|
```
|
|
[Unit]
|
|
Description=subsyt archival of yt subscribtions
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStart=/home/varl/yt/yt-dlp -U
|
|
ExecStart=/home/varl/yt/subsyt
|
|
WorkingDirectory=/home/varl/yt
|
|
|
|
```
|
|
|
|
`~/.config/systemd/user/subsyt-archival.timer`
|
|
```
|
|
[Unit]
|
|
Description=subsyt archival on boot and daily
|
|
|
|
[Timer]
|
|
OnCalendar=*-*-* 4:00:00
|
|
Persistent=true
|
|
AccuracySec=1us
|
|
RandomizedDelaySec=30
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
```
|
|
|
|
## container
|
|
|
|
```
|
|
podman run --rm \
|
|
--env='CONFIG=/data/config.toml' \
|
|
--volume=path/to/config:/data/config.toml \
|
|
--volume=path/to/opml:/data/opml.xml \
|
|
--volume=path/to/cookies:/data/cookies.txt
|
|
registry.meatbag.se/varl/subsyt
|
|
```
|
|
|
|
## result
|
|
|
|
```
|
|
.
|
|
├── Technology Connextras
|
|
│ ├── archive.txt
|
|
│ ├── fanart.jpg
|
|
│ ├── poster.jpg
|
|
│ ├── s2024
|
|
│ │ ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.nfo
|
|
│ │ ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.webm
|
|
│ │ ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.nfo
|
|
│ │ ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.webm
|
|
│ │ ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.nfo
|
|
│ │ └── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.webm
|
|
│ ├── s2025
|
|
│ │ ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.nfo
|
|
│ │ ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.webm
|
|
│ │ ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.nfo
|
|
│ │ └── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.webm
|
|
│ └── tvshow.nfo
|
|
```
|