Viktor Varland
6ff2d53c2d
Some checks are pending
build / build (push) Waiting to run
This uses RSS to fetch a list of videos to avoid the vid being invisible due to "restrictions", then downloads the videos one-by-one instead of scraping and parsing the channel page using yt-dlp. We lose metadata for the entire channel (show-level) so introducing a hack to download just the metadata of a channel.
292 lines
8.4 KiB
Markdown
292 lines
8.4 KiB
Markdown
# subsyt
|
|
|
|
## description
|
|
|
|
`subsyt` is a wrapper around `yt-dlp`[1] to download youtube channels
|
|
based on a OPML file containing all your subscriptions, sorting the
|
|
channels into `{show}/{season}` folders, and generates `nfo` files,
|
|
extracts thumbnails, downloads posters, banners, and fanart so the media
|
|
should plug into media libraries well-enough, e.g. Jellyfin and Kodi.
|
|
|
|
A quick rundown on how to use it:
|
|
|
|
- download `subsyt` or build it into a binary yourself[2]
|
|
- install `yt-dlp`[3]
|
|
- patch it with POT support (POT optional -- yet recommended) [4]
|
|
- generate and download a OPML file[5]
|
|
- setup a config file[6]
|
|
- run `subsyt`[7]
|
|
|
|
[1]: https://github.com/yt-dlp/yt-dlp
|
|
[2]: https://git.meatbag.se/varl/subsyt#install
|
|
[3]: https://git.meatbag.se/varl/subsyt#yt-dlp
|
|
[4]: https://git.meatbag.se/varl/subsyt#pot
|
|
[5]: https://git.meatbag.se/varl/subsyt#generate-opml
|
|
[6]: https://git.meatbag.se/varl/subsyt#config
|
|
[7]: https://git.meatbag.se/varl/subsyt#running
|
|
|
|
## install
|
|
|
|
```
|
|
go install git.meatbag.se/varl/subsyt@latest
|
|
```
|
|
|
|
## yt-dlp
|
|
|
|
Install `pipx` on your system.
|
|
|
|
```
|
|
sudo apt install pipx # debian
|
|
sudo pacman -Syu python-pipx # archlinux
|
|
|
|
pipx install yt-dlp
|
|
```
|
|
|
|
## running
|
|
|
|
Configuration can be loaded from a file specified either by the env
|
|
variable `CONFIG` or `--config` flag.
|
|
|
|
The `--config` flag has priority over `CONFIG` environment variable.
|
|
|
|
```
|
|
CONFIG="/path/to/config.json" ./subsyt
|
|
|
|
./subsyt --config="/patch/to/config"
|
|
|
|
./subsyt # assumes "./config.json"
|
|
```
|
|
|
|
## build
|
|
|
|
We want a statically linked binary so disable CGO.
|
|
|
|
```sh
|
|
CGO_ENABLED=0 go build
|
|
```
|
|
|
|
## config
|
|
|
|
Full `config.json`:
|
|
|
|
```json
|
|
{
|
|
"daemon": true,
|
|
"dry_run": true,
|
|
"out_dir": "./vids",
|
|
"provider": {
|
|
"youtube": {
|
|
"verbose": false,
|
|
"cmd": "./yt-dlp",
|
|
"quality": "res:1080",
|
|
"output_path_template": "s%(upload_date>%Y)s/%(channel)s.s%(upload_date>%Y)Se%(upload_date>%m%d)S.%(title)s.%(id)s.%(ext)s",
|
|
"url": "https://www.youtube.com",
|
|
"throttle": 5,
|
|
"cookies_file": "",
|
|
"opml_file": "./youtube_subs.opml",
|
|
"po_token": "",
|
|
"schedule": "",
|
|
"bgutil_server": "http://127.0.0.1:4416"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Minimal `config.json`:
|
|
|
|
```json
|
|
{
|
|
"out_dir": "./vids",
|
|
"provider": {
|
|
"youtube": {
|
|
"cmd": "./yt-dlp",
|
|
"throttle": 5,
|
|
"opml_file": "./youtube_subs.opml"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## generate opml
|
|
|
|
Use this javascript snippet:
|
|
https://github.com/jeb5/YouTube-Subscriptions-RSS to generate a file
|
|
that has the format:
|
|
|
|
```xml
|
|
<?xml version="1.0"?>
|
|
<opml version="1.1">
|
|
<body>
|
|
<outline ...>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
</outline>
|
|
<outline ...>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
<outline text="" title="" xmlUrl="" .../>
|
|
</outline>
|
|
</body>
|
|
</opml>
|
|
```
|
|
|
|
## cookies
|
|
|
|
> [!WARNING]
|
|
> Your account **MAY** be banned when using cookies ! Consider using a
|
|
> throw-away account.
|
|
|
|
Install an extension that can download cookies per site, e.g. for
|
|
firefox: https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/
|
|
|
|
The steps for the browser is:
|
|
|
|
1. install cookie export extension, allow in private mode
|
|
1. open a private browsing session (e.g. incognito)
|
|
1. go to youtube.com and login using a (throw-away) account
|
|
1. export the cookies using extension, save to disk
|
|
1. close private browsing session
|
|
1. point `cookies_file` in `config.json` to the cookies-file
|
|
|
|
Cookies may need to be refreshed if/when they expire, if so, repeat
|
|
steps 2-5.
|
|
|
|
You can also `yt-dlp` to do it for you, though that exports all the
|
|
cookies in the browser:
|
|
|
|
```
|
|
yt-dlp --cookies-from-browser {browser} --cookies cookies.txt
|
|
```
|
|
|
|
## pot
|
|
|
|
Youtube has started requiring proof-of-origin tokens for some players,
|
|
and it may help not getting hit with the "sign in to confirm you are not
|
|
a bot" together with cookies.
|
|
|
|
Either add a manually generated POT to the config: `po_token =
|
|
"{TOKEN}"` or, set up bgutils[8] with the youtube extractor to do POT
|
|
generation automatically, in which case, leave the `po_token` as an
|
|
empty string (`""`).
|
|
|
|
```
|
|
# assumes pipx was used to install yt-dlp
|
|
pipx inject yt-dlp bgutil-ytdlp-pot-provider
|
|
```
|
|
|
|
Then change the `provider.youtube` option for `cmd` to the `yt-dlp`
|
|
binary in the modified venv, e.g. `/home/varl/.local/bin/yt-dlp`.
|
|
|
|
On the same machine, run the bgutils http server, e.g. with compose:
|
|
|
|
```
|
|
bgutil:
|
|
image: brainicism/bgutil-ytdlp-pot-provider
|
|
container_name: bgutil
|
|
restart: unless-stopped
|
|
ports:
|
|
- 4416:4416
|
|
```
|
|
|
|
If using default ports and it's available on localhost, `yt-dlp` will
|
|
pick up the plugin automatically and can be verified in the logs.
|
|
|
|
[8]: https://github.com/Brainicism/bgutil-ytdlp-pot-provider
|
|
|
|
## scheduling
|
|
|
|
### systemd
|
|
|
|
> [!TIP]
|
|
> Remember to change the `ExecStart` path to the venv'ed `yt-dlp`
|
|
> binary if using it.
|
|
|
|
`~/.config/systemd/user/subsyt-archival.service`
|
|
```
|
|
[Unit]
|
|
Description=subsyt archival of yt subscribtions
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStart=/home/varl/yt/yt-dlp -U
|
|
ExecStart=/home/varl/yt/subsyt
|
|
WorkingDirectory=/home/varl/yt
|
|
|
|
```
|
|
|
|
`~/.config/systemd/user/subsyt-archival.timer`
|
|
```
|
|
[Unit]
|
|
Description=subsyt archival on boot and daily
|
|
|
|
[Timer]
|
|
OnCalendar=*-*-* 4:00:00
|
|
Persistent=true
|
|
AccuracySec=1us
|
|
RandomizedDelaySec=30
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
```
|
|
|
|
## container
|
|
|
|
```
|
|
podman run --rm \
|
|
--volume=path/to/opml:/data/opml.xml \
|
|
--volume=path/to/vids:/data/vids \
|
|
registry.meatbag.se/varl/subsyt
|
|
```
|
|
|
|
## compose
|
|
|
|
Runs in scheduled mode (0400 hours daily), with automatic POT generation
|
|
and the bgutil-ytdlp-pot-provider server.
|
|
|
|
```
|
|
services:
|
|
subsyt:
|
|
image: registry.meatbag.se/varl/subsyt:latest
|
|
container_name: subsyt
|
|
user: 1000:1000
|
|
volumes:
|
|
- /opt/subsyt/youtube_subs.opml:/data/opml.xml
|
|
- /media/videos/youtube:/data/vids
|
|
|
|
bgutil:
|
|
image: brainicism/bgutil-ytdlp-pot-provider
|
|
container_name: bgutil
|
|
restart: unless-stopped
|
|
ports:
|
|
- 4416:4416
|
|
```
|
|
|
|
## result
|
|
|
|
```
|
|
.
|
|
├── Technology Connextras
|
|
│ ├── archive.txt
|
|
│ ├── fanart.jpg
|
|
│ ├── poster.jpg
|
|
│ ├── s2024
|
|
│ │ ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.nfo
|
|
│ │ ├── Technology_Connextras.s2024e0611.Connextras_dishwasher_follow_up_the_sequel.0Kp3bjm55xw-1080p.webm
|
|
│ │ ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.nfo
|
|
│ │ ├── Technology_Connextras.s2024e0712.Here_s_what_Numitron_tubes_in_an_actual_product_look_like.XgzL05Gojfw-1080p.webm
|
|
│ │ ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.nfo
|
|
│ │ └── Technology_Connextras.s2024e0909.Answering_your_pinball_questions_-_Williams_Aztec_Q_A.P3Y4d2aHnNE-1080p.webm
|
|
│ ├── s2025
|
|
│ │ ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.nfo
|
|
│ │ ├── Technology_Connextras.s2025e0330.Renewable_energy_means_we_can_stop_setting_money_on_fire_silly_billy.Y2qSaD1v4cQ-1080p.webm
|
|
│ │ ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p-thumb.jpg
|
|
│ │ ├── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.nfo
|
|
│ │ └── Technology_Connextras.s2025e0331.An_unplanned_trip_from_Chicago_to_Milwaukee_in_an_electric_car.3GUQdrpduo0-1080p.webm
|
|
│ └── tvshow.nfo
|
|
```
|