Commit graph

74 commits

Author SHA1 Message Date
Jesse Bannon
017db953bf
[FEATURE] Automatically handle playlists ordered in reverse (#948)
Playlists have always been a pain-point with ytdl-sub. If an author adds new videos to the end of a playlist, as opposed to the front, it breaks ytdl-sub's intuition of incremental scraping by breaking on the first (oldest) video. This update now makes it possible to handle this in the prebuilt TV Show presets:
- Add each URL variable (`url`, `url2`, ...) into the `download` portion of the subscription twice
- First definition is what we all know and use, simply scrapes first-to-last, then downloads last-to-first
- Second definition does the following:
  - Check to see if a URL is a YouTube playlist URL, if so...
    - Set the field to download, but with modifications to scrape last-to-first, then download first-to-last
  - Otherwise...
    - Set the field to an empty string, which means ytdl-sub will skip it
2024-04-01 04:24:41 -07:00
Jesse Bannon
d5e647554e
[DEV] Fix resolve_once script bug with custom functions (#952)
* [DEV] Fix resolve_once script bug with custom functions

* function name

* fix adding custom functions
2024-03-30 00:53:05 -07:00
Jesse Bannon
e3158583ca
[FEATURE] Toggleable plugin field enable for all dict-based plugins (#897)
For key/value-based plugins, add an `enable` field to make it easier for child presets to disable their functionality
2024-01-11 00:40:07 -08:00
Jesse Bannon
2076de3074
[BUGFIX] Smarter subscription validation (#895)
With the right functions, it was possible for subscription validation to raise a false-positive error. This should hopefully resolve that issue and give validation a small performance increase
2024-01-10 18:16:12 -08:00
Jesse Bannon
9382be5c42
[FEATURE] --match flag to only run subset of subscriptions (#881)
Implements https://github.com/jmbannon/ytdl-sub/issues/880
and maybe fixes https://github.com/jmbannon/ytdl-sub/issues/827

Can now do `ytdl-sub sub --match SubA SubB` or `ytdl-sub sub --match SubA --match SubB`, which will only run subscriptions that contain `SubA` or `SubB` in their names
2024-01-08 13:37:07 -08:00
Jesse Bannon
4d0123c8b0
[BACKEND] Optimize memory usage (#872)
Reduce memory usage significantly when pulling large channels all at once by
- Not loading entry script until it is being processed
- Deleting the entry memory once fully processed
- Disable yt-dlp from keeping entry info in memory since ytdl-sub reads it from file
2024-01-05 13:04:01 -08:00
Jesse Bannon
f2d55fdb23
[REFACTOR] Organize variable definitions (#860) 2023-12-29 14:39:19 -08:00
Jesse Bannon
731d4e444c
[DOCS] Tooling to auto-generate custom sphinx docs (#849)
Tooling + updating docs
2023-12-28 22:39:59 -08:00
Jesse Bannon
1fcaba9274
[DOCS] Update links to reflect doc-upgrade (#858) 2023-12-27 11:07:11 -08:00
Jesse Bannon
873ecc0147
[BUGFIX] Fix recursion limit when applying large reduce functions (#851)
Python would think a recursive error occurred, but in reality, the stack got too large due to poor optimization when executing array reduce functions. Thanks Melissa from Discord for the bug report!
2023-12-20 23:46:17 -08:00
Jesse Bannon
c0e2ded1ae
[BACKEND] Improved error messages when script resolution does not work (#846) 2023-12-19 00:22:14 -08:00
Jesse Bannon
e92b1cd12a
[BACKEND][HUGE] Function Support in variable syntax (#838)
A complete gutting of the internals of ytdl-sub to support functions in our variable syntax, in addition to being able to access a yt-dlp entry's .info.json fields using functions. Functionally, ytdl-sub should still look and behave the same from a user-perspective.

With so many lines of code changed (+8927, -2708), no doubt there will be new issues. Please make a GH issue or reach out on Discord if your config/subscriptions break in any way/shape/form.

Details on how to use function support will come soon in the form of proper documentation in our readthedocs.
2023-12-18 16:08:15 -08:00
Jesse Bannon
056b111be6
[FEATURE] Prebuilt Music Videos (#787)
Creates standardized prebuilt presets for music videos. Preset names are:

```
"Kodi Music Videos"
"Jellyfin Music Videos"
"Plex Music Videos"
```

Usage can be found in the `examples/` directory
2023-10-28 00:12:22 -07:00
Jesse Bannon
83ecd19c77
[FEATURE] Process all subscriptions even if one or more error (#771)
Closes GH Issue #766, partially closes #520

Prior to this release, no other subscriptions would download if a subscription before it had an error. Now, subscriptions will continue to download even if one has an error, and will be reported in the output summary.
2023-10-21 12:25:38 -07:00
Jesse Bannon
3d9c35519d
[BACKEND] More explicit error message for PermissionError (#736)
Do not print the full stack trace when a PermissionError occurs
2023-09-21 23:06:57 -07:00
Jesse Bannon
3c87dc9f8e
[BUGFIX] Fix date_range plugin as a match-filter (#726)
Bugfix to the prior release. Convert yt-dlp date formats (i.e. `today-2months`) into a date string with format `YYYYMMDD`. This will make it compatible with match-filter syntax
2023-09-20 08:30:29 -07:00
Jesse Bannon
0008bf3aca
[FEATURE] More verbose error logs when using ytdl-sub commands (#717)
Add a more informative error message when the config file is missing, and when no command is supplied (i.e. sub, dl, view)
2023-09-15 12:35:34 -07:00
Jesse Bannon
85c98f2eb8
[BACKEND] Include yaml exception in error (#678)
* [BACKEND] Include yaml exception in error

* unit test

* escape
2023-07-29 23:05:34 -07:00
Jesse Bannon
5f9b8afc4b
[BUGFIX] Do not hard-fail if the thumbnail fails to download/convert (#675)
* [BUGFIX] Do not fail with thumbnail issues

* helper

* refactor

* dry-run

* regen fixture

* download arhcive txt -> json in docs, fixtures

* return if no download thumb path

* better test

* more test coverage
2023-07-28 13:12:46 -07:00
Jesse Bannon
a8841bf150
[BUGFIX] Ignore extra data streams with -dn in ffmpeg (#637) 2023-06-24 15:42:29 -07:00
Jesse Bannon
0f39b38dc3
[DOCS] Actionable message for file lock errors (#634)
* [DOCS] Actionable message for file lock errors"

* pylint
2023-06-17 13:42:36 -07:00
Jesse Bannon
d5d9978cdb
[BUGFIX] Fix splitting chapters extracted from comments (#593) 2023-04-19 19:27:27 -07:00
Jesse Bannon
e1242eaa61
[BUGFIX] Fix parsing chapters from comments (#592) 2023-04-19 18:44:25 -07:00
Jesse Bannon
1588725bb1
[BUGFIX] Fix Windows split by chapters not reading chapter titles correctly (#591) 2023-04-19 18:32:13 -07:00
Jesse Bannon
7385aac1ee
[FEATURE][EXPERIMENTAL] --update-with-info-json (#557)
* [FEATURE
] Reformat existing downloads

* working?!?!

* reformat in output dir

* close, need info json enabled by default

* asdfadsfsaf

* lint

* closer

* multi created variables do not exist for update command!

* info json downloader

* [REFACTOR] Shared plugin and download options class

* more refactoring

* lint

* put in initialize plugins

* changes

* [REFACTOR] simplify enhanced download archive

* working!

* in main

* simplified, many url tests working

* ready?
2023-03-21 08:05:08 -07:00
Jesse Bannon
baebd9cba7
[BUGFIX] Add timeout for bad ffmpeg image conversions (#554)
* [BUGFIX] Add timeout for bad ffmpeg image conversions

* add -y

* lint
2023-03-16 23:45:11 -07:00
Jesse Bannon
f4807aa9a4
[BUGFIX] Do not load YAML unless dict is returned (#541) 2023-03-14 23:06:51 -07:00
Jesse Bannon
e69933d525
[REFACTOR] Split metadata and download into two separate steps (#540) 2023-03-13 22:41:09 -07:00
Jesse Bannon
f9318b9d07
[BUGFIX] Handle truncating file names if too long (#524)
* [BUGFIX] Handle truncating file names if too long

* always import os

* test fix

* fix for windows

* lint
2023-03-10 07:20:43 -08:00
Jesse Bannon
ce877a32f9
[BUGFIX] Fix quiet log level not showing warning logs (#516) 2023-03-08 16:30:42 -08:00
Jesse Bannon
91e9fe7602
[FEATURE] Add summary to console output (#515) 2023-03-08 15:34:58 -08:00
Jesse Bannon
648027204f
[FEATURE] Persisted subscription logs (#512) 2023-03-08 00:17:11 -08:00
Jesse Bannon
896a9e2073
[BACKEND] Reset debug log on every subscription (#497) 2023-03-02 18:57:13 -08:00
Jesse Bannon
798bbbc62c
[BACKEND] Windows support (#476) 2023-02-28 17:07:03 -08:00
Joshua M. Boniface
69df95e0e9
[BACKEND] Always perform copy/delete on OSError (#446)
* Always perform copy/delete on OSError

Extends the fix in #400 to handle all potential OSError errno values and
attempt a copy/delete in all cases. For example, a combination
cross-device move and a permissions change would throw errno 1 instead
of errno 18, which wasn't caught here. Instead of messing with
individual errno numbers, just always attempt a copy/delete and let
those throw their own exceptions if they don't work either.

* Remove unused os_error_exc variable
2023-02-19 21:06:48 -08:00
Jesse Bannon
c84496910f
[BUGFIX] Cross-device patch for ffmpeg file writes (#407) 2023-01-10 14:19:45 -08:00
Jesse Bannon
c3d9d45c18
[BUGFIX] Invalid cross-device link workaround (#400) 2023-01-04 15:34:26 -08:00
Jesse Bannon
c8c66b5660
[BUGFIX] Create dir before converting channel thumbnails (#381) 2022-11-30 15:06:56 -08:00
Jesse Bannon
96e741c71d
[BACKEND] Proactively clean working directory (#379)
* fix playlist last thumbnail move to copy

* playlist working

* channel working

* working directory getting cleaned

* working dir e2e fixture

* cleanup files in any case

* clean up if pre-split returns None
2022-11-30 10:36:13 -08:00
Jesse Bannon
9f5ee153b7
[BACKEND] Add URL thumbnail download timeout (#367) 2022-11-26 23:44:40 -08:00
Jesse Bannon
ffaf07d628
[FEATURE] Add support to scrape chapters from comments (#347)
* [FEATURE] Add support to scrape chapters from comments

* tested for comments

* remove timestamps file

* change order of docstrings
2022-11-21 11:19:17 -08:00
Jesse Bannon
70c78dc109
[FEATURE] Embed thumbnails into audio files as cover art (#345)
* [FEATURE] Embed thumbnails into audio files as cover art

* lint

* fix ytdl options
2022-11-19 23:36:15 -08:00
Jesse Bannon
ed08b2f591
asdf (#336) 2022-11-16 14:04:40 -08:00
Jesse Bannon
5b9a5dae50
[FEATURE] Prebuilt presets (#264) 2022-10-01 22:41:36 -07:00
Jesse Bannon
897ea52e33
[BACKEND] Separate directory files in transaction log (#260) 2022-09-30 13:59:43 -07:00
Jesse Bannon
839192059b
[FEATURE] Add playlist thumbnail support (#250) 2022-09-27 23:48:18 -07:00
Jesse Bannon
c08ea64b5a
[BACKEND] Add generic.collection download strategy, use it for soundcloud, youtube channel/playlist (#230) 2022-09-14 15:36:14 -07:00
Jesse Bannon
2c86d640a4
[FEATURE] Have notion of modified files. Do not include files that have not changed (#225) 2022-09-08 12:24:46 -07:00
Jesse Bannon
819bddc9d0
[FEATURE] Support duplicate NFO keys (#220) 2022-09-07 14:06:13 -07:00
Jesse Bannon
6383d133dc
[BUGFIX] Retry channel image downloads, continue if it fails (#218) 2022-09-06 16:12:28 -07:00