Commit graph

428 commits

Author SHA1 Message Date
Jesse Bannon
fc2da4d525
[FEATURE] square_thumbnail plugin (#1244)
Adds the ability to make thumbnails square, both file and embedded. Usage:

```
my_preset:
  square_thumbnail: True
```

Will soon enable this by default for all music-based presets with a toggle to disable.
Huge thanks to @Kentaro1043 for crafting the ffmpeg command for this!

Closes https://github.com/jmbannon/ytdl-sub/issues/383
2025-06-18 22:37:06 -07:00
Jesse Bannon
f6bce88ebd
[FEATURE] height and width variables (#1243)
Adds `height` and `width` entry variables, representing pixels to know resolution. For audio, defaults to 0.
2025-06-18 20:56:25 -07:00
Jesse Bannon
a68d5e4539
[FEATURE] Emby prebuilt presets (#1232)
Gives Emby first class support by creating prebuilt presets for each variant, including
- `Emby TV Show by Date`
- `Emby TV Show Collection`
- `Emby Music Videos`

Emby is identical to Jellyfin except in TV Show Collection, where it requires a `season.nfo`. A new plugin `StaticNfoTags` has been created to create those.

Huge thanks to @Kamaroth92 for the initial code for this feature!
2025-06-01 09:56:48 -07:00
Jesse Bannon
0bd97dbe96
[FEATURE] release_date support for date_range (#1231)
Adds the ability to toggle whether to use `upload_date` or `release_date` in the date_range plugin, which is used to cull older videos.

```
date_range:
  type: "upload_date"
```

If using the Only Recent preset, it will be automatically applied based on whether your season/episode ordering uses release date or upload date.

Closes https://github.com/jmbannon/ytdl-sub/issues/1182
2025-06-01 09:28:42 -07:00
Jesse Bannon
66fd7cf41c
[FEATURE] Add scripting print functions (#1230)
Add the ability to print custom messages in override scripts. Adds the functions
- `print`
- `print_if_true`
- `print_if_false`

Will be used to make under-the-hood preset things more verbose.
2025-05-31 12:15:57 -07:00
Jesse Bannon
36e23839f3
[FEATURE] Ability to change season and episode ordering via variables, support for release_date (#1220)
Adds the ability to change season and episode ordering using override variables, like so:

```
       "~Kids Toys Play":
          url: "https://www.youtube.com/@KidsToysPlayChannel"
          tv_show_by_date_season_ordering: "upload-year-month"
          tv_show_by_date_episode_ordering: "upload-day"
```

In addition, adds support for release-date based episode ordering (and culling if using Only Recent).
More info here: https://ytdl-sub.readthedocs.io/en/latest/prebuilt_presets/tv_shows.html

Closes https://github.com/jmbannon/ytdl-sub/issues/1182
2025-05-31 10:26:42 -07:00
Jesse Bannon
bc15fe09bd
[FEATURE] Throttle protection per request (#1229)
Allows adding a sleep inbetween every request during metadata scrape.

```
throttle_protection:
  sleep_per_request_s:
    min: 4
    max: 6
```

Closes https://github.com/jmbannon/ytdl-sub/issues/1222
2025-05-31 09:41:14 -07:00
Jesse Bannon
987b1cd028
[FEATURE] Ability to customize date used for deleting old files (#1221)
Implements https://github.com/jmbannon/ytdl-sub/issues/1182

Adds the ability to change the `keep_files_date_eval` (traditionally upload_date_standardized) to any date, so long as its in the form of YYYY-MM-DD.

Will be able to use this to have episodes get deleted based on their release_date or epoch_date, instead of their upload_date.
2025-05-31 08:52:15 -07:00
Jesse Bannon
a4293398b7
[FEATURE] Add join and strip string functions to scripting (#1218) 2025-05-04 18:19:32 -07:00
Jesse Bannon
dfcbadb69d
[BACKEND] Make kodi_safe nfo overrideable (#1216)
For an upcoming change to make preset selection easier
2025-05-03 21:46:31 -07:00
Jesse Bannon
d5e8157b6c
[FEATURE] Add ability to suppress colors via CLI arg (#1215)
Not all terminals/logs displayed colors correctly. Adds the ability to suppress any color via `--suppress-colors` cli arg.
Thanks @drewski3420 for the contribution 🎉
2025-05-03 19:56:46 -07:00
Jesse Bannon
19d28e9baa
[FEATURE] Throw more actionable error if users supply the wrong type of url for TV show presets (#1210)
Don't let users supply `url` for TV Show Collection presets, or `s01_url` for TV Show by Date presets.
2025-04-17 22:14:37 -07:00
Tyler Dakin
bb2bc49a49
Support duplicate chapter timestamps by merging (#1179)
Co-authored-by: Tyler Dakin <tyler@dakin.one>
2025-02-21 08:34:46 -08:00
marvin8
279ab4a566
Make total delete videos in Total Stats be negative (#1175) 2025-02-16 19:49:05 -08:00
Jesse Bannon
53d84d8c1e
[FEATURE] Toggleable keyword evaluation behavior (#1130)
With the `Filter Keywords` prebuilt preset, can now set keyword evaluation to be either `ANY` or `ALL`. More info here: https://ytdl-sub.readthedocs.io/en/latest/prebuilt_presets/helpers.html#filter-keywords
2024-11-26 08:20:42 -08:00
Jesse Bannon
8c3f8528ad
[BACKEND] More media quality presets + docs (#1127)
Adds common media quality prebuilt presets and documentation on how to use them
2024-11-24 15:28:36 -08:00
Jesse Bannon
d3295cef86
[BUGFIX] Do not perform nested scripting on file path outputs (#1123)
Fixes a bug where curly braces in file-names causes scripting to think its a variable that does not exist
2024-11-18 00:22:20 -08:00
Jesse Bannon
663fb76132
[FEATURE] Multi-url support for YouTube music presets (#1113)
Adds support to specify multiple URLs for the YouTube-based music presets. More info here: https://ytdl-sub.readthedocs.io/en/latest/prebuilt_presets/music.html
2024-10-30 17:46:42 -07:00
Jesse Bannon
6a756cfecf
[BUGFIX] Move unescape parsing to separate function (#1108)
Fixes #1107 
```
FunctionRuntimeException: Runtime error occurred when executing the function %from_json: Invalid control character
```
2024-10-27 15:26:25 -07:00
Jesse Bannon
80054aa77b
[FEATURE] Support newlines and tabs in scripting strings (#1105)
Allows the usage of `\n` and `\t` in scripting strings
2024-10-27 08:09:05 -07:00
Jesse Bannon
1bdc65f2e1
[BUGFIX] Custom function ordering (#1104)
Fixes a bug where custom functions would throw an error if they were used out-of-order from their definition
2024-10-26 21:16:17 -07:00
Jesse Bannon
88110a816d
[BUGFIX] Fix playlists not downloading correctly
Due to url ytdl-options not materializing
2024-10-25 20:47:18 -07:00
Jesse Bannon
1499d7486c
[FEATURE] Override variable support in ytdl_options (#1087)
Adds the ability to use override variables in the [ytdl_options](https://ytdl-sub.readthedocs.io/en/latest/config_reference/plugins.html#ytdl-options) section of the config.
2024-10-05 21:25:50 -07:00
Jesse Bannon
28c29683cb
[FEATURE] Filter Keywords prebuilt preset (#1086)
Adds ``Filter Keywords``, a preset that can include or exclude media with any of the listed keywords. Both keywords and title/description are lower-cased before filtering.

Supports the following override variables:

* ``title_include_keywords``
* ``title_exclude_keywords``
* ``description_include_keywords``
* ``description_exclude_keywords``

For best usage, use the `~` tilda subscription mode to set a subscription's list override variables.
Tilda mode allows override variables to be set directly underneath it.

```
Plex TV Show by Date | Filter Keywords:
  
  = Documentaries:
    "~NOVA PBS":
       url: "https://www.youtube.com/@novapbs"
       title_exclude_keywords:
         - "preview"
         - "trailer"  

    "~To Catch a Smuggler":
       url: "https://www.youtube.com/@NatGeo"
       title_include_keywords:
         - "To Catch a Smuggler"
```
2024-10-05 01:03:52 -07:00
Jesse Bannon
c0ba30b220
[DEV] More OutputOption unit tests (#1082) 2024-10-02 21:37:54 -07:00
Jesse Bannon
97054043c4
[DEV] Refactor and simplify prebuilt preset tests (#1080) 2024-10-01 01:02:51 -07:00
Jesse Bannon
01e1f46b86
[DEV] Remaining throttle unit tests (#1075) 2024-09-30 22:04:32 -07:00
Jesse Bannon
f2ec26d77e
[BUGFIX] Fix variable collision when disabling plugins (#1072)
Fixes https://github.com/jmbannon/ytdl-sub/issues/1071
2024-09-29 22:38:26 -07:00
Jesse Bannon
1b57d79c8f
[BREAKING CHANGE] Remove regex plugin (#1067)
Regex plugin has been removed in favor of scripting. The function [regex_capture_many](https://ytdl-sub.readthedocs.io/en/latest/config_reference/scripting/scripting_functions.html#regex-capture-many) has been created to replicate the plugin's behavior. See the following converted example:

`regex, now deprecated`
```
    regex:
      from:
        title:
          match:
            - ".*? - (.*)"  # Captures 'Song' from 'Emily Hopkins - Some - Song'
          capture_group_names:
            - "captured_track_title"
          capture_group_defaults:
            - "{title}"
    overrides:
      track_title: "{captured_track_title}"
```

`scripting equivalent`
```
    overrides:
      # Captures 'Song' from 'Emily Hopkins - Some - Song'
      captured_track_title: >-
        {
          %regex_capture_many(
            title,
            [ ".*? - (.*)" ],
            [ title ]
          )
        }
      track_title: "{%array_at(captured_track_title, 1)}"
```

## Motivation:
Regex was a unique plugin that added custom variables based on user input. This made the backend need to support both `overrides` and 'plugin user variables'. Removing `regex` plugin will consolidate this logic into only `overrides`, making it much easier to maintain.
2024-09-29 08:41:09 -07:00
Jesse Bannon
fd5882051a
[DEV] Unit test filter plugin (#1066) 2024-09-26 11:54:15 -07:00
Jesse Bannon
79ba60021c
[BACKEND] Proper functions with docstrings (#1061)
Reimplement some of the 'hidden' functions, including `%regex_capture_many`, as standard functions with docstrings
2024-09-25 23:24:49 -07:00
Jesse Bannon
a37d9ed10e
[DEV] Set ffmpeg/ffprobe paths via ENV variable (#1060) 2024-09-24 22:13:22 -07:00
Jesse Bannon
6510fd0999
[BACKEND] Add /usr/local/bin/ffmpeg as possible default location (#1056)
For better cross-platform support
2024-09-20 14:22:33 -07:00
Jesse Bannon
10cb82a69a
[BACKEND] %contains_any function, list + dict support for tilda overrides mode (#1054)
Adds the function `%contains_any`, and list support in a tilda override subscription. The end-goal of these features are to more easily add a title exclude list, like so:

```
__preset__:
  filter_exclude:
    - "{%contains_any( %lower(title), exclude_title_strings )}"

...
Jellyfin TV Show by Date:
  ~History Documentaries:
    url: "https://..."
    exclude_title_strings:
      - "trailer"
      - "preview"
```

A proper prebuilt preset or built-in functionality will follow this change.
2024-09-20 09:23:22 -07:00
Parker Moore
759948361f
[BACKEND] get_file_md5_hash: read small blocks instead of the entire file (#1052)
This prevents MemoryError when running on systems with less memory than the file being processed.

Fixes #1051. Thanks @parkr
2024-09-17 20:24:33 -07:00
0xBLERKO
941fa43bb9
[BUGFIX] Find ext when info.json wrong (#1045)
Fixes a bug where YouTube's info.json sometimes say ext: mkv but download is not mkv. Thanks @0xBLERKO for the contribution!
2024-09-08 13:19:05 -07:00
Jesse Bannon
0ed8aec83d
[BACKEND] Show ffmpeg conversion errors (#1016) 2024-07-04 10:03:34 -07:00
Jesse Bannon
5e9472986a
[FEATURE] Enhance music video presets using specialized map syntax (#808)
Enhances the music video presets by supporting the following syntax:

`subscriptions.yaml`
```
__preset__:
  overrides:
    music_video_directory: "/music_videos"

# Choose between Jellyfin/Kodi/Plex Music Videos preset:
#   - Plex Music Videos:
#   - Jellyfin Music Videos:
#   - Kodi Music Videos:
#
"Plex Music Videos":

  = Pop:  # Sets genre tag to "Pop"
    "Rick Astley": "https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc"
    "Michael Jackson": "https://www.youtube.com/playlist?list=OLAK5uy_mnY03zP6abNWH929q2XhGzWD_2uKJ_n8E"

  = Rock:
    # Prefixing with '+' puts the subscription into 'map-mode'.
    # Music video presets in map-mode support grouping videos into different
    # categories, which get set on the album field.
    #
    # URLs can either be strings, or maps that can overload title, year, date
    "+ Guns N' Roses":
      Music Videos:
        - "https://www.youtube.com/playlist?list=PLOTK54q5K4INNXaHKtmXYr6J7CajWjqeJ"
      Concerts:
        - title: "Live at The Ritz - New York City"
          year: "1988"
          url: "https://www.youtube.com/watch?v=OldpIhHPsbs"
        - title: "Live at The Hollywood Bowl"
          date: "2023-01-11"
          url: "https://www.youtube.com/watch?v=Z7hutGlvq9I"
```
The 'map-mode' (denoted by `+`) lets you specify a map of music video categories. Under each category, a URL can be specified as-is or with additional metadata (title, year/date, url).

Music videos and concerts, especially older ones, are typically uploaded by random users with inconsistencies in their titles. This syntax aims to make it easy to specify individual URLs with the ability to overwrite their title and year, and group them by category.
2024-06-06 11:50:33 -07:00
Jesse Bannon
5866c12104
[BUGFIX] Do not run all branches of if functions in scripts (#999)
Will only run branch script code of `if` statements if the condition evaluates to that branch
2024-06-03 15:02:59 -07:00
Jesse Bannon
5e335b195c
[FEATURE] Allow YAML maps and lists in overrides, convert to script format (#956)
Adds the ability to create map and list-based override variables. 

For example, you can now create lists like this:
```
overrides:
  urls:
    - "https://...1"
    - "https://...2"
```
which is equivalent to:
```
overrides:
  urls: >-
    {
      [
        "https://...1",
        "https://...2",
      ]
    }
```

Likewise, maps can now look like:
```
overrides:
  music_video_category:
    concerts:
      - "https://...1"
      - "https://...2"
    interviews:
      - "https://...3"
```
which is equivalent to:
```
overrides:
  music_video_category: >-
    {
      "concerts": [
        "https://...1",
        "https://...2"
      ],
      "interviews": [
        "https://...3"
      ]
    }
```
2024-06-02 20:02:48 -07:00
Jesse Bannon
6e31ad3600
[BACKEND] Include .info.json files for Music video presets (#996)
Music video presets now include .info.json files by default
2024-06-02 10:57:32 -07:00
Jesse Bannon
30a2ad7a63
[BUGFIX] Prevent corrupt writes to download archive (#983)
Attempts to make writes to the download archive safer (https://github.com/jmbannon/ytdl-sub/issues/982).

`ytdl-sub` will now *copy* the download archive from the working directory to the output directory with a temp name, then perform a *move* to store it with its final expected name. This will drastically lower the window of time where  the process could die mid-write and corrupt it on the next read.
2024-06-02 10:53:19 -07:00
Jesse Bannon
74625c293d
[DEV] Update fixtures (#995)
* [DEV] Update fixtures

* thumbs fixed, longer timeout

* ugh

* fanart
2024-06-02 00:05:44 -07:00
Jesse Bannon
325d229061
[BUGFIX] Fix usage of ~ in paths (#981)
Fixes path with tildes in them, i.e. `~/videos/youtube`
2024-05-10 19:57:48 -07:00
dependabot[bot]
64d3082a8a
[DEV] Bump pylint from 2.13.5 to 3.1.0 (#972)
* Bump pylint from 2.13.5 to 3.1.0

Bumps [pylint](https://github.com/pylint-dev/pylint) from 2.13.5 to 3.1.0.
- [Release notes](https://github.com/pylint-dev/pylint/releases)
- [Commits](https://github.com/pylint-dev/pylint/compare/v2.13.5...v3.1.0)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* bend knee to pylint

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Bannon <jbann1994@gmail.com>
2024-04-28 10:40:15 -07:00
Tomas Babej
1d176050a7
[FEATURE] %regex_sub built-in script function (#971)
This implements %regex_sub built-in function to enhance string-processing capabilities, allowing users to perform substitutes like:

- removing non-ascii characters
- replacing subsequent whitespace characters with a single whitespace

Thanks @tbabej !
2024-04-27 22:47:29 -07:00
dependabot[bot]
ec58a80660
[DEV] Bump black from 22.3.0 to 24.4.2 (#973)
* Bump black from 22.3.0 to 24.4.2

Bumps [black](https://github.com/psf/black) from 22.3.0 to 24.4.2.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.3.0...24.4.2)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* run linter

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Bannon <jbann1994@gmail.com>
2024-04-27 15:46:47 -07:00
Jesse Bannon
ad72e1a6de
[BACKEND] Debug log exception on retry (#957)
Help diagnose retry errors better
2024-04-02 17:27:21 -07:00
Jesse Bannon
c622cf68b5
[DEV] Escapable curly braces (#955) 2024-04-01 22:40:56 -07:00
Jesse Bannon
598c574b8a
[DOCS] Show ytdl_options per url (#954) 2024-04-01 04:41:12 -07:00