There was a small crack in the fast evaluation logic where a phase 2
candidate's IDAT could be compressed below the best size from phase 1,
but the estimated output size (which includes chunks like PLTE) was
above. This candidate could end up getting returned from the evaluator
if no other filter actually did compress better, and would then be used
for the final compression even though the phase 1 candidate was actually
better.
The issue could potentially affect small, indexed images in `--fast`
mode (o2 and below).
Fixes#753.
Let me know if I should add these to the `Cargo.toml`.
All `cargo test` tests pass on the latest x86_64-unknown-linux-gnu
nightly.
---------
Co-authored-by: Alejandro González <me@alegon.dev>
- Fix docs for entropy filter (fixes#733)
- Improve docs for `--fix` (fixes#728)
- Add tests for alpha optimisation
- Following a recent change to the code here, I realised we don't
actually have any tests covering alpha optimisation. I've adjusted some
of the test images to include lines with both leading transparent and
all transparent pixels, and added dedicated tests for them.
- Change x86_64 macOS build to use the macos-15 runner
- There is a new macos-15-intel runner available but we don't actually
need an intel runner to build x86_64. (As support for x86_64 winds down,
we can probably drop these builds eventually - homebrew is the preferred
installation method for macOS anyway)
This PR allows the Brute filter parameters to be configured rather than
using fixed constants. It also tweaks the presets to take advantage of
these new options:
o3: level=1, lines=3 (fractionally worse/faster)
o4: level=1, lines=4 (same as before)
o5: level=4, lines=4 (fractionally better/slower)
o6: level=5, lines=8 (slightly better/slower)
The parameters can be explicitly set in both the API and the CLI.
Adds an option to skip files when the uncompressed IDAT (or the file
size itself) exceeds a certain size. This provides an effective way to
limit resource usage in constrained environments.
Closes#411.
This is a collection of all the BC breaks mentioned in #714, except for
11 which I'm not including for now.
Fixes#658.
Fixes#660.
It might be best to review each commit individually, referencing the
notes in #714 and #660 (I just didn't want to create a dozen separate
PRs).
This is an experiment I started a while ago before life happened. It
reduces memory usage of fast evaluation (-o2 and lower), bringing it
inline with normal (slow) evaluation. It does this by not retaining the
filtered image data of the evaluations, but instead retaining the row
filters that were used in each line so it can be quickly re-filtered
when required. This does incur a tiny performance penalty, but it's
negligible even at `-o0`.
Although this is fully functional, there are a few rough spots in the
code so I'm just opening it as a draft for now.
```
PR -sao6
70.08 real 614.36 user 2.66 sys
2024243200 maximum resident set size
1839339520 peak memory footprint
PR -sao2
11.16 real 60.85 user 1.33 sys
1982562304 maximum resident set size
1842173824 peak memory footprint
PR -sao2 -t1
55.11 real 53.85 user 0.76 sys
429457408 maximum resident set size
245008064 peak memory footprint
master -sao6
67.70 real 616.07 user 2.72 sys
2043379712 maximum resident set size
1838340416 peak memory footprint
master -sao2
11.53 real 60.63 user 1.25 sys
2753396736 maximum resident set size
2283741440 peak memory footprint
master -sao2 -t1
54.29 real 53.55 user 0.72 sys
626311168 maximum resident set size
305252544 peak memory footprint
```
Note that this involves some refactoring of `RowFilter` and the new
`FilterStrategy`. These are breaking changes so it will ultimately be
destined for v10. One advantage to this new structure is it opens the
door for future changes such as allowing the Brute strategy to take a
parameter for the number of lines.
#719 is failing tests due to requiring a newer version of rust than we
currently specify. This PR updates to 1.85.1 and sets the edition to
2024.
I've also updated dependencies and runner images, using the ubuntu arm
runner which removes the need for qemu and other hacks.
Closes#719.
This adds a new `final_round` parameter to the evaluator, which allows
us to determine when to keep the `filtered` data and when to keep the
compressed `idat_data`. By only keeping one of these we can
significantly improve memory usage.
Ref #683
Two key changes here:
- Removed some defunct reduction benches and added some more alpha
reduction benches.
- Changed use of `chunks()` to `chunks_exact()` where appropriate to
improve performance. (As a byproduct, this also fixes a potential crash
if a malformed file has a PLTE chunk that isn't a multiple of 3.)
If no palette reduction occurred (or palette reductions were off), a
fully transparent palette entry may not have been zeroed to black. By
ensuring this gets cleaned in the `indexed_to_channels` transformation,
we may able to achieve a grayscale conversion that would otherwise have
been RGB.
This is the final piece of the puzzle in #649 to achieve better file
sizes on the first run and avoid further changes on a second run.
Results from the images in #649 (combining this PR and #673):
9.1.3 1st run: 2,191,773
9.1.3 2nd run: 2,191,629
PR 1st run: 2,191,380
PR 2nd run: 2,191,380
If multiple edge colours were tied for most popular, the one chosen
would depend on the current palette order, which could cause it to
change if run a second time.
This further improves determinism for #649. There is still one more
issue I've found affecting the images there which I'll try to address in
another PR.
The evaluator uses the `estimated_output_size` internally but we were
passing only the idat size in the fast filter evaluator. At level `o2`,
or higher levels when using `--fast`, this could cause a slight bias
towards the None and Bigrams filters for indexed images. The actual
impact to results is incredibly minor but it's definitely incorrect so
we should fix it.
This changes the evaluator to prefer a later image instead of an earlier
one in the case of a tie. This gives more consistent results since the
baseline is always added last.
I also removed bit depth from the key since the uncompressed size should
cover that.
Fixes#649.
(Mostly. I did discover another possible but very rare issue: if
multiple colours tie for "most popular edge colour" in the luma sort,
the sort may affect which one is actually picked and multiple runs may
flip-flop back and forth)
I don't know if this is a bug in the mangen but it seems to require
explicitly setting the value name to display correctly.
Before:
```
--zi [default: 15]
Set the number of iterations to use for Zopfli compression.
```
After:
```
--zi=iterations [default: 15]
Set the number of iterations to use for Zopfli compression.
```
This does not affect MANUAL.txt or the built-in help.
#648 may have been a bit hasty - I realised afterward that there's a
simpler way to achieve the same thing, and include the Brute filter as
well.
This reverts #648 and instead just picks None up front if the line is
all zeros. This is guaranteed to be the chosen filter for MinSum,
Entropy, Bigrams and BigEnt. It's almost certainly true for Brute as
well but this is harder to prove. I've tested this across hundreds of
images and found no change in output.
