There was a small crack in the fast evaluation logic where a phase 2
candidate's IDAT could be compressed below the best size from phase 1,
but the estimated output size (which includes chunks like PLTE) was
above. This candidate could end up getting returned from the evaluator
if no other filter actually did compress better, and would then be used
for the final compression even though the phase 1 candidate was actually
better.
The issue could potentially affect small, indexed images in `--fast`
mode (o2 and below).
Fixes#753.
This is a collection of all the BC breaks mentioned in #714, except for
11 which I'm not including for now.
Fixes#658.
Fixes#660.
It might be best to review each commit individually, referencing the
notes in #714 and #660 (I just didn't want to create a dozen separate
PRs).
This is an experiment I started a while ago before life happened. It
reduces memory usage of fast evaluation (-o2 and lower), bringing it
inline with normal (slow) evaluation. It does this by not retaining the
filtered image data of the evaluations, but instead retaining the row
filters that were used in each line so it can be quickly re-filtered
when required. This does incur a tiny performance penalty, but it's
negligible even at `-o0`.
Although this is fully functional, there are a few rough spots in the
code so I'm just opening it as a draft for now.
```
PR -sao6
70.08 real 614.36 user 2.66 sys
2024243200 maximum resident set size
1839339520 peak memory footprint
PR -sao2
11.16 real 60.85 user 1.33 sys
1982562304 maximum resident set size
1842173824 peak memory footprint
PR -sao2 -t1
55.11 real 53.85 user 0.76 sys
429457408 maximum resident set size
245008064 peak memory footprint
master -sao6
67.70 real 616.07 user 2.72 sys
2043379712 maximum resident set size
1838340416 peak memory footprint
master -sao2
11.53 real 60.63 user 1.25 sys
2753396736 maximum resident set size
2283741440 peak memory footprint
master -sao2 -t1
54.29 real 53.55 user 0.72 sys
626311168 maximum resident set size
305252544 peak memory footprint
```
Note that this involves some refactoring of `RowFilter` and the new
`FilterStrategy`. These are breaking changes so it will ultimately be
destined for v10. One advantage to this new structure is it opens the
door for future changes such as allowing the Brute strategy to take a
parameter for the number of lines.
#719 is failing tests due to requiring a newer version of rust than we
currently specify. This PR updates to 1.85.1 and sets the edition to
2024.
I've also updated dependencies and runner images, using the ubuntu arm
runner which removes the need for qemu and other hacks.
Closes#719.
This adds a new `final_round` parameter to the evaluator, which allows
us to determine when to keep the `filtered` data and when to keep the
compressed `idat_data`. By only keeping one of these we can
significantly improve memory usage.
Ref #683
This changes the evaluator to prefer a later image instead of an earlier
one in the case of a tie. This gives more consistent results since the
baseline is always added last.
I also removed bit depth from the key since the uncompressed size should
cover that.
Fixes#649.
(Mostly. I did discover another possible but very rare issue: if
multiple colours tie for "most popular edge colour" in the luma sort,
the sort may affect which one is actually picked and multiple runs may
flip-flop back and forth)
* Keep track of number of pixels in each scanline
* Expand depth to 8-bit
* Attempt expand bit depth
* Simplify tracking of reduction_occurred
* Fix test
* Simplify depth handling in reductions
* Add tests for disabled reductions
* Refactor filters as enum
* Include filter byte in filter output
* Add entropy filter
* Add bigrams filter
* Add bigram entropy filter
* Add brute filter
* Replace bit-vec
* Add tests and benches
* Show filters in help
* Use FxHasher in color to palette
* Use windows function for minor improvement
* Switch main compressor to libdeflate
* Use libdeflater in evaluate
* Use libdeflater to inflate
* Use libdeflater crc
* Tidy up
* Fix benches
* Allow libdeflater/freestanding feature
* Fix building without zopfli
* Switch to crossbeam-channel + rayon::spawn
* Remove thread_spawn for evaluation altogether
This allows to avoid a deadlock when there is only one Rayon thread, and doesn't sacrifice performance, since the caller of .get_result() had to always block on the iterator to be finished anyway, and all the messages are already sent from separate threads.
* Fix `verbose_mode` test
This one is easier to "fix", since we're in control of the rayon pool - just adding one extra thread to the default number to make sure we always can one root "spawn" call without stealing from the actual pool.
In order to avoid problems with mutable references,
the approach is to use the eval sender for non-parallel.
Computations will still be done synchronously
with the fake synchronous rayon module.
* Skip baselines earlier in the evaluator
We know that we'll throw them away anyway, so there is no point in even sending them to Sender and comparing them with others once we used them for the best candidate size.
* Simplify comparison logic
* Avoid using time API when we don't need it
This avoids a syscall to the time API when the result is ignored later anyway.
This allows to use the library with default options on wasm32-unknown-unknown, where the unimplemented syscall would panic otherwise.
* eval_send doesn't need to be an Option
We can drop the value manually, thus avoiding unwrap on each access.
* Keep single `use rayon::prelude::*`
If either `rayon` is already imported, then `rayon::prelude::*` should always resolve.
* Extract comparator
* Fully enable non-parallel mode
* Fix verbose message
* No cloning when restoring original data
* Make reductions return a new uncompressed image
Partially fixes#145
* Async reduction evaluator
* Assert
* Faster bit depth check
* Also try 4-bit depth for small-depth images
* Skip test when using miniz
* Ensure palette is trimmed after depth reduction
Fixes#159
* Fudge factor for reductions to prefer better reductions even if gzip estimation says otherwise
