Let me know if I should add these to the `Cargo.toml`.
All `cargo test` tests pass on the latest x86_64-unknown-linux-gnu
nightly.
---------
Co-authored-by: Alejandro González <me@alegon.dev>
This is a collection of all the BC breaks mentioned in #714, except for
11 which I'm not including for now.
Fixes#658.
Fixes#660.
It might be best to review each commit individually, referencing the
notes in #714 and #660 (I just didn't want to create a dozen separate
PRs).
This is an experiment I started a while ago before life happened. It
reduces memory usage of fast evaluation (-o2 and lower), bringing it
inline with normal (slow) evaluation. It does this by not retaining the
filtered image data of the evaluations, but instead retaining the row
filters that were used in each line so it can be quickly re-filtered
when required. This does incur a tiny performance penalty, but it's
negligible even at `-o0`.
Although this is fully functional, there are a few rough spots in the
code so I'm just opening it as a draft for now.
```
PR -sao6
70.08 real 614.36 user 2.66 sys
2024243200 maximum resident set size
1839339520 peak memory footprint
PR -sao2
11.16 real 60.85 user 1.33 sys
1982562304 maximum resident set size
1842173824 peak memory footprint
PR -sao2 -t1
55.11 real 53.85 user 0.76 sys
429457408 maximum resident set size
245008064 peak memory footprint
master -sao6
67.70 real 616.07 user 2.72 sys
2043379712 maximum resident set size
1838340416 peak memory footprint
master -sao2
11.53 real 60.63 user 1.25 sys
2753396736 maximum resident set size
2283741440 peak memory footprint
master -sao2 -t1
54.29 real 53.55 user 0.72 sys
626311168 maximum resident set size
305252544 peak memory footprint
```
Note that this involves some refactoring of `RowFilter` and the new
`FilterStrategy`. These are breaking changes so it will ultimately be
destined for v10. One advantage to this new structure is it opens the
door for future changes such as allowing the Brute strategy to take a
parameter for the number of lines.
#719 is failing tests due to requiring a newer version of rust than we
currently specify. This PR updates to 1.85.1 and sets the edition to
2024.
I've also updated dependencies and runner images, using the ubuntu arm
runner which removes the need for qemu and other hacks.
Closes#719.
This adds a new `final_round` parameter to the evaluator, which allows
us to determine when to keep the `filtered` data and when to keep the
compressed `idat_data`. By only keeping one of these we can
significantly improve memory usage.
Ref #683
The evaluator uses the `estimated_output_size` internally but we were
passing only the idat size in the fast filter evaluator. At level `o2`,
or higher levels when using `--fast`, this could cause a slight bias
towards the None and Bigrams filters for indexed images. The actual
impact to results is incredibly minor but it's definitely incorrect so
we should fix it.
Allow the creation of PNGs with uncompressed deflate streams via level 0
of libdeflate. If you want a glorified BMP with delta filters, this
change will make your dreams come true 🙂.
This PR adds a build script to generate a man page using clap_mangen, as
per this example:
https://github.com/sondr3/clap-man-example/blob/main/build.rs
I'm not sure what to actually do with the man file from here, I guess
it's up to the packaging process to do something with it?
See
https://github.com/shssoichiro/oxipng/issues/69#issuecomment-1963352536
Note I couldn't see a way to include the `DISPLAY` chunk names from the
constant as we did before. They're now just hardcoded into the help and
will require manually updating if the list changes.
Closes#526
---------
Co-authored-by: Alejandro González <me@alegon.dev>
The filters used during reduction evaluations are normally set to None &
Bigrams, regardless of any options. This PR makes a slight change so
that if only one filter is specified in the options, this filter will be
used for reduction evaluations too.
This resolves an odd situation affecting lower levels (when --fast is
enabled) where you may try to force the filter to a specific value but
it actually ends up different because a reduction evaluation was
smaller. It's particularly helpful if you're wanting it to be as fast as
possible by using `-o0 -f0` which will now exclusively use None instead
of trying the slower Bigrams as well.
As another use, you could try to brute force oxipng by iterating each
filter separately, though this may not actually achieve anything 😂
[edit] I also pulled the options out into a separate file, though this
wasn't relevant to the filter change.
A detailed read to the [Rust documentation for the `fs::Permissions`
struct](https://doc.rust-lang.org/stable/std/fs/struct.Permissions.html)
and a little digging into its implementation in the standard library
have shown that this code didn't work as expected in any platform, and
was a bit weird to begin with:
- It first read the permissions from the input file metadata.
- Then it fetched the output file metadata.
- After that, it changed the permissions for that output file metadata.
- It then performed a sanity check that the output file had the expected
permissions.
Barring the fact that the sanity check in step 4 is not needed, the
overall approach is wrong because setting the permissions in a file
metadata struct does not actually persist those changes anywere; it's
just an in-memory change only, so these operations were useless. The
Rust documentation explicitly mentions that the `set_readonly` method
"does not modify the files attributes" [sic], but it's easy to miss that
warning and not realize that it also applies to the methods offered by
the `PermissionsExt` trait. The code only appeared to work because in
most cases the default permissions for new files happen to match the
input file permissions, so the sanity check passed.
To fix this, use the `set_permissions` method on `File` to actually set
the output file permissions to be the same as the input file
permissions, which is both much simpler and robust.
These changes were tested in the context of issue #576, and fix#576.
Tidy up the API by removing a couple of options we don't really need.
Backup was discussed in #542
Check was discussed in #439
@shssoichiro Just say if you prefer to keep either of these 🙂
This PR is addressing #220. It's not super important but it's a breaking
change, so if it's something we want to do then I thought I should get
it in now before the next release.
- [x] pretend can become another variant of OutFile, probably
OutFile::None, as that's what it essentially is - just another output
destination and not a separate option
- [x] ~~backup and~~ preserve_attrs should become properties of
OutFile::Path variant (so that it would contain Path { path, ~~backup,~~
preserve_attrs }) as they don't have any effect on any other output and
so semantically belong there best
Closes#220
This PR makes two changes:
- `--nz` (`idat_recoding`) now disables all zlib recompression,
including iCCP and fDAT chunks. (Perhaps we should rename the option to
`zlib_recompression`?)
- `--nx` now also disables the default deinterlacing, though it can
still be overridden with `-i`.
`--nx --nz` does disable all optimisations in the v8 release and we
should ensure it continues to do so in the next release. (This is
related to discussions around removing the `--check` option.)
This adds a new palette sorting algorithm that attempts to minimise
entropy by an approximate solution to the Traveling Salesman Problem.
The algorithm comes from "An efficient Re-indexing algorithm for
color-mapped images" by Battiato et al
(https://ieeexplore.ieee.org/document/1344033).
It's fast and effective and works in addition to the luma sort (which
remains the single most effective sort). In order to keep lower presets
fast though, I've only enabled this for o3 and higher.
Results on a set of 190 indexed images at `-o5`:
18,932,727 bytes - master
18,578,306 bytes - PR
18,559,863 bytes - PR + #509
(These images may be particularly suited to alternative sorting methods
- the gains here are not necessarily what should be expected on average)
Note I looked into the 120 different palette sorting methods from
TruePNG, as mentioned in #74 (and seen in action in the Zopfli KrzYmod
fork). They're... largely ineffective. The combination of all 120
methods are outperformed by just the existing luma sort plus this new
one. That's not to say there's nothing further to be gained from them,
but trying to brute force all the combinations definitely seems like a
bad idea. There are other algorithms I hope to explore in future...
@ace-dent Thought this might interest you
UPDATE: I realised a quick tweak to alpha values in the luma sort can
provide a great improvement on images with transparency. The following
numbers were taken with PR #509 as base.
`-o2`:
19,065,549 bytes - base (luma sort)
18,949,747 bytes - modified luma sort
`-o5`:
18,922,165 bytes - base (luma sort)
18,559,863 bytes - new sorting algorithm + luma sort
18,544,813 bytes - new sorting algorithm + modified luma sort
This PR makes the oxipng binary process multiple files in parallel,
finally fulfilling #275. There seemed to be some debate about whether
oxipng _should_ do this or not but there's a couple of reasons I think
it makes sense:
1. The concern seemed mostly around the complexity of such a feature.
Not to worry, it was trivial* 🙂
2. Since then, oxipng has dropped from a max of something like 180
simultaneous compression trials down to 10, which is very much a good
thing but it does mean it's not utilising any more cores than that.
Some benchmarks on around 100 files on a machine with 8 cores:
Level | Master time | PR time
-|-|-
2 | 28.303 | 19.005
3 | 36.507 | 23.089
5 | 1:10.86 | 1:16.01
*Some additional changes were required in order to make sure sensible
output is printed to the terminal, since things won't be in order
anymore. Here's some example output from before:
```
Processing: tests/files/fully_optimized.png
file size = 67 bytes (0 bytes = 0.00% decrease)
File already optimized
Processing: tests/files/corrupted_header.png
Invalid PNG header detected
Processing: tests/files/verbose_mode.png
file size = 102480 bytes (12228 bytes = 10.66% decrease)
Output: tests/files/verbose_mode.png
```
And after:
```
Processing: tests/files/verbose_mode.png
Processing: tests/files/fully_optimized.png
Processing: tests/files/corrupted_header.png
tests/files/corrupted_header.png: Invalid PNG header detected
tests/files/fully_optimized.png: Could not optimize further, no change written
102480 bytes (10.66% smaller): tests/files/verbose_mode.png
```
Closes#275, #84, #169, #196 and #419.
[edit] This is the last thing I wanted to land before the next release 🥳
* Add .whitesource configuration file
* Experimental: allow Zopfli to use any size BufWriter
* Allow user to specify the output buffer size as well
* Allow user to specify maximum block splits
* Reformat and fix warnings
* Use deflater on iCCP chunk as well
* Bug fix: need to implement Zlib format
* Make functions const when possible
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Cargo fmt
* Fix compilation
* Fix tests
* Fix more lints
* Fix more lints
* Fix compilation more
---------
Co-authored-by: mend-bolt-for-github[bot] <42819689+mend-bolt-for-github[bot]@users.noreply.github.com>
Co-authored-by: Chris Hennick <hennickc@amazon.com>
Co-authored-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* Keep track of number of pixels in each scanline
* Expand depth to 8-bit
* Attempt expand bit depth
* Simplify tracking of reduction_occurred
* Fix test
* Simplify depth handling in reductions
* Add tests for disabled reductions
* Fix bKGD conversion from gray to palette
* Allow grayscale reduction from 16 to 4 or less
* Refactor reduction evaluation sequence
* Separate palette into new file
* Refactor ColorType
Move transparency and palette data into the ColorType
* Fixup tests
* Make more use of helper functions
* Change BitDepth to u8 representation with TryFrom
* Fix clippy lints
* Don't use unstable language features
* Restore documentation on transparency/palette
* Make dependency on `image` optional
After PR https://github.com/shssoichiro/oxipng/pull/481 was merged, the
`image` dependency became unused when building with debug assertions
disabled, as it is only used to implement output sanity checks when such
assertions are enabled.
The `image` crate transitively pulls a significant amount of
dependencies, so it's useful for OxiPNG users to get rid of them when
not needed.
[Cargo does not allow specifying dependencies that are only pulled when
debug assertions are
enabled](https://github.com/rust-lang/cargo/issues/7634), so the next
best way to give users some flexibility is to gate those debug
assertions behind a feature flag.
These changes add a `sanity-checks` feature flag that controls whether
the `image` crate and the related sanity checks are compiled in. This
feature is enabled by default to keep debug builds useful to catch
problems during development.
* Fix Clippy lints
* Run tests with new sanity-checks feature enabled
