The filters used during reduction evaluations are normally set to None &
Bigrams, regardless of any options. This PR makes a slight change so
that if only one filter is specified in the options, this filter will be
used for reduction evaluations too.
This resolves an odd situation affecting lower levels (when --fast is
enabled) where you may try to force the filter to a specific value but
it actually ends up different because a reduction evaluation was
smaller. It's particularly helpful if you're wanting it to be as fast as
possible by using `-o0 -f0` which will now exclusively use None instead
of trying the slower Bigrams as well.
As another use, you could try to brute force oxipng by iterating each
filter separately, though this may not actually achieve anything 😂
[edit] I also pulled the options out into a separate file, though this
wasn't relevant to the filter change.
This is a minor change that allows using both `--strip` and `--keep` at
the same time.
E.g. `--strip safe --keep eXIf` will strip chunks while preserving both
the ones that aren't "safe" to remove *and* eXIf. Essentially it's a
convenience to allow extending the default list used by `--strip safe`.
Specifying chunk names for both options is not permitted, e.g. `--strip
eXIf --keep eXIf` will error.
Use of `--strip all` with `--keep` is redundant, but is permitted.
A detailed read to the [Rust documentation for the `fs::Permissions`
struct](https://doc.rust-lang.org/stable/std/fs/struct.Permissions.html)
and a little digging into its implementation in the standard library
have shown that this code didn't work as expected in any platform, and
was a bit weird to begin with:
- It first read the permissions from the input file metadata.
- Then it fetched the output file metadata.
- After that, it changed the permissions for that output file metadata.
- It then performed a sanity check that the output file had the expected
permissions.
Barring the fact that the sanity check in step 4 is not needed, the
overall approach is wrong because setting the permissions in a file
metadata struct does not actually persist those changes anywere; it's
just an in-memory change only, so these operations were useless. The
Rust documentation explicitly mentions that the `set_readonly` method
"does not modify the files attributes" [sic], but it's easy to miss that
warning and not realize that it also applies to the methods offered by
the `PermissionsExt` trait. The code only appeared to work because in
most cases the default permissions for new files happen to match the
input file permissions, so the sanity check passed.
To fix this, use the `set_permissions` method on `File` to actually set
the output file permissions to be the same as the input file
permissions, which is both much simpler and robust.
These changes were tested in the context of issue #576, and fix#576.
This PR brings a big overhaul to oxipng's help, with new long form
descriptions of many options.
The full output (--help) is added as a text file MANUAL.txt. Critiques
welcome.
The short output (-h) is simplified and appears as follows:
```
Losslessly improve compression of PNG files
Usage: oxipng [OPTIONS] <files>...
Arguments:
<files>... File(s) to compress (use '-' for stdin)
Options:
-o, --opt <level> Optimization level (0-6, or max) [default: 2]
-r, --recursive Recurse input directories, optimizing all PNG files
--dir <directory> Write output file(s) to <directory>
--out <file> Write output file to <file>
--stdout Write output to stdout
-p, --preserve Preserve file permissions and timestamps if possible
-P, --pretend Do not write any files, only show compression results
-s Strip safely-removable chunks, same as '--strip safe'
--strip <mode> Strip metadata (safe, all, or comma-separated list)
CAUTION: 'all' will convert APNGs to standard PNGs
--keep <list> Strip all metadata except in the comma-separated list
-a, --alpha Perform additional alpha channel optimization
-i, --interlace <type> Set PNG interlacing type (0, 1, keep) [default: 0]
--scale16 Forcibly reduce 16-bit images to 8-bit (lossy)
-v, --verbose... Run in verbose mode (use twice to increase verbosity)
-q, --quiet Run in quiet mode
-f, --filters <list> Filters to try (0-9; see '--help' for details)
--fast Use fast filter evaluation
--zc <level> Deflate compression level (1-12)
--nb Do not change bit depth
--nc Do not change color type
--np Do not change color palette
--ng Do not change to or from grayscale
--nx Do not perform any transformations
--nz Do not recompress unless transformations occur
--fix Disable checksum validation
--force Write the output even if it is larger than the input
-Z, --zopfli Use the much slower but stronger Zopfli compressor
--timeout <secs> Maximum amount of time to spend on optimizations
-t, --threads <num> Set number of threads to use [default: num CPU cores]
-h, --help Print help (see more with '--help')
-V, --version Print version
Run `oxipng --help` to see full details of all options
```
---------
Co-authored-by: Alejandro González <me@alegon.dev>
This update brings several parameterization and usage flexibility
improvements on Zopfli, allowing users to choose to limit execution time
by a number of iterations without improvement, and exposing a more
advanced `ZlibEncoder` struct to tune compression block sizes and
DEFLATE block types. Some minor microoptimizations were also made.
For now, I don't expect this PR to substantially affect how OxiPNG
compresses images using its Zopfli mode, but the additional parameter
customization may come in handy for future work improving how Zopfli is
used in OxiPNG.
Tidy up the API by removing a couple of options we don't really need.
Backup was discussed in #542
Check was discussed in #439
@shssoichiro Just say if you prefer to keep either of these 🙂
Closes: https://github.com/shssoichiro/oxipng/issues/556
Adds a statement saying that `--timeout` isn't as useful for compression
algorithms which use fewer and slower rounds, which OxiPNG tends to use
nowadays.
---------
Co-authored-by: andrews05 <andrew@digerati.co.nz>
This PR is addressing #220. It's not super important but it's a breaking
change, so if it's something we want to do then I thought I should get
it in now before the next release.
- [x] pretend can become another variant of OutFile, probably
OutFile::None, as that's what it essentially is - just another output
destination and not a separate option
- [x] ~~backup and~~ preserve_attrs should become properties of
OutFile::Path variant (so that it would contain Path { path, ~~backup,~~
preserve_attrs }) as they don't have any effect on any other output and
so semantically belong there best
Closes#220
As commented in issues https://github.com/shssoichiro/oxipng/issues/444
and https://github.com/shssoichiro/oxipng/issues/518, there is some user
interest for distributing binaries for each unstable commit, and target
ARM64 platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.
These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.
When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.
In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:
- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
test`, which substantially speeds up their execution. (On my development
workstation, `cargo test --release` takes around 10.67 s, while `cargo
nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
favor of running Cargo commands directly, or using
`giraffate/clippy-action` for pretty inline annotations for Clippy. This
gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
to take advantage of the latest Clippy lints and codegen improvements.
In my experience, when not relying on specific nightly features or
compiler internals, Rust does a pretty good job at making it possible to
rely on a rolling-release compiler for CI, as breakage is extremely rare
and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
takes less of a toll on total workflow run minutes.
## Pending tasks
- [x] Generate universal macOS binaries with `lipo` (i.e., containing
both `aarch64` and `x64` code)
- [x] Tirelessly fix the stupid errors that tend to happen when
deploying a new CI workflow for the first time
- [x] Think what to do with the `deploy.yml` workflow. Should it fetch
artifacts from the CI job instead of building them again?
- [x] Maybe bring back 32-bit Windows binaries. Are they actually useful
for somebody, or just a way to remember the good old days?
---------
Co-authored-by: Josh Holmer <jholmer.in@gmail.com>
This PR makes two changes:
- `--nz` (`idat_recoding`) now disables all zlib recompression,
including iCCP and fDAT chunks. (Perhaps we should rename the option to
`zlib_recompression`?)
- `--nx` now also disables the default deinterlacing, though it can
still be overridden with `-i`.
`--nx --nz` does disable all optimisations in the v8 release and we
should ensure it continues to do so in the next release. (This is
related to discussions around removing the `--check` option.)
This adds a new palette sorting algorithm that attempts to minimise
entropy by an approximate solution to the Traveling Salesman Problem.
The algorithm comes from "An efficient Re-indexing algorithm for
color-mapped images" by Battiato et al
(https://ieeexplore.ieee.org/document/1344033).
It's fast and effective and works in addition to the luma sort (which
remains the single most effective sort). In order to keep lower presets
fast though, I've only enabled this for o3 and higher.
Results on a set of 190 indexed images at `-o5`:
18,932,727 bytes - master
18,578,306 bytes - PR
18,559,863 bytes - PR + #509
(These images may be particularly suited to alternative sorting methods
- the gains here are not necessarily what should be expected on average)
Note I looked into the 120 different palette sorting methods from
TruePNG, as mentioned in #74 (and seen in action in the Zopfli KrzYmod
fork). They're... largely ineffective. The combination of all 120
methods are outperformed by just the existing luma sort plus this new
one. That's not to say there's nothing further to be gained from them,
but trying to brute force all the combinations definitely seems like a
bad idea. There are other algorithms I hope to explore in future...
@ace-dent Thought this might interest you
UPDATE: I realised a quick tweak to alpha values in the luma sort can
provide a great improvement on images with transparency. The following
numbers were taken with PR #509 as base.
`-o2`:
19,065,549 bytes - base (luma sort)
18,949,747 bytes - modified luma sort
`-o5`:
18,922,165 bytes - base (luma sort)
18,559,863 bytes - new sorting algorithm + luma sort
18,544,813 bytes - new sorting algorithm + modified luma sort
This PR makes the oxipng binary process multiple files in parallel,
finally fulfilling #275. There seemed to be some debate about whether
oxipng _should_ do this or not but there's a couple of reasons I think
it makes sense:
1. The concern seemed mostly around the complexity of such a feature.
Not to worry, it was trivial* 🙂
2. Since then, oxipng has dropped from a max of something like 180
simultaneous compression trials down to 10, which is very much a good
thing but it does mean it's not utilising any more cores than that.
Some benchmarks on around 100 files on a machine with 8 cores:
Level | Master time | PR time
-|-|-
2 | 28.303 | 19.005
3 | 36.507 | 23.089
5 | 1:10.86 | 1:16.01
*Some additional changes were required in order to make sure sensible
output is printed to the terminal, since things won't be in order
anymore. Here's some example output from before:
```
Processing: tests/files/fully_optimized.png
file size = 67 bytes (0 bytes = 0.00% decrease)
File already optimized
Processing: tests/files/corrupted_header.png
Invalid PNG header detected
Processing: tests/files/verbose_mode.png
file size = 102480 bytes (12228 bytes = 10.66% decrease)
Output: tests/files/verbose_mode.png
```
And after:
```
Processing: tests/files/verbose_mode.png
Processing: tests/files/fully_optimized.png
Processing: tests/files/corrupted_header.png
tests/files/corrupted_header.png: Invalid PNG header detected
tests/files/fully_optimized.png: Could not optimize further, no change written
102480 bytes (10.66% smaller): tests/files/verbose_mode.png
```
Closes#275, #84, #169, #196 and #419.
[edit] This is the last thing I wanted to land before the next release 🥳
* Add .whitesource configuration file
* Experimental: allow Zopfli to use any size BufWriter
* Allow user to specify the output buffer size as well
* Allow user to specify maximum block splits
* Reformat and fix warnings
* Use deflater on iCCP chunk as well
* Bug fix: need to implement Zlib format
* Make functions const when possible
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Cargo fmt
* Fix compilation
* Fix tests
* Fix more lints
* Fix more lints
* Fix compilation more
---------
Co-authored-by: mend-bolt-for-github[bot] <42819689+mend-bolt-for-github[bot]@users.noreply.github.com>
Co-authored-by: Chris Hennick <hennickc@amazon.com>
Co-authored-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* Keep track of number of pixels in each scanline
* Expand depth to 8-bit
* Attempt expand bit depth
* Simplify tracking of reduction_occurred
* Fix test
* Simplify depth handling in reductions
* Add tests for disabled reductions
* Fix bKGD conversion from gray to palette
* Allow grayscale reduction from 16 to 4 or less
* Refactor reduction evaluation sequence
* Separate palette into new file
* Refactor ColorType
Move transparency and palette data into the ColorType
* Fixup tests
* Make more use of helper functions
* Change BitDepth to u8 representation with TryFrom
* Fix clippy lints
* Don't use unstable language features
* Restore documentation on transparency/palette
* Make dependency on `image` optional
After PR https://github.com/shssoichiro/oxipng/pull/481 was merged, the
`image` dependency became unused when building with debug assertions
disabled, as it is only used to implement output sanity checks when such
assertions are enabled.
The `image` crate transitively pulls a significant amount of
dependencies, so it's useful for OxiPNG users to get rid of them when
not needed.
[Cargo does not allow specifying dependencies that are only pulled when
debug assertions are
enabled](https://github.com/rust-lang/cargo/issues/7634), so the next
best way to give users some flexibility is to gate those debug
assertions behind a feature flag.
These changes add a `sanity-checks` feature flag that controls whether
the `image` crate and the related sanity checks are compiled in. This
feature is enabled by default to keep debug builds useful to catch
problems during development.
* Fix Clippy lints
* Run tests with new sanity-checks feature enabled
* Refactor filters as enum
* Include filter byte in filter output
* Add entropy filter
* Add bigrams filter
* Add bigram entropy filter
* Add brute filter
* Replace bit-vec
* Add tests and benches
* Show filters in help
* Use FxHasher in color to palette
* Use windows function for minor improvement
* Switch main compressor to libdeflate
* Use libdeflater in evaluate
* Use libdeflater to inflate
* Use libdeflater crc
* Tidy up
* Fix benches
* Allow libdeflater/freestanding feature
* Fix building without zopfli
* Update and optimize dependencies
These changes update the dependencies to their latest versions, fixing
some known issues that prevented doing so in the first place.
In addition, the direct dependency on byteorder was dropped in favor
of stdlib functions that have been stabilized for some time in Rust, and
the transitive dependency on chrono, pulled by stderrlog, was also
dropped, which had been affected by security issues and improperly
maintained in the past:
- https://github.com/cardoe/stderrlog-rs/issues/31
- https://www.reddit.com/r/rust/comments/ts84n4/chrono_or_time_03/
* Run rustfmt
* Bump MSRV to 1.56.1
Updating to this patch version should not be cumbersome for end-users,
and it is required by a transitive dependency.
* Bump MSRV to 1.57.0
os_str_bytes requires it.
* Add initial support for changing Zopfli iterations
PR https://github.com/shssoichiro/oxipng/pull/445 did some dependency
updates, which included using the latest zopfli version. The latest
version of this crate exposes new options in its API that allow users to
choose the desired number of Zopfli compression iterations, which
may greatly affect execution time. In fact, other optimizers such as
zopflipng dynamically select this number depending on the input file
size (see: https://github.com/shssoichiro/oxipng/issues/414).
As a first step towards making OxiPNG deal with Zopfli better, let's add
the necessary options for libraries to be able to choose the number of
iterations. This number is still fixed to 15 as before when using the
CLI.
* Fix Clippy lint
Co-authored-by: Josh Holmer <jholmer.in@gmail.com>
* Update and optimize dependencies
These changes update the dependencies to their latest versions, fixing
some known issues that prevented doing so in the first place.
In addition, the direct dependency on byteorder was dropped in favor
of stdlib functions that have been stabilized for some time in Rust, and
the transitive dependency on chrono, pulled by stderrlog, was also
dropped, which had been affected by security issues and improperly
maintained in the past:
- https://github.com/cardoe/stderrlog-rs/issues/31
- https://www.reddit.com/r/rust/comments/ts84n4/chrono_or_time_03/
* Run rustfmt
* Bump MSRV to 1.56.1
Updating to this patch version should not be cumbersome for end-users,
and it is required by a transitive dependency.
* Bump MSRV to 1.57.0
os_str_bytes requires it.
The for loop in apply_preset_5 was not including 9.
apply_preset_6 now builds now calls apply_preset_4 instead of
apply_preset_5, and adds all compression levels from 1 to 9.
Co-authored-by: Nino Burini <nburini@jabra.com>
For RGB(A) images that contain gray colors, this reduction can achieve
significant space savings. However, in the absence of gamma correction
data, some PNG decoders assume more exotic color spaces for grayscale
images instead of the ubiquitous sRGB. This results in gamma
miscorrection, and for the end user this means that colors will look
wrong, like "washed-out". Java's ImageIO class, which is popular in the
JVM world to read PNG files, uses rather unconventional defaults, as
explained in this StackOverflow question: https://stackoverflow.com/questions/31312645/java-imageio-grayscale-png-issue
Gamma miscorrection problems aside, OxiPNG currently tries hard to
reduce RGB(A) images to grayscale, because it expects that reduction to
be quite effective. However, in some cases, OxiPNG generates smaller
PNG files when reducing grasycale RGB(A) images to paletted color than
actual grayscale color. For example, let's say that "~/gray.png" is a
256x256 RGBA image entirely filled with (119, 119, 119, 255) pixels.
OxiPNG, by default, reduces this image to grayscale and achieves a
68.23% decrease:
$ cargo build --release && target/release/oxipng -omax --out ~/out.png ~/gray.png
Processing: /home/user/gray.png
256x256 pixels, PNG format
4x8 bits/pixel, RGBA
IDAT size = 604 bytes
File size = 661 bytes
Reducing image to 1x4 bits/pixel, Grayscale
Trying: 144 combinations
Found better combination:
zc = 6 zs = 0 f = 0 153 bytes
IDAT size = 153 bytes (451 bytes decrease)
file size = 210 bytes (451 bytes = 68.23% decrease)
Output: /home/user/out.png
However, if the --ng option that this commit adds is used to skip the
grayscale reduction step, OxiPNG reduces to a single color palette
instead, which is much more efficient, achieving a 84.42% decrease:
$ cargo build --release && target/release/oxipng -omax --ng --out ~/out.png ~/gray.png
Processing: /home/alejandro/gray.png
256x256 pixels, PNG format
4x8 bits/pixel, RGBA
IDAT size = 604 bytes
File size = 661 bytes
Reducing image to 1 bits/pixel, 1 colors in palette
Trying: 144 combinations
Found better combination:
zc = 3 zs = 3 f = 0 31 bytes
IDAT size = 31 bytes (573 bytes decrease)
file size = 103 bytes (558 bytes = 84.42% decrease)
Output: /home/alejandro/out.png
While OxiPNG should arguably be made smarter to better handle these
cases, in the meantime, adding an option to manually skip that grayscale
reduction can't hurt. In fact, it may even help users achieving the most
out of current versions of OxiPNG, and developers reasoning about what
makes a grayscale-like RGB(A) image compress better with a color
palette.
Due to the reasons stated above, this adds a simple "grayscale_reduction"
option to the Options struct, and a "no-grayscale-reduction" command
line switch, that makes OxiPNG not try this problematic grayscale
reduction on RGB(A) images.
* Switch to crossbeam-channel + rayon::spawn
* Remove thread_spawn for evaluation altogether
This allows to avoid a deadlock when there is only one Rayon thread, and doesn't sacrifice performance, since the caller of .get_result() had to always block on the iterator to be finished anyway, and all the messages are already sent from separate threads.
* Fix `verbose_mode` test
This one is easier to "fix", since we're in control of the rayon pool - just adding one extra thread to the default number to make sure we always can one root "spawn" call without stealing from the actual pool.
