The filters used during reduction evaluations are normally set to None &
Bigrams, regardless of any options. This PR makes a slight change so
that if only one filter is specified in the options, this filter will be
used for reduction evaluations too.
This resolves an odd situation affecting lower levels (when --fast is
enabled) where you may try to force the filter to a specific value but
it actually ends up different because a reduction evaluation was
smaller. It's particularly helpful if you're wanting it to be as fast as
possible by using `-o0 -f0` which will now exclusively use None instead
of trying the slower Bigrams as well.
As another use, you could try to brute force oxipng by iterating each
filter separately, though this may not actually achieve anything 😂
[edit] I also pulled the options out into a separate file, though this
wasn't relevant to the filter change.
This is a minor change that allows using both `--strip` and `--keep` at
the same time.
E.g. `--strip safe --keep eXIf` will strip chunks while preserving both
the ones that aren't "safe" to remove *and* eXIf. Essentially it's a
convenience to allow extending the default list used by `--strip safe`.
Specifying chunk names for both options is not permitted, e.g. `--strip
eXIf --keep eXIf` will error.
Use of `--strip all` with `--keep` is redundant, but is permitted.
These changes resolve#575 by setting the OxiPNG binary that's about to
be put in a release tarball to be world-executable during the release
workflow, as such permissions are lost when fetching them from
artifacts.
A detailed read to the [Rust documentation for the `fs::Permissions`
struct](https://doc.rust-lang.org/stable/std/fs/struct.Permissions.html)
and a little digging into its implementation in the standard library
have shown that this code didn't work as expected in any platform, and
was a bit weird to begin with:
- It first read the permissions from the input file metadata.
- Then it fetched the output file metadata.
- After that, it changed the permissions for that output file metadata.
- It then performed a sanity check that the output file had the expected
permissions.
Barring the fact that the sanity check in step 4 is not needed, the
overall approach is wrong because setting the permissions in a file
metadata struct does not actually persist those changes anywere; it's
just an in-memory change only, so these operations were useless. The
Rust documentation explicitly mentions that the `set_readonly` method
"does not modify the files attributes" [sic], but it's easy to miss that
warning and not realize that it also applies to the methods offered by
the `PermissionsExt` trait. The code only appeared to work because in
most cases the default permissions for new files happen to match the
input file permissions, so the sanity check passed.
To fix this, use the `set_permissions` method on `File` to actually set
the output file permissions to be the same as the input file
permissions, which is both much simpler and robust.
These changes were tested in the context of issue #576, and fix#576.
This PR brings a big overhaul to oxipng's help, with new long form
descriptions of many options.
The full output (--help) is added as a text file MANUAL.txt. Critiques
welcome.
The short output (-h) is simplified and appears as follows:
```
Losslessly improve compression of PNG files
Usage: oxipng [OPTIONS] <files>...
Arguments:
<files>... File(s) to compress (use '-' for stdin)
Options:
-o, --opt <level> Optimization level (0-6, or max) [default: 2]
-r, --recursive Recurse input directories, optimizing all PNG files
--dir <directory> Write output file(s) to <directory>
--out <file> Write output file to <file>
--stdout Write output to stdout
-p, --preserve Preserve file permissions and timestamps if possible
-P, --pretend Do not write any files, only show compression results
-s Strip safely-removable chunks, same as '--strip safe'
--strip <mode> Strip metadata (safe, all, or comma-separated list)
CAUTION: 'all' will convert APNGs to standard PNGs
--keep <list> Strip all metadata except in the comma-separated list
-a, --alpha Perform additional alpha channel optimization
-i, --interlace <type> Set PNG interlacing type (0, 1, keep) [default: 0]
--scale16 Forcibly reduce 16-bit images to 8-bit (lossy)
-v, --verbose... Run in verbose mode (use twice to increase verbosity)
-q, --quiet Run in quiet mode
-f, --filters <list> Filters to try (0-9; see '--help' for details)
--fast Use fast filter evaluation
--zc <level> Deflate compression level (1-12)
--nb Do not change bit depth
--nc Do not change color type
--np Do not change color palette
--ng Do not change to or from grayscale
--nx Do not perform any transformations
--nz Do not recompress unless transformations occur
--fix Disable checksum validation
--force Write the output even if it is larger than the input
-Z, --zopfli Use the much slower but stronger Zopfli compressor
--timeout <secs> Maximum amount of time to spend on optimizations
-t, --threads <num> Set number of threads to use [default: num CPU cores]
-h, --help Print help (see more with '--help')
-V, --version Print version
Run `oxipng --help` to see full details of all options
```
---------
Co-authored-by: Alejandro González <me@alegon.dev>
This update brings several parameterization and usage flexibility
improvements on Zopfli, allowing users to choose to limit execution time
by a number of iterations without improvement, and exposing a more
advanced `ZlibEncoder` struct to tune compression block sizes and
DEFLATE block types. Some minor microoptimizations were also made.
For now, I don't expect this PR to substantially affect how OxiPNG
compresses images using its Zopfli mode, but the additional parameter
customization may come in handy for future work improving how Zopfli is
used in OxiPNG.
Tidy up the API by removing a couple of options we don't really need.
Backup was discussed in #542
Check was discussed in #439
@shssoichiro Just say if you prefer to keep either of these 🙂
Closes: https://github.com/shssoichiro/oxipng/issues/556
Adds a statement saying that `--timeout` isn't as useful for compression
algorithms which use fewer and slower rounds, which OxiPNG tends to use
nowadays.
---------
Co-authored-by: andrews05 <andrew@digerati.co.nz>
This PR is addressing #220. It's not super important but it's a breaking
change, so if it's something we want to do then I thought I should get
it in now before the next release.
- [x] pretend can become another variant of OutFile, probably
OutFile::None, as that's what it essentially is - just another output
destination and not a separate option
- [x] ~~backup and~~ preserve_attrs should become properties of
OutFile::Path variant (so that it would contain Path { path, ~~backup,~~
preserve_attrs }) as they don't have any effect on any other output and
so semantically belong there best
Closes#220
As commented in issues https://github.com/shssoichiro/oxipng/issues/444
and https://github.com/shssoichiro/oxipng/issues/518, there is some user
interest for distributing binaries for each unstable commit, and target
ARM64 platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.
These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.
When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.
In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:
- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
test`, which substantially speeds up their execution. (On my development
workstation, `cargo test --release` takes around 10.67 s, while `cargo
nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
favor of running Cargo commands directly, or using
`giraffate/clippy-action` for pretty inline annotations for Clippy. This
gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
to take advantage of the latest Clippy lints and codegen improvements.
In my experience, when not relying on specific nightly features or
compiler internals, Rust does a pretty good job at making it possible to
rely on a rolling-release compiler for CI, as breakage is extremely rare
and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
takes less of a toll on total workflow run minutes.
## Pending tasks
- [x] Generate universal macOS binaries with `lipo` (i.e., containing
both `aarch64` and `x64` code)
- [x] Tirelessly fix the stupid errors that tend to happen when
deploying a new CI workflow for the first time
- [x] Think what to do with the `deploy.yml` workflow. Should it fetch
artifacts from the CI job instead of building them again?
- [x] Maybe bring back 32-bit Windows binaries. Are they actually useful
for somebody, or just a way to remember the good old days?
---------
Co-authored-by: Josh Holmer <jholmer.in@gmail.com>
This PR makes two changes:
- `--nz` (`idat_recoding`) now disables all zlib recompression,
including iCCP and fDAT chunks. (Perhaps we should rename the option to
`zlib_recompression`?)
- `--nx` now also disables the default deinterlacing, though it can
still be overridden with `-i`.
`--nx --nz` does disable all optimisations in the v8 release and we
should ensure it continues to do so in the next release. (This is
related to discussions around removing the `--check` option.)
This adds a new palette sorting algorithm that attempts to minimise
entropy by an approximate solution to the Traveling Salesman Problem.
The algorithm comes from "An efficient Re-indexing algorithm for
color-mapped images" by Battiato et al
(https://ieeexplore.ieee.org/document/1344033).
It's fast and effective and works in addition to the luma sort (which
remains the single most effective sort). In order to keep lower presets
fast though, I've only enabled this for o3 and higher.
Results on a set of 190 indexed images at `-o5`:
18,932,727 bytes - master
18,578,306 bytes - PR
18,559,863 bytes - PR + #509
(These images may be particularly suited to alternative sorting methods
- the gains here are not necessarily what should be expected on average)
Note I looked into the 120 different palette sorting methods from
TruePNG, as mentioned in #74 (and seen in action in the Zopfli KrzYmod
fork). They're... largely ineffective. The combination of all 120
methods are outperformed by just the existing luma sort plus this new
one. That's not to say there's nothing further to be gained from them,
but trying to brute force all the combinations definitely seems like a
bad idea. There are other algorithms I hope to explore in future...
@ace-dent Thought this might interest you
UPDATE: I realised a quick tweak to alpha values in the luma sort can
provide a great improvement on images with transparency. The following
numbers were taken with PR #509 as base.
`-o2`:
19,065,549 bytes - base (luma sort)
18,949,747 bytes - modified luma sort
`-o5`:
18,922,165 bytes - base (luma sort)
18,559,863 bytes - new sorting algorithm + luma sort
18,544,813 bytes - new sorting algorithm + modified luma sort
This PR makes the oxipng binary process multiple files in parallel,
finally fulfilling #275. There seemed to be some debate about whether
oxipng _should_ do this or not but there's a couple of reasons I think
it makes sense:
1. The concern seemed mostly around the complexity of such a feature.
Not to worry, it was trivial* 🙂
2. Since then, oxipng has dropped from a max of something like 180
simultaneous compression trials down to 10, which is very much a good
thing but it does mean it's not utilising any more cores than that.
Some benchmarks on around 100 files on a machine with 8 cores:
Level | Master time | PR time
-|-|-
2 | 28.303 | 19.005
3 | 36.507 | 23.089
5 | 1:10.86 | 1:16.01
*Some additional changes were required in order to make sure sensible
output is printed to the terminal, since things won't be in order
anymore. Here's some example output from before:
```
Processing: tests/files/fully_optimized.png
file size = 67 bytes (0 bytes = 0.00% decrease)
File already optimized
Processing: tests/files/corrupted_header.png
Invalid PNG header detected
Processing: tests/files/verbose_mode.png
file size = 102480 bytes (12228 bytes = 10.66% decrease)
Output: tests/files/verbose_mode.png
```
And after:
```
Processing: tests/files/verbose_mode.png
Processing: tests/files/fully_optimized.png
Processing: tests/files/corrupted_header.png
tests/files/corrupted_header.png: Invalid PNG header detected
tests/files/fully_optimized.png: Could not optimize further, no change written
102480 bytes (10.66% smaller): tests/files/verbose_mode.png
```
Closes#275, #84, #169, #196 and #419.
[edit] This is the last thing I wanted to land before the next release 🥳
* Add .whitesource configuration file
* Experimental: allow Zopfli to use any size BufWriter
* Allow user to specify the output buffer size as well
* Allow user to specify maximum block splits
* Reformat and fix warnings
* Use deflater on iCCP chunk as well
* Bug fix: need to implement Zlib format
* Make functions const when possible
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Cargo fmt
* Fix compilation
* Fix tests
* Fix more lints
* Fix more lints
* Fix compilation more
---------
Co-authored-by: mend-bolt-for-github[bot] <42819689+mend-bolt-for-github[bot]@users.noreply.github.com>
Co-authored-by: Chris Hennick <hennickc@amazon.com>
Co-authored-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* Keep track of number of pixels in each scanline
* Expand depth to 8-bit
* Attempt expand bit depth
* Simplify tracking of reduction_occurred
* Fix test
* Simplify depth handling in reductions
* Add tests for disabled reductions
* Update Zopfli and several other depenencies
I've just published a new Zopfli release, v0.7.3, which includes several
new features and internal refactors. Performance should be a tad bit
better, but I didn't test it throughly, so YMMV. Perhaps more
importantly for OxiPNG, its dependency tree is smaller, and
Gzip-exclusive compression code can be excluded at compile time thanks
to new feature switches. As I mentioned on
https://github.com/shssoichiro/oxipng/pull/495#issuecomment-1552669505,
I plan on delivering more significant Zopfli performance improvements at
some point, but for now I think it's good to give the new release more
real-world usage and testing 😄
While at it, I've upgraded other dependencies that are not
performance-critical to their latest semver-compatible versions. This
excludes `libdeflater` on purpose, as its performance characteristics
are said to be somewhat different.
* Update Zopfli to v0.7.4
v0.7.3 was superseeded shortly after v0.7.3 was released to address a
last minute change to the new API it introduced. OxiPNG is not affected
by this, but I think it's good practice to update Zopfli anyway.
* Fix bKGD conversion from gray to palette
* Allow grayscale reduction from 16 to 4 or less
* Refactor reduction evaluation sequence
* Separate palette into new file
