This PR makes 3 changes that together reduce binary size by around 25%:
- Sets lto="fat" in cargo.toml
- Sets panic="abort" in cargo.toml
- Sets location-detail=none in RUSTFLAGS
Closes#571
An unrelated change: I've replaced the zopfli test file with a smaller
one that runs much faster, as well as removing the slow test for
issue-133 which was related to an older alpha optimisation that is no
longer relevant.
The filters used during reduction evaluations are normally set to None &
Bigrams, regardless of any options. This PR makes a slight change so
that if only one filter is specified in the options, this filter will be
used for reduction evaluations too.
This resolves an odd situation affecting lower levels (when --fast is
enabled) where you may try to force the filter to a specific value but
it actually ends up different because a reduction evaluation was
smaller. It's particularly helpful if you're wanting it to be as fast as
possible by using `-o0 -f0` which will now exclusively use None instead
of trying the slower Bigrams as well.
As another use, you could try to brute force oxipng by iterating each
filter separately, though this may not actually achieve anything 😂
[edit] I also pulled the options out into a separate file, though this
wasn't relevant to the filter change.
This PR is addressing #220. It's not super important but it's a breaking
change, so if it's something we want to do then I thought I should get
it in now before the next release.
- [x] pretend can become another variant of OutFile, probably
OutFile::None, as that's what it essentially is - just another output
destination and not a separate option
- [x] ~~backup and~~ preserve_attrs should become properties of
OutFile::Path variant (so that it would contain Path { path, ~~backup,~~
preserve_attrs }) as they don't have any effect on any other output and
so semantically belong there best
Closes#220
As commented in issues https://github.com/shssoichiro/oxipng/issues/444
and https://github.com/shssoichiro/oxipng/issues/518, there is some user
interest for distributing binaries for each unstable commit, and target
ARM64 platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.
These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.
When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.
In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:
- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
test`, which substantially speeds up their execution. (On my development
workstation, `cargo test --release` takes around 10.67 s, while `cargo
nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
favor of running Cargo commands directly, or using
`giraffate/clippy-action` for pretty inline annotations for Clippy. This
gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
to take advantage of the latest Clippy lints and codegen improvements.
In my experience, when not relying on specific nightly features or
compiler internals, Rust does a pretty good job at making it possible to
rely on a rolling-release compiler for CI, as breakage is extremely rare
and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
takes less of a toll on total workflow run minutes.
## Pending tasks
- [x] Generate universal macOS binaries with `lipo` (i.e., containing
both `aarch64` and `x64` code)
- [x] Tirelessly fix the stupid errors that tend to happen when
deploying a new CI workflow for the first time
- [x] Think what to do with the `deploy.yml` workflow. Should it fetch
artifacts from the CI job instead of building them again?
- [x] Maybe bring back 32-bit Windows binaries. Are they actually useful
for somebody, or just a way to remember the good old days?
---------
Co-authored-by: Josh Holmer <jholmer.in@gmail.com>
This adds a new palette sorting algorithm that attempts to minimise
entropy by an approximate solution to the Traveling Salesman Problem.
The algorithm comes from "An efficient Re-indexing algorithm for
color-mapped images" by Battiato et al
(https://ieeexplore.ieee.org/document/1344033).
It's fast and effective and works in addition to the luma sort (which
remains the single most effective sort). In order to keep lower presets
fast though, I've only enabled this for o3 and higher.
Results on a set of 190 indexed images at `-o5`:
18,932,727 bytes - master
18,578,306 bytes - PR
18,559,863 bytes - PR + #509
(These images may be particularly suited to alternative sorting methods
- the gains here are not necessarily what should be expected on average)
Note I looked into the 120 different palette sorting methods from
TruePNG, as mentioned in #74 (and seen in action in the Zopfli KrzYmod
fork). They're... largely ineffective. The combination of all 120
methods are outperformed by just the existing luma sort plus this new
one. That's not to say there's nothing further to be gained from them,
but trying to brute force all the combinations definitely seems like a
bad idea. There are other algorithms I hope to explore in future...
@ace-dent Thought this might interest you
UPDATE: I realised a quick tweak to alpha values in the luma sort can
provide a great improvement on images with transparency. The following
numbers were taken with PR #509 as base.
`-o2`:
19,065,549 bytes - base (luma sort)
18,949,747 bytes - modified luma sort
`-o5`:
18,922,165 bytes - base (luma sort)
18,559,863 bytes - new sorting algorithm + luma sort
18,544,813 bytes - new sorting algorithm + modified luma sort
This PR makes the oxipng binary process multiple files in parallel,
finally fulfilling #275. There seemed to be some debate about whether
oxipng _should_ do this or not but there's a couple of reasons I think
it makes sense:
1. The concern seemed mostly around the complexity of such a feature.
Not to worry, it was trivial* 🙂
2. Since then, oxipng has dropped from a max of something like 180
simultaneous compression trials down to 10, which is very much a good
thing but it does mean it's not utilising any more cores than that.
Some benchmarks on around 100 files on a machine with 8 cores:
Level | Master time | PR time
-|-|-
2 | 28.303 | 19.005
3 | 36.507 | 23.089
5 | 1:10.86 | 1:16.01
*Some additional changes were required in order to make sure sensible
output is printed to the terminal, since things won't be in order
anymore. Here's some example output from before:
```
Processing: tests/files/fully_optimized.png
file size = 67 bytes (0 bytes = 0.00% decrease)
File already optimized
Processing: tests/files/corrupted_header.png
Invalid PNG header detected
Processing: tests/files/verbose_mode.png
file size = 102480 bytes (12228 bytes = 10.66% decrease)
Output: tests/files/verbose_mode.png
```
And after:
```
Processing: tests/files/verbose_mode.png
Processing: tests/files/fully_optimized.png
Processing: tests/files/corrupted_header.png
tests/files/corrupted_header.png: Invalid PNG header detected
tests/files/fully_optimized.png: Could not optimize further, no change written
102480 bytes (10.66% smaller): tests/files/verbose_mode.png
```
Closes#275, #84, #169, #196 and #419.
[edit] This is the last thing I wanted to land before the next release 🥳
* Add .whitesource configuration file
* Experimental: allow Zopfli to use any size BufWriter
* Allow user to specify the output buffer size as well
* Allow user to specify maximum block splits
* Reformat and fix warnings
* Use deflater on iCCP chunk as well
* Bug fix: need to implement Zlib format
* Make functions const when possible
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Cargo fmt
* Fix compilation
* Fix tests
* Fix more lints
* Fix more lints
* Fix compilation more
---------
Co-authored-by: mend-bolt-for-github[bot] <42819689+mend-bolt-for-github[bot]@users.noreply.github.com>
Co-authored-by: Chris Hennick <hennickc@amazon.com>
Co-authored-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* Keep track of number of pixels in each scanline
* Expand depth to 8-bit
* Attempt expand bit depth
* Simplify tracking of reduction_occurred
* Fix test
* Simplify depth handling in reductions
* Add tests for disabled reductions
* Fix bKGD conversion from gray to palette
* Allow grayscale reduction from 16 to 4 or less
* Refactor reduction evaluation sequence
* Separate palette into new file
* Refactor ColorType
Move transparency and palette data into the ColorType
* Fixup tests
* Make more use of helper functions
* Change BitDepth to u8 representation with TryFrom
* Fix clippy lints
* Don't use unstable language features
* Restore documentation on transparency/palette
* Make dependency on `image` optional
After PR https://github.com/shssoichiro/oxipng/pull/481 was merged, the
`image` dependency became unused when building with debug assertions
disabled, as it is only used to implement output sanity checks when such
assertions are enabled.
The `image` crate transitively pulls a significant amount of
dependencies, so it's useful for OxiPNG users to get rid of them when
not needed.
[Cargo does not allow specifying dependencies that are only pulled when
debug assertions are
enabled](https://github.com/rust-lang/cargo/issues/7634), so the next
best way to give users some flexibility is to gate those debug
assertions behind a feature flag.
These changes add a `sanity-checks` feature flag that controls whether
the `image` crate and the related sanity checks are compiled in. This
feature is enabled by default to keep debug builds useful to catch
problems during development.
* Fix Clippy lints
* Run tests with new sanity-checks feature enabled
`crossbeam-channel` is only used if the `parallel` flag is enabled, so
it can be gated behind that feature flag to potentially reduce the size
of the build dependency tree for dependent projects.
While at it, I've fixed several warnings emmitted in tests/flags.rs.
* Refactor filters as enum
* Include filter byte in filter output
* Add entropy filter
* Add bigrams filter
* Add bigram entropy filter
* Add brute filter
* Replace bit-vec
* Add tests and benches
* Show filters in help
* Use FxHasher in color to palette
* Use windows function for minor improvement
* Switch main compressor to libdeflate
* Use libdeflater in evaluate
* Use libdeflater to inflate
* Use libdeflater crc
* Tidy up
* Fix benches
* Allow libdeflater/freestanding feature
* Fix building without zopfli
* Update and optimize dependencies
These changes update the dependencies to their latest versions, fixing
some known issues that prevented doing so in the first place.
In addition, the direct dependency on byteorder was dropped in favor
of stdlib functions that have been stabilized for some time in Rust, and
the transitive dependency on chrono, pulled by stderrlog, was also
dropped, which had been affected by security issues and improperly
maintained in the past:
- https://github.com/cardoe/stderrlog-rs/issues/31
- https://www.reddit.com/r/rust/comments/ts84n4/chrono_or_time_03/
* Run rustfmt
* Bump MSRV to 1.56.1
Updating to this patch version should not be cumbersome for end-users,
and it is required by a transitive dependency.
* Bump MSRV to 1.57.0
os_str_bytes requires it.
* Add initial support for changing Zopfli iterations
PR https://github.com/shssoichiro/oxipng/pull/445 did some dependency
updates, which included using the latest zopfli version. The latest
version of this crate exposes new options in its API that allow users to
choose the desired number of Zopfli compression iterations, which
may greatly affect execution time. In fact, other optimizers such as
zopflipng dynamically select this number depending on the input file
size (see: https://github.com/shssoichiro/oxipng/issues/414).
As a first step towards making OxiPNG deal with Zopfli better, let's add
the necessary options for libraries to be able to choose the number of
iterations. This number is still fixed to 15 as before when using the
CLI.
* Fix Clippy lint
Co-authored-by: Josh Holmer <jholmer.in@gmail.com>
* Switch to crossbeam-channel + rayon::spawn
* Remove thread_spawn for evaluation altogether
This allows to avoid a deadlock when there is only one Rayon thread, and doesn't sacrifice performance, since the caller of .get_result() had to always block on the iterator to be finished anyway, and all the messages are already sent from separate threads.
* Fix `verbose_mode` test
This one is easier to "fix", since we're in control of the rayon pool - just adding one extra thread to the default number to make sure we always can one root "spawn" call without stealing from the actual pool.
This allows to configure or compile away logging in the library from a single place in Rust apps.
For the CLI side, the usage and output remained the same, except it's now colour-coded.
Fixes#217.
* Improve performance consistency
Switch from HashMap / HashSet to IndexMap / IndexSet for consistent iteration order of various options and, as a result, more predictable performance.
