Let me know if I should add these to the `Cargo.toml`.
All `cargo test` tests pass on the latest x86_64-unknown-linux-gnu
nightly.
---------
Co-authored-by: Alejandro González <me@alegon.dev>
- Fix docs for entropy filter (fixes#733)
- Improve docs for `--fix` (fixes#728)
- Add tests for alpha optimisation
- Following a recent change to the code here, I realised we don't
actually have any tests covering alpha optimisation. I've adjusted some
of the test images to include lines with both leading transparent and
all transparent pixels, and added dedicated tests for them.
- Change x86_64 macOS build to use the macos-15 runner
- There is a new macos-15-intel runner available but we don't actually
need an intel runner to build x86_64. (As support for x86_64 winds down,
we can probably drop these builds eventually - homebrew is the preferred
installation method for macOS anyway)
This PR allows the Brute filter parameters to be configured rather than
using fixed constants. It also tweaks the presets to take advantage of
these new options:
o3: level=1, lines=3 (fractionally worse/faster)
o4: level=1, lines=4 (same as before)
o5: level=4, lines=4 (fractionally better/slower)
o6: level=5, lines=8 (slightly better/slower)
The parameters can be explicitly set in both the API and the CLI.
Adds an option to skip files when the uncompressed IDAT (or the file
size itself) exceeds a certain size. This provides an effective way to
limit resource usage in constrained environments.
Closes#411.
This is a collection of all the BC breaks mentioned in #714, except for
11 which I'm not including for now.
Fixes#658.
Fixes#660.
It might be best to review each commit individually, referencing the
notes in #714 and #660 (I just didn't want to create a dozen separate
PRs).
This is an experiment I started a while ago before life happened. It
reduces memory usage of fast evaluation (-o2 and lower), bringing it
inline with normal (slow) evaluation. It does this by not retaining the
filtered image data of the evaluations, but instead retaining the row
filters that were used in each line so it can be quickly re-filtered
when required. This does incur a tiny performance penalty, but it's
negligible even at `-o0`.
Although this is fully functional, there are a few rough spots in the
code so I'm just opening it as a draft for now.
```
PR -sao6
70.08 real 614.36 user 2.66 sys
2024243200 maximum resident set size
1839339520 peak memory footprint
PR -sao2
11.16 real 60.85 user 1.33 sys
1982562304 maximum resident set size
1842173824 peak memory footprint
PR -sao2 -t1
55.11 real 53.85 user 0.76 sys
429457408 maximum resident set size
245008064 peak memory footprint
master -sao6
67.70 real 616.07 user 2.72 sys
2043379712 maximum resident set size
1838340416 peak memory footprint
master -sao2
11.53 real 60.63 user 1.25 sys
2753396736 maximum resident set size
2283741440 peak memory footprint
master -sao2 -t1
54.29 real 53.55 user 0.72 sys
626311168 maximum resident set size
305252544 peak memory footprint
```
Note that this involves some refactoring of `RowFilter` and the new
`FilterStrategy`. These are breaking changes so it will ultimately be
destined for v10. One advantage to this new structure is it opens the
door for future changes such as allowing the Brute strategy to take a
parameter for the number of lines.
#719 is failing tests due to requiring a newer version of rust than we
currently specify. This PR updates to 1.85.1 and sets the edition to
2024.
I've also updated dependencies and runner images, using the ubuntu arm
runner which removes the need for qemu and other hacks.
Closes#719.
If no palette reduction occurred (or palette reductions were off), a
fully transparent palette entry may not have been zeroed to black. By
ensuring this gets cleaned in the `indexed_to_channels` transformation,
we may able to achieve a grayscale conversion that would otherwise have
been RGB.
This is the final piece of the puzzle in #649 to achieve better file
sizes on the first run and avoid further changes on a second run.
Results from the images in #649 (combining this PR and #673):
9.1.3 1st run: 2,191,773
9.1.3 2nd run: 2,191,629
PR 1st run: 2,191,380
PR 2nd run: 2,191,380
I've replaced a few tests with synthetic images and removed a number
that I don't believe are relevant anymore.
I've also removed the #553 test without replacement as I don't know how
to synthesise a replacement and the issue is still open anyway.
Going forward I think we should avoid adding any new "issue" tests and
instead synthesise something suitable in one of the other collections.
Closes#655.
This PR makes 3 changes that together reduce binary size by around 25%:
- Sets lto="fat" in cargo.toml
- Sets panic="abort" in cargo.toml
- Sets location-detail=none in RUSTFLAGS
Closes#571
An unrelated change: I've replaced the zopfli test file with a smaller
one that runs much faster, as well as removing the slow test for
issue-133 which was related to an older alpha optimisation that is no
longer relevant.
The filters used during reduction evaluations are normally set to None &
Bigrams, regardless of any options. This PR makes a slight change so
that if only one filter is specified in the options, this filter will be
used for reduction evaluations too.
This resolves an odd situation affecting lower levels (when --fast is
enabled) where you may try to force the filter to a specific value but
it actually ends up different because a reduction evaluation was
smaller. It's particularly helpful if you're wanting it to be as fast as
possible by using `-o0 -f0` which will now exclusively use None instead
of trying the slower Bigrams as well.
As another use, you could try to brute force oxipng by iterating each
filter separately, though this may not actually achieve anything 😂
[edit] I also pulled the options out into a separate file, though this
wasn't relevant to the filter change.
This PR is addressing #220. It's not super important but it's a breaking
change, so if it's something we want to do then I thought I should get
it in now before the next release.
- [x] pretend can become another variant of OutFile, probably
OutFile::None, as that's what it essentially is - just another output
destination and not a separate option
- [x] ~~backup and~~ preserve_attrs should become properties of
OutFile::Path variant (so that it would contain Path { path, ~~backup,~~
preserve_attrs }) as they don't have any effect on any other output and
so semantically belong there best
Closes#220
As commented in issues https://github.com/shssoichiro/oxipng/issues/444
and https://github.com/shssoichiro/oxipng/issues/518, there is some user
interest for distributing binaries for each unstable commit, and target
ARM64 platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.
These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.
When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.
In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:
- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
test`, which substantially speeds up their execution. (On my development
workstation, `cargo test --release` takes around 10.67 s, while `cargo
nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
favor of running Cargo commands directly, or using
`giraffate/clippy-action` for pretty inline annotations for Clippy. This
gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
to take advantage of the latest Clippy lints and codegen improvements.
In my experience, when not relying on specific nightly features or
compiler internals, Rust does a pretty good job at making it possible to
rely on a rolling-release compiler for CI, as breakage is extremely rare
and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
takes less of a toll on total workflow run minutes.
## Pending tasks
- [x] Generate universal macOS binaries with `lipo` (i.e., containing
both `aarch64` and `x64` code)
- [x] Tirelessly fix the stupid errors that tend to happen when
deploying a new CI workflow for the first time
- [x] Think what to do with the `deploy.yml` workflow. Should it fetch
artifacts from the CI job instead of building them again?
- [x] Maybe bring back 32-bit Windows binaries. Are they actually useful
for somebody, or just a way to remember the good old days?
---------
Co-authored-by: Josh Holmer <jholmer.in@gmail.com>
This adds a new palette sorting algorithm that attempts to minimise
entropy by an approximate solution to the Traveling Salesman Problem.
The algorithm comes from "An efficient Re-indexing algorithm for
color-mapped images" by Battiato et al
(https://ieeexplore.ieee.org/document/1344033).
It's fast and effective and works in addition to the luma sort (which
remains the single most effective sort). In order to keep lower presets
fast though, I've only enabled this for o3 and higher.
Results on a set of 190 indexed images at `-o5`:
18,932,727 bytes - master
18,578,306 bytes - PR
18,559,863 bytes - PR + #509
(These images may be particularly suited to alternative sorting methods
- the gains here are not necessarily what should be expected on average)
Note I looked into the 120 different palette sorting methods from
TruePNG, as mentioned in #74 (and seen in action in the Zopfli KrzYmod
fork). They're... largely ineffective. The combination of all 120
methods are outperformed by just the existing luma sort plus this new
one. That's not to say there's nothing further to be gained from them,
but trying to brute force all the combinations definitely seems like a
bad idea. There are other algorithms I hope to explore in future...
@ace-dent Thought this might interest you
UPDATE: I realised a quick tweak to alpha values in the luma sort can
provide a great improvement on images with transparency. The following
numbers were taken with PR #509 as base.
`-o2`:
19,065,549 bytes - base (luma sort)
18,949,747 bytes - modified luma sort
`-o5`:
18,922,165 bytes - base (luma sort)
18,559,863 bytes - new sorting algorithm + luma sort
18,544,813 bytes - new sorting algorithm + modified luma sort
This PR makes the oxipng binary process multiple files in parallel,
finally fulfilling #275. There seemed to be some debate about whether
oxipng _should_ do this or not but there's a couple of reasons I think
it makes sense:
1. The concern seemed mostly around the complexity of such a feature.
Not to worry, it was trivial* 🙂
2. Since then, oxipng has dropped from a max of something like 180
simultaneous compression trials down to 10, which is very much a good
thing but it does mean it's not utilising any more cores than that.
Some benchmarks on around 100 files on a machine with 8 cores:
Level | Master time | PR time
-|-|-
2 | 28.303 | 19.005
3 | 36.507 | 23.089
5 | 1:10.86 | 1:16.01
*Some additional changes were required in order to make sure sensible
output is printed to the terminal, since things won't be in order
anymore. Here's some example output from before:
```
Processing: tests/files/fully_optimized.png
file size = 67 bytes (0 bytes = 0.00% decrease)
File already optimized
Processing: tests/files/corrupted_header.png
Invalid PNG header detected
Processing: tests/files/verbose_mode.png
file size = 102480 bytes (12228 bytes = 10.66% decrease)
Output: tests/files/verbose_mode.png
```
And after:
```
Processing: tests/files/verbose_mode.png
Processing: tests/files/fully_optimized.png
Processing: tests/files/corrupted_header.png
tests/files/corrupted_header.png: Invalid PNG header detected
tests/files/fully_optimized.png: Could not optimize further, no change written
102480 bytes (10.66% smaller): tests/files/verbose_mode.png
```
Closes#275, #84, #169, #196 and #419.
[edit] This is the last thing I wanted to land before the next release 🥳
* Add .whitesource configuration file
* Experimental: allow Zopfli to use any size BufWriter
* Allow user to specify the output buffer size as well
* Allow user to specify maximum block splits
* Reformat and fix warnings
* Use deflater on iCCP chunk as well
* Bug fix: need to implement Zlib format
* Make functions const when possible
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Switch to using zopfli::Options in prep for https://github.com/zopfli-rs/zopfli/pull/21
* Cargo fmt
* Fix compilation
* Fix tests
* Fix more lints
* Fix more lints
* Fix compilation more
---------
Co-authored-by: mend-bolt-for-github[bot] <42819689+mend-bolt-for-github[bot]@users.noreply.github.com>
Co-authored-by: Chris Hennick <hennickc@amazon.com>
Co-authored-by: Chris Hennick <4961925+Pr0methean@users.noreply.github.com>
* Keep track of number of pixels in each scanline
* Expand depth to 8-bit
* Attempt expand bit depth
* Simplify tracking of reduction_occurred
* Fix test
* Simplify depth handling in reductions
* Add tests for disabled reductions
* Fix bKGD conversion from gray to palette
* Allow grayscale reduction from 16 to 4 or less
* Refactor reduction evaluation sequence
* Separate palette into new file
* Refactor ColorType
Move transparency and palette data into the ColorType
* Fixup tests
* Make more use of helper functions
* Change BitDepth to u8 representation with TryFrom
* Fix clippy lints
* Don't use unstable language features
* Restore documentation on transparency/palette
* Make dependency on `image` optional
After PR https://github.com/shssoichiro/oxipng/pull/481 was merged, the
`image` dependency became unused when building with debug assertions
disabled, as it is only used to implement output sanity checks when such
assertions are enabled.
The `image` crate transitively pulls a significant amount of
dependencies, so it's useful for OxiPNG users to get rid of them when
not needed.
[Cargo does not allow specifying dependencies that are only pulled when
debug assertions are
enabled](https://github.com/rust-lang/cargo/issues/7634), so the next
best way to give users some flexibility is to gate those debug
assertions behind a feature flag.
These changes add a `sanity-checks` feature flag that controls whether
the `image` crate and the related sanity checks are compiled in. This
feature is enabled by default to keep debug builds useful to catch
problems during development.
* Fix Clippy lints
* Run tests with new sanity-checks feature enabled
`crossbeam-channel` is only used if the `parallel` flag is enabled, so
it can be gated behind that feature flag to potentially reduce the size
of the build dependency tree for dependent projects.
While at it, I've fixed several warnings emmitted in tests/flags.rs.
