This still won't help if there's one really huge file that takes
several seconds to write, but if there's a bunch of files together
that is less than a file_chunk_count but take more than a half-
second to write, update anyway
If there's thousands of files in a squashfs (e.g. 400000 like on
some ArcoLinux ISOs) then progress would be reported every
4000 files, which can take quite some time to write. Reduce
file_chunk_count to at most 500, so that progress is reported
more often even if that wouldn't lead to a visible change
in the percentage progress: instead we **do** get a change
in files-transferred numbers.
- The total weight is only needed by the UnpackOperation,
not by each entry.
- Use a chunk size of 107 so that the number-complete seems busy:
the whole larger-or-smaller chunk size doesn't really matter.
- The progress-report was missing the weight of the current
module, so would report way too low if weight > 1. This affects
ArcoLinux configurations where one entry is huge and one is a
single file, so weights 50 and 1 are appropriate.
When there are multiple entries, the overall weight of the
module is divided between the entries: currently each entry
takes an equal amount of space in the overall progress.
When there are multiple entries which take wildly different
amounts of time (e.g. a squash-fs and a single file) then
the progress overall looks weird: the squash-fs gets half
of this module's weight, and the single file does too.
With the new *weight* key for entries, that division can
be tweaked so that progress looks more "even".
- point to main Calamares site in the 'part of' headers instead
of to github (this is the "this file is part of Calamares"
opening line for most files).
- remove boilerplate from all source files, CMake modules and completions,
this is the 3-paragraph summary of the GPL-3.0-or-later, which has
a meaning entirely covered by the SPDX tag.
- Slice overall progress into chunks, with each chunk of equal size
(as long as we have no overall count information) and place
the progress of the current chunk into its own slice.
- The entry knows where it should be mounted, and can remember that
- mount_entry() didn't use self, so made no sense as a method
of the Operation class
- Follow original patch from Gabriel Craciunescu: just drop
the *bufsize* parameter and stick to binary reads.
Text mode was associated in my testing with multiple hangs,
which didn't show up during binary-reads.
- Give the whole entry to file_copy, not just the
destination. This will allow file_copy to work
with local excludes.
- Pluck entry.destination out immediately, to keep
code changes minimal.
- Document the parameters.
- list_excludes() turns the extra mounts from global storage
into --exclude parameters for rsync; it doesn't do anything
with the destination parameter.
- while here rename to something more descriptive
- it's ok if item one creates directories where item two will write,
so don't check for existence of all directories on start-up.
Reported by ArcoLinux.
- Just use the existing rsync code, which can do both
files and directory trees.
- The existing code assumed we were always copying directories.
Now double-check beforehand.
If we don't have/need an image for the rootfs, we might want to
configure the `/` directory as a source for unpackfs. Unfortunately,
this raises an error:
- unpackfs first creates a temporary directory
- it then creates a subdirectory for each source, using the source
path's basename
- when the source is `/`, the basename is an empty string, therefore
the module tries to create an already existing directory
In order to prevent this error, we use the `os.makedirs` function with
parameter `exist_ok=True` instead of `os.mkdir`.
Signed-off-by: Arnaud Ferraris <arnaud.ferraris@collabora.com>
This variable is declared in `if m:`. Of course if this codepath doesn't
run, the variable is not declared an Python doesn't like this kind of
surprise...
Signed-off-by: Arnaud Rebillout <arnaud.rebillout@collabora.com>
src/modules/unpackfs/main.py (UnpackOperation.mount_image): Check
whether entry.source is a regular file or a device and only use
`-o loop` on regular files, not devices.
At least on Fedora >= 29, `-o loop` fails on the read-only device
`/dev/mapper/live-base` (though `-o loop,ro` would be accepted).
- rsync reports its own progress, and reports on files that
find -type f doesn't. This meant that the numbers didn't
match what was stored in entry.total
- The ir-phase adds files to be handled; to-phase happens once
ir-phase is over and the remaining files are processed.
By adding the to-phase files, percentages over 100% were
reported (in part because the number of files doesn't match).
- Update expected entries total from rsync output.
- Re-jig computation of how done everything is: tally it
up in integers, and do only one global progress percentage.
- The mismatch between "ir-chk" and the comment "to-check" led me
to check (ha!) the output of rsync, and it outputs "to-chk"
during small transfers; make sure the comment reflects what
is actually being used to track progress (which is "ir-chk").
Adriaan de Groot
Arnaud Ferraris
Arnaud Rebillout
Kevin Kofler