Resume your build - Tomas-M.com

The Build Slax feature now supports resume operation. That means you can now use your favorite download manager which supports resume, and it will finally work. You'll download correct build even if the connection to Slax server is broken several times during the download.

I thought it will be a big problem to implement this feature, and the first attempts failed due to incredibly bad performance.

What is actually so difficult? My goal was to get the output from mkisofs or tar, and before it is sent to the browser (to you), to strip (throw away) the leading and trailing part which is not needed. I do not like to cache the ISOs on server, and I don't also like to store all the data in RAM just in order to process them further. So a pipe has to be created, which filters the output.

My first attempt proved to be wrong, since I tried to use 'cut -b'. Unfortunately cut works on line-by-line basis, so the output were not correct. This didn't work at all.

Then I tried 'dd', because it supports 'skip' parameter. Unfortunately, the skip parameter reads number of BLOCKS to be skipped, so if one wishes to skip any amount of bytes, the BLOCK size has to be 1. It worked correctly, but the performance was really slow, since it had to read the whole ISO byte by byte, resulting in 200 000 system calls to read() and write() per file. That was bad.

Finally, I found that the best performing way to strip leading bytes from the output: head. Using head -c 1000, you can easily echo first 1000 bytes of the data. But I needed the opposite - to throw them away and print just the remaining part. So how to pipe this into mkisofs? The syntax is following:

mkisofs $parameters | (head -c 12345 > /dev/null && cat)

This will take the output of mkisofs and will throw away first 12345 bytes. The rest is simply sent to the browser. You may also need to strip trailing bytes, it's the same using head, just add a minus sign before the numeric parameter, like 'head -c -12345'.

There is only one glitch - the resulting ISO is not exactly the same between two mkisofs calls, even if executed with the same parameters. Because of that, the resumed download may contain some inconsistencies in the header of the downloaded ISO file. This may affect first (approximately) 150 KBytes of the ISO file, so if your download is resumed at the immediate beginning, I suggest you to start it again from scratch. I do not know what are the differencies of the headers, maybe a date is stored there or something? That would be completely harmless...

So please test it and let me know if you find any problems. Thank you!

[1] Ivo Georgiev wrote 293 days ago:

Sorry for the offtopic message, but will the new Slax be built from the packages from the new official Slackware 64 bit port? If so, will you provide 32 version?

[4] Marco. wrote 292 days ago:

Ok, I feel so dumb for having set up a needlessly complicated scheme to achieve the same goal (see http://www.tomas-m.com/blog/349-Build-Slax-resumable-download.html#comments ).

However, at this point why don't you use a simple tail --bytes=+N instead of head && cat ?
Am I missing something?

Secondly, I confirm the issue on varying iso between successive builds, I encountered that phenomenon when working on the aforementioned dumbass approach. I guess too it's some iso metadata field about creation time.

[5] Tomas M wrote 291 days ago:

Why not to use tail instead of head && cat? Well the answer is very simple - tail doesn't know where is the actual end of the data comming from stdin, so it would have to buffer most of it (200MB) in RAM, before deciding how much to output. That is not the ideal approach.

[6] Marco. wrote 291 days ago:

I see, sorry for the silly question. I tested the feature with success, comparing the contents of a smoothly-downloaded iso versus its several-times-resumed counterpart (both were a minimal build with just core module in it, 57M in size):

bash-3.1# mkdir /mnt/iso /mnt/iso-nobreak
bash-3.1# mount -o loop ~/slax.iso /mnt/iso
bash-3.1# mount -o loop ~/slax-nobreak.iso /mnt/iso-nobreak/
bash-3.1# find /mnt/iso -type f -printf "%P\n" |xargs -I string diff /mnt/iso/string /mnt/iso-nobreak/string
bash-3.1#

Thanks for the good job.

[7] Marco. wrote 291 days ago:

Ok, drawing inspiration from Tomas brilliant idea for headless cat, I investigated a bit the issue of different iso upon subsequent idential takes:

bash-3.1# I=100000; until diff <(cat /root/slax.iso | (head --bytes=$I > /dev/null && cat)) <(cat /root/slax-nobreak.iso | (head --bytes=$I > /dev/null && cat)) > /dev/null ; do I=$(( I + 100 )); done ;echo $I
112800

More precisely:

bash-3.1# I=112789; diff <(cat /root/slax.iso | (head --bytes=$I > /dev/null && cat)) <(cat /root/slax-nobreak.iso | (head --bytes=$I > /dev/null && cat))
Binary files /dev/fd/63 and /dev/fd/62 differ

bash-3.1# I=112790; diff <(cat /root/slax.iso | (head --bytes=$I > /dev/null && cat)) <(cat /root/slax-nobreak.iso | (head --bytes=$I > /dev/null && cat))
bash-3.1#

, where the iso is the 57Mb one of my previous post.

[8] Tomas M wrote 290 days ago:

So you found that the difference is in bytes before 112KB.
It would be good to understand what makes the difference.

[9] kstation (Vic Elliott) wrote 290 days ago:

How do you go about using the resume feature? Also, how do you restore saved build.tct files? How do you reset the selections in the builder?

Thanks!

PS:thus is my first post though i have been using slax for a number of years and just wanted to say thank you, Tomas M for producing an awesome and flexible distro and still maintaining a laid-back attitude. If I ever earn any money in this life, I'll send you some ;-)

[10] Semyon wrote 289 days ago:

I tried to download my build of Slax 5 times, but every time download crashes. :-(
I used Opera and web-browser 2.26.1.

[11] Tomas M wrote 287 days ago:

That is the reason for RESUME implementation - if your download crashes, simply RESUME it and only the missing part will download and append to what you have downloaded before.

[12] Semyon wrote 287 days ago:

But sometimes the dounload process doesn't want to resume correctly. I resume it, but then it finishes (not crashes) too early and the file with slax became be unreally small.

[13] Martin S wrote 283 days ago:

Sorry for stupid question, but
when I built my own Slax, there is option "(save your selection for later)" . I can save it to file built.txt on my HDD, but I do not know how can I resume. So when I for example restart my internet browser, or even computer there are defaults modules, but no my modules. Has anybody any ideas?

[14] Marco wrote 281 days ago:

I think that feature (save your selection for later) hasn't been implemented yet; it's there just as a placeholder. I think it would be nice to have it working. For example one could prepare a showcase liveCD of one particular program (eg openoffice) and then spread just the tiny selection file to his audience, leaving them download the cd directly from slax server, without preemptively upload to a separate location.

[15] sami wrote 279 days ago:

sorry to tell ya that resume built doesNOT work I've tried it with a super built of 1.3Gb and I were hoping to get it with no problems but my net connection crushed after 2hrs and no resume was allowed,