|
written by Tomas M. 2 years ago The Build Slax feature now supports resume operation. That means you can now use your favorite download manager which supports resume, and it will finally work. You'll download correct build even if the connection to Slax server is broken several times during the download.
I thought it will be a big problem to implement this feature, and the first attempts failed due to incredibly bad performance.
What is actually so difficult? My goal was to get the output from mkisofs or tar, and before it is sent to the browser (to you), to strip (throw away) the leading and trailing part which is not needed. I do not like to cache the ISOs on server, and I don't also like to store all the data in RAM just in order to process them further. So a pipe has to be created, which filters the output.
My first attempt proved to be wrong, since I tried to use 'cut -b'. Unfortunately cut works on line-by-line basis, so the output were not correct. This didn't work at all.
Then I tried 'dd', because it supports 'skip' parameter. Unfortunately, the skip parameter reads number of BLOCKS to be skipped, so if one wishes to skip any amount of bytes, the BLOCK size has to be 1. It worked correctly, but the performance was really slow, since it had to read the whole ISO byte by byte, resulting in 200 000 system calls to read() and write() per file. That was bad.
Finally, I found that the best performing way to strip leading bytes from the output: head. Using head -c 1000, you can easily echo first 1000 bytes of the data. But I needed the opposite - to throw them away and print just the remaining part. So how to pipe this into mkisofs? The syntax is following:
mkisofs $parameters | (head -c 12345 > /dev/null && cat)
This will take the output of mkisofs and will throw away first 12345 bytes. The rest is simply sent to the browser. You may also need to strip trailing bytes, it's the same using head, just add a minus sign before the numeric parameter, like 'head -c -12345'.
There is only one glitch - the resulting ISO is not exactly the same between two mkisofs calls, even if executed with the same parameters. Because of that, the resumed download may contain some inconsistencies in the header of the downloaded ISO file. This may affect first (approximately) 150 KBytes of the ISO file, so if your download is resumed at the immediate beginning, I suggest you to start it again from scratch. I do not know what are the differencies of the headers, maybe a date is stored there or something? That would be completely harmless...
So please test it and let me know if you find any problems. Thank you! 
[1] Ivo Georgiev wrote 2 years ago:
Sorry for the offtopic message, but will the new Slax be built from the packages from the new official Slackware 64 bit port? If so, will you provide 32 version? [2] Tomas M wrote 2 years ago:
That has not been decided yet. [3] Dude wrote 2 years ago:
It works! [4] Tomas M wrote 2 years ago:
Why not to use tail instead of head && cat? Well the answer is very simple - tail doesn't know where is the actual end of the data comming from stdin, so it would have to buffer most of it (200MB) in RAM, before deciding how much to output. That is not the ideal approach.
[5] Marco. wrote 2 years ago:
I see, sorry for the silly question. I tested the feature with success, comparing the contents of a smoothly-downloaded iso versus its several-times-resumed counterpart (both were a minimal build with just core module in it, 57M in size):
bash-3.1# mkdir /mnt/iso /mnt/iso-nobreak
bash-3.1# mount -o loop ~/slax.iso /mnt/iso
bash-3.1# mount -o loop ~/slax-nobreak.iso /mnt/iso-nobreak/
bash-3.1# find /mnt/iso -type f -printf "%P\n" |xargs -I string diff /mnt/iso/string /mnt/iso-nobreak/string
bash-3.1#
Thanks for the good job. [6] Marco. wrote 2 years ago:
Ok, drawing inspiration from Tomas brilliant idea for headless cat, I investigated a bit the issue of different iso upon subsequent idential takes:
bash-3.1# I=100000; until diff <(cat /root/slax.iso | (head --bytes=$I > /dev/null && cat)) <(cat /root/slax-nobreak.iso | (head --bytes=$I > /dev/null && cat)) > /dev/null ; do I=$(( I + 100 )); done ;echo $I
112800
More precisely:
bash-3.1# I=112789; diff <(cat /root/slax.iso | (head --bytes=$I > /dev/null && cat)) <(cat /root/slax-nobreak.iso | (head --bytes=$I > /dev/null && cat))
Binary files /dev/fd/63 and /dev/fd/62 differ
bash-3.1# I=112790; diff <(cat /root/slax.iso | (head --bytes=$I > /dev/null && cat)) <(cat /root/slax-nobreak.iso | (head --bytes=$I > /dev/null && cat))
bash-3.1#
, where the iso is the 57Mb one of my previous post.
[7] Tomas M wrote 2 years ago:
So you found that the difference is in bytes before 112KB.
It would be good to understand what makes the difference. [8] kstation (Vic Elliott) wrote 2 years ago:
How do you go about using the resume feature? Also, how do you restore saved build.tct files? How do you reset the selections in the builder?
Thanks!
PS:thus is my first post though i have been using slax for a number of years and just wanted to say thank you, Tomas M for producing an awesome and flexible distro and still maintaining a laid-back attitude. If I ever earn any money in this life, I'll send you some ;-) [9] Semyon wrote 2 years ago:
I tried to download my build of Slax 5 times, but every time download crashes. :-(
I used Opera and web-browser 2.26.1. [10] Tomas M wrote 2 years ago:
That is the reason for RESUME implementation - if your download crashes, simply RESUME it and only the missing part will download and append to what you have downloaded before. [11] Semyon wrote 2 years ago:
But sometimes the dounload process doesn't want to resume correctly. I resume it, but then it finishes (not crashes) too early and the file with slax became be unreally small. [12] Martin S wrote 2 years ago:
Sorry for stupid question, but
when I built my own Slax, there is option "(save your selection for later)" . I can save it to file built.txt on my HDD, but I do not know how can I resume. So when I for example restart my internet browser, or even computer there are defaults modules, but no my modules. Has anybody any ideas? [13] Marco wrote 2 years ago:
I think that feature (save your selection for later) hasn't been implemented yet; it's there just as a placeholder. I think it would be nice to have it working. For example one could prepare a showcase liveCD of one particular program (eg openoffice) and then spread just the tiny selection file to his audience, leaving them download the cd directly from slax server, without preemptively upload to a separate location.
[14] sami wrote 2 years ago:
sorry to tell ya that resume built doesNOT work I've tried it with a super built of 1.3Gb and I were hoping to get it with no problems but my net connection crushed after 2hrs and no resume was allowed, | 