Short: GUI HTTP search/download + disk writer Author: Timm S. Müller Uploader: tmueller neoscientists org Type: comm/www Version: 2.3 Architecture: m68k-amigaos Lubricator 2.3 --------------------------------------------------------------------- GUI HTTP search, download and disk writing 1. About 2. Features 3. Minimum requirements 4. Installation 5. GUI usage 6. Command-line usage 7. Icon tooltypes 8. Troubleshooting 9. Privacy concerns A. Server exchange protocol B. Changelog C. Licenses and Copyright 1. About --------------------------------------------------------------------- Lubricator is a combined GUI and command-line tool that allows you to search, download and unarchive Amiga software, and to write disk images directly from the internet to disks on your Amiga. It works fine on machines with just a 68000 CPU and very little memory. Supported disk image formats are ADF and DMS. Also supported is smart unpacking of LHA and ZIP archives, and the creation of bootable disks for running executables. Included in the GUI is (by default) an Aminet and Pouët search facility. Lubricator can deal with simple HTML files and display the links from a web server's directory listing, for example. The principal idea is to eliminate the need for a companion PC, and to get software onto every Amiga as easily and quickly as possible - bypassing the hassle of physical media or setting up network services locally. For demo connoisseurs, Lubricator makes it as convenient as possible to watch demos (including trackmos) from the real machine, so it is easier to resist the temptation of clicking on internet videos, or ending up in an emulator. See also: C. Licenses and Copyright 2. Features --------------------------------------------------------------------- - Smart unpacking: Unpacks archives without producing file clutter. - Async I/O, no temporary files for disk writing: Interleaves trackdisk I/O and pumping disk images all the way through the internet right onto your disks - it won't get any more efficient. - Resource-saving and hardened for low memory conditions: For instance, you can start Lubricator with just 200k of free memory and search and download from Aminet. - Integrated packdisk maker for producing collections of executables (such as intros, demos and games) on a disk with a small menu. - Special index files (dubbed 'amindex') use a simplified format that can contain links to downloads and other addresses. If you are interested in building your own web, download an example amindex file, edit it, save it locally or upload it to your webserver and share it with your friends. 3. Minimum requirements --------------------------------------------------------------------- 68000 CPU, Kickstart 2.x (v37), bsdsocket.library v3, httpclient.library v4 (included) Every Amiga can be connected to the network, use for example - a PCMCIA network adapter (A600, A1200), - or Plipbox or an ACA500+/X-Surf-500 for the A500, - or a Zorro-II/III network card on an A2000/3000/4000. Good TCP/IP stacks for 68000 Amigas are Roadshow and AmiTCP. AmiTCPv4 is also part of the X-Surf-500 software package. Lubricator should also work with the demo version of Roadshow and the freely downloadable AmiTCPv3 from Aminet. HTTPS and FTP downloads (as well as Pouët and Aminet searches) are redirected to a server in my vestibule. So if you download from an URL with the https:// scheme, the download is not really encrypted all the way to your computer, I am man-in-the-middle. But you can set up your own search and download server. See also: 9. Privacy concerns 4. Installation --------------------------------------------------------------------- Not strictly required - can run with httpclient.library in the current directory. To install, copy httpclient.library to LIBS:, and Lubricator wherever you want. For unpacking, the commands lha and unzip are expected in the path. Obviously, you can download them with Lubricator: From GUI: - Change to the 'Aminet' facility, enter "lha.run" as the search term. Double-click the entry to download it to the download directory. Change to the 'Local' facility, double-click lha.run, run it 'Normal'. 'R'eload the 'Local' facility. Double-click LhA.guide for LhA's manual. Using a shell or file-manager, copy lha_68k to C:lha . - Provided that LhA is already installed, change to the 'Aminet' facility, enter "unz550" as the search term, double-click unz550xA.lha to download and unarchive. Using a shell or file-manager, copy UnZip to C: . From command-line: # lubricator search aminet lha.run # lubricator search aminet unz550 # lubricator get http://aminet.net/util/arc/lha.run # lha.run # copy lha_68k c:lha # lubricator unpack http://aminet.net/util/arc/unz550xA.lha # copy unzip c: Note on UnZip: UnZip 5.50 reports warnings for no other reason than an unset timezone (TZ) env variable. See timezone.doc in the UnZip archive. TL;DR: To silence these warnings, set e.g. Central European time as follows: # setenv TZ "MET-1MEST" # copy ENV:TZ ENVARC: See also: 8. Troubleshooting 5. GUI usage --------------------------------------------------------------------- Self-explanatory, unless noted otherwise. Small button captions: 'R' - Reload 'H' - Homepage or download directory Click the cycle button (by default showing 'Index') to get to the 'Pouët' and 'Aminet' search facilities. On these facilities, use the text field to search. Reload with an empty text field to get a list of the latest uploads to the respective facility. Select the 'Local' facility to browse the download directory. Double-click items in the list to invoke their default action (e.g. Unarchive, Write, Download, Show/Execute). Additional keyboard shortcuts: Tab - Cycle through facilities Return - Activate address/search field ESC - Abort downloading, writing, etc. Menu functions: - "Run Executable..." allows you to launch an executable. This function will also set a file's executable bit if this was forgotten somehow (by UnZip, for example). - "Suggest Random Demos..." uses the Pouët search to suggest a random bunch of demos. - "Open Index file..." - to open a local 'amindex' file. - "Make Disk from Executable..." - this function performs a quick format on a disk, copies an executable on it, and makes it bootable. This can greatly improve chances to get a filedemo running. - "Packdisk Maker..." - With the packdisk maker, you can create a disk with multiple demos/intros and a small menu. Double-click executables in the main list to add them to the packdisk maker. Click entries in the packdisk maker to edit their names (as they should appear in the menu). - Bookmarks: Up to nine bookmark entries (edited in icon tooltypes). See also: 7. Icon tooltypes for homepage and bookmarks 6. Command-line usage --------------------------------------------------------------------- Without an address, path or search term, Lubricator starts up with the GUI. Start the GUI in a given facility: # lubricator SHOW local Search in a facility: # lubricator SEARCH aminet cranker Show recent uploads (SHOW and SEARCH can be used interchangeably): # lubricator SHOW pouet "" # lubricator SEARCH aminet "" Download a file: # lubricator GET http://some.host/some.file # lubricator GET http://some.host/some.file TO filename # lubricator GET http://some.host/some.file TO RAM: Download and unpack an archive: # lubricator UNPACK http://some.host/some.archive TO RAM: Write a disk image: # lubricator WRITE http://some.host/some.dms TO df0: # lubricator WRITE hallo.adf TO ff1: Read a disk to a file: # lubricator READ hallo.adf FROM df0: (Warning: Disk writing and reading starts immediately) In addition to that, you can specify the VERBOSE option for more information on network activity. See also: Privacy concerns 7. Icon tooltypes --------------------------------------------------------------------- Lubricator's configuration is stored in its icon tooltypes. - 'STARTPAGE' - The home URL in the 'Index' facility. - 'STARTFACILITY' - Name of the facility to start up with. For example "local", "aminet", "pouet". Default: "index". - 'DOWNLOADDIR' - Download directory in the 'Local' facility, default: SYS:T - 'DEVICE0', 'DEVICE1' - Devices for writing disk images, by default df0: and df1:. It might be a good idea to assign a virtual floppy to the second device, see for example fmsdisk.device from Aminet. RAD: works too. - 'CREATEICONS' - Create drawer icons for unpacked archives. - 'NUMLINES' - Number of lines in the main list. Default: 12 - 'NOAGA' - Remove AGA-only releases from search results. - 'BOOKMARK1' ... 'BOOKMARK9' - Bookmarks available through the bookmarks menu. - 'SEARCHPROXY' - Set the search proxy. The default search proxy handles the facilities 'pouet' and 'aminet' (see below). - 'DOWNLOADPROXY' - Set the download proxy. - 'HTTPPROXY' - Set the HTTP proxy (rarely needed nowadays). Default: none - 'FACILITIES' - List of external search facilities, separated by commas, default: "Pouët=pouet,Aminet". For example, if you are interested in Aminet and games searches only, you could specify "Aminet,Games". A search facility is an arbitrary name sent to a search proxy as an additional argument. Each facility can be preceded by an alias (delimited by an equal sign), which will be shown in the GUI. By default, searches are sent to SEARCHPROXY. In addition to that, each facility can have its own specialized search proxy, by prefixing it with the facility name, e.g. GAMESSEARCHPROXY, AMINETSEARCHPROXY, etc. - 'MULTIVIEW' - An external program to display various files. Default: "SYS:Utilities/Multiview". Kick 2.x users will need to find another program, e.g. "viewer >NIL:" from aminet.net/text/show/viewer.lha. - 'UNLHACMD' - Command for unpacking lha archives. Default: lha -aNq x "%s" "%s/" - 'UNZIPCMD' - Command for unpacking zip archives. Default: unzip -q -o "%s" -d "%s/" By default, unpacking produces an output window only in case of problems. Check out the LhA and UnZip manuals to produce more verbose output. - 'OUTPUT' - Output window specification when starting external programs. Default: "CON:20/20/600/140/Output/AUTO/CLOSE/WAIT" 8. Troubleshooting --------------------------------------------------------------------- Q: The program doesn't start, in the shell it complains about httpclient.library. A: Delete the old version from AmiTCP:libs/httpclient.library. Q: ZIP archives always seem to produce an output window, why? A: Set the TZ env variable. See also 3. Installation, and timezone.doc in the UnZip distribution. 9. Privacy concerns --------------------------------------------------------------------- Privacy policy: This software does not collect private data. If you use this program in the default configuration, queries, results and downloads are roaming the internet unencrypted between the search server and your Amiga. Requests will be logged by the default server for debugging purposes, and deleted automatically after one week or when the protocol grows too large, whatever comes first. Please note: Http servers often redirect downloads to https - so you do not always recognize by its URL if a download needs to be rerouted through the download proxy. If in doubt, you can create your own search and download server. A. Server exchange protocol --------------------------------------------------------------------- If you want to create your own proxy or download server, you need to implement Lubricator's server exchange protocols. These have been kept markedly simple. Download: GET ?url=&token=lubricator The download proxy can pass response headers to the client more or less unmodified, but it should keep their number and size reasonably small. Only the Content-* headers are really interesting. (Note: Future versions of Lubricator and its default server might implement some token negotiation scheme.) Search: GET ?search=&type=&format=lubricator Additionally, the argument exclude=aga can be provided. The search response is a list in ISO-8859-1 text. (Note: Future versions of Lubricator and its search server might implement an encoding argument for other ISO-8859 variants, such as ISO-8859-7.) "key1": attr1="val1" attr2="val2" ... "key2": attr3="val3" attr4="val4" ... ... The key is the list's visual constituent, and relatively freeform. The key length should not significantly exceed 50 characters. A fixed-width font can be expected for display, and some limited ASCII art is possible. Attribute names shouldn't be longer than 12 characters. For escaping quotation marks inside names and values, use ", for the backslash character, use \. Relevant attributes are URI and Type. URI can be absolute or relative. Conventions for the Type keys: "index" - Address, link to some other index "download" - Some file, unspecified "archive" - LhA or ZIP archive "disk" - ADF disk image "diskarchive" - DMS disk archive "packdisk" - ADF disk image Example line from an Aminet response: " comm/net 68k NetFS-revised.lha": Short="Remote filesystem & AREXX between Amigas" Arch="m68k-amigaos" Type="archive" URI="http://aminet.net/comm/net/NetFS-revised.lha" Path="comm/net" Size="620K" Date="" Example line from a Pouët response: "23 Fortocalypse by TEK ++++": Short="OCS demo by The Electronic Knights 2023-05-15" Date="2023-05-15" Kind="demo" Votes="66 ++++" Filename="tek-fortocalypse" Type="download" URI="http://archive.scene.org/pub/demos/groups/tek/tek-fortocalypse" The first attribute (here, "Short") should contain as much additional information as possible, as it is getting displayed witout user interaction. Do not expect significantly more than 40 characters to be displayed here. Please keep the number of additional attributes and line lengths reasonably small. Use the command-line option VERBOSE to monitor request activity. See also: Node.js proxy server examples See also: 9. Privacy concerns B. Changelog --------------------------------------------------------------------- 2.3 - Added READ argument for reading disks to files locally, requested by Henne/AC; improved determination of local filenames from URLs; host header did not provide port numbers, fixed in httpclient.library 4.3. Node.js proxy server examples added. 2.2 - Unpacking of archives could hang during icon creation, fixed; did not actually set the agent string to Lubricator, fixed with httpclient.library 4.1 2.1 - Added command-line UNPACK feature; can now download and unpack to a destination directory from command-line; command-line exits with WARN if a destination file or unpack directory already exists; fixed \ and " escaping in parser; reduced memory consumption for DMS unpacking; fixed minor quirks in documentation generator; improved error reporting and output in command-line disk writing; some cleanup in initialization and exception handling; tries to no longer overwrite a drawer icon provided by an archive 2.0 - Parser bits rewritten in assembler; Minor corrections in signal handling, verbose output, improved documentation; AmigaGuide and Installer added; some requests could be performed twice, corrected; directories are now sorted to top in local facility; now deletes __MACOSX garbage from archives automatically; version number bumped 1.9 - Improved documentation, Added OUTPUT tooltype, improved blocksize calculation, detaches external viewer and executables started in "Normal" mode, added SEARCH=SHOW command 1.8 - facilities now remember their list and address; now uses fixed-width 'System Font' for the main list 1.7 - Added packdisk maker, STARTFACILITY, UNLHACMD, UNZIPCMD 1.6 - Added "Local" facility and browsing the download directory; now avoids unnecessary sub directories when unarchiving; automatically shows unarchived directories; smarter selection of directories and filenames in file requesters; added external file viewing and MULTIVIEW tooltype; improved dialogs for fewer clicks and more intuitive operation; Reload on Aminet and Pouet facilities now lists the current releases; added support for search proxies per facility; improved error reporting, cleanup, many small quirks fixed 1.5 - Did not use the download proxy in command-line, corrected; improved documentation; random demo suggestion did not select the 'pouet', but the first search facility, corrected; added double-click on mainlist actions; added 'Accept' header, added Lubricator as User-Agent; now requires httpclient.library v4 1.4 - Added FACILITIES 1.3 - Corrections in handling of the URL field; configurable search, download and http proxies; when NOAGA is specified, caches are always disabled when starting an executable, otherwise prompts user; "Suggest Random Demos" improved; more meaningful error messages for connection and proxy failures; more verbose executable disks 1.2 - Corrected handling of URI paths without trailing slashes; dramatically improved compatibility when running executables, especially from disk; running executables and making bootable disks from executables with spaces in their filenames corrected; now correctly reports disk errors on read/write/download disk images 1.0 - Improved documentation, some minor bugs fixed 0.93 - Added bookmarks menu; fixed possible illegal memory access during tooltype parsing; improved filetype and html content detection; link texts are now shortened if necessary; passing an URL in the command-line works (again) 0.92 - Fixed crash when clicking 'Cancel' on unpacking DMS to disk; reduced memory consumption; improved path handling, improved error reporting; file downloads can be aborted 0.91 - Added Iconify function; added NUMLINES and NOAGA tooltypes, removed NOASYNC tooltype 0.90 - Bugfix in devicename handling; reduced memory consumption; cleanup 0.89 - Added 'suggest random demos' function; DMS unpacker in 'heavy' mode slightly optimized in assembler 0.88 - Added CREATEICONS tooltype; DMS unpacker optimized in assembler a bit; some fixes and cleanup 0.87 - DMS archives can now be unpacked directly from HTTP to disk, without downloading before. httpclient.library updated to v3.0. 0.86 - Added a function to create a bootable disk for running an executable 0.85 - The DMS unpacker is now integrated, faster, and requires less memory. xdms and the PIPE: handler are no longer needed. 0.84 - Added support for downloading from https and ftp. 0.83 - Added async disk writing and NOASYNC tool type option. 0.82 - For known archives, the default action is now "Unarchive". Progress displays no longer overwrite the link field, but are now shown in the former Type field. Many minor bugs and quirks fixed. 0.81 - added support for writing ADF and DMS from menu and command-line. 0.80 - added unpacking lha, zip, and dms. 0.79 - added rudimentary HTML browsing capabilities. 0.73 - you can now enter an URL in the link field directly, too 0.72 - added VERBOSE option, fixed a crash in property management, link field is now editable, allowing the user to change the link. Added autorewrite to http in search proxy for scene.org servers 0.7 - added proxy support; added httpclient.library 2.1, which fixes some proxy issues 0.6 - added device tooltypes, home button 0.5 - added some Pouët and Aminet search support 0.4 - added some command-line support, added icon, icon tooltype, progress display, abort writing 0.3 - initial version C. Licenses and Copyright --------------------------------------------------------------------- Written and Copyright by Timm S. Müller. Contact: Timm S. Mueller This software is provided "as is", without warranty of any kind, express or implied. Lubricator is freely distributable if the software package including this documentation remains complete and intact. Node.js proxy server examples kindly provided by Bodo Hinüber/Rabenauge. Thanks! xDMS - compiled into Lubricator --------------------------------------------------------------------- Maintainer: Heikki Orsila Author: Andre Rodrigues de la Rocha xDMS is released as public domain software. You can spread it, modify it and do anything with it without even asking first. But the author(s) would like to know if you do something cool with it. httpclient.library - HTTP client Amiga shared library --------------------------------------------------------------------- (c) 2018-2023 individual Computers Jens Schoenfeld GmbH Based on HTTPClient 1.1 by Eitan Michaelson. HTTPClient's original MIT License is reproduced below: Copyright (c) 2006 Eitan Michaelson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.