Changelog - Jeffchen54/KMP-Kemono-Downloader GitHub Wiki

• Added fix for #16, simplified installation instructions.
  
  

• Fix to --track switch broken in v0.6.1
  
  

• Compatibility update for changes made to how Kemono party stores urls.
  
  

• Improved file preload performance significantly
• Changed hashing algorithm which reduces collisions in file scanning use case significantly
• Added reupdate feature, this will redownload everything from the update db
• Added switch to use different db name
• Added switch to change download format types
• Updater ignores update entries with a nonexistant path
• Slightly decreased memory usage for file preload (~9% decrease)
• Now only scans text files created by the program.
  
  

• HTTP retries no longer infinite by default
• Post content with embedded containerlized elements like embedded GDrive files links are saved to file
• Duplicate files with now have space in between filename and () to conform to windows standards.
• Duplicate file algorithm now takes into account locally available files, this includes files with old dupe and new dupe naming scheme.
• Duplicate file algorithm no longer overwrites local files
• Temporarily removed resume download when stopped due to HTTP errors, download now completely restarts
• Fixed issue where program hangs up on certain post attachments due to not being supported for download
• Database is now backed up when database related switches are used
• Database updated to contain 2 new entries "artist" and "config", old databases are automatically updated to this new format
• Database is now updated at the end of program execution to lower locking issues
• Increased download thread ceiling to 5
• HTTP code 502 added to HTTP retry list
  
  

• Hotfix for download issue
  
  

• Adjusted thread count to a more reasonable amount.
• Default thread count now is 1 thread, default settings are meant to succeed most of the time.
• Updated user agent.
• Added time between downloads.
• Added beta updater sqllite3 module.
• Confirmed compatibility with Boosty (more testing needed for complete compatibility)
• Added secondary switches for each switch
• Added various omittion switches (omitcomments, omitposttextcontent, minattachmentsize)
• Removed some unused files.
  
  

• Support added for website redesign
  
  

• URL scraping for all services is now multithreaded, significantly decreases web scraping time*
• Slightly improved url web scraping efficiency
• Fixed Experimental mode bug where download count started at 1 instead of zero, main thread will exit early and if there is any download thread that was still active, program will hang up.
• Fixed bug where Pixiv and other non kemono links would lead to infinite retries due to program thinking kemono is trying to rate limit
• Fixed rare data race bug where 2 threads attempt to create a directory with the same name at the same time
• Each thread will now have their own session due to sessions not being thread safe
• Slight program optimizations
• Improved internal documentation and removed some useless/cluttered bits
• Now after a download fails to complete, retries continue from where the download left off, solves issue where retries become infinite in some cases
• Fixed rare discord bug where program appends prefix on url that already contains https://kemono in it
• Max thread count of 20 implemented
• Fixed cases where failed downloads were not being registered even though failures were being written to log

*Correction to all services from non discord services made. 0.5.5 is for all services, documentation was wrong.

• Added experimental download mode
• Fixed several edge cases in discord attachment downloads and improved error messaging
• Fixed bug where password protected zips caused program to hang
  
  

• Pre existing download folder check
• Better logging filename reflecting human readable UTC time
• Fixed issues where zip file check checked more than the extension
• Post file check for partial unpack
• Set HTTP codes to retry
• Extracted files now saved to their own file (similar to extract file option in 7z), prevents certain file overwrite situations
• Extracted dupe file handling made much more robust
• Now deletes empty extracted folder if an error occured while extracting a zip
• post name based exclusion
• Link exclusion
• maintain server fname switch
• Organized switches and made logging prefix more useful
• Fix edge case where \ is at the end of the download folder string
  
  

• HTML 429 retry and config timeout
• Improved error messages
• Error logging
  
  
  

• Vastly improved code organization
• Fixed possible bug where file download count not accurate if file was not downloaded entirely
• Added unpacked download mode, each work will not be placed in their own folder and will instead be placed in a main folder
• Now downloads Discord content in the correct order!!!!!
• Fixed an uncommon bug where an exception could occur due to extracting files of same name by chance
• Fixed issues where an undefined char in a filename results in an nonexistant directory being accessed.
• Removed inconsistency where / is used instead of \ in file paths
• Removed shared resource leak between KMP instances
• File extension exclusion
• Contentless file switch
  
  

• Significantly improved cases where multiple zip files unzip to the exact same destination, creates second directory instead of overwritting existing files
• Fix file_text.txt issue where random numbers are scraped up
• Emoji support added
• Fixed case where post__content.txt was created even though work does not contain any post content
• No longer downloads patreon url file links in downloads segment
• Adjust numbering algorithm to prevent overwriting, note that bad img links will not be downloaded, leaving gaps in the naming
• Fixed bug where post comments not downloaded if downloading of images occurred
• Limited discord functionality added, downloads every image, and all text. Hyperlinks are recorded as well
  
  

• Added connectivity protocol where program will attempt to reconnect forever until a connection is made
  
  

• Images in Content section downloaded
• Fixed extremely rare bug where human readable download file name is corrupted on server so hashed file name will be scraped instead
• Links and text in file segment now saved to file__text.txt
• Patch cases where Fanbox works have polluted links which cause errors if download is attempted on them
• Patch case where an artist work has multiple zip files containing the same directory
• Fixed bug where file name was not being trimmed enough
  
  

• Post comments are now downloadable
• Reports program running time
• Pre emptive server disconnect is fixed
• Fixed issue where non zip files was unzipped.
• Downloads the human readable filename displayed on Kemono itself for attachments instead of the scrambled text
• Fixed issue where certain file types were downloaded as .bin instead of the correct extension
• Expanded automatic unzipping to .zip, .7z, and .rar.
• Replaced illegal file characters with "" instead of "_"
• Japanese and other unusual file names now show proper names instead of corrupted characters
• Fixed issue where post content was generated when no text is present
• Supports Linux style file paths containing './' and '../'
  
  

• Some edits made to be compatible with Windows
• Program configured to be runnable using command line arguments only
• Now outputs total downloaded files
• Slight boost to url scraping speed
  
  

• Various bug fixes
• Fix issue where multiple posts exists sharing the same name, causing infinite download loop
• Improved dialouge
  
  

• Added a more robust fix to file not being downloaded fully bug
• Skip duplicate file based on file name and size
• Removed time between download chunks due to pdf requests chunks being limited to <8KBs in some instances
  
  

• Vastly improve graphical aspect and progress bar
• Added many more command line switches
• Fixed incomplete download issues
  
  

• Fixed bug where certain downloads crashed threads when they do not have Contain-Length header
• Returned chunked download to save memory and progress indicator
• Add setting to automatically unzip files if they are not password protected
  
  

• Bulk download support of urls for all artist work, single page of artist work, or a single artist work
• Multithreading optimization where all urls in bulk download file can be loaded up into a queue instead of having to wait for a url to finish downloading
• Several robust upgrades
• Slight memory optimization
• Fixed bug where certain files names were not being trimmed properly
• Switch to cfscrape library instead of requests for Kemono's ddos protection
• Removed chunk download support and visable download process temporarily, download process can be monitored in task manager perforance tab in network section
• Disposal of most invalid urls and option to quit ran with no input url
• Removed duplicate file check temporarily
• Added command line switches -f and -d
• Fixed bug where downloading large files would timeout before they were completed.
  
  

• Multithreading support
  
  

• Downloads image(s) from a post
• Orders downloads in numerical order shown on post
• Puts creates a post directory at selected download folder
• Visable progress when downloading
• Supports zip downloads
• Saves post text content
• Works with gifs
• Works with mp4s
• Support downloading all sections from artist home page