Changelog 300-349

version 349

duplicate filter:
the duplicate filter page now has a full-on real-deal file search object to narrow down the duplicate filter, potential duplicate count, and 'show some random dupes' search domains! it also has a 'both files match' checkbox that determines if one of both files of the potential pairs should match the search!
the duplicate filter page has multiple layout changes as a result:
the main management area is now split into two pages--'preparation', for doing maintenance and discovery work, and 'filtering', for actioning the potential dupe pairs found
the 'filtering' page will select by default, but if 'preparation' needs work, its name will become 'preparation (needs work)'
the 'filtering' page now has file search ui and the 'both files' checkbox instead of the file domain button. this search data is saved on a per-page basis
the two pages' status texts are now updated on separate calls which have been rewritten to be asynchronous (with 'updating...' text while they work). both now have explicit refresh buttons to force them to update
the additional non-unknown pair counts listed on the filter area, which were irrelevant to filtering and sometimes confusing, are now gone. it only lists the 'unknown' pair number
the duplicate filter page's help button no longer has the awful 'simple help' entry. the full html help will get a pass in the coming weeks to reflect the new search changes
the duplicate file db code received significant refactoring and improvement to support searching the potential dupe space while cross-referencing the new file search context (and still falling back to the fast code when the search is just blank/system:everything)
misc duplicate file db code cleanup and refactoring
while in advanced mode, you can no longer select 'all known files' file domain for an export folder (and now the duplicate filter page) search context
making a file delete action in the duplicate filter (by hitting delete key or the button on the top hover window, which both trigger a dialog asking to delete one or both) now auto-skips the current pair
.
manage tags:
the manage tags has a new 'siblings and parents' button that will auto-replace incorrect siblings and auto-add missing parents! it works on multi-file selections as well! it gives you a summary yes/no dialog before it fires
the manage tags dialog has a little logic cleanup r.e. siblings and parents and their cog auto-apply options. the auto-application now only applies on add/pend actions
the manage tags dialog has a new cog button option to not trigger 'remove' actions from an autocomplete dropdown or suggested tag input action when the tag already exists
.
the rest:
gave video metadata parsing another pass--it now detects 'hidden' incorrect framerates due to advanced 'skip frame' codec settings and is more accurate at determining frame count and duration, including some fixed offset calculations that was sometimes adding or discounting a few frames
manual video frame count, when needed, is now faster and produces better error text
fixed a critical bug in thumbnail regen that was sometimes potentially looping regen on files with unusual rotation exif information
significant improvements to how the client file manager handles thumbnail identifier information, saving a great deal of time for file import and thumbnail regeneration code of videos
fixed an issue where regenerated file metadata was not propagating up to the ui level in real time
cleaned up some thumbnail cache initialisation code
the 'generate video thumbs this % in' option is moved from the 'media' to 'thumbnails' options page
to simplify code, and in prep for the idle-maintenance-rewrite of this system, the database->regen->thumbnails call is now removed
all three fields of text on serialised pngs now wrap, and they pad a little better as well
added a new option to the 'gui pages' options page to force input text box focus on page changes
fixed a small type issue with the server's session cookie code and some new library versions

version 348

wrote some OR search help to 'getting started with tags' help page
wrote a new multi-reader, single-writer lock object for the client file manager, along with some unit tests for it.
updated the file and thumbnail access and regen and maintenance code to use the new lock. various access is now faster when available and overall safer. there is still work to do here
adjusted file import to be less aggressive about locking, which should reduce some file/thumbnail access lag during heavy imports
the thumbnail space estimate in the migrate database dialog is now more adaptive to the new more flexible thumbnail size system and specificies better that it is an estimate
the client api's /add_tags/add_tags call now collapses siblings and expands parents on an add/pend call. this can be turned off by setting the new optional parameter 'add_siblings_and_parents' to false. the help is updated regarding this and the client api version is incremented to 6
fixed the client api's /add_tags/add_tags call for the 'hashes' parameter, which was failing to parse, and added an accidentally missing unit test to check this in future
the client local services (the booru and client api) now have two new options under their 'manage services' panel: 'support CORS', which turns on cross-orogin support (which is experimental for now, so defaults to False), and 'logs requests', which controls whether your log will be spammed with request reports (this also defaults to False), which should clear up some 100MB+ log hassle for people using the Hydrus Companion browser add-on
hydrus services now respond correctly (albeit sparsely) to OPTIONS requests, and if CORS is enabled, to CORS OPTIONS requests. there are unit tests for this that seem to work ok, but I think we'll need to verify it irl
finished a first version bitmap manager to handle all wx bitmap creation and destruction, including recycling mid-steps
updated all simple wx bitmap creation and destruction calls across the client to use the new bitmap manager, improving stability and saving some CPU
fixed some incorrect button alignment flags that were causing problems for clients set to assert check these values
added a new yiff.party file url class to the defaults that matches a new file attachment format
updated the 'url' content parser so if a parsed url is in the form 'summary_text url', as some booru source fields sometimes specify, the preceding summary text is removed, cleaning up the resultant url
silenced an old server connection lost error that was needlessly loud
silenced the client network engine from additionally log-printing SizeException errors when a downloading file (usually gif) exceeds file import options rules
improved misc window destruction code
updated supported mime list in 'getting started with files' help and website index
misc cleanup

version 347

or search:
under construction OR predicates now present at the top of the regular tag results list, prepended with 'OR: ', and skipping default selection
this new OR line is enter-able, which will present it as-is, rather than adding new preds
hitting escape on a 'search' tag input box that is empty but has an under construction or predicate will cancel the or pred
hitting escape on a 'search' tag input box otherwise should more reliably kill its focus when the dropdown is a float window
improved OR search efficiency significantly with dynamic OR search triggering based on other search predicates. OR searches including negated '-tag' components should be massively faster when paired with non-OR tag or file search predicates
I believe I fixed a search issue that would sometimes return insufficient results when OR preds are mixed with certain other combinations of tags
improved reliability of some thumbnail refresh calls
cleaned up a bunch of OR handling ui code
.
the rest:
after previous weeks' experiments, wrote new double-layer thumbnail loading system--now too-small thumbs will quckly scale up fuzzily straight to screen, and then in the coming seconds, the nice regenerated full-size thumb will be made and drawn in place as ready. it presents much faster and looks better, but there is some cleanup to do here that I will tackle next week
all local file trashing events now record a context-appropriate deletion statement such as "Deleted from Media Viewer." this value is recovered in 'deleted' import status 'notes'. You will mostly see 'Unknown deletion reason.', for files deleted before this new system, but it will populate with appropriate info over time
fixed a search optimisation that was not cross-referencing with file domain, meaning for instance that bare system:rating calls were returning since-deleted files
upnp management window now uses new listctrl
cleaned up some old custom page-naming code
added a 'data' debug call to clear out all cached thumbnails and force an instant ui thumb reload
fixed the trash bmp misalignment, ha ha
removed e-hentai login script from the defaults, since this testing script is not appropriate for new users
dejanked some media viewer video transitions by cleaning up animation bar rendering and smoothing out video buffer initialisation
cleaned out some surplus subprocess wait calls that were hanging some systems on various 'open externally' calls
fixed multiple syncing problems with 'synchronise' export folders that produce files with subdirectories. subdirectory structures should now be synced correctly and empty folders deleted
export folders that collapse multiple file results to the same duplicated name should, after the next run, do less overwriting to this same name
if an export folder or the regular export dialog makes a file destination path that is above the chosen directory (e.g. if the path starts with ../ or ..\), the export job will error out with an explanation
big manual file exports _should_ be politer to the ui and cause fewer hangs
doing page tab drag and drops may have less post-drop ui jank on linux, continued feedback would be appreciated
moved 'reason' handling for all content updates to its own area, which neatens many content update data handling issues
fixed petitioning a tag via a shortcut, which had bad reason handling
fixed an issue with committing pending ipfs items that was overchecking service permissions
fixed some remaining bad wx code in the unit tests
misc file status reporting cleanup

version 346

or search:
extended the search predicate object to handle more OR stuff
extended the tag list to handle list objects that have multiple colours
extended the new OR search predicate to report multiple text-snippet-and-colour pairs based on sub-predicates
extended tag search input to handle prototype OR predicate creation--hold shift when you enter the tag, and it'll start an OR chain. shift-enter continues the chain, enter alone completes it
fleshed out the predicate unit tests to cover more of this
wrote unit tests for OR search predicates. it seems good!
improved some search logic to apply system preds better in certain edge cases and spend less CPU on OR-search-only searches
.
thumbnails:
thumbnails will now queue for load in a more intelligent order based on estimated difficulty to regenerate, which will tend to put more thumbs on screen faster
the decision to regenerate a thumbnail from source is now tempered by how different the current thumbnail is from what is desired--the more similar the two sizes, the more (randomly) likely the client will decide to just use the current (resized) this time. this smooths out change-lag while limiting the number of really fuzzy thumbs you get. feedback on how this works IRL would be appreciated--it uses some voodoo distribution polling to figure it out, which I can definitely tweak
improved visual quality of thumbnail scale-up optimisations
fixed an issue where a multipage thumbnail grid would incorrectly recalculate the new virtual height after a thumbnail size change event, leading to a bit of invalid extra scrollspace (with noclip rendering errors) at the bottom
the thumbnail right-click menu's reparse files entry is now extended to a new 'regenerate' submenu with three options: reparse file and regen thumbs (the old action), force regen thumbs, and regen thumbs if wrong size!
the new 'regen if thumbs wrong size' action sends how many thumbs needed resize up to the popup window, as well
moved some old thumbnail regen code responsibility out of the db and into the files manager
cleaned out some old redundant file/thumbnail code
cleaned and refactored a bunch of general image handling and resizing code
.
the rest:
fixed some bad serialisation code that was making file search objects set their 'include current tags' value to false/true on interleaving loads. on this update, all 'include current tags' values are blanket reset to true
fixed an issue that was drawing animation canvases pure white on various media update events
extended manage urls dialog to support multiple files when launched from a selection of thumbnails. there is a warning in this case, noting that only gallery-style urls are appropriate to be added to multiple files
manage urls dialog now supports multiple selections, including shift-select, and accepts delete key presses for easy mass deletion
when you ask the database migration dialog to move some files, it now pops up a confirmation dialog that also asks if you would like to limit the max time for the job as 10, 30, or 60 minutes
improved file permission setting code across the program to be more sensible for non-Windows
if you are a non-Windows user and were hit with directory permission problems last week on the thumbnail update--which resulted in the rxx directories not being deleted--the update this week will attempt to do the delete again, this time correcting the now missing execute permission bit. if it finds outstanding rxx directories to delete, it will give a popup beforehand summarising the situation and giving you a chance to bail out
fixed yet another problem that was stopping client api url requests from finding the correct page by name
when a client api url request includes fixed tags, these tags should now propagate in all scenarios where the single url produces multiple files
updated sqlite dll and console for windows
misc fixes and cleanup

version 345

or search:
set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support
started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way--we'll work on this as we discover the most common inefficiencies
.
thumbnails:
the client no longer uses both 'master' and 'resized' thumbnails--it uses a single, smarter thumbnail
only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)
the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui
the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens
if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over
due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048
all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed
added more thumbnail-resizing related popup spam to file report mode
fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded
updated database migration help for new concepts and ui
cleaned up some misc storage code
.
the rest:
fixed a problem in the client api with fetching file identifiers from file_ids
fleshed out 'help my db is broke.txt' with more specific clone recovery examples
fixed import support for a variety of single-frame music webms
fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set
network report mode now states url classes of urls about to be parsed
misc small fixes and cleanup

version 344

final v1.0 client api polish:
added optional 'show_destination_page' arg to '/add_urls/add_url', defaulting to False, to control whether an URL-add will select (i.e. jump to) the destination page in the ui. this changes the default behaviour for this command
simplified the routine that finds or creates a watcher or url import page and fixed a bug in the api that was not creating new pages when destination_page_name was specified
some misc cleanup
fixed fetching file_metadata by hashes
fixed the client api help regarding file_metadata response example tags
client api version is now 5
.
the rest:
psd support added! because of this format's potential multi-layer complexity, it will not render natively, but width and height are parsed. it is treated as 'application/x-photoshop'. PSB is also recognised and treated as psd
added a 'open_known_url' shortcut to the 'media' shortcut set that lets you quickly open URLs for files. if there is one recognised known url, it will be launched, and if there are multiple, a list with all known urls will appear to select which one you want
animation scanbars now show an x/y current timestamp! it includes millisecond timings and even works for variable framerate gifs. whether to show a second/shadow caret for timestamp position on variable frame rate is a new discussion to have
fixed an issue where animations would sometimes not resume animation for several seconds after a big scanbar drag
when the thumbnail manager cannot produce a thumbnail due to a storage error (like a missing file), it now only puts up a single, more informative error popup on the first problem. subsequent errors are printed silently to the log. (these errors tend to come in en masse, so this cuts down on spam and error-related ui lag that was making loading a bad session difficult)
improved error reporting when an upload pending command would fail due to service non-functionality--it should now give a popup with error info imediately, rather than obscured through the login system
added temp_dir parameter to the client and server that will override which temporary directory the program will use
cleaned up how no_daemons and no_wal mode are handled internally
no_wal mode now has to be called from the command parameter, the no_wal file hack in the db directory no longer works
missing ffmpeg errors now prompt the user to check if it is installed
searching for numerical ratings should now work for files that were rated when the service had a different number of stars (ratings now searches in 'bands' rather than exact values)
reduced the min height of the new import files frame's list
doubled the decompression bomb test to permit files up to ~179 megapixel, we'll see how it goes
misc cleanup

version 343

client api:
fixed an int/str type mismatch issue with service_names_to_actions_to_tags in /add_tags/add_tags in the client api that meant that argument was not working
fixed up some last /get_files/search_files stuff
added /get_files/file_metadata
added /get_files/file
added /get_files/thumbnail
added help and unit tests to reflect the above
updated client api version to 4
.
the rest:
the list of paths in the manual file import dialog is now sortable. this order will be preserved in regular and 'add tags' ok events for this dialog. it has a new '#' column so you can return to 'parse' order if desired
animation and static image windows in the media viewer canvas are now recycled through media type transitions, making for slightly smoother browsing between mixed media
increased aggression of media viewer image prefetch
added support for 'MM' Tiffs
fixed webm mime parsing for webms with no audio (these were falling back to mkv)
improved the error reports when a serialised png fails to import
the mass-open-urls popup is now pausable as well as cancellable
fixed several recently broken ui unit tests
misc old code cleanup
some misc test controller/constant refactoring

version 342

added support for webp import. it does not yet support animated webps, which, if the local platform supports, will import like apngs used to: just the first frame
added support for tiff import. it works ok for 24bit and 8bit (monochrome) tiffs, but I am not sure how well it will do with 48bit
both webp and tiff should work on the duplicate files system
improved webm detection to include opus audio (previously, these files were falling back to mkv)
fixed an issue where unusual formats with duration but no frames or frames but no duration were being sorted and otherwise presented incorrectly
improved autocomplete job cancelability. this job can now cancel much faster on large jobs, meaning typing searches with large result sets will hit less CPU and return faster on subsequent keystrokes
_all_ of the complicated 'copy url' commands from the thumbnail right-click->known urls menu are now available on the 'open' submenu! if there is more than one url to open (e.g. 'open all of these files blahbooru post urls' on a selection of 50 files), you will be presented with a yes/no dialog to confirm, and it will open one url in your browser every second (with a cancellable popup if num_urls > 5)
by default, system:everything is now hidden if its total files is >10k. you can force it to always show under options->default system predicates
the gallery downloader's list's status column now shows gallery status (deferring to active file status) when appropriate and shows 'done!' when all work is complete
after working back and forth with a user, I _believe_ the linux similar files >0 distance search crash is finally fixed
fixed sorting by media views/viewtiming with collections
a single-selected collection right-click now shows total media views for all files in the collection! you can now see how long you have been viewing an artist!
fixed an issue that lead to export folders not running on always-on clients as often as they should
updated the gelbooru 0.2.5 file page parser to pull rating tag from the correct location (previously, it was pulling from what appears to be a site-wide 'mature' browser hint)
improved memory cleanup stability when animations and other parts of the video rendering pipeline are deleted--this _may_ fix some rare crashes
increased animation rendering aggression overall and particularly in 'future' of frame buffer
if a video renderer that is asked to start some way into the video fails to render anything, it will now fall back to trying to render from the beginning. this is slightly hacky atm and leads to out of phase rendering frames, but it is better than an error
added a '--no_db_temp_files' launch parameter that will force the client or server to return to the recent old behaviour of exclusively using memory for journalling. this is useful if your temp directory is small and/or your available ram is very large. if running in this mode, the client will attempt to check available memory (instead of free space on your temp dir) before performing very large transactions
with the new lighter-weight update transactions, the client now tests for less free space for journalling before running repository update processing
added /get_files/search_files to the client api, which does the first half of file searching. it allows tag search (including -tag negation) and system inbox/archive. since the second half, which will fetch file metadata, is not yet in, this can't do anything interesting yet
updated help and unit tests to support this, client api version is now 3
some misc refactoring

version 341

client api:
added /add_tags/add_tags, which does several kinds of tag content updates
added /add_tags/clean_tags, which shows how hydrus will handle potential tags
added /add_urls/associate_url, which allows you to associate urls with files
added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)
updated client api version to 2
updated client help and unit tests for the above
added a linked contents to the client api help
improved some server error handling, mostly moving 403s to more correct 400s
improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500
.
the rest:
tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user
when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)
added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details
reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements
the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows
when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8
improved serialisability of some pending tag data
watchers can now hold and pass on fixed pending tag data
gallery log objects can now hold and pass on fixed pending tag data
file import objects can now hold and action fixed pending tag data
hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects
improved some url-import page drop-target-selection logic
improved error reporting when dropping/api-adding urls
adjusted some url import workflow so big 'already in db' download lists should work a bit faster
attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made
tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth
fixed a source of linux CRITICAL logspam related to several common dialogs
improved ui stability on boot when file folders are missing (particularly for linux)
improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better
did some more misc stability improvements, particularly in various boot fail scenarios
completely removed an ancient and janky focus catcher widget from the main gui frame
now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory--hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large
fixed approx bitrate sorting for malformed video files with explicitly '0' duration
daemon_profile_mode now spams some more info about export folders
fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status
updated windows build to cv 4.0
misc refactoring and fixes

version 340

client api:
fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly
fixed the 'add_url' api call's instability
the API will now always return JSON on 200. anything else should be presumed to be raw text
'/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week
'/request_access_permissions' now returns JSON
'/add_url' now results JSON on success with more info, 403 on failure
'/get_url_info' now returns the 'normalised_url' in the response JSON
added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url
added '/add_files/add_file', which can import a file from a path or bytes
added '/add_tags/get_tag_services', which will return info on the client's tag services
updated client api help to reflect the above changes and fleshed out the intro a bit
fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time
updated the client api tests to check the above
refactored client api tests to be neater and in their own file
.
the rest:
fixed the page of pages close bug
added a downloader for nijie.info to the client defaults (it needs a login)
updated danbooru file page parsers to get 'rating' tag
added gelbooru 0.1.11 parser for future application
fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction
added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)
wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest unicode replacement characters
the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks
moved some more old daemons to the new job scheduler, deleted some old daemon code
improved some daemon job wake and shutdown code
wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code
simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update
fixed upnp mapping fetching to cope with ipv6 results
improved some memory clearing code to deal with some semi-stubborn objects
improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine
improved menubar stability, both in finding menus and swapping them out
if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup
fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions
fixed two login invalid cookie error handling bugs
maybe made some dupe filter searching more stable
fixed a py2 datatype issue that made the client unbootable when updating the client from <296
the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go
slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB
added experimental secret discord bugfix dnd mode checkbox
improved how html parsing deals with some unexpected bad tag data
turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays
wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work
improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up--this is an ongoing project
improved how some text import line splitting works
misc fixes

version 339

client api:
wrote some ui to handle client api permissions adding and editing
wrote a 'catch a permissions request' mini-dialog for external api permissions adding
wrote api calls:
GET
/api_version
/request_new_permissions
/verify_access_key
/add_urls/get_url_info
POST
/add_urls/add_url
and made a new 'client api' help page to describe in detail what these do
wrote fairly comprehensive unit tests for the new client api
refactored a bunch of 'hydrus network' specific stuff away from general server code that the client api now uses
neatened up 401 vs 403 error handling across the program, and replaced some clientside error handling that was inelegantly borrowing these network errors
deleted very old prototype file/thumbnail client server fetch code, which was no longer in use
.
misc and bug fixes:
added a 'clear ratings' button to the ratings service 'review services' panels. it can clear out ratings for deleted, non-local, or _all_ files
the '3 downloaders are working, is it ok to close the client?' message is now folded into the 'confirm client exit (auto-yes in 15s)' dialog. this merged dialog will still appear for users who have the regular confirm client exit dialog turned off (and still auto-yeses in 15s)
the file url downloader now reports 'downloading file' and 'importing url' text status separately
fixed a typo bug from last week that was breaking asc/desc ratings service sorting
fixed a typo bug from last week that was stopping manage import folders from opening
fixed a typo bug from last week that was breaking setting upnp port on the local booru/client api service management panel
the advanced file reparse-and-re-thumbnailing routine now correctly moves a file to its new extension if its mime changes (e.g. png->apng, or webm/mkv distinctions)
the client file manager now silently detects and auto-repairs instances of missing files where the file actually does exist, just with the wrong extension
fixed a url parsing issue that was normalise-mangling url classes with no path but some query parameters
the network engine now uses utf-8 decoding when no specific encoding is set (previously ISO-8859-1)
fixed an ffmpeg video parsing bug when the video included Shift-JIS metadata. it should work for other unusual encodings as well
maybe cleaned up some menubar management code
the filename tagging dialog now uses a notebook for service choice, like the manage tags dialog, rather than the janky old listbook
fixed a py2-to-3 issue with the admin-only 'is server currently busy' check while a backup is running
improved some dialog button event handling. it may completely fix the 'trytoendmodal' issue some users run into
improved some JSON db serialisation error reporting code, trying to pin down an issue several users have seen with session save
improved thread-safety of serialisable objects as they serialise
misc improvements and cleanup

version 338

after talking with some users, put a bit of time into tag autocomplete wolkflow. both read (search pages) and write (manage tags) autocomplete inputs now operate ~asynchronously~, with the tag fetch working on a separate thread. tag jobs can also now cancel at certain checkpoints in the tag search process if overwritten by a new request. therefore, a variety of tag lookup scenarios _should_ be less painful. this change was executed in a semi-inelegant way, so please report if you encounter bugs from fast typing etc...
I also improved some of the "I hit enter before results were in" code as a result of this. not sure I have it totally nailed, so please give feedback on errors here
wildcard search tags now have an explicit '(wildcard search)' after their label
taglists that have an attached page of media (basically the 'selection tags' box and the 'active search predicates' up top) now have a right-click menu entry to 'select files with (all these tags)' and, if more than one tag is selected, 'select files with (any of these tags)'! This is pretty neat in action, so give it a go!
added a 'sort by approx bitrate' file sorting option that does a basic filesize/duration so you can filter out dense gifs and other short-but-big vids a bit easier. anything without a duration is shoved to the 'smallest' side
some of the file sort options now default to their respective 'biggest' first, see how you like it
folded in updated gelbooru file page parser (fixing the 403 errors that just appeared) and added searches for gelbooru user favourites and pools
if a gallery or watchable url ends up in the file processing queue (and hence fails), it now says the believed-to-be url class name in the error, which should help some false positive url class matching debugging
fixed a focused-file selection issue that meant preview viewtime was frequently counting in file viewing statistics even while a page was not currently in view
fixed the local booru review service panel, which wasn't fully deleting shares when the button was clicked
wrote a service object and basic server skeleton for the client api (basically refactoring the existing local booru code). client api now appears in manage and review services and can boot and present the normal hydrus browser welcome page at '/'
the client api and the local booru now have an 'allow non-local connections' checkbox! defaulting to off and on respectively
updated a wx-thread call function to more safely and universally deal with instances where the responsible window died before the call could be made
removed some old experimental crypto code that isn't used any more--pycryptodome is no longer needed to run the program from source
improved some misc client service code
deleted some old unused code
misc cleanup
updated to ffmpeg 4.1 on windows release
updated to sqlite 3.26 on windows release
updated to wxpython 4.0.4 on all platforms

version 337

fixed another couple of unicode encoding problems with the logging and profiling code
the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding
fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video
fixed some 'max size' download file size testing
'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain: somesite.com') they are waiting on (you might need to hover over to see the tooltip for this)
the downloader easy-importer lain image is now clickable to launch a file selection dialog
if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you
duplicate content merging now applies to pending as well as current tags
the duplicate filter now counts pending tags when saying which file has more tags
advanced content updates now _copy_ both current and pending tags. the other actions now state what they do more clearly
stopped printing long server error text on 304 and 404, where actual response content is uninteresting
removed wx import that accidentally came into server space due to 2to3 check
improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds
fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else
improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements
did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations
reduced db disk cache aggression significantly and added more memory maintenance to the cache population process
fixed an issue where subscriptions were not promptly responding to shutdown events
fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events
added a 'pubsub report mode' debug mode for simpler pubsub review
the db is now less redundantly spammy on certain behind the scenes update notifications
wrote a first version of the client api manager and permissions handling objects
misc fixes, cleaned some shutdown code

version 336

fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)
fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases
fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)
fixed a str vs bytes issue when loading the filename tagging panel
fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)
improved some json dump deserialisation code
fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel
improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data
fixed some hex character processing for system preds
added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)
the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation
made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update
the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system
simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method
wrote some new help.txt in the db dir about hanging startups

version 335

important:
hydrus now runs completely and exclusively on python 3!
for users who are updating, the client has special install instructions for just this week:
if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)
if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)
if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications!
if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out
please check the release post for more detailed instructions here
.
semi-important:
the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from ???? ????? lads appreciated
I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller)
although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source
I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries
.
misc bug fixes:
fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class
removed some popup flicker related to long error messages
fixed some parsing testing ui error handling
cleared up some bad text ctrl event handling that could sometimes cause a recursive loop
listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion
cleaned up some bad external process calling code and improved how child process environment is set up
finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this
.
boring details:
ran 2to3 to auto-convert what could be done
updated environment to python 3
went over a whole ton of unicode encoding/decoding manually to update it to python 3
removed all the old tobytestring/tounicode calls in favour of new python 3 handling
fixed all the file io to do bytes/str as appropriate
corrected a bunch of / vs // int/float stuff
fixed up twisted, which has some str/bytes stuff going on
fixed all the listctrls to deal with column sorting None values amongst ints/strs
fixed png export/import, which had some fun bytes/bytearray/int/str issues
updated the swf header parsing code to py3 (more str/bytes stuff)
misc float/int fixes
fixed up some http.cookies handling, which has changed in py3
improved some ancient loopback connection code that was only really checking to see if ports were in use
cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked
numerous misc other refactoring and so on
updated the new network engine to now decode non-utf-8 responses correctly based on actual response header
removed some old py2 manual http multipart code
removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection
replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess
deleted some more misc old code
tightened up some spammy network error reporting
converted all /r/n to /n in my environment project, ha ha ha
the ui seems to better support rarer unicode characters like ??
updated some of the install/update/backup help for all this, and some misc other stuff as well
fixed misc bugs

version 334

wrote a system:file viewing stats to comprehensively search the new viewing stats--it _should_ also be synced with the exact current values
but for system:everything, inbox, and archive, which remain where they were, system predicates are now sorted alphabetically!
added a _database->file viewing stats_ menu that lets you suspend file view tracking and clear all records permanently
mr. bones now welcomes all users under the help menu
fixed mr. bones's confusion at those who have yet to board the ride
also mr. bones now makes sure to get the latest file viewing stats
moved confirm trash/archive from _options->gui_ to _options->files and trash_
moved a bunch of 'pages' related stuff from _options->gui_ to the new _options->gui pages_
added an option to options->gui pages to change the number of session rolling backups
subscription popups now provide an x/y query progress string in their popup text
the edit subs/sub panels are now a bit shorter by default and the edit sub has its own frame position data, under 'edit_subscription_dialog', and remembers its last size and position by default
fixed an issue where some dupe watcher urls (like url and url#12345) were not being correctly merged on a mouse drag and drop watcher-import
the client will now print up to 512KB of server error info to the log (previously 4KB)
removed the youtube download prototype--if it returns, we'll do a proper youtube-dl solution. as a result, pafy is no longer needed to run the client
network report mode now shows more network error information
gave the 'getting started with subscriptions' help page a complete pass. it now reflects the new system and has up-to-date advice based on my new experience
wrote a 'logins' section to the bottom of the 'getting started with downloading' help page
misc fixes

version 333

added a first version of file viewing statistics! the client db now keeps track of how many times a file is loaded in the preview and full media viewers, and for how long!
you can see the media and preview stats on any single media right-click menu. there are multiple options for how this displays, including hiding it completely, under options->media
viewing stats update as they happen! (although viewtime typically only updates on the end of viewing. I'll likely make this more live, especially if I end up showing this info in the media media viewer)
you can now sort files by total media views/viewtime!
mr. bones's wild ride continues, as well
deleted the old 'file list' way of updating in-ui media objects in favour of a long-planned global media cache. there is now only ever one active copy of any particular media, and all data-level updates need only occur once on that single copy. this saves a bunch of CPU, memory, and overall hassle behind the scenes! various search results/lookups for media already loaded elsewhere now load super fast!
tag siblings refresh is quicker and less memory heavy thanks to this as well
furthermore, the complicated tag changes from tag repository processing and advanced content updates are now reflected immediately in the gui on the job's completion! (as long as you have fewer than 10k files open, ha ha) previously, these required a search refresh to show the results
the file sort choice dropdown on all pages is now sorted alphabetically. it has always been a mess picking what you want from here, so let's see if this helps!
tag and rating sort options are now listed as 'tag:' and 'rating:' respectively
fixed some misc file sort choice code, which was failing to keep certain defaults in certain situations
fixed the tag import options' new 'load from defaults' button to correctly load the tag blacklist
the keyboard icon on the media viewer's top hover window now permits activation of current/default shortcut sets under submenus. it now also omits these entries if no custom shortcut sets exist
cleaned up some of the hover_window-canvas interaction code
fixed some long-time sperg-out buffer-drawing when changing position in a long video
the database->backup actions are now hidden if the current db has non-default file/thumbnail locations. for now, in these cases, only a custom backup is appropriate
fixed some ancient repository admin code that fetches summary account info given an account key
the filename tagging dialog now has a much shorter listctrl by default, so should fit better on smaller monitors
fixed the 'review session cookies' dialog's clear button, which was not deleting sessions after clear. it now also wraps the operation in a yes/no confirmation

version 332

the client serialisation system now supports multiple rolling backups!
client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp!
when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations
gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload
tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch
under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively
attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process
did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system
wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster
export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.
fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work
when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!
the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads
adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well
fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices
fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add
the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it
wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places
improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service
fixed a small typo bug when trying to auto-add url classes and parsers
updated an old deprecated checkboxlist call
clarified the integrity section of 'help my db is broke.txt'
misc fixes

version 331

added a 'do login now' button to the manage logins dialog. it only enables when the selected logins are active and not invalid and so on, and will ok the dialog and queue up some login attempts, which will make report popups as normal
'review session cookies' panels now support drag and drop cookies.txt import! cookies.txt importing will also handle errors a bit better and report total number of added cookies
the 'review session cookies' panel now defaults to not showing sessions with zero cookies. a new checkbox controls this
login scripts can now be rolled into easy import pngs! should work for export and import just like the other objects (although they won't be auto-added based on domain in export dialog)
brushed up some of the 'change login script' code--particularly, it now puts login scripts that have matching domains first in the selection list, for easier selection
after striking a reckless bargain with a daemon from the database-plane, system:num_tags now runs significantly faster and produces accurate tag counts even when searching over multiple tag services that have duplicate tags. if this works out, the immaterial beast promises greater gains for similar jobs with no possibility of anything going wrong
prototyped a new tag cache in the db that affects (and should speed up) many tag fetching routines. let's see how it goes
added complete, global proxy support for the new network engine! there are new options under options->connection (with some explanation text) to handle it. if pysocks is installed, socks4/5 proxies are also available!
updated the e-hentai.org login script to the new one on the github. your existing mappings for e-hentai.org _should_ all be updated right. exhentai.org is likely too difficult to properly support in the current system
the different panels where you enter system predicate information now all run on the new panel sizing system--if you have had problems with these, please let me know how they size now!
added a '4channel thread' url class to support watchers for the new 4channel sfw domain. it works for now, but let's see if their api changes when the split actually happens
the list right-click menu on gallery import and thread watcher panels now has three options to show combined importers' files--presented, new, and all. it also now shows the files (more smoothly) in the same page, clearing any existing highlight.
misc ui improvements
updated 'running from source' in help

version 330

login:
added a proper username/password login script for hentai foundry--double-check your hf filters are set how you want in your profile, and your hydrus should inherit the same rules
fixed the gelbooru login script from last week, which typoed safebooru.com instead of .org
fixed the pixiv login 'link' to correctly say nsfw rather than everything, which wasn't going through last week right
improved the pixiv file page api parser to veto on 'could not access nsfw due to not logged in' status, although in further testing, this state seems to be rarer than previously/completely gone
added login scripts from the github for shimmie, sankaku, and e-hentai--thanks to Cuddlebear and any other users who helped put these together
added safebooru.donmai.us to danbooru login
improved the deviant art file page parser to get the 'full' embedded image link at higher preference than the standard embed, and only get the 'download' button if it looks like an image (hence, deviant art should stop getting 140MB brush zips!)
the manage logins panel now says when a login is expected to expire
the manage logins dialog now has a 'scrub invalidity' button to 'try again' a login that broke due to server error or similar
entering blank/invalid credentials is now permitted in the manage logins panel, and if entered on an 'active' domain, it will additionally deactivate it automatically
the manage logins panel is better at figuring out and updating validity after changes
the 'required cookies' in login scripts and steps now use string match names! hence, dynamically named cookies can now be checked! all existing checks are updated to fixed-string string matches
improved some cookie lookup code
improved some login manager script-updating code
deleted all the old legacy login code
misc login ui cleanup and fixes
.
other:
sped up tag searches in certain situations (usually huge inbox) by using a different optimisation
increased the repository mappings processing chunk size from 1k to 50k, which greatly increases processing in certain situations. let's see how it goes for different users--I may revisit the pipeline here to make it more flexible for faster and slower hard drives
many of the 'select from a list of texts' dialogs--such as when you select a gallery to download from--are now on the new panel system. the list will grow and shrink depending on its length and available screen real estate
.
misc:
extended my new dialog panel code so it can ask a question before an OK happens
fixed an issue with scanning through videos that have non-integer frame-counts due to previous misparsing
fixed a issue where file import objects that have been removed from the list but were still lingering on the list ui were not rendering their (invalid) index correctly
when export folders fail to do their work, the error is now presented in a better way and all export folders are paused
fixed an issue where the export files dialog could not boot if the most previous export phrase was invalid
the duplicate filter page now has a button to more easily edit the default merge options
increased the sibling/parent refresh delay for 1s to 8s
hydrus repository sync fails due to network login issues or manual network user cancel will now be caught properly and a reasonable delay added
additional errors on repository sync will cause a reasonable delay on future work but still elevate the error
converted import folder management ui to the new panel system
refactored import folder ui code to ClientGUIImport.py
converted export folder management ui to the new panel system
refactored export folder ui code to the new ClientGUIExport.py
refactored manual file export ui code to ClientGUIExport.py
deleted some very old imageboard dumping management code
deleted some very old contact management code
did a little prep work for some 'show background image behind thumbs', including the start of a bitmap manager. I'll give it another go later

version 329

login:
the login manager is fully turned on! hentai-foundry click-through and pixiv login now occur fully on the new system
wrote a Deviant Art login script for NSFW downloading--however, it only seems to work on a client that has done some logged-out downloading first (otherwise it thinks you are a robot)
updated the DA file page parser to only NSFW-veto if the user is currently logged out
wrote a danbooru login script for user prefs and special files if you have a gold account
wrote a gelbooru 0.2.x login script for user prefs
pixiv recently(?) allowed non-logged in users to see sfw content, so the login script is updated to reflect this. the login script doesn't detect a failed login any more, so I will revisit this
logging in in the regular order of things now makes a temporary popup message with the overall login status and final result. ~it is cancellable~--and if cancelled, future login attempts will be delayed
logging in in the regular order of things now prints simple started/result lines to the log
deleted old network->login menu and related code such as the custom pixiv login management. gdpr click-through is now under downloaders
subscription login errors will now specify the given login failure reason
subscription login tests will now occur at a better time, guaranteeing the sub will be correctly saved paused if the test fails
login errors will now always specify the domain for which they failed
testing a login script on a fresh edit login script dialog now pre-fills the alphabetically first example domain
the login script test ui now restores its 'run test' button correctly if the test is abandoned early
misc improvements to login error handling and reporting
.
other:
any texts across the program that ellipsize when they are too thin to display what they have will now tooltip their text (this most importantly includes the status on the network job control, which will now display full login problem info)
the copy button on manage tags goes back to copying all if no tags are selected
the remove button on manage tags now removes only selected if some tags are selected. it still removes all if none are selected
the remove button on manage tags is now wrapped in a yes/no dialog (as is hitting the delete key on the list's selection). this can be turned off under the cog button
filename tagging panels now support directory tagging for the last, second last, and third last directories. the related code for handling directory tagging is cleaned up significantly
the export files panel now lets you delete the files from the client after export. this value will be remembered, and if on will prompt a capital letters warning on export, either via the button or the quick-export shortcut
in manage tag parents, where there are multiple parents in a pending action (either by importing via clipboard/file or by putting multiple parents in right-hand box), the action will now be treated as one transaction with one 'enter a reason' confirmation!
in manage tag siblings, when multiple 'better' values are pended in one action via a clipboard/file import, they will now be treated as one transaction with one 'enter a reason' confirmation!
.
misc:
added a new url class that api-links .gifv-style imgur links so they are downloadable like regular imgur single media pages
the pixiv manga page url class now redirects to the new api, so mode=manga pages should now be drag-and-drop importable and generally downloadable if you have any still hanging around in any queues
clients now come with an additional danbooru parser that fetches the webm version of ugoiras
after discovering a pdf that ate indefinite 100% CPU while trying to parse, I have decided to stop pulling num_words for pdfs. it was always a super inaccurate number, so let's wait for a better solution at a later date. hydrus hence no longer requires pypdf2
fixed an issue with monthly bandwidth estimates rolling over to the new year incorrectly
in an attempt to chase down a duplicate files content move/copy bug, the duplicate action content updates got a bit of cleanup work. if you have noticed duplicate actions not copying tags/urls, please let me know the exact process in the ui, including services and merge options, you went through
tag lists should now update their sibling appearance correctly after a tag siblings dialog ok--previously, they were checking for new sibs too early
tag siblings and parents should now refresh their data more efficiently when spammed with new data notifications (this usually happens janitor-side, which approving dozens at once)
copy queries/watcher urls on the download pages' lists' right-click menus no longer double-spaces the copied texts (it just does single spaces)
fixed an issue where certain initialised watchers were erroring out when asked to provide next-check time estimates--in all cases, null timestamps will be dealt with better here
misc tag parents/siblings ui code cleanup
wrote some code to catch and report on an unusual dialog dismissal error

version 328

wrote test ui for edit login script panel
the login system now works and is turned on, although the legacy hardcoded pixiv and hf logins remain in place. it will not do anything very new this week--it is strictly only for advanced users to experiment with for now
cleaned up some messy network code
all subscription and hydrus jobs will no longer wait indefinitely on an invalid login--they will cancel immediately
network jobs will report a bit more info when they are cancelled
subscriptions will now attempt to test login validity before and during file downloads and syncing. if they fail, the sub will pause and stop work and a message will be presented to the user
made a 'thumbnails' options page and moved some things to it
added thumbnail border and margin to that thumbnails page! you can even set 0 border and/or margin and it works
fixed up a heap of bad thumbnail drawing code that didn't work with thicker borders
the tag sibling and parent dialogs now have suggestion buttons in their 'give a reason' dialogs! if your petitions are simple and fit into one of these categories (which is most of them), please use these buttons as they will let janitors (e.g. hydrus dev for the PTR) process them in batches, in fewer clicks
manage tags dialog's checkboxes and advanced buttons are now wrapped into a cog icon! remove/copy/paste buttons are compacted and put on the same row!
manage tags dialog's copy button now only copies selected, not all tags
manage tags dialog now uses the new sizer. some components are smaller by default but will eat up spare pixels better
misc manage tags code cleanup
added 'paged file import queues' to the network->pause menu. this will pause any, hdd, url, simple, gallery, or watcher page from processing its file import queue. it is a bit hacky and will take up to 30s to unpause unless you joggle the respective downloader to wake it up--see how it goes!
added a similar 'gallery searching' to that menu, to pause any url, simple, or gallery page from actioning its gallery-side queue
and 'watcher checking' for watchers
fixed a stupid bug that was causing false-positive and _mostly_ harmless errors for certain pixiv and artstation multi-page downloads
fixed an issue where multi-page data was being mis-metadata'd (mostly, this meant thread watchers were giving the last filename tag to all files, and I think similarly getting the latest source time) due to a mistake in some recent de-duping code
fixed the new pixiv file page api parser to parse source time, which I must have accidentally deleted previously
fixed a no-expand bug in my new sizer when in horizontal orientation
fixed a small bug when making an easy-import downloader png and cancelling the add url class dialog
misc fixes

version 327

login stuff:
finished off some login script data stuff
fleshed out how login credentials and other linked data is stored in the login manager, including script link recovery when the script changes but name does not
improved some initialisation login validation error handling
improved login failure validation error handling
wrote a dialog panel for managing login credentials and reviewing validity and so on
a heap of related session and login tie-in/fix-up work
the login manager will now save changes to the db. it will get the HF and pixiv scripts on db creation/update, and if you have a pixiv login, the login system will pre-fill that info and 'activate' the script (although the login manager will not fire any login scripts yet--if so configured, it'll just delay on a polite error message)
.
other stuff:
with the subscriptions' new more liberal syncing logic, the periodic file limit will now only pop up if the sub does not see any already-seen files
to give more buffer for the new syncing logic, file import caches will now store 250 entries minimum on compaction (was 100 previously)
subscription merging now lets you choose the primary subscription into which the other subs will be merged
cancelling a subscription merge action mid-merge is now safely nullipotent
post urls that use subsidiary page parsers (such as the new pixiv manga parser) will now correctly insert (rather than append) their manga urls into the file import cache
removed a couple of places where urls could accidentally be duplicated in a file import cache
cleaned up some areas where successful file import objects were presumed to have file hashes when they might not (this was causing errors when importing urls that split into multiple url children, like pixiv manga, while also having 'additional tags' set)
updated tag censorship, parents, and siblings dialogs to the new panel system
tag censorship, parents, and siblings panels now use a notebook instead of the layout-borked listbook
tag parents and siblings panels now use the new small-resolution-friendly sizer, are more tight by default, and expand more neatly
refactored a bunch of tag ui code to clientguitags
the client video renderer will now deal with videos with (invalid) duration of 0 more gracefully
finished the 'getting started with downloading' help page, sans the login stuff
bit of other help work

version 326

login:
finished the new login objects. they can deal with multi-step single second-level domain login problems, can pass variables from step to step, and use cookies as success verification
wrote an ton of ui for the new login objects, now under network->downloader definitions and network->logins. it is not 'active' yet, but advanced users are invited to check it out. there is no good test ui yet, which I think I'll have to figure out in the coming weeks
wrote a first attempt at HF and pixiv replacement login scripts--please try importing from defaults on the manage login scripts dialog and look through them to see what I am going for. once the system is flipped on and we are happy these work, I'll remove the old hardcoded legacy login stuff
when a network job that needs a login cannot login, it now waits (rather than bombing out completely), presenting the related error, and checks again every 60 seconds
if a network job thinks it can login but fails to generate a login process, the network engine now catches the error safely and recovers. the job is put on hold as above
.
subs:
the subscription 'have we caught up to where we were before' test is now more complicated--rather than just stopping after five 'already seen' urls are found, it now only stops if at least the _last_ five contiguous urls of the page are already seen. this will catch more late-tagged files that get inserted out of order
fixed the 'get quality info' button on edit sub panel to only get the current selection, not all queries wew
subscriptions can now optionally publish/present their files to a specific label! this is a great way to merge multiple subs to the same final landing page
.
layout:
after a long time thinking about it, wrote a new custom boxsizer that handles resizing multiple expanding items of different reasonable min size by expanding them _beyond minimum size_ by their proportion, rather than forcing them all to have total proportional width/height. I expect to polish this and apply it in multiple locations around the program where tall things were being too tall because something else was forcing it to be (the management panel on the left of most pages was terrible at this, causing a giganto taglist just because the upper panel was tall as well).
changed my custom boxsizer (the box with a bold header) to the new custom boxsizer, so it is all over now--please report any bad layouts you see
in an effort to improve layout, the manage tag parents and siblings panels' preview boxes have shorter minimum height--it will get a bigger layout overhaul soon
.
bigger misc:
thanks to work of user kourraxspam on the discord, fixed the pixiv downloader to use a more stable api and added pixiv tag search
watchers and gallery imports now have a list right-click menu entry to show all selected importers' files in a new page! use this to clear out a bunch of finished queues all at once!
the tag right-click menu now offers 'open new search pages for each in selection' if multiple tags are selected--this will open three search pages each with one tag, as opposed to the original entry, which would only open one page with all three
the edit nested gug panel now uses a checklistbox rather than the menu to select gugs to add, which is more reliable and allows for multiple selections
sped up autocomplete tag fetches' tag sibling integration--irl this may be a reduction in total a/c search time of approx 33%
page parsers will now generate next gallery urls absent any file/post urls if the only type of url they can generate is gallery urls (so a meta-gallery-search like board->threads that only generates subsidiary gallery pages will now work, whereas before it never could because it was missing post urls)
the gallery log now provides a shorthand way to restart and resume failed searches from its right-click menu (if the most recent log entry failed)
'try again (and allow search to continue)' reattempt jobs will now generate next page urls even if no new urls are found (which can happen if a search stopped due to the file limit exactly lining up with the number of files found, for instance, so a reattempt finds nothing new)
gallery downloaders will now specify their 'delay work for a bit' error states in the ui. this usually means 'could not connect', which has a 4-hour timer (I'll prob add a scrub delays button here at some point)
the watcher will now show its 'delay work for a bit' error state in more places in the ui
added a 'media' shortcut 'export_files_quick_auto_export', which will open the export files frame and give you a quick yes/no to confirm you want to export as set. if yes, it will export. then it will close the frame
added a 'show a "N" to short import summaries' option to options->downloading, which will extend the typical 'x/y' status string to 'x/y - zN' for z 'new files' (as opposed to already in db)
improved how the video parser estimates frame rate--it _should_ fix some of those low-framerate, low-framecount slideshow-vids where at current they render everything in a rush and then sit on the last frame for ten secs
.
smaller misc:
network report mode now reports url_to_fetch and parser-to-parse-with info
when the server fails to accept a file upload due to a file parsing issue, it now prints the hash of the file in the error
if the client sees a possible file hash in a server error message from a file upload, ~it will try to show that file in a new page~
fixed an issue where wildcard searches were not finding results if the search text included the normally discarded characters [](){}"'
fixed some domain handling for localhost and other undotted network names
content parsers will now only launch with permissable content types, which for the legacy 'lookup scripts' scripts system means only tags and vetoes, and for the new login system means only temp variables and vetoes
as compaction now happens automatically on sync, removed the 'compact' button from edit subs panel
an unusual network error related to hydrus update files sometimes being cut off mid-stream is now glossed over silently, with the download reattempted after a delay
the initial gui session load now occurs after a 0.25s delay--let's see if it cleans up some initial layout issues some users have had
maybe fixed an odd dictionary-initialisation error related to tag siblings/parents dialog boot
ruggedised against an unusual bandwidth load bug
gave some of the index help a pass
did most of a 'getting started with downloaders' page in the help--I'll finish it next week
updated discord share link to https://discord.gg/3H8UTpb , which should not expire
some listbox add/edit code cleanup
some listctrl delete code cleanup
misc help work
misc cleanup

version 325

added a 'show a popup while working' checkbox to edit subscription panel--be careful with it, I think maybe only turn it off after you are happy everything is set up right and the sub has run once
advanced mode users will see a new 'get quality info' button on the edit subscription panel. this will some ugly+hacky inbox/archived/deleted info on the selected queries to help you figure out if you are only archiving, say, 2% of one query. this is a quickly made but cpu-expensive way of calculating this info. I can obviously expand it in future, so I would appreciate your thoughts
subscription queries now have an optional display name, which has no bearing on their function but if set will appear instead of query text in various presentation contexts (this is useful, for instance, if the downloader query text deals in something unhelpful like integer artist_id)
subscription queries now each have a simple tag import options! this only allows 'additional tags', in case you want to add some simple per-query tags
selecting 'try again' on file imports that previously failed due to 'deleted' will now pop up a little yes/no asking if you would like to first erase these files' previously deleted file record!
the watcher and gallery import panels now have 'retry failed' buttons and right-click menu entries when appropriate
the watcher and gallery import panels will now do some ui update less frequently when they contain a lot of data
fixed the new human-friendly tag sorting code for ungrouped lexicographic sort orders, where it was accidentally grouping by namespace
downloader easy-import pngs can now hold custom header and bandwidth rules metadata! this info, if explicitly present for the appropriate domain, will be added automatically on the export side as you add gugs. it can also be bundled separately after manually typing a domain to add. on the import side, it is now listed as a new type. longer human-friendly descriptions of all bandwidth and header information being bundled will be displayed during the export and import processes, just as an additional check
for advanced users, added 'do not skip downloading because of known urls/hashes' options to downloader file import options. these checkboxes work like the tag import options ones--ignoring known urls and hashes to force downloads. they are advanced and should not be used unless you have a particular problem to fix
improved how the pre-import url/hash checking code is compared for the tag and file import options, particularly on the hash side
for advanced users, added 'associate additional source urls' to downloader file import options, which governs whether a site's given 'source urls' should be added and trusted for downloaded files. turn this off if the site is giving bad source urls
fixed an unusual problem where gallery searches with search terms that included the search separator (like '6+girls skirt', with a separator of '+') were being overzealously de/encoded (to '6+girls+skirt' rather than '6%2bgirls+skirt')
improved how unicode quoted characters in URLs' query parameters, like %E5%B0%BB%E7%A5%9E%E6%A7%98 are auto-converted to something prettier when the user sees them
the client now tests if 'already in db' results are actually backed by the file structure--now, if a the actual file is missing despite the db record, the import will be force-attempted and the file structure hopefully healed
gallery url jobs will no longer spawn new 'next page' urls if the job yielded 0 _new_ (rather than _total_) file urls (so we should have fixed loops fetching the same x 'already in file import cache' results due to the gallery just passing the same results for n+1 page fetches)
in the edit parsing panels, if the example data currently looks like json, new content parsers will spawn with json formulae, otherwise they will get html formulae
fixed an issue with the default twitter tweet parser pulling the wrong month for source time
added a simple 'media load report mode' to the help debug menu to help figure out some PIL/OpenCV load order stuff
the 'missing locations recovery' dialog that spawns on boot if file locations are missing now uses the new listctrl, so is thankfully sortable! it also works better behind the scenes
this dialog now also has an 'add a possibly correct location' button, which will scan the given directory for the correct prefixes and automatically fill in the list for you
fixed some of the new import folder error reporting
misc code cleanup

version 324

downloaders:
after adding some small new parser tools, wrote a new pixiv downloader that should work with their new dynamic gallery's api. it fetches all an artist's work in one page. some existing pixiv download components will be renamed and detached from your existing subs and downloaders. your existing subs may switch over to the correct pixiv downloader automatically, or you may need to manually set them (you'll get a popup to remind you).
wrote a twitter username lookup downloader. it should skip retweets. it is a bit hacky, so it may collapse if they change something small with their internal javascript api. it fetches 19-20 tweets per 'page', so if the account has 20 rts in a row, it'll likely stop searching there. also, afaik, twitter browsing only works back 3200 tweets or so. I recommend proceeding slowly.
added a simple gelbooru 0.1.11 file page parser to the defaults. it won't link to anything by default, but it is there if you want to put together some booru.org stuff
you can now set your default/favourite download source under options->downloading
.
misc:
the 'do idle work on shutdown' system will now only ask/run once per x time units (including if you say no to the ask dialog). x is one day by default, but can be set in 'maintenance and processing'
added 'max jobs' and 'max jobs per domain' to options->connection. defaults remain 15 and 3
the colour selection buttons across the program now have a right-click menu to import/export #FF0000 hex codes from/to the clipboard
tag namespace colours and namespace rendering options are moved from 'colours' and 'tags' options pages to 'tag summaries', which is renamed to 'tag presentation'
the Lain import dropper now supports pngs with single gugs, url classes, or parsers--not just fully packaged downloaders
fixed an issue where trying to remove a selection of files from the duplicate system (through the advanced duplicates menu) would only apply to the first pair of files
improved some error reporting related to too-long filenames on import
improved error handling for the folder-scanning stage in import folders--now, when it runs into an error, it will preserve its details better, notify the user better, and safely auto-pause the import folder
png export auto-filenames will now be sanitized of \, /, :, *-type OS-path-invalid characters as appropriate as the dialog loads
the 'loading subs' popup message should appear more reliably (after 1s delay) if the first subs are big and loading slow
fixed the 'fullscreen switch' hover window button for the duplicate filter
deleted some old hydrus session management code and db table
some other things that I lost track of. I think it was mostly some little dialog fixes :/
.
advanced downloader stuff:
the test panel on pageparser edit panels now has a 'post pre-parsing conversion' notebook page that shows the given example data after the pre-parsing conversion has occurred, including error information if it failed. it has a summary size/guessed type description and copy and refresh buttons.
the 'raw data' copy/fetch/paste buttons and description are moved down to the raw data page
the pageparser now passes up this post-conversion example data to sub-objects, so they now start with the correctly converted example data
the subsidiarypageparser edit panel now also has a notebook page, also with brief description and copy/refresh buttons, that summarises the raw separated data
the subsidiary page parser now passes up the first post to its sub-objects, so they now start with a single post's example data
content parsers can now sort the strings their formulae get back. you can sort strict lexicographic or the new human-friendly sort that does numbers properly, and of course you can go ascending or descending--if you can get the ids of what you want but they are in the wrong order, you can now easily fix it!
some json dict parsing code now iterates through dict keys lexicographically ascending by default. unfortunately, due to how the python json parser I use works, there isn't a way to process dict items in the original order
the json parsing formula now uses a string match when searching for dictionary keys, so you can now match multiple keys here (as in the pixiv illusts|manga fix). existing dictionary key look-ups will be converted to 'fixed' string matches
the json parsing formula can now get the content type 'dictionary keys', which will fetch all the text keys in the dictionary/Object, if the api designer happens to have put useful data in there, wew
formulae now remove newlines from their parsed texts before they are sent to the StringMatch! so, if you are grabbing some multi-line html and want to test for 'Posted: ' somewhere in that mess, it is now easy.

version 323

wrote first version of the new downloader easy-import drop-panel. you drop downloader-encoded pngs on it, and it maybe asks you a question and jumbles its way through auto-importing all the required data to the client
extended this file import to do some cleverer 'example url merging' when parsers are otherwise dupes, rather than spamming similar dupes on import
wrote first version of the new downloader export panel. it takes gugs, url classes and parsers, and predicts sensible sub-objects to include to make functional downloaders, and bundles it into one png
fleshed out help for the new easy import/export system
the client now slows down gallery and watcher processing when the network engine is under heavy load, aiming for no more than 50 jobs in system at once. the solution is a bit hacky for now, but it should alleviate the deadlock issue when there are ~180+ simultaneous gallery/watcher network jobs pending
the multi-watcher panel's list of watchers now supports right-click menu to copy/open urls and pause/play files/checking
the multi-downloader panel's list of downloaders now supports right-click menu to copy query texts and pause/play files/searching
added a 'derpibooru tag search - no filter' GUG that disables the default derpi no-explicit-files rule
added basic gfycat support to default client--drag and drop any typical video page, and it should import ok
fixed the canvas/hover window tag sorting discrepancy--all tags are now sorted with the same code, and the media view sort order should be the same as your default sort order (although in this case incidence has no effect as there are no tag counts)
rewrote the network job control's cog menu to be a bit more dynamic, and added 'override gallery slot requirements for this job' if appropriate
fixed a stupid typo bug in the shutdown maintenance jobs test code that was causing pending repository work to not report right
fixed gallery searches that include unicode characters that end up in the path of the url (rather than the query parameters)
fixed an issue where highlighting a watcher would unpause its checking
generalised the way the new listctrl class can produce right-click menus
fixed some api link calculation that was over-prescribing api link display pairs (this affected the artstation file page url class by default). these pairs are now also sorted in the links dialog
misc png-export improvements to present better with the new easy import/export stuff
the summary texts in the tag filter panel now ellipsize (...), so if the tag filter is complicated, it won't try to boot a superwide edit panel!
the manage subscriptions panel now correctly initially sorts in a case-insensitive way (previously, it was usually sorting A-Za-z, which is different to regular aA-zZ resorting behaviour, so it always sort-flickered after the first edit)
the status bar has a new segment for reporting when the client is 'busy' with different jobs. for most typical usage, it'll just stay blank. let's see how it goes.
fixed mr. bones's wild review when the client currently has no files
punched up the new file report mode to specify full paths where available
improved some misc downloader code

version 322

wrote gugs help
gave url classes and parsers help a pass
wrote e621 html gallery page example help
wrote gelbooru html file page example help
wrote artstation json file page example help
wrote url class links help
gallery logs for the gallery downloader and url downloader now support 'try again' and 'skip' right-click menu for gallery log entries. the try again allows just the one page or also allowing search to continue) so, if a gallery query fails for some reason, you can now try again/continue where it broke. subs/watcher/simple downloader work on more complicated gallery search logic, so their gallery logs will remain read-only for now
all gallery log buttons now support right-click menu to mass-export urls to png or clipboard. non read-only also support import
fixed an issue with gallery searches that rely on both api url conversions and url class next gallery page urls (I think just artstation and tumblr by default) not generating the next page url correctly
improved some misc gallery url processing logic
fixed some issues with gallery url generators with invalid example urls causing problems opening the edit gug and gallery selector panels
fixed an issue where you could only delete a gug if it was in an ngug, ha ha
thanks to a different submission by prkc on the discord, collections now have a _right-click->set collections as groups of alternates_ duplicate action (note the duplicate menu only appears in advanced mode). the related shortcut action duplicate_media_set_alternate_collections is also added
export phrases now support '\' ('/' in linux) in the path export phrase in order to create folders. you should also be able to do \[series]\ to create optional namespace folders. slashes in tags will still be replaced with _
to stop the client sometimes doing laggy vacuum checks every maintenance cycle, vacuums that cannot occur due to limited disk space now will still count as 'done' for the purposes of rescheduling
added 'file report mode' to the help debug menu. This will spam popups as file and thumbnail actions are requested
tightened up some network job status setting to help us debug the 'there are a ton of jobs in network engine, but the three active on this domain seem stalled' issue
wrote a simple 'review threads' panel under help->debug->data actions->review threads. I knocked it together in about ten minutes, and it's likely unstable as hell, but it's pretty neat!
some instances where many file paths are copied quickly (exporting paths to clipboard and drag and drop) no longer do a safety check for file existence, so should be much faster to go. this particularly reduces startup lag for large file drag and drops!
the 'would you like to do maintenance work in this shutdown?' dialog now lists a summary of what it thinks it'll be working on. I _could_ make this more detailed, so let me know how it works for you
tags with numbers should now sort according to the new improved human sorting method--it now shouldn't matter where the numbers are in the tag--as long as the text-and-number-breaks lines up with another tag, they'll be compared each part in turn correctly
fixed some human sorting code for unusual number characters like ?. they will be treated as text, not a number, for now
misc fixes

version 321

downloader overhaul:
the basic downloader overhaul is complete! at this point, any user can create and share the objects required for a completely new downloader! it is still rough in some places, so a round of EZ-import is coming to make adding new downloaders a single easy drag and drop action
rounded out the ngug (nested gugs, which contain multiple gugs) code
updated the edit gug panel to deal with gugs and ngugs on different notebook pages
added a bunch of logic to this panel and backend data handling to deal with missing gugs in ngugs
if an ngug cannot find a gug by its internal identifier key, it will now attempt to fallback to its simple name, and will silently fail if no gug can be found. all gug tracking now uses this 'key first, name later' id method, so downloaders and subs should generally survive gug renames and same-name overwrites
the gallery selector now works in gugs. it has two 'pages', depending on which gugs are set to 'display', and will note if the chosen gug is cannot be found in the current definitions. the gugs have slightly more specific names ('gelbooru tag search', 'hentai foundry artist lookup', etc...) than before
the gallery selector also puts 'non-functional' gugs (i.e. those with no parsable gallery url class) to a third page
moved the gallery downloader gallery and file pipeline completely over to the new system
the gallery downloader will now bundle nested gugs (like hentai foundry artist, which searches both works and scraps) into a single downloader
moved the subs gallery and file pipeline comppletely over to the new system
the subs gallery sync now handles nested gugs (like hentai foundry artist, which searches both works and scraps) in an interleaved manner and make behind-the-scenes checking decisions in a clearer and more logical way
subs should now make correct 'hit limit' stop reason reports and not generate new gallery pages when the current page has exactly enough results to hit the current file limit
artstation artist lookup is now available as a default downloader
newgrounds artist lookup makes a triumphant return. it works pretty well, given how flash and NG has changed since
derpibooru tag lookup is now available as a default downloader. due to unusual search syntax on derpibooru, please enter queries exactly as you would on derpi, using ',' or ' AND ' to separate tags (such as 'rainbow dash,straight')
pixiv now has multiple artist lookup options--either images, manga, ugoira (doesn't work yet!), or everything
the old downloader code is deleted!
the old manage booru dialog is deleted!
'custom' boorus (i.e. new ones you created or imported to 'manage boorus'), cannot be completely automatically updated to the new system. I've figured out a way to generate new gugs and gallery&post parsers, but they will miss url classes to get working again. your custom-booru subs will notice this and safely pause until the issue is fixed. if you rely on custom boorus, please check the release post for info on this--you might like to put off updating
many misc changes and fixes to gugs and overall gallery url handling pipeline
some misc refactoring and concept-renaming in gallery pipeline r.e. gugs
when the downloader tries to import what looks like a raw html file, its error notes will specify this and suggest a parser may be needed
moved the 'media viewer url display' options panel from the manage url match links dialog to the new network->downloaders->manage downloader and url display
this new dialog also hosts a list for managing which downloaders to show in the first list of the downloader selector
.
misc:
gave the video rendering pipeline communication logic a quick pass, cleaning up a bunch of bad code and other decisions. the video renderer should be quicker to respond to various changes in scanbar position, and incidences of the frame buffer suddenly sperging out (usually inexplicably falling behind the current frame position or deciding to regen for no apparent reason) should be greatly reduced if not completely eliminated
the test that stops repository processing if there is not enough disk space now uses half the current size of client.mappings.db for its estimate (previously 1GB) and also tests temp folder location free space (just as the vacuum test does) and reports this nature of the error along with pausing the repo, stopping further attempts
might have fixed another out-of-order dialog close/open event combination during manage tags close->advanced content update open
fixed gallery queries that include '/' (or some other unusual characters) that end up in the 'path' of the url (as opposed to the query). this fixes 'male/female' on e621, for instance
'advanced mode' users now have a 'nudge subs awake' menu entry below 'manage subs'. this simply wakes the subs daemon (which usually only checks once every four hours), in case any subs are due
'db report mode' now reports every db job as it comes in (formerly, it only reported some optimisation esoterica). this makes it a more lightweight version of 'db profile mode' for several debugging tasks
fixed a tiny issue in fetching the 'how boned am I?' stats when the user had zero inbox/everything count
fixed a typo in the default new url class object that was breaking the edit ui panel
highlighted the quiet filename tagging options on edit import folder panel

version 320

clients should now have objects for all default downloaders. everything should be prepped for the big switchover:
wrote gallery url generators for all the default downloaders and a couple more as well
wrote a gallery parser for deviant art--it also comes with an update to the DA url class because the meta 'next page' link on DA gallery pages is invalid wew!
wrote a gallery parser for hentai foundry, inkbunny, rule34hentai, moebooru (konachan, sakugabooru, yande.re), artstation, newgrounds, and pixiv artist galleries (static html)
added a gallery parser for sankaku
the artstation post url parser no longer fetches cover images
url classes can now support 'default' values for path components and query parameters! so, if your url might be missing a page=1 initialsation value due to user drag-and-drop, you can auto-add it in the normalisation step!
if the entered default does not match the rules of the component or parameter, it will be cleared back to none!
all appropriate default gallery url classes (which is most) now have these default values. all default gallery url classes will be overwritten on db update
three test 'search initialisation' url classes that attempted to fix this problem a different way will be deleted on update, if present
updated some other url classes
when checking source urls during the pre-download import status check, the client will now distrust parsed source urls if the files they seem to refer to also have other urls of the same url class as the file import object being actioned (basically, this is some logic that tries to detect bad source url attribution, where multiple files on a booru (typically including alternate edits) are all source-url'd back to a single original)
gallery page parsing now discounts parsed 'next page' urls that are the same as the page that fetched them (some gallery end-points link themselves as the next page, wew)
json parsing formulae that are set to parse all 'list' items will now also parse all dictionary entries if faced with a dict instead!
added new stop-gap 'stop checking' logic in subscription syncing for certain low-gallery-count edge-cases
fixed an issue where (typically new) subscriptions were bugging out trying to figure a default stop_reason on certain page results
fixed an unusual listctrl delete item index-tracking error that would sometimes cause exceptions on the 'try to link url stuff together' button press and maybe some other places
thanks to a submission from user prkc on the discord, we now have 'import cookies.txt' buttons on the review sessions panels! if you are interested in 'manual' logins through browser-cookie-copying, please give this a go and let me know which kinds of cookies.txt do and do not work, and how your different site cookie-copy-login tests work in hydrus.
the mappings cache tables now have some new indices that speed up certain kinds of tag search significantly. db update will spend a minute or two generating these indices for existing users
advanced mode users will discover a fun new entry on the help menu
the hyperlinks on the media viewer hover window and a couple of other places are now a custom control that uses any custom browser launch path in options->files and trash
fixed an issue where certain canvas edge-case media clearing events could be caught incorrectly by the manage tags dialog and its subsidiary panels
think I fixed an issue where a client left with a dialog open could sometimes run into trouble later trying to show an idle time maintenance modal popup and give a 'C++ assertion IsRunning()' exception and end up locking the client's ui
manage parsers dialog will now autosort after an add event
the gug panels now normalise example urls
improved some misc service error handling
rewrote some url parsing to stop forcing '+'->' ' in our urls' query texts
fixed some bad error handling for matplotlib import
misc fixes

version 319

started the new convert-query-text-to-gallery-urls object. these objects, which I was thinking of calling 'Searchers', will be called the more specific and practical 'Gallery URL Generators', or GUGs for short
the first version of GUGs is done, and I've written some test ui for advanced users under network->downloader definitions->manage gugs. this ui doesn't save anything yet, but lets you mess around with different values. if we don't think of anything else needed in the next week, I will fix this code for v320 and start filling in defaults
watchers now have a checking slot, much like the recent change to galleries and subs. it safely throttles dozens of threads so they don't rudely hammer your (or the destination server's) CPU if they all happen to want to go at once (like just after your computer wakes up). the option is similarly under options->downloading
moved the new gallery delay/token management code to the better-fit bandwidth manager (it was in domain manager before)
the gallery delay/token code now works per-domain!
moved the gallery delay/token checking code into the network job proper, simplifying a bunch of import-level code and making the text display now appear in the network job control. token consumption now occurs after bandwidth (it is now the last hoop to jump through, which reduces the chance of a pileup in unusual situations) I expect to soon add some kind of 'force-go' action to the cog menu
the network engine will now not permit more than three jobs active per domain, and the overall limit has been raised from ten to fifteen
the media right-click menu now supports copying: all of a files recognised urls; all of a files urls; all selected files' urls of a specific url class; and all selected files urls
reworked and harmonised a bunch of urlparsing and generation code--all urls should now appear as full unicode across the program, generally without %20-type encoding characters unless explicitly entered by the user. character encoding now all happens on the backend in requests
non-url-class-matched urls now have their query parameters alphabetised as part of the normalisation process
all urls in the db will have their query params alphabetised on update, and any file relationships merged to the new/existing normalised url
the manage urls dialog will now normalise newly added urls (but should also still permit the removal of non-normalised urls)
reworked how gallery hits update file import object caches, particularly for subscriptions
fixed an issue in subscriptions gallery logging where the gallery log would always state it had found the max number of files and typically redundantly generate an 'ignored' stub--it should now say something like 'found 7 files - saw 5 previously seen urls, so assuming we caught up' as originally intended
simplified some gallery->file import object creation
galleries now compact until 100 entries (was 25)
watchers now gallery-compact after a successful check
watchers now show the 'just added'/'already watching' status for 15s, up from 5s
network report mode now reports three time--once each for job addition, start, and successful completion
fixed an issue with the new 'max width' popup sizing calculation that was sometimes not fitting for new height requirements correctly
fixed an issue with the new url class next page generation code
fixed an issue where TIOs with data regarding since-deleted services were failing to initialise at the ui level
misc status text cleanup

version 318

downloaders:
extended url classes to support 'next gallery page' generation--a fallback that predicts next gallery page url if the parser cannot provide it (as is often the case with APIs and unreliable next-page-url galleries such as gelbooru)
integrated this new next page generation into new gallery processing pipeline
updated gelbooru, tumblr api and artstation gallery api url classes to support the new next gallery page business
fixed the url class for xbooru, which wasn't recognising gallery urls correctly
wrote new gallery parsers for rule34.paheal and mishimmie (which are both shimmie but have slightly different gallery layout). this should finally solve the 'one paheal gallery url is being parsed into the file list per page' problem
'fixed' the tumblr parser to fetch the 1280px url (tumblr killed the raw url trick this past week)
misc text/status fixes
wrote a gallery parser for tumblr that fetches the actual tumblr post urls and hence uses the new tumblr post parser naturally! (tumblr post urls are now more neatly associated as 'known urls' on files!)
note that as the tumblr downloader now produces different kinds of urls, your tumblr subs will hit your periodic limits the next time they run. they will also re-download any 1280px files that are different to the previously fetched raws due to the above raw change (protip: keep your subscription periodic file limits low)
cut the 'periodic limit' subscription warning popup down to a much simpler statement and moved the accompanying help to a new help button on the edit sub panel
multi-gallery pages now have an 'added' column like multi-watchers
the new 'pause' ? and 'stop' ? characters shown in the multi-downloader pages are now customisable under options->downloading (some users had trouble with the unicode)
the watcher now shows the 'stop character' if checking is 404/DEAD
fixed an issue where the new gallery imports on the same multi-page were all sharing the same identifier for their ephemeral 'downloader instance' bandwidth tracker, which meant they were all sharing the same '100rqs per 5mins' etc... rules
the page and subscription downloader 'gallery page delay' is now program-wide (since both these things can run in mass parallel). let's see how it goes, maybe we'll move it to per-site
subscription queries now auto-compact on sync! this means that surplus old urls will be removed from their caches, keeping the whole object lean and quick to load/save
gallery logs now also compact! they will remove anything older than twice the current death velocity, but always keep the newest 25 regardless of age
.
misc:
the top-right hover window will now always appear--previously, it would only pop up if the client had some ratings services, but this window now handles urls
harmonised 'known urls' view/copy menu to a single code location and added sorted url class labels to entries (which should reduce direct-file-url misclicks)
greatly sped up manage tags dialogs initial calculation of possible actions on a tag alteration event, particularly when the dialog holds 10k+ tags
greatly sped up the second half of this process, when the action choice is applied to the manage tag dialog's current media list
the buttons on the manage tags dialog action popup dialog will now only show a max of 25 rows on their tooltips
some larger->smaller selection events on large pages with many tags should be significantly faster
subscription popups should now 'blank' their network job controls when not working (rather than leaving them on the old job, and without flickery-ly removing the job control completely)
the file cache and gallery log summary controls now have ... ellipsized texts to reduce their max width
fixed an issue where larger 'overriding bandwidth' status wait times would sometimes show instead of legit regular smaller bandwidth wait times
removed a now-superfluous layer of buffering in the thumbnail grid drawing pipeline--it seems to have removed some slight lag/flicker
I may have fixed the issue where a handful of thumbs will sometimes remain undrawn after several fast scrolling events
gave the some-linux-flavours infinitely-expanding popup message problem another pass. there _should_ be an explicit reasonable max width on the thing now
added a 'html5lib not found!' notification to the network->downloaders menu if this library is missing (mostly for users running from source)
help->about now states if lz4 is present
gave 'running from source' help page another pass, including info on running a virtual environment
in file lookup scripts, the full file content now supports string transformations--if this is set to occur, the file will be sent as an addition POST parameter and the content-type set to 'application/x-www-form-urlencoded'. this is a temp fix to see if we can get whatanime.ga working, and may see some more work
if the free space on the db dir partition is < 500MB, the program will not boot
if the free space on the db dir partition is < 1GB, the client will not sync repositories
on boot the client can now attempt to auto-heal a missing local_hashes table. it will give an appropriate error message
misc post-importing-cleanup refactoring

version 317

completely overhauled the tag filter panel:
the tag filter panel now has 'whitelist' and 'blacklist' pages beside the old 'advanced' sub-panel. these new simple pages are much more human friendly for common workflows and provide easy-select checkboxes for namespace classes (which are compiled from all the namespaces your parsers can currently do)
the tag filter rule entering workflow now stops you from creating overcomplicated rulesets: when adding a blacklist rule, it will now only add an explicit entry if it is not already blocked by a higher rule (otherwise it will just discard from whitelist, if there)--and when adding a whitelist rule, it will now only add an explicit entry if it is already blocked by a higher blacklist rule (otherwise it will just discard from blacklist, if there)
tag filters now provide more human-friendly summary statements
misc improvements to tag filter ui logic
the various help texts surrounding the tag filter panel all got passes
the tag filter panel now uses text-and-paste controls for mass-adding of tags
namespace checkboxes have been completely removed from the tag import options panel and various other related places. any existing TIO with checked namespaces will be automatically updated to 'get tags' with an appropriate filter. this is an important step in the rewrite--everything is now handled in the new tag filter panel
simplified and sped up the actual tag filtering code
.
numerous multi-importer improvements:
the gallery and watcher page lists will now ~dynamically~ resize in height based on number of entries, from roughly four columns to twenty four. this relayout code somehow seems to work on all platforms
sped up the 'results loading' step of gallery/watcher highlighting immensely--on a typical list of a couple hundred files, it should now be about 50ms total (before, depending on presentation rules, it could be 0.8-3s)
added an additional db-skipping optimisation for calculating presentation status
watcher and gallery highlights will now filter out trash and completely deleted files (the ones that appear with a dark default 'hydrus' icon) on reloads
added two checkboxes to options->downloading for 'if nothing is highlighted when I add a new X, highlight that new X' for watchers and galleries
adding or removing a query or watcher from the new multi-lists should now be reflected in the list ui instantly, rather than after a <=1s delay
added url classes and parsers for imgur single and multiple urls--thanks to the community for providing some examples
added url class and parser for derpibooru single file pages--again thanks to the community. derpibooru hence now supports basic drag and drop import
fixed an issue where the watcher was often still checking despite 404 status
watchers and galleries use a little less CPU to update some of their ui
added simple subsidiary page parsing support to file import objects (previously, this only worked in the gallery log)
.
gave the thumbnail scrolling code a pass--it is now a bit cleverer about drawing and uses a larger number of smaller 'tile' bmps rather than pages
added an 'EXPERIMENTAL' option to options->gui to change the number of thumbnails each scroll tick scrolls. it defaults to 1.0, but you _should_ be able to set 0.5, 0.37, whatever. please report any bugs!
added a thumbnail debug mode to help see the new thumbnail layout boundaries
.
misc:
the max subscription file limits are now 10,000 for users in advanced mode
the default subs initial/periodic limit is now 100/100 (bumped up from 100/50)
the file import dialog now has a little cog icon to change whether human sort is applied on path addition events (for e.g. if you want to add in some date order from an explorer window)
humansort now sorts case-insensitive
by default, unmatched urls will no longer display in the top-right of the media viewer. see how you like this and let me know if you would like an option to put them back
the speed text on the right-side of the network job control now dynamically resizes to its min size, which gives the text on the left side (where it is often cut off, saying 'overriding bandwidth ...') more space when available
I think I fixed an issue where the popup frame could spam-resize in odd ways (such as growing a pixel wider every update tick)
watchers will no longer include the '* ' highlight prefix in subject-based sort comparisons
in prep for an eventual major code refactoring, the thumbnails' underlying media object now stores a faster db-based numeric file identifier
'duplicate' calls on the new listctrl will now insert the dupes in the current correct sort location, rather than tacking them on the end
drag and drop imports to the new listctrl will also now insert like this
caught up edit subscriptions panel to the finalised common listctrl panel code, including the import/export/duplicate buttons
the multiple checkboxlist selection dialog now sorts by label
converted all old checkboxlist dialogs to the new panel system
massively sped up certain kinds of parsing that were wasting time hitting a cache test way too often
fixed an old hash filtering system
moved to a simpler and more stable way of calculating certain text extents
fixed an issue where the include directory (which has the original source, which isn't a big deal but is nice to have) wasn't being correctly copied into the linux build
the os x .tar.gz build now has the include directory
refactored some client tags code around
misc cleanup

version 316

gallery:
gallery url classes can now be linked to parsers!
if parsers are linked, gallery pagewalk can now work on the new parsing system. gallery import pipeline has been significantly updated to reflect this
gallery import objects are now 'multiple' gallery imports, much like the multi-watcher, with each separate query having its own entry in a list (they also run in parallel!)
the multi-gallery list will show file/gallery pause status in slender columns, and will show a 'stop' character when gallery parsing is done
wrote a 'gallery selector' button and added it to the new multi-gallery page, so you can spawn queries for ~different sites~ on the same import page! it always defaults to 'deviant art' for now, but when the next 'searcher' overhaul step is done, this will be customisable
the new page selector and related 'pages' menu is now simpler--with the new selector, you just select 'gallery'
added 'new_gallery_downloader_page' shortcut action to the 'main gui' set to allow quick opening of this new page type
wrote a 'gallery import panel', which reviews a single gallery import stream, and added it to the multi-gallery page to show the current highlighted query
as all gallery imports now run in parallel and work on the new system, the now almost-useless 'cancel' gallery pagewalk button is now removed
with the wider availability of the new gallery log for file count and error reports, shifted around and smoothed out some gallery status text presentation
improved the auto url_class->parser linking 'try to fill in gaps' logic to work with gallery urls (this was surprisingly complicated)
fixed a misc stupid waste of time in auto url_class->parser linking
many misc updates to gallery pipeline
.
subscriptions:
wrote a new gallery pagewalk pipeline for subscriptions, which still does oldest-to-newest url addition and obeys file limits and so on
numerous subscription pipeline and error handling tweaks and improvements, particularly in regards to the new code
subscriptions now have max initial and periodic file limits of 1000. existing subs with >1000 or infinite will be cut to 1000. there is a help button on the edit sub panel to explain why you should do large syncs with the manual downloader and not subs
the ugly and dangerous-if-you-scroll-in-the-wrong-place gallery selector mismash control in the edit subscription panel is now replaced with the new gallery selector button
fixed an issue where the edit subs panels could sometimes say '48 years ago' (i.e. displaying a literal time delta since 0, 1970) on initial timestamps
juggled some 'periodic limit' reporting logic to skip an unusual false positive that affect hentai foundry subs for now and more in future
.
urls:
the url downloader now accepts gallery urls and will receive drag-and-dropped gallery urls. at the moment, it only parses the one page (i.e. it doesn't start a new 'searching' pagewalk) and sends the parsed links to its file queue
.
watcher:
finished 'watcher panel', which reviews a watcher, and added it to the multi-watcher page to show the current highlighted watcher
the single watcher page is completely removed--it is only the multiple watcher now. all singles will be converted to multiples on update
some single-watcher options (like watchers naming their own page tabs and the [404]-style page name prefixes) are removed
multiple watcher panel now lists file/checking pause status and has separate buttons to control these paused statuses
fixed some misc watcher highlight code--highlighted watchers should correctly publish to the page from the start of session load now
improved some 'repage' logic in how highlighted threads get removed
.
misc:
discovered a scroll-setup parameter that stops janky scroll-to-click-focus behaviour on all the new scrolling panels, thank the LORD
improved some 'can't parse' error handling for post url parsing
reworked how all importers present their network jobs to the ui, including fast response if the switch happens during a job
in prep for searcher switchover where all downloader sources will be harmonised into one system, booru identifiers now present in several ui locations as 'name', not 'booru: name'
updated the danbooru parser to deal with the new 'next page' markup they use
wrote a gelbooru gallery parser that works with 0.2.0 and 0.2.5 gelb sites--an ancillary issue where gelb-related downloaders could sometimes not accurately figure out the magic '42' next-page offset is hence now fixed
wrote an e621 gallery page parser
these sites hence now support single-page gallery drag-and-drop
added url classes for artstation gallery url and its api counterpart, but didn't go further yet--we aren't quite there with api pagewalking just yet
updated deviant art gallery url classes
added an e621 'search initialisation' gallery url class to improve some future drag-and-drop stuff
url normalisation no longer cuts off 'www.'-style prefixes
url comparison is more careful to test 'www.'-style prefixes, so a file import cache should recognise that 'http://www.blah.com/blah' is the same as 'https://blah.com/blah'
did a bunch of refactoring to further split up the bloated ClientImporting.py
fixed some misc downloader layout that may have been hiding some texts previously
some multi-watcher and multi-gallery events like add/pause query should be a bit snappier
in the parsing ui, url and title priorities are now 50 by default
prepped a little subscription unit test code for when searcher object is done
misc downloader layout improvements
misc listctrl refactoring

version 315

got started on the big gallery update, but decided not to pull the trigger just yet. I hope to do it next week, switching the whole thing over to a two-object multi-watcher kind of deal
updated to wxPython 4.0.3 for all platforms
cleaned up some menubar replacement code, and the update to the new wxPython should also fix a "event for a menu without associated window" bug some gtk2 users were seeing on quick menubar changes
manage default tag import options panel now has copy/paste buttons that work on the listctrl
added some 'paste tag import options' safety code to make sure no one accidentally pastes a subscription or something in there, wew
added default checker options for subscriptions to options->downloading
unified how checker options are edited from their button, much like how file and tag import options work. it also has a summary tooltip on the button
the checker options under options->downloading are now these slimmer buttons
in the manual import dialog (which pops up when you drop a folder/files on the client), the files will now be added in 'human friendly' number sorting, so files of the sort 'Favourites - 10.jpg' will sort [10, 11, ..., 99, 100] rather than the purely lexicographic [10, 100, 11, ..., 99]
gave the migrate database dialog a pass--a bunch of misc presentation changes and a general simplification of workflow, now based more on just increase/decrease location weight
a bunch of texts on page management (left-hand) panels that share horizontal space with buttons should now ellipsize ("downlo...") when they get too long for the width instead of drawing in an ugly way over the buttonspace
moved the manage import folders dialog to the new listctrl and added a 'paused' and better 'check period' column
if a user tries to run a 'paused' import folder specifically from the menu, the import folder will now unpause (I will probably remove this old paused variable in the future--it isn't of much use any more)
tightened up some repository reset code that wasn't deleting all service tables and hence recovering from some service id malformation errors correctly
wrote a 'clear orphan tables' db maintenance routine that kills some spare tables some users who have previously deleted/reset repositories may have floating around
fixed an issue with parsing folders after hitting cancel button on the import files pre-dialog
if watchers encounter non-404 network errors during check, they should now just delay checking for four hours (before, they were also pausing checking completely)
if watchers are in 'delay' mode, they'll also not work on files.
file and gallery downloads that hit a 403 (Forbidden) will now present a simpler error status, like they do for 404
the new post downloader will no longer fail if one of the parsed source urls is not a url. the borked string will also not be associated as a url
regular gallery downloads now override bandwidth for the file download step, which is almost always the second half of a pair of post_url/file downloads, just to keep things in sync in edge cases
cleaned up some timestamp generation and 'overriding in x seconds' strings to be more human friendly
improved some serverside file parse error handling to propagate the actual error description up to the client a bit better
fixed typo causing incorrect num_ignored count in file import status button right-click menu
parseexceptions will now present more data about which page and content parser caused the problem. I am not totally happy about how this solution works and may revisit it
the lz4 import error catching is now more broad to catch some odd problem I discovered in new Linux build environment
the moebooru parser now fetches the original png of an image, if available
added a new tumblr parser that also gets post tags--it _shouldn't_ be the default
the new login pipeline now kicks in for the legacy logins--pixiv and hentai foundry--on a per-url basis, so adding pixiv/hf urls to the url downloader will trigger a login even if needed (previously, this was tied to legacy gallery initialisation, which explains some pixiv 'missing' login stuff some users and I were having trouble with)
if the legacy login system fails in the new pipeline, it now sets a flag and won't try again that client boot
the old 'default tag import options' panel is now completely removed from options->importing. please check 'network->downloaders->manage default tag import options' for the new url-based settings
misc fixes

version 314

tag import options can now be set to 'default', meaning 'use whatever the default is at the time of import', which will be an easier way of managing TIOs for many subs that you'd prefer all share the same TIO settings anyway
updated tag import options ui to enable this default setting where appropriate
updated the newer import pipeline to work with 'default'-set tag import options
new downloaders, subscriptions, watchers, and multi-watchers now start with 'default' tag import options
deleted the old default tag import options management code and put some text up on options->importing making notice about the impending shift. a popup message will also say this on update
tag import options buttons now have a right-click menu with copy/paste/default options for quick assignment and duplication!
added 'overwrite tag import options' to manage subscriptions--once you are comfortable with the new 'default' TIO mode, and after some small tests, you might want to switch all your subs over to 'default'
the 'urls' downloader now has a tag import options--it initialises as 'default'
added furry.booru.org to gelbooru 0.2.0 parser
wrote a hentai foundry file page parser
wrote a moebooru file page parser (this works for konachan, yande.re, and sakugabooru)
wrote a shimmie parser (this works for rule34.paheal, rule34hentai, and mishimmie)
wrote a newgrounds parser
integrated the user-created sankaku parser
wrote a tumblr parser that handles photo, photoset, and video posts, auto-converts to 'raw' urls for those post-2012 urls that can handle it, figures out a creator tag (reverting to the reblog root if it is a reblog post!), and cooks you breakfast
(hence all these above sites now support drag and drop!)
rolled out some new tumblr url classes to handle all this.
added sakugabooru url classes
fixed an issue where url classes were not normalising api urls in all cases, meaning some url classes would not api-link correctly in 'manage url class links' panel
fixed an issue with deviant art legacy gallery parser pulling some funky 'creator:' tags
some misc new downloader error handling improvements
the watcher now uses the new gallery object to parse and generate file import objects
the downloaders with gallery logs should now report non-success gallery fetches, along with error tracebacks (this will include some no-worry 404s the legacy downloader sometimes uses to terminate searches)
added image and thumbnail cache timeout time delta buttons to options->speed and memory
added a 'show the D on short file import summaries' checkbox to options->downloading--it defaults to off
the 'I' on short file import summaries is now 'Ig' to clear up 1/I confusion
added 'copy queries' to the edit subscription panel, which lets you copy all the selected queries' search texts to clipboard, newline separated
added a checkbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on
fixed file import status button right-click, which I messed up somehow last week with the 'retry ignored' add
shook up and collapsed the network menu into neater categories
tightened-up the rarely used pre-parsing conversion panel on the edit page parser panel to just a button with a bit of explaining text
if database errors include the word 'malformed', the client now throws a little extra error text pointing people to the help.txt in the db dir
cleared out some legacy download code
cleared out legacy hard drive import error handling, moving it all to the new file import object
misc refactoring and cleanup

version 313

fleshed out the new gallery log and its constituent log entry objects
added gallery logs to gallery downloaders, subscriptions, url downloaders, simple downloaders and watchers
added very simple gallery log reporting to these downloaders
added first, read-only version of gallery log ui to these downloaders
fleshed out some new gallery/file-object pipeline stuff
wrote a simple danbooru gallery page parser and added it on update. it doesn't do anything yet, but if you are into the new parsing system, please check it out as an example
the url downloader now has a full file import status control with status text
fixed a url count issue on completely fresh gallery downloads that was stopping gallery searches one file (like 199 vs 200) before the file limit
the pixiv downloader now fetches 'type=all' gallery pages, which include specifically manga file pages (as opposed to merely multi-file 'illustrations')
added 'retry ignored' to the file import status button's right-click menu
fixed the deviant art url class and parser to use the new file page format. also added an '(old format)' class to match the old way for legacy purposes (this legacy class also uses an api conversion to connect to the new parser--we'll also figure out a way to convert all these over at the db level en masse later!)
updated some similar deviant art gallery stuff as well
tag import options now has a tag filter to go along with the 'get all tags' checkbox! ('get all tags' is now renamed to 'get all' as a result). this filter lets you make more complicated tag filtering decisions like 'get all tags except "species:" tags'.
the new 'only get tags if they already exist' checkbox now also has a filter, if you want to only apply this test to a subset of tags (like the unwashed mess of unnamespaced tags many boorus and sites provide)
generalised a 'tag filter' button class to make it simpler to edit tag filters across the program, and cleaned up some related status code
fixed a problem with deriving tag import options for specific url classes when that url class was part of an api-url-class chain
if the domain manager cannot now find a url match for a pending download, it now assigns the file post default tag import options to that import
added a new 'duplicates' options page that has a hacky way to edit the weighted scores used to determine which of the pair of files to present file in the duplicates filter
unifed how some file import status generation works, adding a new 'simple status' string to briefly summarise progress in multi-watcher and edit subscriptions columns
cleared out some old redundant status caching in the urls downloader
simplified how almost all timestamp strings are generated
simplified how time delta strings are generated
brushed up some simple common ways to present timestamps as 'human pretty' strings
all places where timestamps would be presented as a mix of '5 days ago' and complete datetime strings will now present as '5 days ago' unless you set the new options->gui 'always show iso' checkbox. going back to simple to clear up confusion in workflow and code. I may revisit this, as turning on ISO mode now spams it all over the place
cleaned up the 'looks like the computer just woke from sleep' check and reduced its grace period to fifteen seconds. foreground daemons (like the subscription daemon) and the network engine will now also obey it
added a 'simulate wake from sleep' debug action to better test the sleep-wake detection code
improved my custom statictext class to auto-wrap text without flickering
used this new autowrapping to improve wrapping and layout of popup message texts
replaced all other st wrapping with this new code
wrote a little helper function to better dedupe lists in future
did a bunch of refactoring to neaten some long common func names
deleted some old unused code

version 312

converted much of the increasingly complicated tag import options to a new sub-object that simplifies a lot of code and makes things easier to serialise and update in future
tag import options now allows you to set whether tags should be applied to new files/already in inbox/already in archive, much like the file import options' 'presentation' checkboxes
tag import options now allows you to set whether tags should be filtered to only those that already have a non-zero current count on that tag service (i.e. only tags that 'already exist')
tag import options now has two 'fetch if already in db' checkboxes--for url and hash matches separately (the hash stuff is advanced, but this new distinction will be of increasing use in the future)
tag import options now applies sibling and parent collapse/expansion before tag filtering, which will improve filtering accuracy (so if you only want creator tags, and a sibling would convert an unnamespaced tag up to a creator, you will now get it)
the old 'all namespaces' checkbox is now removed from some 'defaults' areas, and any default tag import options that had it checked will instead get 'get all' checked as they update
caught up the ui and importer code to deal with these tag import option changes
improved how some 'should download metadata/file' pre-import checking works
moved all complicated 'let's derive some specific tag import options from these defaults' code to the tag import options object itself
wrote some decent unit tests for tag import options
wrote a parser for deviant art. it has source time now, and falls back to the embedded image if the artist has disabled high-res downloading. if it finds a mature content click-through (due to not being logged in), it will now veto and set 'ignored' status (we will revisit this and get high quality nsfw from DA when the login manager works.)
if a check timings object (like for a subscription or watcher) has a 'static' check interval, it will now apply that period to the 'last next check time', so if you set it to check every seven days, starting on Wednesday night, it will now repeatedly check on Wed night, not creep forward a few minutes/hours every time due to applying time to the 'last check completed time'. if you were hit by this, hit 'check now' to reset your next check time to now
the multiple watcher now sorts by status by default, and blank status now sorts below DEAD and the others, so you should get a neat subject-alphabetical sort grouped by interesting-status-first now right from the start
added 'clear all multiwatcher highlights' to 'pages' menu
fixed a typo bug in the new multiple watcher options-setting buttons
added 'retry ignored' buttons to edit subscription/subscriptions panels, so you can retry pixiv manga pages en masse
added 'always show iso time' checkbox to options->gui, which will stop replacing some recent timestamps with '5 minutes ago'
fixed an index-selection issue with compound formulae in the new parsing system
fixed a file progress count status error in subscriptions that was reducing progress rather than increasing range when the post urls created new urls
improved error handling when a file import object's index can't be figured out in the file import list
to clear up confusion, the crash recovery dialog now puts the name of the default session it would like to try loading on its ok button
the new listctrl class will now always sort strings in a case-insensitive way
wrote a simple 'fetch a url' debug routine for the help->debug menu that will help better diagnose various parse and login issues in future
fixed an issue where the autocomplete dropdown float window could sometimes get stuck in 'show float' mode when it spawned a new window while having focus (usually due to activating/right-clicking a tag in the list and hitting 'show in new page'). any other instances of the dropdown getting stuck on should now also be fixable/fixed with a simple page change
improved how some checkbox menu data is handled
started work on a gallery log, which will record and action gallery urls in the new system much like the file import status area
significant refactoring of file import objects--there are now 'file seeds' and 'gallery seeds'
added an interesting new 'alterate' duplicate example to duplicates help
brushed off and added some more examples to duplicates help, thanks to users for the contributions
misc refactoring

version 311

wrote a new parser that muddles its way through pixiv's new dynamic javascript layout. it seems to get everything working again. it gets tags in kanji, although the unnamespaced pixiv tags remain low quality, and you may wish to just not parse them at all anyway
fixed some misc parser text handling, unicode conversion etc...
the new pixiv parser has a 'page' tag stub that should inform tag import options in the old downloader
the multiple watcher now remembers the highlighted watcher through a session restart
the multiple watcher now shows the highlighted watcher's url up top
the multiple watcher now has checker, file import, and tag import options, which it will assign to all new watchers it creates
the multiple watcher now has a 'set options to watchers' button that will force-set the current options to all the selected watchers
the multiple watcher now has an 'added' column with watcher creation time listed. storing this creation time is new, so any existing watchers will get a new creation time of their next load time, but it is remembered henceforth. the listctrl here is now pretty crushed for width, so maybe we'll rejigger some stuff here
watchers added to a multiple watcher will now have a status of 'just added' for five seconds
watchers that are added to a multiple watcher that is already watching them will now have the status of 'already watching' for five seconds
the multiple watcher list now has a much taller minimum height--layout here is another work in progress
fixed the inkbunny parser (and a related tweak to the inkbunny url class)--it now uses the new 'multiple-file-per-post' import object generation to actually walk through the pages of the mini-gallery (which for inkbunny have -p2- suffixes on the url) to fetch only the correct files and url-associate them neatly
tag import options now has a 'get all tags' checkbox, which can override the normal namespace checkboxes. it gets all tags, even those with namespaces not listed, which happens for several reasons in the new download system. (eventually, the namespace list may be replaced with a slightly different system)
watcher tag import options no longer list 'filename' under their namespace checkboxes--they just have this 'get all tags', which works for everything (so watching yiff.party pages should now get tags)
simplified and sped up similar files search at the db level
sped up some ratings search code
generalised some common file search optimisations, meaning they now apply in more situations and can take advantage of some other speed-ups:
similar files system predicate is now faster
inclusive ratings searches are now faster
duplicate relationship count searches with non-zero-inclusive count are now faster
removed some clumsy old ratings search optimisation code
exporting serialised objects as pngs is a bit easier--now, it displays current export path better, will remember the last export location used, and for single png exports will pre-fill the filename and 'title' value with a reasonable default
the content parser, page parser, and url class listctrls now accept serialised png files when drag and dropped!
the simple downloader should recover and continue better from malformed urls during a page parse
the url downloader should now recover better from various situations where it cannot not derive some tag import options (including urls with a 'file' url class, such as 4ch/8ch direct file links)
parse test results will now state the priority value of urls
gave the 'updating' section of help a pass and wrote a little more on how to do a big-version-gap update
when a new multi-file import object inserts its child file import objects while being looked at in the ui, the listctrl should now correctly refresh the displayed indices
subscriptions will now wait up to 90s for bandwidth (was 30s before, I think) before quitting, which should avoid a few more early-quit events
cleaned up some server decompression bomb testing
users with admin-level accounts can now upload decompression bombs to file repositories, better options on this will be avaliable in future
the manage urls dialog will now OK on the same 'manage_file_urls' shortcut action that can open it (like manage tags and ratings already do)
fixed the string converter for new file lookup parsing scripts
started work on some in-the-background mass file reparsing, but I want to get some nicer ui going before I pull the trigger on any of it
file reparsing now repopulates the table for md5, sha1, and sha512 hashes if they are missing
improved some ffmpeg error parsing
moved from basic list to a pop-faster collections.deque for importable path parsing and duplicate search branch regen
added a BUGFIX option to options->gui that forces minimum width for popup messages in the continuing attempt to deal with some funny fit/layout calculation in certain Linux WMs
fixed how some 'unrepairable db' error messages are displayed in Linux systems
cleaned up a ton of old tuple-stripping code from the db
updated to new sqlite for windows build
misc improvements

version 310

updated the inkbunny file page url class to acknowledge that inbunny pages can have multiple files
updated the inkbunny file page parser to handle multiple file urls (although they may be out of order and possibly sometimes include the artist profile image--this was not super easy)
added a parser for twitter tweets (only images supported atm, but it can handle multiple!) (hence tweet drag and drop now works!)
updated the artstation file page url class to redirect to a new api url class
wrote an artstation file page parser that also handles multiple file urls
updated/added pixiv file page, manga page, and mange_big url classes
updated pixiv file page parsers to be ok with manga links
wrote parsers for pixiv manga and manga_big pages to fetch manga files (with page tags)!
file import objects can now create semi-duplicate children for multi-file post urls and insert them just after themselves in the file import queue.
file import objects can now receive and remember referral urls. this referral url is associated with the file if appropriate. the watcher and simple downloader now uses this in addition to the multi-file post system
jumbled around some parameters and merged the two new file import url commands (import 'file' vs import 'post') into one single simple 'work on this url, thanks' call that is now used across the program
the parsing system's 'content parser' no longer fetches file urls and post urls, but 'download urls' and 'source urls'. this helps some pipeline logic and also lets post urls be download urls
when file import objects parse post urls as the urls to download, it now creates 1-n new import objects, just like if multiple file urls.
improved some file import object file association code
the new parsing system will de-dupe parsed urls
refactored the 'seed' code, which handles all basic file import objects, to the new ClientImportSeeds.py
added a new string transformation type, 'integer addition', for shifting page number tags up and down
fixed thumbnail generation for some videos that failed to do the new x%-in generation--it reverts more reliably just to the old frame 0 method
file reparsing popup now has a stop button
fixed an issue where extremely thin or wide (ratio > 200:1) images would not generate a full-size or resized thumbnail
the file reparsing/re-thumbnailing now reports errors better (including with full path) and does not abandon the larger job as it works
misc thumbnail generation code improvements
improved some thumbnail and file regeneration/moving code when the existing file has read-only status
the multiple watcher now has a 'check now' button
added a checkbox to options->gui that will put new notebook page tabs on the left
for all file download network jobs working in the new download system, the file import options for min size, max size, and max size (gifs) are now applied _during the download_! if the server tells the client the exact file size in the response headers, it will test max and min size before the content is actually downloaded--otherwise, it will test the max size as it downloads. if the server clearly says the file is a gif, the max gif size rules will also be tested in the same way
cleaned up some bandwidth announcement code--now, if bandwidth is due in less time than override time, that will now correctly be the status text
the bandwidth status no longer says 'in in' typo
fixed up some tag repair code from last week
the 'print garbage' debug function now dumps a whole bunch more data to the log
the thumbnail cache should now be a bit more stoic about missing repository thumbnails--it should now just present the hydrus default backup without error popup spam
the repository thumbnail sync will now get as thumbs in blocks as high as 10k at a time, rather than the old 100
hydrus network requests no longer generate web domain network contexts (and so won't have a default one-request-per-second bandwidth limit and should stream through thumbnails a bit faster)
hydrus network services are now willing to wait longer for bandwidth, so big thumbnail queues should keep working even if other bottlenecks pause them for a bit
hydrus network services will no longer sometimes have double-sync popups if synced from the advanced 'sync now' button in review services
changed the default global 'stop-accidents' bandwidth rule of 120rqs per minute to 512MB per minute. this only affects new users, but users trying to sync to large file repos might like to make a similar change manually
doing giant full file delete (i.e. purge from trash) jobs should now be a bit gentler on the gui
improved how the client deletes paths, clarifying in the code when and when not to allow recycle (usually disabled for thumb disposal)
switched the hacky text widgets on the popup system to a newer object. seems to still render ok, so lets see if it fixes some unusual layout issues some users have seen
if the temp folder cannot be created on boot, the client will continue anyway
fixed some url-domain text handling in db storage that was also breaking v309 update for some users
fixed some additional domain generation error handling at the db level
the list of url classes in the system:url panel is now the list of all url classes that are considered associable (before, it was file and post urls)
if a url class now api-links to itself or otherwise forms a loop with n other api url classes, the client will now throw an error (rather than lock up in an infinite loop!)
in the parsing ui, tag parse test results are now cleaned before being displayed
fixed misc url matching error reporting bug
when consulting the current file limit, the gallery page downloader will now try, when it has that number, to consult the total number of urls found it the current search (old behaviour is to only consult the number of _new_ urls, which lead to some bad edge-case workflows)
misc refactoring

version 309

wrote a fix for the tumblr GDPR issue under _network->DEBUG: misc->do tumblr GDPR click-through_. you will also get a popup about this on update
the tumblr downloader will try to detect the GDPR problem and present a similar popup guiding you to the GDPR click-through solution
the client and server now generate video (but not gif yet) thumbnails 35% in by default. the client can now change this percentage value under options->media. this was highly requested and was being put off for a longer rewrite, but I figured out a simple way to hack it in. please let me know if you get failures
on adding a parent, all files with the child tag will now also get all applicable grandparents (with no limit on recursive generations and dealing with accidental loops)
on adding a sibling, all files with any of the siblings will now also get all applicable parents and grandparents for the whole group. a maintenance call to retroactively fill in the sibling/parent gaps that are now filled will also come soon
this logic still does not apply in cross-service situations, which _will_ likely have to wait for a big data/gui overhaul and us figuring out what we actually want here
added a simple pause/play button to the multiple watcher
if the multiple watcher is set to catch watchable url drag and drop events and the current page is a multiple watcher, this current page will catch those new urls (as opposed to the _leftmost_ multiple watcher)
improved some thread unpause logic which was failing to lock pause during 404 status
the multiple watcher should now ignore case when it sorts by subject
added url class and file page parser for inkbunny (so this site is now supported in drag and drop!). it fetches creator tag, some artist-made unnamespaced tags, source time, and md5
added file page parser for gelbooru 0.2.0, which by default works for rule34.xxx, tbib, xbooru but certainly should work for a bunch of others. it fetches source time and source url
html formula parsing rules can now additionally test the tag 'string' using a standard StringMatch object. this greatly helps to parse otherwise indistinguishable 'a' tags that have string 'Original image' and so on
the 'have I seen this url's file before?' pre-import test is now much more strict and will cause fewer accidental false-positive 'already in db'/'deleted' results:
the url pre-import test now does not trust source urls if they do not have a url class
the url pre-import test now no longer trusts urls that are supposed to only be mapped to one file but are actually mapped to multiple
this url pre-import test now treats url-classless original post urls and intended file urls with a special level of trust
urls are now stored in the db in a more powerful and in-future easily searchable way--your db will take a moment to convert to the new format on update
did some prep work for multi-file post urls (like pixiv manga) but did not have time to finish it
the filename tagging options panel (in the 'add tags based on filename' of file import dialog and import folder dialog) now updates its tags/list 0.5s after the last change event, which means typing on a giant list will not cause megalag
improved stability of some client-screen coordinate conversion
misc bmp handling stability improvements
improved some parsing ui stability when example data gets set after the dialog is closed
improved some misc dialog close stability
converted all but one final ui update timer to the new job scheduling system
there are still problems with linux stability--I will continue to work on it
an ugly (but basically harmless) shutdown exception sometimes caused by Animations being a bit slow on deleting their underlying bmps _should_ be fixed
the export files dialog now generates its paths in sort order, meaning (1), (2) de-dupe filename suffixes should now be generated nicely in order
the network domain manager should now always chase API URL links to get the right parser
made some 'the db is broke, let's try to fix it' tag recovery code more forgiving
misc improvements to some media indexing backend, which may fix some unusual session ghost files
fixed the 'sure it is ok to close this importing page' dialog to also veto on a 'cancel' event, rather than just a 'no'
added a guide to database_migration.html on how to move the db from just an HDD to straddle both an SSD and HDD.
cleaned up the help->debug menu a bunch
added run fast/slow memory maintenance calls to help->debug->data actions
misc cleanup

version 308

the multiple watcher will now discard new urls if it is already watching them
the multiple watcher will list x/y progress as just 'x' if x==y (making it easier to scan the list)
the multiple watcher now lists a couple of 'total' summary lines on its ui--the top lists total number of watchers and queue progress, the bottom lists the usual '23 successful, 3 deleted' line, but summed for all watchers
the multiple watcher will now warn you if you try to remove the highlit, alive or un-caught-up watchers
the multiple watcher will now resort if the thread subject (or rather, any data in the current sort column) changes (which usually happens right after it is added, when you see it change from from 'unknown subject' to 'mlp is kino')
fixed an issue where multiple watchers were not unscheduling down their update job correctly on page close
the booru selector in the edit subscription panel should now be in the tab traversal order for keyboard/automated focusing tasks
the boorus in that selector are now alphabetised
tag import options namespaces are now alphabetised
removed/renamed pretty much all references to 'thread' in the watcher code and ui presentation, since it can now do a bunch of other stuff. it is now just the 'watcher' and the 'multiple watcher'
deleted a bunch of old static thread watcher and page of images code from the old downloading system
added an experimental 'compact' button to advanced mode users' manage subscriptions panels. this removes urls from the selected subscriptions' caches that are no longer useful, keeping their load/save snappy. this is still in testing--be careful with it!
the hydrus splash screen now has a bare frame caption and will appear in the taskbar--which helps with some alt-tab and 'where the hell did it go?' stuff if you need to enter a password
wrote five 'reasonable defaults' buttons for the 'check timings' options panel for quick entry for different thread/subscription scenarios.
added a checkbox to this panel that will swap the reactive options with a simpler single checkbox
also clarified/fleshed out the help button on this panel
fixed an important source of program instability related to page alive/dead status checking that was inadvertantly talking subtly to the main gui frame even on non ui threads
improved how some 'page is closed but not destroyed' test logic for pages inside a closed-but-not-destroyed notebook
fixed another small place where the db was talking to the main gui object about status bar updates in a potentially unstable way
fixed another small place where the foreground daemons were talking to the main gui frame in a trivial but potentially unstable way
played around with some taglist sizer and layout settings
the gallery and simple download pages are now a little shorter--the pause and cancel buttons are now just to the right of the status texts, rather than on their own row beneath the network job controls.
the various bandwidth-overriding network jobs in the download system--like gallery page downloading--now wait 30s before overriding their bandwidth. hence these jobs will now obey the usual bandwidth rules up to a point
the simple downloader also obeys the usual bandwidth rules for 30s but no longer has a static wait, so it can run much faster in certain situations
network jobs that will override bandwidth in the future will now report that countdown in their status texts
fixed a bug in the old booru code that meant some boorus were superfluously requesting the 0th indexed page of a gallery more frequently than needed in order to reestablish a 'page size' cache. this value is now cached globally and will be replaced by a completely different system in the new gallery downloader
added a decent tooltip to the 'gallery fixed delay' widgets in the options->downloading panel
the autocomplete input should clear itself after a 'broadcast' event a bit quicker and stop some dupe inputs in certain edge cases
the tumblr url class now recognises that tumblr posts can have multiple files, which helps some source url lookup logic
added a url class for artstation file pages
the primary file import url (the one listed in the file import list) will now correctly not associate with the resulting file if its url class is so set
all the import objects now have much lower idle CPU time and thread needs and start in slightly offset times, smoothing out the thread count spikes
all the import objects will now respond quickly to changes to the underlying file import cache (like right-click->try again events)
the new job scheduling system now uses two queues--fast and slow, in order to reduce some resort/insert overhead
a couple more improvements to the new job scheduling system to smooth out spikes
if the temporary path override does not exist, the client will now compain with spammy popup messages and fall back to the default
if the temporary path override does not exist or is not writeable-to on options dialog ok, a veto exception will be raised
refactored the watcher and multiple watcher to their own file, ClientImportWatchers
misc fixes

version 307

wrote a gelbooru 0.2.5 (which matches gelbooru itself) parser in the new system. it now has some more redundancy and produces md5 hash and source urls
fixed the e621 parser for flash files
manage tags is now a notebook rather than a listbook!
a problem where OS X was displaying the wrong label in the manage tags listbook is now fixed (as it no longer uses the buggy old listbook)
improved autocomplete focus setting in OS X in general
moved all paged importer loops to the new job scheduling system, which will massively cut down on idle time thread count (and some idle CPU usage) on clients with a bunch of import pages open
the main thread pool (which the job scheduling system uses) can now temporarily grow much larger (200 threads) when it gets hammered
recalibrated some thread watcher code to deal with being embedded in a larger object better (and sharing a page with other watcher importers)
finished the first version of a multiple watcher. it is ugly and only for advanced users for now--please check release post for more information
added 'use the multiple watcher on DnD events' to options->downloading, which will send thread url drag and drops straight to a new/existing multiple watcher!
gui session load now 'shows' the first page but loads everything to the right in the background, which saves some CPU and memory for large sessions. this means it now starts focused on the leftmost page rather than the right--see if you like it or not
'clear and load session' now fully deletes old pages and takes a little break between the clear and load step to make sure old large sessions are deleted tidily before loading the new stuff
cleaned up the new leaner popup message display code to react better to (and re-layout) sub-changes in its popups (so now when a subscription popup spawns a new progress gauge, it _should_ be positioned in the right place immediately)
if an import folder/subscription publishes files to a page while the main gui is minimised (and the action would result in a new page creation), the page add will be delayed until the gui is no longer minimised (new page layout in this case was broken due to minimised parent frame)
the edit subscription query panel will now spawn with the query text input focused
added a patch to ensure new booru import pages get query input focus after a second if the focus attempt on init failed (this will be fixed better in the new downloader system)
restored the 'clear deleted files record' button to the 'combined local files' service on _review services_, which was previously hidden during a significant service rewrite.
if critical master tables are missing on boot, the client will warn and provide info and then abandon the boot
if cache similar files tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway (this replaces some old repair code that wasn't always kicking in at a good time)
if cache autocomplete tables are missing on boot, the client will warn and provide info and offer to recreate and repopulate them
if mappings tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway
the emergency 'repair missing file locations' panel now runs in a more stable way
this emergency dialog will now also recognise 'I fixed all the file paths but cannot do the thumbnail paths' situations and will present the same 'ok, make sure you run regen thumbs after boot' dialog as if all the missing paths were thumbnail related
you can now turn off x/y page name import progress under options->gui
fixed an issue where tags that begin with ':' like ':p' were not getting through the new importing system
fixed an issue where tags that begin with ':' like ':p' could sometimes get prepended extra escape characters and end up looking like ':::p'
fixed sibling tag searching (e.g. if a->b exists and you search for b, results with a but not b should also appear), which the recent tag search optimisation accidentally disabled
whole bunch of refactoring and cleanup of ClientData and ClientGUICommon
cleaned up some spammy splash text status setting that was flooding some debug info with useless garbage
gave getting_started_tags.html's tag repo section and access_keys.html a pass, updating the ancient screenshot and linking advanced users to the latest QuickSync location
some misc help updates
misc autocomplete logic improvements
misc ui display fixes
misc wx destroy code stability improvement
updated ffmpeg for windows builds to the new 4.0 release
updated sqlite for windows builds

version 306

the file import status list now has 'open selected import files in a new page', which should show up where it is possible. this is a bit prototype and ugly--it'll show _all_ files, including in-trash and permanently deleted (which will show up with the hydrus thumbnail)
the file import status list now prefixes the already in db/deleted notes with 'url' or the hash type that lead to the recognition
these redundant/deleted notes now also propagate up from 'during import' recognition phase as well
the 'delete seeds of type x' entries on the file import status button's right-click menu are now split into three smaller individual tyes and are more explicit about exactly which status types they will remove
like import folders, subscriptions can now optionally publish their files to pages as well as popup buttons. also, subscriptions can optionally publish their files separately for each query instead of all merged together
sped up multiple tag queries significantly
sped up simple (file size, mime, etc...) system predicate queries that also include a tag/namespace/wildcard predicate significantly
added a pixiv parser that pulls the japanese tags to the defaults--users can switch to this if they prefer under network->manage url class links
fixed the 4chan parser to get part of comment as backup subject/page title
removed the 'newgrounds' entry from the normal gallery page creation ui, as the basic gallery parser no longer works due to a dynamic loading change on their end. I hope to have it back with the new gallery parsing system I will soon be writing
the edit url classes panel now has a little text box to put in example urls and see which class, if any, that they match to
improved layout of edit url class links panel
all url types are now displayable in the media viewer--only post url classes are default on
the new (x/y) import page page_name progress count is now updated on all alterations to this value (previously, this was not updating when a user interacted with the import queue, only when the natural downloader loop cycled)
added 'can produce multiple files' option to post url url classes, which informs client url-checking logic whether the url can be relied upon for 'already in db/deleted' calculations
the pixiv file page url class now has 'can produce multiple files' checked, meaning some bad pixiv url association logic due to other sites referencing it as a source url is now fixed
added a 'twitter tweet' url class, which is also a 'can produce multiple files' post url
added a 'sync known urls?' action choice to the duplicate merge options panel, which governs whether urls should be copied from worse to better or in both directions
gave the edit duplicate merge options panel a layout pass
the edit duplicate merge options panel will now disable pointless/over-complicated choices on non-custom actions, let me know if this is a pain for your workflow
added a 'manual' web browser path override to the 'files and trash' options panel, which fixes the new share->open->in web browser option for Windows and also fixes some #anchor link propagation
consolidated all URL/Path web browser launching code to one location
'open in web browser' is now available for non-advanced_mode users and the 'open' submenu of the share menu is available in the preview window and the media viewer
fixed a bug that was causing import folders to publish incorrect file identifiers, which was poisoning popup buttons and import page destinations
gui sessions that fail to load a page will recover and continue to attempt loading the rest of their pages. some popups detailing the page's serialised data and error will be presented
gui sessions that fail to save a page will recover and continue to attempt saving the rest of their pages. some popups detailsing the page's rough info and error will be presented
the core controller inside all media pages will now present itself in a more beautiful way when asked to dump itself to a log (which should beautify the above save error a bit)
wrote a subsidiary database->check->just repo update files that tests integrity of only repository update files
fixed an issue where default tag import options were sometimes not being saved from the new dialog in the networking menu
wrote a couple of layers of bad tag protection to help the new downloader deal with some occasional bad output from the old downloader
network jobs can now reattempt connection attempts up to three times on POST requests (if you ever got inexplicable immediate 'could not connect' errors on repository uploads, this should now be fixed)
replaced some archaic misc old import code with the new system, cleaning up a bunch of stuff and making space for further refactoring along the way
fixed tags blacklist not being inherited in the old (through options dialog) system
improved some invalid domain error handling
fixed an animation update issue that would pause naturally updating controls on non-main-gui frames when there were no regular media pages open on the main gui
added a BUGFIX option to 'files and trash' option page to override the default temp path for almost all client temp path requests
the minimum value for the 'vacuum period' in maintenance and processing options is now 28 days. the control also has a little explanatory tooltip
the 'try to auto-link url classes and parsers' function now always preferences parsers alphabetically
fixed a typo in the string transformations prettyfication code that incorrectly summarised 'take the last x characters' as the opposite
misc fix to file hash generation and status checking code
the 'export tags to .txt files' checkbox on the export files panel will no longer bother you with a dialog as you uncheck it
wrote some code to make it easier and more fool-proof to update the domain manager with new url classes and parsers on my end
improved some popup message manager ok-to-alter-ui logic when the main ui is minimised and so on
fixed some potential crash conditions (affecting linux mostly, seems like) in the service credential testing and access key fetching ui code
fixed a bug when 'stopping' a gallery parse during a long error pause (like when it holds on '404')
sped up some old set intersection code
some import file presentation refactoring
some url content application pipeline cleanup
misc cleanup

version 305

fixed the pixiv url class, which was unintentionally removing a parameter
wrote a pixiv parser in the new system, fixing a whole bunch of tag parsing along the way, and also parses 'source time'! by default, pixiv now fetches the translated/romaji versions of tags
finished a safebooru parser that also handles source time and source urls
finished an e621 parser that also handles source time and source urls and hash!
wrote a danbooru parser that also handles source time and source urls and hash!
as a result, danbooru, safebooru, e621, and pixiv post urls are now drag-and-droppable onto the client!
finished up a full yiff.party watcher from another contribution by @cuddlebear on the discord, including url classes and a full parser, meaning yiff.party artist urls are now droppable onto the client and will spawn thread watchers (I expect to add some kind of subscription support for watchers in the future). inline links are supported, and there is source time and limited filename: and hash parsing
fixed some thread watcher tag association problems in the new system
when pages put an (x) number after their name for number of files, they will now also put an (x/y) import total (if appropriate and not complete) as well. this also sums up through page of pages!
if a call to close a page of pages or the application would present more than one page's 'I am still importing' complaint, all the complaints are now summarised in a single yes/no dialog
url downloader pages now run a 'are you sure you want to close this page' when their import queues are unfinished and unpaused
if the subscriptions for 'manage subscriptions' take more than a second to load, a popup will come up with load progress. the popup is cancellable
added a prototype 'open in web browser' to the thumbnail right-click share menu. it will only appear in windows if you are in advanced mode, as atm it mostly just launches the file in the default program, not browser. I will keep working on this
harmonised more old download code into a single location in the new system
created a neater network job factory system for generalised network requests at the import job level
created a neater presentation context factory system for generalised and reliable set/clear network job ui presentation at the import job level
moved the new downloader simple-file-download-and-import to the new file object and harmonised all downloader code to call this single location where possible
did the same thing with download-post-and-then-fetch-tags-and-file job and added hooks for in the subscription and gallery downloader loops (where a parser match for the url is found)
the simple downloader and urls downloader now use 'downloader instance' network jobs, so they obey a couple more bandwidth rules
harmonised how imported media is then presented to pages as thumbnails through the new main import object
the new post downloader sets up referral urls for the file download (which are needed for pixiv and anything else picky) automatically
improved file download/import error reporting a little
entering an invalid regex phrase in the stringmatch panel (as happens all the time as you type it) will now present the error in the status area rather than spamming popups
fixed a bug in the new parsing gui that was prohibiting editing a date decode string transformation
fixed enabling of additional date decode controls in the string transformations edit panel
added a hyperlink to date decoding controls that links to python date decoding explainer
if a source time in the new parsing system suggests a time in the future, it will now clip to 30s ago
misc downloader refactoring and cleanup
fixed an issue where new file lookup scripts were initialising with bad string transformation rows and breaking the whole dialog in subsequent calls, fugg
hid the 'find similar files' menu entry for images that have duration (gifs and apngs), which are not yet supported
added 'flip_debug_force_idle_mode_do_not_set_this' to main_gui shortcut set. only set it if you are an advanced user and prepared for the potential consequences
silenced a problem with newgrounds gallery parser--will fix it properly next week
fixed some old busted unit test code
rejiggered some thumb dupe menu entry layout

version 304

renamed the new 'tagcensor' object to 'tagfilter' (since it will end up doing a bunch of non-censoring jobs) and refactored it into clienttags
attached a tag filter object to all tag import options to act as a tag blacklist. all tags that go through the import pipeline (except for a couple of old legacy instances) are now checked against the blacklist, and if a bad tag is found, the file vetoes! tag import options has some new ui to handle this and background code to deal with inheritance from defaults and so on
new file import urls that have url classes, no matter their source, are now normalised!
all new file import urls are now tested against both the original and normalised version of the url, so even though previously parsed urls remain un-normalised, new urls that are pre-normalised the same will not count as new! -fingers crossed-
on update, the db will get normalised copies of all existing urls. this means many files will now have two versions of its urls--some ui to collapse everything down to only the normalised version (after some human eyes have passed in front of this big change) will come in the coming weeks
some sites where normalisation is a consistent problem for later redownloads (like e621, which appends 'preview' tags to the post url) _should_ now be caught reliably!
the 'allow subdomains' on edit url class panel is now named 'match subdomains' and has a tooltip to better explain how it works
'keep subdomains' is now 'keep matched subdomains' and has a tooltip as well
the 'keep matched subdomains' enabled behaviour (and some normalisation calculation) is now additionally governed by the 'associate url with files' value and api url conversion info rather than just 'match subdomains' and raw url type
fixed an issue that was stopping the 'associate url with files' option sticking in edit url class panel
edit url matches now resorts after an add or edit action
all listctrls with a wrapper panel now resort after an import from clipboard, png, or defaults call
url matches now match against www*. versions of their domain regardless of 'match subdomains' settings
updated xbooru url classes to prefer https
the manage url class links panel now has a 'clear' button to clear a url_class->parser link
introduced three new simple downloader parsers for yiff.party, thanks to @cuddlebear on discord for the submission
the old 'uninteresting mime' status has been expanded to a wider 'vetoed' status to represent all file imports that are abandoned without a particular error (e.g. tag blacklist, wrong filesize or resolution)
the import system now reports the total of 'num vetoed' as 'num ignored' in its summary statements
it now also reports 'num skipped'
the 'num successful' and 'num already in db' are now folded more neatly together in import cache summary statements
file downloads that are cancelled will now set a 'veto' state rather than a 'skip' state
improved file import exception handling across the board
improved how single-file-result parsing vetoes propagate up to the file import status cache
404 network errors will now provide a 'veto' status rather than an 'error'
vetoes will not count as errors when deciding whether a subscription should be abandoned early (so a bunch of decomp bombs or 404s will no longer stutter a subscription!)
misc fixes and improvements to the new download stuff
wrote a new parsing cache that saves a lot of work in the new parsing system
improved the 'is this url known?' test to better deal with situations where all the given urls are galleries or unrecognised--a better aggregate of file status is formed, and 'already in db'/'deleted' statuses will apply if there is no evidence otherwise (the dev got the new logic for this from a legit nightmare about urls downloading over and over, so let's hope it works out)
the 'is this url known?' logic also recovers from 1->n url->hash relationships where it does not expect them, trying to find 'already in db' hashes over 'deleted' ones
to clear up some ambiguity, galleries or subscriptions now give a different 'checking in x seconds' status when waiting on the first page of a query
the 'noneablebytescontrol', as seen in edit file import options, will now correctly disable/enable its bytes sub-control when it is none'ed
a persistent issue with the new network engine sometimes failing to correctly error after certain broken connections (the computer going to sleep mid-download was a common cause here) should now be recovered from and the connection naturally reattempted
added three new shortcuts to the 'main_gui' shortcut set that allow for opening a new 'urls', 'simple', or 'thread watcher' downloader page
added two more shortcuts to 'main_gui' for new 'page of pages' and 'duplicate filter page'
moved some old 'new page' menu code to the new application command system
added numerous 'duplicates' shortcuts to the 'media' shortcut set that will work on selections of thumbnails
the thumbnail duplicates menu actions now go through the new application command system
fixed an issue where the current tag parents caches was not refreshing when notified
inputting a short invalid syntactic input on a 'read' tag autocomplete such as '-' will now clear the system predicates list--system preds should now only show on a completely empty input
fixed an issue where certain combinations of 'remove a tag, then re-add it' nullipotent actions in a single manage tags dialog transaction were not applying reliably (sometimes, the subsequent mirror action was not occuring due to a processing re-order optimisation at the db level)
made some animation code a little safer and quieter as a test for some users who were getting blitzed with some deadwindow error spam in certain situations--let's see if this changes anything
replaced all the em dashes in the help with double hyphens as github pages was rendering them wrong
added CrystalDiskInfo recommendation to 'help my db is broke.txt'
misc cleanup

version 303

file post url classes can now be linked to parsers!
the 'raw url' downloader is now just the 'url' downloader. if a dropped url is matched as a 'file post' url and links to a parser, it will download it with that parser and default tag import options
url drag and drop now recognises file post urls that have linked parsers and will send them on to a url import page, just like 'raw' urls! they will use the new parsing system to parse tags and known urls and all that
wrote a way for the new download system to store and edit default tag import options--this can now be found under network->manage default tag import options, although it does not do much yet. eventually the old options will be transferred here, and the different downloaders will consult it more
refactored and cleaned up how some default tag import option calculation is done
the fixed 5 second wait time between gallery page fetches is now editable for downloader pages and subscriptions under options->downloading. subscriptions default remains 5s, gallery default is now 15s
the gallery downloader will present its 'x urls found so far' string in a more uniform and reactive way. it will also count down until the next page fetch
subscriptions will now similarly react quicker while the gallery parse step is ongoing, and it will count down to the next page fetch
the manage subscriptions dialog will now note in BIG RED TEXT if subscriptions are currently globally paused
subscriptions now consume less parallel-timeslot overhead, meaning they sync much faster when they have no work to do (which is most of the time)
separating subscriptions is now more sophisticated--if you wish, you can now choose just to separate a subset of the large subs's queries, and if you do so, you can decide whether to merge the extracted subs into a new larger sub or just have them as individuals. also, you will be asked for name/base_name for what to name the new subs
since separating subs is more complicated, the button is only enabled when a single large sub is selected. please do your separating one sub at a time!
merging subs will now ask if you want to rename the merged subs as it goes
refactored a giant heap of duplicate import code into single locations in the new file import object--things like applying tags and generating pre-import status are all now done in one place. a number of weird behaviours (like not applying known url associations in certain circumstances) should now be a lot better and unified
the way the db and other import testers handle whether a pending import is new or unknown is simplified
optimised the way the importers figure out whether to display a new import
the 'has the client seen this url before?' test used to figure out whether to download a file now explicitly ignores anything it isn't certain is a single-file File URL or Post URL (according to current url classes). associating 'gallery' known urls is no longer catastrophic to this system
the simple downloader now associates the entered url with the files subsequently imported
file import objects working in the new parsing system are aware of the new 'should this url be associated' option and will be given gallery urls to hold on to as appropriate
url classes now have an explicit option as to whether they should be associated with files imported from them, which is not appropriate for dynamic CDN File URLs but is appropriate for multi-image tweet links (which will likely be Gallery URLs in the new system), for instance
general importer code cleanup across the board
threw together some early ui to show current jobs in the network engine under network->review network jobs
popup messages will now try to set a max width of roughly 56 characters wide rather than 400 pixels. this value is editable under options->gui
updated the listctrl in the export files frame and the way this frame generates and stores export filenames. let's see if linux users who had crashes with this have better luck
refactored stuff out of the bloated ClientImporting file into seperate new files
refactored client network bandwidth and session code out to their own files as well
refactored client network objects to its own file
cleaned up some open externally debug code linked to callto report mode
a common grid layout used across the program will copy control tooltips to the string on the left
removed the 'downloader' network context type, which was never fully introduced and wasn't turning out to offer much in the new bandwidth tracking system

version 302

improved how simple downloader parsing formulae are stored and passed around
the edit simple downloader parsing formulae panel now has an 'add defaults' menu button
the simple downloader formula edit panel handles its buttons better and can now do import from/export to clipboard/png and reimport from the defaults, and fixes duplicate names automatically
simple downloaders now remember their current parser through a session save/load
simple downloaders will set a new default parser selection for new simple downloader pages on any choice event
renamed all the default simple downloader parsers to more clearly explain what they do
added 'all files linked by images in page' to the simple downloader, which does the old page of images' behaviour, and is hence pretty decent for most imageboards
added a simple mewch thread html parse formula to the simple downloader
added a very simple webmshare parse formula to the simple downloader
added 'imgur image album' to the simple downloader, which will fetch the first 'page' of results from an image album. full parsing will have to wait for the gallery downloader update
subscriptions can now run simultaneously! you can set the max number at options->downloading page--the default remains 1, so default behaviour will not change
if subscriptions guess they can eat some bandwidth in the next 30s but it turns out they can't, they will bandwidth-override that last job after the 30s expires. this will stop some edge cases from causing subs to potentially hang for hours just on one last request
'explicit tags' are now renamed to 'additional tags'
you can now set media to initially scale to 100% even when it is larger than the media or preview canvas
the html parsing formula can now 'ascend' as well as search down the DOM tree--so, for instance, you can find all 'img' below an 'a' and then jump back up to the 'a' to fetch the 'href' of that image link!
html parsing formulae can now search without a tag name--so, for instance, 'find all tags with class = "whatever"' is doable, without having to specify div or span etc..
the html parsing formula rule panel is hence a bit richer. it also presents a preview of the rule's string as you edit
the client no longer needs lxml to boot, and if html5lib is present, it will prefer that for html parsing. if both libraries are missing and html parsing is requested, it will raise an exception explaining the error
the builds now include html5lib, which is better at recovering from some bad markup than lxml, and as a result some parsing jobs that formerly did not work (such as 'akaiha_(akaihasugk)' on danbooru) now do
rewrote how services test their functional status to better account for different states. the logic here should be a bit better now--repositories will still be able to process if their network side is down, but everything will pause if any pause is set, and it should all propagate up to higher levels of error catching and status reporting better
hydrus can now deal with really big decompression bombs (>~180 megapixel)
filtered out en-dashes (as an alternative to standard hyphens in a "? + -" link-prefix) from booru tag parsing
fixed a string generation issue that was stopping date decode string transformations from summarising themselves
fixed some catastrophic boot failure gui reporting
cleaned up a double-exit error on dialogs that could occur when spamming exit events with a script like AutoHotkey
improved some dead cpu thread clearout code
updated misc old code
misc cleanup

version 301

after discussions with Sankaku Complex about their recent bandwidth problems, added a new 64MB/day default bandwidth rule for sankakucomplex.com--please check the release post for more information
the 'page of images downloader' is now called the 'simple downloader' that uses the new parsing system (particularly, a single formula to parse urls)
the simple downloader supports multiple named parsers--currently defaulting to: html 4chan and 8chan threads, all images, gfycat mp4, gfycat webm, imgur image, imgur video, and twitter images (which fetches the :orig and also works on galleries!)
there is some basic editing of these parsing formulae, but it isn't pretty or easy to import/export yet
the new parsing test panel now has a 'link' button that lets you fetch test data straight from a URL
added a 'gather to this page of pages->dead thread watchers' menu to the page of pages right-click menu--it searches for all 404/DEAD thread watchers in the current page structure and puts them in the clicked page of pages!
cleaned up some page tab right-click menu layout and order
fixed tag parents, which I previously broke while optimising their load time fugg
the new favourites list now presents parents in 'write' tag contexts, like manage tags--see if you like it (maybe this is better if hidden?)
sped up known_url searches for most situations
fixed an unusual error when drag-and-dropping a focused collection thumbnail to a new page
fixed a problem that was marking collected thumbnails' media as not eligible for the archive/delete filter
wrote a 'subscription report mode' that will say some things about subscriptions and their internal test states as they try (and potentially fail) to run
if a subscription query fails to find any files on its first sync, it will give a better text popup notification
if a subscription query finds files in its initial sync but does not have bandwidth to download them, a FYI text popup notification will explain what happened and how to review estimated wait time
delete key now deletes from file import status lists
default downloader tag import options will now inherit the fetch_tags_even_if_url_known_and_file_already_in_db value more reliably from 'parent' default options objects (like 'general boorus'->'specific booru')
the db maintenance routine 'clear file orphans' will now move files to a chosen location as it finds them (previously, it waited until the end of the search to do the move). if the user chooses to delete, this will still be put off until the end of the search (so a mid-search cancel event in this case remains harmless)
the migrate database panel should now launch ok even if a location does not exist (it will also notify you about this)
brushed up some help (and updated a screenshot) about tag import options
fixed a problem that stopped some old manage parsing scripts ui (to content links) from opening correctly
improved some parsing test code so it can't hang the client on certain network problems
misc ui code updates
misc refactoring

version 300

wrote system:known url to find files that have--or do not have--certain types of urls. it works but is still a little slow--I can optimise it later!
added exact match, domain, regex, and url class search types for system:known url
added a button to the top media viewer hover window that will start a file export drag and drop event if dragged from
moved the autocomplete dropdown results list down into a paged notebook
wrote a new 'favourites' page tab for the autocomplete dropdown results
hitting left or right arrow keys on an empty text input will move between the results tabs
hitting arrow up/down/page up/down/home/page or passing mouse scroll events will now go to the current selected page
typing regular search text into the input will automatically return the current page to the search results list
moved the 'tag suggestions' part of the 'tags' options page to a new page
added 'tag favourites' to the 'tags' options page to edit which tags show in this new tab
added import/export buttons to the tag siblings and parents dialogs. they'll export to clipboard or .txt file, and import from the same with an additional option to add_only (i.e. to not delete/petition conflicts with the existing list)
added some quick-and-dirty 'set as alternates/same/notdupes' buttons to the duplicate filter, which will quickly apply that status to the dupes and show some more dupes
sped up db loading time of tag siblings and parents significantly
added a short delay check to tag siblings/parents regeneration so rapid regenerations (such as when processing certain admin-side petitions) can be merged
fixed an issue where similar_to searches could return results not in the current file domain
fixed some spinctrls that were sizing to thin
fixed a bug in the manage server services dialog that was incorrectly dealing with port conflicts on edit service dialog ok
added a clientside and serverside assertion to test that all the services on a serverside modify services call have unique ports
fixed an issue where hydrus network services without access keys would sometimes try to sync their accounts (this was messing up some admin server setup)
fixed some misc dialog window structure
messed around a little with how the autocomplete dropdown hides and shows when in float mode--I _think_ it will now be less flickery and will otherwise position itself and receieve focus better
converted the 'export files' dialog to the new sizing system and also made it non-modal (i.e. you can now interact with the rest of the program while it is open)
wrote a more rigorous force-fit-all-tlws command to the debug menu
misc fixes
misc refactoring

Revision #1
Created 12 March 2021 17:05:16 by CuddleBear
Updated 12 March 2021 18:24:07 by CuddleBear