Changelog

• Changelog 400>
• Changelog 350-399
• Changelog 300-349
• Changelog 250-299
• Changelog 200-249
• Changelog 150-199
• Changelog 100-149
• Changelog 50-99
• Changelog <49

Changelog 400>

• version 432

• tag sorting:
• the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select
• my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'
• when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.
• I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control
• the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically
• fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed
• if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'
• .
• misc:
• the client now gives a once-per-boot warning popup if your session size exceeds 500k. for those who cannot reduce session size conveniently, this popup can be turned off under _options->gui pages_
• when the file import options prohibit a file due to filesize or resolution etc.., it should now always record that as an 'ignored' result rather than an 'error'
• fixed an unusual error popup in thread watcher display that could occur during session load. this problem seems to have been around for a long time, but it required a watcher in a previously saved and still valid 'wait a bit' error state and was only vulnerable for a few milliseconds, so it hadn't come up before. in any case, it is fixed
• subscriptions with small 'first run' file limits now work better: if you create a subscription with a fairly small 'first run' file limit (this typically matters when the number is smaller than one of the site's gallery page's worth of results), subsequent normal checks with larger file limits will be more aggressive about noticing that they 'caught up' to that small initial sync (previously, they would sometimes incorrectly think the site just got some files tagged out of order and bump right past that initial 'already in db' batch and keep going until they hit their own file limit)
• .
• advanced string processing:
• added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!
• the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it
• the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly
• the new String Slicer edit UI similarly gets all the strings at that stage of processing
• future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage
• .
• server/client api core improvements:
• I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416
• the client now understands 416 ("can't do that requested range m8") errors
• I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly
• .
• boring:
• fixed up a handful of typo-borked unit tests
• fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao
• started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful
• updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance
• updated sqlite dll on the Windows release to 3.34.1

• version 431

• misc:
• when parents are hidden in the edit/write taglists (e.g. in manage tags), there is now a '(n parents)' suffix
• thread watchers that are DEAD or 404 but still have files downloading now report 'working' status until that is done
• search terms with ';' like 'steins;gate' should now work in downloaders. sorry for the trouble!
• fixed an issue where un-ideal tags were sometimes becoming non-searchable when they were entirely replaced in manage tags with their 'ideal' siblings (i.e. their autocomplete count went to 0). this was due to overzealous deletion in the new tag definitions cache not filtering out sibling/parent chain members. a small routine will run on update to resynchronise affected tags
• fixed an issue when loading up files in the main 'import files' dialog where a critical error (as opposed to a nice 'couldn't figure it out, sorry') in mime detection would cause the whole job to hang
• that main 'import files' dialog now counts 'missing' files separately in the error count
• fixed tags not updating on the filename tagging dialog when double-clicking to remove from the simple taglists
• fixed the sort on the manage tags dialog's suggestion taglists--they now preserve their original sort, rather than alphabetising once sibling/parent data is populated
• the manage parser test panel now catches all network errors. error data back from the server is presented better, and the traceback is now viewable in a special new button on the network job control
• the edit shortcut set dialog now gives a veto text popup if you try to ok with a shortcut set twice (previously, it would ok and merge down to one command randomly). support for multiple commands per shortcut will come in future
• entering alt+number in the shortcut entry dialog on windows will no longer spam some errors about 'null character'
• fixed a 'this object is too huge' check in the database, which mostly affects gui sessions with millions of objects, to check against 1 billion bytes max size rather than 1GB, as here https://sqlite.org/limits.html (issue #816)
• fixed the 8chan.moe parser, which was pulling hash incorrectly. it should now save more bandwidth
• updated the e621 parser to pull their (new?) 'lore' tags, which all end in _(lore) and refer to canonical gender and some spice. they come with the 'lore' namespace for now, we'll see how it works out. a user reports these are useful for blacklists
• .
• new string processing sort step (for advanced users):
• string processing objects have a new processing step: String Sorter
• this sorter can sort the whole list of strings, either strict lexicographic or 'human sort' that does numbers properly, asc/desc
• it can also take a regex for the sort 'key', so you can sample just the number or name you want for sort purposes
• content parsers no longer have the 'sort formulae results' controls. any content parser with existing sort has been converted so its string processing object has a String Sort step appended
• the string processing UI is still built around single string processing, so the test UI here is essentially non-functional, but you can see the sort happen in the formula test parse panel
• I will add a String Slicer in future to sample the list of strings, so you'll be able to grab the top item etc...
• .
• boring code cleanup:
• refactored the refresh call in filename tagging dialog to nicer Qt signals
• the add/remove taglists on the simple panel are also moved to Qt signals
• and so are 'filter' taglists
• fixed some typos in new help text
• removed a 'needs restart' string in 'gui pages' options that no longer does
• .
• here is a hieroglyph falcon:
• 𓅃

• version 430

• misc:
• fixed 'unusual character' collapse logic for short text inputs in tag autocomplete lookups. in human, this means typing 'a' now correctly gives you the tag '/a/' and _vice versa_ (issue #799)
• to make this work, an old database subtag map cache is revived this week in a more efficient form. if you sync with the PTR, it will take a couple minutes to update. the regen routine is also added to the database->regen menu, in case it ever desynchronises in future
• absent an override referral url, api-linked url fetches now use the original url as referrer. previously they were sending no referrer. this fixes watching spicy boards on 8chan.moe
• updated a 'get all this stuff' database routine to report more info, and a handful of supermassive jobs (mostly db maintenance regen) now report x/y progress with y, rather than just a nebulous increasing x
• fixed an odd bug in a common UI text-clearing call that was causing real text not to show up for a while after the clear. this was most apparent in the downloader highlight panels, where status text on file/gallery/network status could sometimes stay blank until a change
• the manage tags dialog's "there are several things you can do" button box when you enter tags in complicated situations is now clearer. there are several sorts of intro text on the dialog, the button labels are clearer, and button tooltips have more action information
• fixed the tumblr downloader! sorry for the trouble here, I hadn't realised the situation from some reports. if you have tumblr subs, please go into them and set to 'try again' any recent urls that say 'Found 0 new URLs.'
• .
• taglists:
• you can now right-click any edit/write taglist (like those across the manage tags dialog) and choose to hide/show the implied parents that now hang underneath tags
• you can set whether this defaults to hide or show, separately for the regular taglists and the autocomplete results dropdown, under options->tags
• the taglist now sorts lexicographically using sibling tag data where available. I had expected to make options here to use storage or ideal tag, but once I tried it out, using the ideal all the time felt proper to me, so let's see how it goes
• fixed the routine that removes mutually exclusive predicates (e.g. system:inbox/archive) when adding to the active search predicates taglist. this fixes the 'exclude xxx from search' menu action and other add/swap actions (issue #815)
• gave the taglist right-click menu another quick pass. since there are all sorts of actions that may or not appear, and menu items can get pretty wide with tag text, I am trying out an intentionally short and thin top-level menu of 'verbs' that is quick to navigate with your mouse, and then tuck longer and taller stuff in secondary menus
• .
• boring code cleanup:
• cleaned and unified a bunch of the new taglist sibling and parents display logic and other legacy variables. it now basically all derives from one storage/display state, so behaviour across the program should be more unified. this may cause confusion in some more advanced dialogs, so let me know anywhere it looks weird
• the 'favourites' autocomplete tab in 'edit/write' a/c dropdowns now show siblings and parents for the current display service
• the tag suggestions favourites dropdowns and taglists in the options now show siblings/parents according to the current service
• the 'url class precedence' routine, which tests more 'specific' url classes first when trying to match an url, has a subtle logic change--now, url classes are first considered more 'specific' according to number of path components and parameters that have no default. this stops an url class with multiple optional parameters overriding another with a single fixed parameter (this is what affected the tumblr downloader above). the specific (descending) sort key is now (required components, total components, required parameters, total parameters, len normalised example url)
• refactored client object serialisation access routines to a new db module
• refactored database transaction code and status tracking to a separate object
• refactored some more tag definition routines to the master tag module

• version 429

• misc:
• fixed a bug in the new taglist backend that would sometimes error out in a paint event(!) on display initialisation or data changes for some clients
• improved the taglist 'tag' vs 'copyable string' copy/select/action menu logic. e.g. 'namespace:*' is copyable, but it is not a tag
• thread watchers now skip/clean up unactioned check log entries (this usually happens when a check is due during network traffic paused, queueing the job, and then the client shuts down). if you noticed some odd perpetually 'pending' checkers in last week's status overhaul, this was the issue, and they should clean up. this was always harmless, just revealed with new status code
• thread watchers now record serious network error detail in the check log
• thread watchers are quicker about notifying UI on checker log changes
• thread watchers now report 'time delta' as their simple status when waiting to check, rather that 'checking in (time delta)'. let's see if that fits better in the columns
• fixed an issue where several dialogs with multi-column lists would reset their 'last column' size to the minimum three characters on the next load if they did not receive certain size events while they were open. you should just have to fix any broken dialogs once and you'll be good again
• I believe I also improved/fixed the issue of dialogs with multi-column lists sometimes shrinking by a few pixels every open/close
• the 'we just woke from sleep' detection is now more aggressive. it should now detect a wake after sleeps as short as 60 seconds (down from 5 mins). let's see if we get any false positives during maintenance or other busy periods
• if you have a complicated database (one stored across multiple locations), the 'database' menu now has a label in place of the simple database's 'backup/restore' commands
• improved the 'directory is writeable-to' check used in the program. on windows, due to some python tempfile weirdness, this was actually hanging on Program Files.
• improved the related 'is db dir writeable-to' test in the boot script. if you try to run the program on a custom non-writeable db directory, the crash error should now be nicer, and running a straight client.exe installed to 'Program Files' should now auto-place your db in your user folder, no complaints, like the macOS App
• corrected 'writable' typo to 'writeable' across the program lmao
• fixed the new header links in the FAQ file, which I accidentally messed up
• started work on updating neighbouring .txt tag sidecar export. it isn't ready yet, but it will add tag filters and tag display type to sidecar export with easier expansions in future, and fold it nicely into Export Folders
• improved some log-off detection + clean shutdown code, but I do not yet have nice multiplatform support
• .
• filetypes:
• the stacked expand/collapse checkbox widget that lets you select filetypes now always starts collapsed. also, some 'partially clicked' logic is improved when you click through filetype group
• application/clip (clip studio paint) files are now supported! thanks to a user for helping out here
• just a side note: I looked into animated webp support this week, but it turns out decoding support is rarer than encoding. my normal and fairly new FFMPEG can't reliably render subsequent frames or figure out duration, nor can PIL or OpenCV. I think we will simply have to wait for an update on one of their ends
• .
• boring db cleanup:
• wrote a local hashes cache to store hashes for all the files on your disk, much like the tag one. this should speed up all normal searches and other common file lookups in the db
• the raw storage mapping tables are spun off to their own module
• basic file info and inbox is spun off to its own module
• improved and sped up some inboxing file count calculations
• cleaned up some more misc file metadata and inbox code
• improved logic in local tags cache

• version 428

• interesting taglist changes:
• taglists work way better behind the scenes
• when siblings display with the '(will display as xxx)' suffix, this text is now coloured by the correct namespace!
• parents now show in 'manage tags dialog' taglists! they show up just like in a write/edit tag autocomplete results list
• the tag right-click menu has had a pass. 'copy' is now at the top, the 'siblings and parents' menu is split into 'siblings' and 'parents' with counts on the top menu label and the submenus for each merged, and the 'open in new page' commands are tucked into an 'open' submenu. the menu is typically much tighter than before
• when you hit 'select files with these tags' from a taglist, the thumbgrid now takes keyboard focus if you want to hit F7 or whatever
• custom tag presentation (_options->tag presentation_, when you set to always hide namespaces or use custom namespace separator in read/search views) is more reliable across the program. it isn't perfect yet, but I'll keep working
• a heap of taglist code has been cleaned up. some weird logical issues should be better
• now the code is nicer to work with, I am interested in feedback on how to further improve display and workflows here
• .
• the rest:
• added two mirrors for nitter, whose main site is failing due to load. I added them randomly from the page here: https://github.com/zedeus/nitter/wiki/Instances . if you have nitter subs, please move their download source to one of the mirrors or set up your own url classes to other mirror addresses. thanks to a user for providing other parser fixes here
• gallery download pages now show the 'stop' character in the small file column when the files are done
• gallery download pages now report their 'working' status without flicker, and they report 'pending' when waiting for a download slot (this situation is a legacy hardcoded bottleneck that has been confusing)
• thread watchers also now have the concept of 'pending', and also report when they are next checking
• improved the new grouped status sort on gallery downloader and watcher pages. the ascending order is now DONE, working, pending, checking later (for watchers), paused
• the network request delay after a system resume is now editable under the new options->system panel. default is 15 seconds
• the 'wait on files too' option is moved from 'files and trash' to this panel
• when the 'just woke' status is active, you now get a little popup with a cancel button to override it
• 'open similar-looking files' thumbnail menu entry is moved up from file relationships to the 'open' menu
• the duplicate filter right-hand hover window no longer has both 'previous' and 'next' buttons, since they both act as 'flip', and the merged button is moved down, made bigger, and has a new icon
• added 'view next' to the duplicate filter shortcut set, so you can set a custom 'flip between pair' mapping just for that filter
• thanks to a user helping me out, I was able to figure out a set of lookups in the sibling/parent system that were performing unacceptably slow for some users. this was due to common older versions of sqlite that could not optimise a join with a multi-index OR expression. these queries are now simpler and should perform well for all clients. if your autocomplete results from a search page with thumbs were achingly slow, let me know how they work now!
• the hydrus url normalisation code now treats '+' more carefully. search queries like 6+girls should now work correctly on their own on sites where '+' is used as a tag separator. they no longer have to be mixed with other tags to work
• .
• small/specific stuff:
• the similar files maintenance search on shutdown now reports file progress every 10 files and initialises on 0. it also has faster startup time in all cases
• when a service is deleted, all currently open file pages will check their current file and tag domains and update to nicer defaults if they were pointed at the now-missing services
• improved missing service error handling for file searches in general--this can still hit an export folder pointed at a missing service
• improved missing service error handling for tag autocomplete searches, just in case there are still some holes here
• fixed a couple small things in the running from source help and added a bit about Visual Studio Build Tools on Windows
• PyOpenSSL is now optional. it is only needed to generate the crt/key files for https hosting. if you try to boot the server or run the client api in https without the files and without the module available to generate new ones, you now get a nice error. the availability of this library is now in the client's about window
• the mpv player will no longer throw ugly errors when you try to seek on a file that its API interface cannot support
• loading a file in the media viewer no longer waits on the file system lock on the main thread (it was, very briefly), so the UI won't hang if you click a thumb just after waking up or while a big file job is going on
• the 'just woke' code is a little cleaner all around
• the user-made downloader repository link is now more obvious on Lain's import dialog
• an old hardcoded url class sorting preference that meant gallery urls would be matched against urls before post, and post before file, is now eliminated. url classes are now just preferenced by number of path components, then how many parameters, then by example url length, with higher numbers matching first (the aim is that the more 'specific' and complicated a url class, the earlier it should attempt to match)
• updated some of the labelling in manage tag siblings and parents
• when you search autocomplete tags with short inputs, they do not currently give all 'collapsed' matching results, so an input of 'a' or '/a/' does not give the '/a/' tag. this is an artifact of the new search cache. after looking at the new code, there is no way I can currently provide these results efficiently. I tested the best I could figure out, but it would have added 20-200ms lag on all PTR searches, so instead I have made a plan to resurrect an old cache in a more efficient way. please bear with me on this problem
• tag searches that only include unusual characters like ? or & are now supported without having to lead the query with an asterisk. they will be slower than normal text search
• fixed a bug in the 'add tags before import' dialog for local imports where deleting a 'quick namespace' was not updating the tag list above
• .
• windows clean install:
• I moved to a new windows dev machine this week and a bunch of libraries were updated. I do not believe the update on Windows _needs_ a clean install this week, as a new dll conflict actually hits the coincidentally now-optional PyOpenSSL, but it is worth doing if you want to start using the Client API soon, and it has been a while, so let's be nice and clean. if you extract the release on Windows, please check out this guide: https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#clean_installs
• the Windows installer has been updated to remove many old files. it should now do clever clean installs every week, you have nothing to worry about!™
• .
• boring db breakup:
• the local tags cache, which caches tags for your commonly-accessed hard drive files, is now spun off to its own module
• on invalid tag repair, the new master tags module and local tags cache are now better about forgetting broken tags
• the main service store is spun off to its own module. several instances of service creation, deletion, update and basic fetching are merged and cleaned here. should improve a couple of logical edge cases with update and reset
• .
• boring taglist changes:
• taglists no longer manage text and predicates, but a generalised item class that now handles all text/tag/predicate generation
• taglist items can occupy more than one row. all position index calculations are now separate from logical index calculations in selection, sizing, sorting, display, and navigation
• all taglist items can present multiple colours per row, like OR predicates
• items are responsible for sibling and parent presentation, decoupling a heap of list responsibility mess
• tag filter and tag colour lists are now a separate type handled by their own item types
• subordinate parent predicates (as previously shown just in write/edit autocomplete result lists) are now part of multi-row items. previously they were 'quiet' rows with special rules that hung beneath the real result. some related selection/publish logic is a bit cleaner now
• string tag items are now aware of their parents and so can present them just like autocomplete results in write/edit contexts
• the main taglist content update routines have significantly reduced overhead. the various expansions this week add some, so we'll see how this all shakes out
• the asynchronous sibling/parent update routine that populates sibling and parent data for certain lists is smarter and saves more work when data is cached
• old borked out selection/hitting-skipping code that jumped over labels and parents is now removed
• 'show siblings and parents' behaviour is more unified now. basically they don't show in read/search, but do in write/edit
• a heap of bad old taglist code has been deleted or cleaned up

• version 427

• ghost pending tags:
• fixed another ghost pending tags bug. this may have been new or there since the display cache started, I am not sure, but it shouldn't happen again. it was occuring when a pending tag was being committed to 'current' and another tag in its sibling group already existed as a current tag for that file. the pending tag and its count would not clear for non-'all known files' domains, causing ghosts to appear in search pages but not typically manage tags. you may have noticed ghost tags hanging around after a pending commit--this was it. the true problem here was in the 'rescind pending' action that occurs just before an add/commit. a new unit test tests for this situation both for two non-ideal tags being pend-merged, and a non-ideal tag being pend-merged into the existing ideal
• wrote a routine to regenerate _pending_ tag storage and autocomplete counts from scratch for the combined and specific display tag caches. this job is special in that it regens tags instantly without having to reset sibling/parent sync. you can run the job from the database->regen menu. this is the start of 'fix just this tag' maintenance ability
• the pending regen routines will occur on update. it shouldn't take long at all, unless you have five million tags pending, where it could be a couple minutes
• .
• autocomplete shortcuts:
• there is a new shortcut set under _file->shortcuts_ just for tag autocomplete shortcuts. any 'switch searching immediately' shortcut previously on 'main gui' will be migrated over
• the tag autocomplete input text box is now plugged into the new shortcut system and uses this set
• migrated previously hardcoded autocomplete shortcuts to the shortcut system (defaults):
• - force search now, for when you have automatic searching turned off (ctrl+space)
• - enable IME-friendly mode (insert)
• - if input empty, move left/right a tab (left/right arrow)
• - if input empty, move left/right a service page (up/down arrow)
• - if input empty and on media viewer, move to previous/next media (page up/down)
• misc improvements to my shortcut handler
• misc shortcut code cleanup
• .
• the rest:
• I fixed a bad example url in the new gelbooru file page parser that was sometimes leading to a link to the gallery url class. this was an artifact of an old experiment with md5-search parsing, now fixed with newer redirection tech. the updated parser is folded into update, and if you ended up with the incorrect link, it should be detected, dissolved, and re-linked with the file page parser
• thanks to a user report, wrote a new url class for 420chan's newer thread url format
• sorting a gallery downloader or thread watcher multi-column list by 'status' should now group 'done' and 'paused' items separately
• fixed a bug in the /add_tags/add_tags Client API call when checking some petitioned tags data types. cleaned all that code, it was ugly (issue #788)
• added unit tests for /add_tags/add_tags to test the service_names_to_actions_to_tags parameter better and repository actions, including petitioning with and without specified reason
• .
• code refactoring:
• finally addressing the near-1MB ClientDB file, I have started a framework to break the db into separate modules with their own creation/repair/work responsibilities. this will make the file easier to work on, maintain, update, and test. this week starts off simple, with the master definitions being peeled off into hashes, tags, urls, and texts submodules
• cleaned some misc code around here, including a bunch of related decoupling
• ClientDB.py is now in its own 'db' module as well. the db will further fracture and this module will gain more files in future
• the boot code in the launch scripts is now migrated to the 'hydrus' directory, with the actual launch scripts now doing nicer __main__ checks to not launch the program if you want to play around with importing hydrus. more work to come here
• finished the help's header linking job--all headers across the help are now #fragment links
• misc help cleanup

• version 426

• misc:
• thanks to help from Codexx at 8chan.moe, the old 8kun board is completely migrated and archived at 8chan.moe /hydrus/. going forward I will be maintaining a Hydrus Network General there on /t/ for merged release posts, Q&A, and Bug Reports. the plan is that whenever it fills up, it will be moved to the /hydrus/ archive. the links across the program and help are updated, please let me know if I missed any. Endchan /hydrus/ remains as a bunker
• fixed a bug where subtag entries in the new tag fast search cache were being deleted for all namespaces when a single namespaced version was went to count 0. it meant some autocomplete results were not appearing, often after some sibling changes. a new 'repopulate' job has been added to the database regenerate menu to fix this efficiently if something like it happens again. this routine will be run on update to fix all users, it shouldn't take long (issue #785)
• fixed a bug where the new network objects would throw an error on save when a 'dirty' object was quickly deleted. I think this was typically sessions that only have ephemeral session cookies being created in the final five minutes of the program and then being cleared during program exit
• when an archive/delete filter finishes, it now fires off all its changes in one go. previously they would go in ~64-file chunks over the next few hundred milliseconds. this will add a small amount of 'refresh lag', delaying page refreshes etc..., on bigger filter jobs for some clients, but it will guarantee that if you hit F5 real quick after finishing filtering on a processing page with non-random sort, you won't see the same files again at the top, only for them to be swiftly archived/deleted as you watch. trash file performance is much better these days, let me know how this goes for you if you do megafilters
• the tag import options whitelist now checks subtags of parsed tags. if you add 'samus aran' to the whitelist, but the site delivers 'character:samus aran', this now passes the whitelist
• thanks to a user's submission, the gelbooru 0.2.5 post parser is updated and should get tags again, for those users who stopped getting them last week--however, I never experienced this myself, so please let me know if you still have trouble. there could be something more complicated going on here
• updated the gelbooru 0.2.x gallery parser to handle an alternate form of gelbooru pools--we did not figure out why different users are being given different markup, it wasn't as simple as being logged in or not, but there is a difference for some. this parser is folded in on update, so the gelb pool downloader should be fixed for users who had trouble with it
• also updated the gelbooru pool gallery url class to infer next page url, as in the alternate form the next page is difficult to parse
• the 'clear all closed pages' command under the 'undo' menu now asks for yes/no confirmation
• added a 'callto' profile mode, which will be very useful in diagnosing GPU lag in future. the 'callto' jobs are little off-main-thread things like image rendering and async panel preparation. should help us figure out where big download pages etc... are eating up CPU
• the different profile modes in the debug menu now all show popup messages, but only when their job exceeds the particular profile's interesting time, usually 3-20ms. this should reduce spam
• the 'this session' bandwidth tracker on the status bar is now a special tracker that only includes data from boot. previously, it was using the 'global' tracker, which after certain time intervals (four minutes, three hours, three days), will compress bandwidth history into larger time windows to save space. if one of these windows covered time before the client started, it could spookily report a little bandwidth used on a client started with network traffic paused
• bandwidth data usage in times shorter than the last ten seconds (which are smoothed out to avoid bumps) now also get the 'don't get bandwidth from the future on motherboards that had a briefly crazy system clock' fix from last week
• fixed some disengaged database tuning that was leading to worse cancel times on certain jobs
• updated a whole bunch of the help so section headings are links with nice #fragment/anchor ids, making it easy to link other users to a particular section. I will continue this work, and future help will follow this new format
• fixed some bad character encodings in the changelog document, siblings help, and tagging schema help. these should now be utf-8 valid
• .
• object load improvements:
• the client now detects serialisable (saveable) objects that were generated in a future version format your client does not yet support. this mostly affects downloader objects like parsers, where you might import an object a user in a much newer version of the client made. for instance, this week some users imported a fixed gelbooru parser in an older client, which was then saved and double-updated later on, and that caused other problems down the line. downloader imports deal with this situation cleanly, but otherwise it mostly makes a popup notifying you of the problem and asking to contact me. there are about 170 places in the program where objects are deserialised and I am not ready to make this a fullblown error until I know more about people's IRL situations. let's hope this is not widespread. if you run into this, please let me know!
• if you were running an older client and manually imported the updated gelbooru parser that was going around, and then you got errors about 'md5', hex' or 'additional_info' something, it _should_ be automatically fixed on update. you should be able to update from previous to ~422, see it in network->downloader components->manage parsers, and it should just work. many users will have the entry overwritten anyway in the above gelb update I am rolling in. if any of this does still give you trouble, please delete and re-import the affected object(s)
• importing one of these future-versioned serialised objects using the import/export buttons on a multi-column list, either clipboard, json, or png, will cleanly discard future objects with a non-spammy notification
• the Lain drag-and-drop easy downloader import does the same
• the parser 'show what this can parse in nice text' routine now fails gracefully
• multi-column lists now handle a situation where either the display or sort data for a row cannot be generated. a single error popup per list will be generated so as not to spam, bad sorts will be put at the top, and 'unable to render' will occupy all display cells
• .
• network server stuff:
• fixed being able to delete an account type in the server admin menu
• the way accounts are checked for permissions serverside now works how the client api does it, unified into a neater system that checks before the job starts
• did some misc server code cleanup, and clientside, prepped for restoring account modification and future improvements

• version 425

• optimisations:
• I fixed the new tag cache's slow tag autocomplete when in 'all known files' domain (which is usually in the manage tags dialog). what was taking about 2.5 seconds in 424 should now take about 58ms!!! for technical details, I was foolishly performing the pre-search exact match lookup (where exactly what you type appears before the full results fetch) on the new quick-text search tables, but it turns out this is unoptimised and was wasting a ton of CPU once the table got big. sorry for the trouble here--this was driving me nuts IRL. I have now fleshed out my dev machine's test client with many more millions of tag mappings so I can test these scales better in future before they go live
• internal autocomplete count fetches for single tags now have less overhead, which should add up for various rapid small checks across the program, mostly for tag processing, where the client frequently consults current counts on single tags for pre-processing analysis
• autocomplete count fetch requests for zero tags (lol) are also dealt with more efficiently
• thanks to the new tag definition cache, the 'num tags' service info cache is now updated and regenerated more efficiently. this speeds up all tag processing a couple percent
• tag update now quickly filters out redundant data before the main processing job. it is now significantly faster to process tag mappings that already exist--e.g. when a downloaded file pends tags that already exist, or repo processing gives you tags you already have, or you are filling in content gaps in reprocessing
• tag processing is now more efficient when checking against membership in the display cache, which greatly speeds up processing on services with many siblings and parents. thank you to the users who have contributed profiles and other feedback regarding slower processing speeds since the display cache was added
• various tag filtering and display membership tests are now shunted to the top of the mappings update routine, reducing much other overhead, especially when the mappings being added are redundant
• .
• tag logic fixes:
• I explored the 'ghost tag' issue, where sometimes committing a pending tag still leaves a pending record. this has been happening in the new display system when two pending tags that imply the same tag through siblings or parents are committed at the same time. I fixed a previous instance of this, but more remained. I replicated the problem through a unit test, rewrote several update loops to remain in sync when needed, and have fixed potential ghost tag instances in the specific and 'all known files' domains, for 'add', 'pend', 'delete', and 'rescind pend' actions
• also tested and fixed are possible instances where both a tag and its implication tag are pend-committed at the same time, not just two that imply a shared other
• furthermore, in a complex counting issue, storage autocomplete count updates are no longer deferred when updating mappings--they are 'interleaved' into mappings updates so counts are always synchronised to tables. this unfortunately adds some processing overhead back in, but as a number of newer cache calculations rely on autocomplete numbers, this change improves counting and pre-processing logic
• fixed a 'commit pending to current' counting bug in the new autocomplete update routine for 'all known files' domain
• while display tag logic is working increasingly ok and fast, most clients will have some miscounts and ghost tags here and there. I have yet to write efficient correction maintenance routines for particular files or tags, but this is planned and will come. at the moment, you just have the nuclear 'regen' maintenance calls, which are no good for little problems
• .
• network object breakup:
• the network session and bandwidth managers, which store your cookies and bandwidth history for all the different network contexts, are no longer monolithic objects. on updates to individual network contexts (which happens all the time during network activity), only the particular updated session or bandwidth tracker now needs to be saved to the database. this reduces CPU and UI lag on heavy clients. basically the same thing as the subscriptions breakup last year, but all behind the scenes
• your existing managers will be converted on update. all existing login and bandwidth log data should be preserved
• sessions will now keep delayed cookie changes that occured in the final network request before client exit
• we won't go too crazy yet, but session and bandwidth data is now synced to the database every 5 minutes, instead of 10, so if the client crashes, you only lose 5 mins of login/bandwidth data
• some session clearing logic is improved
• the bandwidth manager no longer considers future bandwidth in tests. if your computer clock goes haywire and your client records bandwidth in the future, it shouldn't bosh you _so much_ now
• .
• the rest:
• the 'system:number of tags' query now has greatly improved cancelability, even on gigantic result domains
• fixed a bad example in the client api help that mislabeled 'request_new_permissions' as 'request_access_permissions' (issue #780)
• the 'check and repair db' boot routine now runs _after_ version checks, so if you accidentally install a version behind, you now get the 'weird version m8' warning before the db goes bananas about missing tables or similar
• added some methods and optimised some access in Hydrus Tag Archives
• if you delete all the rules from a default bandwidth ruleset, it no longer disappears momentarily in the edit UI
• updated the python mpv bindings to 0.5.2 on windows, although the underlying dll is the same. this seems to fix at least one set of dll load problems. also updated is macOS, but not Linux (yet), because it broke there, hooray
• updated cloudscraper to 1.2.52 for all platforms

• version 424

• new tag caches:
• as 2020 ended, I attempted but failed to tune fast search for all kinds of clients, big and small and simple and complex. unable to guarantee decent speeds with just code, I have redesigned the tag text search cache. rather than checking the gigantic master table for all namespace and subtag lookups, the client can now zoom in on a small fast cache limited to the current search context, so doing a clever lookup on 'my tags' will no longer be hampered by having PTR beside it, and doing a solid lookup on the PTR or 'all known tags' will no longer be accidentally hampered by an optimisation for another situation
• the 424 update will take some time to generate the new caches for your existing data. if you don't sync with the PTR, it should be a few seconds. if you do sync, it will be about ten minutes on an SSD (seems about 30,000 definitions a second), and somewhat longer on an HDD. it will count up the tags as it goes, and on the PTR there will be a bit of deletion work, then one or two counts up to perhaps a million, and then one big count up to about 16 million.
• in my initial tests, this cache adds about 1-2% additional processing time to mass tag changes, but a wide variety of tag lookups and file searches are now significantly faster, have much nicer worst-case lag spikes, and should cancel quicker. these are best in any specific tag domain, although 'all known tags' should still be much better. a future expansion of the tag cache is planned to finally address clean and accurate 'all known tags' searches
• summary; all these should be faster and cancel faster:
• autocomplete searches for 'subtag*' (most normal searches) are optimised
• autocomplete searches for 'namespace:*' are optimised, including when the namespace itself is a wildcard
• autocomplete searches for wildcards with an asterisk in the middle of the subtag are optimised
• autocomplete searches for wildcards with an asterisk at the beginning of the subtag are optimised (but this is still generally the slowest query)
• autocomplete searches for namespace and subtag wildcard combinations are optimised, with either or both as a wildcard of any type
• autocomplete searches for '*' are optimised
• tag file searches without a namespace (i.e. in file search, with any namespace) are optimised
• namespace file searches are optimised, including when the namespace is a wildcard
• wildcard file searches are optimised, for all the classes of wildcard above
• 'tag as number' file searches are optimised
• 'has ><= x namespace tags' file searches are optimised for speed, including when the namespace is a wildcard, but still have bad cancelability on large domains. I'll work on this more
• .
• other tag cache info:
• the 'tag text search cache' regeneration routine under the _database->regenerate_ menu is replaced with a service specific routine for the new cache
• on boot, if the client sees any of the new cache tables are missing, it notifies you and regenerates the affected subsection of the cache
• an old method of performing complex wildcard searches was using surplus data and has been eliminated. these searches are now also computationally cheaper beyond the other domain-based optimisations this week
• I have identified the next bottleneck in the tag search pipeline and have a plan to speed all the above up even further, which can all be done in code
• thanks to user feedback, I have also identified other wasteful overhead in tag processing. I'll keep working!
• while the planned 'all known tags' cache will be useful since most file searches are in this domain, it will be a bit of work, so I will first let this new lookup cache breathe for a bit. 'all known tags' will not be nearly as big as the 'all known files/combined file' caches that have hit us with so much CPU recently. I expect it to increase the client.caches.db size by about 5%
• unified all increments or decrements to autocomplete count caches, no matter the service domain, to one location
• unified how autocomplete counts are fetched across different service domains
• optimised specific and combined autocomplete count cache update overhead for new, existing, and deleted tags
• optimised display autocomplete count cache updates for tags with multiple siblings or parents
• optimised the 'local tags cache', which does fast tag text fetching for local files, when new tags or files are added/removed from the 'all local files' domain. this now occurs in the same unified autocomplete count update process. it now also caches pending tags that have no current count
• merged 'exact match' autocomplete tag searching code into generalised wildcard search
• misc autocomplete and other tag code cleanup and harmonisation
• ditched some old mass UNION queries that were not cancelling well
• .
• the rest:
• when you paste queries into a sub, the summary 'these were/were not added' dialog now always appears, and if you paste empty whitespace, it now says so
• the manage siblings/parents dialogs now specify which services apply which siblings, whether they are fully synced, the current display tag sync maintenance settings, and ultimately whether you can expect changes to apply quickly after dialog ok
• when a text entry dialog comes with suggestion buttons, it now focuses the text box by default. sorry for the trouble here! (issue #765)
• updated a couple petition reason suggestions in manage tags and parents
• added a shortcut to 'main window' to refresh _manage tags'_ related tags suggestions with 'thorough' duration. in future, these dialog-specific actions will be moved out of 'main window', these have just been a 'temporary' patch
• updated the 'running from source' and 'install' help with some new numbers and info about mpv, and updated the 'server' help with a document helpfully provided by a user explaining that the server does not do what many new users think
• sped up 'has tags' file searches in certain situations, mostly when there are few if any other search predicates
• the default e621 parser now pulls meta tags, thank you to a user for providing this
• the default nitter timeline url classes are updated, thank you to a user for providing this
• the new little hook that takes 'file:///' off of paths pasted into the filename tagging path text now also normalises the path, so if you are on Windows, the URI's slashes will be Windows-corrected to backlashes. it also now removes wrapping quotes
• the hydrus logger again correctly restores stdout and stderr after it is closed on program exit (this was disabled for some reason, but fingers crossed it seems fine now!)
• an issue where automatically started duplicate potentials file search could not cancel when shutdown 'stop work' button was clicked or where idle maintenance mode turned off should be fixed
• the shutdown maintenance work for the first client shutdown now has a little text saying it is just some quick initialisation work
• for hopefully the last and completely final time, I think I fixed the invalid tag repair function for certain sorts of tags applied to currently local files
• improved the way a job thread was pulling new jobs (issue #750)

• version 423

• tag autocomplete searches:
• the 'fetch results as you type' and 'do-not-autocomplete character threshold' options are moved from _options->speed and memory_ to _tags->manage tag display and search_. they are now service specific!
• getting the raw '*' autocomplete is now hugely faster when both file and tag domains are specific (i.e. not 'all known xxx')
• getting the raw '*' autocomplete is now hugely faster in 'all known tags' domain. this is likely still bonkers on any decent sized client that syncs with the PTR, but if you have a small client that once synced with the PTR, this is now snappy
• the cancelability of 'namespace:*' and 'match namespaces from normal search' searches should be improved
• 'namespace:*' queries are now much faster in some situations, particularly when searching in a specific tag domain (typically this happens in manage tags dialog) or a small-file client, but is still pretty slow for clients with many files, and I think some scenarios are still bananas. I am not happy here and have started a plan to improve my service domain caches to deal with several ongoing problems with slow namespace and subtag lookup in different situations
• fixed an issue with advanced autocomplete result matching where a previously cached 'character:sam' result could match 'char:sam' search text
• some misc code cleanup and UI label improvements in autocomplete
• .
• the rest:
• the siblings & parents tag menu, which proved a little tall after all, is now compressed to group siblings, parents, and children by the shared services that hold them. it takes less space, and odd exceptions should be easy to spot
• this menu also no longer has the 'this is the ideal tag' line
• added 'sort pages by name a-z/z-a' to page right-click menu and tucked the sorts into a submenu
• the parsing test panel now shows up to 64KB of what you pulled (previously 1KB)
• the parsing test panel now shows json in prettier indented form
• when the parsing test panel is told to fetch a URL that is neither HTML or JSON, this is now caught more safely and tested against permitted file types. if it was really a jpeg, it will now say 'looks like a jpeg' and disable parse testing. if the data type could not be figured out, it tries to throw the mess into view and permits parse testing, in case this is some weird javascript or something that you'll want to pre-parse convert
• the dreaded null-character is now eliminated in all cases when text is decoded from a site, even if the site has invalid unicode or no encoding can be found (e.g. if it is truly a jpeg or something and we just end up wanting to throw a preview of that mess into UI)
• the 'enter example path here' input on import folders' filename tagging options edit panel now uses placeholder text and auto-removes 'file:///' URL prefixes (e.g. if your paste happens to add them)
• the 'fix invalid tags' routine now updates the tag row in the local tags cache, so users who found some broken tags were not updating should now be sorted
• added --db_cache_size launch parameter, and added some text to the launch_parameters help about it. by default, hydrus permits 200MB per file, which means a megaclient under persistent heavy load might want 800MB. users with megamemory but slow drives might want to play with this, let me know what you find
• updated to cloudscraper 1.2.50

• version 422

• advanced tags:
• fixed the search code for various 'total' autocomplete searches like '*' and 'namespace:*', which were broken around v419's optimised regular tag lookups. these search types also have a round of their own search optimisations and improved cancel latency. I am sorry for the trouble here
• expanded the database autocomplete fetch unit tests to handle these total lookups so I do not accidentally kill them due to typo/ignorance again
• updated the autocomplete result cache object to consult a search's advanced search options (as under _tags->manage tag display and search_) to test whether a search cache for 'char' or 'character:' is able to serve results for a later 'character:samus' input
• optimised file and tag search code for cases where someone might somehow sneak an unoptimised raw '*:subtag' or 'namespace:*' search text in
• updated and expanded the autocomplete result cache unit tests to handle the new tested options and the various 'total' tests, so they aren't disabled by accident again
• cancelling a autocomplete query with a gigantic number of results should now cancel much quicker when you have a lot of siblings
• the single-tag right-click menu now shows siblings and parents info for every service, and will work on taglists in the 'all known tags' domain. clicking on any item will copy it to clipboard. this might result in megatall submenus, but we'll see. tall seems easier to use than nested per-service for now
• the more primitive 'siblings' submenu on the taglist 'copy' right-click menu is now removed
• right-click should no longer raise an error on esoteric taglists (such as tag filters and namespace colours). you might get some funky copy strings, which is sort of fun too
• the copy string for the special namespace predicate ('namespace:*anything*') is now 'namespace:*', making it easier to copy/paste this across pages
• .
• misc:
• the thumbnail right-click 'copy/open known urls by url class' commands now exclude those urls that match a more specific url class (e.g. /post/123456 vs /post/123456/image.jpg)
• miniupnpc is no longer bundled in the official builds. this executable is only used by a few advanced users and was a regular cause of anti-virus false positives, so I have decided new users will have to install it manually going forward.
• the client now looks for miniupnpc in more places, including the system path. when missing, its error popups have better explanation, pointing users to a new readme in the bin directory
• UPnP errors now have more explanation for 'No IGD UPnP Device' errortext
• the database's boot-repair function now ensures indices are created for: non-sha256 hashes, sibling and parent lookups, storage tag cache, and display tag cache. some users may be missing indices here for unknown update logic or hard drive damage reasons, and this should speed them right back up. the boot-repair function now broadcasts 'checking database for faults' to the splash, which you will see if it needs some time to work
• the duplicates page once again correctly updates the potential pairs count in the 'filter' tab when potential search finishes or filtering finishes
• added the --boot_debug launch switch, which for now prints additional splash screen texts to the log
• the global pixmaps object is no longer initialised in client model boot, but now on first request
• fixed type of --db_synchronous_override launch parameter, which was throwing type errors
• updated the client file readwrite lock logic and brushed up its unit tests
• improved the error when the client database is asked for the id of an invalid tag that collapses to zero characters
• the qss stylesheet directory is now mapped to the static dir in a way that will follow static directory redirects
• .
• downloaders and parsing (advanced):
• started on better network redirection tech. if a post or gallery URL is 3XX redirected, hydrus now recognises this, and if the redirected url is the same type and parseable, the new url and parser are swapped in. if a gallery url is redirected to a non-gallery url, it will create a new file import object for that URL and say so in its gallery log note. this tentatively solves the 'booru redirects one-file gallery pages to post url' problem, but the whole thing is held together by prayer. I now have a plan to rejigger my pipelines to deal with this situation better, ultimately I will likely expose and log all redirects so we can always see better what is going on behind the scenes
• added 'unicode escape characters' and 'html entities' string converter encode/decode types. the former does '\u0394'-to-'Δ"', and the latter does '&'-to-'&'
• improved my string converter unit tests and added the above to them
• in the parsing system, decoding from 'hex' or 'base64' is no longer needed for a 'file hash' content type. these string conversions are now no-ops and can be deleted. they converted to a non-string type, an artifact of the old way python 2 used to handle unicode, and were a sore thumb for a long time in the python 3 parsing system. 'file hash' content types now have a 'hex'/'base64' dropdown, and do decoding to raw bytes at a layer above string parsing. on update, existing file hash content parsers will default to hex and attempt to figure out if they were a base64 (however if the hex fails, base64 will be attempted as well anyway, so it is not critically important here if this update detection is imperfect). the 'hex' and 'base64' _encode_ types remain as they are still used in file lookup script hash initialisation, but they will likely be replaced similarly in future. hex or base64 conversion will return in a purely string-based form as technically needed in future
• updated the make-a-downloader help and some screenshots regarding the new hash decoding
• when the json parsing formula is told to get the 'json' of a parsed node, this no longer encodes unicode with escape characters (\u0394 etc...)
• duplicating or importing nested gallery url generators now refreshes all internal reference ids, which should reduce the liklihood of accidentally linking with related but differently named existing GUGs
• importing GUGs or NGUGs through Lain easy import does the same, ensuring the new objects 'seem' fresh to a client and should not incorrectly link up with renamed versions of related NGUGs or GUGs
• added unit tests for hex and base64 string converter encoding

• version 421

• misc:
• thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation
• fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again
• fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices
• prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start
• when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)
• the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later
• the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on
• fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist
• I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search
• when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast
• fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later
• reduced overhead of tag filtering, which should improve display speed of taglist for very large pages
• parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster
• .
• duplicates search improvements:
• potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going
• the search distance in duplicates pages is now synchronised across all pages--when one updates, they all do
• all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often
• misc cleanup for duplicates page
• .
• database modes:
• a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
• added the '--db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY
• ensured --db_synchronous_override was hooked up correctly
• the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code
• fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true
• db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_
• .
• technical database nonsense:
• PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn
• hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done
• hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see
• in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations
• when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour
• after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%
• the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated
• improved launch switch code generally
• boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values
• the client and server no longer disconnect from the db to check whether it is possible to vacuum databases

• version 420

• misc:
• fixed the bad position indexing when drag-selecting taglists that were scrolled down. this also caused some weird selection when scrolled and clicks included a little mouse movement. sorry for the trouble!
• ctrl+drag-select now deselects!
• fetching tag autocomplete results when you have thumbnails and 'searching immediately' on, which has been way too slow recently, now cancels much faster. in some large page situations, it was adding multi-second lag on the first character-press. it also runs faster overall
• hydrus should now deal better with invalid tags that contain the null character (there is one we know about on the PTR, from a decode of botched Shift JIS, which could crash the client from too many errors during critical paint periods). when a tag like this turns up in a taglist, thumbnail, or canvas background, it now renders as an appropriate 'invalid tag' string, and a one-time 'woah, bad tag, run fix tags now' popup appears
• regular tag cleaning now looks for and removes null characters, so all new sources of these bad tags should now be eliminated
• _database->check and repair->fix invalid tags_ now fixes tags with the null character. it also fixes tags so broken that after cleaning they have no subtag left. it also now forces a full media tag reload when it is done for all media
• the 'regen storage mappings', 'regen display mappings', and repopulate from cache' database routines now have an additional step where you can order them only to work on one tag service, so regenning or repopulating local tags, which usually takes a couple seconds, doesn't need to wait two hours for the PTR to go as well
• added some menu help to the 'profile modes' debug menu, and gave 'reducing program lag' help page a pass
• fixed virtual display regeneration on service delete
• .
• parents and siblings:
• fixed situations where some grandparent and sibling relationships would not appear in the virtual system. it was a bug when certain links of a multi-part display 'chain' were updated at different times. when repopulating chain data, the sibling and parent update routines now correctly chase their complete chains both when wiping ideal data and repopulating from raw data, hitting all levels of the chain, ensuring to go back up and down chains when there are multiple grandsiblings/children/parents, and chasing parents where one or both members have better siblings. thank you to the several users who reported and helped figure out this problem, which was not simple to reproduce (issue #725)
• your ideal display data will be regenerated on update, which should not take more than a couple of seconds. it will likely correct some siblings and add some grandparents to be filled in by the siblings/parents sync. my PTR test environment went up from about 189,000 display rows to 192,000
• while sibling and parent lookup is more thorough (and hence more expensive), I also optimised many parts of lookup week. I believe tag display sync and tag processing will be much faster for tags with simple sibling and parent relationships, and slightly slower for tags with complex relationships and many instances to files on your drive. as always, let me know what sort of processing speeds and lag you get, and if you know how to make a db profile, please send them in when it gets bad
• when a 'write' autocomplete results list includes parent expansion rows (as in _manage tags_), parents now show duplicated and properly for all the tags that have them, including siblings and other children/grandchildren (previously, a parent label could only exist once in a list, which meant parents were ending up hanging off the last valid tag for which they applied)
• 'write' autocompletes now show results that exactly match the text entry, and all their siblings, when they do not have count but do have sibling or parent data. so, if you type in 'samus aran', and it has a sibling to 'character:samus aran', but 'samus aran' doesn't actually have count, you now get it and all siblings anyway. this may need tuning, but it solves a persistent and annoying lookup and quick-sibling-access problem in _manage tags_
• copying tags and their indented parents now removes the parent indent whitespace
• tag sync display now takes way longer breaks (now 30 seconds, was 2.5) between 'normal' background work periods. this thing was hammering people far harder than needed and could clog up db write/commit time and nobble UI responsitivity when big bumps collided
• the tag display maintenance manager now also tries to detect when many siblings or parents are streaming in (from a migration or a repository process with a heap of data), and pauses work while that continues
• greatly sped up mass imports of sibling and parent data, either from tag migration or big dialog pastes. what was 40 rows/s should now be about 1,000 rows/s
• fixed the database menu's 'regenerate tag parents lookup cache', which wasn't hooked up
• .
• boring changes:
• gave tag parents and siblings update, regen, and chain fetch a full pass, correcting bad queries to fix the above, fixing raw pair chain level navigation and parent-sibling idealisation, and optimised these lookups as well
• fixed some tag_id vs ideal_tag_id nomenclature (and related bugs) in tag parents cache
• optimised 'all known tags' autocomplete count fetching a little. tag autocomplete and search should be a bit faster in this domain
• reduced display sync pre-processing overhead by about 30% with a better random pair sampling routine
• reduced the overhead of my now very commonly used single integer memory table select optimisation. this now recycles tables after use, which reduces overhead about 50% in small number scenarios. all features of the database will enjoy this speed improvement, particularly small repetitive tag lookup jobs (such as the new display sync and repository tag processing)
• reduced overhead on some sibling chain lookup code
• reduced overhead on the sibling lookup used by manage tag dialog taglist
• reduced overhead on some parent chain lookup code
• tiny optimisation on single sibling chain lookup
• sped up the ancient OG single tag->tag-id fetching routine, seems to work about twice as fast now
• more misc optimisations, mostly list/set/dict comprehension rewriting to reduce overhead, across virtual sibling and parent code
• added a full combined siblings and parents unit tests for the main missing parent chain link problem solved this week
• added a full combined siblings and parents unit test for large real world data added in multiple pieces
• 'a file identifier was missing!' critical errors now print a stack trace to the log for further debugging info
• updated the 'help my db is broke.txt' document with a couple new comments

• version 419

• tag lists and editing predicates:
• you can now set the default value for any editable system predicate. a star button beside each panel lets you set or clear the custom default
• all editable system predicate panels now put 'recent' predicate buttons up top, for the five most entered predicates of the respective types. this is a little jank and grows pretty tall with multi-pred-type panels, but let me know what you think
• all tag lists now support drag-selection!
• taglists now have 'open a new OR page' menu entry when more than one tag is selected
• when taglists can change the current search, they now have an 'add an OR to current search' menu entry when more than one tag is selected
• OR Predicates are now editable! they launch their own little autocomplete input that is a little jank because you can technically make nested ORs, but it works!
• system:rating is now editable! it launches the whole stack every time. the stack alignment is messed up though :/
• invertible predicates (inbox/archive, tag/-tag, etc...) now flip on double-click only if you have one selected. if you have more than one selected, they appear as invertible buttons along with the rest of the edit UI
• the active search predicates taglist now has an 'edit search terms' menu entry, if you find shift+double-click a pain
• when you shift+double-click on more than one tag to add them to the current search, this is now added as an OR
• similarly if you shift+middle-click on more than one tag, the new page is now an OR
• when editing predicates, edited predicates now stay selected
• shift+clicking on an already selected tag no longer adds any new selections (i.e. shift+click filling-in). this should make it nicer to do shift+double-click on selections. furthermore, the 'last clicked' focus ghost (from which a shift+click selection cascade starts) on tag lists is now cleared on edits or removes, which should reduce some other crazy/annoying select behaviour here
• the list of active search predicates now correctly initialises sorted
• entering hex hashes into system:hash or :similar_to now has unified hash parsing, auto-removes 'md5:'-style prefixes, and presents detailed error information when a hash is too long or short
• .
• faster and snappier file and tag searching:
• searching for files by complicated wildcard (i.e. a search phrase that includes an asterisk in a non-rightmost character position) is now greatly optimised when the tag does not start with an asterisk (e.g. 'sm*l' is now much faster, '*all' is still hellmode), and now cancels (due to hitting the stop button or changing the query before results come in) much faster thanks to a new unified results fetching and cancel-checking routine
• rewrote my autocomplete tag search to use the new namespace and subtag lookup code from the virtual siblings and parents system, unifying lookup logic and benefitting from the same new complicated wildcard optimisation and fast-cancel tech
• autocomplete tag count aggregation (a later step, after the initial lookup) benefits from a little faster cancel tech
• all file queries based on tag, wildcard, namespace, tag count, and tag existence now use the new fast-cancel tech. if you put in a 'has >4 tags' query and it is taking ages, changing the query or just hitting the 'stop' button should now free up the db pretty fast
• related tags suggestions also gets the cancel tech and is now more timing precise for tags with either huge or tiny count
• .
• client api:
• the /get_files/file_metadata call now returns a service_names_to_statuses_to_displayed_tags structure, which reflects the sibling-collapsed and parent-added tags, as displayed to the user in UI. the help is updated to reflect this
• the client api version is now 15
• .
• the rest:
• fixed an issue where regenerating the tag definition search cache would not tidy up the 'I am busy' modal dialog once it was done, resulting in a soft lock
• fixed another upnp error handling bug, this time in the upnp daemon
• updated Qt to 5.15.2 on Windows and Linux builds. this should fix the unusual button clicking area problem for some custom styles
• .
• boring specific code changes:
• wrote widgets to edit invertible preds and OR preds
• pulled the messy rating code out of the rating system predicate ui code to their own widgets
• wrote some special predicate ui definitions and initialisation handling for OR preds and grouped 'multiple' preds (for ratings)
• refactored search and predicate ui code to a new 'search' module
• refactored collect and sort widgets away from search code
• misc layout improvements for system pred edit ui

• version 418

• almost all system predicates are now editable if you shift+double-click them! you can also edit several at once in the same dialog
• if you double-click on any predicate type that is not editable but does have an inverse version (e.g. archive/inbox, has audio/no audio, and tag/-tag), the inverse version(s) will be swapped in
• all legacy custom system predicate defaults are eliminated this week. the panels now show a fixed default on launch, and will get a flexible favourites system in future, along with 'recently entered' quick buttons
• restored the 'show system:everything' and 'hide archive/inbox' options, which were inadvertantly hidden when file system predicate defaults were hidden, to the new _options->search_ panel
• fixed the borked list height for the file viewing statistics system pred panel checkbox lists
• fixed an issue where namespace:anything predicates would not propagate to new pages on 'open page with these tags' commands
• .
• boring code specifics:
• updated almost all the system predicate panels to take arbitrary initialisation values, and wrote a 'can I edit this' test for all predicate types to help some finnicky which-panel-and-pred-to-use issues
• wrote some new filtering code and a little UI to handle editing of system preds
• cleaned up some of the taglist item activation code

• version 417

• the hydrus network version is increased this week from 18 to 19. clients and servers can only talk to each other when they are on the same version, so please update your clients if you wish to keep talking to the PTR, and your own servers if you have a home network setup or similar. if a server and client are on different versions, you will get a polite error when they next try to talk, and sync will be paused
• added 'run all export folders now' shortcut command to 'main window' shortcut set
• added shortcuts to the 'main gui' shortcut set for navigating the currently selected page. you can move left, right, to the leftmost on the current row, or to the rightmost. the left and right will cycle up a page of pages layer when at left/rightmost boundaries, letting you iterate through all pages in a depth-first manner
• updated the default newgrounds parser to deal with artists with more than 60/70 items in one art gallery (essentially, some clever 'next page' fetching now occurs to get older info that in your browser is drawn in as you scroll down). if you have some subscriptions for artists where you know this is true, try doing a full reset on them
• added realbooru to the hydrus defaults. they also apparently just switched away from a gelbooru 0.2.x site, so if you have a gelbooru parser with a realbooru example URL, I remove that example URL
• updated the page initial media load routine to my new async job
• updated the imported file presentation page-publish routine to my new async job
• when drag and drop or import file presentation now wants to add files to a page that is not yet fully loaded (rare, but possible for large sessions), that page now remembers the files it should add and appends them once load is done. these files-to-be-added are also preserved through a session save, if the client is closed before this long-loading page is initialised
• updated windows mpv, the reported api version is now 1.09
• updated windows ffmpeg to 4.3.1
• updated windows release to sqlite 3.33.0
• updated windows opencv to 4.4.0
• just a little thing--I took the source links out of the release post. anyone running from source is probably pulling straight from the github repo anyway
• cleaned up some misc inelegant string code
• misc other cleanup
• .
• macOS shortcuts:
• the client's shortcut system now detects macOS-specific 'scroll start/end' states, and will not spam scrolls or errors when these states are held
• the client's shortcut system now attempts to detect artificial trackpad scroll/wheel events, and adapts the relative speed of scroll event generation according to the respective trackpad velocity. let me know how the hell this works for you in media viewer etc... (issue #710)
• the client's shortcut system now detects Control and Command as separate and reliable modifiers in macOS, with correct shortcut string rendering (issue #717)
• .
• upnp:
• fixed the awful typos in the upnp add-mapping error handling I changed last week. I am sorry for this!
• improved the async mappings and external ip fetch routines in upnp dialog. closing the dialog while a job is going on should now be completely ok
• upnp dialog add, edit, and delete actions are now async (they won't hang the UI while they work)!
• all the upnp async jobs should now disable the main list controls while they work
• fixed the 'edit' action on upnp dialog to correctly remove old and existing mappings depending on what was edited
• when adding a mapping for an (external_port,protocol) that is already mapped, the upnp dialog now asks if you want to overwrite, rather than just failing with a notification
• after an async action in upnp dialog, and a mappings refresh triggered, the cached external IP should now be properly restored to the status area
• pulled parsing code out of upnp code and wrote some proper unit tests for this so stupid typo errors should not happen again
• .
• parsing:
• subsidiary page parser separation formulae that throw an exception will now be ignored, as if they parsed nothing. in the weird case that you might receive json or html, you can now create subsidiary parsers for both types, and the one that fails will do so gracefully and silently
• URL Classes now have a key->value 'header override' value. any time one of these URLs is hit, these headers are added!!! be careful with this, but it may solve some tough problems. also, sorry, the URL Class UI is becoming a hellstack, I need to break it into tabs or similar
• .
• client api:
• added documentation for the new add_files commands, delete_files, undelete_files, archive_files, and unarchive_files
• added unit tests for the new commands

• version 416

• misc:
• the new siblings and parents taglist menus now copy just the actual tag when you click, excluding the 'ideal/child/parent:' prefixes
• added a checkbox to _options->files and trash_ that allows you to automatically prefix hashes copied to clipboard with their hash type in a booru-lookup friendly manner, such as "md5:2496dabcbd69e3c56a5d8caabb7acde5"
• the media viewer now remembers if it was previously maximised when you set it to un-fullscreen (before, it would always restore-window-ise)
• fixed the 'test address' button in _manage services_ for hydrus administration services
• improved the 'add upnp mapping' error handling to better catch 'already mapped' error, with separate errors for redundant, already-on-but-wrong-port, and already-on-another-computer
• improved error handling when saving objects to the database, particularly for encoding or giganto-size-session errors
• rewrote my tag sibling lookup unit tests to deal with more situations
• wrote similar fairly comprehensive tag parent lookup unit tests
• .
• new downloaders:
• rolling in a user-created thread watcher for warosu. it may be CloudFlare hampered depending on your situation
• rolling in a prolikewoah thread watcher
• rolling in a smuglo.li thread watcher
• .
• multi-column lists:
• spent a bunch of time cleaning out how I calculate multi-column list preferred initial width/preferred current width/minimum width, and made the final column more flexible in its resizing. instances of dialog suddenly getting gigantic because of a final column that wants to size itself at 1,000px should be completely gone, and lists that are shrunk due to non-last-column resizing will now adapt to this situation and not try to flex back to total initial width.
• multi-column lists now have horizontal scrollbars again for those situations where the parent window is thinner than their (now better calculated) minimum size
• improved the multi-column list num_rows height calculation, it should have less empty space at the bottom for lists that grow as items are entered into them (such as in the download pages)
• .
• manage tags megajob speedup:
• sped up manage tags final application step when entering many tags for many thousands of files at once
• optimised UI-side per-file tag cache (re)generation, reducing overhead and surplus work
• granularised UI-side per-file tag cache (re)generation based on the four current tag display contexts--now, if a system (e.g. manage tags dialog) only needs storage tags, the different display tags do not need to be regenerated
• optimised all tag filtering, which is also used in UI-side tag cache regen
• overall, giganto manage tag dialog jobs should now be faster in several ways. on my dev machine, adding 6 tags to 10k reasonably tagged files went down from 52s to 4.8. even larger jobs will still need a lump of CPU time, but they should scale more efficiently (what was previously O( num_tag_changes x num_total_mappings ) is now O( num_total_mappings ), and better at that)
• when a huge number of tags is added at once in the manage tags dialog, 'recent tags' is now populated more carefully

• version 415

• in _options->gui pages_ you can now set the main window's page tab alignment to top/left/right/bottom (previously it was just top/left). this property now updates for all page of pages on options ok, it no longer needs client restart (issue #642)
• the maintenance task that migrates tag display from the current values to the ideal application now works in significantly smaller steps. big lag from adding hundreds of childen to one parent (or similar siblings) should now be radically reduced
• rejiggered some layout in the new tag display dialogs
• added green/red texts to the new tag display dialogs to talk about when sync can work atm and how fast to expect changes to apply
• reordered the new tag 'siblings/parents info' right-click menu so the dynamic 'has x siblings/parents' submenus are on the bottom
• added basic client api calls for /add_files/..., delete_files, undelete_files, archive_files, and unarchive_files. they take 'hash' and 'hashes' parameters. I am throwing these out at the end of the week, so they don't have documentation or proper unit tests, but feel free to play with them (issue #393)
• sped up some UI refresh on content update for very large sessions
• sped up right-click tag/file menu any/all select actions on very large file sessions

• version 414

• tl;dr: you don't have to do anything. if you haven't heard of a tag parent before, no worries. the database should work better now
• .
• top level:
• parents are now completely virtual! this means that when you add a tag parent, the tags that 'fill in' to make it show do not really exist in storage, only in a computed cache. if you decide to undo the parent, the implications are recalculated and the virtual tags disappear, with no permanent changes made. also, petitioning a parent will 'preview' the delete, just as siblings now does
• siblings and parents are now unified, and the logic is improved. all parents apply to all siblings, so no more worries about retro-active filling-in. the siblings and parents code is now basically 'nice'. this was a lot of quite complicated work, and it solves a number of lingering issues from the original prototypes I made several years ago. I will still do some smaller work and little fixes I am sure in the near future, but the 'big' siblings and parents work is done!
• like with the recent siblings change, the client no longer needs to do the 'loading parent tags' step when booting--everything is now handled at the db level
• like with the recent siblings change, you can now edit which services apply their parents to which service, now under _services->manage where siblings and parents apply_
• in the _manage tags_ dialog (and some other places), tags with parent implications now show a '(x parents)' after their label, much like the 'will display as' sibling suffix. I do not like this, but I ran out of time. I hope to add a more advanced actual listing of virtual tags with a nice 'ghostly' colour or similar in future
• right-clicking on a tag in a specific tag domain now shows a 'siblings and parents' submenu with detailed info on all known siblings and parents in that domain
• 'tag' menu entries are moved from the top 'services' menu to a new 'tags' menu. 'pending', when available, is also moved right
• the process of changing siblings or parents, or which services apply where, is no longer a CPU-laggy process! actual changes, however, may not appear immediately. a maintenance task now tracks what is currently applied and what is 'ideal', and slowly migrates to the ideal in the background in little chunks. in most situations, the changes are very quick, but if you are behind due to big recent changes, they may be delayed. you can manage when this maintenance runs and see the current status under _tags->siblings/parents sync_. this is an entirely new thing, so feedback on IRL work would be appreciated--there may be some kinds of siblings or parents that cause a whole bunch of annoying lag
• the PTR has a lot of non-virtual parents that were hard-added in older versions over the years. most are fine, but some are like the 'shadow'->'shadow the hedgehog' debacle. now the source of the problem is fundamentally solved, this problem will reduce over time. with luck, before the end of the year, no more will be added at all, and thanks to the janitors, the worst offenders should be chipped away
• during all this work, a bug with tag siblings and parents repository processing has been revealed (some users do not 'get' all siblings/parents for some reason). now the system is nice and undoable, this will be more easily addressed in coming weeks, with automatic retroactive fixes rolling out to all clients
• .
• boring details:
• like with siblings, wrote a parents structure object that constructs the parents tree without loops more simply and reliably. it populates a new parents quick-lookup table in the database, for which a full suite of lookup and maintenance methods are written
• parents and siblings virtual tag presentation is now unified into a single 'display' (i.e. vs 'storage') system with a more granular tag implication algebra (essentially 0-n rows of 'if A is in storage, show B in display' for every tag) that can calculate new and updated display tags and counts without having to do the expensive 'clear-and-regen' that 408-413 used
• wrote functions to quickly add or remove a display implication to the 'all known files' or specific file service tag display cache
• migrated all the combined and specific tag display cache update code (add/remove files, add/remove mappings, add/remove sibling/parent, add/remove sibling/parent application, and misc regen maintenance calls) to use the implication system instead of the sibling 'ideal' system (basically moving from 1->1 to 1->n)
• completely rewrote the complicated 'all known files' cache 'with tags' and 'with and without tags' lookup routines to use much less overhead in general and to use a single, albeit complicated, count-based query that carefully chooses whether to select the 'with tags' and 'without tags' portions using tags or files where available as the primary selector based on existing autocomplete count data
• replaced all usage of the old ui-side 'tag parents manager' object. as parents pop in virtually and do not need to be bundled intentionally to various content updates, this was mostly just clearing now-surplus code, but for instance in 'write' autocomplete searches, the parents that appear below search results are now generated at the db level on first search, rather than looked-up live in UI time
• the parents and siblings lookup tables are now split into two views: what the display cache currently holds, and what it ideally should hold. when adding new sibling or parent data, only the fast ideal table is changed
• a new complicated maintenance function now takes actual and ideal data for a particular unsynced tag, hashes out the implication changes needed to effect a migration, and performs it
• a new maintenance manager and accompanying db code now track and manage calls to migrate actual to ideal display presentation, and to update UI afterwards
• as tag display changes are now more frequent, I have made the routine that refreshes tag UI after sibling/parent changes more efficient. tag display now only refreshes for files that have the affected tags in a particular change
• wrote the UI panel and dialog to show and hurry up current sync status, and all the background hooks for that
• added 'tag parents lookup' entry to the database 'regenerate' menu. this routine and the 'siblings' variant are now very quick thanks to the new actual/ideal maintenance system
• updated my sibling unit tests to deal with the new actual->ideal syncing
• improved the speed of mappings cache updates when deleting files
• deleted all the old combined/specific 'regen chain' code and the sibling-based 'get sole/any tagged files' search code
• optimised and generally cleaned a bunch of the new cache code, particularly cutting out overhead for unusual/small situations
• fixed a counting bug with 'all known files' tag counts when rescinding pending tags
• fixed a bug in the siblings display code where deleting or pend-rescinding all of the multiple tags that have the same ideal sibling in the same transaction (e.g. if both A and B sibling to C, and a file has both A and B, and you remove them in one manage tags dialog apply) would not remove the current/pending ideal (issue #571)
• the 'add_siblings_and_parents' parameter on /add_tags/add_tags client api command is now obsolete! the help is updated to reflect this
• cleaned up just a bunch of db/ui/tag code mate
• .
• the rest:
• fixed an issue where long-running 'similar files' search was not cleaning its memory use properly as the job was going on, resulting in out-of-mem errors on very large clients (issue #669)
• thanks to user submission, rolling in a fix for the default pixiv tag search downloader
• cloudscraper updated to 1.2.48
• removed surplus executables from linux and macOS builds (win32 upnpc exe was causing anti-virus false positive on mac lmao)

• version 413

• added 'sort by number of files in collection' file sort type. it obviously only does anything interesting if you are collecting by something
• when you enter a tag from a manage tags suggested tags column with a double-click, the tag input box is now immediately focused. entering it with a keyboard action does not move the focus
• wrote a new routine for the 'check and repair' database menu that scans for and fixes invalid tags. this might be some system:tag that snuck in, superfluous unicode whitespace, or some weird website encoding that results in null characters, or any other old tag that has since become invalid. tag translations are written to the log
• added an experimental 'post_index' CONTEXT VARIABLE to subsidiary page parsers--whenever a non-vetoed post has pursuable URLs, this value is incremented by one. this is an attempt to generate a # 0,1,2,3 series. feedback on this would be appreciated, so I can formalise and document it
• added 'no_proxy' option for the options->connection page. it uses comma-separated host/domains, just like for curl or the NO_PROXY environment variable. it defaults to 127.0.0.1. in future, options will be added to auto-inherit proxy info from environment variables
• fixed an error when subscriptions try to publish to a page name when a 'page of pages' already has that name
• activated some old 'clean url' parsing tech I wrote but never plugged in that helps parsing urls from source fields on sites that start with non-url gubbins
• fixed the v411->v412 update step to account for a tags table that has duplicate entries (this shouldn't ever happen, but it seems some legacy bug or storage conversion indicent may have caused this for some users). if a unique constraint error is raised, the update step now gives a little message box and does dedupe work
• fixed an issue where the 'will display as' tag was rendering without namespace when 'hide namespace in normal views' was on
• fixed a recent character encoding routine that was supposed to filter out null characters
• fixed some UPnP error reporting
• _may_ have fixed an odd and seemingly rare 'paintevent' issue when expanding the popup toaster from collapsed state--it may also have been a qt bug, and fixed in the new qt:
• updated qt to 5.15.1 for windows and linux builds. it fixes a couple of odd issues like 'unclicking' to select a menu item (issue #296)
• added session save to holistic ui test suite
• misc code cleanup
• .
• client api:
• wrote a client test for the help menu so I can test some basic functions holistically, hoping to stop some recent typo bugs from happening again
• did a couple of hotfixes for v412 to deal with some client api url pending bugs. the links in the 412 release now point to new fixed builds
• fixed an issue setting additional tags via the client api when the respective service's tag import options are not set to get anything
• fixed a 500 error with /add_tags/add_tags when a tags parameter is an empty list
• fixed the /manage_pages/get_page_info client api help to show the 'page_info' key in the example response

• version 412

• client api:
• added Hydrus Web, https://github.com/floogulinc/hydrus-web, to the Client API page. It allows you to access your client from any web browser
• added Anime Boxes, https://www.animebox.es/, to the Client API page. This booru-browsing application can now browse hydrus!
• the /add_urls/add_url command's 'service_names_to_tags' parameter now correctly acts like 'additional' tags, and is no longer filtered by any tag import options that may apply. that old name still works, but the more specific synonym 'service_names_to_additional_tags' is now supported and recommended (issue #456)
• the /add_urls/add_url command now takes a 'filterable_tags' parameter, which will be merged with any parsed tags and will be filtered in the same per-service way according to the current tag import options.
• the client api help is updated to talk about this, and the client api version is now 14
• updated client api help to talk about http/https
• .
• the rest:
• the 407->408 update step now opens a yes/no dialog before it happens to talk about the big amount of CPU and HDD work coming up. it offers the previous 'full' version that takes all the work, and a 'lite' version that applies no siblings and is much cheaper. if you have been waiting on a PTR-syncing HDD client, this should let you update in significantly less time. there is still some copy work in lite mode, but it should not be such a killer
• the 'manage where tag siblings apply' dialog now has big red warning text talking about the current large CPU/HDD involved in very big changes
• a bunch of file-location loading and searching across the program has the opportunity to run very slightly faster, particularly on large systems. update will take a few seconds to make these new indices
• namespace and subtag tag searches and other cross-references now have the opportunity to run faster. update will take another couple of minutes to drop and remake new indices
• gave tag and wildcard search a complete pass, fixing and bettering my recent optimisations, and compressing the core tag search optimisation code to one location. thank you for the feedback everyone, and sorry for the recent trouble as we have migrated to the new sibling and optimisation systems
• gave untagged/has_tags/has_count searches a similar pass, mostly fixing up namespace filtering
• gave the new siblings code a similar pass, ensuring a couple of fetches always run the fast way
• gave url search and fetch code a similar pass, accounting better for domain cross-referencing and file cross-referencing
• fixed a typo bug when approving/denying repository file and mapping petitions
• fixed a bug when right-clicking a selection of multiple tags that shares a single subtag (e.g. 'samus aran' and 'character:samus aran')
• thanks to some nice examples of unusual videos that were reported as 1,000fps, I improved my fallback ffmpeg metadata parsing to deal with weird situations more cleverly. some ~1,000fps files now reparse correctly to sensible values, but some either really produce 1000 updates a second due to malformation or bad creation, or are just handled that way due to a bug in ffmpeg that we will have to wait for a fix for
• the hydrus jpeg mime type is now the correct image/jpeg, not image/jpg, thanks to users for noticing this (issue #646)
• searching for similar files now requires up to 10,000x less sqlite query initiation overhead for large queries. the replacement system has overhead of its own, but it should be faster overall
• improved error handling when a database cannot connect due to file system issues
• the edit subscription(s) panels should be better about disabling the ui while heavy jobs, like large subscription resets, are running
• the edit subscription(s) panels now do not allow an 'apply' if a big job is currently disabling the ui
• cancelling a manage subscriptions call when missing query logs were detected no longer causes a little error
• if a long-running asynchronous subscription job lasts beyond its parent's life, it now handles errors better
• .
• boring details:
• improved a pre-optimisation decision tool for tag search that consults the autocomplete cache for expected end counts in order to make a better decision. it now handles subtag searches and multiple namespace/subtag searches such as for wildcards
• wrote fast tag lookup tools for subtag and multiple namespace/subtag
• fixed some bad simple tag search optimisation code, which was doing things in the wrong order!
• optimised simple tag search optimisations when doing subtag searches
• polished simple tag search code a bit more
• added brief comments to all the new cross joins to reinforce their intention
• greatly simplified the multiple namespace/subtag search used by wildcards
• fixed and extended tag unit tests for blacklist, filterable, additional, service application, overwrite deleted filterable, and overwrite deleted additional
• added a unit test for tag whitelist
• extended the whole 'external tags' pipeline to discriminate between filterable and additional external tags, and cleaned up several parts of the related code
• moved the edit subscription panel asynchronous info fetch code to my new async job object
• cleaned up one last ugly 'fetch query log containers' async call in edit subscriptions panel
• moved the edit subscription(s) panels asynchronous log container code to my new async job object
• misc code cleanup

• version 411

• misc:
• fixed the 'system:(like/dislike) rating = x' search predicate string, which was saying 'unknown' rather than 'like/dislike' in several cases
• fixed a 'current_count' error in the new file search optimisation code for tag searches where the tag did not exist for any files in the domain (i.e. autocomplete count=0). thank you to users for helpful reports here
• fixed the recent file search optimisation code to handle 'system:time imported' when it was mixed with tags or search predicates that would pre-populate the query file pool with file domain cross-referenced files. sorry for the trouble!
• the forced delay overhead for table analysis is reduced from 0.1s to 0.02s. whenever many mostly empty tables need to be analyzed (like on first boot shutdown, when it is usually 100+ tables), it now zips by
• .
• siblings/tag improvements:
• typing a shorthand sibling like 'lotr' into an 'all known tags' 'read' autocomplete - like on a default search page - now reliably discovers and matches text entry to ideal sibling results like 'series:lord of the rings'. this was previously buggy and unreliable--it now allows the match using better db knowledge, even when the merged 'all known tags' services involved disagree on siblings
• when typing tags into a 'searching immediately' page that has media, the autocomplete count results that only refer to that media will now match shorthand sibling inputs to the ideal result. media-populated tag search now takes a little bump of extra CPU to fetch results (they are now passed through the db to get nice siblings info), so it is also cached for the duration of your typing (previously, the counts were re-computed on every new keystroke, so this should be significantly smoother to work with on large pages even if that first keystroke takes a moment to give results)
• when typing into a 'write' autocomplete, like in manage tags, the process that promotes the entry text and known siblings to that entry and a potential ideal sibling result to the top of the list is now more sane. it now also only uses results with nonzero count. we'll see how this last change works out IRL
• when typing into a 'read' or 'write' autocomplete, the pre-search tag insert no longer has sibling insertion/swapping. it was unreliable before, with weird sibling-swapping in the short moment before real results returned. if you have slow results and often quick-add tags into search pages or manage tags, let me know how this works for you
• the 'additional tags' tag input dialog off the tag import options edit panel now shows the 'will display as' label
• the 'favourite', 'file lookup', and 'recent' tag suggestion panels now show the 'will display as' label
• the 'related' suggestion panel, which works on a slightly different system, now shows the 'will display as' label
• the 'tag suggestions' options panel's 'favourite tags' edit lookup and list now displays 'will display as' labels and correctly finds service-specific siblings in its results (e.g. you type 'lotr', it also finds 'series:lord of the rings')
• all autocomplete tag filtering should be just that little bit faster as you type
• filtering cached autocomplete results based on subsequent search text is now faster
• autocomplete inputs should no longer return 'ghost' results that have no current/pending count when one of the 'include current/pending' buttons is deactivated
• the new database autocomplete predicate generation routine now checks for 'cancel search' signals, saving CPU time as you type
• the slow 'regen chains' maintenance tasks now process sibling chains in random order, smoothing out the 500/100,000 progress label, which previously took about 80% of time on the first 20% of ids due to IRL tag distribution
• .
• the last UI-side siblings work is cleared:
• the UI-side tag siblings cache is no longer used. the sometimes multi-second 'loading tag siblings' step of boot no longer happens!
• media autocomplete fetches are now asynchronously populated with siblings data via the db
• the exact-match and sibling 'insert' predicates at the top of pre-load and post-load read and write autocompletes now rely exclusively on db data for sibling matching
• taglists now present 'will display as' labels asynchronously and are better about updating those labels when the list's underlying tag service changes
• the taglist right-click menu that shows siblings to copy now fetches that submenu's contents asynchronously from the database
• the test panel on a blacklisting tag filter now asynchronously fetches tag siblings to test against from the database
• the actual blacklist tag filter test now fetches tag siblings to test against from the database
• reworked my custom tag listbox to handle asynchronous text decoration, and unified sibling decoration for media taglists and string taglists
• updated my old async updater class to be more flexible for different job types, and cleaned the code that already used it
• wrote a simple class for one-shot async jobs
• wrote a simple db lookup for UI-side tag sibling chain members
• wrote a simple db lookup for UI-side tag ideal siblings
• bunch of misc sibling, db, and ui work and cleanup to make all this work

• version 410

• general work:
• fixed a bug in the new file service filtering code that was stopping file upload commands to file repositories or ipfs services from sticking
• fixed an issue with the export files dialog auto-close-when-done function
• I think I fixed a possible bug in the boot file location repair/recovery dialog sometimes not saving corrected paths on unusual file systems
• file migration cancel button and shut off timer should work a bit more reliably, more to come here
• copying subscription quality csv info to clipboard no longer does nice human numbers (you now get 1234, not csv-breaking 1,234)!
• may have fixed a very rare 'or predicate' error when opening a dialog with a 'read' autocomplete input, like export folder or file maintenance jobs dialogs
• all pages are better about dealing with missing (i.e. recently deleted) services on load, and autocompletes also
• error handling from servers with strange character encodings should be better about dealing with null characters
• cleaned up the combined display regen chain code
• deleted some obselete db code
• .
• optimisation review:
• after more profiling, and thanks to additional input from users, I have done another round of optimisation for the new caches. using a new technique, more than just mappings are sped up this week - a number of queries that were prone to lag spikes should now have much more reliable speed and also be faster when hammered often
• .
• join and analyse db optimisations:
• these are mostly forcing table join orders, which reduces lag spikes, and reducing some related pre-query analysis overhead, which speeds things up more the faster your drive is (up to double processing speed on an ssd). they will affect different clients to different extents, but if your 'related tags' were taking more than a second to load, it should be sorted this week. systems affected:
• archiving files
• fetching 'related' suggested tags
• tag siblings regen/update in about ten places
• all mappings processing
• additional mappings processing for add/delete, pend/rescind_pend
• importing or deleting files that have tags
• loading medias' tags for the first time or on regen
• loading any media for the first time
• num notes searches
• similar files search tree maintenance
• many general file hash lookups
• many general tag lookups
• .
• other optimisations:
• mappings processing
• sibling processing
• wildcard tag searches, with and without namespaces, particularly when mixed with other search terms
• 'tag as number' searches, with and without namespaces, particularly when mixed with other search terms
• searching for tags when mixed with other search terms
• has notes/no notes
• searching files on 'all known files' with general file metadata system predicates (like size, filetype)
• url class, url domain, and url regex file searches, particularly when mixed with other search terms
• num tag file searches when mixed with other search terms
• has/not has tags file searches when mixed with other search terms
• sped up specific display chain regen significantly, with similar separate current/pending optimisations as last week's for combined
• converted specific display cache overall regen to use a copy followed by the new chain regen rather than additive file import
• sped up combined display chain regen a little bit
• the splash window now updates itself with less UI overhead, so spammy updates (like the new tag regen code) use a little less CPU and fewer UI context switches

• version 409

• siblings:
• the slowest of the new sibling regen & update code has received a full optimisation pass. some sections take 10% less time, some 90%, and one critical query takes 99% less time. overall, several big jobs work much faster, and ptr processing, which slowed significantly for many users, should be back up to a good speed. uploading pending tags (which tend to be for local files) should be much faster in particular. let's do another round of IRL observation and profiling this week, and I'll keep at it
• the various 'display' regeneration routines now provide more progress status text, drilling down to the x/y siblings being collapse-counted, or number of files added to a cache, and generally all tag sibling regen got a status update polish pass
• optimised the way tag sibling application is set--now, only the tag siblings that are changed need to have their counts regenerated. hence, if you just apply (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes
• .
• the rest:
• fixed the annoying issue with media viewer mouseovers stealing focus/activation from the manage tags dialog. this can now only happen if current focus is on a hover window. sorry for the delay!
• updated manage tag parents dialog to state the pairs being petitioned on the 'petition reason entry' dialog
• updated manage tag parents and siblings dialogs to have appropriate 'reason' suggestions for petitions (previously, they were inheriting the same suggestions as for add)
• ipfs network jobs now have a minimum 'reply' connection timeout of two hours (so giganto directory pushes won't throw an error). connection timeout remains the same, so if the server is hanging on that, it'll still notice
• fixed the 'test address' button on the IPFS manage services panel
• petitioning an IPFS file when there is no IPFS multihash entry in the db no longer causes an error. now, in this case, the file entry is removed with no change made.
• when pending to or petitioning from a file service, a quick filter is now applied to discard invalid files (i.e. (not) already in the service). any weird logical holes where this might occur should now be fixed
• export folders now catch and report missing file errors more nicely
• export folders now remember the last error they encountered and report that in the edit export folders dialog
• .
• boring tag siblings optimisations:
• optimised the tag manager generation routine to use any common file domains for fast cache lookup for any subset of the files available, rather than falling back to 'all known files' domain when there is no single common file domain
• optimised the new 'all known files' display autocomplete cache to use similar faster specific files cache lookups when available
• optimised how the 'all known files' display cache regenerates tag sibling chains. it now takes a shortcut when given non-sibling tags and tags where all but one sibling member have zero count, and it can count current and pending counts separately according to the most efficient counting method (e.g. most pre-display pending counts are 0 across the board, so even if current count is a million, the pending count can often be assumed without lookup overhead). furthermore, the 'clever' count has better query planning and less non-sqlite data overhead, and with experimental data is now chosen more carefully. what was previously a 22s job on a test database now takes 5s
• deduplicated how new mappings are filtered to all the specific cache domains, significantly reducing overhead
• massively optimised a critical - and the slowest - part of the new 'combined' cache that handles add/pend mappings pre-insert presence testing, speeding up the core query about 100x!
• reduced some overhead when doing file service_id normalisation in repository processing
• split up specific chain regen into groups to reduce memory usage
• optimised specific display tag cache 'add file' updates, and thereby basic cache regeneration, to be just a little faster for files that have multiple sibling tags
• all predicates made in the database are now populated with ideal and chain sibling information, and this is used for '(will display as xxx)' labels and autocomplete tag search filtering (e.g. you type in 'lotr', it matches an autocomplete result of 'lord of the rings'). there are still some ui-made predicates to figure out, so the old system remains as a fallback
• related tags lookup is a tiny bit faster and now populates its predicates with ideal and chain sibling info at the db level
• cleaned up some 'fetch related tags' code, might make it a bit faster for large tag counts
• cleaned up the way some mapping tables are fetched
• unified table/table_name nomenclature in the db code
• updated an old data->ui status presentation method (it typically does stuff like "regenning some stuff: 500/10,000"), to not hog so much UI time and not yield worker threads so often when new statuses are coming in real fast
• several late optimisations based on IRL testing

• version 408

• tag siblings cache:
• tl;dr: siblings are faster and better now, you don't have to do anything. some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon
• wrote the first version of a 'tag display' cache, which stores not your tags as they are, but how they appear after display rules such as siblings, parents, and filtering are applied, meaning this data need not be calculated every time on thumbnail load. this week marks the first concrete step forward in an improvement of siblings and parents storage, and begins with just siblings. all siblings and front-end tags work should be generally faster and more accurate
• part one is for tag domains cross-referenced with file domains. it maintains virtual sibling-collapsed mappings and autocomplete counts through mappings added, deleted, pended or pend-rescinded, files added/deleted, and siblings added/removed
• part two is for tag domains on the 'all known files' domain (i.e. no file domain). it maintains virtual sibling-collapsed autocomplete counts through mappings added, deleted, pended or pend-rescinded, and siblings added/removed
• both parts also support full table drop/regen (under the new database->regenerate->tag display mappings cache) for when my logic inevitably miscounts something. the existing regen 'tag mappings cache'/'tag siblings lookup' commands also regen the display mappings cache, since it relies on them
• when tag siblings on a repository are petitioned to be deleted, they are now instantly discounted from tag sibling application (previously, they had to be uploaded and committed to count, now both pending and petitioned offer a quick preview of outcome)
• the display cache supports the tag sibling service application rules under the 'services menu', and regen when that changes, so you can now turn siblings on and off, and apply them across services. as a result, the old 'apply all siblings to all services' option is now gone! as parents will undergo a similar change soon, and the siblings changes this week may lead to some undesired parents in the interim, the 'apply all parents to all services' option is also gone
• tag autocomplete counts in the form (x-y) due to siblings are eliminated. it will still do it when combining the same merged tag across different services, or when an unnamespaced tag includes how many potential namespaced will also be found
• the following search types now obey tag sibling application rules accurately: number of tags search, namespace:anything search, wildcard search, tag search (on a per-tag-domain basis, previously it was globally hacked to all siblings), tag-as-number search. for instance, if you search series:anything, a file that has 'metroid' tag-siblinged to 'series:metroid' will now correctly appear.
• the above search types are now exact to how the tag displays. if you have for files that are tagged 'samus' on either tag service A or B, and service B has a sibling for that to 'character:samus aran', searching for 'samus' gets the results in A, 'character:samus aran' gets the results in B. previously it was an expensive logical mish-mash of 'sure, try and get everything behind the scenes'. now it searches what you see
• searching for files in the advanced 'all known files' domain currently has no sibling support for the above search types. autocomplete counts should be good, and the results that come up should have the correct tag display, but the actual results are calculated based on storage tags. getting this to work without doubling the size of the db is tricky, so it will have to be ongoing work
• all tag siblings are now completely virtual. this means that when a tag comes in via a downloader or other means, it will not be automatically coerced to its ideal sibling in the actual db storage tables (the true tags you see in manage tags dialog), but remain as it is. there is no change in sibling appearance in normal operation--it still _displays_ and searches as its ideal sibling. the same will happen to parents in the coming months, and in the interim period, parents no longer apply across siblings. as siblings can come and go from anywhere, they are now divorced from actual stored tag mappings. in a similar way, the manage tags dialog no longer supports the 'hard-replace siblings and parents' command, nor the 'auto-replace with sibling' command. this may be jarring to workflows and preferences, so please bear with me and let me know what feels particularly bad. and please don't worry too much about parents not always being added in the meantime--I hope to do the same transition for them in four of five weeks, and all gaps will be filled in. also, in the coming weeks, I expect to improve manual tagging workflow by indent-grouping edit-view siblings together (ditching the old 'will display as' text) for easier review and selection, a bit like parents. actual 'hard' siblings and parents that do always get irreversibly renamed/added in storage will come in the future as a separate system
• the 'add_siblings_and_parents' client api parameter no longer adds siblings, and soon will be retired completely
• I had wanted to completely eliminate the old UI level siblings manager this week, but there are still some systems, mostly tag autocomplete work, that need it and are tricky to swap. I stripped it down, at least, and reduced its update delay to 2 seconds. therefore, the 'loading tag siblings' step of boot still occurs, albiet a _little_ faster. I hope to have it gone soon
• this is some complicated code affecting core systems. almost everything 'siblings' is now different. there are likely to be laggy parts, awkward new workflow, and possibly some update or miscount bugs as I iron it out. the good news, now they are all virtual, is that problems are undoable. please report any issues, and I will work on it as I keep pushing on this and on parents
• please expect your client.caches.db file to expand in size about 10-30% or so this week. the update itself will take a few minutes as the improved tag lookups and new caches are regenerated from empty
• .
• boring mostly db optimisation list:
• after some thought, moved those new options for tag sibling application down to the db. previously, they were stored in an UI object for convenience, but since everything is going down to the db, it is worth doing it properly down there. thus they reset this week to the default
• I also removed that complicated 'all known tags' page in the tag sibling application options--it wasn't doing enough to justify itself
• tag siblings lookup cache now obeys the tag sibling application rules and regenerates the appropriate cache when the options change
• tightened up the db tag siblings lookup cache and wrote more tools for it. it had a couple of blind spots for getting all siblings in a chain. also optimised the lookup for en-masse tag operations
• tightened up my tag sibling structure builder object, which was not eliminating loops but collapsing them to (generally harmless, but not desireable) (A,A) pairs
• extended mappings and siblings lookup caches to perform various sorts of tag sibling collapse filtering, to determine files that do or do not have another tag mapping on a tag sibling chain
• optimised the existing mappings cache in several ways
• optimised cross-domain file cache mappings filtering, and cleaned the code
• optimised autocomplete count fetching from the mappings cache, particularly for large result sets
• optimised how the combined autocomplete count generates from nothing
• optimised how tags are loaded for search results (thumbnails)
• optimesed basic tag search
• greatly optimised how the mappings caches request cross-domain file cache filtering
• broke up the rescind_pending/add mappings job into simpler separate parts, which was needed for accurate display cache counting. this may end up fixing the other weird pending miscount bug we had
• the cached 'display' tags are now loaded with regular media results, not generated on the fly on first request (unless in the advanced 'all known files' domain, where it is done quickly on first load at the db level)
• converted the db over to using its local sibling lookup cache for all sibling jobs
• all data-level content updates to media result objects now occur in the database loop, reducing lag and eliminating a single UI event loop gap when the objects the UI relies on were desynchronised
• optimised how the tag and hash id-to-definition cache maintains itself
• cleaned up cache code generally
• wrote a ton of unit tests to cover construction, tag, and tag sibling operations on the siblings and display caches
• wrote a second optimised method for regenerating 'all known files' display cache autocomplete counts from nothing, which, when multiple siblings have wildly different counts (e.g. 50, 100, 200000), instead of counting them all, counts the smaller tags sans the largest, and adds this to the already pre-computed largest count
• the old ui level siblings manager has been pared down to some final tools that will be trickier to replace
• .
• the rest:
• fixed the stupid manage tag siblings dialog input/ok bug I introduced last week
• fixed the pair preview label in manage tag siblings dialog when it asks to enter a reason for a remove petition
• I believe I fixed the annoying recent bug where the top-right hover window would sometimes not position itself correctly on a window size/move until the top hover was shown once
• fixed a bug where the 'do you want to do shutdown work?' dialog was not abandoning shutdown if cancelled (rather than yes/no)
• updated the 'has free space to do db transaction?' checker, which needs to test device partitions, to do two sweeps--first only fast local devices, then potentially mega laggy network discovery if the mount point is not found (hydev was wondering why it was suddenly taking nine seconds to close his test client!)
• fixed another issue with double-clicking some addremove/queue listboxes when no edit button is set--now in this case they all delete on a double-click
• fixed a little bad error handling on pending content upload. an error with petitioning certain IPFS uploads is not yet fixed

• version 407

• sibling prep:
• I am preparing for a new siblings database cache for v408. this will ultimately make siblings (and parents) faster, more accurate, more powerful, and simple to undo. I have decided, as part of it, to make siblings and parents completely virtual (i.e. the tags won't exist for real, they'll be implied). better tools to manage hard-replace siblings and parents will come later, as trying to support both situations at once has not been excellent
• .
• created options to hold per-service sibling and parent preferences, so you'll be able to set up '"my tags" siblings and then "ptr" siblings apply to "my tags"' or 'no parents apply to this service'
• wrote UI for the sibling options under 'services->manage where tag siblings apply'. you can play with it if you like, and it saves values, but it is not plugged in yet and makes no changes
• siblings logic is a little tighter. the db and gui side of siblings structure calculation is more unified, petitioned siblings are discounted properly on all generation, and the db side now resolves conflict decisions the same on every regen. the gui-side still runs on an older structure, but will be updated to exactly mirror the db
• updated and unified how large numbers of raw tag siblings are fetched in the database. it also supports fast tag slicing, speeding up sibling cache maintenance. the siblings lookup cache now uses this method for regeneration and update calls
• .
• the rest:
• tag right-click menu copying now supports all combinations of selected/all, tags/subtags, and no_count/with_counts where appropriate (issue #325)
• if the media viewer is too thin for the top hover window to fit into its space, the top-right hover now drops down below it. I don't really like how this looks, and will probably instead figure out a flow layout so the toolbar buttons always fit, but at least they are now accessible (issue #388)
• altered the above fix--if the top-right hover window can be shrunk to fit in the available space, it will now squeeze in, only bumping down if it can't
• moving the mouse off an activated (e.g. clicked) hover window now instantly activates the main canvas. this should fix up some fast swallowed clicks and annoying click-to-activate issues with the center-right duplicates hover window, which does not hide (issue #384)
• the duplicates hover window now positions correctly if its min size is too wide to fit in a thin media window
• if you make changes to a parser or content parser, there is now a yes/no confirmation when trying to cancel the dialog
• fixed an issue where 'queue' listboxes with no edit button would throw an error on double-click. now double-click in this case deletes
• fixed a couple of timestamp convertions that were doing YYYY/MM/DD instead of the more ISO-nice YYYY-MM-DD. also, when in UTC, they'll correctly say UTC now instead of GMT (issue #369)
• fixed some borked centered text layout on ratings dialog and import folder dialog
• fixed the manage services dialog's wrong headers for type/name columns
• added links in the official help to the new user-written simple help guide at https://github.com/Zweibach/text/blob/master/Hydrus/Hydrus%20Help%20Docs/00_tableOfContents.md
• moved object tag and ratings code to a new client module, 'metadata', and pulled various ratings gui code into a new separate file
• refactored some more manager code around to generally more sensible locations
• did a bit more work chasing down the highlight-downloader ui deadlock, which unfortunately still exists
• reduced the number of db hits some paged downloaders need, particularly on highlight and init
• updated some test code to support cleverer db testing
• updated mpv for windows build. api version is now 1.109. this fixes at least one weird linux vm audio driver issue

• version 406

• subscription management:
• the manage subscriptions dialog now has a 'deduplicate' button. it is enabled whenever your subs of a particular downloader contain duplicate queries. it launches a semi-bananas but thorough 4-step process to ask if you want to do upper/lower-case deduplication, then which downloader, then which queries, then which master sub(s) to retain the queries
• subscription dedupe within the same sub keeps the query with the most files
• the manage subscriptions dialog also now has a 'lowercase' button that coerces all queries of the selected subs to lowercase
• when pasting a list of queries into a subscription, the 'already in sub' test is now caseless. pasting "Samus_Aran" into a sub already with "samus_aran" will not add anything
• .
• misc:
• url classes now have a checkbox to keep fragment data (e.g. "#kwGFb3xhA3k8B") during URL normalisation. this data is not sent to the server and is not useful for almost all sites, but for sites like Mega, it contains useful clientside javascript navigation or access info if you open the URL in your browser
• fixed video resolution parsing for some unusual SAR files. this stretches a video slightly, usually when the vid was created or converted with older analog tech (e.g. NTSC)
• fixed rating system predicate label for 'rated/not rated'
• the issue where miscounts in pending upload data would persist, sometimes leading to an annoying 'pending (13)' style menu that would not clear without debug action, is now fixed in a cheap way. on any upload action, this cached count is reset. a fix for the actual unusual miscount situation will have to come later
• the different in-memory manager objects now save changes at different time intervals--lightweight things like favourite searches still save not long after any change, but column widths, network sessions, and bandwidth use now save only every ten minutes
• I _may_ have fixed an issue with favourite tags not sticking correctly or resetting when added en masse via the tag right-click menu
• I believe I fixed a rare but permanent ui hang on highlighting a gallery or watcher when that same downloader was spamming through a largely 'already in db/previously deleted' list
• copying tags 'with counts' now works correctly for simple tag views (previously, it only worked for 'predicate' views)
• copying tags now preserves the tag order as in the list (previously, it did a human sort)
• to stop status-sorted gallery and watcher list entries bouncing around so much, they now just say 'working' in their status column when they are working. the highlight panel still reports granular file/gallery info. galleries also now say a more solid DONE when complete, to spot them more easily
• the gallery and watcher search/checking column now includes stop status in sort
• fixed the dowloader link in the help to https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders
• added that same link to the Lain dowloader import panel's help button
• updated cloudscraper to 1.2.46
• updated cloudscraper interfacing code to adapt for new reCaptcha->Captcha object names
• .
• boring code cleanup:
• refactored downloader gui code to its own file
• refactored network gui code to its own file
• refactored service gui code to its own file
• finished import reordering. now all files import in a cleaner order
• further reworked all hydrus imports to be more breadth-first, loading core modules earlier and catching potential errors in nicer places
• checkbox selection is now wrapped in the 'quick' dialog system, and all checkbox selections now use this single method
• simplified and unified a variety of layout code, and fixing some odd layout expanding bugs
• misc code cleanup
• deleted some old unused ui code

• version 405

• tag search:
• system:number of tags now supports namespaces, for example 'find files with two character tags'! (issue #280)
• it also supports wildcard namespaces, as now do regular namespace search predicates. both run faster. "crea*:anything" is now possible
• system:number of tags has been optimised, and in many cases is now ten to a hundred times faster
• system:number of tags still does not support siblings, something I hope to start correcting as of v408
• both tag existence (numtags =0 or greater than 0) and tag count database routines now respond quickly to 'cancel search' commands, so if you do run a slow query (a bare 'has creator tag' search on 'all known files' on the PTR, for instance), you can now back out quickly after the 'stop' button appears
• note that 'system:number of character tags greater than 0' and '= 0' are equivalent to +/-character:anything, which will be swapped in if you enter these. also, +/-unnamespaced:anything can now appear
• the program is a bit better about determining =0 and greater than 0 and less than 1 being 'none' and 'any but none', when it needs to determine optimisations and special labels
• unfortunately, I am taking away the default value for system:num tags in the options page (edit: I am killing the whole panel now). this old ugly mess of stacked predicate edit panels works on ancient, difficult to update code, so I will retire it and replace it with a unified system that is easy to use, supports in-search system predicate editing, and keeps up with changes automatically
• system:number of tags is now comfortable with redundancies--if you add >2 and >4, it now knows that >4 is the true lower bound (previously, the one used was random)
• boring code changes here:
• updated tag existence and tag count searches to take advantage of the tag cache when in a specific file domain (which is pretty much all the time), which should speed them up significantly
• updated tag existence and tag count searches to more carefully plan their queries, speeding them up both in advantageous and difficult situations
• cleaned up tag existence and tag count code significantly
• updated all edit system predicate panels to return full predicate objects, a step towards decoupling them and allowing in-place system predicate editing
• wrote a new number test object to hold and help with number range test values. num tags now uses it, and eventually all range predicates will too
• the namespace existence search code ('anything' queries) is now folded into the new generalised tag existence search code
• streamlined how the search context propagates through all database tag searching--now, most queries do not know or care about domain or current/pending status--they just iterate over n tables as determined by a specialised routine
• added a handful of unit tests for the new namespace num tag searching
• .
• database repair:
• the database menu has a new entry, 'repopulate truncated mappings tables', under the newly renamed 'check and repair' submenu, which will try its best to 'fix' a client.mappings.db file that has been truncated due to hard drive fault by repopulating from the local-file-only tag cache. do not run this unless you know you need to
• the 'help my db is broke.txt' document has a full update pass. the language is clearer, common issues and questions are better addressed, two new recovery routines are added, a section on the stages after boot recovery (like the new repopulate job above) is added, and I added my stock 'now become a backup patrician' nag at the end
• the debug routine to clear cached service info numbers is now moved to the 'regenerate' database menu. this thing fixes hanging incorrect 'pending' counts until I can fix it properly
• .
• the rest:
• fixed an issue where when you pasted queries into a subscription, those that were already in the sub (and got the dialog saying so), were being added anyway! I believe this bug came in the last few weeks, after the data storage rewrite. please check your pasted-into subs for dupes
• fixed tab double middle-click behaviour (so you can spam page close), which I thought I had fixed last week but actually messed up completely right at the end (issue #314)
• cleaned up some more of the page tab event code--it was a mess all around. should all be on Qt now, no wx hacks
• network jobs will no longer wait for and consume bandwidth start tokens while all network traffic is paused. all bandwidth competition now halts. (previously, they would continue to consume tokens according to current rules and then all rush to start as soon as traffic was resumed)
• fixed some client booru/client api requests to correctly 404 on missing file results, rather than 500
• cleaned up some file sort code and fixed the sort string conversion, which was rendering the opposite sort direction (asc/desc) in summary labels (e.g. on manage favourite searches)
• cleaned up some ui layout stretching code, including some borked tag import options expand sizing
• improved some button and padding layout definitions, and improved, slightly, the way the top-right media viewer hover window lays itself out and changes its size on media change
• improved some review services layout. should be fewer weird heights and widths in unusual situations, and the new multi-column list fits better
• the manage subs dialog now saves its changes to db more cleanly and atomically
• updated the default derpibooru parser to pull species tags. ten points if you can guess what that is most of the time

• version 404

• column lists:
• all multi-column lists across the program now remember the widths of their columns when they are next recreated
• the last column of any list is now universally the 'stretching' column, which should correctly initialise with its preferred/previous size, but also grows and shrinks with the parent window
• while all lists retain their initial rows height, and those in the gallery and watcher management panels will continue to grow and shrink in a fixed way, all lists in dialog windows can now be shrunk down to four rows
• the minimum size of any column is now much smaller, about three characters
• all column headers now tooltip their name
• lists should be better about sizing in non-100% OS UI scalings
• the lists that are automatically sorted (e.g. the download pages, and manage subs) now remember the last sort you gave them
• future plans, now within reach:
• all lists will sort, sort arrows will appear on the header, and sort will be faster
• columns will be rearrangeable
• columns will be hide/showable, and initially hidden complicated columns will be available
• there will be some, maybe optional, capability to have lists sync live, so if you edit one, the others do the same
• num rows height memory, maybe--we'll see how the above shakes out first
• boring code changes:
• moved list code to a new sub-module
• wrote a status object for column list current columns, widths, and sort, and plugged it into list code
• wrote a manager to handle column statuses, and plugged it into the main controller and db
• wrote definitions for every list (66 or so different lists!) and all their columns, and unified width, name, default sort, and future hideabality and default hide/show status to that one easy to edit and extend location
• rewrote list column and sort initialisation to work off the new status object and added hooks for list sorting and column resizing to save new status back
• rewrote every list column instantiation to use the new system
• numerous misc column list code cleanup
• .
• the rest:
• double middle-clicking on the page tab bar should now correctly close two tabs in a row (rather than opening the rename page dialog on the second)
• entering an odd number of hex characters into system:hash no longer causes an error. this will be changed in future to properly highlight and explain badly pasted or incorrect-length hashes in future
• the new red text for non-functional status texts in review services now properly re-colours itself between normal/red when an error or resolution occurs while the panel is open
• hydrus now knows if it is running in the Haiku operating system and has preliminary platform specificity. if you are interested in helping to get hydrus running properly in Haiku, please join in with github issue #358
• cleaned up a mix of smaller code, unused variables and imports and so on

• version 403

• shortcuts:
• shortcuts have a backend update this week. a bunch of hacky stuff is now cleaner behind the scenes, and the related UI has some cleanup as well
• converted all 100-odd simple shortcut commands from hacky text ids to a proper enumerated id system, and across every single instance across the program
• wrote nicer descriptive labels for all simple shortcuts. gone is 'focus_media_viewer', now is 'keyboard focus: to the media viewer'
• if you have no like/dislike or numerical services, the respective application command edit panels now say so and do not allow an ok action
• like/dislike rating sub-panels now start with 'like' checked
• when a like/dislike or numerical rating sub-panel is set to 'remove', the action dropdown is set to 'set' (rather than flip) and is disabled, as is the numerical slider
• application commands now state better "3/5" information about rating actions, rating than the underlying "0.6" float implementation
• all application commands existing in shortcuts or elsewhere are updated to the new enumerated id system
• refactored ApplicationCommand (the side of shortcuts that holds the actual action to be done) and its edit UI to new separate files
• completely refactored the application command edit panel, pulling the simple/tag/rating sub-panels into their own decoupled classes, simplifying the tangle and permitting easier future expansion
• rearranged some application command functions and contant definitions to more appropriate locations
• improved how application commands are interrogated by the objects that process them
• added plenty of type hinting around application command processing code
• cleaned up a bunch of shortcut and application command code, including some wx->Qt updates as well
• .
• menu and UI cleanup:
• removed an old wx hack that prohibited last-second ui updates. the exit splash screen now reports final db shutdown info
• if a service or account is currently non-functional (e.g. all repositories are paused), the appropriate status text is now in red
• if there is work to do the first time a duplicate page is opened or looked at, it now moves to the 'preparation' tab
• doing a 'migrate database' file migration now temp-closes the migrate db dialog and hides the main gui while it goes on
• brushed up the tag filter ui a bit--now only one of the tag_filter/blacklist test phrases only show up, in the appropriate context, and the test text input now supports multiple newline-separated tags (e.g. if you want to paste a bunch)
• every panel on review services now has a refresh button to force an update
• the 'clear trash' button on the trash review services panel is now disabled when there is nothing to clear
• updated edit subscription panel to point to the main html help and brushed up that help to talk about file limits more, also the earlier downloader help has a little section to highlight subscriptions and their use
• reworked the 'restore from db backup' command--it is now integrated into client shutdown proper, and reports its basic restore progress to the exit splash screen
• reorganised the 'network' menu. manage subs is now up top, downloader submenus are now split better into high-level vs component-level, and login stuff is pulled to its own submenu
• put 'network traffic' at the top of the network->pause menu
• rearranged some of the 'gui' and 'gui pages' option pages and tucked everything into box sections for clarity
• the search pause/play button on search page tag autocomplete now has a simpler 'search paused' label when paused. the code has a similar nomenclature change, and eventually this will turn into a simple pause/play icon button or similar
• fixed some weirdness with floating autocomplete dropdowns sometimes not appearing on dialogs on first load
• fixed some focus logic so set-focus calls on downloader pages should work again on the query input text box and elsewhere
• unified all numerical rating->stars and stars->rating calculations across the program. this may have fixed some edge-case bugs
• unified all rating string generation across the program
• .
• the rest:
• the disk cache options under _options->speed and memory_ are now default off and force-set off for all users on update. as more users are on decent ssds where these options are of limited value (and sometimes negative value), I now only recommend them for users on HDDs
• added two options for autocomplete results list height to 'gui pages' option page, under the new 'controls' section
• fixed a critical issue where the client api could duplicate-add tags with url imports to multiple services. the potential service duplicate cascade order was pseudorandom and particular to a client. thanks to a user for figuring this issue out (issue #317)
• added a 'tag whitelist' to downloader tag import options. its edit button is below the blacklist. when there are no tags in the list, it does nothing, but if tags are added, then files that do not have at least one of the given tags at the download source will not be imported. for instance, if you have a username-based downloader (where you can't add more tags to the query to filter serverside), and you only want their metroid content, you can now filter it simply hydrusside (issue #279)
• if you are both in advanced mode and a mad lad, the basic blacklist tag filter now allows you to show the 'whitelist' and 'advanced' panels again, if you have a complicated blacklist to set
• the local booru and client api now support the same https as the hydrus server, using self-signed certificates stored in the db directory. just set the checkbox in manage services and you should be good. self-signed certificates are free and will work on a server hosted off an IP address, but they are imperfect. they are also likely to require special permission to be accepted by the web browser or whatever you want to talk to the https service. however, if you host your client from a real DNS domain and have your own fully signed cert+key files, you can swap them in no problem
• local booru and client api urls adjust scheme for the new option, and unified and cleaned up how booru share urls are generated internally
• the way cert+key files are generated is moved from server code to common hydrus code
• cleaned up how additional db files like certificate files and the mpv conf are managed for backup/restore operations
• cleaned up some ancient http urls to https. mostly stuff like the regex tutorial links
• when files are appended to a regular search page (e.g. from a subscription publish to an existing page, or from a mouse drag and drop), the search context will now pause. this is to stop accidental F5 or mass refresh signals wiping out the changed page
• to break advanced-case gallery search loops, gallery url jobs now have a 'run' identity token. galleries pass their token down to 'next page' or 'sub-gallery' urls they generate, meaning all urls of a particular search run share the same url. gallery logs now ignore to-be-added urls that already exist for their token, terminating loops. new tokens are generated if a search is restarted or similar, meaning duplicate urls can exist in a gallery log, just not from the same starting point (issue #302)
• improved simple gallery url deduplication in several stages of the downloader pipeline
• when right-clicking on multiple thumbs, the info lines off the top menu item now list the files' combined viewtimes (this previously only showed when one file was selected)
• fixed some error reporting problems with adding urls to import via the client api--some url class exceptions were being converted from 400 to 500 errors unintentionally
• a new stylesheet, 'Hydracula', is added to the default install. check it out under options->style. thank you to a user for contributing this
• subscriptions are better about calculating a 90 second forgiveness window for bandwidth rules. they should schedule and startup more effectively, and the edit subscriptions and single edit subscription panels should also no longer show bandwidth delays below the next 90s, which are often a technical situation of regular work breaks that are better ignored for the purposes of the dialog
• went back up to pyinstaller 3.6 again on windows, as 3.5 caused its own Qt bindings dll problems. if you had trouble with 3.6 (401), let me know how this works for you, as there are additional dll-finding fixes included (issue #329)
• fixed an issue where under some conditions, file save dialogs were only happy with filenames that already existed (issue #319)
• fixed an issue with the 'client already running' system sometimes not closing the client process correctly when told to cancel the boot
• bumped the 'space needed for vacuum' estimate up to 120% (was 100%) of estimated final file size, just to catch some edge cases
• rolling out updated danbooru parsers that pull associable urls correctly, thank you to a user for this fix
• rolling out an updated deviant art parser that finds some unusual file urls when other methods fail, thank you to another user for this fix (issue #295)
• upgraded cloudscraper to 1.2.42
• improved some type hinting
• fixed up some unit tests for new command and rating data

• version 402

• in many situations--such as a search result that gives no results, or a search cancel, or a downloader page cleared of a highlight--pages will now report a special status text rather than '0 files', such as 'no results for this search' or 'search cancelled!' (issue #277)
• new pages, and the first page of a loaded session, should now correctly publish their status text to the status bar immediately after initialisation, (previously blank until first change)
• clicking the 'searching immediately' button while a search is ongoing now correctly cancels a search, cleaning up status and page and buttons, rather than just stopping current work immediately
• added 'copy_xxx_hash' shortcuts to the media shortcut set for 'md5', 'sha1', and 'sha512'
• when copying file hashes to clipboard, a popup appears for two seconds to verify what happened
• when copying file hashes to clipboard, recovery from missing hashes is more graceful, with multiple error report states
• the way the client shuts down is untangled. the order in which the gui, managers, threads, database are shut down is smoothed out, with better error handling and fewer potential logical holes
• the 'should I do shutdown work?' dialog is now only presented in the clean shutdown pipeline
• menu labels now elide at 128 characters, extended from 64 previously. hopefully this strikes a better balance between fixed texts we do want to read while still not letting long dynamic texts go nuts (issue #276)
• gallery and watcher pages now have 'show file/gallery log' on their menus, which directly zoom in to the edit dialogs for the top-most selected query or watcher (issue #256)
• when file maintenance is forced to run from the thumbnail menu or file maintenance job panel, it now provides x/y progress text and gauge based on total jobs, e.g. 1,234/10,000, rather than out of the 256-job batches (issue #264)
• the simple downloader page now updates its pending jobs list more efficiently, and supports multiple selection, and presents a yes/no confirmation on delete
• most lists with clipboard/png import/export buttons can now also do .json files. they also accept json files in a drag and drop. you can mix json and png files in a multi-file drag and drop
• when selecting a parser for a url class in 'manage url class links', those parsers with example urls that match the url class are now separately listed at the top of the choice dialog
• in the recent autocomplete rewrite, the hidden repository update file domain was accidentally exposed in the file domain button. after some testing, it actually works(!), but as this is an advanced topic, it is now hidden behind advanced mode
• the way services are deleted or completely reset is now changed to what should be a significantly faster and smaller operation
• the latest user-made nitter/twitter downloader is rolled in to the update. some little fixes and adds support for mobile.twitter.com url imports
• fixed an issue where uninitialised repositories thought they were caught up
• to reflect that it does nothing in this case, the mouse shortcut edit panel now disables the press/release choice on double-click or scroll
• fixed file save dialogs not filling in the default filename properly
• removed an old wx safety hack where new pages would silently not create while the client was minimised. this fixes issues with large session loading and subscriptions publishing files to page names that do not yet exist while the client is minimised
• removed an old wx safety hack where some tag lists would not regen their current tag display while the client was minimised
• in lieu of a future better bit of html subscription help that I link to from the subscription panel, the 'file limits' help button has temporarily briefer text so it doesn't make such a giant popup
• moving back to pyinstaller 3.5 (from 3.6) for the windows build, which appears to fix some dll loading for some users (issue #244)
• the windows and linux builds are updated to Qt 5.15 (from 5.13.2). it does not seem to have the odd problems 5.14 gave us. let me know if you have any trouble or if any weird graphical issues magically fix themselves
• .
• client api:
• the /get_files/file_metadata call has a new true/false parameter, 'detailed_url_information', default false, that adds 'detailed_known_urls' structure to list the known urls results as in /add_urls/get_url_info. it has a help example and a unit test and everything (issue #235)
• the client api version is now 13
• .
• boring cleanup details:
• reshuffled the shutdown code. now the controller takes the lead, booting splash as appropriate and commanding gui to save and close, and then proceeds to other shutdown
• fast and normal shutdown code is unified, just run differently
• shutdown calls should now always be idempotent
• a catch for some OS-level shutdown commands, normally user log-off, also hooks into the newer UI-free fast shutdown
• SIGINT and SIGTERM also hook better into the new shutdown, and are thread safe
• performing multiple SIGINTS on shutdown should no longer throw an error after the gui is deleted
• more potential startup/shutdown errors are now caught and presented to the user and saved to log, with subsequent shutdown urgency accelerated afterwards
• critical errors on a fast shutdown no longer present to the user--they just save to log
• updated how an emergency shutdown state is tested
• updated how a 'clean exit complete' state is set and tested
• various unusual shutdown states now skip human interaction and jump straight to guaranteed fast shutdown
• refactored splash window to its own file
• wrote a new qlistwidget subclass to do some common data storage/retrieval/selection. it will eventually replace most lists across the program
• the 'queue' list widget that has up/delete/down and add/edit buttons beside a list has nicer backend code and now initialises with its buttons correctly disabled due to no selection
• the similar 'add/edit/delete' list widget is updated to use the nicer backend
• some wx->Qt list hacks, which were themselves using borked old display-string-based indexing, are deleted
• the repository download/process daemon has been moved to the newer job scheduler. it should start up and close out on program exit a bit more neatly
• untangled some messy value-change radio button code in the shortcut edit panel
• updated the way page status text propagates up from the thumbnail grid to the main gui to Qt signals instead of the old inefficient pubsub
• all UI file hash clipboard copying code is now unified and improved
• added a new subscription file publish debug test to help->debug->gui
• refactored some client specific time delta rendering code out of core to client
• misc event cleanup code
• misc code style cleanup

• version 401

• subscriptions:
• as subs can now load more flexibly, previously hardcoded waits are now eliminated:
• - the subscriptions manager now only waits three seconds after initial session load to boot (previously 15)
• - the subscriptions manager now wakes as soon as the subscriptions dialog is ok'd or cancelled
• - a timing calculation that would delay the work of a sub up to five or fifteen minutes if more queries would come due for sync in that time window (in order previously to batch to reduce read/write) is now eliminated--subs will now start as soon as any query is due. if you were ever confused why a query that seemed due did not boot after dialog ok or other wake-up event, this _should_ no longer happen
• re-added the import/export/duplicate buttons to manage subs. export and dupe may need to do db work for a couple of seconds and will have a yes/no confirmation on larger jobs
• the import button on manage subs accepts and converts the old 'legacy' subscription object format, including a copy/paste of the objects backed up to disk in the v400 update
• fixed an issue where creating a subscription query and then deleting it in the same manage subs dialog session would result in surplus data being written to the db (which the next dialog launch would note and clear out)
• an unusual error with pre-run domain checking, exposed by the new subscription code and e621 subs, where the gallery url has also recently changed, is now fixed
• .
• issue tracker:
• the Github issue tracker (https://github.com/hydrusnetwork/hydrus/issues) is turned on again! it is now run by a team of volunteer users. the idea is going to be to try to merge duplicate feature suggestions with the proper platform and put some discussion and cognition and prioritisation into idea development before it gets to my desk, so I can be more focused and productive and so 95% of feature suggestions do not simply get banished to the shadow realm of the back of my todo
• this is mostly intended for wishlist and other suggestions, as the tsunami was just getting too much for me to handle, but we'll see how it goes for things like bug reports as well. I'll still take any sort of report through my normal channels, if you are uncomfortable with github, or if you wish for me to forward an item to the issue tracker anonymously
• the website, help documents, and hydrus help menu links have been updated regarding the issue tracker
• .
• the rest:
• improved how the database 'update default downloader objects' job works, ensuring that new defaults are better at simply take the place of existing objects and do not break/reset existing url class to parser links
• tightened up how automatic url class to parser linking works, eliminating some surplus and potentially bad data related to api links. furthermore, whenever the links between url classes and parsers update, existing surplus data, which may creep in when api links change, is now cleaned from the data structure
• rolling out updated e621 url class and parser to deal with their alternate gallery url format
• rolling out an updated derpibooru parser that will link to the new api class correctly
• thanks to a user's submission, rolling out updated versions of the new default nitter parsers that pull creator:username tags
• before every subprocess launch, and when waiting for all subprocess communication (e.g. to ffmpeg), now tests regularly for program shutdown. if an unusual situation develops where a subscription is doing a file import job while the OS is shutting down, and that system shut down would hang or is hanging on a 'ffmpeg can't be launched now' dialog, the hydrus client should now notice this and bomb out, rather than going for that never-running ffmpeg. this may not fix all instances of this issue, and further feedback on the client not closing down cleanly with the OS is welcome.
• when adding a new path to the 'migrate database' panel, any symbolic links will be converted to canonical equivalents
• added some location checks and appropriate errors when the database is doing file storage rebalancing
• fixed an issue uploading swfs, video, or audio to the server when it is launched from a frozen executable build
• misc code cleanup

• version 400

• subscription data overhaul:
• the formerly monolithic subscription object is finally broken up into smaller pieces, reducing work and load lag and total db read/write for all actions
• subscriptions work the same as before, no user input is required. they just work better now™
• depending on the size and number of your subscriptions, the db update may take a minute or two this week. a backup of your old subscription objects will be created in your db directory, under a new 'legacy_subscriptions_backup' subdirectory
• the manage subscriptions dialog should now open within a second (assuming subs are not currently running). it should save just as fast, only with a little lag if you decide to make significant changes or go into many queries' logs, which are now fetched on demand inside the dialog
• when subscriptions run, they similarly only have to load the query they are currently working on. boot lag is now almost nothing, and total drive read/write data for a typical sub run is massively reduced
• the 'total files in a sub' limits no longer apply. you can have a sub with a thousand queries and half a million urls if you like
• basic subscription data is now held in memory at all times, opening up future fast access such as client api and general UI editing of subs. more work will happen here in coming weeks
• if due to hard drive fault or other unusual situations some subscription file/gallery log data is missing from the db, a running sub will note this, pause the sub, and provide a popup error for the user. the manage subscription dialog will correct it on launch by resetting the affected queries with new empty data
• similarly, if you launch the manage subs dialog and there is orphaned file/gallery log data in the db, this will be noticed, with the surplus data then backed up to the database directory and deleted from the database proper
• subscription queries can now handle domain and bandwidth tests for downloaders that host files/posts on a different domain to the gallery search step
• if subs are running when manage subs is booted, long delays while waiting for them to pause are less likely
• some subscription 'should run?' tests are improved for odd situations such as subs that have no queries or all DEAD queries
• improved some error handling in merge/separate code
• the 'show/copy quality info' buttons now work off the main thread, disabling the sub edit dialog while they work
• updated a little of the subs help
• .
• boring actual code changes for subs:
• wrote a query log container object to store bulky file and gallery log info
• wrote a query header object to store options and cache log summary info
• wrote a file cache status object to summarise important info so check timings and similar can be decided upon without needing to load a log
• the new cache is now used across the program for all file import summary presentation
• wrote a new subscription object to hold the new query headers and load logs as needed
• updated subscription management to deal with the new subscription objects. it now also keeps them in memory all the time
• wrote a fail-safe update from the old subscription objects to the new, which also saves a backup to disk, just in case of unforeseen problems in the near future
• updated the subscription ui code to deal with all the new objects
• updated the subscription ui to deal with asynchronous log fetching as needed
• cleaned up some file import status code
• moved old subscription code to a new legacy file
• refactored subscription ui code to a new file
• refactored and improved sub sync code
• misc subscription cleanup
• misc subscription ui cleanup
• added type hints to multiple subscription locations
• improved how missing serialisable object errors are handled at the db level
• .
• client api:
• the client api now delivers 'is_inbox', 'is_local', 'is_trashed' for 'GET /get_files/file_metadata'
• the client api's Access-Control-Allow-Headers CORS header is now '*', allowing all
• client api version is now 12
• .
• downloaders:
• twitter retired their old api on the 1st of June, and there is unfortunately no good hydrus solution for the new one. however thanks to a user's efforts, a nice new parser for nitter, a twitter wrapper, is added in today's update. please play with it--it has three downloaders, one for a user's media, one for retweets, and one for both together--and adjust your twitter subscriptions to use the new downloader as needed. the twitter downloader is no longer included for new hydrus users
• thanks to a user's submission, fixed the md5 hash fetching for default danbooru parsers
• derpibooru gallery searching _should_ be fixed to use their current api
• .
• the rest:
• when the client exits or gets a 'modal' maintenance popup window, all currently playing media windows will now pause
• regrettably, due to some content merging issues that are too complicated to improve at the moment, the dupe filter will no longer show the files of processed pairs in the duplicate filter more than once per batch. you won't get a series of AB, AC, AD any more. this will return in future
• the weird bug where double-clicking the topmost recent tags suggestion would actually remove the top two items _should_ be fixed. general selection-setting on this column should also be improved
• middle-clicking on a parent tag in a 'write' autocomplete dropdown no longer launches a page with that invalid parent 'label' tag included--it just does the base tag. the same is true of label tags (such as 'loading...') and namespace tags
• when changing 'expand parents on autocomplete' in the cog button on manage tags, the respective autocomplete now changes whether it displays parents
• this is slightly complicated: a tag 'write' context (like manage tags) now presents its autocomplete tags (filtering, siblings, parents) according to the tag service of the parent panel, not the current tag service of the autocomplete. so, if you are on 'my tags' panel and switch to 'all known tags' for the a/c, you will no longer get 'all known tags' siblings and parents and so on presented if 'my tags' is not set to take them. this was sometimes causing confusion when a list showed a parent but the underlying panel did not add it on tag entry
• to reduce blacklist confusion, when you launch the edit blacklist dialog from an edit tag import options panel, now only the 'blacklist' tab shows, the summary text is blacklist-specific, and the top intro message is improved. a separate 'whitelist' filter will be added in the near future to allow downloading of files only if they have certain tags
• 'hard-replace siblings and parents' in _manage tags_ should now correctly remove bad siblings when they are currently pending
• network->downloaders->manage downloader and url display now has a checkbox to make the media viewer top-right hover show unmatched urls
• the '... elide page tab names' option now applies instantly on options dialog ok to all pages
• added 'copy_bmp_or_file_if_not_bmpable' shortcut command to media set. it tries copy_bmp first, then copy_file if not a static image
• fixed some edit tag filter layout to stop long intro messages making it super wide
• fixed an issue where tag filters could accept non-whitespace-stripped entries and entries with uppercase characters
• fixed a display typo where the 'clear orphan files' maintenance job, when set to delete orphans, was accidentally reporting (total number of thumbnails)/(number of files to delete) text in the file delete step instead of the correct (num_done/num_to_do)
• clarified the 'reset repository' commands in review services
• when launching an external program, the child process's environment's PATH is reset to what it was at hydrus boot (removing hydrus base dir)
• when launching an external program from the frozen build, if some Qt/SSL specific PATH variables have been set to hydrus subdirectories by pyinstaller or otherwise, they are now removed. (this hopefully fixes issues launching some Qt programs as external file launchers)
• added a separate requirements.txt for python 3.8, which can't handle PySide2 5.13.0
• updated help->about to deal better with missing mpv
• updated windows mpv to 2020-05-31 build, api version is now 1.108
• updated windows sqlite to 3.32.2

Changelog 350-399

• version 399

• improvements:
• the media viewer and thumbnail _right-click->manage_ menus now have a _viewing stats->clear_ action, which does a straight-up delete of all viewing stats record for the selected files. 'edit' will be added to this menu in future
• extended the tag autocomplete options with a checkbox to allow 'namespace:' to match all tags, without the explicit asterisk
• tag autocomplete options now permit namespace searches if the 'search namespaces into full tags' option is set
• the tag autocomplete options panel now disables and checks the namespace checkboxes when one option overrules another
• cleaned up some tag search logic to recognise and deal with 'namespace:' as a query
• added some more unit tests for tag autocomplete options
• the html and json parsing formulae now support negative indexing, to select the nth last item from a list
• extended the '1 -> "1st"' ordinal string conversion code to deal with negative indices
• the 'hide tag' taglist menu actions are now wrapped in yes/no dialogs
• reduced the activation-to-click-accept time that the shortcuts handler uses to ignore activating clicks from 100ms to 17ms
• clicking the media viewer's top hover window's zoom buttons now forces the 'media viewer center' zoom centerpoint, so if you have the mouse centerpoint set, it won't zoom around the button where you are clicking!
• added a simple 8chan.moe watcher to the defaults, all users will get it on update
• the default bandwidth rules for download pages, subs, and watchers are now more liberal. only new users will get these. various improvements to db and ui update pipeline mean the enforced breaks are less needed
• when a manage tags dialog moves to another media, if it has a 'recent tags' suggestion list with a selection, the selection now resets to the top item in the list
• the mpv player now tracks when a video is fully loaded and only reports seek bar info and allows seeks when this is so (this should fix some seekbar errors on broken/slow-loading vids)
• added 'undelete_file' to media shortcut commands
• file delete and undelete are no longer hardcoded in the media viewer and media thumbnail grid. these actions are now handled entirely in the media shortcut set, and added to all clients by default (this defaults to (shift +) delete key, and also backspace on macos, so likely no changes)
• ctrl+mouse wheel is no longer hardcoded to zoom in the media browser. these actions are now handled entirely in the 'all' media viewer shortcut set (this defaults to ctrl+wheel or +/-, so likely no changes)
• deleted some old shortcut processing code
• tightened up some update timers to better halt work while the client is minimised to system tray. this _may_ improve some users' restore hanging issues
• as Qt is happier than wx about making pages on a non-visible client, subscriptions and various url import operations are now permitted to create pages while the client is minimised to taskbar or system tray. if this applies to your situation, please let me know how you get on here, as this may relieve some restore hanging as the pending new-file jobs are no longer queued up
• .
• fixes:
• clicks on hover window greyspace should no longer propagate up to the media viewer. this was causing weird archive/delete filter actions
• mouse scroll on hover window taglist should no longer propagate up to the media viewer when the taglist has no more to scroll in that direction
• fixed an issue that meant preview windows were initialising about twenty pixels too short for the first page loaded in a session, and also pages created within nested page of pages. also cleaned up some logic for unusual situations like hidden preview windows. one more cycle of closing and reopening the client will fix the option value here
• cleaned and unified some page sash setting code, also improving the 'hide preview window' option reliability for advanced actions
• fixed a bug that meant file viewtime was still being recorded on the duplicate filter when the special exception option was off
• reduced some file viewtime manager overhead
• fixed an issue with database repair code when local_tags_cache is missing
• fixed an issue updating a very old db not recognising that local_tags_cache does not yet exist for proper reason and then trying to repair it before update code runs
• fixed the annoying issue introduced in the recent string match overhaul where a 'fixed character' string match edit panel would not want to ok if the (now hidden) example string input did not have the same fixed char data. it now validates no matter what is in the hidden input
• potentially important parsing fix: JSON parsing, when set to get strings, no longer converts a 'null' value to 'None'
• the JSON parsing formula now allows you to select the nth indexed item of an Object (a JSON key->value dictionary). due to technical limitations, it alphabetises the keys, not selecting them as-is in the JSON itself
• images that do not load in PIL no longer cause mime exceptions if they are run through the decompression bomb check
• .
• misc:
• boosted the values of the decompression bomb check anyway, to reduce false positives. it generally now has a problem with images with a bmp > 1GB memory
• by default, new file import options now start with decompression bombs allowed. this option is being reduced to a stopgap for users with less memory
• 'MimeException' is renamed to 'UnsupportedFileException'
• added 'DamagedOrUnusualFileException' to handle normally supported files that cannot be parsed or loaded
• 'SizeException' is split into 'TagSizeException' and 'FileSizeException'
• improved some file exception inheritance
• removed the 'experimental' label from sub-gallery page url type in parsing system
• updated some advanced help regarding bad files
• misc help updates
• updated cloudscraper to 1.2.40

• version 398

• new tag search options:
• there are several new options for tag autocomplete under the newly renamed _services->tag display and search_:
• for 'manage tags'-style 'write' autocompletes, you can now set which file service and tag service each tag service page's autocomplete starts with (e.g. some users have wanted to say 'start my "my tags" service looking at "all known files" and "ptr"' to get more suggestions for "my tags" typing). the default is 'all known files' and the same tag service
• the old blanket 'show "all known files" in write autocompletes' option under _options->tags_ is removed
• you now can enable the following potentially very slow and expensive searches on a per-tag-domain basis:
• - you can permit namespace-autocompleting searches, so 'ser' also matches 'ser*:*', i.e. 'series:metroid' and every other series tag
• - you can permit 'namespace:*', fetching all tags for a namespace
• - you can permit '*', fetching all tags （╬ಠ益ಠ)
• '*' and 'namespace:*' wildcard searches are now significantly faster on smaller specific tag domains (i.e. not "all known tags")
• short explicit wildcard searches like "s*" now fire off that actual search, regardless of the 'exact match' character threshold
• queries in the form "*:xxx" are now replaced with "xxx" in logic and display
• improved the reliability of various search text definition logic to account for wildcard situations properly when doing quick-enter tag broadcast and so on
• fixed up autocomplete db search code for wildcard namespaces with "*" subtags
• simplified some autocomplete database search code
• .
• string processing:
• the new string processor is now live. all parsing formulae now use a string processor instead of the string match/transformer pair, with existing matches and transformers that do work being integrated into the new processor
• thus, all formulae parsing now supports the new string splitter object, which allows you to split '1,2,3' into ['1','2','3']
• all formulae panels now have the combined 'string processing' button, which launches a new edit panel and will grow in height to list all current processing steps
• the stringmatch panel now hides its controls when they are not relevent to the current match type. also, setting fixed match type (or, typically, mouse-scrolling past it), no longer resets min/max/example fields)
• the string conversion step edit panel now clearly separates the controls vs the test results
• improved button and summary labelling for string tools across the program
• some differences in labelling between string 'conversion' and 'transformation' are unified to 'conversion' across the program
• moved the test data used in parsing edit panels to its own object, and updated some of the handling to support passing up of multiple example texts
• the separation formula of a subsidiary page parser now loads with current test data
• the string processing panel loads with the current test data, and passes the first example string of the appropriate processing step to its sub-panels. this will be expanded in future to multiple example testing for each panel, and subsequently for note parsing, multiline testing
• added safety code and unit tests to test string processing for hex/base64 bytes outcomes. as a reminder, I expect to eliminate the bytes issue in future and just eat hashes as hex
• cleaned up a variety of string processing code
• misc improvements to string processing controls
• .
• the rest:
• double-clicking a page tab now opens up the rename dialog
• system:time imported now has quick buttons for 'since 1/7/30 days ago'
• all hydrus downloaders now accept percent-encoded characters in the query field, so if you are on a site that has tags with spaces, you can now enter a query like "simple%20background red%20hair" to get the input you want. you can also generally now paste encoded queries from your address bar into hydrus and they should work, with the only proviso being "%25", which is "%", when all bets are off
• duplicates shut down work (both tree rebalancing and dupe searching) now quickly obeys the 'cancel shutdown work' splash button
• fixed a signal cleanup bug that meant some media windows in the preview viewer were hanging on to and multiplying a 'launch media' signal and a shortcut handler, which meant double-clicking on the preview viewer successively on a page would result in multiple media window launches
• fixed an issue opening the manage parsers dialog for users with certain unusual parsers
• fixed the 'hide the preview window' setting for the new page layout method
• updated the default gelbooru gallery page parser to fix gelb gallery parsing
• updated the newgrounds parser to the latest on the github. it should support static image art now
• if automatic vacuum is disabled in the client, forced vacuum is no longer prohibited
• updated cloudscraper for all builds to 1.2.38
• .
• boring code cleanup:
• all final mouse event processing hackey is removed from the media viewers, and the shortcut system is now fully responsible. left click (now with no or any modifier) is still hardcoded to do drag but does not interfere with other mapped left-click actions
• the duplicates filter no longer hardcodes mouse wheel to navigate--whatever is set for the normal browser, it now obeys
• cleaned up some mouse move tracking code
• clicking to focus an unfocused media viewer window will now not trigger the associated click action, so you can now click on archive/delete filters without moving on!
• the red/green on/off buttons on the autocomplete dropdown are updated from the old wx pubsub to Qt signalling
• updated wx hacks to proper Qt event processing for splash window, mouse move events in the media viewer and the animation scanbar
• cleaned up how some event filtering and other processing propagates in the media viewer
• deleted some old unused mouse show/hide media viewer code
• did some more python imports cleanup
• cleaned up some unit test selection code
• refactored the media code to a new directory module
• refactored the media result and media result cache code to their own files
• refactored some qt colour functions from core to gui module
• misc code cleanup

• version 397

• regular changelog:
• added 'system:has/has no note with name xxx' to search for specific note names
• in the normal system predicate list, the notes pred is now the generic 'system:notes' to launch a combined dialog for both num notes and named notes
• favourite tag suggestions are now sorted in manage tags dialog according to the default tag sort
• page names will now middle...elide when there are too many to fit into a row (and normally left/right buttons would be added). if the elided tabs still do not fit, the buttons will pop up as before. added a checkbox to options->gui pages to turn this text eliding off
• pulled the 'page name' options on that panel into their own box and added some text regarding the 'my big row of import page tabs keeps scrolling weird' issue
• when files are pixel duplicates, the filesize and age comparison statements will now have 0 score and thus be coloured neutral blue
• the standard text entry dialog now always selects any default text it starts with, so you can now type to immediately overwrite. see how you like it and if there are some places where you think an exception should be made
• updated the IPFS interface to work with the new IPFS 5.0. all api requests are now POST so it doesn't 405, and the User-Agent is overridden to one that IPFS will not 403 at, and I fixed a typo the new api is more strict about
• a hack to get page splitters to lay out correctly on session load is rewritten from a hammer to a scalpel. pages now set their splitter positions on their first individual visible selection. this both reduces some minor ui lag on session/page load and improves splitter positions for clients that open minimised to the system tray
• a long-time odd issue where loaded sessions would initially select the top-left-most non-page of pages is fixed. now the bottom-left-most page of any kind is selected
• fixed tag autocomplete selecting the bottom-most pre-loading result. it now correctly selects at the top
• fixed an issue setting certain values (typically loading a default) to a tag import options panel
• the client is now more aggressive about clearing subscriptions from memory when they are finished running
• in windows, the main method that copies files now checks for modified time of the source file. if it is before 1980-01-01 UTC, it does not copy the file metadata, as some Windows has trouble with this lmaoooo
• cleaned up how some thumbnail 'current focus' media determination code works. should have fixed some weird errors when hitting certain shortcuts on collections
• cleaned up basic list/sort code across the program
• the 'queue' and add/edit/delete listboxes now emit change signals when new items are added or imported
• pyparsing, a helper for cloudscraper, is now correctly bundled in the built releases. a new line in help->about displays this
• help->about now lists cloudscraper version
• updated the discord link to the new https://discord.gg/wPHPCUZ
• .
• upcoming string processing changes for advanced users:
• I extended string parsing code this week, but I am not yet ready to turn it on. when it does come on, it will change all formulae from the fixed string match/converter pair a combined general string processing 'script' of n steps
• wrote a new 'string splitter' object that takes one strings and splits it into up to n strings based on a separator phrase (such as ' ,')
• wrote an edit panel for string splitters
• wrote a new 'string processor' object that holds n ordered string match/converter/splitter objects and filters/converts/splits x strings into y strings based on those steps
• wrote an edit panel for string processors. it has a notebook that live updates with test results for each step on every update
• wrote unit tests for string match
• wrote unit tests for string converter
• wrote unit tests for string splitter
• wrote unit tests for string processor
• refactored string conversion edit panels to their own file
• refactored string conversion controls to their own file
• misc string processing cleanup and labelling improvements
• .
• technical url parsing stuff:
• urls are now stripped of leading and trailing whitespace during normalisation, just in case a paste contains some extra whitespace. previously, it would sometimes throw a 'doesn't start with http' error
• the hydrus url normalisation process now normalises the hostname according to the NKFC unicode format, meaning unusual characters like ？and e◌́ are now replaced with their normalised visual equivalent ? and é, and hence these urls will no longer throw errors when they are added
• if '?' or '#' end up in a hostname (which are invalid characters), it is now converted to _, just to stop complete parse mangling when weird urls are submitted. this character replacement may become more sophisticated in future
• the hydrus downloader should now support search terms that include '#'
• download query parameters that contain '%23' ('#', encoded) are now not unquoted in url normalisation

• version 396

• notes:
• the file notes system is more mature. files now store multiple named notes
• the edit notes ui is now a tabbed window with add/edit_name/delete buttons
• media results now load with their notes, so note access is instant
• thumbnails now show a notes icon when they have notes
• the media viewer top-right area shows a notes icon when the current file has notes
• clicking the media viewer top-right notes icon opens edit notes
• the edit notes menu entry now lists the number of current notes if there are notes
• added a 'system:number of notes' predicate. it has easy 'has/no notes' buttons for quick filtering
• the file notes database table will be updated on update, it shouldn't take long. existing notes will get the default 'notes' name
• duplicate notes now share the same storage space in the database
• in prep for a future search expansion, notes are now cached in the database for fast text search
• in prep for note parsing, wrote a 'note import options' object. it doesn't do anything in the program yet, but it supports multiple note conflict resolutions, note extension detection, and global and specific note renaming
• wrote unit tests for the new note import options
• .
• some tag search stuff:
• hydrus now maintains an internal mapping of direct 'searchable' versions of tags to the tags themselves, which allows it to now do fast exact-match (short search) and complicated wildcard lookups of tags with unusual characters. 'f' and '/f/' will now return '/f/' and 'board:/f/' quickly, 'board:f' and 'board:/f/' will return 'board:/f/' quickly, and 'te*a*' will correctly return 'test-tag'
• it will take a few minutes to regenerate this new cache on update
• complex wildcards like 's*m*' are now treated the same as simple ones like 'sam*' and should match unusual subtag characters in all cases
• wildcard tag file search predicates are now plugged into the new cache, so the search preds '/f/*', 'board:/f/', 'board:/f/ast;', 'b*d:/f/' and 'b*d:/f/*' now all match files with 'board:/f/', as do wildcards that include replacement characters, so the same should be true above for 'f' instead of /f/' in all cases
• new wildcard search preds do not collapse their characters for their presentation string, so 'date:2*-01-01' now renders like that, not 'date:2* 01 01'
• wildcard file search predicates are now faster for simple (just an asterisk on the end) subtag wildcards
• the fts search cache is moved from 'master' to 'caches' db this week, it will take a few moments on update
• the 'repopulate tag search cache' db regen job now repopulates the fts cache, the new 'searchable' cache, and the integer tag cache
• the database repair code now checks for the fts cache and new 'searchable' cache on boot and, if they are missing, warns the user and creates empty tables
• .
• improvements:
• fixed the unsorted tags in tag suggestion boxes
• clicking the inbox icon in the top-right hover window now archives the file
• system:dimensions now has quick buttons for 16:9, 9:16, 4:3, 1:1, 1080p, 720p, and 4k
• system:known url searches are now better about fetching www and non-www urls for the domain or url class
• the edit shortcut sets panel now has nicer english names for reserved shortcut sets, and also sorts them in a more logical way
• you no longer have to be in advanced mode to copy file hashes from thumbnails
• users in advanced mode can copy the internal file_id of files from the thumbnail/viewer copy menus (this is most useful for the client api)
• system num_frames, num_words, and num_notes now display alternate 'has/no xxx' labels when they search for =0 or >0
• you can now search for 0 with system:num_frames
• .
• fixes:
• users who could restore from system tray using the menu but had trouble with clicking _should_ now have better luck with clicking
• fixed some instances where fps could be calculated as 0, which would lead to other problems down the line. now a missing or 0 fps is remapped to 1
• fixed system:framerate for '<' queries
• the status bar cells now get expanded tooltips to describe what they do
• fixed some media result caching code that could in rare cases cause an error in content update processing when the result disappeared from the cache during processing
• the 'hard-replace siblings and parents' button on 'manage tags' now makes a submenu so its actions' long labels show better
• fixed a handful of tables that were not starting sorted
• a variety of credential parse and other server failures that were formerly returning 403 now properly return 400 and 409
• in order to improve default 'open externally' behaviour on Linux/macOS, if the environment variable XDG_DATA_DIRS is not preserved through a hydrus build launch env, hydrus now sets a simple 'default' value for this before running xdg-open
• if the client is booted from a windows shortcut to a built release, the program restart command is slightly more reliable
• .
• misc:
• cleaned up some db update error reporting code, it should now more reliably make an english-friendly popup text box before splurging technical info
• refactored some media object code, cleaned some class definitions, and added typing hints
• misc code cleanup
• the 'getting started' help files now have anchor definitions, so their sections can now be #linked to
• added several links in the 'getting started' help to the user-created video guides here: https://github.com/CuddleBear92/Hydrus-guides thank you for making these!
• added a link to the help for the user-made 'other archiving software' guide here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/0-Alternative-Programs-and-Resources#software thank you for making this!
• fixed link to AUR package in the help
• updated cloudscraper in all builds to 1.2.36
• updated windows mpv to a significantly newer dll, it now reports api version 1.108
• included libgpg-error.so.0 in Linux build, which will improve some Linux situations (more reports from Ubuntu 20.04 or others about missing/conflicting .so files are welcome)

• version 395

• some more suggested tags fixes/qol:
• favourite tags now correctly refreshes on new media
• the tag suggestion lists in manage tags now discard current and pending tags that _all_ the current media already have, and all tag suggestion lists update this filter any time the media gets a tag content update! they _should_ update live now
• all tag and predicate taglists now try to move the selection to a 'nice' neigbour when a keyboard enter activation results in the current selection being removed (e.g. as in these tag suggestion lists). the nice selection should be the tag after, before, or at the top of the list, and should make it nicer to keep navigating the list and add tags with your keyboard
• all tag and predicate taglists now try to preserve selection on simple clear-and-set data refreshes
• .
• deleted tags overwrite update:
• due to an unfortunate oversight, until now tag parsing has not filtered out previously deleted tags from the tags it parses and sends to the local database
• as the majority of downloaded files are parsed once per site per user and in a similar time window before manual editing ever occurs, and most non-tag-sibling-eligible bad tags are site specific or not parsed to begin with, and as these undesired tags were not broadcast up to the tag repository, this problem has not been very obvious and I believe has not affected most users too much. this is however a reason why some users who have more recently downloaded many older files are seeing smaller 'deleted mappings' counts on their ptr review panel (and some low quality tags in their db), as they have been re-adding previously deleted tags to their local store
• this has been fixed. tag import options now load the pending importee file's metadata before tags are filtered and discard currently deleted tags from those to be added or pended. this applies to parsed tags, additional tags, and those tags added through special other means, such as from a parent gallery page.
• if you do wish to allow parsed or additional tags to overwrite currently deleted tags for a particular job, the cog icons on the edit tag import options panel now allow you to permit overwrite for either
• tags added via hard drive imports or the migrate tags tool still overwrite deleted tags as before
• as this is a local-only problem, there is thankfully a retroactive fix for this issue for tag repository domains, involving a content reprocess run to re-apply deleted tags. I am not activating this automatically this week as this is a heavy job for the ptr and I need to study the true fallout of the problem more, but I may in future, likely as a smaller and more targeted maintenance job. advanced users can do it now under the ptr's review services panel
• I regret missing this, and I am sorry for any inconvenience. I only discovered it through the serendipity of some users recently reporting unusual deleted counts and a personal item in my todo to check the reliability of deleted mapping filtering for local tag domains--turns out it never got added, and we never specifically noticed, fugg
• there are now unit tests for the improved tag filtering pipeline and both of these new overwrite options
• .
• the rest:
• hydrus can now use several different zoom 'centerpoints' about which to expand and shrink a zooming file. this was previously hardcoded to the center of the media. under options->media, you can now set it to be the media window center (the new default, which feels much nicer after a pan), the mouse cursor, the old media center, or the media top-left corner
• cleaned up the related zoom positioning code, and removed the jarring old re-centering off-screen rescue hack when zooming out to canvas zoom
• added a warning about big zooms to the media options page
• fixed tag autocomplete filtering in python 3.7 so 'character:aran' matches 'character:samus aran' again
• when the hover windows on a media viewer have focus, they _should_ now pass up all options->shortcuts shortcuts to the media viewer
• mouse back/forward buttons _should_ now be supported in the shortcuts system, as much as your OS allows them to work like regular clicks
• fixed a rare crash with the 'clear trash' button
• the client will now not re-analyze tables that have been previously scanned with at least 100k rows in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
• the client will now not vacuum database files greater than 1GB in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
• the new 'cannot vacuum because xxxx' log entry is now only ever printed once per boot. however due to the above change, it likely won't appear in the normal maintenance cycle anyway now
• cleaned up some vacuum code
• reworked the panel system to better test data validity vs 'woah, you sure you want to do this?' tests and generally cleaned and simplified the canok/cancancel/isvalid testing logic for all panels. panels like manage siblings will now not produce two message boxes if you try to ok them on an uncommited pair and then back out of the ok
• refactored the top level window code and improved scrollable panel code typing
• more standalone gui function code refactoring
• fixed a click-selection-test bug when clicking on certain whitespace in certain predicate lists
• the text of the cloudflare-specific error when encountering a captcha page is improved
• cleaned up some tag list menu copy and select code, both the menu labels and the copy action, for unusual tags. the 'copyable tags' fetching code is now flexible and unified for menu and action
• cleaned up the taglist sibling copy code, eliminating the chance of dupes
• fixed a _little_ of the wording on the discard/exclude tag list menu labels for negated predicates, it still feels a bit awkward and I will keep working here
• cleaned up some old media metadata fetching code
• misc import code typing
• misc list/iterable typing improvements
• added some misc media-tag tool code
• unified the tag import options tag filtering pipeline somewhat to deal with the deleted overwrite situation
• improved a debug ui test to no longer need window focus
• misc help cleanup

• version 394

• autocomplete cleanup:
• the text you type into tag autocomplete is now parsed in a unified object. all the variants of empty text, invalid text, valid text, namespace text, and wildcard text are all tested and fetched in one simple location with better code
• autocomplete results caching is now a unified object that tracks and filters results in one location. wildcard searches are now never cached by accident, and switching from tag cache to system predicate cache and to non-initialised cache is instant and more reliable
• when an autocomplete, either in a search page or a context that manages tags, has results include multiple sibling variants of the typed text, they are now all elevated to the top of the list. the ideal is at the top, the entered text is next, and any known siblings follow
• the search character 'collapse' that ensures quote marks and hyphens and other odd characters are unified across tags now applies uniformly to all non-complicated-wildcard search tags, with namespace not collapsed and subtag always collapsed
• when entering an explicit wildcard search, both strict and autocomplete versions (whether they end with an asterisk) are now displayed
• the way tag results are filtered is now more accurate for some unusual wildcards
• it is now more difficult to slip cpu-killer search tags (weird asterisk combinations) through
• the quick-broadcast that happens when the user hits enter before any results have started loading now uses the unified object and chooses a safer and more reliable broadcast value. the test whether to do the quick-broadcast is also more reliable, particularly in unusual situations where a recent search was cancelled or delayed. note that for many users, the cache and search tech is fast enough that this very rarely triggers
• searching with a wildcard below the autocomplete threshold can no longer trigger a full search, nor an invalid exact-text search
• namespace count merging is now unified across db tag fetches and media fetches
• include current/pending buttons now filter down to media-based tag autocomplete counts
• namespace tag autocomplete queries will no longer show up some unusual siblings below the 'anything' tag
• deleted a whole bunch of old a/c and caching code
• added comprehensive unit tests for the new parsed autocomplete text object
• added comprehensive unit tests for the new predicate results cache object
• .
• the rest:
• fixed a stupid typo bug in the new domain checking code that was stopping subscriptions with incomplete file queues from starting. I apologise for this
• network error responses 502 (Bad Gateway) and 503 (Service Unavailable) are now treated as a retryable. the 503 is assuming it is not a CF challenge page. if they fail all retries, they are considered a network infrastructure error
• all other misc 5xx http responses are now treated as instant network infrastructure errors and will be logged in the new domain health tracker
• the exit splash screen now opens a bit earlier, so you now shouldn't have any momentary uncertainty where no windows are open
• clients that start minimised to system tray _should_ be better about restoring splitter positions on first show
• the various 'management panels', the panels on the left of main gui pages, now have smaller minimum width where available. the gallery and watcher panels are still the widest, which is a limitation of the current list tech. when it gets better column sizing code and selection memory, this will improve
• fixed an issue loading gifs with some OpenCV versions
• brushed up some running from source help
• deleted the Py2To3 script that attempts to detect a legacy python 2 install
• improved all the gui files' import order
• cleaned up and refactored some subscription code
• added a bunch of type hints to edit panel code
• misc code cleanup
• .
• environment updates:
• did second step of hydrus project structure improvement--now the project is split into subdirectories for core/client/server/misc and some client subdirs. work here will continue
• linux build gets some new libraries, cv is up to 4.2.0
• it isn't important, but hydrus is now built in python rather than directly from command line. my build scripts now include cloudscraper and the new hydrus source code tree in the build as they are, rather than hardcoded copying

• version 393

• cloudflare and network:
• the hydrus client now has an experimental hook to the cloudscraper module, which is now an optional pip module for source users and included in all built releases. if a CF challenge page is downloaded, hydrus attempts to detect and solve it with cloudscraper and save the CF cookies back to the session before reattempting the request. all feedback on this working/breaking irl would be welcome. current expectation for this prototype is it can pass the basic 'wait five seconds' javascript challenge, but only a handful of the more complicated captcha ones
• if a CF challenge page is not solvable, the respective fail reason for that URL will be labelled appropriately about CloudFlare and have more technical information
• .
• the hydrus network engine now has the capability to remember recent serious network infrastructure errors (no connection, unsolvable cloudflare problem, etc..) on a per domain basis. if many serious errors have happened on a domain, new jobs will now wait until they are clear. this defaults to three or more such errors in the past ten minutes, and is configurable (and disableable) under options->connection. this will be built out to a flexible system in future, with per-domain options+status ui to see what's going on and actions to scrub delays
• basically, if a server or your internet connection goes down, hydrus now throttles down to limit the damage
• subscriptions now test if a domain is ok in order to decide whether they can start or continue file work, just like with bandwidth
• serverside bandwidth alerts (429 or 509) are now classified as network infrastructure errors
• I expect this system will need more tuning
• .
• the hydrus downloader system now recognises when an expected parseable document is actually an importable file. when this is true, the file is imported. this hopefully solves the situation where a site may deliver a post url or a file
• .
• the rest:
• the windows build of hydrus is now in python 3.7.6, up from 3.6. this rolls in a host of small improvements, including to network stability and security (e.g. TLS 1.3), and possibly a couple of new bugs in more unusual hydrus systems
• similarly, all the windows libraries are now their latest versions. opencv is now 4.2
• greatly sped up several file searches that include no tags such as bare system:rating, most system file metadata predicates, or bare system:inbox, when the result size is much smaller than the total number of files in the file domain
• thanks to some excellent work by a user, the Deviant Art downloader gets another pass--it can now get high res versions of images where they are available, and video, and flash, and pdf! the only proviso is that you need to be logged in to DA to get most content, otherwise you get 404. the current hydrus DA login script _seems_ to work ok
• tag import options blacklists now test unnamespaced rules against namespaced tags. so if you blacklist 'metroid', a 'series:metroid' will be caught and the blacklist veto signal sent. this can be escaped with the 'advanced' exception panel, which now permits you to add 'redundant' rules
• the edit tag filter panel now explains the blacklist rules explicitly and has a second 'test' green/red text to display test results for a tag import options blacklist, with the new sibling and namespace check
• added some unit tests to test the new tag import options blacklist namespace rule
• when 'default' tag import options are set, the edit panel now hides the per-service options, rather the the previous disable
• the system tray icon now destroys itself when no longer needed, rather than hiding itself. it should now be more reliable in OSes that do not support system tray icon hide/show. if your OS still doesn't get rid of them, and you get a whole row of them, I recommend just leaving it always on
• the system tray now has a tooltip with the main hydrus title and pause statuses
• the timer that hides the mouse on the media viewer is now fired off when the window first opens (previously it would only initiate on the first mouse move over the window), so users who navigate mostly by keyboard should now see their cursors nicely hide on their own
• added some semi-hacky import/export/duplicate buttons to edit shortcuts. I'll keep working on this, it'd be nice to have import/export for whole shortcut sets
• added a semi-hacky duplicate button to the 'manage http headers' dialog
• the 'clear' recent tag suggestions button is now wrapped in a yes/no dialog
• a new checkbox under options->gui now lets you set it so when new cookies are sent from the API, or cookies are cleared, a popup message summarises the change. the popup dismisses itself after five seconds
• the client api now also returns 'ext' on /get_files/file_metadata calls, just as a simpler alternative if the 'mime' is a pain
• fixed a bug when petitioning tags through the client api, with or without reasons
• fixed an error where subscriptions that somehow held invalid URLs would not be able to predict some bandwidth stuff, which would not allow the edit subs dialog to open
• the string transformation dialog's step subdialog is now ok with example strings that are bytes. even then, this str/bytes dichotomy is an old artifact of python 2 and I will likely clean it up sometime so string transformers (and downloaders) only ever work utf-8 and hashes just work off utf-8 hex
• added a BUGFIX checkbox to options->gui that tells the UI to use Qt file/directory picker dialogs, instead of the native OS one. users who have crashes on file selection are encouraged to try this out
• updated running from source help with cloudscraper, a new pip masterline, and some windows venv info
• the 'import with tags' button on 'import files' dialog gets another rename for new users, this time to 'add tags before the import >>'. it also gets a tooltip
• handled an unusual rare error that could occur when switching out a media player inside a media viewer, perhaps during media viewer shutdown

• version 392

• db-level tag sibling cache:
• the hydrus client db now maintains a fast cache of current+pending tag-to-ideal-tag sibling relationships. it works for specific services and 'all known tags'. this is a nice tool and the first step in having a proper hard-baked siblings mappings cache
• the new sibling cache can be regenerated under _database->regenerate_. the 'autocomplete cache' entry under that menu is also renamed to the now more appropriate 'tag mappings cache'
• the db repair system can regenerate this new cache if any part is missing on boot
• the lookup that finds tag sibling matches for autocomplete uses this and is now faster, specific to the searched service, more accurate about status, and now includes pending siblings
• wrote a new unified object to manage a collection of tag siblings, it is now in use at the db level
• as I continue to develop this new fast tech, the old 'apply all sibs to all services' option, which was always buggy, may sometimes not apply in it. I will ultimately replace it with a fuller per-service choice system that will work quickly and properly and in the same unified way
• fixed a bug where only one local tag service's siblings would be matched at the ui level when looking at 'all known tags'
• fixed a bug in the file search code where searching for a tag that had an unnamespaced sibling going to it would result in searching for all possible namespaces of that sibling (e.g. searching for 'character:samus aran' when 'samus aran'->'character:samus aran' sibling existed would result in effectively 'anything:samus aran')
• when tag services are deleted, they are now better about cleaning up their siblings and parents quickly
• optimised some tag and hash id->value database cache population routines to improve performance for large queries (e.g. when fetching all the tag parents/siblings on boot). also these caches are now larger, 100k instead of 25k
• all cache regen code now forces an immediate analyze of the new tables to speed up imminent access
• .
• the rest:
• updated the default e621 file page parser to get rating tags again (looks like their markup just changed again)
• updated the default sankaku file page parser to get their recently redefined 'genre' tags
• in edit subscriptions, the 'overwrite tag import options/check options' actions now initialise their dialogs with the current value for the first subscription, rather than the global program default
• in the edit subscription panel, the checker options button is moved down to the file/tag import options
• when not in advanced mode, the edit tag import options panel now has some red-text at the top to reinforce to new users that they should generally use the defaults
• the tag import options blacklist now secondarily checks against all known siblings of the parsed tags, rather than just the 'collapsed' ideal siblings
• subscriptions are now more aggressive about clearing out old urls from their file import caches--instead of clearing the 251st url after it has aged twice the death period, now they use just one DP. also, checkers with static checker timings will use five times that check period as DP if that is smaller. static checkers, or those that never die, will use a flat value of six months as DP if that is smaller
• moved a bunch of the debug 'data actions' to a new 'memory actions' menu
• significantly reduced how often the system tray regenerates its menu, which seems to improve stability
• fixed an issue where guis that were maximised before a minimise were restoring from a system tray icon click to normal view
• double-clicking the system tray when the ui is hidden should no longer do a fast show/hide
• fixed an issue where if the gui was minimised, the main animation timer would not run for other windows (e.g. a separate media viewer)
• improved ui shown/hidden tracking logic for the new system tray icon for different OSes
• fixed the 'refresh_page_of_pages_pages' shortcut action, which had faulty old wx code in it
• fixed a wx->Qt bug where modal popups that cannot be cancelled, and thus pop up a 'sorry, you can't dismiss this' text when you try to close them, were nonetheless still closing afterwards
• the hydrus client and server now attempt to listen their servers on both IPv4 and IPv6, failing gracefully if IPv6 is not available
• the 'is this a localhost request?' check now understands IPv6 localhost (::1 or ::ffff:127.0.0.1)
• may have solved a 100% cpu repaint issue with the a/c dropdown in some qt environments
• added info to installing help about Windows N and clean installs
• misc media viewer wx->Qt code cleanup
• misc code cleanup
• .
• experimental hellzone, be wary ye scabs:
• added an experimental 'sub-gallery url' url content type to the parsing and downloading system. this url is queued into the gallery log even if the primary gallery page found no file/post urls, and is meant for galleries that link to galleries. not yet ready for primetime, but feedback would be appreciated
• added an experimental ui-hang relief mode, activated under _help->debug->data actions->db ui-hang relief mode_, which _should_ stop the ui hanging in unusual long-time ui-synchronous db jobs. it may cause other problems, so it is default off. it also prints begin/end statements to log for additional info. users who experience ui hang due to db job processing time are invited to play with this mode and report back results

• version 391

• system tray icon:
• hydrus now can now make a system tray icon for those OSes that support it. it can be buggy/crashy under non-Windows, where it gets some warning labels
• under the new options->system tray page, you can set whether to show the system tray all the time, minimise the main gui to system tray, close-button the main gui to system tray, and start the program minimised to the system tray
• right-clicking the icon brings up a menu to show/hide the ui, pause/unpause network traffic or subscriptions, and to exit hydrus
• the main file menu now has an option to minimise to system tray
• double-clicking or middle-clicking the icon will show/hide the whole hydrus ui as long as there are no dialogs open
• clicking it will restore the main gui from minimise or raise it to the front
• on an ui hide, the current preview window will be blanked and media viewers will be paused, so any ongoing noise/cpu from them should stop
• a new 'global' shortcut 'hide_to_system_tray' is now available
• starting the client minimised may have some layout issues on first show--I particularly had to fix splitter layouts--please report any more you discover
• .
• framerate and num frames:
• system:framerate search added to system:duration panel. precise framerate is tricky with current hydrus info, so it searches +/- 5% of a given value
• system:number of frames added to system:duration panel
• sort by number of frames added
• duration/framerate/num frames sort moved to their own 'duration' submenu
• framerate added to generic media metadata summary string (which appears in status bar and media viewer, etc...). precise framerate is tricky with current hydrus info, so it is rounded to the nearest integer
• .
• the rest:
• rolling in new danbooru file page parsers that should fix file downloads, thank you to a user for the submission
• rolling in a e621 login script, thank you to a user for the submission
• gave tag autocomplete results fetch code a pass, cleaning up several instances of incorrect or inefficient timing and caching logic and I believe fixing the issue where system preds would sometimes not be loaded after entering a tag
• improved reliability of autocomplete dropdown hiding on background pages (some edge cases where these could still hang around _should_ be fixed)
• improved 'hide' tests in several parts of the program related to the new system tray icon, which should help some other cases--e.g. weird shutdowns now _shouldn't_ ever leave a bunch of floating popup messages
• fixed a bug where pages set to open with all known files/tags domains, which is not supported, was incorrectly substituting tag domain with a file domain, which is even more not supported
• cleaned up some sort code--I believe this has fixed the odd issue where a 'time imported' sort would not work on some pages (such as one loaded from a favourite search)
• fixed the 'related' tag suggestion box not knowing about new pending tags added in a manage tags dialog open on a media viewer after next/previous media transitions while the dialog is open. also it and the file lookup's lists now clear when a new lookup starts
• the tag suggestion boxes are now add-only and remove what you add as you add them! let me know if this feels nice or not!
• the splash window now has a different 'booting/exiting' window title, if you would like to hook it with a window manager
• went over all the 'prep url for display', 'filter urls', and 'normalise url' requests across the program to deal with invalid url (e.g. garbled text) better
• you can now no longer add invalid urls via the client api associate_url call--you'll get 400 instead
• cleaned some thumbnail selection and rendering code, particularly fixing some edge case 'where that media go?' issues where collect-by calls happen during thumbnail waterfalls and so on
• cleaned up some page file domain setting code and misc page management code
• improved accuracy of rendered image cache memory footprint calculations
• fixed some Qt signal object definitions that were causing errors for some users who run from source

• version 390

• fixed a bug that was causing potential duplicates to be sometimes re-added between media groups that were previously set as false positive/not related. I apologise for the inconvenience this bug has caused. if you were hit by this, please reset your potential duplicate pairs (hit the cog button on the dupes page) and re-search, and the bad pairs should not be re-added again
• fixed an issue where tag autocomplete entry in the form 'namespace-blah:' was replacing the hyphen or other 'collapsable' character to space, which then was not searching correctly for the _anything_ namespace search
• 'namespace:anything' searches now work when the namespace itself has a wildcard
• fixed 'write' autocompletes not matching inputs with UPPERCASE letters
• fixed adding tags that start with a colon (e.g. ":D") in 'write' autocompletes
• it should now be impossible to enter some 'kill my cpu' queries into tag autocomplete, such as '[asterisk]:anything', even if accidentally entered through the fast-add system
• the 'cancel search' stop button that appears after a search takes three seconds is back to being neatly embedded beside the tag autocomplete input box
• hitting the cancel search button now clears the non-interactable 'Loading' thumbnail media page (with its misleading 'Loading...' statusbar) and returns you to an empty thumbnail page
• loading a favourite search with non-immediate search no longer loads the 'loading...' page. it also saves that new non-immediate status to the page session more reliably
• reworked my linux build environment (pyinstaller=3.5, virtualenv=16.1) so that you can launch the built exe using a symlink
• rolling out a first version of a requirements.txt, any feedback would be appreciated
• rolling out another version of the derpibooru file page parser that no longer duplicates namespaced tags as unnamespaced, thank you to the user who submitted this
• .
• boring stuff:
• .
• moving old pubsub system to Qt signals:
• all the 'refresh query' calls that do changes to the current file search across the program
• the current file and tag domain update calls for search pages
• the clear/set file focus calls when launching and exiting the media viewer browser or archive/delete filter
• the way thumbnails send the current focused media to the preview media window
• the way widgets with shortcuts-based tooltips are notified to update those tooltips when shortcuts change
• the way a thumbgrid sends the current tags to be displayed in the 'selection tags' list
• the way a thumbgrid adds newly imported files' tags to the 'selection tags' list as they fade in
• the way the 'searching/waiting' search button is flipped on and off by shortcut. btw what should be the correct name + label for this button? should it really be an icon?
• .
• cleanup:
• NOTE: the 'include' folder is renamed this week to 'hydrus'. if you have source patches, please update. as I further disentagle code in future, hydrus will ultimately move to typical nested folder/module structure
• decoupled the shortcuts edit ui code from the controller and db, unified how shortcuts are edited, and eliminated db wait when booting shortcuts editing
• decoupled the shortcuts manager from the controller, cleaned all the code, and moved to a nicer reference with proper typing hints
• refactored the frame and media controls of clientguicanvas into separate files
• renamed the hoverframes file to canvashoverframes and updated its classnames to 'canvas' rather than the old 'fullscreen'
• fixed two wx->Qt typo artifacts in the login script edit ui
• reduced some occasional idle memory bloat of clients that have large subscriptions
• cleaned up how media-based taglists are appended with new media
• removed some old booru object update code
• some misc setmedia/clearmedia cleanup
• misc search code cleanup
• misc typing hints to clear up pylint confusion
• misc tag autocomplete code cleanup
• misc 'global' variable cleanup
• misc gui code refactoring, cleanup and typing

• version 389

• downloaders:
• the e621 file page parser is updated again, thanks to a user's contribution. this one gets md5 and file url more reliably, and also gets rating tag
• added a 'e621 file page (old format)' url class to help match and search for files downloaded with the old format. please be aware there is no good solution to auto-convert old urls to a new format yet, so this connection does not (yet) solve the old/new comparison test
• updated deviant art file post parser to use their json api. this should be more resilient to their current layout changes
• the nijie.info login script appears no longer to function. as with exhentai last week, it has been removed to make it easier to log in with hydrus companion. please use hydrus companion if you would like to log into nijie.info
• updated file lookup scripts for 'iqdb danbooru' and 'danbooru md5' thanks to a user's contribution
• .
• the rest:
• the way the mpv.conf works changes this week. it is now correctly fully portable, stored in the db directory beside the .db files. if this file does not exist, the 'default' as stored under the install_dir/static/mpv-conf folder will auto-populate it. if you have been using a non-default mpv conf, please re-set it one time after update, and you should be good
• the code that loads mpv.conf is now more graceful on 'missing file' errors, which now means when both the db conf and the default conf are missing
• hitting escape on a tag autocomplete input that has text will now clear that text! note that hitting escape on an _empty_ a/c input will still do 'lose focus' and then 'close dialog'
• updated the slideshow logic so that if a media with duration has a shorter duration than the slideshow duration (e.g. a gif that lasts 0.5s on a 10s slideshow), the media will keep looping until the duration is up. media that has duration longer than the slideshow time will continue to play through once completely, delaying slideshow progression and then stopping promptly when it has finished
• the string transformation system now allows 'url percent encoding' under the encode/decode type!
• fixed the 'only add existing tags' filter in the tag import options, which was denying all the tested tags. it seems to have been hit by a typo in the last three months
• the 'favourite searches' defaults now include an 'empty page' entry, which is a convenient way to simply clear a page. all users will also get this on update, feel free to delete if you don't like/need it
• opening a new search page from a tag or an active search predicate ('open a new search page for...' or middle-click) now copies the file service (e.g. looking at trash) from the original page
• opening a new search page in the 'all known files' file domain when the tag domain should be 'all known tags' (a currently unsupported combination) now coerces the tag domain to 'all local tags'
• checkboxes should now appear again on the collect-by dropdown in Fusion (and hopefully any other) style
• fixed an issue where entering 'namespace:*' explicitly would show the much less efficient wildcard search rather than the efficient 'anything' namespace search
• fixed an issue where wildcard search could include multiple asterisks in a row
• fixed an issue with page duplication where the main management object was not being duplicated properly until a session reload, meaning the two pages would sometimes share signals and changes
• an old wx delayed hide/show performance hack is removed, making the floating autocomplete dropdown now update more smoothly to resize or move requests, such as when the main gui window is dragged
• the program base installation directory is now calculated more accurately, both when running from source and the frozen build, and when launched using a symlink
• install dir and db dir are now specified in the help->about window
• the petition page content checkbox list now has a taller minimum height
• improved error text reporting in hydrus service login failure, hydrus service delay reason-setting, and all 'cancelled' errors across the program
• the review services panel now has elided... text. when unusually long errors propagate up to its status texts, it now won't suddenly jump to 2,000 pixels wide. full text appears in tooltips
• code refactoring: the tag autocomplete input now now takes responsibility for the active predicate list above it
• refactored some tag lists and added typing hints to improve how current page predicates are determined
• did some prep work for tag filters supporting wildcards, but it isn't ready yet
• cleaned up some wx->Qt data fetching code
• misc code cleanup

• version 388

• favourite searches:
• hydrus can now save, load, and edit favourite searches. this first system stores searches with a name and an optional folder name, and contains search predicates, file and tag domain, whether the search is live or not, and optionally sort-by and collect-by
• this is program-wide and all accessed through the new 'star' icon menu button beside any 'read' tag autocomplete input on search pages, duplicate pages, export folder ui, and file maintenance selection
• wrote a favourite searches manager
• wrote a dialog to manage favourite searches
• wrote a dialog to edit a single favourite search
• wrote load and save search functionality
• autocomplete dropdowns that have buttons beside them now stretch their floating dropdown windows across the button width also
• cleaned a variety of search code, simplifying objects and responsibility
• cleaned up some collect-by ui code
• refactored sort and collect controls to better location
• refactored search constants
• numerous small search code fixes and cleanup
• renamed clientguipredicates to clientguisearch
• .
• the rest:
• a note from the users managing Hydrus Companion: The Chrome Web Store release of Hydrus Companion is no longer available due to publishing issues. If you have been using it in the past, please install the extension manually as outlined here instead: https://gitgud.io/prkc/hydrus-companion
• the default e621 downloader is updated to their new system, thanks to a user's submission. if you log in to e621 with hydrus or the hydrus companion and discover some tags are now blacklisted, please check your blacklist settings on your account on the site
• an old test e-hentai login script from 2018 that is no longer in the client defaults will be deleted from clients that still have it today. if the user has no other login script for e-hentai, the domain entry will be deleted as well. this removes potential technical barriers for users that wish to use hydrus companion to access e-hentai, which is now the recommended method
• hydrus mpv now has an appropriate stream title, which propagates up to the os-level sound mixer. it was previously the ugly hydrus filename
• improved error handling when mpv is passed an invalid conf
• the default mpv conf now has audio normalisation that seems to work ok
• fixed an issue with the 'delete/move out missing/corrupt file' file maintenance job where record deletes were not processing correctly. it now deletes the file record correctly and also clears that deletion record, to make re-import of the correct file, if found, easier
• all hydrus menu labels are now "middle...elided" when they are greater than 64 characters
• all new hdd, url, and simple download pages should now obey the 'remove files when trashed' rule. pages in existing sessions will not
• updated the user-created CutieDuck darkmode qss file to the latest version, which alters the recent hydrus qss styling colours like green/red button labels
• did a full pass of all service fetching--all file and tag services should now present in lists and tabs in service_type, alphabetical order, e.g. for manage tag siblings, the tabs will always be local_tags, tag_repositories, both in alphabetical order
• fixed an issue where a 'get darker or lighter comparison colour' calculation was not working well for black or very dark colours
• if subscriptions or general network traffic is paused, the bandwidth section of the main gui statusbar now says it
• the status bar now tooltips each section
• clarified some labels on the edit url class panel
• moved all delayed focus-shifting code to a more stable system
• cleaned up how the global icon cache is initialised and referenced
• updated the hydrus project gitignore to hide all db, log, server, recovery, and media files that could be under the db directory
• updated the endchan links in the help to have a .org secondary link
• more general code refactoring

• version 387

• the sort-files-by dropdown is now a button that launches a nested menu. it still supports mouse wheel events. it should now be quicker to find what you want!
• added 'sort by framerate' to regular file sort. it works for file search at the db level as well, when mixed with system:limit
• under options->sort/collect, the namespace sort-by ui has finally had its makeover. it now has add/edit/delete buttons and up/down buttons for reordering how the entries will appear. it also deals with bad input better. furthermore, namespaces that have hyphens (like 'creator-id') are now supported in namespace sort (and hence collect-by dropdowns!)!
• numerical (multi-star) ratings can now be set by dragging the mouse across the line of stars
• added 'duplicate page' to the page tab right-click menu! it just makes a copy of the page or page of pages right beside it
• system:everything will now always show up in non-query-page autocomplete dropdowns (such as in the file maintenance dialog)
• wrote a maintenance routine to repopulate and correct the tag text search cache. it is possible to trigger this (though it is typically pointless) from the database->maintain menu
• updated the characters that are ignored in autocomplete tag text search rules, which help skip over unusual characters and assist word-break discovery for searching for tags like '[intensifies]'. as well as the previous brackets, braces, paretheses, quotes, and double-quotes, now slash, backslash, hyphens, and underscores(!) are ignored. searching for 'bbb' will now match a tag 'aaa-bbb', and searching for 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'!
• to effect the above change, the client will take a few seconds to a minute to update
• the above tag text search rules now collapse contiguous unusual characters, or combinations of whitespace and characters, better
• namespace and simple wildcard search inputs no longer have the tag text search rules applied to them, meaning you can now search for these unusual characters more specifically when desired
• updated the derpibooru gallery search objects to use their api, thanks to a user's submission. this re-enables the 'no filter' mode
• added watcher support for tvch.moe, which works with an existing 4chan-style parser
• the 'add the ptr' help item now warns the user about the ptr's modern drive storage requirements (4GB download+files, 25GB db). the help files are also updated
• I believe I fixed the sometimes crazy fast media drag-move that could happen in archive/delete and duplicate filters
• fixed an old uncaught wx->qt issue with the simple downloader where editing the formulae would throw an error
• fixed a bug in the 'move highlighted thumbnail' code in the rare case where the currently focused thumbnail can not be found
• text input dialogs are now mostly wider
• refactored some ui code, cleaning up core objects and import hierarchy
• did some controller/gui refactoring, pushing on untangling things
• cleaned up a bunch of no-longer-used import statements
• misc ui code cleanup
• slight rewording of database menu
• prepped shortcuts system to ignore a window-activating click (for the media viewer filters), but can't turn it on yet as media viewer clicks are not yet fully plugged in

• version 386

• gifs and mpv:
• the client now parses gifs for loop count metadata (some gifs say they should only be played x times through, usually 1). options->media now has a checkbox to control whether this value should be obeyed. both the native viewer and the mpv viewer should follow this. default value is still to loop indefinitely
• if gifs are set to play with the mpv player, those without duration will now still be loaded in the native image viewer. the media viewing options ui now notes this
• the mpv.conf file used in the mpv window can now be changed under options->media. it _should_ update the conf for all open mpv players on options dialog ok. added to the hydrus static mpv-conf directory are three new 'test' mpv confs for high quality and two audio normalisation tests. all test feedback and recommended conf info is welcome
• .
• ui cleanup and improvements:
• the media viewer mouse autohide time is now customisable under options->media, including disabling it completely. it defaults to 700ms
• improved the timing and reliability of the media viewer mouse autohide code
• the mouse should now never autohide while a dialog is open
• improved the bad colours of the splash screen. it should now be all one colour now, no ugly stand-out white square or other hardcoded colours. hydev also deployed his unparalleled gimp skills to get a white fade around the transparent-background hydrus icon, so it should look correct in darkmodes as well
• created a default_hydrus.qss file in the qss folder in order to handle formerly hardcoded colours using hydrus-specific classnames and properties. as well as being loaded by default, this qss file is prepended to any custom stylesheet, so any custom stylesheet that includes its own versions of the hydrus-specific entries will override the defaults. this qss will get more work in future
• added on/off buttons to hydrus default qss and converted existing object to use class and properties to obey this
• added a variety of valid/invalid/warning text colours to hydrus default qss and converted existing text objects to use classnames to obey this
• added accept/cancel buttons to hydrus default qss and converted all green/red buttons across the program to use classnames to obey this
• the migrate database dialog now has an outright 'remove location' button to reduce confusion and speed up removal of high weight locations
• if a location does not exist on the migrate database dialog, it will now stop throwing multiple error popups every time the list slightly changes, and will complain if file rebalancing is attempted, and will provide different 'remove' yes/no messages if that missing location currently has files or not
• slight ui touch-ups to the migrate database dialog
• if a window that remembers its position attempts to re-position to a location not on a current display, the windowing system now attempts to rescue it to the primary display, with appropriate popup messages given and errors caught more gracefully
• extended these off-screen rescue calls to windows that pull their position from their parent. e.g. if you open the options dialog while the main gui is half over the left side of your screen, it should rescue to the primary display
• windows that position off the center of their parent now calculate that reliably on the parent window, not just the parent widget, which never really worked as intended
• windows that have no position memory and no parent to pull center/topleft position from will now appear center/topleft of the monitor your mouse is on
• the splash screen now appears centered on the monitor your mouse is on
• cleaned up and improved a bunch of window/screen coodinate code, moving 'space on screen' calculations to 'space on screen minus taskbar' and similar
• unified a 'dialog is open' check across the program
• cleaned up the old wx->Qt size, coordinate, and colour conversion code
• cleaned up some old wx->Qt calculation code
• improved 'light' and 'grey' colour detection code to now work in HSV
• improved colour changing code to now work in HSV
• improved some internal single-shot scheduled job code

• version 385

• mpv:
• the mpv window is now plugged into the slideshow system, so when an mpv window has played its media once, a media viewer browser currently slideshow-ing will now correctly know to move to the next media
• slideshows of videos should progress to the next file faster and more smoothly, with no more half-second blit of the start of the movie before the next file loads
• pausing video/audio is no longer cause for the slideshow to move on--now only the 'has played once through' check will naturally trigger it. you can now more reliably seek/scrub a video during slideshow
• the mpv window and native animation player is now better about pausing while a seek drag (scrub) is ongoing
• various music videos that have 1 frame should now show a seekbar correctly for the mpv window, with correct timecode based seeking
• fixed a bug in the video/audio resizing code that meant zoom in/out cycles would move a video player down a few pixels off center
• the blank audio mpv window now has basic hardcoded zoom support and will scale down to a too-small viewer window, so you can still access the seek bar
• fixed some 'has duration' calculations for audio/video that has duration but no frame count
• .
• shortcuts:
• if a shortcut handler on an individual window does not have a double-click mapping for a command--and furthermore if none of its parent windows that are fully plugged in have one either--it will attempt to map the single-click version of the event as a backup. so now if you have an archive/delete filter, you can click fast again and the double-clicks will be interpreted as single-clicks (unless you map double-click to mean something else on the media_window or one of the media_viewer parents)
• the media viewers across the program are now fully plugged into the shortcuts system for key presses, and half-plugged in for mouse clicks
• 'close_media_viewer' is added to all 'media_viewer' shortcut set types. enter/return/escape are now defaults for 'media_viewer' (applying to all), and middle-click/double left-click are now defaults for 'media_viewer_browser'. this is no longer hardcoded. if you are a madlad, you can now unmap all 'close_media_viewer' commands
• double left-click is now assigned to 'keep' in the 'archive_delete_filter' shortcut set. due to the new click/double click rules above, this means that by default double clicking a video/audio in the archive/delete filter will now mean 'keep and move on' on that fast second click!
• edit shortcut set ui now sorts its command list on load
• significant shortcuts refactoring
• general shortcuts code and debug code cleanup and improvement
• .
• the rest:
• local file import pages and most downloaders will now report more file import steps to their status labels. most will blit by too fast to see, but if it hangs for a bit, you will now see the step it is caught up on. I imagine in most cases, this will be metadata generation for large videos
• fixed a variety of searches that could return files not filtered to the current file domain (e.g. files in trash while in my files, or not in ipfs while in ipfs) when the search did not include an inclusive tag
• updated the default danbooru file page parsers to get their new tag format, thank you to a user for submitting these
• a popup message now appears while sessions are loading. it auto-dismisses once the load is complete
• the edit media view options dialog (under options->media, launched from the filetype list) is now better at disabling non-applicable widgets based on filetype
• fixed an issue where clicking from the autocomplete dropdown floating window to the same control's text input could result in a single flicker-frame where the dropdown is hidden
• tightened the size of the splash screen white background. figuring out appropriate colour from the current stylesheet remains elusive lmao
• cleaned up and wx->Qt converted a variety of event handling code
• updated some 'mime' labels across the program to the new 'filetype' wording

• version 384

• shortcuts:
• the shortcut system now supports mouse double-clicks--left, right, or middle
• the shortcuts system now differentiates between press or release single mouse clicks--although complete support for release mouse events may be a bit patchy, as full mouse integration is ongoing
• the shortcut edit ui is now simpler--the command type is selected by a list, and the individual command sub-panels hide and show as appropriate--no more stupid 'set command' buttons
• the shortcut edit ui now has a 'restore defaults' button that will restore an individual set back to default settings
• two new shortcut sets are added--'media_viewer_media_window' and 'preview_media_window'. they control pause, pause/play, open_externally, and close/launch_media_viewer respectively. they work on the static image viewer, the native animation widget, and the new mpv player, and they support mouse clicks. the old pause/play (formerly left-click) and open_externally (double left-click) commands are no longer hardcoded
• by default, the preview window's media window now launches the media viewer on a middle- or double-left-click
• 'media_viewer_browser' shortcut set now has 'release right-click' bound to 'show_menu', a new command, which is no longer hardcoded
• most menus across the program can now be opened with the keyboard context menu key
• the 'global' shortcut set now has 'exit_application', 'exit_application_force_maintenance', and 'restart_application' commands
• fixed the rating increment/decrement command option not hiding in non-'media' shortcut sets
• fixed some issues loading edit ui for shortcuts with rating actions
• significant refactoring and some cleaning of shortcut code
• .
• the rest:
• mpv windows should not longer get a single frame of previous-window-stretch when flicking between one mpv media to another with a different aspect ratio on the same media canvas. when a video is caught in a frame of loading, it should now flicker a frame of black
• switching from one static image or native animation to another of the same type _should_ be less likely to do a single frame of stretch when transitioning. when an image or animation transition is caught on a new frame, it _should_ now flicker a frame the same colour as the media canvas background
• the string transformation edit panel's individual transformation rule edit panel has had some more work: much like with shortcuts, the controls now hide and show based on transformation type, the controls' text labels now change based on transformation type, and the example text now updates on any widget change. the manual 'update example' button is removed
• fixed a typo that caused an error when establishing the correct mouse cursor to use over the volume control when hydrus was using PyQt5 (rather than PySide2)
• in order to reduce accidental micro-drags that cause mpv load-pause issues, starting a thumbnail drag now takes more pixels and requires a smoother drag to start, let's see how it goes
• improved the show/hide logic of the floating autocomplete dropdown window. it should now also reliably detect when window focus goes from the dropdown itself to another window
• fixed a bug where clearing the deletion record of a deleted tag would not remove the record from the fast cache that populates thumbnail tags (making it seem on most file loads that the tag still existed). if you were hit by this previously, please hit _database->regen->a/c cache_ one time to resync the cache
• relatedly, thumbnails should now correctly live-update their deleted tags on deletion record clearance updates
• if mpv is not available, opening the about window will now make a popup with the actual import error trace
• significant refactoring of various ui code

• version 383

• mpv:
• updated the prototype volume/mute controls on the top media viewer hover window to be a proper 'speaker' icon button for mute with a volume slider that pops up or down on mouse-over
• the new volume control is on the hover window and any media that has audio
• the right-click menu of the preview viewer and media viewer now have volume submenus to set mute/volume
• the client now has multiple volumes and mutes:
• for mute, there is a global mute which overrides everything, and the preview and media viewers have their own mutes that just apply there.
• under options->audio, you can choose whether preview windows have their own separate volume value, default is yes they do
• there is a new shortcut set called 'global', which applies on the main gui and the media viewer both, and which currently has actions to alter global mute. by default, ctrl+g flips global mute
• after reports of unusual rendering bugs for some users, the default mpv.conf is now more barebones. more work will happen here
• .
• linux:
• the linux release is now built on Ubuntu 18.04 (was 16.04). unfortunately, my build packager bundled in a variety of surplus libraries, so the archive has bloated somewhat--I have removed some that I am confident are not needed, but I may have made a mistake, and there are likely more that can be taken away
• the linux release now comes with mpv support
• please let me know if you have any errors running this build or loading mpv. early tests seem good though!
• .
• the rest:
• the launch/exit splash screen now uses a cleaner Qt-compatible layout system. It resizes and obeys stylesheets better, colouring text and background according to current style
• removed the 'has duration' text label option from 'audio and duration' options panel as it is no longer used, and renamed the panel back to just 'audio'
• the string transformation edit panel's individual transformation edit panel now shows that transformation step's example string and the transformed string, which is updated by button. this edit panel will get some more love soon, including dynamic hide/show of applicable controls and live updates of the example transformation as you type
• misc ui layout improvements
• misc ui improvements

• version 382

• mpv:
• rolled Qt back from 5.14.0 to 5.13.0 on the releases, which seems to have fixed our 'event queue sometimes halts until mouse move' issue that occurs after initial mpv load. some other ui and media viewer resize jank seems to be cleared up. I dunno what happened with 5.14, and I don't suspect it as the problem nearly as much as my currently borked Qt event processing code, but rolling back seems the easiest solution for now
• fixed an issue that was crashing non-windows that were able to load mpv
• mpv now loads an mpv.conf from install_dir/static/mpv-conf. please feel free to swap in another conf or edit that one as you like. I would be interested in feedback
• default mpv conf is now set to specifically enable some hardware acceleration to improve playback for some users, and to never load sidecar files like subtitles as this was introducing incredibly large load lag for users with large/high latency file storage
• fixed a new issue where preview windows were not unloading media (particularly significant for mpv with audio) on page change and client shutdown
• fixed an issue with global volume propagation to multiple open mpv widgets
• .
• the rest:
• added two dark qss stylesheets from the user-creation github repo to the default install
• when zooming out from a zoom that makes the media bigger than the media viewer canvas to a zoom where it fits, the media will now recenter. see if you like this, maybe it should be an option
• to help forestall unnamespaced filename tag spam in various new-user scenarios, the 'filename' checkbox-and-namespace widget on the filename tagging options panel now initialises with 'filename' as the namespace
• fixed a recent window sizing issue with the 'the client is already running' dialog not appearing
• file export paths that include subdirectories that could possibly have empty text, like "[creator]/[page]", will no longer error when this is so (e.g. if a file in this case has no creator tags)--they will eliminate the subdirectory entirely, becoming "[page]". this should work for all platforms and for any nested subdirectory
• fixed an issue with some fractional dataspeeds below 1KB/s displaying with many significant figures
• improved some custom event handling definition code
• reworked hydrus's internal object publisher/subscriber messaging system to be more Qt-happy
• if the file import tagger is given a neighbouring .txt file to pull tags from that does not decode to utf-8 nicely, it now catches and reports the error more gracefully
• reworded a bit of the installing help and first-start popup to emphasise that hydrus does not auto-update
• added links to https://github.com/Zweibach/text/blob/master/Hydrus/PTR.md , a new guide for the PTR, to the help
• removed the old 'hardcoded shortcuts' help entry, since it is increasingly irrelevent

• version 381

• mpv:
• mpv is now available and the default for all windows users
• I believed I have eliminated the final reported mpv crash
• mpv load and unload delays are greatly reduced. initial load still takes about half a second, but subsequent loads are now as quick as native renderers
• mpv seems to work well for gif and apng
• added a very simple global volume slider and audio mute checkbox to the media viewer top hover window. this was a quick patch--much better controls and shortcuts will come in future
• mpv windows now properly re-show the cursor on mouse movement
• unified mpv mouse press/release handling with native animation--click down now does pause/play and starts a drag event
• unfortunately, in some cases embedding mpv requires overriding local OS number rendering (e.g. 1,234 vs 1.234). hydrus number rendering is now coerced to the english style with commas until we can figure out a better solution--sorry!
• cleared up an issue where simple clicks on page tabs would trigger micro-page drags that were immediately cancelled. this situation was exacerbated when the page being left had an active mpv window. the flicker of page drag cursor is now gone, and some weird situations where static clicks during busy time could move a tab should be fixed
• eliminated the recent issue in the media viewer where transitioning from one media type to another through navigation, particularly mpv->other, would flicker a single frame of the last 'other' media shown(!)
• fixed a bug where repeated mpv views in the preview viewer could disable client file drag and drop
• the bug where thumbnails may not waterfall in unless the mouse is moving after some mpv videos are loaded for a page is relieved but not completely fixed
• if the preview window is collapsed and hidden, media will no longer ever load into it
• fixed an edge-case bug where the mpv window would not like being told to show nothing when it was already showing nothing
• wrapped mpv load errors in a basic graceful catch
• fixed an issue some users had with loading mpv's dll
• .
• file types:
• a new file metatype, 'animation', is added, for gif and apng. these are no longer considered 'image' for a variety of purposes
• the filetype selection panel, which is used in system:filetype and import folder UI, has had an overhaul--it now has tristate 'mime group' checkboxes to represent a half-filled group and expand/collapse buttons to hide the tall filetype lists. individual filetype lists will start hidden unless their default value is a partially filled group
• the media view options have a similar overhaul: they are now collapsed to general filetypes by default. you set view and zoom options for the generalised 'video' type under options->media, and if you want to set specific options for webm or anything else, you can add/delete those types to override the general default
• the new default options for a fresh client are just for these general types. if mpv is available, video, animations, and audio now start with mpv as the default viewer. video and animation zoom is now flexible (not fixed to 50%, 100%, 200%) and will fill the media canvas
• all media view options will be reset to this simple default on update! if you have specific zoom or display preferences, please reset them after the update--but you might like to play with mpv a bit first, as it renders at large and smooth zooms very well
• .
• the rest:
• the new thumbnail right-click file selection routine will now only focus and scroll to the first member of the selection if no other members of the new selection are already in view
• fixed some caching code and sped up the new select/remove menu count generation (which can lag for very large pages) by two to six times
• sped up file filter counting code by about ten percent
• fixed weird layout on: migrate database panel, duplicates page (left and right), edit shortcuts, edit import folder, and the filename tagging panel
• fixed an issue where the media viewer's hover windows might flicker into view for one frame when the mouse moved over the center of the media viewer for the first time
• fixed a media viewer shutdown issue that would sometimes lead to the first file in the list being opened in the shutting-down viewer for an instant or highlighted as the new thumb focus
• the file maintenance system that queues up missing/broken files' urls for redownload will no longer re-select the download page on every new url
• fixed an issue where a downloader's tag blacklist was not being applied on the child files of certain kinds of multiple-file post (such as with pixiv)
• deleting a very long tag should no longer create a very wide confirmation dialog in the manage tags dialog
• fixed some 'the panel grew a bit, but the parent window didn't grow quite enough and now it has scrollbars for two pixels of extra content' sizing issues
• fixed some dialog sizing calculations when the parent window was borderless fullscreen
• maybe fixed a rare event processing bug
• improved quality of some misc data comparison code across the program
• did some significant backend event/pubsub code cleanup, mostly related to getting mpv working a bit cleaner
• improved thumbnail rendering time
• improved smoothness of thumbnail fade animations (at least for when they are working right, ha ha!)
• misc fixes

• version 380

• basic mpv support is added. it comes with the windows build this week, and is a prototype meant for initial testing. the library is optional. users who run from source will want 'python-mpv' added via pip and libmpv available on their PATH, more details in running_from_source help
• took an qt-mpv example kindly provided by a user, updated it to work with the hydrus environment, and integrated it into the client as a new choosable view type under audio/video filetypes under options->media for advanced users
• reworked how the 'start paused' and 'start with embed button' media viewer options work under options->media. these are now separate checkboxes, not combined with the underlying 'show action'. existing embed/paused show actions should be converted automatically to the correct new values
• unfortunately, due to some python/qt/libmpv wrapper mouse interaction issues, mpv's 'on screen controller' overlay is not available
• for now, left click pause/plays the mpv window, just like the native mpv window.
• preview/next frame shortcuts should work for the mpv window when playing video
• no volume/mute controls yet, these will come in the coming weeks, including global mute settings
• updated media show and sizing code to account for mpv widgets
• reworked my animation scanbar to talk to mpv, and for my mpv window to talk back to it
• improved the animation scanbar to be more flexible when frame position and num_frames are not available, both in displaying info and calculating scanbar seek clicks
• mpv api version added to help->about
• .
• new downloader objects:
• thanks to a user, updated the 'pixiv artist page' url class to a new object that covers more situations. the defunct 'pixiv artist gallery page' url class is removed
• added 8kun and vch.moe download support. I got started on julay, smug, and endchan, but they were a little more tricky and I couldn't finish them in time--fingers crossed, next week
• .
• menu quality of life:
• a right-click on thumbnail whitespace will now not send a 'deselect all' event! feel free to right-click in empty space to do an easy remove->selected
• remorked the tag menu layout to move less frequently used actions down:
• - moved the discard/require/permit/exclude search predicate actions down
• - moved 'open in a new page' below select and copy
• - moved copy above select
• and some misc menu layout improvement on this menu
• fixed some labelling with the discard/require/permit/exclude verbs on negated tags
• right-clicking on system search predicates now shows the 'copy' menu correctly
• system predicates that offer easy inverse versions (like inbox/archive) should now offer the 'exclude' verb
• when right-clicking on a single tag that has siblings, its siblings and those siblings' subtags will now be listed in the copy menu!
• copying 'all' tags from a list menu, with or without counts, will now always copy them in the list order
• across the program, all menu 'labels' (menu text items that do not have a submenu and have no associated action, like 'imported 3 years 7 months ago') will now copy their text to the clipboard. let's see how it goes
• .
• other ui quality of life:
• across the program's UI, filetypes are now referred to with simpler terms rather than technical mimetypes. instead of 'image/jpg', it is now typically just 'jpeg'
• the 'remove selected' buttons on the gallery and watcher pages are now smaller trash icon buttons
• the new page chooser will now auto-dismiss if it loses focus--so if you accidentally launch it with a middle-/double-click somewhere, just click again and it'll go away
• hitting enter or return on the new page chooser now picks the 'first' button, scanning from the top-left. hitting enter twice now typically opens a new 'my files' search page
• added pause_media and pause_play_media shortcuts to the media_viewer shortcut set. new clients will start with space keypress performing pause_play_media
• added pause_play_slideshow shortcut to the media_viewer_browser shortcut set. this shortcut is no longer hardcoded by space keypress
• the six default shortcut sets now have a small description text on their edit panels
• the options->media edit panels now enable/disable widgets better based on current media/preview action
• added a checkbox to _options->gui pages_ to set whether middle-clicking a tag in the media viewer or a child tag manager to open a tag search page will switch to the main gui. default is false
• mr bones now reports total files, total filesize, and average filesize
• mr bones now loads your fate asynchronously
• .
• the rest:
• added tentative and simple realvideo (.rm) and realaudio (.ra) support--seems to work ok, but some weirder variable bit rate formats may not, and I have collapsed the various different extensions just down to .rm or .ra
• added trueaudio (.tta) audio support
• fixed a bug from the recent search optimisations where a bare inbox search would not cross-reference with the file domain (so some trash could show up in a simple inbox/'my files' query)
• fixed an issue with searching for known urls by url class where the class was for a third-or-higher-level domain and was not set to match subdomains (this hit 4chan file urls for a few users)
• fixed the issue with 'open externally' button panel not clearing their backgrounds properly
• fixed some of the new unusual stretchy layouts in the options dialog
• removed overhead from subscriptions' 'separate' operation, which should stop super CPU hang when trying to split a subscription with hundreds of thousands of urls
• fixed an issue where the advanced file delete dialog would not show the simple 'permanent delete' option when launched from the media viewer's right-click menu
• fixed the select/remove actions for local/remote
• fixed 'set_media_focus' from manage tags to correctly activate the underlying media viewer as well as set focus
• stopped the 'file lookup script' status control from resizing so wide when it fetches a url
• fixed a rare mouse wheel event handling bug in the media viewer
• reduced db overhead of the 'loading x/y' results generation routine. this _may_ help some users who had very slow media result loading
• cleaned up how the server reports a bootup-action error such as 'cannot shut down server since it is not running'--this is now a simple statement to console, not a full error with trace
• improved client shutdown when a system session shutdown call arrives at the same time as a user shutdown request--the core shutdown routine should now only occur once
• fixed an issue with thumbnail presentation on collections that have their contents deleted during the thumbnail generation call
• misc wx->Qt layout conversion improvements
• updated the github readme to reflect some new links and so on
• misc code cleanup

• version 379

• downloaders:
• the right-click menus from gallery and watcher page lists now provide a 'remove' option
• gallery and watchers now provide buttons and menu actions for 'retry ignored'
• activating a file import status list (double-clicking or hitting enter on a selection of rows) now opens the selection in a new page
• file import status buttons now have show new/all files on their right-click menus
• on gallery and watcher pages, the highlight, clear highlight, pause files, and pause search/check buttons are now smaller bitmap buttons
• as the old default pixiv login script is completely broken, any client with this active will have it deactivated and receive an update popup explaining the situation and suggesting to use Hydrus Companion for login instead
• updated the derpibooru downloader
• .
• search:
• when search predicates are added to the active search list, they are now better able to remove existing mutually exclusive/redundant predicates:
• - system:limit, hash, and similar to predicates now remove other instances of their type
• - system:has audio now removes system:no audio and vice versa
• - any search predicate will remove system:everything (see how you feel about this)
• improved 378's db optimisation to do tag searches in large file domains faster
• namespace search predicates ('character:anything' etc...) now take advantage of the same set of temporary file domain optimisations that tag predicates do, so mixing them with other search predicates will radically improve their speed
• wildcard search predicates, which have been notoriously slow in some cases, now take full advantage of the new tag search optimisations and are radically faster when mixed with other search predicates
• simple tag, namespace, or wildcard searches that are mixed with a very large system:inbox predicate are now much faster
• a variety of searches that include simple system predicates are now faster
• integer tag searches also now use the new tag search optimisation tech, and are radically faster when mixed with other search predicates
• system:known url queries now use the same temporary file domain search optimisation, and a web-domain search optimisation. this particularly improves domain and url class searches
• fixed an issue with the new system:limit sorting where sort types with non-comprehensive data (like media views/viewtime, where files may not yet have records) were not delivering the 'missing' file results
• improved the limit/sort_by logic to only do sort when absolutely needed
• fixed the system:limit panel label to talk about the new sorted clipping
• refactored tag searching code
• refactored namespace searching code
• refactored wildcard searching code and its related subfunctions
• cleaned all mappings searching code further
• .
• the rest:
• m4a files (and m4b) are now supported and recognised as separate audio-only mp4 files. files with a single jpeg frame for their video stream (such as an album cover) should also be recognised as audio only m4a for hydrus purposes for now. better single-frame audio support, including functional thumbnails and display, is planned for the future. please send in any m4a or m4b files that detect incorrectly
• the remove thumbnail menu has been moved to a new, cleaner file filtering system. it now presents remove options for different file services and local/remote when available (most of the time, this will be 'my files'/'trash' appearing when there is a mix), including with counts for all options
• the select thumbnail menu is also moved to this same file filtering system. it has a neater menu, with counts for each entry. also, when there is no current focus, or it is to be deselected, the first file to be selected is now focused and scrolled to
• for thumbnail icon display and internal calculations, collections now _merge_ the locations of their members, rather than intersecting. if a collection includes any trash, or any ipfs members, it will have the appropriate icon. this also fixes some selection-by-file-service logic for collections
• import folders, export folders, and subscriptions now explicitly only start after the first session has been loaded (so as well as freeing up some boot CPU competition, a quick import folder will now not miss publishing a file or two to a long-loading session)
• the subscription manager now only waits 15s before starting first work (previously, the buffer was 60 seconds)
• rearranged migrate tags panel so action comes before destination and added another help text line to clarify how it works. the 'go' confirmation dialog now summarises tag filtering as well
• tag filter buttons now have a prefix on their labels and tooltips to better explain what they are doing
• the duplicate filter right-center hover window should now shorten its height appropriately when the pairs change
• fixed a couple of bugs that could appear when shutting down the duplicate filter
• hackily 'fixed' an issue with duplicates processing that could cause too many 'commit and continue?' dialogs to open. a better fix here will come with a pending rewrite
• dejanked a little of how migrate tags frame is launched from the manage tags dialog
• updated the backup help a little and added a note about backing up to the first-start popup
• improved shutdown time for a variety of situations and added a couple more text notifications to shutdown splash
• cleaned up some exit code
• removed the old 'service info fatten' maintenance job, which is not really needed any more
• misc code cleanup
• updated to Qt 5.14 on Windows and Linux builds, OpenCV 4.1.2 on all builds

• version 378

• if a search has system:limit, the current sort is now sent down to the database. if the sort is simple, results are now sorted before system:limit is applied, meaning you will now get the largest/longest/whateverest sample of the search! supported sorts are: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. this does not apply to searches in the 'all known files' file domain.
• after identifying a sometimes-unoptimal db access routine, wrote a new more reliable one and replaced the 60-odd places it is used in both client and server. a variety of functions will now have less 'spiky' job time, including certain combinations of regular tag and system search predicates. some jobs will have slightly higher average job time, some will be much faster in all common situations
• added additional database analysis to some complicated duplicate file system jobs that adds some overhead but should reduce extreme spikes in job time for very large databases
• converted some legacy db code to new access methods
• fixed a bug in the new menu generation code that was not showing sessions in the 'pages' menu if there were no backups for these sessions (i.e. they have only been saved once, or are old enough to have been last saved before the backup system was added)
• fixed the 'click window close button should back out, not choose the red no button' bug in the yes/no confirmation dialogs for analyze, vacuum, clear orphan, and gallery log button url import
• fixed some checkbox select and data retrieval logic in the checkbox tree control and completely cleared out the buggy ipfs directory download workflow. I apologise for the delay
• fixed some inelegant multihash->urls resolution in the ipfs service code that would often mean a large folder would lock the client while parsing was proceeding
• when the multihash->urls resolution is going on, the popup now exposes the underlying network control. cancelling the whole job mid-parse/download is now also quicker and prettier
• when a 'downloader multiple urls' popup is working, it will publish its ongoing presented files to a files button as it works, rather than just once the job is finished
• improved some unusual taglist height calculations that were turning up
• improved how taglists set their minimum height--the 'selection tags' list should now always have at least 15 rows, even when bunched up in a tall gallery panel
• if the system clock is rewound, new objects that are saved in the backup system (atm, gui sessions) will now detect that existing backups are from the future and increase their save time to ensure they count as the newest object
• short version: 'remove files from view when trashed' now works on downloader thumbs that are loaded in from a session. long version: downloader thumb pages now force 'my files' file domain for now (previously it was 'all local files')
• the downloader/thread watcher right-click menus for 'show all downloaders xxx files' now has a new 'all files and trash' entry. this will show absolutely everything still in your db, for quick access to accidental deletes
• the 'select a downloader' list dialog _should_ size itself better, with no double scrollbars, when there are many many downloaders and/or very long-named downloaders. if this layout works, I'll replicated it in other areas
• if an unrenderable key enters a shortcut, the shortcut will now display an 'unknown key: blah' statement instead of throwing an error. this affected both the manage shortcuts dialog and the media viewer(!)
• SIGTERM is now caught in non-windows systems and will initiate a fast forced shutdown
• unified and played with some border styles around the program
• added a user-written guide to updating to the 'getting started - installing' help page
• misc small code cleanup

• version 377

• qt:
• all non-menubar menus across the program now launch on click release. some previously launched on click press. a variety of related click event behaviour is cleaned up, particularly with thumbnail/tag selection on the click down. this also fixes some users' menus immediately activating the first entry on slow clicks in some ui styles
• I think I fixed the annoying single-frame delayed size-down resize on media viewer hover frames when changing media!
• the vast majority of old wx panel background colour hacks are removed, so custom stylesheets should now cover much more of the UI
• improved the new custom style and stylesheet setting, resetting, and error handling code, particularly for not re-applying the same style or stylesheet twice, and for handling un-re-settable styles (seems to be defaults initialised by third-party OS-wide Qt style) gracefully
• fixed hyperlinks not using the custom web browser launch path as set in the options
• fixed the 'migrate entire db' and 'set thumb location' buttons in the migrate database dialog
• fixed a typo bug when launching the url selection tree after adding an ipfs directory to download
• fixed two typo bugs when editing regex favourites and simple downloader formulae
• fixed an issue where custom shortcut sets could not be deleted
• fixed a typo in the edit account type panel
• fixed sorting the login listctrl when there are session logins mixed with non-session logins
• removed some old media viewer hover window display/raise hacks
• retired the 'always show hover windows' debug mode
• the media viewer will no longer perform any drag calculations on anything but left-click drag
• misc Qt code refactoring/cleanup
• .
• url searching:
• the database now stores 'known url' domain information more efficiently. it will take a few moments/minutes to reshape the db when updating
• system:known url's exact url search now runs extremely fast. this will only affect new predicates of this type, not those in existing sessions
• system:known url's domain search now runs much faster and matches subdomains of the given domain. this will only affect new predicates of this type, not those in existing sessions
• system:known url's url class search now runs much faster. this will only affect new predicates of this type, not those in existing sessions
• when entering a regex system:known url predicate, the dialog will now not OK (throwing up an error dialog) if the regex is invalid
• .
• the rest:
• the shortcut system now allows all text characters. if it has text, it should work, but it is the wild west in terms of modifier labelling. anything unusual on your keyboard like ctrl+alt+e to make æ will _display_ as ctrl+alt+æ, but the same key combination will match up in the program all correct
• added shortcut actions 'pan_top_edge', 'pan_bottom_edge', 'pan_left_edge', 'pan_right_edge' to the media viewer shortcut set that will move the current image so the respective edge is aligned with the larger canvas's
• added shortcut actions 'pan_horizontal_center' and 'pan_vertical_center' to do as above but center on that axis
• session save now hangs the UI significantly less, whether triggered by user command or auto-saving 'last session'
• saving of last/exit sessions on client close is a little faster
• the call to refresh thumbnail file info (and redraw if needed) when a file is imported or has metadata-regenererating file maintenance done will now only call for files that are actually loaded, run faster per file, run faster when the client has large collections in its session, and not hang the ui thread when waiting for the new media info to arrive
• like regular popups, modal popups (like those created when big vacuum/analyze jobs jump in) will now only appear if the main gui or an on-parent child has OS focus
• the main gui/on-parent child OS focus test now includes misc child windows like the autocomplete results hover window
• network jobs that fail for one reason or another will now be more reliably cleaned up, and their connections returned to the connection pool. this may fix the 'too many open file handles' errors some users were seeing after long term unreliable network traffic
• fixed an issue where some thumbnails that were trashed or physically deleted were being removed from 'all known files' and file repository views when it was not appropriate
• connection and downloader retry time options now have a wider min/max range when in advanced mode, with an accompanying warning label for the connection panel
• checker options times now have a wider min/max range when in advanced mode, with an accompanying warning label
• cleaned up some shutdown reporting text
• misc debug improvements

• version 376

• subscriptions:
• wrote a new subscription manager to better look after subscription scheduling
• rather than checking every four hours or after manage subs dialog close, subscriptions now record an indication of when they are next due for work, whether that is the estimated next check time or when bandwidth is free on remaining file downloads, and launch in a fifteen-minute window around that time. delays due to previous errors or user cancels are also taken into account. this reduces background cpu and i/o greatly for clients with large subs
• if a sub is paused, or all its queries are paused, it will now never be reloaded after first load until a change via the manage subs dialog
• furthermore, if a single sub takes a very long time to work, the whole sublist can re-cycle if they come up due for more work before it is finished
• if a sub query is DEAD but still has outstanding files to download, it will no longer automatically pause
• subs now clean up more tidily if they are running on a program exit
• the subscription popup now shows check/file progress based on the number of queries that appear to have pending work. instead of 'query 300/450' with 420 that aren't due, you'll get 'query 12/30'. if a query becomes due during a round of checking, another round of checking will run
• if a subscription fails to load from the db, the error is handled better and no more subs will run in that boot
• improved subscription startup checking logic, tightening up various paused/dead/cansync tests
• improved subscription interrupt checking logic, tightening checks on global network pause and various shutdown scenarios
• cleaned up some more subscription code in prep for data storage breakup
• .
• qt:
• added experimental Qt style settings to the new options->style page! all users should now be able to set Fusion style, and perhaps some alternate OS styles. advanced users are invited to play around with QSS stylesheets (although be warned that some of hydrus's custom colour system overrides QSS, so more work is needed here), which will be extended and made user-friendly in coming weeks
• fixed tab position calculations for all tab/media drag and drops for tab bars that are centered or otherwise positioned far off top-left alignment
• fixed tab drag and drop event object handling for macOS. tab and media DnD is now enabled for macOS
• the popup toaster can now unhide if an on-top-of-parent non-modal frame (like review services) is focused (so hitting 'process now' should show you the work)
• fixed a variety of old hacky wx close-window veto tests. the 'close client?' confirmation dialog will now reliably veto a close requent on 'no'/cancel, dialog close events that are vetoed (such as closing the manage tags dialog with pending tags) will now veto more than just the first time, and several bad media viewer archive/dupe filtering cancel and end-of-window events should now work more cleanly and correctly. users who had crashes at the end of filtering may find they are stable again
• as a quick patch against some multiline notes and statuses, list controls now force single-line text in all cells
• list controls now tooltip all cells
• fixed the shutdown splash not updating after the daemons shut down (lmao)
• 'modal' message dialogs, which are created by blocking maintenance tasks such as vacuum, will no longer raise the program to the foreground on creation
• should have fixed the taglist vertical positioning jank that could occur in the row after a tag with a tall emoji unicode character (and also sometimes kanji/hangul)
• fixed a typo bug that was throwing an error for the upnp port widget in the local client server management panel when 'allow non local connections' was checked
• improved stability of bandwidth review panel bandwidth rules refresh
• improved stability of review services bandwidth rules refresh
• improved some dialog cleanup code
• reverted a bad environment-setting change put in last week that was causing some running-from-source users trouble
• misc qt code cleanup
• .
• the rest:
• updated the default pixiv tag search downloader to one submitted by a user. it now uses their api
• updated the default twitter username lookup to a downloader submitted by a user. it fetches just the media tweet feed, making it more efficient. also added (but not linked by default) is a new tweet parser that can fetch most videos using a third-party site, advanced users may wish to play with this
• added a {file_id} term for file export phrases that substitutes a unique and permanent numerical file identifier
• fixed an issue where idle maintenance jobs could sometimes sneak in a few milliseconds of work during certain long shut down pauses, such as while waiting for a 'should I do shutdown work?' dialog to return. program shutdown should be snappier for many users as forced startup delays in these calls will no longer trigger
• added a date 'encode' string transformation rule, which takes an integer timestamp and converts it to a pretty date string. the date rules are now renamed to the clearer 'datestring to timestamp' and vice versa
• fixed page parser edit panel's 'test parse' when string transformations perform pre-parsing conversion. the handling and passing of test data for all the panels here is cleaned up throughout
• system:limit predicate edit panel now has a small label describing its sampling behaviour
• updated the various 8chan links in the client and help to 8kun, let me know if I missed any, and added Endchan bunker link to help menu
• improved some misc status text handling across the program
• refactored cache and manager code into different, simpler files
• updated sqlite on windows build to 3.30.1

• version 375

• qt:
• disabled the failed legacy high dpi scaling mode experiment (which was scaling up thumbnails and media in an ugly way) and returned to font-size-based natural ui scaling as set by the OS. a couple of non-font things like bitmap buttons and various layout margins are too small on >100% UI scale, and the splash screen is borked again, but it looks clear again. I'll keep working on this
• fixed the custom taglist at >100% UI scale, which was spacing its tags at the wrong text height. this should survive changing ui scale while the program is open and environments with multiple monitors at different ui scale
• re-fixed a critical old media-viewer-close-on-video memory leak from wx code to qt code. this was also a cause for some child ffmpeg processes not being terminated
• fixed the media viewer not redrawing correctly when the media size completely exceeds the canvas window size
• fixed the loading of the shortcut edit panel when the shortcut set a tag
• fixed some url class edit path component ui
• fixed and cleaned up some 'safe window size/position' calculations that were missing out the total frame geometry, meaning some dialogs were not moving up and left enough to show entirely on screen, and dialogs with parent-dimension gravity were not calculating initial size accurately
• fixed focusing on the already-open manage tags text input when you hit 'manage tags' on a canvas with a manage tags dialog already open
• fixed the html formula rule edit ui actually rendering html tag labels, lmao
• updated boot-password entry to use the normal hydrus text entry dialog, and fixed a hydrus password cancel not setting a 'clean' exit for the next boot
• fixed page layout splitter sash positions not resetting nicely from the menu command
• fixed keyboard delete in the manage urls dialog
• popup message titles are now in bold
• popup message titles should now multiline correctly and fill available width
• the popup messages manager should now set its min/fixed width more sensibly
• subscription popups now will be wider if space is available
• wrote a new class to manage better asynchronous updates for future Qt ui presentation
• the file, pages, and pending menubar menus, which all require a db hit to generate, now operate on this new update class. all three should update faster when able and more politely and smoothly wait when the db is busy
• reduced some accidental blocking in an old ui-update routine that kicked in when it was running hard
• if the media_viewer frame type is set not to remember its 'last size', it will now instantiate with a small min size
• when pasting new queries into a sub, if there are more than 5 or 50 that are already in or new, they will be rendered in a more compact way in order to stop the notification dialog growing too tall
• improved stability of page update, splash screen update, and perhaps pubsub update
• .
• new file maintenance jobs:
• added a new 'check for missing files' file maintenance job, where if the file is missing and has urls, those urls will be queued up in a new url downloader for redownload. the file record is not removed, preserving archive/inbox and import time
• added a new 'check for invalid files' file maintenance job that does the same deal as above with an additional expensive byte-for-byte content check if the file is not missing
• added a new 'check for invalid files' file maintenance job that only cares about invalidity--if the file is present and invalid, it is moved out but the file record is not removed
• .
• the rest:
• network jobs that receive low-bandwidth error codes from the server now use a separate wait routine (previously, they piggybacked on the connection fail retry system). they have a separate cog-menu action to override these waits
• the time delay multiple for connection errors and serverside bandwidth problems are now editable under options->connection. old default was 10 seconds base, now 15 and 60 seconds respectively
• updated the danbooru login script
• improved the precision of the thumbnail size estimate in database migration
• the alphabetisation of a url class's GET paramaters on normalise is now optional. it is a new checkbox on the url class edit panel
• when a default object fails to load from a png path, a simple error is now written to the log
• misc cleanup

• version 374

• qt environment/build:
• macOS build is useable! tab drag and drop position calculation doesn't work yet, so intra-client file DnDs and tab rearrange DnDs are disabled for now. borderless fullscreen is also disabled, feedback on this vs maximise would be appreciated
• fixed a critical bug in the macOS release that was resulting in 100% CPU repaint loop for the canvas viewer when media was loaded (wew). this may have affected certain other platforms in some situations
• the linux build has a variety of common library files removed, letting your OS rely on higher compatibility system defaults. this _should_ clean up font and other issues for users running on very new/old system libraries. if you cannot run 374, please let me know your distro and version and any error messages
• the special linux running from source document is updated, including info about Arch and PyQt5
• fixed a windows build issue that meant some animated gifs were not able to load and render correctly
• fixed a precise time fetching issue for users running from source with python 3.8
• high dpi scaling should have improved support. please report on bad layout issues and other artifacts
• fixed creating a serialised object png when using PyQt5
• fixed file save dialogs with filetype filters when using PyQt5
• fixed an important menubar related memory leak
• _seem_ to have fixed an important media viewer memory leak
• .
• qt ui fixes:
• fixed pages not collecting and sorting on creation if they do not have to, which restores the 'preserve flat unsorted order' behaviour of session loads and file drag and drop page tab creations
• fixed the cursor not unhiding on move in the media viewer when over an animation or static image
• fixed the issue where a new thumbnail panel would double-up with the old one for half a second if a menu caused the panel swap
• reworked the elided (text that cuts off...) label code to more reliably work on single lines, which fits our purposes. the network job control (esppecially on subscription popups) and top hover window should now show their long statuses without changing their parent panel's layout
• updated a variety of old text-wrap-width wx-hacks texts to instead auto-fill available space
• the various downloaders should now be careful about handling large status texts. if a multiline error or html page slips in to a status somewhere, your download pages' lists should no longer go nuts with very tall spam-filled status cells
• hydrus->discord drag and drop should be fixed if the BUGFIX is on!
• fixed page tab drag and drop to do live drag selection with 'do not follow' behaviour (this is switched by holding down shift during drag), and, in this case, got it to return to the original page's neighbour/parent once the drop is complete
• fixed 'center' dialogs positioning on the center of their parent windows, rather than the center of the primary screen
• fixed the hover windows not passing shortcuts up to the media viewer when not consumed
• fixed some misc 'can I consume a shortcut' focus/active checking code
• fixed the various hide/parents/siblings tag menu items for tags with counts
• fixed the main gui and other non-dialog windows remembering their pre-maximise/fullscreen sizes if set to remember size and previously closing while maximised/fullscreened
• menubar menus should now show description text in the main gui statusbar on mouseover of their items
• fixed a bad menu initialisation in the canvas preview panel
• fixed a little page splitter bork and improved size of preview window on initial boot
• fixed the edit notes dialog when launched from the media viewer
• fixed a couple of text edit issues in edit url class panel
• fixed page up/down scroll for taglists
• fixed page down scroll for thumbnail grid, and fixed page up/down distance
• fixed thumbnails not scrolling into view if they are keyboard-selected slightly off screen but within the scroll option percentage threshold
• misc layout and style cleanup
• misc refactoring
• .
• misc:
• you can now set the maximum size of duplicate filter pair batches (default 250) under options->duplicates
• when an ipfs service fails to pin a file and returns no hash or the empty multihash, this is now recognised, info dumped to log, a simple popup message sent, and the job continued. this is just a patch--better error handling here will come later
• if the client or server are launched with a custom temp_dir that does not exist, it will now attempt to create it (previously errored out)
• fixed a clean exit after certain client boot fail error handling, and repeated cleaner exit for the server
• added some new memory profiling actions to the help->debug menu
• parallel subscriptions should now initialise with less of an aggresive CPU spike
• if the client or server crash before the application can be launched, the crash log is now called hydrus_crash.log. if the db dir is not yet established, it will now try to find and put it in your desktop and, failing that, then your user dir
• the client no longer prints 'booting db' twice
• a variety of misc code cleanup and fixes

• version 373

• qt:
• hydrus now uses Qt for its client's user interface, migrating from wx. this is thanks to a huge effort by a user, who delivered converted code for hydrus dev to finish off
• a number of hacks and patches remain to compensate for old systems, which hydrus dev will slowly clean up in normal work. ui bug and layout issue reports would be greatly appreciated
• shortcut storage had to be converted from fixed wx enums to an independant system. there is a small chance that one of your shortcuts, particularly if it is on the numpad, may have been converted wrong (unusual Enter/Return buttons may be hit here). if one is not working, please check what hydrus thinks it is and try re-entering it
• added tentative support for 'Mode_switch' keyboard modifier, for X11 users (and perhaps some users' AltGr?)
• autocomplete results can now float in a popup window in dialogs like manage tags! they'll still embed by default, but there are now separate float/embed options for 'main gui' and 'other frame' a/cs
• autocomplete results can now float in linux and macOS ok!
• page drag and drop now navigates as you drag, so dropping into a page of pages works by you hovering over it and then dropping in the tabbar below, inserting exactly where you want the page to be
• a couple of text inputs in the program--the watcher and gallery search pages' text inputs, particularly--now use nicer 'placeholder' text, which isn't real and only shows as grey text when the input is empty
• for now, moved to icons for thumbnail 'has audio/duration' indicators, rather than the custom labels
• to run the hydrus client from source, qtpy is now needed. either pyside2 (default) or qtpy5 is needed. QtCharts is optional. wx and matplotlib are no longer needed
• .
• misc:
• 'archive/delete filter' now appears even when no file is focused. it also appears when no files are selected--and will apply to everything
• the system predicate edit panels now support static buttons for easy one-click select for common predicates. duration, has audio, limit, and num tags now have these
• system:duration and system:num tags now render a special label if they are >0 or =0
• system:untagged is now removed from the normal list
• fixed a critical cpu inefficiency in the file maintenance manager's new always-on maintenance, which was lagging several users' browsing sessions while it was working
• fixed ctrl+mousewheel tag autocomplete results navigation to skip over multirow parent results
• fixed an issue where resetting to default bandwidth rules for a network context would not update the ui properly
• fixed a bug when adding a parent/sibling from autocomplete results list
• the serialised png export folder now catches when a manually inputted export path's directory does not exist
• reduced metadata update lag of pages with very large media collection groups
• the inaccurate 'add tags based on filename' button is now called 'import with tags'
• fixed a database UNIQUE issue when two duplicate gui session save calls happen within one second
• the server's lock_off command now works with the Hydrus-Key header auth (rather than hanging indefinitely wew)
• the server now caches hashed access keys in the session manager, in memory, to avoid a db hit on access-key based reauthentication, and in instances where this authentication requires a db hit, now cleanly provides an appropriate 'serverbusy' error
• improved some media object memory management and speedy cleanup
• improved boot fail graceful exit
• removed a bunch of defunct flash (swf) hacks from media viewer code
• bunch of misc non-qt cleanup as I went through the code
• fixed a bug with rendering network credentials for human display
• cleared out the ancient tag archive sync advanced help and added a stub for the new tag migration window
• various help updates around wx->Qt

• version 372

• petitions processing page:
• the selection taglist now displays the raw 'storage' tag view, before siblings are applied
• added a noneable spinctrl to control how many files are shown on a petition row double-click. it samples randomly and defaults to 256
• I think I fixed the issue where the petitions taglist sometimes hangs on to some old tags after a petition process event
• .
• the rest:
• you can now customise the animation scanbar height and nub width under options->media
• all users now see the number of open pages in the pages menu
• added approx total session 'weight' to the pages menu. this is an early test and will do more and update more frequently in future
• added add/remove tag to favourites to the taglist right-click menu
• collapsed the taglist right-click menu a little, as it was getting a bit tall
• added https://gitgud.io/koto/hydrus-archive-delete, a web browser archive/delete filter, to client api help
• added clipboard import/export buttons to the edit tag filter panel for the new favourites
• added 'open in a new page' to media viewer right-click menus, just to put the current single media in a new page
• fixed the url class edit panel not initialising with the new referral options correctly
• the call that publishes new subscription/import folder media to pages now does so more politely to the gui when the db is busy
• subscriptions will no longer start if global network traffice is paused
• the 'hard-replace siblings/parents' action under manage tags is now a local-only operation for tag repositories. clients with unusual sibling and parent application will no longer affect the repos they sync with
• on program shutdown, if a daemon takes more than thirty seconds to shutdown (which can happen in odd situations, like if a subscription run is paused by global network traffic pause, leading to shutdown deadlock), the client will stop waiting and continue with other shutdown tasks
• fixed an error with client 'already running' fast exit
• fixed a different error with server 'already running' exit choice
• updated some ffmpeg calls to fix certain OS problems
• fixed a help link to todolist recommendation and added link to new ptr guidelines

• version 371

• the edit tag filter panel now has load/save/delete buttons at the top to manage tag filter favourites. it starts with a handful of examples
• sorting thumbnails by num tags or namespaces now uses the 'single' tag display context
• the 'sort by media views/viewtime' sorts now do not put the other (viewtime/views) as an implicit secondary sort, so as to better let the user's secondary sort be used
• highlighting a downloader should now not be able to create a page with duplicate thumbnails
• all thumbnail pages now do an addition de-dupe check when they are created with media
• when a gallery page parser now adds new urls to a file import list, urls that are invalid will now be skipped (previously, they threw an error and failed the parse
• fixed a bug where if a default collect is set, pages without a collect (e.g. download pages) would nonetheless initialise with collected+sorted initial media on session load
• file imports now publish the same 'refresh existing media metadata' call as the file maintenance system, meaning if the import already exists in the gui session as an 'unknown thumb', it should now refresh itself correctly
• if the media canvas is called to display an invalid media (due to mime mixup or a faulty parse that slips through), it should now better recognise that and skip/dump out
• fixed import of videos that have 'Duration:' in their title metadata
• improved the error reporting when the old options object fails to save
• removed some old ratings dialog position options storage that was causing errors on certain ratings dialog ok events
• url classes now support options regarding the 'referer' http header they send (their referral url). you can set an optional converter to generate a referral url based on the url class's url and choose to always use the given referrer if available, never use a referrer, use the converter if no referrer is available, or always use the converter
• the network report mode now reports on referral urls used in requests
• the 'quoted' referral url (a unicode workaround) is now only applied if the referral url cannot be encoded to latin-1
• the janitorial petitions processing page now lets you copy tags and left/right tags of pairs with a right-click on selected checkbox rows
• cleaned a little server code
• improved how the server sets and releases its 'currently busy' mode
• the server no longer does <5min vacuums in a backup command
• added a specific 'vacuum' server POST command that forces a full vacuum
• added 'lock_on' and 'lock_off' server POST commands to lock the server and shut down the db, and restart
• the new vacuum, lock_on, lock_off, and a 'is server busy?' check commands are added to the services->admin menu
• added 'pause and disconnect' ability to the database mainloop
• added some unit tests for url classes and the new referral url conversions and server commands
• cleaned some of the thumbnail banner/icon drawing code
• some misc label fixes

• version 370

• tag display updates:
• the old tag censorship system is now replaced by a broader tag display manager that will deal with tag storage and presentation settings. this is the first step, and some of it is not yet completely functional or as efficient as intended
• management for the new tag display manager can be found under services->manage tag display. you can set per-service and all-services filters for 'multiple' display like the 'tag selection' boxes and for 'single' file views, like thumbnails and the media viewer
• existing censorship rules will be added to the new manager and will apply to 'selection list' and 'single media' display rules
• censorship/display rules no longer apply to underlying storage views, which are unfilterable for now, so the manage tags dialog and the autocomplete lookup will now show all tags
• page and media viewer taglists now have new right-click menu options for hiding--they will provide hide options for the specific tag clicked, and its namespace more broadly, and will apply immediately to that kind of taglist (previously, this was just the tag, and launched the tag censorship edit panel)
• all 'tag manager' objects behind every media object in the client now pre-compute cache layers for different tag presentation contexts. operations such as sibling collapse are now only done on file load or new siblings
• for now, initial media load will take slightly longer, but various tag display updates and autocomplete tag fetches on media will be faster
• changes to siblings and the new tag display rules will now trigger a reliable (although, for siblings, delayed by a few seconds) and complete tag list and thumbnail and media viewer refresh
• changes to tag presentation will now correctly update collection thumbnails
• some complicated sibling display and counts are now more precise
• cleaned up some tag/siblings/thumb refresh notification code
• cleaned up all tag manager access code
• cleaned up a variety of related tag fetching, counting, and display code
• .
• the rest:
• added 'system:modified date'. it works just like system:time imported
• files with duration but no audio will now have a ' ▶ ' label in the top-left of their thumbnails, like the 'has audio' one. you can edit this label under options->audio and duration. I don't really like how this looks, so maybe we'll go to icons. let me know what you think
• fixed an iteration timing bug in the new asynchronous repository mappings processing that meant large lists of mappings within an update object may be occasionally truncated, leaving some mappings unprocessed. this would more affect users on slower machines running 'process now'
• any tag repository content updates issued in the last eight weeks will be scheduled for reprocessing to cover the above issue and fill in gaps for most user situations. since the vast majority of the data was added as intended, they should catch up very fast
• added two pixiv url classes for their new url format
• the edit subs panel now recommends users break up subs with >200k urls
• the 'separate' subscriptions button now has a new 'break in half' option. subscriptions that have more than 100 queries will auto-choose this to separate
• the 'quality info' button for advanced users' edit subscription panels now gives the option to additionally copy the info to your clipboard in CSV format
• the lists on the gallery downloader, thread watcher, subscription, and subscriptions panels now sort their progress column by ( y, x ) (given a total status of x/y). previously, this was preceded by a percent-done sort
• the hydrus network engine now recognises 429 bandwidth responses
• on 429 or 509 bandwidth responses, network jobs will now go through the regular reconnection delay loop and try again later (previously, they just failed)
• added 'tag migration' to 'services' menu for quick-launch
• tag migration's 'go' action now skips the second confirmation if you are in advanced mode
• expanded the 'reset' review services button (only visible to advanced users) to allow 'softer' resets that simply reprocess definition/content without deletion (a 'filling in the gaps' command)
• fixed the 'process now' review services button disable check, which was being overzealous
• cleaned up some of the new repo 'caught up' checking code
• improved stability of review services 'refresh account' call
• the client api /manage_pages/get_page_info call now returns a list of hash_ids beside the list of hashes, in simple or not simple mode
• fixed a bug where tag import options that still had a secret deleted service reference were causing tag-parse errors on import jobs
• fixed some other places that were not handling service disappearance neatly
• added a note to the install/backup help to mention not to use continuous cloud-sync backups on your live db directory
• misc unit test refactoring

• version 369

• file maintenance:
• the file maintenance manager now works continuously in the background, optionally in both idle and active time, with two different throttles, which are now always active
• as usual, the default throttles are low-load (1 heavy job every 2 (idle) or 20 (normal) seconds), so as not to interfere with your browsing or other programs--feel free to speed them up as you wish
• the options for file maintenance under 'maintenance and processing' are updated, and quick-pause actions are now available under database->maintain->file maintenance
• the file maintenance manager no longer works on shutdown
• the file maintenance manager will now only make a popup if it is started by the user--it otherwise now works silently in the background
• the file maintenance manager now weights its jobs, so quick jobs will run faster and heavy jobs will space out more. exact weights, if you are interested, are now under the 'see description' button on the maintenance panel
• file maintenance jobs now report to the debug file report mode
• improved some misc file maintenance code, particularly with how the panel talks to the manager
• media with new metadata will now refresh their thumbnails (for now, this means updating the has_audio icon)
• .
• modified timestamps:
• the client now records file modified timestamps of all file imports!
• on update, the retroactive population of this data for all existing local files will be scheduled on the file maintenance system, which has a new job type for this
• the modified time now appears on a file's information lines that present on a right-click
• the modified time can be sorted with the new 'file: modified time' sort
• .
• the rest:
• added lexicographic sort by subtag (ignoring namespace) to the normal taglist sort selection
• reworded the sort by lexicogrphic (grouped by namespace), to be (group unnamespaced)
• the export files panel now has an explicit button to change the neighbouring .txt file tag services
• on duplicate merge action options panel, 'sync archive' is no longer disabled for advanced users' 'alternates' duplicate action
• split the download and process sync components of repositories a little
• added a 'download now' button to repositories' review services panels, to hurry up metadata/update download when possible
• the 'process now' button's enable/disable states should now be more reliable
• the 'refresh account' button now disables when a repository is paused
• improved stability of 'process now' button post-job updating
• added a subscription option to the downloading option panel to change how many file-fails in a run will cause a sync to stop working early
• re-added the truncated image loading mode to the debug->data actions menu. this has hung indefinitely with some bad files, so it not on by default
• fixed an issue with copying an external local booru url with a upnp port
• fixed an unrecoverable ui hang when a modal popup wants to self-terminate while a child yesno is open
• if on a hydrus request the session key is invalid (due, for instance, to a recent serverside session clearing :^)), the session key cookie will now correctly be cleared clientside so a new one can be generated automatically on the next request
• hydrus services can now take the access key as their credential using the 'Hydrus-Key' header. more options will come here, basically the same as the client api
• network jobs waiting on a login process now continue faster once the login is complete (5s sleep cycle down to 1s)
• perhaps fixed some linux problems with tag migration panel, perhaps not
• caught and silenced a rare unimportant services shutdown error
• updated to opencv 4.1.1 on the linux build
• updated windows ffmpeg to 4.2.1

• version 368

• multiple local tag services:
• you can now add additional local tag services under services->manage services!
• new local tag services will appear in manage tags and tag import options and so on, just like when you add a tag repository
• you can also delete local tag services, but you must have at least one
• the default local tag service created for a new client is now renamed from 'local tags' to 'my tags'. any existing user with their local tag service called 'local tags' will be renamed on update to 'my tags'
• .
• ptr migration:
• the ptr has been successfully migrated to user management! hydrus dev is no longer involved in running or administering it. the old bandwidth limits are removed! it has the same port and access key, but instead of hydrus.no-ip.org, it is now at ptr.hydrus.network
• on update, if you sync with the ptr, you will get a yes/no asking if you want to continue using it at the new location. on yes, it'll update your server's address automatically. on no, it'll leave it as-is and pause it. if you still have a connection to my old read-only file repo, that will be paused
• changed the auto repo setup command to be _help->add the public tag repository_. it points to the new location
• as repo processing and related maintenance is now nicer, and secondarily since bandwidth limits are less a problem for the ptr specifically, the default clientside hydrus bandwidth limit of 64MB/day is lifted to 512MB/day. any users who are still on the old default will be updated
• updated the help regarding the public tag repository, both in general description and the specific setup details
• a copy of the same sanitized and frozen PTR db used to start the new PTR, and convenient tag archives of its content, are now available at https://mega.nz/#F!w7REiS7a!bTKhQvZP48Fpo-zj5MAlhQ
• .
• the rest:
• fixed a small bug related to the new 'caught up' repository mechanic for clients that only just added (or desynced) a repository
• rewrote the tag migration startup job to handle specific 'x files' jobs better--they should now start relatively instantly, no matter the size of the tag service
• on 'all known files' tag migrations, a startup optimisation will now be applied if the tag service is huge
• fixed the tag filter's advanced panel's 'add' buttons, which were not hooked up correctly
• the internal backup job now leaves a non-auto-removing 'backup complete!' message when finished
• on update, server hydrus repositories will collapse all their existing content timestamps to a single value per update. also, all future content uploads will collapse similarly, meaning all update content has the same timestamp. this adds a further layer of anonymity and is a mid-step towards future serverside db compaction (I think I can ultimately reduce server.mappings.db filesize by ~33%). if you have a tag repo with 10M+ mappings, this will take some time
• hydrus servers now generate new cert/key files on boot if they are missing. whenever they generate a new cert/key, they now print a notification to the log
• misc help fixes and updates, and removed some ancient help that referred to old systems
• corrected journalling->journaling typo for the new experimental launch parameter

• version 367

• tag migration:
• added htpa and tag service sources for parents/siblings migration that support filtering for the left and right tag of each pair
• added htpa and tag service destinations for parents/siblings migration
• added unit tests for all parent/siblings migration scenarios
• misc improvements to mappings migration code
• reworded some of the tooltip/tag filter message text to more clearly explain how the filter applies to migrations
• the tag filter edit panel now has a 'test' area where you can put in an example tag to see if it passes or is blocked by the current filter
• .
• the rest:
• fixed an issue with auto-no-ing yes/no dialogs throwing errors on exit. I am sorry for the inconvenience!
• thumbnails now show the 'has audio' string on their thumbnails
• 'sort by file: has audio' added!
• icons drawn on thumbnails are now adjusted to sit inside the border
• added increment/decrement numerical ratings actions for media shortcuts! if a file hit by this action has no rating, it will initialise with 0/1 stars or max stars. please forgive the ugly expanding ui in the shortcuts panel here--I'll rewrite this to layout more dynamically in future
• client repository services now track whether they are 'caught up' to their repos, which for now means processed up until at least two weeks ago, and will prohibit uploading new content until the client is caught up
• repository review services panels will now display the 'caught up' status below the 'processed' progress gauge
• repository review services panels will no longer duplicate 'account' status problems in the 'this client's network use' status line--both lines now refer to service/account functionality separately
• repositories will now put in 'unknown error' when an empty error reason slips through the 'no requests until x time' reporting process
• the new thumbnail and media viewer right-click menus now collapse the selection info lines at the top to just the top line and places all the rest (and in complicated file domain situations, this can be a long list) in a submenu off that line
• the new thumbnail 'remove' submenu has separators after 'selected' and 'all' to reduce misclicks
• reworded a couple of things in the manage shortcuts panel to be more clear
• added wildcard support ('eva*lion') and namespace wildcards (like 'character:*') to the advanced OR text input parsing
• fixed a rare issue with the duplicate filter being unable to go back or retreat from an interstitial confirm/forget/cancel dialog when every pair in the current batch cannot be displayed (such as if at least one of the pair has been physically deleted). the filter now catches this situation, informs the user, and closes itself gracefully
• added two extremely advanced and dangerous launch parameters for database access testing
• couple of misc fixes and cleanup

• version 366

• tag migration:
• wrote a unified mass-migrate pipeline to make moving large amounts of data in and out of the client more powerful and more pleasant like other recent non-interrupting changes
• the advanced content update dialog is now renamed to 'migrate tags', both on the review service panel and the manage tags dialog, and has been completely revamped to reflect the new migration pipeline, which works on a location-agnostic (content type, source, filtering, destination, action) model
• the advanced content update dialog is now a non-modal frame--you can keep using the client while it is open, and it will not prohibit the popup from appearing while it works
• added hta and tag service mapping sources to the new migration pipeline, with hash conversion, content status filtering, file domain filtering, specific hash filtering, and tag filtering
• added mappings clear deletion record action to the normal content update pipeline
• added hta and tag service mapping destinations to the new migration pipeline, with appropriate choosable content action (add for HTAs, add, delete, clear deletion record for local tags, and pend, petition for tag repositories)
• added stub list sources and destinations for testing the new migration pipeline
• wrote comprehensive unit tests to test mappings migrations with hta source and destination, including tests for file domain, hash type conversion, and tag filtering
• wrote comprehensive unit tests to test mappings migrations with local tags or tag repository service source and destination, including tests for file domain, hash type conversion, tag filtering, and content status
• adjusted the tag archives to have an optimise call separate from the commit call, so you can do several big jobs in a row faster, or pre-optimise, or top up a well-optimised db without wasting time re-optimising every commit
• added a Close method to Tag Archives just for a nicer explicit exit
• wrote a HydrusTagPairArchive to store sibling and parent tag pairs
• deleted old tag db migration and archive syncing code, which is no longer used
• deleted some related old hta import/export code
• removed the 'tags' export option from the thumbnail share->export menu
• accessing 'migrate tags' is no longer gated by advanced mode on the manage tags dialog cog menu
• the hydrus database now creates a name.temp.db database while it is running to support long-term temp jobs such as the new migration. this file is otherwise unimportant and is deleted on a clean exit
• .
• thumbnail menu rework:
• shifted the thumbnail menu around to group 'view' vs 'action' commands together more sensibly and bury less frequent commands away from the top list
• the 'remove' menu command is now a submenu with filters for 'selected/not selected/ all/inbox/archive'. it also displays on a right-click with no focused file
• 'delete' now has separator on both sides to reduce accidental clicks
• 'share->open' is now moved up a level and inherits 'open in new page' and 'open externally'
• the 'remote services', 'file relationships' and 'regenerate' submenus are moved to the 'manage' submenu
• made similar changes to the media viewer menus, including grouping the zoom/fullscreen/slideshow commands together and making the zoom commands a submenu
• misc wording and count changes to this menu
• .
• the rest:
• added 'main_gui' shortcuts for 'refresh_all_pages' and 'refresh_page_of_pages'. the second shortcut refreshes all pages under the most immediate page of pages parent
• fixed an issue where under-construction OR predicates were not displaying with the system predicate list
• the 'import local files' frame, which pops up when you drop some paths on the client, is now on the new panel system and sizes and cleans itself up more sensibly as a result
• refactored the remaining 100-odd yesno dialogs to the one that works on the new panel system
• misc yesno dialog message and logic cleanup
• fixed an issue where the export files dialog could hang indefinitely if the filename phrase involved long shared tags that resulted in duplicate paths for the first 245 characters in length. now, the long-filename truncation is done before the de-duping ' (n)' text is appended, and the length limit is reduced to 240
• removed the now-defunct 'regen similar files metadata' command under the database menu--this is now handled in the new file maintenance processing system
• improved database optimisation code to better check if empty/small tables have suddenly grown and improved quality of optimisation data for frequently emptying tables
• fixed the 'clear orphan file records' maintenance task, which was not performing the final clear on the correct domain
• fixed some bad error presentation
• fixed the duplicate page not refreshing maintenance numbers after maintenance job completion
• cleared out a bunch of old py2to3 safety code, maybe sped up some sibling/parent/mappings stuff
• misc ui cleanup and small fixes

• version 365

• new repo processing:
• improved the new asynchronous repository processing system in several ways:
• - it now uses the time it is allotted more accurately. when it has 0.45s to work, it hits this mark more often, especially on slower machines
• - it is now politer to the ui if plenty of other work is going on--if the db is loading search results or you are viewing video, it should pause for milliseconds as needed
• - it can now work up to 90% of the time during a manual 'process now' run
• - when it is working faster than expected, it accelerates its load to operate more efficiently
• as a result, the new system should now have faster rows/s and lag out the ui less
• .
• client api:
• improved how parameters are fetched and tested against expected type and given default values if appropriate, and updated all client api code to use this new system
• added /manage_pages/get_page_info, which gives simple or detailed info on a given page, found by page_key
• added page info response to hdd importers, simple downloaders, url downloaders, gallery downloaders, watcher downloaders--they say their pause status and file/gallery import info as appropriate
• added page info response to file import caches--they say their status and num_done/num_to_do, and in detailed mode report file import item info, which is url/path, created, modified, and source times, current status, and any note
• added page info response to gallery logs--they say their status and num_done/num_to_do, and in detailed mode report gallery log info, which is url, created and modified times, current status, and any note
• added page info response to thumbnail panes--they say their total num files, and in detailed mode list their ordered hashes
• started some help for this expansion, but it will need some feedback and more work to finish
• the client api now sorts /get-files/search_files results by import time, newest to oldest. this first hardcoded sort comes to help implement booru-like pagination, but will be expanded to support more types as I flesh out the ui side (as below) as well
• hydrus services, including the client, should now be able to handle larger request header+path total size (16KB->1MB). this helps some larger GET queries in the client api. let's see how this goes
• client api is now version 11
• .
• collect improvements:
• the collect data attached to pages is updated to its own object. the default value and existing page settings should update. all ui now handles the new clean object, rather than the old messy list
• the new collect object supports an option for whether to collect 'unmatched' thumbs or to leave them separate. this displays in the ui as a dropdown beside the collect-by checkboxlist
• to better distinguish between unmatched singles and matched collections with just one item, all one-item collections will now act as collections, with the little '1' collection icon in their corner (previously, they were split into singles). if this is annoying, I will add another option to control whether this occurs
• removed some old 'integrate media into existing collected structure code' that was complicated, never used, and now broken
• misc sort/collect refactoring
• deleted some old unused collection code
• .
• the rest:
• entering tags in the filename tagging panel, either for all or just selected, now pushes those tags to the 'recent tags' list in later manage tags dialogs
• added a framework to start sorting search results before the system:limit is applied--I will soon extend this to start catching the current ui sort (say, 'largest files first', and cut a system:limit appropriately, rather than the current random sample)
• added a faster table size check on the analyze maintenance call that will recognise fast-growing tables (e.g. initially empty/tiny repository processing tables that may have seen a ton of recent work) and schedule them better (this was previously potentially hanging certain maintenance checks/shutdown by several minutes when hitting a surprisingly giant table)
• reduced the analyze frequency for established tables
• the client will now explicitly count as 'idle' and 'very idle' during shutdown maintenance time, in case any shutdown job is considering that for how greedy it should be with work time
• fixed an issue where appending new media (thumbnails) to a page that already had that media but within a collection could create a duplicate media entry and invalidate some internal data links to the old media
• subscriptions will no longer print full traceback information when a network error causes a sync fail
• updated to yet another deviant art file page parser. title tags and embedded image links should be fixed again, post/source time is not fixed
• the deviant current art login script is confirmed to work for some users. my guess is certain people are getting cloudflare blocked or aren't being shown the new login page all the time yet, please send in any more info you discover
• the client will now recover from a missing options object by putting in a fresh one with default values, including a popup notifying you of the error and giving you a chance to bail out
• added a warning and link to the quicksync to the access_keys help page
• if the os commands the client to close due to a log off or system shut down, the client will kindly ask for a bit more time do to so if it is available
• updated the WTFPL license to v3

• version 364

• repo processing makeover:
• repository processing is now no longer a monolithic atomic database job! it now loads update files at a 'higher' level and streams packets of work to the database without occupying it continuously! hence, repository processing no longer creates a 'modal' popup that blocks the client--you can keep browsing while it works, and it won't hang up the client!
• this new system runs on some different timings. in this first version, it will have lower rows/s in some situations and higher in others. please send me feedback if your processing is running significantly slower than before and I will tweak how this new routine decides to work and take breaks
• multiple repos can now sync at once, ha ha
• shutdown repository processing now states the name of the service being processed and x/y update process in the exit splash screen
• the process that runs after repository processing that re-syncs all the open thumbnails' tags now works regardless of the number of thumbnails open and works asynchronously, streaming new tag managers in a way that will not block the main thread
• 'process now' button on review services is now available to all users and has a reworded warning text
• the 1 hour limit on a repo processing job is now gone
• pre-processing disk cache population is tentatively gone--let's see how it goes
• the 10s db transaction time is raised to 30s. this speed some things up, including the new repo processing, but if a crash occurs, hydrus may now lose up to 30s of changes before the crash
• .
• the rest:
• users in advanced mode now have a 'OR' button on their serch autocomplete input dropdown panels. this button opens a new panel that plugs into prkc's neat raw-text -> CNF parser, which allows you to enter raw-text searches such as '( blue eyes and blonde hair ) or ( green eyes and red hair )' into hydrus
• fixed the silent audio track detection code, which was handling a data type incorrectly
• improved the silent audio track detection code to handle another type of silence, thank you to the users who submitted examples--please send more false positives if you find them
• fixed an issue where thumbnails that underwent a file metadata regeneration were not appearing to receive content updates (such as archive, or new tags/ratings) until a subsequent reload showed they had happened silently. this is a long-time bug, but the big whack of files added to the files maintenance system last week revealed it
• the 'pause ui update cycles while main gui is minimised' change from last week now works on a per-frame basis. if the main gui is minimised, media viewers that are up will still run videos and so on, and vice versa
• a few more ui events (e.g. statusbar & menubar updates) no longer occur while the client is minimised
• duplicate processing pages will now only initialise and refresh their maintenance and dupe count numbers while they are the current page. this should speed up session load for heavy users and those with multiple duplicate pages open
• gave the new autocomplete 'should broadcast the current text' tests another pass--it should be more reliable now broadcasting 'blue eyes' in the up-to-200ms window where the stub/full results for, say, 'blue ey' are still in
• fixed an accidental logical error that meant 'character:'-style autocomplete queries could do a search and give some odd results, rather than just 'character:*anything*'. a similar check is added to the 'write' autocomplete
• fixed an issue with autocomplete not clearing its list properly, defaulting back to the last cached results, when it wants to fetch system preds but cannot due to a busy db
• fixed GET-argument gallery searches for search texts that include '&', '=', '/', or '?' (think 'panty_&_stocking_with_garterbelt')
• removed the pixiv login script from the defaults--apparently they have added a captcha, so using Hydrus Companion with the Client API is now your best bet
• the client's petition processing page will now prefer to fetch the same petition type as the last completed job, rather than always going for the top type with non-zero count
• the client's petition processing page now has options to sort parent or sibling petitions by the left side or right--and it preserves check status!
• the client's petition processing page now sorts tags by namespace first, then subtag
• the client now starts, restarts, and stops port-hosted services using the same new technique as the server, increasing reliability and waiting more correctly for previous services to stop and so on
• the client now explicitly commands its services to shut down on application close. a rare issue could sometimes leave the process alive because of a client api still hanging on to an old connection and having trouble with the shut-down db
• the file maintenance manager will no longer spam to log during shutdown maintenance
• sketched out first skeleton of the new unified global maintenance manager
• improved some post-boot-error shutdown handling that was also doing tiny late errors on server 'stop' command
• added endchan bunker links to contact pages and github readme
• updated to ffmpeg 4.2 on windows

• version 363

• has audio:
• wrote a detection routine that can determine if a video has audio. it reads actual audio data and should be able to detect videos with a 'fake' silent audio track and consider them as not having audio
• extended the client database, file import pipeline, and file metadata object to track the new has_audio value
• flash files and audio files (like mp3) are considered to always have audio
• all 'maybe' audio files (atm this means video) are queued up for a file metadata reparse in the files maintenance manager. your existing videos will start off as not having audio, but once they are rescanned, they will get it. this is one of the first big jobs of the new maintenance system, and I expect it will need some different throttling rules to finish this job in reasonable time--by default it does 100 files a day, but if you have 50,000 videos, that's a long time!
• files now show if they have audio in their info string that appears on thumbnail right-click or the top of the media viewer. it defaults to a unicode character, but can be edited under the new 'sound' options page
• added a system:has audio predicate to search for files with/without audio
• updated file import unit tests to check 'has audio' parsing, and added tests for system:has audio
• .
• client api:
• the /get_files/file_metadata call now provides has_audio info
• the /get_files/file_metadata call now provides known_urls!
• added 'cookie management' permission
• added /manage_cookies/get_cookies to get current cookies by domain
• added /manage_cookies/set_cookies to set or clear current cookies
• added/updated unit tests for the above
• updated help for the above
• client api version is now 10
• .
• the rest:
• system:hash and system:similar to now accept multiple hashes! so, if you have 100 md5s, you can now search for them all at once
• the thumbnail right-click->file relationships->find similar files now works for multiple selections!
• when system:hash was just one hash, it would run before anything else and complete a search immediately on finding a match, but now it works like any other predicate, checking for file domain and ANDing with other predicates in the search
• the 'complete' file maintenance regen job now only does file metadata, not a complete thumb regen. its name and labels are updated to reflect this, and any existing job in the system will get the separate thumb regen job
• the file maintenance manager now has a couple of how-to sentences at the top, and a new 'see description' button will talk more about each job type
• the login script testing system now uses a duplicate of the existing domain manager (rather than a fresh empty one), so it will inherit current http headers such as default User-Agent, the lacking of which was messing up some tests
• fixed the login script testing system not showing downloaded data
• subscriptions with multiple queries now publish the files they have imported as soon as each query has finished, rather than waiting for the whole sub to be done
• subscriptions now publish the files they have imported to page/popup even if they have an error
• added 9:16, 2:3, and 4:5 to the duplicate comparison statement system, for various vertical social media types
• the autocomplete tag search 'read', which appears on places like search pages, should now more reliably accept the current entered text when there are no search results yet to show
• the autocomplete tag search 'write', which appears on places like the manage tags dialog, should now correctly accept the input (including appropriate sibling-collapse) when you select a 'stub' result while other results are still loading, rather than broadcasting the exact current text
• fixed the deviant art file page parser to get source time--however the login script may now be broken/unreliable
• fixed a missing dialog import when deleting a string transformation
• reduced the base network connection error reattempt time to 10s (from 60s). there will be more work here in future
• network jobs that are waiting on a connection error now have a reattempt wait override option in their cog icon menus
• the post-bad-shutdown 'open your default session or a blank page' dialog will now auto-choose to open your default session in 15 seconds
• a variety of ui-update events will now not fire as long as the main gui is minimised. as well as saving a sliver of resources, I believe this may fix an issue where long-running subscriptions and other import pipelines could sometimes put the ui in an unrecoverable state due to too many thumb-fade etc... events when the currently focused page was receiving new files while the main gui was minimised
• maybe fixed a rare problem with deleting old pages
• cleaned some misc code

• version 362

• duplicates work finished:
• updated the duplicates help text and screenshots to reflect the new system
• duplicate files search tree rebalancing is now done automatically on the normal idle maintenance routine, and its over-technical UI is removed from the duplicates page
• the duplicate filter's resolution comparison statement now specifies 480p, 720p, 1080p, and 4k resolutions and highlights resolutions with odd (i.e. non-even) numbers
• if the files are of different resolution, a new 'ratio' comparison statement will now show if either have a nice ratio, with current list 1:1, 4:3, 5:4, 16:9, 21:9, 2.35:1
• added a 'stop filtering' button to the duplicate hover frame
• made the ill-fitting 'X' button on top hover frame a stop button and cleaned up some misc related ui layout
• added a 'remove this file's potential pairs' command to the thumbnail file relationships menu
• if in advanced mode, multiple thumbnail selection right-click menus' file relationships submenus will now offer mass remove/reset commands for the whole selection. available commands are: 'reset search', 'remove potentials', 'dissolve dupe groups', 'dissolve alt groups', 'remove false positives'
• .
• the rest:
• added link to https://gitgud.io/koto/hydrus-dd/ , a neat neural net tagging library that uses the DeepDanbooru model and has several ways of talking to hydrus, to the client api help
• cleaned up a little of the ipfs file download code, mostly improving error/cancel states
• rewrote some ancient file repository file download code, which ipfs was also using when commanded to download via a remote thumbnail middle-click. this code and its related popup is now cleaner, cancellable, and session-based rather than saving download records to the db (which caused a couple of edge-case annoyances for certain clients). I think it will need a bit more work, but it is much saner than it was previously
• if you do not have the manage tags dialog set to add parents when you add tags, the autocomplete input will no longer expand parents in its results list
• fixed an issue displaying the 'select a downloader' list when two GUGs have the same name
• hitting apply on the manage parsers or url classes dialogs will now automatically do a 'try to link' action as under manage url class links
• fixed (I think!) how the server services start, which was broken for some users in 361. furthermore, errors during initial service creation will now cancel the boot with a nice message, and the 'running ... ctrl+c' message will appear strictly after the services have started ok the first time, and services will shut down completely before the db is asked to stop
• improved how the program recognises shutdowns right after boot errors, which should speed up clean shutdowns after certain bad server starts
• the server will use an existing server.crt and server.key pair if they exist on db creation, and complain nicely if only one is present
• the 'ensure file out of the similar files system' file maintenance job result will now automatically remove from/dissolve the file's duplicate group, if any, and clear out outstanding potential pairs
• a system language path translation error that was occuring in some unusual filesystems when checking for free disk space before big jobs is now handled better
• like repository processing, there is now a 1 hour hard limit on any individual import folder run
• fixed an issue where if a gallery url fetch produced faulty urls, it could sometimes invalidate the whole page with an error rather than just the bad file url items
• subscriptions will now stop a gallery-page-results-urls-add action early if that one page produces 100 previously seen before urls in a row. this _should_ fix the issue users were seeing with pixiv artist subs resyncing with much older urls that had previously been compacted out of the sub's cache
• until we can get better asynch ui feedback for admin-level repository commands (like fetching/setting account types), they now override bandwidth rules and only try the connection once for quicker responses
• misc code cleanup

• version 361

• duplicates:
• the duplicate filter now compares the pixel content of static image pairs of the same resolution--if they have the exact same pixels, a comparison statement is added, and if one file is a png and the other not (i.e. the png is likely a useless clipboard copy), the statement notes this and a strong duplicate score is applied
• added 'system:is/is not best file of its group' to search for file kings
• renamed 'system:num duplicate relationships' to 'system:num file relationships'
• wrapped the two file relationship system predicates into one 'system:file relationships' stub predicate that opens to a dialog with two pred panels
• added a 'add potential pairs' command to the thumbnail right-click file relationships menu, which will force-queue files for the duplicates filter
• the duplicate filter now ensures the two medias' zoom is locked so they have the same width through a transition. furthermore, their current dragged top-left position is pinned in the same location. this ensures files that have slightly different resolution ratios (especially when they are just a couple of pixels off) still remain reasonably comparable when switching back and forth
• reworked and simplified how position/drag delta is handled in the media canvas to support the above
• fixed the 'custom action' button on the duplicate filter, which had no 'delete neither' choice and whose 'forget it' button cancelled the whole custom operation, making it impossible to custom action without deleting something. I have added a 'delete neither' green-text button to the front, as the default action
• mr bones now reports on your potential, duplicate, and alternates numbers
• .
• tag autocomplete:
• greatly sped up tag autocomplete search when fetching from a current media view (i.e. from thumbnails in the search page)--it had some CPU-inefficient testing/counting that mattered at high media/tag counts
• greatly improved cancelability of tag autocomplete search when pulling from a current media view--this was resulting in high lag when typing fast with multi-thousand results
• fixed the gui-level tag matching test to match namespaced search inputs with offset subtags (e.g. 'character:aran' now matches 'character:samus aran'), both for wildcard and specific namespaces
• when typing an explicit wildcard tag search that does not end in a *, you will now be presented with two wildcard options--one with the implicit * suffix, one without
• fixed 'write' tag autocomplete inputs (like in manage tags) being able to search for chunky 'namespace:*' explicit wildcard searches
• .
• the rest:
• fixed the ipfs nocopy path translation control saving rows for client file paths outside of the main install path for non-Windows, where it was forgetting on save
• renamed 'system:size' to 'system:filesize'
• sped up some system:inbox searches
• disabled a PIL 'load truncated images' backup mode, which on the current version can seemingly lead to infinite load hangs
• file report mode now prints info when it deletes/recycles a path, including stack traces
• fixed a long-running and silent 'port already running' bug related to setting services on the server that was stopping successful service-set-restart from the client in many situations. 'port is already running' checks that conflict with other processes will now give an immediate error to the client without saving any changes
• the server now prints to the log as it stops/starts/has started its services
• improved how the server can report certain 500 errors
• the 'critical service tag/file reference' repository processing error has been improved: rather than reset the whole repository, it now pauses the repo and resets processing status for just the repo's 'definition' update files (without deleting any existing entries, so they should ultimately reprocess super fast) and also schedules a complete integrity and metadata check for all updated files
• keyboard interrupts from the console should now trigger a clean exit request for the client
• polite and forced shutdown requests when logging off should now trigger a fast exit (i.e. no yes/no dialog, no shutdown maintenance, but otherwise session saved and so on) for the client. this fast exit is noted in the log
• moved the tag and rating service listctrls in duplicate merge options panel to the new listctrl object
• moved the manage regex favourites listctrl to the new object
• updated a bunch of yes/no dialogs to the new panel system
• deleted some old unused dialog code and related unit tests
• fixed up deletion-and-reimport file location handling for lingering media objects, which were not correctly forgetting combined local file deletion record on the reimport
• improved shutdown error handling during repo processing
• deleted the mishimmie default downloader

• version 360

• tag autocomplete:
• after various tag autocomplete async work, fetch timings get a complete overhaul this week. the intention is for a/c jobs to appear as fast as possible, with good ui feedback, without interrupting ui while they work. feedback on how this works IRL would be appreciated
• there are now just two autocomplete options under options->speed and memory:
• - whether autocomplete results are ever fetched automatically, defaults to true
• - the max number of characters in the input that will cause just exact results vs. full autocomplete results, defaults to 2, can be None
• namespaces are no longer searched from an unnamespaced query ('char' no longer matches 'character:samus aran'). this proved too slow for real use, and remains better available with explicit namespace searches such as 'character:*' or 'char*:*'
• the 'exact results' character limit now also applies to subtags of namespace searches! so, entering 'character:a' will deliver the same short exact match results as just 'a'--no more gigantic lists when you put in a simple namespace
• improved tag results caching to deal with the new non-namespace matching on subtag input
• tag autocomplete dropdowns will now display a non-selectable 'loading results...' label when results take more than 200ms to load.
• tag autocomplete dropdowns will now also display 'static' tags, such as 'namespace:*anything*' for 'read' inputs and the exact entered text and possible siblings/parents for 'write' inputs, during loading. so, typing 'character:' just to get the special 'character:*anything*' predicate is now simple and does not need a whole load wait to enter!
• cleaned up some tag listbox code to handle parent selection and navigation better along with the new label type
• greatly improved autocomplete search logic in the critical text search portion, collapsing it into one cleverer and more easily cancellable query rather than two or three simpler ones with potentially gigantic lists thrown back and forth
• improved speed of autocomplete cancel for certain large lists with many siblings
• .
• file maintenance:
• the new file maintenance ui now shows scheduled jobs in a listctrl, and only shows jobs that have outstanding work. you can clear/do work on multiple selected jobs
• the file manager should now try to guarantee at least 644 permission on file imports (previously, it was only trying to add 600, which lead to problems with nocopy ipfs running on another user etc...)
• added a file maintenance job to check and fix file permissions
• added a file maintenance job to regenerate similar files metadata
• added a file maintenance job to check if a file should be in the similar files system--if it should and isn't, it is queued to get its metadata data regenerated, and if it is and shouldn't be, it is removed
• the previous bulky similar files metadata regen job from the duplicates page is now removed, and any outstanding scheduled regen will be transferred to the new file maintenance manager on update
• .
• client api:
• added POST /manage_pages/focus_page, which makes the given page the current page in the main gui
• added help and unit tests for this new call
• client api is now version 9
• .
• the rest:
• fixed an issue recording media viewtimes when no max viewtime is set
• fixed the new missingdirectory errors not printing the missing path
• fixed an issue with some human-started repository actions waiting silently on bandwidth when it was not intended (e.g. account refresh)
• export folders now raise proper errors and pause themselves if their path is not set, does not exist on the file system, or is not a directory (previously, they silently stopped work without error)
• cleaned up some misc import folder code, and put in additional protections to the delete/move code to ensure folders cannot be so actioned if they somehow end up in the path import queue
• when unpinning a file or directory from ipfs, the clientside service now first checks that the current daemon considers it pinned (previously, this 500 errored when the object was not pinned due to a reinitialised daemon etc...)
• fixed an issue with the new ipfs path translation control, which was forgetting values when the clientside path was outside of the default db structure
• media objects that transition from trashed to physically deleted but remain in view will now correctly be aware of their complete previously-deleted status (rather than being simply remote, as they were before until a client restart)
• improved some of the recent duplicates db update code to pre-optimise the new tables on update (some users were getting slow behaviour due to mis-scheduled analysis maintenance)
• extended the new panel system to deal with custom button panels and moved the duplicate filter 'commit and continue?' dialog to the new panel system
• moved the archive/delete and duplicate filter 'commit and finish' dialog to the new panel system
• wrote a new question panel for the typical yes/no dialog used across the program and started a cleanup job to migrate all 140-odd instances of this over
• fixed an issue where a program instance that quit due to a user deciding to leave an already running instance in place would clear the original instance's 'running' file in its shutdown, meaning subsequent runs would charge ahead and hit 'database is locked' problems on db init!
• wrote a new 'similar files metadata generation report mode' to provide debug info on this cpu/gpu intensive routine
• added 'why use sqlite?' entry to the help faq, with a link to prkc's excellent document about the subject, https://gitgud.io/prkc/hydrus-why-sqlite/blob/master/README.md
• also added prkc's excellent Linux package requirements information to the 'running from source' help page
• fixed some old py 2.7 references in running from source help and an old link in ipfs help
• moved the 'file viewing statistics' menu down on the database menu
• fixed some dialog Escape key event handling
• fixed some ui ancestory testing code
• improved some misc similar files system code

• version 359

• ipfs nocopy:
• wrote a new panel to better show ipfs daemon status and added it to the review and manage ipfs service panels
• added nocopy config review and enable status and buttons to this new panel
• added an EXPERIMENTAL 'use nocopy' checkbox to the ipfs manage services panel
• added accompanying WEWLAD path translation ui to enable nocopy when your hydrus media storage paths are inaccessible to the ipfs daemon for nocopy purposes. a help button explains this more--it currently needs some symlinking, so non-advanced users should stay away
• if everything is set up, ipfs nocopy seems to work! I am not totally happy about the setup required here, so feedback from advanced ipfs-fluent users would be appreciated and we can iterate on this
• improved stability of ipfs daemon/version checking code
• .
• file maintenance:
• wrote some proper file maintenance ui under database->maintain->review scheduled file maintenance!
• for existing work, the new file maintenance ui shows how much work is scheduled for each job type and lets you cancel that work or run it manually
• for new work, the new file maintenance ui lets you queue up work of any type for files you select with the standard tag autocomplete search interface! you can schedule all pngs to be rescanned in case they are truly apngs, or regen thumbs for all files imported before a certain date, or whatever you wish. you can also queue up repository update files
• the file maintenance manager can now deal with repository update files when it does a complete file metadata regen
• the file maintenance manager now takes responsibility for checking file presence and file integrity. the old 'check file integrity' options under database->maintenance, which did all files in one go, are now gone
• file integrity checks will now always export broken files and missing/broken files' known urls to .txt files to your db_dir/missing_and_invalid_files. appropriate popups and log data will be sent as well. also, the known urls will be both exported on a per-file .txt basis and appended to one unified .txt
• if a file now fails to parse on a metadata reparse, it is now automatically checked for file data integrity
• if a repository encounters a missing, invalid, or incorrect filetype update during update processing, it now schedules all updates in the repo to be appropriately rescanned by the file maintenance manager
• if the storage subdirectory directory does not exist on a client file path request or thumbnail-add attempt, a special error will now be raised with instructions to reconnect the location or shut the client down immediately
• cleaned up some ffmpeg mime-detection logspam
• .
• duplicates:
• added several single-file thumbnail right-click dissolve/reset duplicate actions:
• - reset search status
• - remove from duplicate group (if in one and not the king)
• - dissolve duplicate group (if in a group)
• - remove from alternate group (if in one)
• - dissolve alternate group (if in one)
• - clear false-positive relations (if it has some)
• added some new code to deal with dissolution and member extraction at the db level
• when a member is extracted from alternate group, its constituent files are now requeued for potential search
• multi-selection duplicate right-click actions are now available to non-advanced-mode users
• wrote some unit tests for the new dissolve/reset actions
• cleaned up some misc duplicates code
• .
• the rest:
• fixed a recent bug in the file lookup script GET call--I apologise for the mistake
• the main gui page tab menu now lets you sort page tabs by the number of files they have
• deviant art seem to be rolling out a new page format. this week hydrus introduces completely new deviant art downloader objects that, fingers crossed, will update any existing users smoothly and also provide new tag search functionality. users who are still logged in may still be getting the old page format. if this is you, and this update does not work (although I _think_ it should, even so), please try clearing your existing login and logging in again
• new deviant art login script, artist + tag GUGs, gallery url classes, file and gallery parsers
• updated the shimmie file page parser to pull source time and md5
• improved the 'process now' advanced button to only focus on actual specific outstanding processing. previously, it was also checking for new metadata when due, which, when the server was not available, could seemingly idle for a time before actually processing updates due to the new delaying connection retry code
• wrote a new 'file import report mode' mode to help->debug->report modes
• fixed a progress display issue with the janitorial petitions processing page
• improved accuracy of sibling and parent petition counts, and properly capped them at 1000
• mapping petitions are now grouped by namespace, and will come in more manageable chunks
• fixed the server launch-and-init test debug code
• misc string-to-string control improvements to support the new ipfs edit ui
• removed the old 'continual tag archive sync' legacy code from tag services, which has been semi/non-functional for a long time
• cleaned up the annoying separator hanging on the end of certain tag right-click menus
• cleared out the 'Exception ignored in' spam that is often printed after the log closes

• version 358

• duplicates:
• the final large data storage overhaul work of the duplicates work big job is done--potential duplicate information is now stored more sensibly and efficiently. potential pair information is now stored between duplicate file groups, rather than files themselves. when duplicate file groups are merged, or alternate or false positive relationships set, potentials are merged and culled appropriately
• your existing potential data will be updated. the current potential pairs queue size will shrink as duplicate potential relationships are merged
• the duplicate filter now presents file kings as comparison files when possible, increasing pair difference and decision value
• potential pair information is now stored with the 'distance' between the two files as found by the similar-files search system. the duplicate filter will serve files with closer distance first, which increases decision value by front-loading likely duplicates instead of alts. distance values for existing potential pair info is estimated on update, so if you have done search distance 2 or greater and would like to fill in this data accurately to get closer potentials first, you might like to reset your potential duplicates under the cog icon (bear in mind this reset will schedule a decent whack of CPU for your idle maintenance time)
• setting alternate relationship on a pair is now fixed more concretely, ensuring that in various search expansions or resets that the same pair will not come up again. this solves some related problems users have had trying to 'fix' larger alternate groups in place--you may see your alternates compared one last time, but that should be the final go. these fixed relationships are merged as intra-alternate group members merge due to duplicate-setting events
• a variety of potential duplicates code has been streamlined based on the new duplicate group relationship
• improved how a second-best king representative of a group is selected in various file relationship fetching jobs when the true king is not permitted by search domain
• one critical part of the new potential duplicates system is more complicated. if you experience much slower searches or count retrievals IRL, please let me know your details
• expanded duplicates unit tests to test potential counts for all tested situations
• fixed a bug where alternate group merging would not cull now-invalid false-positive potential pairs
• the rest:
• updated the default pixiv parser to work with their new format--thank you to a user for providing this fix
• fixed the issue where mouse scroll events were not being processed by the main viewer canvas when it did not have focus
• file page parsers that produce multiple urls through subsidiary page parsers now correctly pass down associated urls and tags to their child file import items
• updated to wx 4.0.6 on all built platforms--looks like a bunch of bug fixes, so fingers-crossed this improves some stability and jank
• updated the recent server access-key-arg-parsing routine to check access from the header before parsing args, which fixes an issue with testing decompression bomb permission on file POST requests on the file repository. generally improved code here to deal more gracefully with failures
• the repositories now max out at 1000 count when fetching pending petition counts (speeding up access when there are large queues)
• the repositories now fetch petitions much faster when there are large queues
• frames and dialogs will be slightly more aggressive about ensuring their parents now get focus back when they are closed (rather than the top level main gui, which sometimes happens due to window manager weirdness)
• rewrote a bad old legacy method of refocusing the manage tags panel that kicks in when the 'open manage tags' action is processed by the media viewer canvas but the panel is already open
• hitting 'refresh account' on a paused service now gives a better immediate message rather than failing after delay on a confusing 'bad login' error
• improved login errors' text to specify the exact problem raised by the login manager
• fixed a problem in the duplicates page when a status update is called before the initial db status fetch is complete
• the manage tag siblings panel now detects if the pair you wish to add connects to a loop already in the database (which is a rare but possible case). previously it would hang indefinitely! it now cancels the add, communicates the tags in the loop, and recommends you break it manually
• added a link to https://github.com/cravxx/hydrus.js , a node.js module that plugs into the client api, to the help
• a variety of user-started network jobs such as refreshing account and testing a server connection under manage services now only attempt connection once (to fail faster as the user waits)
• the 'test address' job under manage services is now asynchronous and will not hang the ui while it waits for a response
• fixed some unstable thread-to-wx code under the 'test access key' job under manage services
• improved some file handling to ensure open files are closed more promptly in certain circumstances
• fixed some unstable thread-to-wx communication in the ipfs review services panel
• improved the accuracy of the network engine's 'incomplete download' test and bandwidth reporting to work with exact byte counts when available, regardless of content encoding. downloads that provide too few bytes in ways that were previously not caught will be reattempted according to the normal connection reattempt rules. these network fixes may solve some broken jpegs and json some users have seen from unreliable servers
• fixed watcher entries in the watcher page list not reporting their file and check download status as they work (as the gallery downloader does)
• the client api will now deliver cleaner 400 errors when a given url argument is empty or otherwise fails to normalise (previously it was giving 500s)
• misc cleanup

• version 357

• client api:
• the client api can now receive the access key through a GET or POST parameter rather than the header
• the client api now supports GET /session_key, which provides a temporary key that gives the same access as its permanent access key with the Hydrus-Client-API-Session-Key name through header or GET/POST param. it expires after 24 hours of inactivity or if the client is restarted
• the GET /manage_pages/get_pages call now returns the unique 'page_key' identifier that will be useful in future page management when multiple pages share a name
• the POST /add_urls/add_url command now takes 'destination_page_key' to exactly specify which page you would like a URL to end up on. if the page is not found, or it is the incorrect type, the standard page selection/creation rules will apply
• cleaned up some serverside request processing code
• cleaned up some misc client api permission checking code
• updated client unit tests to check the new changes
• updated client api help to reflect the new changes
• cleaned up some GET and POST parameter parsing
• client api version is now 8
• .
• shortcut and hover window fixes:
• moved the canvas shortcut processing code more towards the new shortcut system
• the OS X shortcut-in-media-viewer issue, which was being boshed in a similar way to the main gui last week, should now be fixed
• when the hover windows have focus, they now pass shortcuts up to the canvas parent more reliably
• removed a legacy menu highlight-tracking system that was malfunctioning and generally throwing a slow-memory-leaking wrench in several places, particularly some non-Windows situations
• the 'menubar is open' test code is now only active for Windows. the other platforms have mixed reliability with menubar open/close events
• some related OS X hover-window flickering and hiding-under the main page problems (having problems due to thinking menus were open) are also fixed
• some hover window flicker on certain focus changes due to clicking focus windows should be fixed
• hover windows now try to size themselves a little better on init, which reduces some initial flicker or false-positive single-frame display on some systems
• extended the hover report mode to report some 'ideal' pos/size info as well
• under file->shortcuts, custom shortcuts are now hidden for non-advanced-mode users
• .
• the rest:
• the popup message toaster now always shows its 'dismiss all' summary bar whenever any messages are being displayed. the summary bar now also has a ▼/▲ button to collapse/expand its messages!
• added duplicate comparison score options (under options->duplicates) for the new jpeg quality estimator
• fixed the default duplicate comparison score values, which appeared to be reversed for higher vs much higher--they will be reset to these new defaults on update, so recheck them if you prefer different
• in manage tag siblings and parents, the filename tagging dialog, and some misc options panels, tag autocomplete input controls now have 'paste' buttons to make entering many results much easier
• to reduce update flicker, the downloader and watcher pages do not list seconds in their 'added' column ('12 minutes 24 seconds ago' is now '12 minutes ago')
• improved clipboard access cleanup on in-clipboard errors, which was sometimes leading to error popups or clipboard lockup
• rather than the simple 'yes', the review bandwidth usage dialog now puts the waiting estimate (like '12 minutes 50 seconds') in the 'blocked?' column
• improved external program launch code for non-Windows to remove hydrus-specific LD_LIBRARY_PATH completely when no OS default exists to restore it to. this should fix ffmpeg connection for certain installs
• fixed a rare bug when initial media results of a page failed to load due to a subset of unexpectedly unfetchable file records
• gave the rare 'ui freezup on dialog close' event yet another pass. closing via escape key should now be immune to this
• 'remote' files that were once in the client but since deleted now have the 'trashed' icon and will state so on their right-click info summary lines
• fixed various instances where selection-from-list dialogs were failing to sort their list based on underlying data object uncomparibility. an example of this was when selecting which queries to pull from a separating subscription
• on the edit parser panels, fetching test data successfully via the quick button or the manual URL entry will now set that URL in the example parsing context
• on the edit parser panels, the subsidiary page parser's separation formula now launches with the correct example data (the original data from the parent dialog, rather than the post-separated data) on which to test separation. this should nest correctly for multiple subsidiary page parsers
• to reduce server load spikes, clientside petition processing now approves very large mapping and file petitions (such as a petition to delete one tag from 50k files) as a sequence of smaller chunks

• version 356

• duplicates:
• moved better/worse/same quality duplicates relationships to the new 'king' group-based model. rather than tracking every relationship, duplicates are now stored in groups with a single 'best' file
• as a result, duplicate relationships are now transitive! saying that one king is duplicate to another will merge groups. the 'better' king is the new king, and 'same quality' kings choose one of the kings pseudorandomly. advanced exceptions: saying that a king is better than a basic member of another group or saying that two members are same quality is still valid but will simply 'poach' the non-king member from the other group in order to ensure the wrong king doesn't end up on top in the eventual merge. saying KingA is same quality as MemberB will merge the groups with KingB as the new king (since it is presumably same/better quality to all members of A)
• the thumbnail right-click 'duplicates' entry is now renamed to 'file relationships' and is no longer advanced mode only. the 'find similar files' entry is folded into this
• the thumbnail 'file relationships' menu now shows a simple 'duplicates' count rather than the old messy better/worse/equal. it will show all the members of a duplicates group when clicked. the menu also notes if the focused file is the best file of its group. if it is not, you will get the option to show the best file or make the focused file the best
• as a result, it is now much simpler to view a group of duplicates and overrule a 'best quality' member as needed
• added the 'media' shortcut 'duplicate_media_set_focused_king' to shortcut setting a 'best quality' file
• the system:num duplicate relationships now has the simpler 'duplicates' entry, to search on size of the entire group. searching for kings/not kings will come soon
• due to the new duplicate transitivity rules, potential pairs are now eliminated at a much faster rate!
• setting duplicate relationships will overrule false positive or alternate relationships already in place
• manually setting alternate relationships to more than two thumbnails at once will now set each file as alternate to every other file in the selection, completely eliminating potential pairs within the group. if you try to do this to large groups of files you will get a longer yes/no confirmation message just to make sure you aren't overwriting some potential dupes by accident
• all existing better/worse/same relationships will be converted to the new group storage in this update, with appropriate kings determined. potential pair queue counts will be reduce accordingly, and the temporary alternate/duplicate confusion from the alternates update will be auto-resolved by merging truly duplicate 'alternates' together
• fleshed out the duplicate test code significantly to handle the new dupe groups and their interactions with the recent false positive and alternates changes
• refactored some db test code into separate client/server/duplicates files and cleaned up dupe tests readability
• potential pairs are now the only component of the new system still on the old pairs system. the duplicate filter will still serve up some inefficient (i.e. non-king) comparisons
• .
• the rest:
• fixed the issue where many clipboard-watcher-caught URLs that did not match were producing false-positive 'could not generate new page for that URL' error popups
• the clipboard text-fetcher now tests against incompatible clipboard types (like a screenshot) better, and all instances of text fetching now report errors more gracefully and with more information
• fixed the unusual OS X issue where many shortcuts were not being processed after client boot until the top menubar was opened and closed. a variety of other blocking-while-menubar-is-open issues that were false-positive misfiring are now fixed as well, please let me know if you still have trouble here
• the file menu now has an 'exit and force shutdown maintenance' option to force-run outstanding maintenance jobs
• when shutdown maintenance work is going on, the shutdown splash screen now has a 'stop shutdown maintenance' button!
• cleaned up some file maintenance manager maintenance locking and shutdown cancel logic
• moved all the idle-mode maintenance checks to a new system that explicitly defines idle/shutdown/forced maintenance work and tests those states in a unified manner, checking idle mode and the new splash cancel button status and so on more reliably. a lot of maintenance should cancel out quicker when appropriate
• misc shutdown logic cleanup
• added a 'file maintenance' option to the database->maintenance menu that forces the new file maintenance manager to run its queue. it'll make a little popup as it works, or a note that no work is due
• the 'regenerate' thumbnail menu is also available to all users
• jpeg quality estimates are now available for all users in the duplicate filter. they only display when the two jpegs' quality have different labels
• the jpeg quality estimator now handles some unusual jpegs that load with empty quantization table arrays
• the duplicate filter now handles bad jpeg quality estimations gracefully
• cleaned up some ffmpeg communication code
• the ffmpeg debug text that spawns on a help->about call that fails to discover ffmpeg version information now prints stderr output as well. if you have been hit by this, please give it another go and let me know what you get
• the same ffmpeg 'no response' error on file parse now popups and prints some debug info and returns a better error
• dialogs and windows on the new panel system now support a new pre-close tidying system
• the manage tags dialog and window will now cancel any pending large tag autocomplete queries on close
• regular gui pages now support a new pre-close tidying system
• search pages will now cancel any pending search results loading or tag autocomplete queries on close
• improved reliability of the popup message manager chasing the main gui when it is sent to another screen by a keyboard shortcut (such as shift+win+arrow on Windows). it should work now if the mouse cursor is in either window. please let me know if this causes trouble for virtual display navigation
• the network engine now waits significantly longer--60s--on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster
• I believe I have fixed/improved a situation where media viewer hover windows would sometimes disappear immediately after appearing on some Linux window managers
• improved hover window report mode to state more focus info in case the above is insufficient
• to better link the two requests and consume bandwidth under strict rules more precisely, the override bandwidth rule that kicks in when a file page has a single file is now 3 seconds instead of 30
• updated options->connection page to specify that 'socks4a'/'socks5h' is needed to force remote dns resolution
• sped up tag parents initialisation
• repositories now group tag sibling and parent petitions by the parent/better tag's namespace
• removed some old network 'death time' code that is no longer useful and was interfering with heavy petition processing
• the log now flushes itself to disk every 60s rather than 300s
• misc fixes and cleanup

• version 355

• duplicates:
• retuned the new alternate and false positive setting code to be less aggressive about removing potential pairs. users with alternate groups who updated to 354 may have lost some intra-alternate potential pairs, which I expect to fill back in once the potential pairs system is updated
• the duplicate filter center-right hover window now has a trash button, which is moved to its own row with the cog icon where they will fit on thinner windows better
• users who are in advanced mode now have access to duplicate merge options for 'alternates' again--but be careful with this, it is advanced. if you had merge options for alternates set up before, your old options _should_ return, but make sure to check it
• if you are in advanced mode, the duplicate filter can now show an experimental jpeg quality comparison statement. if this works out, I will add custom scores and improve this otherwise based on feedback , so comments would be appreciated
• .
• client api:
• the client api has a new 'manage pages' permission!
• /manage_pages/get_pages now fetches a structure of the current page session! there's some help for it
• in future, I expect to add a 'select page' command and get/remove/highlight URL(s) for downloader pages, and reveal the unique page identifier for better targeting here and for add_url commands
• client api is now version 7
• .
• the rest:
• fixed a stupid issue related to drawing collection thumbnails that was causing a lot of errors. I apologise for the inconvenience and have added pre-build tests to catch these simple mistakes in future
• added a clipboard url watcher! there are two options now under network->downloaders--one for watcher urls, and one for all other _recognised_ urls (i.e. post and gallery urls, and file urls that have matching url classes). it checks every second, sends urls to pages according to the same rules as a drag and drop event, and will handle newline-separated lists of urls
• when the client is minimised and needs to create a new downloader page because of an 'add url' command through this new clipboard watcher or the client api, it will now queue up the url and its page creation for when the client is next restored (page creation while minimized causes layout bugs). it returns an appropriate result text to the client api in this case
• added four 'show_and_focus_manage_tags_XXX' shortcuts to the 'main_gui' shortcut set that let you select and focus the four possible suggested tags panels on the manage tags dialog. these are slightly special focus events that do some 'first tag selection' work as well and should let you make the whole process keyboard-only
• added a 'focus_media_viewer' shortcut to the 'main_gui' shortcut set that focuses the media viewer from a media viewer's manage tags window
• file viewing statistics gets an update: it now has its own options page where you can set min/max for preview and media viewing times. the global turn on/off control is moved from database menu to this page, as is a new control for enabling it on the duplicate filter (where you will be flicking back and forth and perhaps do not want lots of views recorded). duplicate filter file stats now default to disabled
• added a new command to database->file viewing statistics that will cull your current stats based on the new min/max values to compensate for previous accidental '3day 4hour' preview view that snuck in. a yes/no dialog explains how it works before it goes
• wrote a 'mixin' class to let my buttons automatically note current shortcut mapping information to the bottom of button tooltips
• the duplicate commands and a variety of other media viewer buttons that work in the new shortcut system now report their current shortcuts in their tooltips!
• quick-entering 'namespace:' in a search page's tag autocomplete input will now automatically swap in the special 'namespace:*anything*' predicate
• cleaned up code around setting a custom temp_dir through the "temp_dir" command, and made it test the given path is indeed a directory and writable-to. if not, the program dumps out with an error popup
• the client and server now similarly test that the directory db is a dir and writable-to
• rearranged some critical boot error handling code and made the client's boot error handling throw up some ui
• fixed the client's repairdb function when needing to regen autocomplete caches
• autocomplete regen now publishes status updates to the splash screen
• fixed the client's repairdb to wait correctly for the ui in case the user wants to bail before repair
• the client's repair db now repairs the new local_tags_cache table if missing
• fixed an issue with showing the manage logins panel when the domain had some unusual cookie expiry numbers
• to reduce initial load gui-event crunch (which affects some systems' layout reliability), initial session load now happens after a 0.5s delay
• cleaned up some image load error handling
• when FFMPEG version information cannot be understood in help->about, a popup note appears and debug information is printed to the log
• the advanced content update panel now puts up a 'working...' dialog while it processed a job
• in the client
• all 4XX and 5XX network exceptions' error texts are now prepended with the actual status code
• extended the hover window debug report mode
• improved some misc temp and permission testing code
• improved some misc testing code

• version 354

• duplicates important:
• duplicates 'false positive' and 'alternates' pairs are now stored in a new more efficient structure that is better suited for larger groups of files
• alternate relationships are now implicitly transitive--if A is alternate B and A is alternate C, B is now alternate C
• false positive relationships remain correctly non-transitive, but they are now implicitly shared amongst alternates--if A is alternate B and A is false positive with C, B is now false positive with C. and further, if C alt D, then A and B are implicitly fp D as well!
• your existing false positive and alternates relationships will be migrated on update. alternates will apply first, so in the case of conflicts due to previous non-excellent filtering workflow, formerly invalid false positives (i.e. false positives between now-transitive alternates) will be discarded. invalid potentials will also be cleared out
• attempting to set a 'false positives' or 'alternates' relationship to files that already have a conflicting relation (e.g. setting false positive to two files that already have alternates) now does nothing. in future, this will have graceful failure reporting
• the false positive and alternate transitivity clears out potential dupes at a faster rate than previously, speeding up duplicate filter workflow and reducing redundancy on the human end
• unfortunately, as potential and better/worse/same pairs have yet to be updated, the system may report that a file has the same alternate as same quality partner. this will be automatically corrected in the coming weeks
• when selecting 'view this file's duplicates' from thumbnail right-click, the focus file will now be the first file displayed in the next page
• .
• duplicates boring details:
• setting 'false positive' and 'alternates' status now accounts for the new data storage, and a variety of follow-on assumptions and transitive properties (such as implying other false positive relationships or clearing out potential dupes between two groups of merging alternates) are now dealt with more rigorously (and moreso when I move the true 'duplicate' file relationships over)
• fetching file duplicate status counts, file duplicate status hashes, and searching for system:num_dupes now accounts for the new data storage r.e. false positives and alternates
• new potential dupes are culled when they conflict with the new transitive alternate and false positive relationships
• removed the code that fudges explicit transitive 'false positive' and 'alternate' relationships based on existing same/better/worse pairs when setting new dupe pairs. this temporary gap will be filled back in in the coming weeks (clearing out way more potentials too)
• several specific advanced duplicate actions are now cleared out to make way for future streamlining of the filter workflow:
• removed the 'duplicate_media_set_false_positive' shortcut, which is an action only appropriate when viewing confirmed potentials through the duplicate filter (or after the ' show random pairs' button)
• removed the 'duplicate_media_remove_relationships' shortcut and menu action ('remove x pairs ... from the dupes system'), which will return as multiple more precise and reliable 'dissolve' actions in the coming weeks
• removed the 'duplicate_media_reset_to_potential' shortcut and menu action ('send the x pairs ... to be compared in the duplicates filter') as it was always buggy and lead to bloating of the filter queue. it is likely to return as part of the 'dissolve'-style reset commands as above
• fixed an issue where hitting 'duplicate_media_set_focused_better' shortcut with no focused thumb would throw an error
• started proper unit tests for the duplicates system and filled in the phash search, basic current better/worse, and false positive and alternate components
• various incidences of duplicate 'action options' and similar phrasing are now unified to 'metadata merge options'
• cleaned up 'unknown/potential' phrasing in duplicate pair code and some related duplicate filter code
• cleaned up wording and layout of the thumbnail duplicates menu
• .
• the rest:
• tag blacklists in downloaders' tag import options now apply to the parsed tags both before and after a tag sibling collapse. it uses the combined tag sibling rules, so feedback on how well this works irl would be appreciated
• I believe I fixed the annoying issue where a handful of thumbnails would sometimes inexplicitly not fade in after during thumbgrid scrolling (and typically on first thumb load--this problem was aggravated by scroll/thumb-render speed ratio)
• when to-be-regenerated thumbnails are taken off the thumbnail waterfall queue due to fast scrolling or page switching, they are now queued up in the new file maintenance system for idle-time work!
• the main gui menus will now no longer try to update while they are open! uploading pending tags while lots of new tags are coming in is now much more reliable. let me know if you discover a way to get stuck in this frozen state!
• cleaned up some main gui menu regeneration code, reducing the total number of stub objects created and deleted, particularly when the 'pending' menu refreshes its label frequently while uploading many pending tags. should be a bit more stable for some linux flavours
• the 'fix siblings and parents' button on manage tags is now a menu button with two options--for fixing according to the 'all services combined' siblings and parents or just for the current panel's service. this overrides the 'apply sibs/parents across all services' options. this will be revisited in future when more complicated sibling application rules are added
• the 'hide and anchor mouse' check under 'options->media' is no longer windows-only, if you want to test it, and the previous touchscreen-detecting override (which unhid and unanchored on vigorous movement) is now optional, defaulting to off
• greatly reduced typical and max repository pre-processing disk cache time and reworked stop calculations to ensure some work always gets done
• fixed an issue with 'show some random dupes' thumbnails not hiding on manual trashing, if that option is set. 'show some random dupes' thumbnail panels will now inherit their file service from the current duplicate search domain
• repository processing will now never run for more than an hour at once. this mitigates some edge-case disastrous ui-hanging outcomes and generally gives a chance for hydrus-level jobs like subscriptions and even other programs like defraggers to run even when there is a gigantic backlog of processing to do
• added yet another CORS header to improve Client API CORS compatibility, and fixed an overauthentication problem
• setting a blank string on the new local booru external port override option will now forego the host:port colon in the resultant external url. a tooltip on the control repeats this
• reworded and coloured the pause/play sync button in review services repository panel to be more clear about current paused status
• fixed a problem when closing the gui when the popup message manager is already closed by clever OS-specific means
• misc code cleanup
• updated sqlite on windows to 3.28.0
• updated upnpc exe on windows to 2.1

• version 353

• duplicate filter:
• duplicate action options no longer handle file deletion
• renamed 'not duplicates' across the program to 'not related' or 'false positive'
• 'alternates' and 'not related/false positive' duplicate actions no longer have duplicate action options. no merge content update now occurs on these actions
• the duplicate filter hover panel now splits 'this is better' decisions into two buttons--whether to delete or keep the worse file
• when selecting 'custom action' in the duplicate filter hover panel, it now asks if you would like to delete the current file, the other file, or both
• the 'duplicate_filter_this_is_better' shortcut action will be auto-updated to 'duplicate_filter_this_is_better_and_delete_other'. an alternate 'duplicate_filter_this_is_better_but_keep_both' is now also available
• the 'duplicate_filter_not_dupes' shortcut action will be auto-updated to 'duplicate_filter_false_positive'
• separated the buttons on the duplicate filter hover panel to more carefully split 'yes, files are duplicates' vs other decisions
• in prep for the duplicate db overhaul, refactored all PHash search code and Duplicate management code apart
• misc other prep work for duplicate db overhaul
• .
• file maintenance:
• wrote a new unified manager to handle various long-term file maintenance tasks like regenerating file metadata and thumbnails
• options to govern how this manager can run are now in options->maintenance and processing. you can enable it for idle and shutdown maintenance time and give it a throttle to limit how fast it will work on files, defaulting to 200 per day
• unified the previous db-level attempts at file maintenance to the new system, which supports async job queueing, and moving regen code up to the new manager, out of the db lock
• unified a variety of file and thumbnail regen code to work through the new simpler and saner path
• the right-click->regen thumbnail commands now run through the new manager and no longer need a modal popup. you can keep browsing while they work. they will also not hang the ui as the old system could on big jobs
• when right-click->regenning on more than 50 thumbnails, you now get a dialog asking if you want to do the job now or put it off later
• file maintenance tasks can now run in shutdown time! you will get previews of the jobs with file counts and status progress reports on the shutdown splash
• cleaned up some file extension renaming and dupe-removing code
• in future, I will move the current file integrity check to this new system and have some ui to prompt and set up other big jobs, like fixing various historical misparsing issues
• thumbnail resizing during thumbnail fade that resizes down is now more efficient
• moved the ClientFilesManager to ClientFiles.py
• .
• the rest:
• the 'manage upnp' dialog now moves the duplicated external ip display from the column up to the status text at the top. it fetches the ip after the initial mappings fetch is done. this ip is no longer affected by the external host override option
• cleaned up options->connection page and removed the now defunct external host override option
• the manage services page for the local booru now has optional override for scheme, host, and port for the 'copy external url' function
• fixed an issue with the recent 'collect by' session saving where a restored session that needed a collect was not sorted
• fixed an issue with collections being sorted by approx bitrate
• added a new checkbox to options->sort/collect to set it so the default sort updates every time you choose a new sort anywhere
• fixed an issue with 'remove trashed files from view', which was incorrectly removing on 'all local files' pages
• the 'all local files' file domain, which is frequently confusing to new users, is now no longer an option for new file pages or the autocomplete file domain if the user is not in advanced mode
• the client now searches for versions of urls both with and without a final '/' character when looking up file url import status at the db level and in import lists. system:known_url is unfortunately still an inefficient mess
• improved how the server code deals with some connectionLost errors
• cleaned up and unified some older dialog button code
• fixed a problem in manage tag siblings when petitioning existing pairs and then cancelling when asked for a reason
• fixed a miscount issue when uploading pending tags while many new tags are coming in. progress would sometimes be -754/1,234, ha ha
• db maintenance, repository sync, and file maintenance processing will all now wake on a force idle mode call
• deleted some old code
• misc fixes and cleanup
• some misc gui layout fixes

• version 352

• the client now supports importing .ico files! (.cur should be supported too)
• finally, 'collect by' is saved for sessions! if your default collect by previously included ratings services, it will forget them this one time--please reset it under the options->sort/collect
• fixed the issue where the media viewer's hover windows were hovering over child dialogs (manage tags, ratings, or known urls)
• improved some os x hover window focus handling for the new always-on-top duplicate action window
• the entries on the 'sort by' list on gui pages are now subcategorised better. it should be a bit easier to find what you are looking for
• the 'sort by file: approximate bitrate' sort option now sorts still images as well by filesize / num_pixels
• to reduce confusion, sort by mime and system:mime are now renamed to 'filetype'
• fixed an issue where the 'unclose_page' shortcut was restoring pages in reverse order (unclosing least-recently-closed-first rather than most-recently-closed-first)
• improved rigour of video framerate estimation
• stopped the video metadata parser from opting to manually frame count videos with size >128MB or num_frames estimate >2,400
• fixed the forced manual frame count to deal with frame counts >9999
• the 'ffmpeg not found' error on file import will now put up a popup message once per boot informing you of this problem more broadly and steps to address it
• fixed some underreporting issues with subprocess_report_mode
• fixed an issue with some yes/no dialogs returning 'no' on escape/window_close_button rather than 'cancel', which affected cancelability some db maintenance questions
• fixed an issue where media that fitted the media viewer canvas width or height exactly at 100% zoom would not respond to zoom switch events to restore non-100% zoom to 100%
• when a local server's CORS mode is turned on, Access-Control-Allow-Origin is now correctly added to GET/POST requests with an Origin request header
• improved reliability of some timestamp rendering code, which should help some users who had trouble opening cookies management page after malformed cookie import
• I believe I fixed an issue with 'open externally' on certain custom paths where the external program could spawn without an ui (flash projector did this). please let me know if your 'open externally' calls start making terminal windows everywhere
• fixed a runtime stability issue with the new duplicates page and slow-updating counts that come in after the page has been deleted

• version 351

• wrote a new (always on top!) hover window for the duplicate filter that sits on the middle-right. the duplicate cog button and action buttons are moved to this new window, as are the file comparison statements
• the duplicate file comparison statements now state the relevant actual metadata along with better '>>'-style operators to highlight differences and green/red/blue colouring based on given score. it is now much easier to see and action clearly better files at-a-glance
• improved some hover window focus display calculations to play with the new always-on-top tech
• both the 'show some random dupes' button and finding dupe pairs for the filter should be a bit faster for very large search domains. the basic file search and indexing still has to run, but the second sampling step in both cases will bail out earlier once it has a decent result
• core image handling functions now uniformly use OpenCV (faster, more accurate) by default, falling to PIL/Pillow on errors. image importing in the client and server should be a bit faster, and some unusual image rotations should now be read correctly
• the server now supports OpenCV for image operations, it _should_ also still work with only PIL/Pillow, if you are running from source and cannot get it
• unified all thumbnail generation code and insulated it from suprises due to unexpectedly-sized source files, fixing a potential client-level thumbnail generation looping bug
• gave all image processing a refactor and general cleanup pass, deleted a bunch of old code
• wrote a new 'local tag cache' for the db that will speed up tag definition lookups for all local files. this should speed up a variety of tag and file result fetching, particularly right after client boot. it will take a minute or two on update to generate
• sped up how fast the tag parent structure builds itself
• the review services panel now uses nested notebooks, rather than the old badly coded listbook control. I don't really like how it looks, but the code is now saner
• similar-files metadata generation now discards blank frames more reliably
• subscription popups now report x/y progress in terms of the current job, discarding historical work previously done. 1001/1003 is gone, 1/3 is in
• made the disk cache more conservative on non-pre-processing calls
• cleaned up some file import code, moving responsibility from the file locations manager to the file import object
• updated the ipfs service listctrl to use the new listctrl object. also cleaned up its action code to be more async and stable
• I believe I fixed a rare vector for the 'tryendmodal' dialog bug
• fixed a bug in presenting the available importable downloader objects in the easy drag-and-drop downloader import when the multiple downloaders dropped included objects of the same type and name--duplicate-named objects in this case will now be discarded
• unified url_match/url_class code differences to url class everywhere
• updated some common db list selection code to use new python string formatting
• plenty of misc code cleanup

• version 350

• the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long
• fixed the duplicate pairs filter's 'ghost pair' issue. it was failing, when 'both files' was unchecked, to remove pairs that included one file that was non-local. this accidental inclusion resulted in incorrect non-zero count and filter/random pairs that could not display correctly
• insulated against potential future iterations of this problem (likely that one of the pair was deleted by another process while a filter is ongoing), with a notification and graceful exiting of the duplicate filter while saving progress
• the 'show random duplicates' button now puts the 'base' of the group (to which all the others are potentially matched) as the first thumbnail
• added a new 'advanced file deletion' section to 'files and trash' options page to turn on a new advanced dialog and set custom file deletion reasons
• if this new dialog is turned on, a delete event from thumbnail grid, regular media viewer, or the duplicate filter's manual delete will launch it. it permits you to delete physically (skipping trash) in one step or delete physically without leaving a deletion record (for easier later re-import) and choose one of the deletion reasons in the new options panel or set a one-time custom reason
• export folders now have more run-controls: 'run regularly', 'paused', and 'run now'
• the file menu now has a 'run export folder now' submenu just like for import folders-- it is simple now to set up an export folder that only runs when you tell it to
• updated the on-boot missing file folder recovery process to automatically 'heal' file location mappings when a missing folder is actually in a location that is known (essentially, you can now manually move a bunch of folders from one known location to another while the client is off and it will recover automatically now). error dialogs will appear in this case summarising the problem and proposed fixes with a chance to bail out if you do not want it to happen
• added a new frame type to 'gui' options page called 'regular_center_dialog' for yes/no style dialogs that are better in the center of the parent window
• the custom web browser launch path and file type 'open externally' paths are moved from 'files and trash' to a new 'external programs' options page
• as the superior '--temp_path' program launch parameter now exists for both client and server, I have removed the limited 'BUGFIX: temp folder override' option from the client's 'files and trash' page and use in the actual code. if this option was important to you, please migrate to the --temp_path launch parameter, which covers temp usage more comprehensively
• as the artstation downloader is now non-functional, apparently by a cloudflare issue, the default gug for new users (and anyone with artstation set atm) is now safebooru
• added dolphin file manager add-on link to the client api help
• some misc file metadata fetching cleanup

Changelog 300-349

• version 349

• duplicate filter:
• the duplicate filter page now has a full-on real-deal file search object to narrow down the duplicate filter, potential duplicate count, and 'show some random dupes' search domains! it also has a 'both files match' checkbox that determines if one of both files of the potential pairs should match the search!
• the duplicate filter page has multiple layout changes as a result:
• the main management area is now split into two pages--'preparation', for doing maintenance and discovery work, and 'filtering', for actioning the potential dupe pairs found
• the 'filtering' page will select by default, but if 'preparation' needs work, its name will become 'preparation (needs work)'
• the 'filtering' page now has file search ui and the 'both files' checkbox instead of the file domain button. this search data is saved on a per-page basis
• the two pages' status texts are now updated on separate calls which have been rewritten to be asynchronous (with 'updating...' text while they work). both now have explicit refresh buttons to force them to update
• the additional non-unknown pair counts listed on the filter area, which were irrelevant to filtering and sometimes confusing, are now gone. it only lists the 'unknown' pair number
• the duplicate filter page's help button no longer has the awful 'simple help' entry. the full html help will get a pass in the coming weeks to reflect the new search changes
• the duplicate file db code received significant refactoring and improvement to support searching the potential dupe space while cross-referencing the new file search context (and still falling back to the fast code when the search is just blank/system:everything)
• misc duplicate file db code cleanup and refactoring
• while in advanced mode, you can no longer select 'all known files' file domain for an export folder (and now the duplicate filter page) search context
• making a file delete action in the duplicate filter (by hitting delete key or the button on the top hover window, which both trigger a dialog asking to delete one or both) now auto-skips the current pair
• .
• manage tags:
• the manage tags has a new 'siblings and parents' button that will auto-replace incorrect siblings and auto-add missing parents! it works on multi-file selections as well! it gives you a summary yes/no dialog before it fires
• the manage tags dialog has a little logic cleanup r.e. siblings and parents and their cog auto-apply options. the auto-application now only applies on add/pend actions
• the manage tags dialog has a new cog button option to not trigger 'remove' actions from an autocomplete dropdown or suggested tag input action when the tag already exists
• .
• the rest:
• gave video metadata parsing another pass--it now detects 'hidden' incorrect framerates due to advanced 'skip frame' codec settings and is more accurate at determining frame count and duration, including some fixed offset calculations that was sometimes adding or discounting a few frames
• manual video frame count, when needed, is now faster and produces better error text
• fixed a critical bug in thumbnail regen that was sometimes potentially looping regen on files with unusual rotation exif information
• significant improvements to how the client file manager handles thumbnail identifier information, saving a great deal of time for file import and thumbnail regeneration code of videos
• fixed an issue where regenerated file metadata was not propagating up to the ui level in real time
• cleaned up some thumbnail cache initialisation code
• the 'generate video thumbs this % in' option is moved from the 'media' to 'thumbnails' options page
• to simplify code, and in prep for the idle-maintenance-rewrite of this system, the database->regen->thumbnails call is now removed
• all three fields of text on serialised pngs now wrap, and they pad a little better as well
• added a new option to the 'gui pages' options page to force input text box focus on page changes
• fixed a small type issue with the server's session cookie code and some new library versions

• version 348

• wrote some OR search help to 'getting started with tags' help page
• wrote a new multi-reader, single-writer lock object for the client file manager, along with some unit tests for it.
• updated the file and thumbnail access and regen and maintenance code to use the new lock. various access is now faster when available and overall safer. there is still work to do here
• adjusted file import to be less aggressive about locking, which should reduce some file/thumbnail access lag during heavy imports
• the thumbnail space estimate in the migrate database dialog is now more adaptive to the new more flexible thumbnail size system and specificies better that it is an estimate
• the client api's /add_tags/add_tags call now collapses siblings and expands parents on an add/pend call. this can be turned off by setting the new optional parameter 'add_siblings_and_parents' to false. the help is updated regarding this and the client api version is incremented to 6
• fixed the client api's /add_tags/add_tags call for the 'hashes' parameter, which was failing to parse, and added an accidentally missing unit test to check this in future
• the client local services (the booru and client api) now have two new options under their 'manage services' panel: 'support CORS', which turns on cross-orogin support (which is experimental for now, so defaults to False), and 'logs requests', which controls whether your log will be spammed with request reports (this also defaults to False), which should clear up some 100MB+ log hassle for people using the Hydrus Companion browser add-on
• hydrus services now respond correctly (albeit sparsely) to OPTIONS requests, and if CORS is enabled, to CORS OPTIONS requests. there are unit tests for this that seem to work ok, but I think we'll need to verify it irl
• finished a first version bitmap manager to handle all wx bitmap creation and destruction, including recycling mid-steps
• updated all simple wx bitmap creation and destruction calls across the client to use the new bitmap manager, improving stability and saving some CPU
• fixed some incorrect button alignment flags that were causing problems for clients set to assert check these values
• added a new yiff.party file url class to the defaults that matches a new file attachment format
• updated the 'url' content parser so if a parsed url is in the form 'summary_text url', as some booru source fields sometimes specify, the preceding summary text is removed, cleaning up the resultant url
• silenced an old server connection lost error that was needlessly loud
• silenced the client network engine from additionally log-printing SizeException errors when a downloading file (usually gif) exceeds file import options rules
• improved misc window destruction code
• updated supported mime list in 'getting started with files' help and website index
• misc cleanup

• version 347

• or search:
• under construction OR predicates now present at the top of the regular tag results list, prepended with 'OR: ', and skipping default selection
• this new OR line is enter-able, which will present it as-is, rather than adding new preds
• hitting escape on a 'search' tag input box that is empty but has an under construction or predicate will cancel the or pred
• hitting escape on a 'search' tag input box otherwise should more reliably kill its focus when the dropdown is a float window
• improved OR search efficiency significantly with dynamic OR search triggering based on other search predicates. OR searches including negated '-tag' components should be massively faster when paired with non-OR tag or file search predicates
• I believe I fixed a search issue that would sometimes return insufficient results when OR preds are mixed with certain other combinations of tags
• improved reliability of some thumbnail refresh calls
• cleaned up a bunch of OR handling ui code
• .
• the rest:
• after previous weeks' experiments, wrote new double-layer thumbnail loading system--now too-small thumbs will quckly scale up fuzzily straight to screen, and then in the coming seconds, the nice regenerated full-size thumb will be made and drawn in place as ready. it presents much faster and looks better, but there is some cleanup to do here that I will tackle next week
• all local file trashing events now record a context-appropriate deletion statement such as "Deleted from Media Viewer." this value is recovered in 'deleted' import status 'notes'. You will mostly see 'Unknown deletion reason.', for files deleted before this new system, but it will populate with appropriate info over time
• fixed a search optimisation that was not cross-referencing with file domain, meaning for instance that bare system:rating calls were returning since-deleted files
• upnp management window now uses new listctrl
• cleaned up some old custom page-naming code
• added a 'data' debug call to clear out all cached thumbnails and force an instant ui thumb reload
• fixed the trash bmp misalignment, ha ha
• removed e-hentai login script from the defaults, since this testing script is not appropriate for new users
• dejanked some media viewer video transitions by cleaning up animation bar rendering and smoothing out video buffer initialisation
• cleaned out some surplus subprocess wait calls that were hanging some systems on various 'open externally' calls
• fixed multiple syncing problems with 'synchronise' export folders that produce files with subdirectories. subdirectory structures should now be synced correctly and empty folders deleted
• export folders that collapse multiple file results to the same duplicated name should, after the next run, do less overwriting to this same name
• if an export folder or the regular export dialog makes a file destination path that is above the chosen directory (e.g. if the path starts with ../ or ..\), the export job will error out with an explanation
• big manual file exports _should_ be politer to the ui and cause fewer hangs
• doing page tab drag and drops may have less post-drop ui jank on linux, continued feedback would be appreciated
• moved 'reason' handling for all content updates to its own area, which neatens many content update data handling issues
• fixed petitioning a tag via a shortcut, which had bad reason handling
• fixed an issue with committing pending ipfs items that was overchecking service permissions
• fixed some remaining bad wx code in the unit tests
• misc file status reporting cleanup

• version 346

• or search:
• extended the search predicate object to handle more OR stuff
• extended the tag list to handle list objects that have multiple colours
• extended the new OR search predicate to report multiple text-snippet-and-colour pairs based on sub-predicates
• extended tag search input to handle prototype OR predicate creation--hold shift when you enter the tag, and it'll start an OR chain. shift-enter continues the chain, enter alone completes it
• fleshed out the predicate unit tests to cover more of this
• wrote unit tests for OR search predicates. it seems good!
• improved some search logic to apply system preds better in certain edge cases and spend less CPU on OR-search-only searches
• .
• thumbnails:
• thumbnails will now queue for load in a more intelligent order based on estimated difficulty to regenerate, which will tend to put more thumbs on screen faster
• the decision to regenerate a thumbnail from source is now tempered by how different the current thumbnail is from what is desired--the more similar the two sizes, the more (randomly) likely the client will decide to just use the current (resized) this time. this smooths out change-lag while limiting the number of really fuzzy thumbs you get. feedback on how this works IRL would be appreciated--it uses some voodoo distribution polling to figure it out, which I can definitely tweak
• improved visual quality of thumbnail scale-up optimisations
• fixed an issue where a multipage thumbnail grid would incorrectly recalculate the new virtual height after a thumbnail size change event, leading to a bit of invalid extra scrollspace (with noclip rendering errors) at the bottom
• the thumbnail right-click menu's reparse files entry is now extended to a new 'regenerate' submenu with three options: reparse file and regen thumbs (the old action), force regen thumbs, and regen thumbs if wrong size!
• the new 'regen if thumbs wrong size' action sends how many thumbs needed resize up to the popup window, as well
• moved some old thumbnail regen code responsibility out of the db and into the files manager
• cleaned out some old redundant file/thumbnail code
• cleaned and refactored a bunch of general image handling and resizing code
• .
• the rest:
• fixed some bad serialisation code that was making file search objects set their 'include current tags' value to false/true on interleaving loads. on this update, all 'include current tags' values are blanket reset to true
• fixed an issue that was drawing animation canvases pure white on various media update events
• extended manage urls dialog to support multiple files when launched from a selection of thumbnails. there is a warning in this case, noting that only gallery-style urls are appropriate to be added to multiple files
• manage urls dialog now supports multiple selections, including shift-select, and accepts delete key presses for easy mass deletion
• when you ask the database migration dialog to move some files, it now pops up a confirmation dialog that also asks if you would like to limit the max time for the job as 10, 30, or 60 minutes
• improved file permission setting code across the program to be more sensible for non-Windows
• if you are a non-Windows user and were hit with directory permission problems last week on the thumbnail update--which resulted in the rxx directories not being deleted--the update this week will attempt to do the delete again, this time correcting the now missing execute permission bit. if it finds outstanding rxx directories to delete, it will give a popup beforehand summarising the situation and giving you a chance to bail out
• fixed yet another problem that was stopping client api url requests from finding the correct page by name
• when a client api url request includes fixed tags, these tags should now propagate in all scenarios where the single url produces multiple files
• updated sqlite dll and console for windows
• misc fixes and cleanup

• version 345

• or search:
• set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support
• started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way--we'll work on this as we discover the most common inefficiencies
• .
• thumbnails:
• the client no longer uses both 'master' and 'resized' thumbnails--it uses a single, smarter thumbnail
• only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)
• the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui
• the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens
• if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over
• due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048
• all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed
• added more thumbnail-resizing related popup spam to file report mode
• fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded
• updated database migration help for new concepts and ui
• cleaned up some misc storage code
• .
• the rest:
• fixed a problem in the client api with fetching file identifiers from file_ids
• fleshed out 'help my db is broke.txt' with more specific clone recovery examples
• fixed import support for a variety of single-frame music webms
• fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set
• network report mode now states url classes of urls about to be parsed
• misc small fixes and cleanup

• version 344

• final v1.0 client api polish:
• added optional 'show_destination_page' arg to '/add_urls/add_url', defaulting to False, to control whether an URL-add will select (i.e. jump to) the destination page in the ui. this changes the default behaviour for this command
• simplified the routine that finds or creates a watcher or url import page and fixed a bug in the api that was not creating new pages when destination_page_name was specified
• some misc cleanup
• fixed fetching file_metadata by hashes
• fixed the client api help regarding file_metadata response example tags
• client api version is now 5
• .
• the rest:
• psd support added! because of this format's potential multi-layer complexity, it will not render natively, but width and height are parsed. it is treated as 'application/x-photoshop'. PSB is also recognised and treated as psd
• added a 'open_known_url' shortcut to the 'media' shortcut set that lets you quickly open URLs for files. if there is one recognised known url, it will be launched, and if there are multiple, a list with all known urls will appear to select which one you want
• animation scanbars now show an x/y current timestamp! it includes millisecond timings and even works for variable framerate gifs. whether to show a second/shadow caret for timestamp position on variable frame rate is a new discussion to have
• fixed an issue where animations would sometimes not resume animation for several seconds after a big scanbar drag
• when the thumbnail manager cannot produce a thumbnail due to a storage error (like a missing file), it now only puts up a single, more informative error popup on the first problem. subsequent errors are printed silently to the log. (these errors tend to come in en masse, so this cuts down on spam and error-related ui lag that was making loading a bad session difficult)
• improved error reporting when an upload pending command would fail due to service non-functionality--it should now give a popup with error info imediately, rather than obscured through the login system
• added temp_dir parameter to the client and server that will override which temporary directory the program will use
• cleaned up how no_daemons and no_wal mode are handled internally
• no_wal mode now has to be called from the command parameter, the no_wal file hack in the db directory no longer works
• missing ffmpeg errors now prompt the user to check if it is installed
• searching for numerical ratings should now work for files that were rated when the service had a different number of stars (ratings now searches in 'bands' rather than exact values)
• reduced the min height of the new import files frame's list
• doubled the decompression bomb test to permit files up to ~179 megapixel, we'll see how it goes
• misc cleanup

• version 343

• client api:
• fixed an int/str type mismatch issue with service_names_to_actions_to_tags in /add_tags/add_tags in the client api that meant that argument was not working
• fixed up some last /get_files/search_files stuff
• added /get_files/file_metadata
• added /get_files/file
• added /get_files/thumbnail
• added help and unit tests to reflect the above
• updated client api version to 4
• .
• the rest:
• the list of paths in the manual file import dialog is now sortable. this order will be preserved in regular and 'add tags' ok events for this dialog. it has a new '#' column so you can return to 'parse' order if desired
• animation and static image windows in the media viewer canvas are now recycled through media type transitions, making for slightly smoother browsing between mixed media
• increased aggression of media viewer image prefetch
• added support for 'MM' Tiffs
• fixed webm mime parsing for webms with no audio (these were falling back to mkv)
• improved the error reports when a serialised png fails to import
• the mass-open-urls popup is now pausable as well as cancellable
• fixed several recently broken ui unit tests
• misc old code cleanup
• some misc test controller/constant refactoring

• version 342

• added support for webp import. it does not yet support animated webps, which, if the local platform supports, will import like apngs used to: just the first frame
• added support for tiff import. it works ok for 24bit and 8bit (monochrome) tiffs, but I am not sure how well it will do with 48bit
• both webp and tiff should work on the duplicate files system
• improved webm detection to include opus audio (previously, these files were falling back to mkv)
• fixed an issue where unusual formats with duration but no frames or frames but no duration were being sorted and otherwise presented incorrectly
• improved autocomplete job cancelability. this job can now cancel much faster on large jobs, meaning typing searches with large result sets will hit less CPU and return faster on subsequent keystrokes
• _all_ of the complicated 'copy url' commands from the thumbnail right-click->known urls menu are now available on the 'open' submenu! if there is more than one url to open (e.g. 'open all of these files blahbooru post urls' on a selection of 50 files), you will be presented with a yes/no dialog to confirm, and it will open one url in your browser every second (with a cancellable popup if num_urls > 5)
• by default, system:everything is now hidden if its total files is >10k. you can force it to always show under options->default system predicates
• the gallery downloader's list's status column now shows gallery status (deferring to active file status) when appropriate and shows 'done!' when all work is complete
• after working back and forth with a user, I _believe_ the linux similar files >0 distance search crash is finally fixed
• fixed sorting by media views/viewtiming with collections
• a single-selected collection right-click now shows total media views for all files in the collection! you can now see how long you have been viewing an artist!
• fixed an issue that lead to export folders not running on always-on clients as often as they should
• updated the gelbooru 0.2.5 file page parser to pull rating tag from the correct location (previously, it was pulling from what appears to be a site-wide 'mature' browser hint)
• improved memory cleanup stability when animations and other parts of the video rendering pipeline are deleted--this _may_ fix some rare crashes
• increased animation rendering aggression overall and particularly in 'future' of frame buffer
• if a video renderer that is asked to start some way into the video fails to render anything, it will now fall back to trying to render from the beginning. this is slightly hacky atm and leads to out of phase rendering frames, but it is better than an error
• added a '--no_db_temp_files' launch parameter that will force the client or server to return to the recent old behaviour of exclusively using memory for journalling. this is useful if your temp directory is small and/or your available ram is very large. if running in this mode, the client will attempt to check available memory (instead of free space on your temp dir) before performing very large transactions
• with the new lighter-weight update transactions, the client now tests for less free space for journalling before running repository update processing
• added /get_files/search_files to the client api, which does the first half of file searching. it allows tag search (including -tag negation) and system inbox/archive. since the second half, which will fetch file metadata, is not yet in, this can't do anything interesting yet
• updated help and unit tests to support this, client api version is now 3
• some misc refactoring

• version 341

• client api:
• added /add_tags/add_tags, which does several kinds of tag content updates
• added /add_tags/clean_tags, which shows how hydrus will handle potential tags
• added /add_urls/associate_url, which allows you to associate urls with files
• added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)
• updated client api version to 2
• updated client help and unit tests for the above
• added a linked contents to the client api help
• improved some server error handling, mostly moving 403s to more correct 400s
• improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500
• .
• the rest:
• tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user
• when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)
• added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details
• reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements
• the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows
• when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8
• improved serialisability of some pending tag data
• watchers can now hold and pass on fixed pending tag data
• gallery log objects can now hold and pass on fixed pending tag data
• file import objects can now hold and action fixed pending tag data
• hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects
• improved some url-import page drop-target-selection logic
• improved error reporting when dropping/api-adding urls
• adjusted some url import workflow so big 'already in db' download lists should work a bit faster
• attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made
• tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth
• fixed a source of linux CRITICAL logspam related to several common dialogs
• improved ui stability on boot when file folders are missing (particularly for linux)
• improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better
• did some more misc stability improvements, particularly in various boot fail scenarios
• completely removed an ancient and janky focus catcher widget from the main gui frame
• now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory--hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large
• fixed approx bitrate sorting for malformed video files with explicitly '0' duration
• daemon_profile_mode now spams some more info about export folders
• fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status
• updated windows build to cv 4.0
• misc refactoring and fixes

• version 340

• client api:
• fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly
• fixed the 'add_url' api call's instability
• the API will now always return JSON on 200. anything else should be presumed to be raw text
• '/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week
• '/request_access_permissions' now returns JSON
• '/add_url' now results JSON on success with more info, 403 on failure
• '/get_url_info' now returns the 'normalised_url' in the response JSON
• added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url
• added '/add_files/add_file', which can import a file from a path or bytes
• added '/add_tags/get_tag_services', which will return info on the client's tag services
• updated client api help to reflect the above changes and fleshed out the intro a bit
• fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time
• updated the client api tests to check the above
• refactored client api tests to be neater and in their own file
• .
• the rest:
• fixed the page of pages close bug
• added a downloader for nijie.info to the client defaults (it needs a login)
• updated danbooru file page parsers to get 'rating' tag
• added gelbooru 0.1.11 parser for future application
• fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction
• added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)
• wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest unicode replacement characters
• the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks
• moved some more old daemons to the new job scheduler, deleted some old daemon code
• improved some daemon job wake and shutdown code
• wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code
• simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update
• fixed upnp mapping fetching to cope with ipv6 results
• improved some memory clearing code to deal with some semi-stubborn objects
• improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine
• improved menubar stability, both in finding menus and swapping them out
• if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup
• fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions
• fixed two login invalid cookie error handling bugs
• maybe made some dupe filter searching more stable
• fixed a py2 datatype issue that made the client unbootable when updating the client from <296
• the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go
• slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB
• added experimental secret discord bugfix dnd mode checkbox
• improved how html parsing deals with some unexpected bad tag data
• turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays
• wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work
• improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up--this is an ongoing project
• improved how some text import line splitting works
• misc fixes

• version 339

• client api:
• wrote some ui to handle client api permissions adding and editing
• wrote a 'catch a permissions request' mini-dialog for external api permissions adding
• wrote api calls:
• GET
• /api_version
• /request_new_permissions
• /verify_access_key
• /add_urls/get_url_info
• POST
• /add_urls/add_url
• and made a new 'client api' help page to describe in detail what these do
• wrote fairly comprehensive unit tests for the new client api
• refactored a bunch of 'hydrus network' specific stuff away from general server code that the client api now uses
• neatened up 401 vs 403 error handling across the program, and replaced some clientside error handling that was inelegantly borrowing these network errors
• deleted very old prototype file/thumbnail client server fetch code, which was no longer in use
• .
• misc and bug fixes:
• added a 'clear ratings' button to the ratings service 'review services' panels. it can clear out ratings for deleted, non-local, or _all_ files
• the '3 downloaders are working, is it ok to close the client?' message is now folded into the 'confirm client exit (auto-yes in 15s)' dialog. this merged dialog will still appear for users who have the regular confirm client exit dialog turned off (and still auto-yeses in 15s)
• the file url downloader now reports 'downloading file' and 'importing url' text status separately
• fixed a typo bug from last week that was breaking asc/desc ratings service sorting
• fixed a typo bug from last week that was stopping manage import folders from opening
• fixed a typo bug from last week that was breaking setting upnp port on the local booru/client api service management panel
• the advanced file reparse-and-re-thumbnailing routine now correctly moves a file to its new extension if its mime changes (e.g. png->apng, or webm/mkv distinctions)
• the client file manager now silently detects and auto-repairs instances of missing files where the file actually does exist, just with the wrong extension
• fixed a url parsing issue that was normalise-mangling url classes with no path but some query parameters
• the network engine now uses utf-8 decoding when no specific encoding is set (previously ISO-8859-1)
• fixed an ffmpeg video parsing bug when the video included Shift-JIS metadata. it should work for other unusual encodings as well
• maybe cleaned up some menubar management code
• the filename tagging dialog now uses a notebook for service choice, like the manage tags dialog, rather than the janky old listbook
• fixed a py2-to-3 issue with the admin-only 'is server currently busy' check while a backup is running
• improved some dialog button event handling. it may completely fix the 'trytoendmodal' issue some users run into
• improved some JSON db serialisation error reporting code, trying to pin down an issue several users have seen with session save
• improved thread-safety of serialisable objects as they serialise
• misc improvements and cleanup

• version 338

• after talking with some users, put a bit of time into tag autocomplete wolkflow. both read (search pages) and write (manage tags) autocomplete inputs now operate ~asynchronously~, with the tag fetch working on a separate thread. tag jobs can also now cancel at certain checkpoints in the tag search process if overwritten by a new request. therefore, a variety of tag lookup scenarios _should_ be less painful. this change was executed in a semi-inelegant way, so please report if you encounter bugs from fast typing etc...
• I also improved some of the "I hit enter before results were in" code as a result of this. not sure I have it totally nailed, so please give feedback on errors here
• wildcard search tags now have an explicit '(wildcard search)' after their label
• taglists that have an attached page of media (basically the 'selection tags' box and the 'active search predicates' up top) now have a right-click menu entry to 'select files with (all these tags)' and, if more than one tag is selected, 'select files with (any of these tags)'! This is pretty neat in action, so give it a go!
• added a 'sort by approx bitrate' file sorting option that does a basic filesize/duration so you can filter out dense gifs and other short-but-big vids a bit easier. anything without a duration is shoved to the 'smallest' side
• some of the file sort options now default to their respective 'biggest' first, see how you like it
• folded in updated gelbooru file page parser (fixing the 403 errors that just appeared) and added searches for gelbooru user favourites and pools
• if a gallery or watchable url ends up in the file processing queue (and hence fails), it now says the believed-to-be url class name in the error, which should help some false positive url class matching debugging
• fixed a focused-file selection issue that meant preview viewtime was frequently counting in file viewing statistics even while a page was not currently in view
• fixed the local booru review service panel, which wasn't fully deleting shares when the button was clicked
• wrote a service object and basic server skeleton for the client api (basically refactoring the existing local booru code). client api now appears in manage and review services and can boot and present the normal hydrus browser welcome page at '/'
• the client api and the local booru now have an 'allow non-local connections' checkbox! defaulting to off and on respectively
• updated a wx-thread call function to more safely and universally deal with instances where the responsible window died before the call could be made
• removed some old experimental crypto code that isn't used any more--pycryptodome is no longer needed to run the program from source
• improved some misc client service code
• deleted some old unused code
• misc cleanup
• updated to ffmpeg 4.1 on windows release
• updated to sqlite 3.26 on windows release
• updated to wxpython 4.0.4 on all platforms

• version 337

• fixed another couple of unicode encoding problems with the logging and profiling code
• the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding
• fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video
• fixed some 'max size' download file size testing
• 'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain: somesite.com') they are waiting on (you might need to hover over to see the tooltip for this)
• the downloader easy-importer lain image is now clickable to launch a file selection dialog
• if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you
• duplicate content merging now applies to pending as well as current tags
• the duplicate filter now counts pending tags when saying which file has more tags
• advanced content updates now _copy_ both current and pending tags. the other actions now state what they do more clearly
• stopped printing long server error text on 304 and 404, where actual response content is uninteresting
• removed wx import that accidentally came into server space due to 2to3 check
• improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds
• fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else
• improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements
• did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations
• reduced db disk cache aggression significantly and added more memory maintenance to the cache population process
• fixed an issue where subscriptions were not promptly responding to shutdown events
• fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events
• added a 'pubsub report mode' debug mode for simpler pubsub review
• the db is now less redundantly spammy on certain behind the scenes update notifications
• wrote a first version of the client api manager and permissions handling objects
• misc fixes, cleaned some shutdown code

• version 336

• fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)
• fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases
• fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)
• fixed a str vs bytes issue when loading the filename tagging panel
• fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)
• improved some json dump deserialisation code
• fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel
• improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data
• fixed some hex character processing for system preds
• added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)
• the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation
• made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update
• the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system
• simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method
• wrote some new help.txt in the db dir about hanging startups

• version 335

• important:
• hydrus now runs completely and exclusively on python 3!
• for users who are updating, the client has special install instructions for just this week:
• if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)
• if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)
• if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications!
• if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out
• please check the release post for more detailed instructions here
• .
• semi-important:
• the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from ???? ????? lads appreciated
• I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller)
• although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source
• I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries
• .
• misc bug fixes:
• fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class
• removed some popup flicker related to long error messages
• fixed some parsing testing ui error handling
• cleared up some bad text ctrl event handling that could sometimes cause a recursive loop
• listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion
• cleaned up some bad external process calling code and improved how child process environment is set up
• finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this
• .
• boring details:
• ran 2to3 to auto-convert what could be done
• updated environment to python 3
• went over a whole ton of unicode encoding/decoding manually to update it to python 3
• removed all the old tobytestring/tounicode calls in favour of new python 3 handling
• fixed all the file io to do bytes/str as appropriate
• corrected a bunch of / vs // int/float stuff
• fixed up twisted, which has some str/bytes stuff going on
• fixed all the listctrls to deal with column sorting None values amongst ints/strs
• fixed png export/import, which had some fun bytes/bytearray/int/str issues
• updated the swf header parsing code to py3 (more str/bytes stuff)
• misc float/int fixes
• fixed up some http.cookies handling, which has changed in py3
• improved some ancient loopback connection code that was only really checking to see if ports were in use
• cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked
• numerous misc other refactoring and so on
• updated the new network engine to now decode non-utf-8 responses correctly based on actual response header
• removed some old py2 manual http multipart code
• removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection
• replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess
• deleted some more misc old code
• tightened up some spammy network error reporting
• converted all /r/n to /n in my environment project, ha ha ha
• the ui seems to better support rarer unicode characters like ??
• updated some of the install/update/backup help for all this, and some misc other stuff as well
• fixed misc bugs

• version 334

• wrote a system:file viewing stats to comprehensively search the new viewing stats--it _should_ also be synced with the exact current values
• but for system:everything, inbox, and archive, which remain where they were, system predicates are now sorted alphabetically!
• added a _database->file viewing stats_ menu that lets you suspend file view tracking and clear all records permanently
• mr. bones now welcomes all users under the help menu
• fixed mr. bones's confusion at those who have yet to board the ride
• also mr. bones now makes sure to get the latest file viewing stats
• moved confirm trash/archive from _options->gui_ to _options->files and trash_
• moved a bunch of 'pages' related stuff from _options->gui_ to the new _options->gui pages_
• added an option to options->gui pages to change the number of session rolling backups
• subscription popups now provide an x/y query progress string in their popup text
• the edit subs/sub panels are now a bit shorter by default and the edit sub has its own frame position data, under 'edit_subscription_dialog', and remembers its last size and position by default
• fixed an issue where some dupe watcher urls (like url and url#12345) were not being correctly merged on a mouse drag and drop watcher-import
• the client will now print up to 512KB of server error info to the log (previously 4KB)
• removed the youtube download prototype--if it returns, we'll do a proper youtube-dl solution. as a result, pafy is no longer needed to run the client
• network report mode now shows more network error information
• gave the 'getting started with subscriptions' help page a complete pass. it now reflects the new system and has up-to-date advice based on my new experience
• wrote a 'logins' section to the bottom of the 'getting started with downloading' help page
• misc fixes

• version 333

• added a first version of file viewing statistics! the client db now keeps track of how many times a file is loaded in the preview and full media viewers, and for how long!
• you can see the media and preview stats on any single media right-click menu. there are multiple options for how this displays, including hiding it completely, under options->media
• viewing stats update as they happen! (although viewtime typically only updates on the end of viewing. I'll likely make this more live, especially if I end up showing this info in the media media viewer)
• you can now sort files by total media views/viewtime!
• mr. bones's wild ride continues, as well
• deleted the old 'file list' way of updating in-ui media objects in favour of a long-planned global media cache. there is now only ever one active copy of any particular media, and all data-level updates need only occur once on that single copy. this saves a bunch of CPU, memory, and overall hassle behind the scenes! various search results/lookups for media already loaded elsewhere now load super fast!
• tag siblings refresh is quicker and less memory heavy thanks to this as well
• furthermore, the complicated tag changes from tag repository processing and advanced content updates are now reflected immediately in the gui on the job's completion! (as long as you have fewer than 10k files open, ha ha) previously, these required a search refresh to show the results
• the file sort choice dropdown on all pages is now sorted alphabetically. it has always been a mess picking what you want from here, so let's see if this helps!
• tag and rating sort options are now listed as 'tag:' and 'rating:' respectively
• fixed some misc file sort choice code, which was failing to keep certain defaults in certain situations
• fixed the tag import options' new 'load from defaults' button to correctly load the tag blacklist
• the keyboard icon on the media viewer's top hover window now permits activation of current/default shortcut sets under submenus. it now also omits these entries if no custom shortcut sets exist
• cleaned up some of the hover_window-canvas interaction code
• fixed some long-time sperg-out buffer-drawing when changing position in a long video
• the database->backup actions are now hidden if the current db has non-default file/thumbnail locations. for now, in these cases, only a custom backup is appropriate
• fixed some ancient repository admin code that fetches summary account info given an account key
• the filename tagging dialog now has a much shorter listctrl by default, so should fit better on smaller monitors
• fixed the 'review session cookies' dialog's clear button, which was not deleting sessions after clear. it now also wraps the operation in a yes/no confirmation

• version 332

• the client serialisation system now supports multiple rolling backups!
• client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp!
• when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations
• gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload
• tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch
• under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively
• attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process
• did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system
• wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster
• export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.
• fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work
• when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!
• the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads
• adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well
• fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices
• fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add
• the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it
• wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places
• improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service
• fixed a small typo bug when trying to auto-add url classes and parsers
• updated an old deprecated checkboxlist call
• clarified the integrity section of 'help my db is broke.txt'
• misc fixes

• version 331

• added a 'do login now' button to the manage logins dialog. it only enables when the selected logins are active and not invalid and so on, and will ok the dialog and queue up some login attempts, which will make report popups as normal
• 'review session cookies' panels now support drag and drop cookies.txt import! cookies.txt importing will also handle errors a bit better and report total number of added cookies
• the 'review session cookies' panel now defaults to not showing sessions with zero cookies. a new checkbox controls this
• login scripts can now be rolled into easy import pngs! should work for export and import just like the other objects (although they won't be auto-added based on domain in export dialog)
• brushed up some of the 'change login script' code--particularly, it now puts login scripts that have matching domains first in the selection list, for easier selection
• after striking a reckless bargain with a daemon from the database-plane, system:num_tags now runs significantly faster and produces accurate tag counts even when searching over multiple tag services that have duplicate tags. if this works out, the immaterial beast promises greater gains for similar jobs with no possibility of anything going wrong
• prototyped a new tag cache in the db that affects (and should speed up) many tag fetching routines. let's see how it goes
• added complete, global proxy support for the new network engine! there are new options under options->connection (with some explanation text) to handle it. if pysocks is installed, socks4/5 proxies are also available!
• updated the e-hentai.org login script to the new one on the github. your existing mappings for e-hentai.org _should_ all be updated right. exhentai.org is likely too difficult to properly support in the current system
• the different panels where you enter system predicate information now all run on the new panel sizing system--if you have had problems with these, please let me know how they size now!
• added a '4channel thread' url class to support watchers for the new 4channel sfw domain. it works for now, but let's see if their api changes when the split actually happens
• the list right-click menu on gallery import and thread watcher panels now has three options to show combined importers' files--presented, new, and all. it also now shows the files (more smoothly) in the same page, clearing any existing highlight.
• misc ui improvements
• updated 'running from source' in help

• version 330

• login:
• added a proper username/password login script for hentai foundry--double-check your hf filters are set how you want in your profile, and your hydrus should inherit the same rules
• fixed the gelbooru login script from last week, which typoed safebooru.com instead of .org
• fixed the pixiv login 'link' to correctly say nsfw rather than everything, which wasn't going through last week right
• improved the pixiv file page api parser to veto on 'could not access nsfw due to not logged in' status, although in further testing, this state seems to be rarer than previously/completely gone
• added login scripts from the github for shimmie, sankaku, and e-hentai--thanks to Cuddlebear and any other users who helped put these together
• added safebooru.donmai.us to danbooru login
• improved the deviant art file page parser to get the 'full' embedded image link at higher preference than the standard embed, and only get the 'download' button if it looks like an image (hence, deviant art should stop getting 140MB brush zips!)
• the manage logins panel now says when a login is expected to expire
• the manage logins dialog now has a 'scrub invalidity' button to 'try again' a login that broke due to server error or similar
• entering blank/invalid credentials is now permitted in the manage logins panel, and if entered on an 'active' domain, it will additionally deactivate it automatically
• the manage logins panel is better at figuring out and updating validity after changes
• the 'required cookies' in login scripts and steps now use string match names! hence, dynamically named cookies can now be checked! all existing checks are updated to fixed-string string matches
• improved some cookie lookup code
• improved some login manager script-updating code
• deleted all the old legacy login code
• misc login ui cleanup and fixes
• .
• other:
• sped up tag searches in certain situations (usually huge inbox) by using a different optimisation
• increased the repository mappings processing chunk size from 1k to 50k, which greatly increases processing in certain situations. let's see how it goes for different users--I may revisit the pipeline here to make it more flexible for faster and slower hard drives
• many of the 'select from a list of texts' dialogs--such as when you select a gallery to download from--are now on the new panel system. the list will grow and shrink depending on its length and available screen real estate
• .
• misc:
• extended my new dialog panel code so it can ask a question before an OK happens
• fixed an issue with scanning through videos that have non-integer frame-counts due to previous misparsing
• fixed a issue where file import objects that have been removed from the list but were still lingering on the list ui were not rendering their (invalid) index correctly
• when export folders fail to do their work, the error is now presented in a better way and all export folders are paused
• fixed an issue where the export files dialog could not boot if the most previous export phrase was invalid
• the duplicate filter page now has a button to more easily edit the default merge options
• increased the sibling/parent refresh delay for 1s to 8s
• hydrus repository sync fails due to network login issues or manual network user cancel will now be caught properly and a reasonable delay added
• additional errors on repository sync will cause a reasonable delay on future work but still elevate the error
• converted import folder management ui to the new panel system
• refactored import folder ui code to ClientGUIImport.py
• converted export folder management ui to the new panel system
• refactored export folder ui code to the new ClientGUIExport.py
• refactored manual file export ui code to ClientGUIExport.py
• deleted some very old imageboard dumping management code
• deleted some very old contact management code
• did a little prep work for some 'show background image behind thumbs', including the start of a bitmap manager. I'll give it another go later

• version 329

• login:
• the login manager is fully turned on! hentai-foundry click-through and pixiv login now occur fully on the new system
• wrote a Deviant Art login script for NSFW downloading--however, it only seems to work on a client that has done some logged-out downloading first (otherwise it thinks you are a robot)
• updated the DA file page parser to only NSFW-veto if the user is currently logged out
• wrote a danbooru login script for user prefs and special files if you have a gold account
• wrote a gelbooru 0.2.x login script for user prefs
• pixiv recently(?) allowed non-logged in users to see sfw content, so the login script is updated to reflect this. the login script doesn't detect a failed login any more, so I will revisit this
• logging in in the regular order of things now makes a temporary popup message with the overall login status and final result. ~it is cancellable~--and if cancelled, future login attempts will be delayed
• logging in in the regular order of things now prints simple started/result lines to the log
• deleted old network->login menu and related code such as the custom pixiv login management. gdpr click-through is now under downloaders
• subscription login errors will now specify the given login failure reason
• subscription login tests will now occur at a better time, guaranteeing the sub will be correctly saved paused if the test fails
• login errors will now always specify the domain for which they failed
• testing a login script on a fresh edit login script dialog now pre-fills the alphabetically first example domain
• the login script test ui now restores its 'run test' button correctly if the test is abandoned early
• misc improvements to login error handling and reporting
• .
• other:
• any texts across the program that ellipsize when they are too thin to display what they have will now tooltip their text (this most importantly includes the status on the network job control, which will now display full login problem info)
• the copy button on manage tags goes back to copying all if no tags are selected
• the remove button on manage tags now removes only selected if some tags are selected. it still removes all if none are selected
• the remove button on manage tags is now wrapped in a yes/no dialog (as is hitting the delete key on the list's selection). this can be turned off under the cog button
• filename tagging panels now support directory tagging for the last, second last, and third last directories. the related code for handling directory tagging is cleaned up significantly
• the export files panel now lets you delete the files from the client after export. this value will be remembered, and if on will prompt a capital letters warning on export, either via the button or the quick-export shortcut
• in manage tag parents, where there are multiple parents in a pending action (either by importing via clipboard/file or by putting multiple parents in right-hand box), the action will now be treated as one transaction with one 'enter a reason' confirmation!
• in manage tag siblings, when multiple 'better' values are pended in one action via a clipboard/file import, they will now be treated as one transaction with one 'enter a reason' confirmation!
• .
• misc:
• added a new url class that api-links .gifv-style imgur links so they are downloadable like regular imgur single media pages
• the pixiv manga page url class now redirects to the new api, so mode=manga pages should now be drag-and-drop importable and generally downloadable if you have any still hanging around in any queues
• clients now come with an additional danbooru parser that fetches the webm version of ugoiras
• after discovering a pdf that ate indefinite 100% CPU while trying to parse, I have decided to stop pulling num_words for pdfs. it was always a super inaccurate number, so let's wait for a better solution at a later date. hydrus hence no longer requires pypdf2
• fixed an issue with monthly bandwidth estimates rolling over to the new year incorrectly
• in an attempt to chase down a duplicate files content move/copy bug, the duplicate action content updates got a bit of cleanup work. if you have noticed duplicate actions not copying tags/urls, please let me know the exact process in the ui, including services and merge options, you went through
• tag lists should now update their sibling appearance correctly after a tag siblings dialog ok--previously, they were checking for new sibs too early
• tag siblings and parents should now refresh their data more efficiently when spammed with new data notifications (this usually happens janitor-side, which approving dozens at once)
• copy queries/watcher urls on the download pages' lists' right-click menus no longer double-spaces the copied texts (it just does single spaces)
• fixed an issue where certain initialised watchers were erroring out when asked to provide next-check time estimates--in all cases, null timestamps will be dealt with better here
• misc tag parents/siblings ui code cleanup
• wrote some code to catch and report on an unusual dialog dismissal error

• version 328

• wrote test ui for edit login script panel
• the login system now works and is turned on, although the legacy hardcoded pixiv and hf logins remain in place. it will not do anything very new this week--it is strictly only for advanced users to experiment with for now
• cleaned up some messy network code
• all subscription and hydrus jobs will no longer wait indefinitely on an invalid login--they will cancel immediately
• network jobs will report a bit more info when they are cancelled
• subscriptions will now attempt to test login validity before and during file downloads and syncing. if they fail, the sub will pause and stop work and a message will be presented to the user
• made a 'thumbnails' options page and moved some things to it
• added thumbnail border and margin to that thumbnails page! you can even set 0 border and/or margin and it works
• fixed up a heap of bad thumbnail drawing code that didn't work with thicker borders
• the tag sibling and parent dialogs now have suggestion buttons in their 'give a reason' dialogs! if your petitions are simple and fit into one of these categories (which is most of them), please use these buttons as they will let janitors (e.g. hydrus dev for the PTR) process them in batches, in fewer clicks
• manage tags dialog's checkboxes and advanced buttons are now wrapped into a cog icon! remove/copy/paste buttons are compacted and put on the same row!
• manage tags dialog's copy button now only copies selected, not all tags
• manage tags dialog now uses the new sizer. some components are smaller by default but will eat up spare pixels better
• misc manage tags code cleanup
• added 'paged file import queues' to the network->pause menu. this will pause any, hdd, url, simple, gallery, or watcher page from processing its file import queue. it is a bit hacky and will take up to 30s to unpause unless you joggle the respective downloader to wake it up--see how it goes!
• added a similar 'gallery searching' to that menu, to pause any url, simple, or gallery page from actioning its gallery-side queue
• and 'watcher checking' for watchers
• fixed a stupid bug that was causing false-positive and _mostly_ harmless errors for certain pixiv and artstation multi-page downloads
• fixed an issue where multi-page data was being mis-metadata'd (mostly, this meant thread watchers were giving the last filename tag to all files, and I think similarly getting the latest source time) due to a mistake in some recent de-duping code
• fixed the new pixiv file page api parser to parse source time, which I must have accidentally deleted previously
• fixed a no-expand bug in my new sizer when in horizontal orientation
• fixed a small bug when making an easy-import downloader png and cancelling the add url class dialog
• misc fixes

• version 327

• login stuff:
• finished off some login script data stuff
• fleshed out how login credentials and other linked data is stored in the login manager, including script link recovery when the script changes but name does not
• improved some initialisation login validation error handling
• improved login failure validation error handling
• wrote a dialog panel for managing login credentials and reviewing validity and so on
• a heap of related session and login tie-in/fix-up work
• the login manager will now save changes to the db. it will get the HF and pixiv scripts on db creation/update, and if you have a pixiv login, the login system will pre-fill that info and 'activate' the script (although the login manager will not fire any login scripts yet--if so configured, it'll just delay on a polite error message)
• .
• other stuff:
• with the subscriptions' new more liberal syncing logic, the periodic file limit will now only pop up if the sub does not see any already-seen files
• to give more buffer for the new syncing logic, file import caches will now store 250 entries minimum on compaction (was 100 previously)
• subscription merging now lets you choose the primary subscription into which the other subs will be merged
• cancelling a subscription merge action mid-merge is now safely nullipotent
• post urls that use subsidiary page parsers (such as the new pixiv manga parser) will now correctly insert (rather than append) their manga urls into the file import cache
• removed a couple of places where urls could accidentally be duplicated in a file import cache
• cleaned up some areas where successful file import objects were presumed to have file hashes when they might not (this was causing errors when importing urls that split into multiple url children, like pixiv manga, while also having 'additional tags' set)
• updated tag censorship, parents, and siblings dialogs to the new panel system
• tag censorship, parents, and siblings panels now use a notebook instead of the layout-borked listbook
• tag parents and siblings panels now use the new small-resolution-friendly sizer, are more tight by default, and expand more neatly
• refactored a bunch of tag ui code to clientguitags
• the client video renderer will now deal with videos with (invalid) duration of 0 more gracefully
• finished the 'getting started with downloading' help page, sans the login stuff
• bit of other help work

• version 326

• login:
• finished the new login objects. they can deal with multi-step single second-level domain login problems, can pass variables from step to step, and use cookies as success verification
• wrote an ton of ui for the new login objects, now under network->downloader definitions and network->logins. it is not 'active' yet, but advanced users are invited to check it out. there is no good test ui yet, which I think I'll have to figure out in the coming weeks
• wrote a first attempt at HF and pixiv replacement login scripts--please try importing from defaults on the manage login scripts dialog and look through them to see what I am going for. once the system is flipped on and we are happy these work, I'll remove the old hardcoded legacy login stuff
• when a network job that needs a login cannot login, it now waits (rather than bombing out completely), presenting the related error, and checks again every 60 seconds
• if a network job thinks it can login but fails to generate a login process, the network engine now catches the error safely and recovers. the job is put on hold as above
• .
• subs:
• the subscription 'have we caught up to where we were before' test is now more complicated--rather than just stopping after five 'already seen' urls are found, it now only stops if at least the _last_ five contiguous urls of the page are already seen. this will catch more late-tagged files that get inserted out of order
• fixed the 'get quality info' button on edit sub panel to only get the current selection, not all queries wew
• subscriptions can now optionally publish/present their files to a specific label! this is a great way to merge multiple subs to the same final landing page
• .
• layout:
• after a long time thinking about it, wrote a new custom boxsizer that handles resizing multiple expanding items of different reasonable min size by expanding them _beyond minimum size_ by their proportion, rather than forcing them all to have total proportional width/height. I expect to polish this and apply it in multiple locations around the program where tall things were being too tall because something else was forcing it to be (the management panel on the left of most pages was terrible at this, causing a giganto taglist just because the upper panel was tall as well).
• changed my custom boxsizer (the box with a bold header) to the new custom boxsizer, so it is all over now--please report any bad layouts you see
• in an effort to improve layout, the manage tag parents and siblings panels' preview boxes have shorter minimum height--it will get a bigger layout overhaul soon
• .
• bigger misc:
• thanks to work of user kourraxspam on the discord, fixed the pixiv downloader to use a more stable api and added pixiv tag search
• watchers and gallery imports now have a list right-click menu entry to show all selected importers' files in a new page! use this to clear out a bunch of finished queues all at once!
• the tag right-click menu now offers 'open new search pages for each in selection' if multiple tags are selected--this will open three search pages each with one tag, as opposed to the original entry, which would only open one page with all three
• the edit nested gug panel now uses a checklistbox rather than the menu to select gugs to add, which is more reliable and allows for multiple selections
• sped up autocomplete tag fetches' tag sibling integration--irl this may be a reduction in total a/c search time of approx 33%
• page parsers will now generate next gallery urls absent any file/post urls if the only type of url they can generate is gallery urls (so a meta-gallery-search like board->threads that only generates subsidiary gallery pages will now work, whereas before it never could because it was missing post urls)
• the gallery log now provides a shorthand way to restart and resume failed searches from its right-click menu (if the most recent log entry failed)
• 'try again (and allow search to continue)' reattempt jobs will now generate next page urls even if no new urls are found (which can happen if a search stopped due to the file limit exactly lining up with the number of files found, for instance, so a reattempt finds nothing new)
• gallery downloaders will now specify their 'delay work for a bit' error states in the ui. this usually means 'could not connect', which has a 4-hour timer (I'll prob add a scrub delays button here at some point)
• the watcher will now show its 'delay work for a bit' error state in more places in the ui
• added a 'media' shortcut 'export_files_quick_auto_export', which will open the export files frame and give you a quick yes/no to confirm you want to export as set. if yes, it will export. then it will close the frame
• added a 'show a "N" to short import summaries' option to options->downloading, which will extend the typical 'x/y' status string to 'x/y - zN' for z 'new files' (as opposed to already in db)
• improved how the video parser estimates frame rate--it _should_ fix some of those low-framerate, low-framecount slideshow-vids where at current they render everything in a rush and then sit on the last frame for ten secs
• .
• smaller misc:
• network report mode now reports url_to_fetch and parser-to-parse-with info
• when the server fails to accept a file upload due to a file parsing issue, it now prints the hash of the file in the error
• if the client sees a possible file hash in a server error message from a file upload, ~it will try to show that file in a new page~
• fixed an issue where wildcard searches were not finding results if the search text included the normally discarded characters [](){}"'
• fixed some domain handling for localhost and other undotted network names
• content parsers will now only launch with permissable content types, which for the legacy 'lookup scripts' scripts system means only tags and vetoes, and for the new login system means only temp variables and vetoes
• as compaction now happens automatically on sync, removed the 'compact' button from edit subs panel
• an unusual network error related to hydrus update files sometimes being cut off mid-stream is now glossed over silently, with the download reattempted after a delay
• the initial gui session load now occurs after a 0.25s delay--let's see if it cleans up some initial layout issues some users have had
• maybe fixed an odd dictionary-initialisation error related to tag siblings/parents dialog boot
• ruggedised against an unusual bandwidth load bug
• gave some of the index help a pass
• did most of a 'getting started with downloaders' page in the help--I'll finish it next week
• updated discord share link to https://discord.gg/3H8UTpb , which should not expire
• some listbox add/edit code cleanup
• some listctrl delete code cleanup
• misc help work
• misc cleanup

• version 325

• added a 'show a popup while working' checkbox to edit subscription panel--be careful with it, I think maybe only turn it off after you are happy everything is set up right and the sub has run once
• advanced mode users will see a new 'get quality info' button on the edit subscription panel. this will some ugly+hacky inbox/archived/deleted info on the selected queries to help you figure out if you are only archiving, say, 2% of one query. this is a quickly made but cpu-expensive way of calculating this info. I can obviously expand it in future, so I would appreciate your thoughts
• subscription queries now have an optional display name, which has no bearing on their function but if set will appear instead of query text in various presentation contexts (this is useful, for instance, if the downloader query text deals in something unhelpful like integer artist_id)
• subscription queries now each have a simple tag import options! this only allows 'additional tags', in case you want to add some simple per-query tags
• selecting 'try again' on file imports that previously failed due to 'deleted' will now pop up a little yes/no asking if you would like to first erase these files' previously deleted file record!
• the watcher and gallery import panels now have 'retry failed' buttons and right-click menu entries when appropriate
• the watcher and gallery import panels will now do some ui update less frequently when they contain a lot of data
• fixed the new human-friendly tag sorting code for ungrouped lexicographic sort orders, where it was accidentally grouping by namespace
• downloader easy-import pngs can now hold custom header and bandwidth rules metadata! this info, if explicitly present for the appropriate domain, will be added automatically on the export side as you add gugs. it can also be bundled separately after manually typing a domain to add. on the import side, it is now listed as a new type. longer human-friendly descriptions of all bandwidth and header information being bundled will be displayed during the export and import processes, just as an additional check
• for advanced users, added 'do not skip downloading because of known urls/hashes' options to downloader file import options. these checkboxes work like the tag import options ones--ignoring known urls and hashes to force downloads. they are advanced and should not be used unless you have a particular problem to fix
• improved how the pre-import url/hash checking code is compared for the tag and file import options, particularly on the hash side
• for advanced users, added 'associate additional source urls' to downloader file import options, which governs whether a site's given 'source urls' should be added and trusted for downloaded files. turn this off if the site is giving bad source urls
• fixed an unusual problem where gallery searches with search terms that included the search separator (like '6+girls skirt', with a separator of '+') were being overzealously de/encoded (to '6+girls+skirt' rather than '6%2bgirls+skirt')
• improved how unicode quoted characters in URLs' query parameters, like %E5%B0%BB%E7%A5%9E%E6%A7%98 are auto-converted to something prettier when the user sees them
• the client now tests if 'already in db' results are actually backed by the file structure--now, if a the actual file is missing despite the db record, the import will be force-attempted and the file structure hopefully healed
• gallery url jobs will no longer spawn new 'next page' urls if the job yielded 0 _new_ (rather than _total_) file urls (so we should have fixed loops fetching the same x 'already in file import cache' results due to the gallery just passing the same results for n+1 page fetches)
• in the edit parsing panels, if the example data currently looks like json, new content parsers will spawn with json formulae, otherwise they will get html formulae
• fixed an issue with the default twitter tweet parser pulling the wrong month for source time
• added a simple 'media load report mode' to the help debug menu to help figure out some PIL/OpenCV load order stuff
• the 'missing locations recovery' dialog that spawns on boot if file locations are missing now uses the new listctrl, so is thankfully sortable! it also works better behind the scenes
• this dialog now also has an 'add a possibly correct location' button, which will scan the given directory for the correct prefixes and automatically fill in the list for you
• fixed some of the new import folder error reporting
• misc code cleanup

• version 324

• downloaders:
• after adding some small new parser tools, wrote a new pixiv downloader that should work with their new dynamic gallery's api. it fetches all an artist's work in one page. some existing pixiv download components will be renamed and detached from your existing subs and downloaders. your existing subs may switch over to the correct pixiv downloader automatically, or you may need to manually set them (you'll get a popup to remind you).
• wrote a twitter username lookup downloader. it should skip retweets. it is a bit hacky, so it may collapse if they change something small with their internal javascript api. it fetches 19-20 tweets per 'page', so if the account has 20 rts in a row, it'll likely stop searching there. also, afaik, twitter browsing only works back 3200 tweets or so. I recommend proceeding slowly.
• added a simple gelbooru 0.1.11 file page parser to the defaults. it won't link to anything by default, but it is there if you want to put together some booru.org stuff
• you can now set your default/favourite download source under options->downloading
• .
• misc:
• the 'do idle work on shutdown' system will now only ask/run once per x time units (including if you say no to the ask dialog). x is one day by default, but can be set in 'maintenance and processing'
• added 'max jobs' and 'max jobs per domain' to options->connection. defaults remain 15 and 3
• the colour selection buttons across the program now have a right-click menu to import/export #FF0000 hex codes from/to the clipboard
• tag namespace colours and namespace rendering options are moved from 'colours' and 'tags' options pages to 'tag summaries', which is renamed to 'tag presentation'
• the Lain import dropper now supports pngs with single gugs, url classes, or parsers--not just fully packaged downloaders
• fixed an issue where trying to remove a selection of files from the duplicate system (through the advanced duplicates menu) would only apply to the first pair of files
• improved some error reporting related to too-long filenames on import
• improved error handling for the folder-scanning stage in import folders--now, when it runs into an error, it will preserve its details better, notify the user better, and safely auto-pause the import folder
• png export auto-filenames will now be sanitized of \, /, :, *-type OS-path-invalid characters as appropriate as the dialog loads
• the 'loading subs' popup message should appear more reliably (after 1s delay) if the first subs are big and loading slow
• fixed the 'fullscreen switch' hover window button for the duplicate filter
• deleted some old hydrus session management code and db table
• some other things that I lost track of. I think it was mostly some little dialog fixes :/
• .
• advanced downloader stuff:
• the test panel on pageparser edit panels now has a 'post pre-parsing conversion' notebook page that shows the given example data after the pre-parsing conversion has occurred, including error information if it failed. it has a summary size/guessed type description and copy and refresh buttons.
• the 'raw data' copy/fetch/paste buttons and description are moved down to the raw data page
• the pageparser now passes up this post-conversion example data to sub-objects, so they now start with the correctly converted example data
• the subsidiarypageparser edit panel now also has a notebook page, also with brief description and copy/refresh buttons, that summarises the raw separated data
• the subsidiary page parser now passes up the first post to its sub-objects, so they now start with a single post's example data
• content parsers can now sort the strings their formulae get back. you can sort strict lexicographic or the new human-friendly sort that does numbers properly, and of course you can go ascending or descending--if you can get the ids of what you want but they are in the wrong order, you can now easily fix it!
• some json dict parsing code now iterates through dict keys lexicographically ascending by default. unfortunately, due to how the python json parser I use works, there isn't a way to process dict items in the original order
• the json parsing formula now uses a string match when searching for dictionary keys, so you can now match multiple keys here (as in the pixiv illusts|manga fix). existing dictionary key look-ups will be converted to 'fixed' string matches
• the json parsing formula can now get the content type 'dictionary keys', which will fetch all the text keys in the dictionary/Object, if the api designer happens to have put useful data in there, wew
• formulae now remove newlines from their parsed texts before they are sent to the StringMatch! so, if you are grabbing some multi-line html and want to test for 'Posted: ' somewhere in that mess, it is now easy.

• version 323

• wrote first version of the new downloader easy-import drop-panel. you drop downloader-encoded pngs on it, and it maybe asks you a question and jumbles its way through auto-importing all the required data to the client
• extended this file import to do some cleverer 'example url merging' when parsers are otherwise dupes, rather than spamming similar dupes on import
• wrote first version of the new downloader export panel. it takes gugs, url classes and parsers, and predicts sensible sub-objects to include to make functional downloaders, and bundles it into one png
• fleshed out help for the new easy import/export system
• the client now slows down gallery and watcher processing when the network engine is under heavy load, aiming for no more than 50 jobs in system at once. the solution is a bit hacky for now, but it should alleviate the deadlock issue when there are ~180+ simultaneous gallery/watcher network jobs pending
• the multi-watcher panel's list of watchers now supports right-click menu to copy/open urls and pause/play files/checking
• the multi-downloader panel's list of downloaders now supports right-click menu to copy query texts and pause/play files/searching
• added a 'derpibooru tag search - no filter' GUG that disables the default derpi no-explicit-files rule
• added basic gfycat support to default client--drag and drop any typical video page, and it should import ok
• fixed the canvas/hover window tag sorting discrepancy--all tags are now sorted with the same code, and the media view sort order should be the same as your default sort order (although in this case incidence has no effect as there are no tag counts)
• rewrote the network job control's cog menu to be a bit more dynamic, and added 'override gallery slot requirements for this job' if appropriate
• fixed a stupid typo bug in the shutdown maintenance jobs test code that was causing pending repository work to not report right
• fixed gallery searches that include unicode characters that end up in the path of the url (rather than the query parameters)
• fixed an issue where highlighting a watcher would unpause its checking
• generalised the way the new listctrl class can produce right-click menus
• fixed some api link calculation that was over-prescribing api link display pairs (this affected the artstation file page url class by default). these pairs are now also sorted in the links dialog
• misc png-export improvements to present better with the new easy import/export stuff
• the summary texts in the tag filter panel now ellipsize (...), so if the tag filter is complicated, it won't try to boot a superwide edit panel!
• the manage subscriptions panel now correctly initially sorts in a case-insensitive way (previously, it was usually sorting A-Za-z, which is different to regular aA-zZ resorting behaviour, so it always sort-flickered after the first edit)
• the status bar has a new segment for reporting when the client is 'busy' with different jobs. for most typical usage, it'll just stay blank. let's see how it goes.
• fixed mr. bones's wild review when the client currently has no files
• punched up the new file report mode to specify full paths where available
• improved some misc downloader code

• version 322

• wrote gugs help
• gave url classes and parsers help a pass
• wrote e621 html gallery page example help
• wrote gelbooru html file page example help
• wrote artstation json file page example help
• wrote url class links help
• gallery logs for the gallery downloader and url downloader now support 'try again' and 'skip' right-click menu for gallery log entries. the try again allows just the one page or also allowing search to continue) so, if a gallery query fails for some reason, you can now try again/continue where it broke. subs/watcher/simple downloader work on more complicated gallery search logic, so their gallery logs will remain read-only for now
• all gallery log buttons now support right-click menu to mass-export urls to png or clipboard. non read-only also support import
• fixed an issue with gallery searches that rely on both api url conversions and url class next gallery page urls (I think just artstation and tumblr by default) not generating the next page url correctly
• improved some misc gallery url processing logic
• fixed some issues with gallery url generators with invalid example urls causing problems opening the edit gug and gallery selector panels
• fixed an issue where you could only delete a gug if it was in an ngug, ha ha
• thanks to a different submission by prkc on the discord, collections now have a _right-click->set collections as groups of alternates_ duplicate action (note the duplicate menu only appears in advanced mode). the related shortcut action duplicate_media_set_alternate_collections is also added
• export phrases now support '\' ('/' in linux) in the path export phrase in order to create folders. you should also be able to do \[series]\ to create optional namespace folders. slashes in tags will still be replaced with _
• to stop the client sometimes doing laggy vacuum checks every maintenance cycle, vacuums that cannot occur due to limited disk space now will still count as 'done' for the purposes of rescheduling
• added 'file report mode' to the help debug menu. This will spam popups as file and thumbnail actions are requested
• tightened up some network job status setting to help us debug the 'there are a ton of jobs in network engine, but the three active on this domain seem stalled' issue
• wrote a simple 'review threads' panel under help->debug->data actions->review threads. I knocked it together in about ten minutes, and it's likely unstable as hell, but it's pretty neat!
• some instances where many file paths are copied quickly (exporting paths to clipboard and drag and drop) no longer do a safety check for file existence, so should be much faster to go. this particularly reduces startup lag for large file drag and drops!
• the 'would you like to do maintenance work in this shutdown?' dialog now lists a summary of what it thinks it'll be working on. I _could_ make this more detailed, so let me know how it works for you
• tags with numbers should now sort according to the new improved human sorting method--it now shouldn't matter where the numbers are in the tag--as long as the text-and-number-breaks lines up with another tag, they'll be compared each part in turn correctly
• fixed some human sorting code for unusual number characters like ?. they will be treated as text, not a number, for now
• misc fixes

• version 321

• downloader overhaul:
• the basic downloader overhaul is complete! at this point, any user can create and share the objects required for a completely new downloader! it is still rough in some places, so a round of EZ-import is coming to make adding new downloaders a single easy drag and drop action
• rounded out the ngug (nested gugs, which contain multiple gugs) code
• updated the edit gug panel to deal with gugs and ngugs on different notebook pages
• added a bunch of logic to this panel and backend data handling to deal with missing gugs in ngugs
• if an ngug cannot find a gug by its internal identifier key, it will now attempt to fallback to its simple name, and will silently fail if no gug can be found. all gug tracking now uses this 'key first, name later' id method, so downloaders and subs should generally survive gug renames and same-name overwrites
• the gallery selector now works in gugs. it has two 'pages', depending on which gugs are set to 'display', and will note if the chosen gug is cannot be found in the current definitions. the gugs have slightly more specific names ('gelbooru tag search', 'hentai foundry artist lookup', etc...) than before
• the gallery selector also puts 'non-functional' gugs (i.e. those with no parsable gallery url class) to a third page
• moved the gallery downloader gallery and file pipeline completely over to the new system
• the gallery downloader will now bundle nested gugs (like hentai foundry artist, which searches both works and scraps) into a single downloader
• moved the subs gallery and file pipeline comppletely over to the new system
• the subs gallery sync now handles nested gugs (like hentai foundry artist, which searches both works and scraps) in an interleaved manner and make behind-the-scenes checking decisions in a clearer and more logical way
• subs should now make correct 'hit limit' stop reason reports and not generate new gallery pages when the current page has exactly enough results to hit the current file limit
• artstation artist lookup is now available as a default downloader
• newgrounds artist lookup makes a triumphant return. it works pretty well, given how flash and NG has changed since
• derpibooru tag lookup is now available as a default downloader. due to unusual search syntax on derpibooru, please enter queries exactly as you would on derpi, using ',' or ' AND ' to separate tags (such as 'rainbow dash,straight')
• pixiv now has multiple artist lookup options--either images, manga, ugoira (doesn't work yet!), or everything
• the old downloader code is deleted!
• the old manage booru dialog is deleted!
• 'custom' boorus (i.e. new ones you created or imported to 'manage boorus'), cannot be completely automatically updated to the new system. I've figured out a way to generate new gugs and gallery&post parsers, but they will miss url classes to get working again. your custom-booru subs will notice this and safely pause until the issue is fixed. if you rely on custom boorus, please check the release post for info on this--you might like to put off updating
• many misc changes and fixes to gugs and overall gallery url handling pipeline
• some misc refactoring and concept-renaming in gallery pipeline r.e. gugs
• when the downloader tries to import what looks like a raw html file, its error notes will specify this and suggest a parser may be needed
• moved the 'media viewer url display' options panel from the manage url match links dialog to the new network->downloaders->manage downloader and url display
• this new dialog also hosts a list for managing which downloaders to show in the first list of the downloader selector
• .
• misc:
• gave the video rendering pipeline communication logic a quick pass, cleaning up a bunch of bad code and other decisions. the video renderer should be quicker to respond to various changes in scanbar position, and incidences of the frame buffer suddenly sperging out (usually inexplicably falling behind the current frame position or deciding to regen for no apparent reason) should be greatly reduced if not completely eliminated
• the test that stops repository processing if there is not enough disk space now uses half the current size of client.mappings.db for its estimate (previously 1GB) and also tests temp folder location free space (just as the vacuum test does) and reports this nature of the error along with pausing the repo, stopping further attempts
• might have fixed another out-of-order dialog close/open event combination during manage tags close->advanced content update open
• fixed gallery queries that include '/' (or some other unusual characters) that end up in the 'path' of the url (as opposed to the query). this fixes 'male/female' on e621, for instance
• 'advanced mode' users now have a 'nudge subs awake' menu entry below 'manage subs'. this simply wakes the subs daemon (which usually only checks once every four hours), in case any subs are due
• 'db report mode' now reports every db job as it comes in (formerly, it only reported some optimisation esoterica). this makes it a more lightweight version of 'db profile mode' for several debugging tasks
• fixed a tiny issue in fetching the 'how boned am I?' stats when the user had zero inbox/everything count
• fixed a typo in the default new url class object that was breaking the edit ui panel
• highlighted the quiet filename tagging options on edit import folder panel

• version 320

• clients should now have objects for all default downloaders. everything should be prepped for the big switchover:
• wrote gallery url generators for all the default downloaders and a couple more as well
• wrote a gallery parser for deviant art--it also comes with an update to the DA url class because the meta 'next page' link on DA gallery pages is invalid wew!
• wrote a gallery parser for hentai foundry, inkbunny, rule34hentai, moebooru (konachan, sakugabooru, yande.re), artstation, newgrounds, and pixiv artist galleries (static html)
• added a gallery parser for sankaku
• the artstation post url parser no longer fetches cover images
• url classes can now support 'default' values for path components and query parameters! so, if your url might be missing a page=1 initialsation value due to user drag-and-drop, you can auto-add it in the normalisation step!
• if the entered default does not match the rules of the component or parameter, it will be cleared back to none!
• all appropriate default gallery url classes (which is most) now have these default values. all default gallery url classes will be overwritten on db update
• three test 'search initialisation' url classes that attempted to fix this problem a different way will be deleted on update, if present
• updated some other url classes
• when checking source urls during the pre-download import status check, the client will now distrust parsed source urls if the files they seem to refer to also have other urls of the same url class as the file import object being actioned (basically, this is some logic that tries to detect bad source url attribution, where multiple files on a booru (typically including alternate edits) are all source-url'd back to a single original)
• gallery page parsing now discounts parsed 'next page' urls that are the same as the page that fetched them (some gallery end-points link themselves as the next page, wew)
• json parsing formulae that are set to parse all 'list' items will now also parse all dictionary entries if faced with a dict instead!
• added new stop-gap 'stop checking' logic in subscription syncing for certain low-gallery-count edge-cases
• fixed an issue where (typically new) subscriptions were bugging out trying to figure a default stop_reason on certain page results
• fixed an unusual listctrl delete item index-tracking error that would sometimes cause exceptions on the 'try to link url stuff together' button press and maybe some other places
• thanks to a submission from user prkc on the discord, we now have 'import cookies.txt' buttons on the review sessions panels! if you are interested in 'manual' logins through browser-cookie-copying, please give this a go and let me know which kinds of cookies.txt do and do not work, and how your different site cookie-copy-login tests work in hydrus.
• the mappings cache tables now have some new indices that speed up certain kinds of tag search significantly. db update will spend a minute or two generating these indices for existing users
• advanced mode users will discover a fun new entry on the help menu
• the hyperlinks on the media viewer hover window and a couple of other places are now a custom control that uses any custom browser launch path in options->files and trash
• fixed an issue where certain canvas edge-case media clearing events could be caught incorrectly by the manage tags dialog and its subsidiary panels
• think I fixed an issue where a client left with a dialog open could sometimes run into trouble later trying to show an idle time maintenance modal popup and give a 'C++ assertion IsRunning()' exception and end up locking the client's ui
• manage parsers dialog will now autosort after an add event
• the gug panels now normalise example urls
• improved some misc service error handling
• rewrote some url parsing to stop forcing '+'->' ' in our urls' query texts
• fixed some bad error handling for matplotlib import
• misc fixes

• version 319

• started the new convert-query-text-to-gallery-urls object. these objects, which I was thinking of calling 'Searchers', will be called the more specific and practical 'Gallery URL Generators', or GUGs for short
• the first version of GUGs is done, and I've written some test ui for advanced users under network->downloader definitions->manage gugs. this ui doesn't save anything yet, but lets you mess around with different values. if we don't think of anything else needed in the next week, I will fix this code for v320 and start filling in defaults
• watchers now have a checking slot, much like the recent change to galleries and subs. it safely throttles dozens of threads so they don't rudely hammer your (or the destination server's) CPU if they all happen to want to go at once (like just after your computer wakes up). the option is similarly under options->downloading
• moved the new gallery delay/token management code to the better-fit bandwidth manager (it was in domain manager before)
• the gallery delay/token code now works per-domain!
• moved the gallery delay/token checking code into the network job proper, simplifying a bunch of import-level code and making the text display now appear in the network job control. token consumption now occurs after bandwidth (it is now the last hoop to jump through, which reduces the chance of a pileup in unusual situations) I expect to soon add some kind of 'force-go' action to the cog menu
• the network engine will now not permit more than three jobs active per domain, and the overall limit has been raised from ten to fifteen
• the media right-click menu now supports copying: all of a files recognised urls; all of a files urls; all selected files' urls of a specific url class; and all selected files urls
• reworked and harmonised a bunch of urlparsing and generation code--all urls should now appear as full unicode across the program, generally without %20-type encoding characters unless explicitly entered by the user. character encoding now all happens on the backend in requests
• non-url-class-matched urls now have their query parameters alphabetised as part of the normalisation process
• all urls in the db will have their query params alphabetised on update, and any file relationships merged to the new/existing normalised url
• the manage urls dialog will now normalise newly added urls (but should also still permit the removal of non-normalised urls)
• reworked how gallery hits update file import object caches, particularly for subscriptions
• fixed an issue in subscriptions gallery logging where the gallery log would always state it had found the max number of files and typically redundantly generate an 'ignored' stub--it should now say something like 'found 7 files - saw 5 previously seen urls, so assuming we caught up' as originally intended
• simplified some gallery->file import object creation
• galleries now compact until 100 entries (was 25)
• watchers now gallery-compact after a successful check
• watchers now show the 'just added'/'already watching' status for 15s, up from 5s
• network report mode now reports three time--once each for job addition, start, and successful completion
• fixed an issue with the new 'max width' popup sizing calculation that was sometimes not fitting for new height requirements correctly
• fixed an issue with the new url class next page generation code
• fixed an issue where TIOs with data regarding since-deleted services were failing to initialise at the ui level
• misc status text cleanup

• version 318

• downloaders:
• extended url classes to support 'next gallery page' generation--a fallback that predicts next gallery page url if the parser cannot provide it (as is often the case with APIs and unreliable next-page-url galleries such as gelbooru)
• integrated this new next page generation into new gallery processing pipeline
• updated gelbooru, tumblr api and artstation gallery api url classes to support the new next gallery page business
• fixed the url class for xbooru, which wasn't recognising gallery urls correctly
• wrote new gallery parsers for rule34.paheal and mishimmie (which are both shimmie but have slightly different gallery layout). this should finally solve the 'one paheal gallery url is being parsed into the file list per page' problem
• 'fixed' the tumblr parser to fetch the 1280px url (tumblr killed the raw url trick this past week)
• misc text/status fixes
• wrote a gallery parser for tumblr that fetches the actual tumblr post urls and hence uses the new tumblr post parser naturally! (tumblr post urls are now more neatly associated as 'known urls' on files!)
• note that as the tumblr downloader now produces different kinds of urls, your tumblr subs will hit your periodic limits the next time they run. they will also re-download any 1280px files that are different to the previously fetched raws due to the above raw change (protip: keep your subscription periodic file limits low)
• cut the 'periodic limit' subscription warning popup down to a much simpler statement and moved the accompanying help to a new help button on the edit sub panel
• multi-gallery pages now have an 'added' column like multi-watchers
• the new 'pause' ? and 'stop' ? characters shown in the multi-downloader pages are now customisable under options->downloading (some users had trouble with the unicode)
• the watcher now shows the 'stop character' if checking is 404/DEAD
• fixed an issue where the new gallery imports on the same multi-page were all sharing the same identifier for their ephemeral 'downloader instance' bandwidth tracker, which meant they were all sharing the same '100rqs per 5mins' etc... rules
• the page and subscription downloader 'gallery page delay' is now program-wide (since both these things can run in mass parallel). let's see how it goes, maybe we'll move it to per-site
• subscription queries now auto-compact on sync! this means that surplus old urls will be removed from their caches, keeping the whole object lean and quick to load/save
• gallery logs now also compact! they will remove anything older than twice the current death velocity, but always keep the newest 25 regardless of age
• .
• misc:
• the top-right hover window will now always appear--previously, it would only pop up if the client had some ratings services, but this window now handles urls
• harmonised 'known urls' view/copy menu to a single code location and added sorted url class labels to entries (which should reduce direct-file-url misclicks)
• greatly sped up manage tags dialogs initial calculation of possible actions on a tag alteration event, particularly when the dialog holds 10k+ tags
• greatly sped up the second half of this process, when the action choice is applied to the manage tag dialog's current media list
• the buttons on the manage tags dialog action popup dialog will now only show a max of 25 rows on their tooltips
• some larger->smaller selection events on large pages with many tags should be significantly faster
• subscription popups should now 'blank' their network job controls when not working (rather than leaving them on the old job, and without flickery-ly removing the job control completely)
• the file cache and gallery log summary controls now have ... ellipsized texts to reduce their max width
• fixed an issue where larger 'overriding bandwidth' status wait times would sometimes show instead of legit regular smaller bandwidth wait times
• removed a now-superfluous layer of buffering in the thumbnail grid drawing pipeline--it seems to have removed some slight lag/flicker
• I may have fixed the issue where a handful of thumbs will sometimes remain undrawn after several fast scrolling events
• gave the some-linux-flavours infinitely-expanding popup message problem another pass. there _should_ be an explicit reasonable max width on the thing now
• added a 'html5lib not found!' notification to the network->downloaders menu if this library is missing (mostly for users running from source)
• help->about now states if lz4 is present
• gave 'running from source' help page another pass, including info on running a virtual environment
• in file lookup scripts, the full file content now supports string transformations--if this is set to occur, the file will be sent as an addition POST parameter and the content-type set to 'application/x-www-form-urlencoded'. this is a temp fix to see if we can get whatanime.ga working, and may see some more work
• if the free space on the db dir partition is < 500MB, the program will not boot
• if the free space on the db dir partition is < 1GB, the client will not sync repositories
• on boot the client can now attempt to auto-heal a missing local_hashes table. it will give an appropriate error message
• misc post-importing-cleanup refactoring

• version 317

• completely overhauled the tag filter panel:
• the tag filter panel now has 'whitelist' and 'blacklist' pages beside the old 'advanced' sub-panel. these new simple pages are much more human friendly for common workflows and provide easy-select checkboxes for namespace classes (which are compiled from all the namespaces your parsers can currently do)
• the tag filter rule entering workflow now stops you from creating overcomplicated rulesets: when adding a blacklist rule, it will now only add an explicit entry if it is not already blocked by a higher rule (otherwise it will just discard from whitelist, if there)--and when adding a whitelist rule, it will now only add an explicit entry if it is already blocked by a higher blacklist rule (otherwise it will just discard from blacklist, if there)
• tag filters now provide more human-friendly summary statements
• misc improvements to tag filter ui logic
• the various help texts surrounding the tag filter panel all got passes
• the tag filter panel now uses text-and-paste controls for mass-adding of tags
• namespace checkboxes have been completely removed from the tag import options panel and various other related places. any existing TIO with checked namespaces will be automatically updated to 'get tags' with an appropriate filter. this is an important step in the rewrite--everything is now handled in the new tag filter panel
• simplified and sped up the actual tag filtering code
• .
• numerous multi-importer improvements:
• the gallery and watcher page lists will now ~dynamically~ resize in height based on number of entries, from roughly four columns to twenty four. this relayout code somehow seems to work on all platforms
• sped up the 'results loading' step of gallery/watcher highlighting immensely--on a typical list of a couple hundred files, it should now be about 50ms total (before, depending on presentation rules, it could be 0.8-3s)
• added an additional db-skipping optimisation for calculating presentation status
• watcher and gallery highlights will now filter out trash and completely deleted files (the ones that appear with a dark default 'hydrus' icon) on reloads
• added two checkboxes to options->downloading for 'if nothing is highlighted when I add a new X, highlight that new X' for watchers and galleries
• adding or removing a query or watcher from the new multi-lists should now be reflected in the list ui instantly, rather than after a <=1s delay
• added url classes and parsers for imgur single and multiple urls--thanks to the community for providing some examples
• added url class and parser for derpibooru single file pages--again thanks to the community. derpibooru hence now supports basic drag and drop import
• fixed an issue where the watcher was often still checking despite 404 status
• watchers and galleries use a little less CPU to update some of their ui
• added simple subsidiary page parsing support to file import objects (previously, this only worked in the gallery log)
• .
• gave the thumbnail scrolling code a pass--it is now a bit cleverer about drawing and uses a larger number of smaller 'tile' bmps rather than pages
• added an 'EXPERIMENTAL' option to options->gui to change the number of thumbnails each scroll tick scrolls. it defaults to 1.0, but you _should_ be able to set 0.5, 0.37, whatever. please report any bugs!
• added a thumbnail debug mode to help see the new thumbnail layout boundaries
• .
• misc:
• the max subscription file limits are now 10,000 for users in advanced mode
• the default subs initial/periodic limit is now 100/100 (bumped up from 100/50)
• the file import dialog now has a little cog icon to change whether human sort is applied on path addition events (for e.g. if you want to add in some date order from an explorer window)
• humansort now sorts case-insensitive
• by default, unmatched urls will no longer display in the top-right of the media viewer. see how you like this and let me know if you would like an option to put them back
• the speed text on the right-side of the network job control now dynamically resizes to its min size, which gives the text on the left side (where it is often cut off, saying 'overriding bandwidth ...') more space when available
• I think I fixed an issue where the popup frame could spam-resize in odd ways (such as growing a pixel wider every update tick)
• watchers will no longer include the '* ' highlight prefix in subject-based sort comparisons
• in prep for an eventual major code refactoring, the thumbnails' underlying media object now stores a faster db-based numeric file identifier
• 'duplicate' calls on the new listctrl will now insert the dupes in the current correct sort location, rather than tacking them on the end
• drag and drop imports to the new listctrl will also now insert like this
• caught up edit subscriptions panel to the finalised common listctrl panel code, including the import/export/duplicate buttons
• the multiple checkboxlist selection dialog now sorts by label
• converted all old checkboxlist dialogs to the new panel system
• massively sped up certain kinds of parsing that were wasting time hitting a cache test way too often
• fixed an old hash filtering system
• moved to a simpler and more stable way of calculating certain text extents
• fixed an issue where the include directory (which has the original source, which isn't a big deal but is nice to have) wasn't being correctly copied into the linux build
• the os x .tar.gz build now has the include directory
• refactored some client tags code around
• misc cleanup

• version 316

• gallery:
• gallery url classes can now be linked to parsers!
• if parsers are linked, gallery pagewalk can now work on the new parsing system. gallery import pipeline has been significantly updated to reflect this
• gallery import objects are now 'multiple' gallery imports, much like the multi-watcher, with each separate query having its own entry in a list (they also run in parallel!)
• the multi-gallery list will show file/gallery pause status in slender columns, and will show a 'stop' character when gallery parsing is done
• wrote a 'gallery selector' button and added it to the new multi-gallery page, so you can spawn queries for ~different sites~ on the same import page! it always defaults to 'deviant art' for now, but when the next 'searcher' overhaul step is done, this will be customisable
• the new page selector and related 'pages' menu is now simpler--with the new selector, you just select 'gallery'
• added 'new_gallery_downloader_page' shortcut action to the 'main gui' set to allow quick opening of this new page type
• wrote a 'gallery import panel', which reviews a single gallery import stream, and added it to the multi-gallery page to show the current highlighted query
• as all gallery imports now run in parallel and work on the new system, the now almost-useless 'cancel' gallery pagewalk button is now removed
• with the wider availability of the new gallery log for file count and error reports, shifted around and smoothed out some gallery status text presentation
• improved the auto url_class->parser linking 'try to fill in gaps' logic to work with gallery urls (this was surprisingly complicated)
• fixed a misc stupid waste of time in auto url_class->parser linking
• many misc updates to gallery pipeline
• .
• subscriptions:
• wrote a new gallery pagewalk pipeline for subscriptions, which still does oldest-to-newest url addition and obeys file limits and so on
• numerous subscription pipeline and error handling tweaks and improvements, particularly in regards to the new code
• subscriptions now have max initial and periodic file limits of 1000. existing subs with >1000 or infinite will be cut to 1000. there is a help button on the edit sub panel to explain why you should do large syncs with the manual downloader and not subs
• the ugly and dangerous-if-you-scroll-in-the-wrong-place gallery selector mismash control in the edit subscription panel is now replaced with the new gallery selector button
• fixed an issue where the edit subs panels could sometimes say '48 years ago' (i.e. displaying a literal time delta since 0, 1970) on initial timestamps
• juggled some 'periodic limit' reporting logic to skip an unusual false positive that affect hentai foundry subs for now and more in future
• .
• urls:
• the url downloader now accepts gallery urls and will receive drag-and-dropped gallery urls. at the moment, it only parses the one page (i.e. it doesn't start a new 'searching' pagewalk) and sends the parsed links to its file queue
• .
• watcher:
• finished 'watcher panel', which reviews a watcher, and added it to the multi-watcher page to show the current highlighted watcher
• the single watcher page is completely removed--it is only the multiple watcher now. all singles will be converted to multiples on update
• some single-watcher options (like watchers naming their own page tabs and the [404]-style page name prefixes) are removed
• multiple watcher panel now lists file/checking pause status and has separate buttons to control these paused statuses
• fixed some misc watcher highlight code--highlighted watchers should correctly publish to the page from the start of session load now
• improved some 'repage' logic in how highlighted threads get removed
• .
• misc:
• discovered a scroll-setup parameter that stops janky scroll-to-click-focus behaviour on all the new scrolling panels, thank the LORD
• improved some 'can't parse' error handling for post url parsing
• reworked how all importers present their network jobs to the ui, including fast response if the switch happens during a job
• in prep for searcher switchover where all downloader sources will be harmonised into one system, booru identifiers now present in several ui locations as 'name', not 'booru: name'
• updated the danbooru parser to deal with the new 'next page' markup they use
• wrote a gelbooru gallery parser that works with 0.2.0 and 0.2.5 gelb sites--an ancillary issue where gelb-related downloaders could sometimes not accurately figure out the magic '42' next-page offset is hence now fixed
• wrote an e621 gallery page parser
• these sites hence now support single-page gallery drag-and-drop
• added url classes for artstation gallery url and its api counterpart, but didn't go further yet--we aren't quite there with api pagewalking just yet
• updated deviant art gallery url classes
• added an e621 'search initialisation' gallery url class to improve some future drag-and-drop stuff
• url normalisation no longer cuts off 'www.'-style prefixes
• url comparison is more careful to test 'www.'-style prefixes, so a file import cache should recognise that 'http://www.blah.com/blah' is the same as 'https://blah.com/blah'
• did a bunch of refactoring to further split up the bloated ClientImporting.py
• fixed some misc downloader layout that may have been hiding some texts previously
• some multi-watcher and multi-gallery events like add/pause query should be a bit snappier
• in the parsing ui, url and title priorities are now 50 by default
• prepped a little subscription unit test code for when searcher object is done
• misc downloader layout improvements
• misc listctrl refactoring

• version 315

• got started on the big gallery update, but decided not to pull the trigger just yet. I hope to do it next week, switching the whole thing over to a two-object multi-watcher kind of deal
• updated to wxPython 4.0.3 for all platforms
• cleaned up some menubar replacement code, and the update to the new wxPython should also fix a "event for a menu without associated window" bug some gtk2 users were seeing on quick menubar changes
• manage default tag import options panel now has copy/paste buttons that work on the listctrl
• added some 'paste tag import options' safety code to make sure no one accidentally pastes a subscription or something in there, wew
• added default checker options for subscriptions to options->downloading
• unified how checker options are edited from their button, much like how file and tag import options work. it also has a summary tooltip on the button
• the checker options under options->downloading are now these slimmer buttons
• in the manual import dialog (which pops up when you drop a folder/files on the client), the files will now be added in 'human friendly' number sorting, so files of the sort 'Favourites - 10.jpg' will sort [10, 11, ..., 99, 100] rather than the purely lexicographic [10, 100, 11, ..., 99]
• gave the migrate database dialog a pass--a bunch of misc presentation changes and a general simplification of workflow, now based more on just increase/decrease location weight
• a bunch of texts on page management (left-hand) panels that share horizontal space with buttons should now ellipsize ("downlo...") when they get too long for the width instead of drawing in an ugly way over the buttonspace
• moved the manage import folders dialog to the new listctrl and added a 'paused' and better 'check period' column
• if a user tries to run a 'paused' import folder specifically from the menu, the import folder will now unpause (I will probably remove this old paused variable in the future--it isn't of much use any more)
• tightened up some repository reset code that wasn't deleting all service tables and hence recovering from some service id malformation errors correctly
• wrote a 'clear orphan tables' db maintenance routine that kills some spare tables some users who have previously deleted/reset repositories may have floating around
• fixed an issue with parsing folders after hitting cancel button on the import files pre-dialog
• if watchers encounter non-404 network errors during check, they should now just delay checking for four hours (before, they were also pausing checking completely)
• if watchers are in 'delay' mode, they'll also not work on files.
• file and gallery downloads that hit a 403 (Forbidden) will now present a simpler error status, like they do for 404
• the new post downloader will no longer fail if one of the parsed source urls is not a url. the borked string will also not be associated as a url
• regular gallery downloads now override bandwidth for the file download step, which is almost always the second half of a pair of post_url/file downloads, just to keep things in sync in edge cases
• cleaned up some timestamp generation and 'overriding in x seconds' strings to be more human friendly
• improved some serverside file parse error handling to propagate the actual error description up to the client a bit better
• fixed typo causing incorrect num_ignored count in file import status button right-click menu
• parseexceptions will now present more data about which page and content parser caused the problem. I am not totally happy about how this solution works and may revisit it
• the lz4 import error catching is now more broad to catch some odd problem I discovered in new Linux build environment
• the moebooru parser now fetches the original png of an image, if available
• added a new tumblr parser that also gets post tags--it _shouldn't_ be the default
• the new login pipeline now kicks in for the legacy logins--pixiv and hentai foundry--on a per-url basis, so adding pixiv/hf urls to the url downloader will trigger a login even if needed (previously, this was tied to legacy gallery initialisation, which explains some pixiv 'missing' login stuff some users and I were having trouble with)
• if the legacy login system fails in the new pipeline, it now sets a flag and won't try again that client boot
• the old 'default tag import options' panel is now completely removed from options->importing. please check 'network->downloaders->manage default tag import options' for the new url-based settings
• misc fixes

• version 314

• tag import options can now be set to 'default', meaning 'use whatever the default is at the time of import', which will be an easier way of managing TIOs for many subs that you'd prefer all share the same TIO settings anyway
• updated tag import options ui to enable this default setting where appropriate
• updated the newer import pipeline to work with 'default'-set tag import options
• new downloaders, subscriptions, watchers, and multi-watchers now start with 'default' tag import options
• deleted the old default tag import options management code and put some text up on options->importing making notice about the impending shift. a popup message will also say this on update
• tag import options buttons now have a right-click menu with copy/paste/default options for quick assignment and duplication!
• added 'overwrite tag import options' to manage subscriptions--once you are comfortable with the new 'default' TIO mode, and after some small tests, you might want to switch all your subs over to 'default'
• the 'urls' downloader now has a tag import options--it initialises as 'default'
• added furry.booru.org to gelbooru 0.2.0 parser
• wrote a hentai foundry file page parser
• wrote a moebooru file page parser (this works for konachan, yande.re, and sakugabooru)
• wrote a shimmie parser (this works for rule34.paheal, rule34hentai, and mishimmie)
• wrote a newgrounds parser
• integrated the user-created sankaku parser
• wrote a tumblr parser that handles photo, photoset, and video posts, auto-converts to 'raw' urls for those post-2012 urls that can handle it, figures out a creator tag (reverting to the reblog root if it is a reblog post!), and cooks you breakfast
• (hence all these above sites now support drag and drop!)
• rolled out some new tumblr url classes to handle all this.
• added sakugabooru url classes
• fixed an issue where url classes were not normalising api urls in all cases, meaning some url classes would not api-link correctly in 'manage url class links' panel
• fixed an issue with deviant art legacy gallery parser pulling some funky 'creator:' tags
• some misc new downloader error handling improvements
• the watcher now uses the new gallery object to parse and generate file import objects
• the downloaders with gallery logs should now report non-success gallery fetches, along with error tracebacks (this will include some no-worry 404s the legacy downloader sometimes uses to terminate searches)
• added image and thumbnail cache timeout time delta buttons to options->speed and memory
• added a 'show the D on short file import summaries' checkbox to options->downloading--it defaults to off
• the 'I' on short file import summaries is now 'Ig' to clear up 1/I confusion
• added 'copy queries' to the edit subscription panel, which lets you copy all the selected queries' search texts to clipboard, newline separated
• added a checkbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on
• fixed file import status button right-click, which I messed up somehow last week with the 'retry ignored' add
• shook up and collapsed the network menu into neater categories
• tightened-up the rarely used pre-parsing conversion panel on the edit page parser panel to just a button with a bit of explaining text
• if database errors include the word 'malformed', the client now throws a little extra error text pointing people to the help.txt in the db dir
• cleared out some legacy download code
• cleared out legacy hard drive import error handling, moving it all to the new file import object
• misc refactoring and cleanup

• version 313

• fleshed out the new gallery log and its constituent log entry objects
• added gallery logs to gallery downloaders, subscriptions, url downloaders, simple downloaders and watchers
• added very simple gallery log reporting to these downloaders
• added first, read-only version of gallery log ui to these downloaders
• fleshed out some new gallery/file-object pipeline stuff
• wrote a simple danbooru gallery page parser and added it on update. it doesn't do anything yet, but if you are into the new parsing system, please check it out as an example
• the url downloader now has a full file import status control with status text
• fixed a url count issue on completely fresh gallery downloads that was stopping gallery searches one file (like 199 vs 200) before the file limit
• the pixiv downloader now fetches 'type=all' gallery pages, which include specifically manga file pages (as opposed to merely multi-file 'illustrations')
• added 'retry ignored' to the file import status button's right-click menu
• fixed the deviant art url class and parser to use the new file page format. also added an '(old format)' class to match the old way for legacy purposes (this legacy class also uses an api conversion to connect to the new parser--we'll also figure out a way to convert all these over at the db level en masse later!)
• updated some similar deviant art gallery stuff as well
• tag import options now has a tag filter to go along with the 'get all tags' checkbox! ('get all tags' is now renamed to 'get all' as a result). this filter lets you make more complicated tag filtering decisions like 'get all tags except "species:" tags'.
• the new 'only get tags if they already exist' checkbox now also has a filter, if you want to only apply this test to a subset of tags (like the unwashed mess of unnamespaced tags many boorus and sites provide)
• generalised a 'tag filter' button class to make it simpler to edit tag filters across the program, and cleaned up some related status code
• fixed a problem with deriving tag import options for specific url classes when that url class was part of an api-url-class chain
• if the domain manager cannot now find a url match for a pending download, it now assigns the file post default tag import options to that import
• added a new 'duplicates' options page that has a hacky way to edit the weighted scores used to determine which of the pair of files to present file in the duplicates filter
• unifed how some file import status generation works, adding a new 'simple status' string to briefly summarise progress in multi-watcher and edit subscriptions columns
• cleared out some old redundant status caching in the urls downloader
• simplified how almost all timestamp strings are generated
• simplified how time delta strings are generated
• brushed up some simple common ways to present timestamps as 'human pretty' strings
• all places where timestamps would be presented as a mix of '5 days ago' and complete datetime strings will now present as '5 days ago' unless you set the new options->gui 'always show iso' checkbox. going back to simple to clear up confusion in workflow and code. I may revisit this, as turning on ISO mode now spams it all over the place
• cleaned up the 'looks like the computer just woke from sleep' check and reduced its grace period to fifteen seconds. foreground daemons (like the subscription daemon) and the network engine will now also obey it
• added a 'simulate wake from sleep' debug action to better test the sleep-wake detection code
• improved my custom statictext class to auto-wrap text without flickering
• used this new autowrapping to improve wrapping and layout of popup message texts
• replaced all other st wrapping with this new code
• wrote a little helper function to better dedupe lists in future
• did a bunch of refactoring to neaten some long common func names
• deleted some old unused code

• version 312

• converted much of the increasingly complicated tag import options to a new sub-object that simplifies a lot of code and makes things easier to serialise and update in future
• tag import options now allows you to set whether tags should be applied to new files/already in inbox/already in archive, much like the file import options' 'presentation' checkboxes
• tag import options now allows you to set whether tags should be filtered to only those that already have a non-zero current count on that tag service (i.e. only tags that 'already exist')
• tag import options now has two 'fetch if already in db' checkboxes--for url and hash matches separately (the hash stuff is advanced, but this new distinction will be of increasing use in the future)
• tag import options now applies sibling and parent collapse/expansion before tag filtering, which will improve filtering accuracy (so if you only want creator tags, and a sibling would convert an unnamespaced tag up to a creator, you will now get it)
• the old 'all namespaces' checkbox is now removed from some 'defaults' areas, and any default tag import options that had it checked will instead get 'get all' checked as they update
• caught up the ui and importer code to deal with these tag import option changes
• improved how some 'should download metadata/file' pre-import checking works
• moved all complicated 'let's derive some specific tag import options from these defaults' code to the tag import options object itself
• wrote some decent unit tests for tag import options
• wrote a parser for deviant art. it has source time now, and falls back to the embedded image if the artist has disabled high-res downloading. if it finds a mature content click-through (due to not being logged in), it will now veto and set 'ignored' status (we will revisit this and get high quality nsfw from DA when the login manager works.)
• if a check timings object (like for a subscription or watcher) has a 'static' check interval, it will now apply that period to the 'last next check time', so if you set it to check every seven days, starting on Wednesday night, it will now repeatedly check on Wed night, not creep forward a few minutes/hours every time due to applying time to the 'last check completed time'. if you were hit by this, hit 'check now' to reset your next check time to now
• the multiple watcher now sorts by status by default, and blank status now sorts below DEAD and the others, so you should get a neat subject-alphabetical sort grouped by interesting-status-first now right from the start
• added 'clear all multiwatcher highlights' to 'pages' menu
• fixed a typo bug in the new multiple watcher options-setting buttons
• added 'retry ignored' buttons to edit subscription/subscriptions panels, so you can retry pixiv manga pages en masse
• added 'always show iso time' checkbox to options->gui, which will stop replacing some recent timestamps with '5 minutes ago'
• fixed an index-selection issue with compound formulae in the new parsing system
• fixed a file progress count status error in subscriptions that was reducing progress rather than increasing range when the post urls created new urls
• improved error handling when a file import object's index can't be figured out in the file import list
• to clear up confusion, the crash recovery dialog now puts the name of the default session it would like to try loading on its ok button
• the new listctrl class will now always sort strings in a case-insensitive way
• wrote a simple 'fetch a url' debug routine for the help->debug menu that will help better diagnose various parse and login issues in future
• fixed an issue where the autocomplete dropdown float window could sometimes get stuck in 'show float' mode when it spawned a new window while having focus (usually due to activating/right-clicking a tag in the list and hitting 'show in new page'). any other instances of the dropdown getting stuck on should now also be fixable/fixed with a simple page change
• improved how some checkbox menu data is handled
• started work on a gallery log, which will record and action gallery urls in the new system much like the file import status area
• significant refactoring of file import objects--there are now 'file seeds' and 'gallery seeds'
• added an interesting new 'alterate' duplicate example to duplicates help
• brushed off and added some more examples to duplicates help, thanks to users for the contributions
• misc refactoring

• version 311

• wrote a new parser that muddles its way through pixiv's new dynamic javascript layout. it seems to get everything working again. it gets tags in kanji, although the unnamespaced pixiv tags remain low quality, and you may wish to just not parse them at all anyway
• fixed some misc parser text handling, unicode conversion etc...
• the new pixiv parser has a 'page' tag stub that should inform tag import options in the old downloader
• the multiple watcher now remembers the highlighted watcher through a session restart
• the multiple watcher now shows the highlighted watcher's url up top
• the multiple watcher now has checker, file import, and tag import options, which it will assign to all new watchers it creates
• the multiple watcher now has a 'set options to watchers' button that will force-set the current options to all the selected watchers
• the multiple watcher now has an 'added' column with watcher creation time listed. storing this creation time is new, so any existing watchers will get a new creation time of their next load time, but it is remembered henceforth. the listctrl here is now pretty crushed for width, so maybe we'll rejigger some stuff here
• watchers added to a multiple watcher will now have a status of 'just added' for five seconds
• watchers that are added to a multiple watcher that is already watching them will now have the status of 'already watching' for five seconds
• the multiple watcher list now has a much taller minimum height--layout here is another work in progress
• fixed the inkbunny parser (and a related tweak to the inkbunny url class)--it now uses the new 'multiple-file-per-post' import object generation to actually walk through the pages of the mini-gallery (which for inkbunny have -p2- suffixes on the url) to fetch only the correct files and url-associate them neatly
• tag import options now has a 'get all tags' checkbox, which can override the normal namespace checkboxes. it gets all tags, even those with namespaces not listed, which happens for several reasons in the new download system. (eventually, the namespace list may be replaced with a slightly different system)
• watcher tag import options no longer list 'filename' under their namespace checkboxes--they just have this 'get all tags', which works for everything (so watching yiff.party pages should now get tags)
• simplified and sped up similar files search at the db level
• sped up some ratings search code
• generalised some common file search optimisations, meaning they now apply in more situations and can take advantage of some other speed-ups:
• similar files system predicate is now faster
• inclusive ratings searches are now faster
• duplicate relationship count searches with non-zero-inclusive count are now faster
• removed some clumsy old ratings search optimisation code
• exporting serialised objects as pngs is a bit easier--now, it displays current export path better, will remember the last export location used, and for single png exports will pre-fill the filename and 'title' value with a reasonable default
• the content parser, page parser, and url class listctrls now accept serialised png files when drag and dropped!
• the simple downloader should recover and continue better from malformed urls during a page parse
• the url downloader should now recover better from various situations where it cannot not derive some tag import options (including urls with a 'file' url class, such as 4ch/8ch direct file links)
• parse test results will now state the priority value of urls
• gave the 'updating' section of help a pass and wrote a little more on how to do a big-version-gap update
• when a new multi-file import object inserts its child file import objects while being looked at in the ui, the listctrl should now correctly refresh the displayed indices
• subscriptions will now wait up to 90s for bandwidth (was 30s before, I think) before quitting, which should avoid a few more early-quit events
• cleaned up some server decompression bomb testing
• users with admin-level accounts can now upload decompression bombs to file repositories, better options on this will be avaliable in future
• the manage urls dialog will now OK on the same 'manage_file_urls' shortcut action that can open it (like manage tags and ratings already do)
• fixed the string converter for new file lookup parsing scripts
• started work on some in-the-background mass file reparsing, but I want to get some nicer ui going before I pull the trigger on any of it
• file reparsing now repopulates the table for md5, sha1, and sha512 hashes if they are missing
• improved some ffmpeg error parsing
• moved from basic list to a pop-faster collections.deque for importable path parsing and duplicate search branch regen
• added a BUGFIX option to options->gui that forces minimum width for popup messages in the continuing attempt to deal with some funny fit/layout calculation in certain Linux WMs
• fixed how some 'unrepairable db' error messages are displayed in Linux systems
• cleaned up a ton of old tuple-stripping code from the db
• updated to new sqlite for windows build
• misc improvements

• version 310

• updated the inkbunny file page url class to acknowledge that inbunny pages can have multiple files
• updated the inkbunny file page parser to handle multiple file urls (although they may be out of order and possibly sometimes include the artist profile image--this was not super easy)
• added a parser for twitter tweets (only images supported atm, but it can handle multiple!) (hence tweet drag and drop now works!)
• updated the artstation file page url class to redirect to a new api url class
• wrote an artstation file page parser that also handles multiple file urls
• updated/added pixiv file page, manga page, and mange_big url classes
• updated pixiv file page parsers to be ok with manga links
• wrote parsers for pixiv manga and manga_big pages to fetch manga files (with page tags)!
• file import objects can now create semi-duplicate children for multi-file post urls and insert them just after themselves in the file import queue.
• file import objects can now receive and remember referral urls. this referral url is associated with the file if appropriate. the watcher and simple downloader now uses this in addition to the multi-file post system
• jumbled around some parameters and merged the two new file import url commands (import 'file' vs import 'post') into one single simple 'work on this url, thanks' call that is now used across the program
• the parsing system's 'content parser' no longer fetches file urls and post urls, but 'download urls' and 'source urls'. this helps some pipeline logic and also lets post urls be download urls
• when file import objects parse post urls as the urls to download, it now creates 1-n new import objects, just like if multiple file urls.
• improved some file import object file association code
• the new parsing system will de-dupe parsed urls
• refactored the 'seed' code, which handles all basic file import objects, to the new ClientImportSeeds.py
• added a new string transformation type, 'integer addition', for shifting page number tags up and down
• fixed thumbnail generation for some videos that failed to do the new x%-in generation--it reverts more reliably just to the old frame 0 method
• file reparsing popup now has a stop button
• fixed an issue where extremely thin or wide (ratio > 200:1) images would not generate a full-size or resized thumbnail
• the file reparsing/re-thumbnailing now reports errors better (including with full path) and does not abandon the larger job as it works
• misc thumbnail generation code improvements
• improved some thumbnail and file regeneration/moving code when the existing file has read-only status
• the multiple watcher now has a 'check now' button
• added a checkbox to options->gui that will put new notebook page tabs on the left
• for all file download network jobs working in the new download system, the file import options for min size, max size, and max size (gifs) are now applied _during the download_! if the server tells the client the exact file size in the response headers, it will test max and min size before the content is actually downloaded--otherwise, it will test the max size as it downloads. if the server clearly says the file is a gif, the max gif size rules will also be tested in the same way
• cleaned up some bandwidth announcement code--now, if bandwidth is due in less time than override time, that will now correctly be the status text
• the bandwidth status no longer says 'in in' typo
• fixed up some tag repair code from last week
• the 'print garbage' debug function now dumps a whole bunch more data to the log
• the thumbnail cache should now be a bit more stoic about missing repository thumbnails--it should now just present the hydrus default backup without error popup spam
• the repository thumbnail sync will now get as thumbs in blocks as high as 10k at a time, rather than the old 100
• hydrus network requests no longer generate web domain network contexts (and so won't have a default one-request-per-second bandwidth limit and should stream through thumbnails a bit faster)
• hydrus network services are now willing to wait longer for bandwidth, so big thumbnail queues should keep working even if other bottlenecks pause them for a bit
• hydrus network services will no longer sometimes have double-sync popups if synced from the advanced 'sync now' button in review services
• changed the default global 'stop-accidents' bandwidth rule of 120rqs per minute to 512MB per minute. this only affects new users, but users trying to sync to large file repos might like to make a similar change manually
• doing giant full file delete (i.e. purge from trash) jobs should now be a bit gentler on the gui
• improved how the client deletes paths, clarifying in the code when and when not to allow recycle (usually disabled for thumb disposal)
• switched the hacky text widgets on the popup system to a newer object. seems to still render ok, so lets see if it fixes some unusual layout issues some users have seen
• if the temp folder cannot be created on boot, the client will continue anyway
• fixed some url-domain text handling in db storage that was also breaking v309 update for some users
• fixed some additional domain generation error handling at the db level
• the list of url classes in the system:url panel is now the list of all url classes that are considered associable (before, it was file and post urls)
• if a url class now api-links to itself or otherwise forms a loop with n other api url classes, the client will now throw an error (rather than lock up in an infinite loop!)
• in the parsing ui, tag parse test results are now cleaned before being displayed
• fixed misc url matching error reporting bug
• when consulting the current file limit, the gallery page downloader will now try, when it has that number, to consult the total number of urls found it the current search (old behaviour is to only consult the number of _new_ urls, which lead to some bad edge-case workflows)
• misc refactoring

• version 309

• wrote a fix for the tumblr GDPR issue under _network->DEBUG: misc->do tumblr GDPR click-through_. you will also get a popup about this on update
• the tumblr downloader will try to detect the GDPR problem and present a similar popup guiding you to the GDPR click-through solution
• the client and server now generate video (but not gif yet) thumbnails 35% in by default. the client can now change this percentage value under options->media. this was highly requested and was being put off for a longer rewrite, but I figured out a simple way to hack it in. please let me know if you get failures
• on adding a parent, all files with the child tag will now also get all applicable grandparents (with no limit on recursive generations and dealing with accidental loops)
• on adding a sibling, all files with any of the siblings will now also get all applicable parents and grandparents for the whole group. a maintenance call to retroactively fill in the sibling/parent gaps that are now filled will also come soon
• this logic still does not apply in cross-service situations, which _will_ likely have to wait for a big data/gui overhaul and us figuring out what we actually want here
• added a simple pause/play button to the multiple watcher
• if the multiple watcher is set to catch watchable url drag and drop events and the current page is a multiple watcher, this current page will catch those new urls (as opposed to the _leftmost_ multiple watcher)
• improved some thread unpause logic which was failing to lock pause during 404 status
• the multiple watcher should now ignore case when it sorts by subject
• added url class and file page parser for inkbunny (so this site is now supported in drag and drop!). it fetches creator tag, some artist-made unnamespaced tags, source time, and md5
• added file page parser for gelbooru 0.2.0, which by default works for rule34.xxx, tbib, xbooru but certainly should work for a bunch of others. it fetches source time and source url
• html formula parsing rules can now additionally test the tag 'string' using a standard StringMatch object. this greatly helps to parse otherwise indistinguishable 'a' tags that have string 'Original image' and so on
• the 'have I seen this url's file before?' pre-import test is now much more strict and will cause fewer accidental false-positive 'already in db'/'deleted' results:
• the url pre-import test now does not trust source urls if they do not have a url class
• the url pre-import test now no longer trusts urls that are supposed to only be mapped to one file but are actually mapped to multiple
• this url pre-import test now treats url-classless original post urls and intended file urls with a special level of trust
• urls are now stored in the db in a more powerful and in-future easily searchable way--your db will take a moment to convert to the new format on update
• did some prep work for multi-file post urls (like pixiv manga) but did not have time to finish it
• the filename tagging options panel (in the 'add tags based on filename' of file import dialog and import folder dialog) now updates its tags/list 0.5s after the last change event, which means typing on a giant list will not cause megalag
• improved stability of some client-screen coordinate conversion
• misc bmp handling stability improvements
• improved some parsing ui stability when example data gets set after the dialog is closed
• improved some misc dialog close stability
• converted all but one final ui update timer to the new job scheduling system
• there are still problems with linux stability--I will continue to work on it
• an ugly (but basically harmless) shutdown exception sometimes caused by Animations being a bit slow on deleting their underlying bmps _should_ be fixed
• the export files dialog now generates its paths in sort order, meaning (1), (2) de-dupe filename suffixes should now be generated nicely in order
• the network domain manager should now always chase API URL links to get the right parser
• made some 'the db is broke, let's try to fix it' tag recovery code more forgiving
• misc improvements to some media indexing backend, which may fix some unusual session ghost files
• fixed the 'sure it is ok to close this importing page' dialog to also veto on a 'cancel' event, rather than just a 'no'
• added a guide to database_migration.html on how to move the db from just an HDD to straddle both an SSD and HDD.
• cleaned up the help->debug menu a bunch
• added run fast/slow memory maintenance calls to help->debug->data actions
• misc cleanup

• version 308

• the multiple watcher will now discard new urls if it is already watching them
• the multiple watcher will list x/y progress as just 'x' if x==y (making it easier to scan the list)
• the multiple watcher now lists a couple of 'total' summary lines on its ui--the top lists total number of watchers and queue progress, the bottom lists the usual '23 successful, 3 deleted' line, but summed for all watchers
• the multiple watcher will now warn you if you try to remove the highlit, alive or un-caught-up watchers
• the multiple watcher will now resort if the thread subject (or rather, any data in the current sort column) changes (which usually happens right after it is added, when you see it change from from 'unknown subject' to 'mlp is kino')
• fixed an issue where multiple watchers were not unscheduling down their update job correctly on page close
• the booru selector in the edit subscription panel should now be in the tab traversal order for keyboard/automated focusing tasks
• the boorus in that selector are now alphabetised
• tag import options namespaces are now alphabetised
• removed/renamed pretty much all references to 'thread' in the watcher code and ui presentation, since it can now do a bunch of other stuff. it is now just the 'watcher' and the 'multiple watcher'
• deleted a bunch of old static thread watcher and page of images code from the old downloading system
• added an experimental 'compact' button to advanced mode users' manage subscriptions panels. this removes urls from the selected subscriptions' caches that are no longer useful, keeping their load/save snappy. this is still in testing--be careful with it!
• the hydrus splash screen now has a bare frame caption and will appear in the taskbar--which helps with some alt-tab and 'where the hell did it go?' stuff if you need to enter a password
• wrote five 'reasonable defaults' buttons for the 'check timings' options panel for quick entry for different thread/subscription scenarios.
• added a checkbox to this panel that will swap the reactive options with a simpler single checkbox
• also clarified/fleshed out the help button on this panel
• fixed an important source of program instability related to page alive/dead status checking that was inadvertantly talking subtly to the main gui frame even on non ui threads
• improved how some 'page is closed but not destroyed' test logic for pages inside a closed-but-not-destroyed notebook
• fixed another small place where the db was talking to the main gui object about status bar updates in a potentially unstable way
• fixed another small place where the foreground daemons were talking to the main gui frame in a trivial but potentially unstable way
• played around with some taglist sizer and layout settings
• the gallery and simple download pages are now a little shorter--the pause and cancel buttons are now just to the right of the status texts, rather than on their own row beneath the network job controls.
• the various bandwidth-overriding network jobs in the download system--like gallery page downloading--now wait 30s before overriding their bandwidth. hence these jobs will now obey the usual bandwidth rules up to a point
• the simple downloader also obeys the usual bandwidth rules for 30s but no longer has a static wait, so it can run much faster in certain situations
• network jobs that will override bandwidth in the future will now report that countdown in their status texts
• fixed a bug in the old booru code that meant some boorus were superfluously requesting the 0th indexed page of a gallery more frequently than needed in order to reestablish a 'page size' cache. this value is now cached globally and will be replaced by a completely different system in the new gallery downloader
• added a decent tooltip to the 'gallery fixed delay' widgets in the options->downloading panel
• the autocomplete input should clear itself after a 'broadcast' event a bit quicker and stop some dupe inputs in certain edge cases
• the tumblr url class now recognises that tumblr posts can have multiple files, which helps some source url lookup logic
• added a url class for artstation file pages
• the primary file import url (the one listed in the file import list) will now correctly not associate with the resulting file if its url class is so set
• all the import objects now have much lower idle CPU time and thread needs and start in slightly offset times, smoothing out the thread count spikes
• all the import objects will now respond quickly to changes to the underlying file import cache (like right-click->try again events)
• the new job scheduling system now uses two queues--fast and slow, in order to reduce some resort/insert overhead
• a couple more improvements to the new job scheduling system to smooth out spikes
• if the temporary path override does not exist, the client will now compain with spammy popup messages and fall back to the default
• if the temporary path override does not exist or is not writeable-to on options dialog ok, a veto exception will be raised
• refactored the watcher and multiple watcher to their own file, ClientImportWatchers
• misc fixes

• version 307

• wrote a gelbooru 0.2.5 (which matches gelbooru itself) parser in the new system. it now has some more redundancy and produces md5 hash and source urls
• fixed the e621 parser for flash files
• manage tags is now a notebook rather than a listbook!
• a problem where OS X was displaying the wrong label in the manage tags listbook is now fixed (as it no longer uses the buggy old listbook)
• improved autocomplete focus setting in OS X in general
• moved all paged importer loops to the new job scheduling system, which will massively cut down on idle time thread count (and some idle CPU usage) on clients with a bunch of import pages open
• the main thread pool (which the job scheduling system uses) can now temporarily grow much larger (200 threads) when it gets hammered
• recalibrated some thread watcher code to deal with being embedded in a larger object better (and sharing a page with other watcher importers)
• finished the first version of a multiple watcher. it is ugly and only for advanced users for now--please check release post for more information
• added 'use the multiple watcher on DnD events' to options->downloading, which will send thread url drag and drops straight to a new/existing multiple watcher!
• gui session load now 'shows' the first page but loads everything to the right in the background, which saves some CPU and memory for large sessions. this means it now starts focused on the leftmost page rather than the right--see if you like it or not
• 'clear and load session' now fully deletes old pages and takes a little break between the clear and load step to make sure old large sessions are deleted tidily before loading the new stuff
• cleaned up the new leaner popup message display code to react better to (and re-layout) sub-changes in its popups (so now when a subscription popup spawns a new progress gauge, it _should_ be positioned in the right place immediately)
• if an import folder/subscription publishes files to a page while the main gui is minimised (and the action would result in a new page creation), the page add will be delayed until the gui is no longer minimised (new page layout in this case was broken due to minimised parent frame)
• the edit subscription query panel will now spawn with the query text input focused
• added a patch to ensure new booru import pages get query input focus after a second if the focus attempt on init failed (this will be fixed better in the new downloader system)
• restored the 'clear deleted files record' button to the 'combined local files' service on _review services_, which was previously hidden during a significant service rewrite.
• if critical master tables are missing on boot, the client will warn and provide info and then abandon the boot
• if cache similar files tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway (this replaces some old repair code that wasn't always kicking in at a good time)
• if cache autocomplete tables are missing on boot, the client will warn and provide info and offer to recreate and repopulate them
• if mappings tables are missing on boot, the client will warn and provide info and offer to recreate them empty and try to boot the client anyway
• the emergency 'repair missing file locations' panel now runs in a more stable way
• this emergency dialog will now also recognise 'I fixed all the file paths but cannot do the thumbnail paths' situations and will present the same 'ok, make sure you run regen thumbs after boot' dialog as if all the missing paths were thumbnail related
• you can now turn off x/y page name import progress under options->gui
• fixed an issue where tags that begin with ':' like ':p' were not getting through the new importing system
• fixed an issue where tags that begin with ':' like ':p' could sometimes get prepended extra escape characters and end up looking like ':::p'
• fixed sibling tag searching (e.g. if a->b exists and you search for b, results with a but not b should also appear), which the recent tag search optimisation accidentally disabled
• whole bunch of refactoring and cleanup of ClientData and ClientGUICommon
• cleaned up some spammy splash text status setting that was flooding some debug info with useless garbage
• gave getting_started_tags.html's tag repo section and access_keys.html a pass, updating the ancient screenshot and linking advanced users to the latest QuickSync location
• some misc help updates
• misc autocomplete logic improvements
• misc ui display fixes
• misc wx destroy code stability improvement
• updated ffmpeg for windows builds to the new 4.0 release
• updated sqlite for windows builds

• version 306

• the file import status list now has 'open selected import files in a new page', which should show up where it is possible. this is a bit prototype and ugly--it'll show _all_ files, including in-trash and permanently deleted (which will show up with the hydrus thumbnail)
• the file import status list now prefixes the already in db/deleted notes with 'url' or the hash type that lead to the recognition
• these redundant/deleted notes now also propagate up from 'during import' recognition phase as well
• the 'delete seeds of type x' entries on the file import status button's right-click menu are now split into three smaller individual tyes and are more explicit about exactly which status types they will remove
• like import folders, subscriptions can now optionally publish their files to pages as well as popup buttons. also, subscriptions can optionally publish their files separately for each query instead of all merged together
• sped up multiple tag queries significantly
• sped up simple (file size, mime, etc...) system predicate queries that also include a tag/namespace/wildcard predicate significantly
• added a pixiv parser that pulls the japanese tags to the defaults--users can switch to this if they prefer under network->manage url class links
• fixed the 4chan parser to get part of comment as backup subject/page title
• removed the 'newgrounds' entry from the normal gallery page creation ui, as the basic gallery parser no longer works due to a dynamic loading change on their end. I hope to have it back with the new gallery parsing system I will soon be writing
• the edit url classes panel now has a little text box to put in example urls and see which class, if any, that they match to
• improved layout of edit url class links panel
• all url types are now displayable in the media viewer--only post url classes are default on
• the new (x/y) import page page_name progress count is now updated on all alterations to this value (previously, this was not updating when a user interacted with the import queue, only when the natural downloader loop cycled)
• added 'can produce multiple files' option to post url url classes, which informs client url-checking logic whether the url can be relied upon for 'already in db/deleted' calculations
• the pixiv file page url class now has 'can produce multiple files' checked, meaning some bad pixiv url association logic due to other sites referencing it as a source url is now fixed
• added a 'twitter tweet' url class, which is also a 'can produce multiple files' post url
• added a 'sync known urls?' action choice to the duplicate merge options panel, which governs whether urls should be copied from worse to better or in both directions
• gave the edit duplicate merge options panel a layout pass
• the edit duplicate merge options panel will now disable pointless/over-complicated choices on non-custom actions, let me know if this is a pain for your workflow
• added a 'manual' web browser path override to the 'files and trash' options panel, which fixes the new share->open->in web browser option for Windows and also fixes some #anchor link propagation
• consolidated all URL/Path web browser launching code to one location
• 'open in web browser' is now available for non-advanced_mode users and the 'open' submenu of the share menu is available in the preview window and the media viewer
• fixed a bug that was causing import folders to publish incorrect file identifiers, which was poisoning popup buttons and import page destinations
• gui sessions that fail to load a page will recover and continue to attempt loading the rest of their pages. some popups detailing the page's serialised data and error will be presented
• gui sessions that fail to save a page will recover and continue to attempt saving the rest of their pages. some popups detailsing the page's rough info and error will be presented
• the core controller inside all media pages will now present itself in a more beautiful way when asked to dump itself to a log (which should beautify the above save error a bit)
• wrote a subsidiary database->check->just repo update files that tests integrity of only repository update files
• fixed an issue where default tag import options were sometimes not being saved from the new dialog in the networking menu
• wrote a couple of layers of bad tag protection to help the new downloader deal with some occasional bad output from the old downloader
• network jobs can now reattempt connection attempts up to three times on POST requests (if you ever got inexplicable immediate 'could not connect' errors on repository uploads, this should now be fixed)
• replaced some archaic misc old import code with the new system, cleaning up a bunch of stuff and making space for further refactoring along the way
• fixed tags blacklist not being inherited in the old (through options dialog) system
• improved some invalid domain error handling
• fixed an animation update issue that would pause naturally updating controls on non-main-gui frames when there were no regular media pages open on the main gui
• added a BUGFIX option to 'files and trash' option page to override the default temp path for almost all client temp path requests
• the minimum value for the 'vacuum period' in maintenance and processing options is now 28 days. the control also has a little explanatory tooltip
• the 'try to auto-link url classes and parsers' function now always preferences parsers alphabetically
• fixed a typo in the string transformations prettyfication code that incorrectly summarised 'take the last x characters' as the opposite
• misc fix to file hash generation and status checking code
• the 'export tags to .txt files' checkbox on the export files panel will no longer bother you with a dialog as you uncheck it
• wrote some code to make it easier and more fool-proof to update the domain manager with new url classes and parsers on my end
• improved some popup message manager ok-to-alter-ui logic when the main ui is minimised and so on
• fixed some potential crash conditions (affecting linux mostly, seems like) in the service credential testing and access key fetching ui code
• fixed a bug when 'stopping' a gallery parse during a long error pause (like when it holds on '404')
• sped up some old set intersection code
• some import file presentation refactoring
• some url content application pipeline cleanup
• misc cleanup

• version 305

• fixed the pixiv url class, which was unintentionally removing a parameter
• wrote a pixiv parser in the new system, fixing a whole bunch of tag parsing along the way, and also parses 'source time'! by default, pixiv now fetches the translated/romaji versions of tags
• finished a safebooru parser that also handles source time and source urls
• finished an e621 parser that also handles source time and source urls and hash!
• wrote a danbooru parser that also handles source time and source urls and hash!
• as a result, danbooru, safebooru, e621, and pixiv post urls are now drag-and-droppable onto the client!
• finished up a full yiff.party watcher from another contribution by @cuddlebear on the discord, including url classes and a full parser, meaning yiff.party artist urls are now droppable onto the client and will spawn thread watchers (I expect to add some kind of subscription support for watchers in the future). inline links are supported, and there is source time and limited filename: and hash parsing
• fixed some thread watcher tag association problems in the new system
• when pages put an (x) number after their name for number of files, they will now also put an (x/y) import total (if appropriate and not complete) as well. this also sums up through page of pages!
• if a call to close a page of pages or the application would present more than one page's 'I am still importing' complaint, all the complaints are now summarised in a single yes/no dialog
• url downloader pages now run a 'are you sure you want to close this page' when their import queues are unfinished and unpaused
• if the subscriptions for 'manage subscriptions' take more than a second to load, a popup will come up with load progress. the popup is cancellable
• added a prototype 'open in web browser' to the thumbnail right-click share menu. it will only appear in windows if you are in advanced mode, as atm it mostly just launches the file in the default program, not browser. I will keep working on this
• harmonised more old download code into a single location in the new system
• created a neater network job factory system for generalised network requests at the import job level
• created a neater presentation context factory system for generalised and reliable set/clear network job ui presentation at the import job level
• moved the new downloader simple-file-download-and-import to the new file object and harmonised all downloader code to call this single location where possible
• did the same thing with download-post-and-then-fetch-tags-and-file job and added hooks for in the subscription and gallery downloader loops (where a parser match for the url is found)
• the simple downloader and urls downloader now use 'downloader instance' network jobs, so they obey a couple more bandwidth rules
• harmonised how imported media is then presented to pages as thumbnails through the new main import object
• the new post downloader sets up referral urls for the file download (which are needed for pixiv and anything else picky) automatically
• improved file download/import error reporting a little
• entering an invalid regex phrase in the stringmatch panel (as happens all the time as you type it) will now present the error in the status area rather than spamming popups
• fixed a bug in the new parsing gui that was prohibiting editing a date decode string transformation
• fixed enabling of additional date decode controls in the string transformations edit panel
• added a hyperlink to date decoding controls that links to python date decoding explainer
• if a source time in the new parsing system suggests a time in the future, it will now clip to 30s ago
• misc downloader refactoring and cleanup
• fixed an issue where new file lookup scripts were initialising with bad string transformation rows and breaking the whole dialog in subsequent calls, fugg
• hid the 'find similar files' menu entry for images that have duration (gifs and apngs), which are not yet supported
• added 'flip_debug_force_idle_mode_do_not_set_this' to main_gui shortcut set. only set it if you are an advanced user and prepared for the potential consequences
• silenced a problem with newgrounds gallery parser--will fix it properly next week
• fixed some old busted unit test code
• rejiggered some thumb dupe menu entry layout

• version 304

• renamed the new 'tagcensor' object to 'tagfilter' (since it will end up doing a bunch of non-censoring jobs) and refactored it into clienttags
• attached a tag filter object to all tag import options to act as a tag blacklist. all tags that go through the import pipeline (except for a couple of old legacy instances) are now checked against the blacklist, and if a bad tag is found, the file vetoes! tag import options has some new ui to handle this and background code to deal with inheritance from defaults and so on
• new file import urls that have url classes, no matter their source, are now normalised!
• all new file import urls are now tested against both the original and normalised version of the url, so even though previously parsed urls remain un-normalised, new urls that are pre-normalised the same will not count as new! -fingers crossed-
• on update, the db will get normalised copies of all existing urls. this means many files will now have two versions of its urls--some ui to collapse everything down to only the normalised version (after some human eyes have passed in front of this big change) will come in the coming weeks
• some sites where normalisation is a consistent problem for later redownloads (like e621, which appends 'preview' tags to the post url) _should_ now be caught reliably!
• the 'allow subdomains' on edit url class panel is now named 'match subdomains' and has a tooltip to better explain how it works
• 'keep subdomains' is now 'keep matched subdomains' and has a tooltip as well
• the 'keep matched subdomains' enabled behaviour (and some normalisation calculation) is now additionally governed by the 'associate url with files' value and api url conversion info rather than just 'match subdomains' and raw url type
• fixed an issue that was stopping the 'associate url with files' option sticking in edit url class panel
• edit url matches now resorts after an add or edit action
• all listctrls with a wrapper panel now resort after an import from clipboard, png, or defaults call
• url matches now match against www*. versions of their domain regardless of 'match subdomains' settings
• updated xbooru url classes to prefer https
• the manage url class links panel now has a 'clear' button to clear a url_class->parser link
• introduced three new simple downloader parsers for yiff.party, thanks to @cuddlebear on discord for the submission
• the old 'uninteresting mime' status has been expanded to a wider 'vetoed' status to represent all file imports that are abandoned without a particular error (e.g. tag blacklist, wrong filesize or resolution)
• the import system now reports the total of 'num vetoed' as 'num ignored' in its summary statements
• it now also reports 'num skipped'
• the 'num successful' and 'num already in db' are now folded more neatly together in import cache summary statements
• file downloads that are cancelled will now set a 'veto' state rather than a 'skip' state
• improved file import exception handling across the board
• improved how single-file-result parsing vetoes propagate up to the file import status cache
• 404 network errors will now provide a 'veto' status rather than an 'error'
• vetoes will not count as errors when deciding whether a subscription should be abandoned early (so a bunch of decomp bombs or 404s will no longer stutter a subscription!)
• misc fixes and improvements to the new download stuff
• wrote a new parsing cache that saves a lot of work in the new parsing system
• improved the 'is this url known?' test to better deal with situations where all the given urls are galleries or unrecognised--a better aggregate of file status is formed, and 'already in db'/'deleted' statuses will apply if there is no evidence otherwise (the dev got the new logic for this from a legit nightmare about urls downloading over and over, so let's hope it works out)
• the 'is this url known?' logic also recovers from 1->n url->hash relationships where it does not expect them, trying to find 'already in db' hashes over 'deleted' ones
• to clear up some ambiguity, galleries or subscriptions now give a different 'checking in x seconds' status when waiting on the first page of a query
• the 'noneablebytescontrol', as seen in edit file import options, will now correctly disable/enable its bytes sub-control when it is none'ed
• a persistent issue with the new network engine sometimes failing to correctly error after certain broken connections (the computer going to sleep mid-download was a common cause here) should now be recovered from and the connection naturally reattempted
• added three new shortcuts to the 'main_gui' shortcut set that allow for opening a new 'urls', 'simple', or 'thread watcher' downloader page
• added two more shortcuts to 'main_gui' for new 'page of pages' and 'duplicate filter page'
• moved some old 'new page' menu code to the new application command system
• added numerous 'duplicates' shortcuts to the 'media' shortcut set that will work on selections of thumbnails
• the thumbnail duplicates menu actions now go through the new application command system
• fixed an issue where the current tag parents caches was not refreshing when notified
• inputting a short invalid syntactic input on a 'read' tag autocomplete such as '-' will now clear the system predicates list--system preds should now only show on a completely empty input
• fixed an issue where certain combinations of 'remove a tag, then re-add it' nullipotent actions in a single manage tags dialog transaction were not applying reliably (sometimes, the subsequent mirror action was not occuring due to a processing re-order optimisation at the db level)
• made some animation code a little safer and quieter as a test for some users who were getting blitzed with some deadwindow error spam in certain situations--let's see if this changes anything
• replaced all the em dashes in the help with double hyphens as github pages was rendering them wrong
• added CrystalDiskInfo recommendation to 'help my db is broke.txt'
• misc cleanup

• version 303

• file post url classes can now be linked to parsers!
• the 'raw url' downloader is now just the 'url' downloader. if a dropped url is matched as a 'file post' url and links to a parser, it will download it with that parser and default tag import options
• url drag and drop now recognises file post urls that have linked parsers and will send them on to a url import page, just like 'raw' urls! they will use the new parsing system to parse tags and known urls and all that
• wrote a way for the new download system to store and edit default tag import options--this can now be found under network->manage default tag import options, although it does not do much yet. eventually the old options will be transferred here, and the different downloaders will consult it more
• refactored and cleaned up how some default tag import option calculation is done
• the fixed 5 second wait time between gallery page fetches is now editable for downloader pages and subscriptions under options->downloading. subscriptions default remains 5s, gallery default is now 15s
• the gallery downloader will present its 'x urls found so far' string in a more uniform and reactive way. it will also count down until the next page fetch
• subscriptions will now similarly react quicker while the gallery parse step is ongoing, and it will count down to the next page fetch
• the manage subscriptions dialog will now note in BIG RED TEXT if subscriptions are currently globally paused
• subscriptions now consume less parallel-timeslot overhead, meaning they sync much faster when they have no work to do (which is most of the time)
• separating subscriptions is now more sophisticated--if you wish, you can now choose just to separate a subset of the large subs's queries, and if you do so, you can decide whether to merge the extracted subs into a new larger sub or just have them as individuals. also, you will be asked for name/base_name for what to name the new subs
• since separating subs is more complicated, the button is only enabled when a single large sub is selected. please do your separating one sub at a time!
• merging subs will now ask if you want to rename the merged subs as it goes
• refactored a giant heap of duplicate import code into single locations in the new file import object--things like applying tags and generating pre-import status are all now done in one place. a number of weird behaviours (like not applying known url associations in certain circumstances) should now be a lot better and unified
• the way the db and other import testers handle whether a pending import is new or unknown is simplified
• optimised the way the importers figure out whether to display a new import
• the 'has the client seen this url before?' test used to figure out whether to download a file now explicitly ignores anything it isn't certain is a single-file File URL or Post URL (according to current url classes). associating 'gallery' known urls is no longer catastrophic to this system
• the simple downloader now associates the entered url with the files subsequently imported
• file import objects working in the new parsing system are aware of the new 'should this url be associated' option and will be given gallery urls to hold on to as appropriate
• url classes now have an explicit option as to whether they should be associated with files imported from them, which is not appropriate for dynamic CDN File URLs but is appropriate for multi-image tweet links (which will likely be Gallery URLs in the new system), for instance
• general importer code cleanup across the board
• threw together some early ui to show current jobs in the network engine under network->review network jobs
• popup messages will now try to set a max width of roughly 56 characters wide rather than 400 pixels. this value is editable under options->gui
• updated the listctrl in the export files frame and the way this frame generates and stores export filenames. let's see if linux users who had crashes with this have better luck
• refactored stuff out of the bloated ClientImporting file into seperate new files
• refactored client network bandwidth and session code out to their own files as well
• refactored client network objects to its own file
• cleaned up some open externally debug code linked to callto report mode
• a common grid layout used across the program will copy control tooltips to the string on the left
• removed the 'downloader' network context type, which was never fully introduced and wasn't turning out to offer much in the new bandwidth tracking system

• version 302

• improved how simple downloader parsing formulae are stored and passed around
• the edit simple downloader parsing formulae panel now has an 'add defaults' menu button
• the simple downloader formula edit panel handles its buttons better and can now do import from/export to clipboard/png and reimport from the defaults, and fixes duplicate names automatically
• simple downloaders now remember their current parser through a session save/load
• simple downloaders will set a new default parser selection for new simple downloader pages on any choice event
• renamed all the default simple downloader parsers to more clearly explain what they do
• added 'all files linked by images in page' to the simple downloader, which does the old page of images' behaviour, and is hence pretty decent for most imageboards
• added a simple mewch thread html parse formula to the simple downloader
• added a very simple webmshare parse formula to the simple downloader
• added 'imgur image album' to the simple downloader, which will fetch the first 'page' of results from an image album. full parsing will have to wait for the gallery downloader update
• subscriptions can now run simultaneously! you can set the max number at options->downloading page--the default remains 1, so default behaviour will not change
• if subscriptions guess they can eat some bandwidth in the next 30s but it turns out they can't, they will bandwidth-override that last job after the 30s expires. this will stop some edge cases from causing subs to potentially hang for hours just on one last request
• 'explicit tags' are now renamed to 'additional tags'
• you can now set media to initially scale to 100% even when it is larger than the media or preview canvas
• the html parsing formula can now 'ascend' as well as search down the DOM tree--so, for instance, you can find all 'img' below an 'a' and then jump back up to the 'a' to fetch the 'href' of that image link!
• html parsing formulae can now search without a tag name--so, for instance, 'find all tags with class = "whatever"' is doable, without having to specify div or span etc..
• the html parsing formula rule panel is hence a bit richer. it also presents a preview of the rule's string as you edit
• the client no longer needs lxml to boot, and if html5lib is present, it will prefer that for html parsing. if both libraries are missing and html parsing is requested, it will raise an exception explaining the error
• the builds now include html5lib, which is better at recovering from some bad markup than lxml, and as a result some parsing jobs that formerly did not work (such as 'akaiha_(akaihasugk)' on danbooru) now do
• rewrote how services test their functional status to better account for different states. the logic here should be a bit better now--repositories will still be able to process if their network side is down, but everything will pause if any pause is set, and it should all propagate up to higher levels of error catching and status reporting better
• hydrus can now deal with really big decompression bombs (>~180 megapixel)
• filtered out en-dashes (as an alternative to standard hyphens in a "? + -" link-prefix) from booru tag parsing
• fixed a string generation issue that was stopping date decode string transformations from summarising themselves
• fixed some catastrophic boot failure gui reporting
• cleaned up a double-exit error on dialogs that could occur when spamming exit events with a script like AutoHotkey
• improved some dead cpu thread clearout code
• updated misc old code
• misc cleanup

• version 301

• after discussions with Sankaku Complex about their recent bandwidth problems, added a new 64MB/day default bandwidth rule for sankakucomplex.com--please check the release post for more information
• the 'page of images downloader' is now called the 'simple downloader' that uses the new parsing system (particularly, a single formula to parse urls)
• the simple downloader supports multiple named parsers--currently defaulting to: html 4chan and 8chan threads, all images, gfycat mp4, gfycat webm, imgur image, imgur video, and twitter images (which fetches the :orig and also works on galleries!)
• there is some basic editing of these parsing formulae, but it isn't pretty or easy to import/export yet
• the new parsing test panel now has a 'link' button that lets you fetch test data straight from a URL
• added a 'gather to this page of pages->dead thread watchers' menu to the page of pages right-click menu--it searches for all 404/DEAD thread watchers in the current page structure and puts them in the clicked page of pages!
• cleaned up some page tab right-click menu layout and order
• fixed tag parents, which I previously broke while optimising their load time fugg
• the new favourites list now presents parents in 'write' tag contexts, like manage tags--see if you like it (maybe this is better if hidden?)
• sped up known_url searches for most situations
• fixed an unusual error when drag-and-dropping a focused collection thumbnail to a new page
• fixed a problem that was marking collected thumbnails' media as not eligible for the archive/delete filter
• wrote a 'subscription report mode' that will say some things about subscriptions and their internal test states as they try (and potentially fail) to run
• if a subscription query fails to find any files on its first sync, it will give a better text popup notification
• if a subscription query finds files in its initial sync but does not have bandwidth to download them, a FYI text popup notification will explain what happened and how to review estimated wait time
• delete key now deletes from file import status lists
• default downloader tag import options will now inherit the fetch_tags_even_if_url_known_and_file_already_in_db value more reliably from 'parent' default options objects (like 'general boorus'->'specific booru')
• the db maintenance routine 'clear file orphans' will now move files to a chosen location as it finds them (previously, it waited until the end of the search to do the move). if the user chooses to delete, this will still be put off until the end of the search (so a mid-search cancel event in this case remains harmless)
• the migrate database panel should now launch ok even if a location does not exist (it will also notify you about this)
• brushed up some help (and updated a screenshot) about tag import options
• fixed a problem that stopped some old manage parsing scripts ui (to content links) from opening correctly
• improved some parsing test code so it can't hang the client on certain network problems
• misc ui code updates
• misc refactoring

• version 300

• wrote system:known url to find files that have--or do not have--certain types of urls. it works but is still a little slow--I can optimise it later!
• added exact match, domain, regex, and url class search types for system:known url
• added a button to the top media viewer hover window that will start a file export drag and drop event if dragged from
• moved the autocomplete dropdown results list down into a paged notebook
• wrote a new 'favourites' page tab for the autocomplete dropdown results
• hitting left or right arrow keys on an empty text input will move between the results tabs
• hitting arrow up/down/page up/down/home/page or passing mouse scroll events will now go to the current selected page
• typing regular search text into the input will automatically return the current page to the search results list
• moved the 'tag suggestions' part of the 'tags' options page to a new page
• added 'tag favourites' to the 'tags' options page to edit which tags show in this new tab
• added import/export buttons to the tag siblings and parents dialogs. they'll export to clipboard or .txt file, and import from the same with an additional option to add_only (i.e. to not delete/petition conflicts with the existing list)
• added some quick-and-dirty 'set as alternates/same/notdupes' buttons to the duplicate filter, which will quickly apply that status to the dupes and show some more dupes
• sped up db loading time of tag siblings and parents significantly
• added a short delay check to tag siblings/parents regeneration so rapid regenerations (such as when processing certain admin-side petitions) can be merged
• fixed an issue where similar_to searches could return results not in the current file domain
• fixed some spinctrls that were sizing to thin
• fixed a bug in the manage server services dialog that was incorrectly dealing with port conflicts on edit service dialog ok
• added a clientside and serverside assertion to test that all the services on a serverside modify services call have unique ports
• fixed an issue where hydrus network services without access keys would sometimes try to sync their accounts (this was messing up some admin server setup)
• fixed some misc dialog window structure
• messed around a little with how the autocomplete dropdown hides and shows when in float mode--I _think_ it will now be less flickery and will otherwise position itself and receieve focus better
• converted the 'export files' dialog to the new sizing system and also made it non-modal (i.e. you can now interact with the rest of the program while it is open)
• wrote a more rigorous force-fit-all-tlws command to the debug menu
• misc fixes
• misc refactoring

Changelog 250-299

• version 299

• wrote ui to review and even edit session cookies by network context. it is still a bit rough but will help with future development.
• added a 'open_selection_in_new_page' shortcut to the 'media' shortcut set that will work on the thumbnail view
• added a 'export_files' shortcut to the 'media' shortcut set that will work on the thumbnail view
• fudged manage siblings logic to not do the borked 'hey, that sibling already exists' as soon as you type the old sibling--it will now auto-petition any existing siblings when you click 'add' with a good automated petition reason that makes sense to the janitor
• manage siblings now also only shows rows appropriate to the current selection like parents does. it gets a new 'notes' column to specify conflicts that will be auto-petitioned as above
• manage siblings and parents now have a laggy 'show all pairs' checkbox to let you quickly review everything like you used to
• fixed a database-level bug that meant petitioned and pending the same left-hand sibling tag (like petition a->b, pend a->c) would sometimes not both save together
• when you middle-click or right-click->open a new page on a selection of tags/search predicates, the new page will now be named after the tags
• if subscriptions hit their periodic file limit, they will now give a little popup message describing what happened and possible causes and actions the user can take
• added a 'make a modal popup in five seconds' action to help->debug
• modal popups will now hide/show other child frames (like review services) rather than minimise/restore, as this latter action can raise the entire progam to the front
• added a 'make a parentless text control dialog' debug entry to test some key event catching
• added a 'layout all tlws' debug entry to help->debug that'll hopefully help figure out some child window sizing/position issues
• added a 'shortcut report mode', which will report caught shortcut keys and their matched commands, if any
• fixed a little shortcut catching bug in the main gui
• finished adding the new bytes control
• updated ffmpeg for windows
• improved sankaku default bandwidth rules to stop a subscription bandwidth rules mismatch that could sometimes make for subscription delays--existing users may like to add a 2GB/day rule for sankakucomplex.com
• improved some service and account error handling to better propagate the exact problem up the exception handling chain
• the clientside 'update A actually had hash B' repository sync error is now dealt with in a less severe way, and the bad update file is saved to disk with a request for it to be forwarded to hydrus dev for further investigation
• file parsing is now more resistant to invalid negative values for properties like width, height, and duration
• improved some focus code that may have been affecting linux stability
• improved a deviant art url-not-found error
• improved a wx version boot test
• misc fixes

• version 298

• wrote a new 'bytescontrol' and related noneablebytescontrol that allows for a uniform way to choose a bytes size value with a wider range--and deployed it all over the place
• hitting a mapped shortcut for 'manage file notes' should now ok the 'manage file notes' dialog as well as open it
• the manage file notes text control will now start with its caret at the end of the document
• manage siblings/parents now state how many pairs they have
• as an experiment, manage parents now only lists pertinent information in its listctrl. this is hacky but potentially a big improvement to the workflow here, feedback would be appreciated
• added a new menu entry, 'network->pause->all new network traffic', which will indefinitely pause any new network jobs. this value will persist through a restart
• 'pause subscriptions' is moved from 'services' to 'network'
• 'pause import/export folders' is moved to 'file' and all the import/export folder stuff is sent down to its own submenu
• in lieu of proper session inspection gui, added some debug 'reset login' entries to the network menu for pixiv and hf
• if a page has files but none are selected, it will now say the total size in the status bar
• the edit bandwidth rules control and its subsidiary dialog use some saner and more user-friendly layout and presentation
• the previous search distance on the 'review bandwidth usage' frame is now remembered
• fixed some bad logic where a 'copy_bmp' event could trigger despite the current media being None or a non-static image
• you can no longer open multiple copies of the subscriptions or import/export folder manage dialogs if you hit the menu entry multiple times while the first is waiting for the jobs to finish
• like import folders and subscriptions, manage export folders now waits for currently running export folders to quit before opening
• export folders run lighter and quit faster on client shutdown
• the domain manager will give a better error if a URL submitted to it lacks a schema (the http or https part)
• fixed a bunch of unicode error handling
• fixed an issue with similar files metadata orphans prohibiting new file imports
• finished up a simple shared shortcut processing object and replaced most old temp duplicate shortcut code with it
• misc ui cleanup
• misc text rendering fixes
• misc shortcut code refactoring

• version 297

• finished a prototype 'file notes' system. thumbnails and media viewer canvas now support 'manage->file notes' in their right-click menus. this launches a simple text box which will save its contents to db
• added 'manage_file_notes' shortcut to the 'media' shortcut set
• tag summary generators now have a simple show/hide checkbox and (for thumbnails) custom colours for background and text including alpha channel!
• fixed a variety of timing and display logic related to subscription query DEAD vs next check time calculation
• all currently dead subscription queries will be revived on update, just in case they were formerly set dead by accident
• the 'fetch tags even if url known and file already in db' option is moved from the download/subscription panel's cog icon to tag import options
• cleaned up tag import options layout, controls, internal workflow, and help button
• added 'select all/none' buttons to tag import options panels with multiple namespaces
• if a subscription is blocked by bandwidth, the manage subscriptions dialog will display that in its 'recent error/delay' column
• the edit subscription dialog will show similar bandwidth blocking info, on a per-query basis, under a new 'recent delays' column
• the review bandwidth usage panel will no longer show some unusual results by default that you can see with 'show all' hit anyway
• the review bandwidth usage panel will show the usage at the current search distance in a new column
• the review bandiwdth usage panel will show number of requests after data usage. this might be info-overload, so I might alter the syntax or roll it back entirely
• fixed an issue with hentai foundry parser pulling images placed in the image description area instead of main image. this particularly affected the artist 'teku'
• tags for deviant art and tumblr and thread watchers, which were formerly stored in volatile session memory--meaning half-completed import queues were losing their tags through a program restart--are now saved to the new import object directly
• removed all the old volatile session memory patch code
• added the new import object through a larger part of the parsing pipeline
• deleted the old remains of the giphy parser--if it comes back, it'll all be rewritten in the new system
• harmonised some other import pipeline code to the new system
• added a new 'management and preview panels' submenu to the 'pages' menu
• added an option to control 'save sash positions on close' to this menu
• added an entry to force-save the current sash positions to this menu
• added an entry to 'restore' the currently saved sash positions to all pages to this menu (this is useful if your window resizes real small and all your pages get crushed up)
• rejiggered how URL Classes are matched with URLs to make sure some Post URLs are not lost (this was affecting Hentai Foundry Post URLs, which were sometimes not displaying in the media viewer despite matching)
• fixed an issue where the duplicate filter page's jobs would not trigger an update after a job finished
• fixed an outside chance of a crash after running a duplicate filter page job
• improved how strings are coerced to unicode--now the preferred system encoding will be tried before utf-16, which should improve support for characters in various non-unicode sources (like neighbouring .txt files)
• fixed an issue with the client's local booru and flash files (and some other file fetching and mime reporting is a bit faster and neater overall)
• the options should be more reliable about redrawing all thumbnail banner summaries on an option ok now
• the options->media->media zooms option will now remove any <=0.0 values when it saves
• fixed up some old test code
• improved how some thread-to-gui update reporting code works
• deleted some old network object code
• converted manage subscriptions panel to an edit panel--a decoupling refactor I will likely ultimately make across the program
• wrote a help page for content parsers
• did the first half of a help page for page parsers
• misc refactoring
• misc cleanup

• version 296

• the 'allow decompression bombs' option is now moved to 'file import options'. it defaults to False
• file import options now allow max size and max resolution rules. they default to None
• file import options now allows a max gif size rule to deal with THE SFM COMMUNITY. it defaults to 32MB
• file imports will give better quality errors if they fail due to file import option exclusion rules
• file import fail errors will now also specify the incorrect amount--e.g. if a file is 64KB but the minimum limit is 256KB, it will state this in the error
• wrote a bunch of unit tests to make sure the new complicated file import options object works good
• finished the tag summary edit ui and added a new 'tag summaries' page to the options
• you can set the thumbnail top, thumbnail bottom right, and media viewer top summaries here, and everything should link up right!
• the old options to hide thumbnail top and thumbnail bottom right text are removed from the options->gui page
• tag summaries now include pending tags again
• tag summaries will order tags according to hydrus numeric tag sorting rules (0 < 0a < 0b < 1 < 9 < 10)
• tag summaries will collapse namespace groups that all start with decimals into a "min-max" phrase instead of the whole list (so 'collecting' a chapter will list p1-23 for the page phrase instead of each number in turn)
• fixed some potential error popups from the autocomplete dropdown when not in float mode (this hit non-windows users more often)
• split the parser help pages up a bit more and updated the client ui html links
• finished off the formulae help pages
• updated to wx 4.0.1 on all platforms. it promises increased stability!
• default tag import options are now moved to the 'importing' options panel, alongside the default file import options
• the database will deal with and recover from rollback exceptions more reliably. rollback-loops should be fixed, but the original event will now also produce a request to restart the client (which will safely reset everything) anyway
• improved how I create and layout the 'help for this panel -->' help buttons
• the job scheduler now regularly checks for and clears out dead wx-related jobs (this saves a bit of memory)
• cleaned up an old tag retrieval method that was wasting CPU. some larger pages with many thousands of tags should do tag calculation after selection and so on a bit faster
• fixed a thread shutdown test that wasn't working until app shutdown time
• improved some more thread shutdown code
• deleted some old thread-inefficient calllater and wx-safe timer code
• emergency server errors that occur during the response writing stage should be handled better by the client
• emergency server errors are more robust in general serverside
• improved the quality of the errors thrown by the new network engine's login system
• improved some test timing and key press simulation code
• misc refactoring
• misc improvements

• version 295

• fixed the runtimeerror popups that would come up on restore from minimise or main gui move after the complete destruction of a general search page
• cleaned up some main gui move code generally, and removed a memory leak on the way
• file queries can now cancel at multiple checkpoints during the first phase, saving a bunch of CPU time on certain large queries that are replaced mid-search
• after a file query has been going three seconds, a little 'stop' button will appear beside the regular autocomplete input. clicking this will cancel the current query! it will stop when it next hits one of the checkpoints above
• the floating autocomplete dropdown should be less flickery in some circumstances
• dejanked some more file query code
• added a 'clear orphan file records' entry to the database->maintain menu. this looks for and purges orphan file rows as you may have seen a notification about recently. this mostly affects the duplicate filter system
• fixed up the delete file code to be a bit more robust--it should lead to fewer orphans in future
• all the parsing edit panels have new layout: they no longer have info panels but instead a help button that points to the html help, and the edit and test panels are now beside each other rather than in notebook pages
• harmonised a bunch of the parser ui test panel code, refactored how the results are stored
• the test panel now presents a better 'preview' of what it contains (the actual text control has like 64KB text limit on some OSes and has unreliable text encoding rules, so using it as the raw container for the example data has lead to problems), and we now read and write the example data with a couple of new copy to/paste from clipboard buttons
• wrote another new test panel for subsidiary page parsers that does the separation formula stuff a bit better. the test results now come back for all posts as well, rather than just the first
• added a new 'deeply_nested_dialog' frame key to options->gui for the parsing ui to better lay out five or six nested dialogs in a nice 'topleft' way
• the 'topleft' frame padding is reduced from 50 to 24 pixels to better fit in deeply nested dialogs
• misc parsing ui improvements and little fixes
• the manage url classes and manager parsers dialogs now have a better 'add defaults' button that allows you to just select the defaults you want (by name) from a checklistbox
• wrote a parser for 420chan and added it to the defaults. it should automatically add and link up when you update
• if the install_dir/db directory is not writable-to (e.g. you have installed the program to a protected location like "C:\Program Files"), the client and server will default to ~/Hydrus as the db directory
• wrote a new 'TagSummaryGenerator' class that will do 'bunch of tags'->'nice summary string' conversions for the thumbnail banners and export filenames
• substituted some static tagsummarygenerators to do thumbnail banners
• did the same for the media viewer top-center namespace summary
• started some edit ui for tagsummarygenerators--I'll have some proper customisable stuff in the near future
• moved the background memory maintenance and misc daemons to the new job scheduler, reducing thread count and idle CPU some more
• added a debug 'show scheduled jobs' entry for the new job scheduler system
• decompression bomb failures will no longer count towards a subscription's fail count, so having a bunch of them won't abandon a sync
• fixed and otherwise improved a potential crash condition when a thumbnail panel closes while a menu is popup'd on it
• to forestall this program instability, the thumbnail window will no longer replace while its menu is open. the behaviour after this delayed window delivery is slightly borked, but it isn't a crash so I'm ok with it for now
• removed some other jank from the thumbnail media panel swap code
• non-cancellable modal popups will no longer have the 'close' button. trying to close them with the dialog's X button will still give the 'sorry lad, can't cancel' error
• rating and file service system predicates for services that no longer exist will now render a neat 'unknown x system predicate' presentation string rather than throwing an error
• searches in 'all known tags'/'specific tag domain' no longer provide system:untagged, wew
• some delayed events are now posted in a more thread-safe way
• misc refactoring

• version 294

• fixed video scan
• fixed up some import folder logic--they now run 'look for new files' checks separate from 'import anything still in the queue', so they can now catch up on outstanding files more easily
• the ten minute file-processing break is reverted--import folders now just 'save' every ten minutes, to forestall lost work on a crash
• import folders now have an explicit 'check regularly' checkbox to control whether you want it to check regularly or only when you tell it to. the paused status now means 'do not ever work'
• import folders now make a 'working' popup, like subscriptions do, that shows new file discovery progress and found file import progress. it has a cancel button that will stop the current job and can be hidden in the import folder's options. existing import folders will default to ON for this, so you'll likely see one of these right after you update
• import folders can now publish their files to a new page! this effectively cuts out the button step. furthermore, if a page already exists with the import folder's name (e.g. a previous page the import folder created is still around), the import folder will publish the new files to _that_ page, updating it
• file popup buttons with the same name will now 'merge'! multiple work cycles of a subscription or import folder will now just update the first button rather than spamming several
• running import folders will respond faster to a client shutdown event
• wrote some better controller-level thread management, including surplus thread deletion--idle CPU should be reduced on import-busy clients
• wrote some code to deal with sub-second times a bit better in certain places
• finished off a new scheduled job queue that collapses an old multiple-thread-when-idle system down into just one
• many things are moved to the new job scheduling system--all the old calllater calls, and the popup message timers as well
• completely removed the old wx timers the autocomplete input was using, as they were just too much of a hassle. any crashing these were causing is now gone--it all works on the simpler scheduled job queue now
• the autocomplete dropdowns are better at judging when to engage their timers and so now use far less idle CPU
• fixed a crash that could occur sometime after starting a duplicate filter maintenance task from the dupe filter page
• fixed several unlikely but plausible crashes on the admin-side of petition processing
• wrote a bunch of help for the url classes and new parsing system--it isn't finished yet, but then neither is the new system!
• if the client fails to initialise the db, it will now try to present the error in a bit of screenshottable-gui before the program quits
• improved thread watcher error handling when the given url is unwatchable
• lots of timer related cleanup and tiny fixes
• misc fixes

• version 293

• fixed the issue with options dialog not opening again after a save--I apologise for the inconvenience
• the default system:time imported predicate (as set in options) is reset to the default '< 7 days'
• fixed another potential crash with the manage tags dialog (in fact with any dialog that has an embedded autocomplete or any other autocomplete with a non-floating dropdown list, such as the import file path tagging dialog)
• the sankaku user-agent has been updated to be a generic Firefox string "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0", which seems to work with their new rules. if your sank has been broken and you haven't touched the settings that do this stuff, your sank should be magically fixed on update
• the default global hydrus user-agent has been updated to the more polite and compatible "Mozilla/5.0 (compatible; Hydrus Client)". if your previous global user-agent is currently 'hydrus client' (the old default), you'll be updated
• import folders now deal with interruptions more skillfully and take a break and save their progress every ten minutes
• the manage import folders dialog now waits properly for any currently running import folders to quit and save themselves before opening itself (like how the manage subs dialog does it)
• import folders deal with some error states in a better way and are able to cancel out of their loops more reliably when an error occurs
• added a new checkbox option to options->tags that lets your 'write' autocomplete inputs (ones where you can submit new tags, like in manage tags) select the first result with non-zero count by default. this means it will skip over the first result if it is just nonsense like 'boyp' that you typed to get the results
• linked all the final behind the scenes stuff in the thread watcher together with the new parsing system. this stuff remains complicated and non-user-friendly, so please feel free to completely ignore it and the following batch of points:
• -
• wrote new url classes and parsers for 4chan and 8chan--they should all be automatically linked up on update
• the new thread parsers will use the first part of the first post's comment if no subject is set
• the domain manager can now cope with URL->API URL links
• manage url class links now displays the current expected api pairs of URL class links and filters out what cannot be currently parsed as a result
• manage url class links now only displays watchable urls in the bottom listctrl and permits parsers to be linked!
• the domain manager can now try to link url classes with parsers based on example urls!
• a button to fill in missing links is now available on the manage url class links panel
• content parsers that produce urls now define different url types--file, post, and 'next gallery page'
• content parsers can now parse a 'source timestamp' type
• veto content parsers now use a StringMatch object to do their matching. existing veto content parsers will be updated and _should_ work about the same as before (unless the previous string was crazy in regex rules) using a regex match
• html formulas now pick up the 'string' value more reliably in complicated or mangled html
• html and json parsing formulas can now work without any rules, in which case they action the top level node (this is useful to replicate the json/html for subsidiary page parsing)
• activated some of the new file import objects' features for the new parsing system--now, they store tags and known hashes as informed by what is parsed. this information is tested and used in import
• the db can now test all the hash types--md5, sha1, sha256, sha512--during pre-import checking
• the new parsing system now takes a temporary 'parsing context' that will eventually be able to receive and serve temporary variables but at the moment just holds a 'url' entry if the parser would like to use the url used to fetch the data anywhere
• all the new parsing ui now has a button to edit the parsing context, and the example parsing context is saved to the topmost page parser object
• wrote some url classes for 420chan.org--they will be added on update
• fixed an issue with the 4chan thread url class
• -
• the standard new listctrl panel wrapper can now now provide a special 'export to pngs' for certain named objects that will quickly export one object per png
• the 'import from png' dialog as launched from the standard new listctrl panel wrapper now allows multiple png selection
• the default parsers (and now the url classes as well) are stored in install_dir/static/defaults. they are read from here when you create a db or ask to load from defaults, so feel free to play around with this
• the manage services dialog now has an additional Yes/No warning/confirmation dialog if it expects to delete any services on dialog ok
• page tabs escape mnemonic characters (typically '&', which in ui labelling rules makes the following character an underlined shortcut) more reliably
• in an attempt to stop subs making so many separate small file popups, the subscription daemon will now only naturally check every four hours (previously one hour) and will wait up to ten minutes for bandwidth to free up before dumping out (previously 30 seconds). these values will be user-configurable in a future update
• fixed import for some unusual apngs
• corrected some more GMT/local time stuff with the new system:time imported 'date' mode
• misc GMT/local time fixes (profile logs names themselves correctly, etc..)
• improved how the thread watcher reports thread-check errors in unusual situations
• fixed an issue generating network contexts when the initial domain is 'localhost'
• improved some misc program stability
• started a new job scheduler object to reduce idle thread count and CPU usage--will play around with it a bit more and see about integrating it in the coming weeks

• version 292

• extended system:age to support searching by fixed calendar date, e.g. "system:age > 2018/01/24"
• 'system:age' is also renamed to 'system:time imported' and presents nicer strings
• I believe I have fixed the manage tags dialog crash
• the manage tags dialog now loads related tags, file lookup script info, and recent tags off the main ui thread, meaning the dialog itself should launch much faster in almost all situations
• cleaned up how a bunch of autocomplete<->manage tags events occur
• 'set_search_focus' works in the manage tags dialog again, drawing from the new shortcut system
• fixed ctrl+scrollwheel on autocomplete entry, which now correctly scrolls the selection over the results again
• cleaned up a bunch of the taglist scroll and selection code, particularly for tag children/parent selection events
• fixed a potential crash bug initiated by en early exit from the export files dialog
• fixed a potential crash bug initiated by an early petition processing page delete
• fixed a potential crash bug initiated by an edit repository close during access key fetching
• the thumbnail view should be a bit less jittery--the scrollbar should fade neater on mouseover, and maybe the thumbs are smoother as well (I turned off a now-superfluous double-buffering mode, please report any visual bugs)
• added 'check_all_import_folders' to main_gui shortcut actions
• menus off the main menubar that are out of date (this often happens to 'pages' after a session addition/delete) will now be disabled (greyed out) until they are back in sync
• the duplicate filter now detects media it cannot display more carefully. if it runs across undisplayable media due to file orphan status, it will skip over it. if there is nothing else to display, it will ask the user to inform hydrus dev
• added json support to the parsing engine!
• wrote a new json parsing formula object
• wrote a new 'compound' parsing formula object that combines multiple formulae together
• wrote new panel for editing formulae with some cleaner button workflow
• added multiple 'separated' content parsing solutions to page parsers and wrote all new gui to reflect that
• expanded the new separated content parsing solutions to be full-blown nested sub-page-parsers, wew
• put some more general work into the new parsing engine--the new page parser object now saves and serialises, so it can be imported and exported. I don't expect to change it much more, but don't rely on that! this is all still under construction
• added 'file hash' content type to content parsers and improved ui support for when string conversion rules generate bytes data etc...
• new parsing objects now render their parsable content correctly in the 'produces' column
• misc cleanup of new and existing parsing dialogs
• fixed the handling of escape/cancel events on the 'select from a list of strings' dialog
• on update, users with a certain kind of orphaned file will be informed of the numbers and a request to inform hydrus dev
• fixed exporting serialisable object to png, which was failing due to a bmp data change in the new wx
• fixed a couple of places where I was switching GMT and local time around
• deleted some old useless tag editing and commandevent code

• version 291

• expanded the permissible shortcut keys to all reasonable ASCII characters, such as comma and ampersand and the types of brackets
• fixed a delayed crash sometimes caused by the file import frame (the one launched by dropping some files on the client or going file->import files)
• the file import frame does its file parsing job in a generally more sensible way, particularly when new drops occur while parsing is ongoing
• fixed a crash that could be initiated by the manage tags dialog setting the favourite tags late
• fixed a crash that could be initiated by manage parents closing before results came in
• fixed a rare possible crash that could be initiated by a review service panel delivering results late
• when media pages load, they will be more thread-crash-polite to wx
• wrote a new 'PageParser' object that will be the primary parser in the new downloader engine. this can deal with a variety of complicated parsing situations. it is still a work in progress, not yet ready for proper use
• started new 'manage parsers' dialog for the new parsing system and arranged a bunch of things around that to make the eventual transition easier
• edited some more old 'parsing scripts' ui to again make the transition easier. file lookup scripts will be converted to something new as we move to the new parsing system
• fixed some bugs in contentparser instantiation
• fixed a bug in the html tag rule edit dialog that wouldn't allow attribute pair editing
• the names in the manage tags file lookup script dropdown are now sorted
• removed some old redundant error-hiding code from the pubsub system
• a common 'is my page closed' check is now a lot simpler and faster
• fixed an edge shutdown bug where log messages were not able to be written to the log file
• some media viewer objects delete themselves more safely
• maybe fixed a rare menu refresh bug in linux
• improved how hydrus deletes temp paths--if they are 'in use by another process', it will now put them aside and try again later. this error typically occurs when file mimetype parsing fails, and due to the error context hanging around, an open file handle is still in scope.
• the client is better about recovering from missing tags in the db--in certain circumstances, it will now generate a 'unknown tag:random hex' as a simple placeholder that can be better used to investigate the error, rather than dumping out
• downgraded os x opencv to 3.1.0 to hoyefully increase compatibility
• the server closes a little more neatly
• upnp errors are more verbose
• misc code cleanup

• version 290

• content (tag and rating) shortcuts in the 'media' shortcut set now work on the thumbnail view!
• tags in lexicophically sorted taglists will now sort numerically if their subtags start with decimals (e.g. page:3 will come before page:20 now)
• trying to add/edit a subscription query that is a dupe will now give you a messagebox saying no, and pasting multiple queries will filter out the dupes and inform you (so, if you aren't sure if some of your list of ten artists to add is already in your big 'artist sub', just paste them anyway--it'll take out the dupes for you)
• string converters now support regex substitution! (I expect to eventually integrate this into the filename tag parsing panel!)
• 'modal' popup messages (that create a new dialog and stop interaction with the program, like the one launched for repository processing) will now minimise non-main-gui frames while they work. these frames will be restored when the job is done (this should relieve the problem of this modal dialog sometimes letting other semi-stay_on_top frames obscure it while still prohibiting interaction, wew
• simplified and sped up how importers decide whether to do file work or wait
• made file importers do ui-polite pauses during the file import loop more intelligently--if the files make no big change to db or ui (e.g file was known already deleted because of its url), no wait will occur at all, meaning redundant lean queue-sections should work real quick
• fixed an issue with multiply-nested submenus sometimes not deleting themselves neatly and resulting in highlight-related RuntimeErrors.
• fixed an issue where the 'select from list' dialog could sometimes be ok'd without a valid entry--I believe this may be have been causing a delayed crash as well
• added a 'ui timer profile' mode, which profiles some common unified ui update loops that draw animations and keep downloader ui synced and so on
• the 'delete processed' file import status right-click menu entry will no longer show if it has the same count as the 'successful' line
• the options->colours panel has some better layout and the namespace listbox asks a yes/no to delete if you double-click/enter a colour
• dropping a raw file url onto the client will now automatically pend it to a raw url importer (previously, it would be put in the input box, but after rethinking how this could go wrong, I concluded it wasn't such a big deal). if you use this a lot, see how you like it, and let me know if you would want an option to revert to the old behaviour
• updated domain manager so it tracks url class identity in a neater way (renaming them will no longer break links or display options)
• improved some widget feedback of edit url class panel
• simplified the manage url class links panel
• url classes now normalise gallery and page urls differently (page urls strip all non-declared data, gallery urls just switch scheme). this may change in future
• url classes can now convert a normalised url to an api url--for instance for informing the program that an imageboard thread url has an api version that'll be better to check
• the domain manager can now determine arbitrary url parsability capability, and this is just starting to be integrated into url drag-and-drop--try dropping a known url class on the client, like a booru link: it'll now tell you it sees what it is but can't yet parse it!
• the 4chan and 8chan thread url classes will reset to defaults again this week--replacing with url classes that convert to the api versions--and you'll get the new api url classes themselves
• if missing, the primary tables in client.caches.db can now regenerate on boot. they aren't properly repopulated yet, but the client will boot and you'll get an error message telling you what to do next
• improved .txt and clipboard tag import unicode support
• removed leading BOM (byte order mark) from .txt and clipboard tag imports
• improved some clipboard text fetching code
• added a new 'hover window profile mode' to the debug menu--it will print the logic info behind any media viewer's hover window show/hide decision
• tweaked the locale-setting code, which was stopping some users (on Win10 creator's update, it seems) from booting the client
• added system:rating to 'all known files' query contexts (so you can now search for deleted files that have ratings)
• the manage import folders dialog now has a redtext warning about subdirectories
• some file importers deal with shutdown events more gracefully
• lz4 is no longer required. if it is absent, or the version is too old, a simple statement will be printed to the log and it will not be used in either the client or server
• updated to ffmpeg 3.4.1 on windows
• updated opencv to 3.4 on windows and linux (had trouble with os x--will continue working on it next week)
• fixed an important grammar typo in the help

• version 289

• fixed an issue where scrollbars were only appearing on taglists after a resize event
• fixed the raw filename component of file drag and drop events from the client to external programs
• fixed the tag lookup scripts
• fixed some wx menuhighlight issues
• improved some shutdown code
• fixed the add/edit namespace colours options panel, which needed to be updated to deal with the new wx's better alpha channel reporting
• fixed an issue when hitting 'open externally' on a media collection
• fixed a crash on client shutdown whenever closed pages were in the undo menu
• think I fixed another shutdown crash
• fixed a rare issue with the collect by dropdown not being able to generate a string to display
• misc wx fixes
• added a 'importing' page to the options dialog, which now sets the default file import options for quiet and loud file import contexts
• the old and no longer used 'exclude deleted files' option is now removed from the 'files and trash' panel
• finished off default url matches for all downloaders that come with the program--these will be set on update, so if you have custom ones, please export them before you update so you can import them again!
• improved how urls are matched and presented for the user in the media viewer
• added a 'delete "successful" file imports from the queue' entry to the file import status button right-click menu--this only removes 'successful' and 'already in db', leaving anomalies
• improved locale instantiation in the client and added locale strings to the help->about dialog
• you can now set the page name prefix for a paused thread checker. it defaults to a unicode pause character: ?
• thread watchers will no longer pause on a network error during a check--they now have a 'delay' system like subscriptions, and on a network error, they will delay four hours (unless you hit 'check now')
• patched in some simple 'connection cutoff' network error handling, we'll see how it does
• wrote in some more proper error handling for a specific connection cutoff error that is being produced
• the similar files search tree regen code now clears out orphaned files. if you have had blank 'unknown' files appear in similar files searches, please run database->regen->similar files search tree
• bitmap buttons on download pages will now update using less CPU and will flicker less
• improved some video rendering error reporting
• fixed the 'author name' regex favourite default, which had a superfluous asterisk. if you would like to fix it yourself, please try: [^\\]+(?=\s-)
• added 'flush log' debug command
• client.pyw now makes a safe 'system' ui error popup if it fails to boot

• version 288

• updated to wxPython Phoenix (4.0) build!
• all builds now require the new wx, so if you run from source, please consult the help files for new info on figuring this out
• did a ton of wx refactoring
• merged a large number of timers
• wrote a new wx-aware timer to replace some of the more awkward old timers that could not be neatly merged
• replaced wx.calllaters (the new ones aren't always garbage collecting nicely) with the new custom timer object
• misc wx deprecation refactoring
• cleaned up some wx test code
• fixed some crashing wx test code
• fixed 'select from list of strings' object, which was no longer processing enter key properly
• cleaned up a bunch of ui object interaction code
• cleaned up a bunch of wx-related shutdown code
• fixed an issue with the splash window not shutting down cleanly
• fixed a shutdown db-gui status report crash
• was unable to get flash embed windows working properly without edge-case crashing on windows, so they are disabled for now (you'll likely get an 'open externally' button instead)
• animations now clean up their memory buffer faster in certain circumstances--all users who view animations in the preview window should notice significantly leaner 'idle' memory usage over time
• fixed a very important inefficiency bug that meant select->inbox/archive on large pages with more than a handful of inbox files was taking extremely long periods of 100% CPU
• wrote a faster way of fetching some media and their paths
• copying files to clipboard is now much faster
• initialising any large file drag-and-drop event is now much faster
• certain redundant image pre-fetching will no longer be done
• 'open externally' panels will no longer draw a (slightly buggy and useless) animation bar if the underlying media would have had one
• added a new 'menu profile mode' that will profile any menu click
• cleaned up the debug menu
• deleted some old options code that is no longer used

• version 287

• thumbnails can now be drag-and-dropped to other pages!
• dragging and dropping thumbs to another page tab will remove them from the source and append them to the destination, maintaining file order!
• DnDing thumbs to a 'page of pages' tab will put the files in the next lowest selected media page
• DnDing thumbs to a blank notebook area (or a page of pages without a selected media page) will create a new page for the thumbs
• holding down ctrl when you drop thumbnails will not remove them from the source
• please forgive the thumbnail DnD cursor, which for now will be in the 'copy' state, despite the internal DnD being move by default
• improved page tab drag and drop drop logic--dropping onto the page area itself will no longer send the page to the right-end of the current notebook
• the 'file import options' object now supports three 'presentation' booleans--for new/already_in_inbox/already_in_archive files--so you can customise whether new thumbnails appear based on each state. page imports will by default show everything, while 'quieter' import queues like import folders and subscriptions will continue to just show only 'new' files in their files popup buttons. if you have a gui page with 10k+ items in its queue, try reducing the presentation to speed it up!
• all existing import queues will be updated when they are next loaded--but please note that for simplicity and safety they will all initialise to the 'quiet' presentation method, so if you have ongoing download pages in any of your gui sessions (including thread watchers!), they will only add 'new' thumbnails unless you edit them. I apologise for the inconvenience
• the regular hdd import now has a file import options button!
• subscription query 'finished' file popups are now merged up to the subscription level--so, a sub with five queries that each get 20 new files in a run will now ultimately leave one popup with 100 files
• file popups (as produced by subscriptions and a couple other places) now preserve their import order!
• if a subscription with many queries runs out of bandwidth, it should now only give you one 'no more bandwidth to download files' message, rather than one for every outstanding query to sync
• added a checkbox to turn on/off the new random subscription syncing to the options->downloading panel
• the file import status button's menu now supports import/export of sources to clipboard/png! it _should_ also support unicode. be careful not to paste paths into a url cache, or urls from one gallery site to another, or you'll just get errors--this is prototype, so please keep like with like for now
• the png import/export system now supports raw string payloads
• the new listctrlpanel can now hang its buttons in multiple rows
• the manage subscriptions panel now has an 'overwrite checker options' button to mass-set checker options
• the manage subscriptions panel now has a 'select subs' button that selects subs based on a basic query text search
• separating merged subscriptions now sets better new subscription names of 'old_sub_name: query_text'
• saving a session from a page of pages with a custom name will no longer suggest a session name prepended by [USER]
• doubled the subscription and downloader instance default bandwidth rules to 400 and 200 rqs/day
• the 'load_images_with_pil' and 'disable_cv_for_gifs' options are now officially BUGFIX in the options--unless you know you need them on, turn them off!
• added some safeguards to the new dialog-panel system's OK stuff, which sometimes catches a duplicate OK event
• shuffled some db update status texts around

• version 286

• simplified how thread watcher assigns DEAD and 404 status--it should do it more reliably now
• thread watchers now publish their page names (including updating dead/404 status) right after they check
• the save session menu under pages now lists the exsting sessions for quick-save--if you select one of these, it will throw up a yes/no overwrite confirmation dialog
• 'appending' sessions now loads them in a new page of pages named as the session name!
• right-clicking on a page of pages now gives you a 'save this page of pages to a session' option
• right-clicking on the page tabs lets you append a session anywhere
• 'load' session is now 'clear and load', and it throws a yes/no dialog for confirmation
• the gallery pending queries box now allows multiple selection with ctrl/shift clicks. it'll even move up/down correctly on non-contiguous selections!
• the gallery query input is now a new combined textctrl-and-paste button control that will appear in other places across the program
• the page of images input now also uses this new control
• the page of images and raw urls queues will now ignore any input that does not start with 'http'
• pasting a list of texts from the clipboard will now typically strip the pasted content of leading or trailing whitespace
• some text and tag cleaning is a bit faster and neater
• the file import status panel now supports 'open sources' on its right-click menu--it opens in your web browser or file explorer as appropriate. (linux can't do file explorer though)
• thumbnails will no longer 'fade' when their pages are not visible, reducing CPU load in several contexts
• thumbnail fade will use less CPU overhead on very fast computers
• the timer loops that all import pages use to keep themselves updated are now harmonised into one loop at the top gui level (this should reduce some idle CPU overhead and improve some ui responsiveness)
• content (tag and rating) shortcuts can now be set to 'flip on/off' or 'set'
• the manage subscriptions dialog now waits for any currently syncing subs to cancel and save themselves before opening itself (keeping it all synced and rollback-proof)
• added a 'watchable' url class type and wrote examples for 4chan/8chan thread urls
• added 4chan/8chan file url classes to the examples, which if added will auto-hide them from the media viewer
• fixed 'add parameter' in edit url match panel
• simplified some stuff in the existing parsing system
• added preliminary 'url type' support to the parsing system
• the html formula parsing object can now return the full child html of the node it finds
• the new url and html stuff doesn't do anything yet--but it will in the new downloader engine
• added a bunch more booru entries to the url classes defaults. I will continue filling these out, hopefully setting comprehensive defaults next week
• the advanced mode thumbnail right-click menu lets you reparse and regen thumbnails for any files! if you have rotated or stretched files, you can now fix them from the menu!
• added new tools to let the client update media file metadata and thumbnails live!
• the new reparse and regen should update file metadata and thumbnails live!
• improved how some 'quiet' errors are printed to the log. they'll now have the full trace of the error as well as the stack
• added a daemon_report_mode debug mode. it throws up a popup every time a daemon fires its callable
• fixed an issue with the editstringtostringdictcontrol
• fixed an issue with loading (and updating to a new version) import pages or import folders with some kinds of unicode paths
• you can now set the 'woah, you should close some pages' warning value under options->gui. its default remains 165 pages
• system:similar_to searches should be a lot faster on clients with many files
• the manage subscriptions and edit subscription panels now both list their num_urls summary column as '22' if all done but '11/22' if not. this column also sorts based on percentage completion, then num total, then num done
• the database migration dialog now uses the new listctrl--its buttons are now also clever, and will disable when they are invalid
• misc fixes
• fixed an index bug in the new listctrl after certain types of 'setdata' call
• cleaned up some search code
• some veto code cleanup
• cleaned and harmonised how text is pulled from the clipboard

• version 285

• added 'network' main gui menu and moved network stuff from services to it
• split the new domain manager stuff up into separate dialogs and menu entries on the new network menu
• manage url classes dialog now lists url type in the listctrl
• added url class links info (which will permit client-specific settings and downloader mappings for url classes) to the domain manager
• wrote a 'url class links' dialog and added it to the new network menu (only the 'display on media viewer' part works atm)
• the domain manager now filters urls on the media viewer depending on whether they have a url match and are set to display in the new links panel
• updated the local booru service code to the new service system
• the local booru's shares can be reviewed again under review services
• the local booru's port and bandwidth settings can be set again under manage services
• the different gui parts of the local booru are updated to new controls
• fixed a local booru 404 reporting error
• the edit subscription panel now has a 'paste queries' button that lets you add queries en masse
• added 'manage_file_urls' to shortcuts system
• added several 'get_similar_to_x' actions to the shortcuts system
• the manage upnp dialog now initialises its mappings on another thread and fails better when UpnP mappings cannot be fetched
• connection and readtimeout network exceptions are now recognised more specifically by the client. subscriptions will only delay for an hour on one of these exceptions
• improved the resilience of the HF login code after wake from sleep (when networking is often not available for a couple of seconds)
• like the recent subscription query randomisation update, subscriptions themselves are now synced in random order (this stops a subscription named 'aadvark' always getting first bite into available bandwidth)
• fixed import for jpegs that have unparsable exif data
• fixed a bug in 'next month' bandwidth estimate calculation when the month is December, wew
• fixed some logic that was setting max page at 165 rather than the intended 200 and added a dialog that even lets you engage the debug override at 200 if you are insane
• all audio mime detection and duration parsing is now done through ffmpeg. hsaudiotag is no longer needed to run the program
• since the old listctrl sort crash bug is still hitting some people, I've disabled sort on the old listctrl class. feel free to try to sort any listctrls now, no matter your situation. I will continue replacing the old class with the new working class over time
• updated another listctrl
• a ton of misc controller options/manager access refactoring
• cleared out some old code
• moved Time controls to their own file and added velocity and checker options stuff as well
• wrote a new edit panel for single controls and updated the time delta button to use it
• misc refactoring

• version 284

• fixed subscription queries turning dead on the initial sync
• all dead subscription queries have been set to check again in case they can revive
• added query file velocity to edit subscription panel
• subscription network contexts now reflect the new multiple subscription query system, and are named "sub_name: query_text". as every query now counts as its own separate subscription network context, this will stop query-heavy subscriptions from throttling so much on bandwidth limits
• finished URLMatch object, which matches and normalises URLs into certain 'classes' like 'gelbooru post url'
• expanded some URLMatch subdomain options
• fixed some test logic in URLMatch
• finished the last of the EditURLMatchPanel
• split the 'manage network rules' dialog into two panels--it now has a 'url classes' tab
• wrote a panel for managing url matches
• added export/import/duplicate buttons to EditURLMatchesPanel
• wrote some URLMatches for hentai-foundry as an initial test of the system and added a temp button to add them--please check them out to see how it all works
• all, invert, inbox, and archive (and none, lol) thumbnail 'select' menu items now have counts
• invert is now at the bottom
• the thumbnail select menu now has local/remote entries if applicable (this typically is only true in 'all known files' file domain)
• added png/clipboard export/import/duplicate code to the generic new listctrl button wrapper panel, which will save a bunch of time as the png/clipboard sharing system expands
• added a human-facing serialisable name to all objects on the new serialisation system and tied the new import/export code into it for png presentation
• the edit import folder dialog will now complain (but not veto) on an ok event if any of the entered paths do not exist
• if you attempt to manually run an import folder while import folders are globally paused, you'll get a little popup telling you so
• added an experimental 'thumbnail fill' setting to options->gui. it zooms the existing thumbnails so they fill the whole thumb space. feedback on this from those who would be interested in a prettier system would be appreciated
• added 'paste tags' buttons to filename tagging options panel
• the paths/urls in the file import cache are now their own object that holds the creation/modified/source times and current status and note. this object can also hold prospective urls, tags, and hashes for future use
• a bunch of file import actions are faster
• all the different importers now use this new file import object
• fixed a screen position calculation in the new drag and drop filtering code that was accidentally including too many possible drop candidates on drops in the top-left corner of the main gui (if you had trouble moving tabs to the left, this should be it fixed!)
• fixed a problem display volume/chapter/page tags that included unicode characters in thumbnail banners and media viewers
• fixed a rare media display bug in the dupe filter
• fixed some 'C++ part of panel has been deleted' bugs in review services if the frame is shut down before delayed db info is fetched
• cleaned up some more 'C++ deleted' errors in import files selection dialog
• fixed the network context custom header panel 'add' action, which wasn't saving the value of the panel
• fixed a bunch of bugs in the newish QueueListBox class
• SynchroniseRepositories daemon will be better about quitting early on application shutdown
• cleaned up some pending pretty timestamp grammar
• when the client cannot clean up a temporary file, it will print more error information
• added pylzma to the 'running from source' library recommendations. this is not required, but if available it adds ZWS flash support

• version 283

• subscription popups show a bit more info about their individual queries
• subscription popups hide the sub-query info in text and file buttons if it is the same as the subscription name
• subscriptions now work through their queries' downloading phases in random order (so a heavy query named 'aardvark' won't repeatedly choke the sub's other queries if bandwidth is tight)
• if a subscription query dies, a notification popup will now be made
• changed a logical edge case that may have been killing some subscription queries after the first sync
• re-enabled pixiv artist search, as it still uses the old gallery style
• added tentative support for 'ZWS' flash files
• if there is more than one import folder, the 'check import folder now' submenu now has a 'check all' entry
• TimeDelta controls that have the 'days' entry (as in thread/subscription checker options panels) now have a max period of 10 years for those _really_ slow threads
• fixed an issue where the 'quick namespaces' section of the new filename tagging options panel was not initialising correctly when it loaded data from a previously serialised import folder (this was the "unhashable type: 'list'" bug in the edit import folder panel)
• the filename tagging dialog now works on the new panel system. it also has an entry under options->gui: 'local_import_filename_tagging'
• polished StringMatch object
• polished EditStringMatchPanel
• html formulas now have a string match object
• improved parsing error system
• miscellaneous improvements to the parsing system
• misc fixes to StringMatch and parsing system
• the main gui will no longer interpret text drops on child frames as url drops. this fix just quashes the drop--it doesn't allow the initially desired drop on the child frame to occur. I will keep thinking about this
• the dupe filter's 'no media to display' text now says 'Looking for more pairs--please wait'
• the 'remove' media action is now labelled 'remove from view' to reduce 'is it a delete?' confusion
• fixed an icon initialisation order problem that meant pre-boot errors were unable to display
• fixed a graceful-shutdown-on-boot-error bug
• the custom 'listboxtags' controls now post an event on their contents changing
• the tag censorship panel now updates its status text when its listboxes change due to a mouse double-click event
• fixed file import 'source time' column sometimes showing '3e-05 years' kind of garbage that was due to a missed float->int conversion
• cleaned up how some page names are thrown around, which should stop some stray [USER] prefixes slipping though
• the adminside petition management checklistbox will now escape mnemonic characters (typically '&') and hence display them correctly on sibling pairs and so on
• wrote a 'help my client will not boot.txt' that explains the new debug builds and some other common fixes to try
• 'all known files' file search domain is now hidden behind advanced mode
• fixed another instance of flashwin causing a failed boot
• cleaned up some old broken unit tests
• some misc help improvements
• updated Windows build to new version of sqlite

• version 282

• rolled back to an older version of pyinstaller that does not seem to have the embedded flash window problems windows users experienced
• added an error handler to wx.lib.flashwin import--if it fails to import, the client will print the error to the log and thereafter show 'open externally' buttons instead of the embedded flash window
• created a 'filename tagging options' object to handle the instance-non-specific auto-tagging parts of the filename tagging dialog
• moved all the appropriate tag generation code to this new object
• extracted the simple and advanced panels in the filename tagging dialog to a separate panel
• wrote a new wrapper panel to edit filename tagging options objects directly
• cleaned up some tag generation code--it'll now all be siblinged/parented/censored the same way
• import folders now use the filename tagging options object, harmonising this tag parsing code
• edit import folder gui now has an add/edit/delete listctrl to manage the new tag_service->filename_tagging_object relationship (this replaces the old .txt management button and summary text)
• finished the StringConverter class in the new parsing engine. it applies a series of transformations to text
• wrote a panel to edit string converters
• wrote a panel to edit string converters' individual transformations
• updated html formulas to use string converters instead of the old cull_and_add system
• html formula edit panels can now edit their new string converters
• file lookup scripts now 'stringconvert' their file identifier strings--this should allow the 'md5:md5_hash' fix we've been talking about (i.e. by first encoding to hex and then prepending 'md5:')
• the help->about window now shows the client's default temp directory
• thumbnail regeneration--particularly full-size regen and resized png gen--should be much faster in the client
• the debug exes are now included in the windows build
• the debug exes are now included in the non-windows builds
• the test exe is no longer included in the windows install (can't remember it ever being useful, and it is like 10MB now)
• unfortunately, a recentish change to how pixiv serves gallery page results has broken the hydrus pixiv downloader. this will have to wait for the downloader overhaul, and even then, it might be _slightly_ tricky. pixiv downloader entries are now hidden, and existing pixiv subscriptions will be paused on update
• the thumbnail select submenu now clarifies 'all (all in x)' if all are in the inbox or archive
• you can now archive/delete filter from the thumbnail menu on a single file
• lowered the http session inactivity timeout even more, to 45 minutes
• fixed a couple of instances of the new subscription dialogs being unable to mass-retry failures
• ffmpeg parsing errors now print more info about broken file path
• some daemons will be snappier about shutting down on application shutdown
• took out the sometimes invalid 'booting' phrase in the during disk cache population status report
• the client will now warn you about page instability at 165 pages after one user reported instability at 175. 200 is still the strict limit.
• downloader pages and subscriptions will fail more gracefully if their downloaders cannot be loaded (e.g. they used a since-deleted booru)
• fixed listctrl panel button enabled/disabled status when the child listctrl starts empty
• the new listctrl can now select data in a cleverer way
• misc fixes
• misc refactoring

• version 281

• subscriptions can now support multiple queries!
• subscriptions now use the new thread checker system to set their check periods!
• improved some check timing calculations to deal with subscription timestamps
• existing subscriptions will update
• subscriptions can now see any 'delay' reason on their edit page and can scrub it there as well
• subscriptions no longer track 'last error'. everything goes through the delay system and keeps track of reason
• added 'last new file time' to the subscription listctrls
• subscriptions now generate separate 'file button' popups for each query, and file button popups now include their name in the button label
• the subscription listctrl is now breddy wide. I will be adding some way of hiding columns (with persistent memory) on the new listctrl at some point in the future, so please bear with it for now
• manage subs now has a wrapper panel and buttons that disable when invalid
• tumblr simple 'video' posts should now be supported! turns out they are just mp4s
• in the client, static image thumbnails are now generated with opencv
• files that give truncated image errors should now import ok. they'll render with errors. if cv is up to date and working ok, they should have good thumbnails as well (otherwise they will have all-black thumbs)
• updated manage regex favourites dialog to the new panel system
• regex favourites are now editable under the new options->regex page
• ongoing import or export folders will now stop working if their respective manage dialogs are opened
• import folders can now stop working if the client shuts down during the initial file parse phase
• export folders will now stop working immediately if the client shuts down
• export folders can now stop working during the file copy stage
• the splash screen is now a little wider and taller
• the splash screen now has a subtext row for misc technical data
• the boot and exit splash screens show a bit more data about several things
• the repository sync daemons should be a bit snappier about responding to a client shutdown
• the import files dialog is now a non-modal frame! (you can now continue to use the client while this frame parses files) you can even open up several at once!
• the 'import file path tagging' dialog should now never be taller than its spawning screen and also slide up and left if it spawns with its bottom-right corner off-screen (this dialog is still on the old system, however, so the newer sizing stuff that does this stuff automatically is not yet available)
• sliding up-and-left should now also occur on spawn for frames on the new system
• the advanced 'process now' repository button will now disable itself if there is no work to do!
• if simple downloads like gallery pages are larger than 100MB, the client will throw an error (this is a stopgap for an ongoing issue)
• the 'maintenance and processing' page has a bit more warning text
• the new listctrl wrapper panel now supports menu buttons
• libpng's iccp terminal warnings from loading the flash/pdf/zip thumbnails should now be gone!
• misc refactoring

• version 280

• the client should now recognise EXIF file rotation and flipping in resolution discovery, thumbnail generation, and image rendering. existing files will be borked, so will have to either be manually deleted&reimported, or wait for a maintenance routine that will retroactively fix all these
• videos with a non 1:1 sample aspect ratio (wew lad) should now import with correct resolutions
• all 'tag import options' are now managed with the new button
• deleted all old tag option management gui, including deleting ClientGUICollapsible entirely
• with the simpler tag import options, increased max page count to 200
• page of images downloader now has two pause separate buttons for queue and files processing
• the page of images downloader should no longer too-quickly blank out its parser status at the end of queue processing
• the default hydrus bandwidth rules now no longer has the choking 'requests per day' limitation that was messing up large mappings uploads. (it is now just 64MB/day)
• your hydrus default bandwidth rules will reset to this on update
• POST requests (which in for our purposes are always user-driven) no longer obey bandwidth rules
• fixed a bug when sometimes hitting the file limit or hitting a siteside 404 in the gallery downloader
• a rare bug due to empty string menu labels is now no longer possible. empty string labels, if they slip through, will be replaced with 'invalid label'
• you can no longer choose the empty string as a gui session name
• improved how network jobs keep track of all applicable network contexts vs the preferred session and login contexts
• cleaned up a bunch of login code, improved verification testing
• fleshed out domain login system
• in the networking engine, the read timeout is now six times the connect timeout--we'll see if this reduces some hydrus and other read timeout problems we've seen
• increased the server transaction period from 10s to 120s--we'll see if this helps reduce some spiky server POST lag or what
• the default tag import options panel now edits on double-click, rather than deletes. it also yes/no dialogs on a delete action. this control is still trash, but it now works more like the rest of the program

• version 279

• moved the hydrus service object over to the new networking engine
• hydrus services now appear in the bandwidth review panel
• hydrus requests now obey larger network bandwidth rules (this mostly means 'global')
• service bandwidth usage and rules are no longer managed from manage services--it is now under review bandwidth usage, like all other network contexts
• updated some network engine stuff for misc hydrus states like 'server busy'
• fixed a bug in the server code where the session key cookie had an invalid expiration timestamp--servers should update this week to make sure new clients can log in properly
• hydrus network requests will force-set User-Agent as 'hydrus client/(network_version)' as the network version is used in the protocol to determine compatibility
• fleshed out the hydrus specific network job, giving it the various bandwidth tracking and version checking responsibilities the service object used to have
• moved session cookie decay (only matters for Hentai Foundry atm) to the new session manager (was previously hacked into the login manager)
• moved some hydrus response parsing stuff around, added content type awareness to the new network job
• updated several unusual hydrus 'static/test' requests used to test credentials and fetch access key and so on to the new system
• added a special 'test service' service to better accomodate these requests
• wrote a static login script for hydrus services
• polished up login management system overall
• import folders now support a 'check now' state, like subscriptions, that will cause them to check immediately
• import folders can be 'check now'ed from the file menu under a new submenu! if you would like to have a 'manual' import folder, try pausing it and just running it from this menu!
• added an optimisation to the file search algorithm to search ratings queries super fast when they lack tag or file system preds to otherwise speed them up
• updated the booru presets that now support https to be https
• added a 'restore defaults' button to the manage boorus dialog--you can restore specifics or all of them
• optimised how fading thumbnails are blitted to screen, which may provide a huge performance boost for high-res/small-thumb clients
• pages that have been renamed by the user will no longer be rename-overwritten by any auto-renaming system (currently just thread watchers, I think, but this will expand in future). unfortunately, this is not retroactive--only pages renamed from now on will be aware that they were user-renamed
• in advanced mode, the pages menu now states how many pages are currently open
• the 'page of images' downloader will now say '(x already in queue)' when it reports how many urls were found, if any were already in the queue. (this should clear up some confusion where it would previously say '0 new urls' even when it found some stuff)
• the gallery downloader will do essentially the same, but on a per-page basis. the text is a little crushed, so I may revisit this
• fixed an issue with the manage services dialog not being able to rename dupe-named services on edit subdialog ok
• the manage services dialog now uses the new listctrl! the listed services are no longer a horrific unsorted mess!
• fixed the file import status button not showing on raw url downloader or import folder edit dialog
• improved how some directory tree merging code deals with read_only files in the source
• maybe fixed some unusual selection behaviour with the booru selection popup dialog. it now has a real ok button, rather than a mickey-mouse hidden one
• completely deleted the old networking engine!
• cleaned out some unused imports from the networking code and related entries from running_from_source.html
• changed up and fixed some odd bugs in how how repositories test some error/isfunctional stuff vs regular paused status
• hydrus network contexts now have a prettier name
• ip address-based network contexts will no longer spam the bandwidth tracker with their useless subdomains
• network jobs that are unable to attempt validation or login will now error immediately rather than waiting indefinitely
• fixed up a bunch of test code that was broken due to the mismatch of network engines
• broke some other test code due to the network engine transfer!
• the client is more resilient about broken 'twisted' installs, and _should_ be able to boot without it. this may or may not apply to the built release--more work can be done here
• some network job refactoring
• clientside service code cleanup
• misc fix that I can't remember
• misc cleanup
• misc refactoring

• version 278

• fixed the tumblr raw url converter to now point to the data.tumblr.com domain
• added a hardcoded ssl verify exception for data.tumblr.com, which has an incorrectly defined ssl cert (at least for public-facing interactions), wew
• all existing db urls and file import cache urls for media.tumblr.com will be updated to data.tumblr.com on update! (everything _should_ just magically work again)
• fixed apng import, which the decompressionbomb detection code was not handling correctly
• collapsed the different instatiations of the 'file import status' button down to one class
• the file import status button now has a right-click menu that supports 'retry failures' and 'delete processed', if applicable
• misc import status cache cleanup and refactoring
• you can now edit or completely turn off the [404] and [DEAD] thread watcher page name prefixes under options->downloading
• thread watchers should more reliably keep 404 status
• 'open selection in a new page' now preserves file order!
• 'view this file's duplicates' now sorts the files!
• options->gui now has an option to change how often 'last session' is saved
• 'last session' will no longer autosave to the database if there are no changes
• tags exported to neighbouring .txt files are now correctly sibling-collapsed
• tags imported or exported via neighbouring .txt files are now correctly tag censored
• the manage tags dialog will now protest with a yes/no dialog if you attempt to cancel it with uncommitted changes
• the manage parents and siblings dialogs will now protest with a yes/no on an ok event if there are 'uncommitted' pairs in the lower boxes (e.g. if you forgot to click the 'add' button)
• fixed an issue that would sometimes stop old sessions from loading properly
• the duplicates page now does its maintenance jobs in modal popups!
• attempting to apply a duplicate status to more than 100 pairs now throws up a warning yes/no dialog
• the manage urls dialog now has copy/paste buttons
• added a (somewhat debug) option to disable the mouse hide&anchor behaviour on slow Windows canvas drags to options->media
• added a 'regen all phashes' command to the database regen menu
• the disk cache options in help now have a help button to explain good values for ssd vs hdd
• the edit bandwidth rules dialog now uses the new listctrl
• merged the old and new login managers
• misc login work
• misc refactoring
• misc cleanup

• version 277

• expanded the domain manager into a legit serialisable object that holds data and saves itself on changes. to begin with, it supports custom http headers for particular on network contexts
• the global User-Agent (which now defaults to 'hydrus client') is set through this new manager, as attached to the 'global' network context
• wrote a basic panel to edit custom http headers under services->manage network rules (this panel will be fleshed out with more in future--please ignore if you are not an advanced user)
• wrote a panel to edit an individual custom http header
• wrote a panel to edit/select network contexts. this will get a bunch more use elsewhere as the overhaul continues
• flushed out and tested the new popup yes/no button system
• sanakucomplex.com has a User-Agent entry in the new custom header system--on the first sank request, you will be presented with a yes/no popup asking if it is ok to use.
• if you ok the user-agent, sankaku now seems to work again!
• sankakucomplex.com network context will get some specific conservative bandwidth rules (80 rqs per 7m, 1 rq per 4s) on update
• added an option to options->gui to reverse the page tab shift behaviour
• 'don't scroll down on key navigation event if thumb is at least this % visible' value is now 75%. it is also editable under options->gui
• the hydrus icon used on frames is now the non-transparent version that shows up better on dark backgrounds
• the standard hydrus.ico used in web favicons and the executable builds is now also non-transparent (this should also propagate up to OS shortcuts and taskbar icons wherever the frame icon is not inherited)
• file export drag and drop events will now defocus the currently focused media on successful drop (e.g. if you drag a video to an external media player, it will stop rendering clientside), just like for open externally
• database analyze maintenance should be more reliable with respect to fresh repository syncs
• reduced default impermanent session timeout in new networking engine to 60m (should fix/reduce some hentai foundry 503s)
• misc ui improvements and speedup
• misc fixes

• version 276

• the new thread watcher object will no longer produce check periods shorter than the time since the latest file. this effectively throttles checking on threads that were posting very fast but have since suddenly stopped completely
• thread watchers now parse their thread subject and place this in the left management panel
• thread watchers now name their pages based on the thread subject, if one exists
• an option to permit or deny thread watchers renaming their pages is now under options->downloading
• dead and 404 threads now disable their checker pause button--to attempt to revive, hit 'check now'
• thread watchers now preface their page name with [DEAD] or [404] when appropriate
• misc thread watcher code improvements
• added basic import support for zip, rar, and 7z files. they get no useful metadata (yet) and have a default 'archive' thumbnail
• the client will now by default detect and not import decompression bombs before they blat your computer. an option to allow them nonetheless is under options->media
• the server will now not parse or accept decompression bomb uploads in POST requests
• added a 'refresh all pages' entry to page of pages's right-click menu
• added 'send this page down to a new page of pages' to page right-click menu
• added 'send all pages to the right to a new page of pages' to page right-click menu
• fixed a page of pages drag and drop issue when dropping the last page of a notebook onto the same notebook tab
• fixed some index calculation problems when DnDing page tabs to the right on the same notebook
• sending a refresh event to a 'show selection in a new page' page (which has no search predicates and so cannot 'refresh' its search) will now trigger a sort event (like importers got last week)
• thumbnails at the bottom of the current view but are at least 90% in view will no longer scroll into view when selected
• click events will no longer scroll thumbnails that are semi-out of view into view
• improved how all 'wait until the client ain't so busy' checks work. importers that have a whole slew of 'already in db' to catch up on should now not clog the gui so much
• similarly, under ideal conditions where nothing is busy, importers will iterate over their files more quickly
• the network engine now has a 'verification' loop that doesn't do anything yet, and a stub domain engine is generated to be consulted in this
• wrote some verification code, extended popup messages to support yes/no questions
• polished some domain engine code
• fixed an issue where file repositories were not recording deleted files in certain cases
• all file repositories will be reset on update
• the date entries on the review bandwidth bar chart now have leading zeroes on 0-9 months to ensure the sort correctly (this month's 2017-10 entry was sorting before 2017-8, wew!)
• the migrate database dialog now shows approximate total thumbnail size
• gave the migrate database help a quick pass
• gave the 'help my db is broke.txt' file a quick pass

• version 275

• if you hold shift down while dropping a page tab, the client will not 'chase' that page to show it (try it out!)
• the gui will be more snappy about dealing with drag-and-drop drops (of types file import, page tab rearrange, and url drop), returning the mouse to normal state instantly on drop and dealing with the event in a subsequent action
• dropping url text on the client will specifically inform the source program not to delete the dragged text (that the operation was copy, not move), if it is interested in hearing that
• page drag-and-drops should transition a little less flickerily
• all file import status objects can now track 'source time', typically to represent upload time
• file imports now populate 'source time' based on the earliest of creation/modified time! (this will be used later to parse as a tag)
• thread watchers now populate 'source time' based on post time!
• finished a watcher options object for the new thread checker system
• wrote a panel to edit watcher options
• converted the thread object to the new watcher system
• thread watchers now have two pause buttons--one for the file queue, one for the checker
• compressed thread watcher ui layout
• converted the thread left-panel ui and options->downloading page to reflect the new watcher system
• improved the watcher options to generate better timings for fresh threads
• cleaned up some thread watcher check time code
• the total/selected mime summary on the status bar is a little prettier and will now report by individual mime sometimes
• generating the total/select mime summaries are now faster on pages with >1,000 files (it'll just use 'file')
• a 'refresh' action on an import page now triggers a sort event
• added 'flip_darkmode' shortcut command to 'main_gui' and 'media_viewer' shortcut sets
• added copy_file/path/hash/bmp actions to the 'media' shortcut set, removed the hardcoded ctrl+c for copy_file. I added ctrl+c to the defaults, but existing users will have to re-add it manually if they want it!
• simplified some page ui update code
• import pages will no longer update their left-panel ui (which uses a bit of cpu) when they are not in view
• polished some new string and url matching code the domain engine will be using
• wrote a panel to edit string match objects
• wrote a new panel to handle simple ordered lists of data in a better way
• wrote most of a panel to edit url match objects
• misc domain manager work
• fixed an issue with the old listctrl where object name de-duplication was sometimes not permitting (1)-type names to be cleaned up
• fixed some inelegant time duration->text conversions
• improved complete process shutdown reliability when some downloads are waiting on bandwidth on shutdown
• did some misc listctrl update work

• version 274

• the help menu now has an easy on/off check entry for the darkmode colourset
• changing any individual colour or the entire colourset will now immediately refresh almost all custom-coloured controls with the new colour
• added a BUGFIX option to options->gui to permanently fix all discord file drag-and-drop events (as long as they contain <= 10 files and total < 50MB)
• you can now set specific 'open externally' launch paths on a per-mime basis under options->files and trash
• improved error reporting on a bad file launch
• improved the network engine to recover from and reattempt in-progress response read errors (previously reported as ReadTimeout)
• fixed the 'scroll to focused thumbnail' calculations on key events (when hitting up/down arrow key on thumbnail grid, the page wasn't scrolling correctly as needed)
• known urls no longer display with the scheme (http or https) in the media viewer top-right summary
• known urls in the media viewer top-right summary now tooltip their full url
• wrote a new button for editing tag import options. it has a good summary tooltip. it is only in the manage import folder dialog atm, but I will replicate it across the rest of the program in the coming weeks
• in some situations, the file import status window will list some timestamp note info for 'already in db' and 'deleted' statuses. see if you like it
• the new listctrl will generally give its data to other consumers in ui sorted order (this fixes some stuff like 'copy sources' in the file import status window, which was copying them in random order)
• manage tag parents now uses the new listctrl (and is hence now safely sortable)
• manage tag sibings now uses the new listctrl (and is hence now safely sortable)
• some behind-the-scenes of manage parents/siblings is a bit neater
• improved some thumbnail internal media structures to fetch specific media based on hash much much faster, particularly for pages with 10,000+ thumbs. this should speed up large imports and other content update events that can result in thumb redraws
• fixed an issue where dismissing a popup message could spawn the entire result of the queue, ignoring the 'show 10 max' rule
• completely finished the menu rewrite! all menus now work on the new system
• deleted a ton of old and now obselete menu event processing code
• some boot/shutdown terminal printing should be more reliable
• misc refactoring
• misc cleanup

• version 273

• converted the gui colour options to the new options system
• added a 'darkmode' colour set and extended the colour options page to support it!
• added a secret discord-compatible drag and drop mode. start a file transfer drag and drop with ctrl held down to initiate. keep ctrl held down, as this is secretly a 'move' drag and drop, and you want to keep it a 'copy' one. it should work for discord dnd, at least to the web version. I have an idea on how to do this better, but feedback on this would be appreciated.
• converted 'import options - files' collapsible panel to a button that launches a dialog. this reduces total gui object count when many importers are open and reduces overall ui lag and limits. the button shows a summary of its current options as its tooltip. furthermore, all of this sometimes confusing nomenclature is now uniformly presented as 'file import options'
• added a BUGFIX option to gui options page to force hover windows to display at all times
• improved last week's gelbooru 404 fix--page urls generated pre-v272 should now be more reliable
• in a further attempt to improve workspace support, popup messages will no longer update in any way if the mouse is not on the same display as the main gui
• fixed an issue where dupe import urls could be queued up in a file import queue if they came in the same batch. this was essentially harmless but lead to some mixed x/y progress counts and row indices
• fixed an issue where url drops were not filling in the url entry when a new raw url page had to be created on a currently selected page of pages. this may have affected a couple of other page spawning situations
• may have fixed an issue with the mouse shortcut-setting button, where for some users scroll events were not registering
• manage subs dialog now has the new help button up top
• manage subs dialog's sub verbs are now wrapping into a single menu button that I can expand more easily in future
• sped up thumbnail access a bit on certain well-defragged hard disk drives
• made thumbnail fading a little smoother in some situations
• rendered images that aren't otherwise pushing any memory limits will now stay cached for less time (10 mins) while thumbnails will hang around for longer (24 hours)--this was previously 20 mins for both. you should now see less thumbnail page 'refreshing' after returning to the client after inactivity
• sketched out basic classes for login and domain engines
• sketched out basic class for url matching
• sketched out basic classes for matching and transforming strings in certain login/domain situations
• updated the manage export folders dialog's listctrl to the new control
• updated the last media viewer menu code to the new system
• removed some old proxy code that was sometimes intruding on the new networking engine. proper proxy support will come in a later version
• fixed some test code
• misc refactoring

• version 272

• finished moving the last misc network consumers to the new networking engine
• greatly simplified and harmonised some of the new network job response processing
• the gelbooru 404 issue should be fixed!
• added a parsing patch for rule34hentai.net's mp4 links, which was causing 'link not found' failures on the hydrus parsing end. please tell your R34 subs to retry their fails
• wrote a patch (actually updated an existing patch that had fallen to bitrot) for the danbooru booru (and extended it to anything else with 'Running Danbooru' in its html) to try to fetch the full-size image if one exists. this may not last long, so I do not recommend regular users rely on it, but feel free to play with it until I have a better fix in the new downloader engine
• you can now add 'thread watcher' to the default import tag options. it has 'filename' and explicit tags options. unlike gallery downloaders, thread watchers will not inherit from the 'default' import tag options entry--you need to set the specific entry
• added regular number key support to the new shortcut system
• fixed the borked splash screen drag coordinate calculation
• heavy session loads should be more polite to CPUs and window managers
• pages should generally spawn with their proper name (rather than a flicker of 'page') no matter their spawning context
• fixed osx page tab drag-and-drop coordinate calculations
• figured a way to make os x page tab drag-and-drop start on a single left-mouse-down event
• in an effort to improve popup display on systems with virtual desktops, the popup message parent window will now not go from hidden to shown while the mouse is on a display screen other than the main gui
• fixed a wacky bug where clicking on a page tab, then on its sort dropdown, then on a different tab, would invoke the Deep Ones to commence a page tab drag and drop event
• fixed an index issue when unclosing some pages in certain orders
• increased pre-processing disk cache population time
• tweaked regular memory maintenance disk cache population timings
• fixed shortcut-driven tag petition events when the tag has a sibling (e.g. if you have sibling pair (a->b) and a shortcut (key->flip a), hitting 'key' will now correctly flip a (appearing as b) on and off in the media viewer)
• fixed some ugly layout sizing calculations on the path tagging dialog
• updated the path tagging listctrl to the new control
• updated the quick namespaces listctrl to the new control
• updated the regexbutton's menu to the new menu system
• fixed some typo errors with the regexbutton's entries!
• removed the old gallery downloader unit tests, as these operated on the old network and downloader engine--new, granular tests will be reintroduced for the new parsing components as they are written
• cleaned up some shortcut key code
• misc refactoring
• misc improvements

• version 271

• fixed an issue where the notebook 'motion' event was being consumed, disallowing the regular OS 'highlighting' on tab mouseover
• fixed page tab drag and drop crash for Linux
• fixed page tab drag and drop coordinate calculation for OS X
• page tab drag and drop should now work for all platforms
• new 'page of pages' pages now start with a blank file search page
• fixed several issues with row display in the new listctrl
• added a # index column to the file import status window, so you can resort the 'to be imported' order reliably
• fixed an issue where the 'delete_file' application command was not firing on an appropriately mapped shortcut
• fixed an issue where automatic bandwidth override would only fire if its page was in view. it now occurs at all times
• fixed an issue where the file import status frame, as launched from manage subscriptions, would sometimes be non-responsive (typically for Linux)
• fixed the 'pattern shortcuts' button on the export files dialog, which was misbehaving for certain choices for some window managers
• fixed an unusual error where middle-clicking to create a new page while a menu was open would lock the application from receiving new key/mouse events until program focus was lost, wew lad
• updated some old network code to the new engine
• added some delay to network connection error reattempts so that instant fails due to local network disconnect (e.g. while computer is waking from sleep) do not consume all the reattempts too quickly
• greatly simplified low-time-delta bandwidth testing and reporting, improving essential accuracy and overall feel
• gallery download errors will now print additional info to the log--let's see if we can gather more info on the gelbooru 404s
• wrote a patch for users of 2.X opencv who were running from source, who were unable to boot
• due to hardware failure, had to make a new Linux build computer. some libraries are updated, but build _should_ be the same--let me know if not
• updated to opencv 3.3 for all releases
• misc old menu code updates

• version 270

• added page tab drag and drop (windows only, have to iron out critical bugs for linux/os x).
• dragging onto the middle of a normal tab will put the source tab there
• dragging onto the edge of a tab will try to insert the tab there
• dragging onto the middle of a page of pages tab will insert the tab onto the end of that page's list
• dragging a page up a notebook level will work ok
• dragging a notebook into itself will do nothing and not crash the client :^)
• new pages that come from the main gui level (such as from pages menu or file import) now open in the deepest open notebook (previously, they would always appear in the top row)
• fixed some misc page of pages bugs
• fixed a bandwidth calculation that meant 1s time delta rules were working at 50% capacity (e.g. 1rq per 1s rule for domains were actually running at 1rq per 2s)
• improved a bandwidth estimate calculation that was cutting out early in some situations for large time deltas
• tag manager's page up/down shortcuts no longer mistakenly navigate the archive/delete filter
• added a network timeout option to the 'connection' options page
• reduced some hover window show/hide flicker when the media viewer is fullscreen and the OS has a taskbar that pops in for non-fullscreen windows (this mostly affected Linux)
• fixed a long-time issue with yes/no dialog layout
• improved some rendering of EXIF-rotated and -flipped jpegs. I expect to add more support for this (and retroactive image metadata-parsing to figure out correct reversed resolution--atm rotated images remain stretched) in the coming weeks
• fixed an error when cancelling the booru-picker dialog from the page chooser dialog
• if no files or all files are selected, the 'invert' select choice will no longer be shown (in this case, it redundantly does the same job as 'all' and 'none')
• fixed an issue where setting namespace sort as default would persist through a client reboot
• fixed an issue where force idle debug mode was not waking sleeping daemons
• increased frequency of mappings processing reporting
• wrote an exception and the needed maintenance code for the 'repairfilesystem' dialog to allow a proceed action if the only remaining incorrect paths are thumbnail paths--in this case, the client will create empty prefix folders and prompt the user to regen thumbnails
• export filenames are now clipped to make <255 total characters in path
• removed the old 'processing phase' option, which was no longer in use
• removed the proxy settings from the 'connection' options page--the new engine does not use this old system, but if there is demand, a more flexible system will return
• misc image rendering pipeline updates
• misc improvements

• version 269

• nested pages now supported!
• moved all page management (session load/save, new/close page, page navigation, page name maintenance, etc...) code from the main gui to the new PagesNotebook object
• expanded the session object to hold nested page information
• added a 'page of pages' page to the 'special' new page entry!
• numerous other gui-notebook page-related event fixes and improvements
• figured out cross-platform menu and other mouse event support for nested notebooks, but there may still be holes--please let me know if your new pages ever appear in the wrong tier!
• ways to move pages up and down rows will come in the coming weeks!
• main gui status bar now shows total bandwidth this session as well as current speed
• added a cog button to network job controls that allow for manual- and auto-bandwidth-override
• added a 'blocked?' column to the review bandwidth panel to quickly see which network contexts are currently available to do work
• added a button to reset 'default' and 'global' bandwidth rules to review bandwidth panel
• merged the network request start test and consumption code into one transaction, stopping some accidental overconsumption when the engine was under heavy load
• did some logic work to make sure unusual network context rule/usage situations are visible on the review bandwidth panel for editing
• network jobs now report more information as they get ready to start, including while they are held up waiting for a download slot
• added a simple 'network profile mode' to the debug menu that atm just prints a summary of new jobs
• fixed an error that could sometimes be a crash when 'review services' was opened while no pages were open in the main gui
• fixed the pixiv login test button with a hacky workaround--I will make extensive proper login testing gui when I move to the login engine
• fixed an issue in the youtube downloader
• if the bandwidth or session managers are missing on boot, empty defaults will be created in their stead
• bumped the max page limit up to 150--I expect to increase it more in the coming months as I rejigger how some gui stuff is laid out
• renamed 'sort by age' to 'time imported'
• fixed and improved some test code
• cleaned up some shutdown code
• should have fixed a rare unicode conversion issue when printing to log
• misc improvements

• version 268

• split the sort dropdown into two, splitting the sort type and sort order
• file sorting works more intelligently behind the scenes in several ways
• added 'sort by number of tags' to sort options
• session pages now remember their sort status!
• sessions also more reliably remember their actual thumbnail order for all pages (typically, this matters for importers atm)
• network job controls now report an estimate of how long they will have to wait for bandwidth
• subscriptions will now show some 'no more bandwidth to download files' text if they have to stop because of bandwidth rules
• subscription 'should I start/continue' testing now has a little padding to forestall some potential unexpected long delays in operation due to edge cases
• the edit subscription dialog now uses the new file import status control
• fixed an issue with hentai foundry filters not applying (they added some categories since the downloader last worked, breaking the POST form), hence hiding most non-vanilla results
• finished the 'database->migrate database' dialog and its help and removed the 'under construction' labels. the new help page is also now linked from the standard index.html
• fixed an issue where pages would sometimes never initialise (due to being queued up after an infinite job like the network engine's mainloop!)
• improved how all long-job threads are spawned
• improved some more thread scheduling logic that meant some long-term jobs could be stacked undesirably
• expanded how the new listctrl updates and deletes its data
• reduced flicker on the new listctrl update events
• the file import status window now uses the new listctrl
• the manage subscriptions dialog now uses the new listctrl
• collections now track their tags more accurately
• moved the 'delete original files after success' checkbox up to reduce misclicks on ok'ing the file import dialog and added a bit of red warning text whenever it is on
• sped up some behind-the-scenes content processing for large pages
• improved how the program cleans some things up during exit
• the issue with some clients not clearing their exit splash screen until a mouseover event should be fixed
• at 120 open pages, the client will inform the user about the approaching max page limit
• by default, import folders no longer delete anything. default is to ignore original files in all cases
• deleted some old unusued code
• misc cleanup
• misc improvements

• version 267

• drag-and-dropping a 4chan or 8chan url onto the client will now automatically open a thread watcher for that url
• fixed an issue where web domain or subscription network contexts that included unicode characters in their context data were unable to serialise and save to the db, causing error spam when the bandwidth manager attempted to save itself
• subscriptions will now show a network job control in their popup as they do network work
• subscriptions will cancel more reliably during gallery parse
• subscriptions now have some 'delay' logic that will stop them sometimes restarting as soon as they are cancelled or otherwise have to stop mid-work
• subscriptions will now tolerate up to five 'already in cache' urls per page parse until it considers the page 'already seen'. this is to catch the odd additional late insert and avoid the problem of a page updating and shuffling everything up one as the subscription walks through pages
• maintenance modal popup messages should no longer appear if their jobs are very quick
• queued up modal popup messages will no longer flicker their dialogs on a de-minimise
• should have fixed an issue where modal popup messages could sometimes error out on a close attempt, locking the client's whole gui and requiring a force quit
• fixed an issue where the modal popup message dialog was OKing on a close attempt of a non-cancellable job, despite presenting user text indicating otherwise
• did a bunch of data work on pages
• page tab names will now be clipped to 20 characters by default
• pages now show (num_files) after their name by default, although you can set this for only import pages or turn it off completely
• options for these new behaviours are in options->gui--and page names will update immediately on dialog ok
• subscription and import folder 'show files' events (and a couple of other misc occasions) will now launch their page tab with the sub/folder name rather than a flat 'files'
• wrote a new listctrl class with custom sort code that does not suffer from the 'crash when sorted while many pages open' bug. it also handles data in a simpler way for hydrus
• review bandwidth listctrl now uses this new class and should now not crash your client. I will replace all the other listctrls with the new class over the coming weeks
• temporarily, no old listctrls will auto-sort themselves (as this causes a crash for many users)! if you do not suffer from the crash, please sort them yourselves for now
• reduced some large-scale gui import lag:
• reduced content processing CPU load on clients with many thumbnails open
• massively reduced content processing CPU load on non-current pages
• offloaded newly-imported file thumbnail detection and generation to a non-gui thread
• migrate database dialog now lets you move the whole database and all portable locations, which requires a client shutdown
• the raw url downloader will no longer have a problem with pasted url lists that include empty newlines
• all the downloaders/importers now sleep on cleverer event objects so they will burn less idle cpu and wake as soon as they have new work to do
• fixed and otherwise improved some gallery downloader timing logic
• hdd import pages now use the file import status control
• the thread watcher's controls now try to wrap in a single sizer. it uses less space, but might sperg out, let me know if it is a problem. I'll replace the whole watcher timing system in the new engine anyway
• fixed a gelbooru (and possibly others) booru parsing bug that meant half of the pages in the gallery walk were being skipped (e.g. 'mogudan' was producing ~350 files when there were actually 690 in the list)
• fixed some dialog panel layout scrollbar-cutoff in sevaral Linux places and perhaps elsewhere
• cleaned debug menu a little and added a save 'last session' entry
• fixed an issue where some kinds of media would error on notification of new url association
• clients will now save small transaction progress reliably within ten seconds no matter how idle they are
• deleted the old 'gui capitalisation' option--I never got around to expanding it beyond a handful of menu labels, it was always too much to work on for too little reward
• improved support for certain broken videos--these will import ok, but full rendering might be borked to different degrees, so let me know how it goes.
• cleared out some old content processing code
• cleared out some old unused db data
• removed all old 'waiting politely' download settings and gui code
• misc prep work for wx update
• misc gui code refactoring
• misc downloader cleanup and timing tweaks
• misc improvements

• version 266

• converted gallery downloaders to the new network engine
• greatly simplified how gallery downloaders report network activity and converted them to show the new network job control as well
• subscriptions also work on the new engine but will not show network gui yet
• hacked hentai foundry and pixiv login to use the new network engine
• successful logins to hf or pixiv now print to the log
• the new network engine now clears temporary session cookies after 90 mins of inactivity
• gallery downloaders and subscriptions now use the new 'downloader instance' and subscription bandwidth rules. by default, this means downloaders will do small bursts every five minutes and that subscriptions will do at most 256MB per day
• subscriptions' bandwidth use is now listed by name in the review bandwidth panel
• subscriptions use a new bandwidth test to determine if they should start or continue based on current bandwidth limits. it should mean subs do a good bit of work and then stop when they are supposed to without ever waiting on bandwidth more than 30s or so
• thread watcher and page of images now ignore bandwidth limits when doing their 'page' fetching part
• gallery page fetching will ignore bandwidth rules (in order to stop gallery walk desyncs from having to wait a long time)--it will fetch one page per five seconds
• the thread watcher and page of images importers now work on their files and page-checking simultaneously--also, the page of images will process its page queue at any time, not only when the current file queue is finished
• the gallery downloader page now works on its files and gallery page parsing simultaneously!
• wrote a 'file import status' control to better wrap up the import summary, progress gauge, and file import status button into one panel
• thread watchers now use the new file status control
• 'page of images' downloaders now use the new file status control
• the gallery downloaders now use the new file status control
• all network jobs will now retry up to four times on the BadStatusLine ConnectionError, which seems to be a TLS (https) negotiation timeout/remote termination
• all requests on the new network engine will now timeout after ten seconds
• they will also retry on generic timeout errors
• popup messages can now be shown in 'modal' mode as a dialog that prohibits interaction with the rest of the client.
• these will not boot while the client is minimised
• database maintenance routines will now all publish messages like this
• 'migrate database' panel now publishes a modal message when it does a file rebalance
• rewrote the controller-side pubsub pipeline to respond faster and consume fewer program resources, particularly for the client
• import pages now update themselves in a less spammy way behind the scenes, meaning more active pages can be open at once without them stepping on each other and clogging things up
• simplified how importers set their status information in several ways
• reduced a swath of pubsub spam related to content updates
• improved how spammy small jobs are written to the profile log
• reduced text flicker on all download pages
• more misc pubsub improvements
• reduced some gui update/refresh spam on hidden pages
• cleaned up a bunch of database->gui message reporting and cleanup code
• added close other/left/right pages to tab right-click menu
• the top-right media hover window will no longer refit-flicker on a ratings change
• wrote a new panel wrapper for listctrls that handles the underlying row of buttons in a neater way and automatically disables them if they are nullipotent (mostly, this means greying out 'delete' buttons when nothing is selected)
• several listctrls use this new panel
• when it is not strictly necessary, videos that are >30MB will no longer use the CPU-expensive manual frame count parsing
• a problem where newly reloaded thread watchers could sometimes stick in a 1/1 initialisation state _should_ be fixed
• fixed gelbooru url parsing (they stopped using the janky redirect.php urls)
• fixed an issue that meant ipfs pin was erroring when trying to show gui-side
• 'page of images' downloaders now protest if they are told to close while working
• some small layout and status text fixes
• adminside petition processing now has a 'flip selected' box to flip checked status of all selected contents
• adminside petition processing contents chechklistbox now supports multiple selection
• improved the reliability of some shutdown code
• cleared out some old unused code
• misc new control cleanup
• misc fixes
• more misc fixes

• version 265

• the bandwidth engine now recognises individual thread watcher threads as a network context that can inherit default bandwidth rules
• tweaked default bandwidth rules and reset existing rules to this new default
• review all bandwidth frame now has a time delta button to choose how the network contexts are filtered
• review all bandwidth frame now updates itself every 20 seconds or so
• review all bandwidth frame now has a 'delete history' button
• review all bandwidth frame now shows if services have specific rules
• review all bandwidth frame now has an 'edit default rules' button that lets you select and set rules for default network contexts
• review network context bandwidth frame now has a bar chart to show historical usage!
• bar chart is optional based on matplotlib availability
• review network context bandwidth frame now lists current bandwidth rules and current usage. it says whether these are default or specific rules
• review network context bandwidth frame now has a button to edit/clear specific rules
• rows of bandwidth rules and current usage, where presented in ui, are now ordered in ascending time delta
• misc bandwidth code improvements
• client file imports are now bundled into their own job object that generates cpu-expensive file metadata outside of the main file and database locks. file imports are now much less laggy and should generally block the feel of the ui much less
• removed the database 'rebalance files' menu entry
• removed the 'client files location' page from options
• db client_files rebalance will no longer occur in idle or shutdown time
• (this stuff is now handled in the migrate database dialog)
• 'migrate database' now uses a dialog, meaning you cannot interact with the rest of the program while it is open
• migrate database now has file location editing verbs--add, remove, +/- weight, rebalance_now. thumbnail location and portable db migration will be added next week
• flushed out the backup guide in the getting started help, including to reflect the new internal process
• the client now saves the 'last session' gui session before running a database backup
• the shutdown maintenance yes/no dialog will now auto-no after 15 seconds
• gave status bar tabs a bit more space for their text (some window managers were cutting them off)
• tumblr api lookups are now https
• tumblr files uploaded pre-2013 will no longer receive the 68. subdomain stripping, as they are not supported at the media.tumblr.com domain (much like 'raw' urls)
• pages will now not 'start' their download queues or thread checkers or whatever data checking loops they have until their initial media results are loaded
• key events started from an autocomplete entry but consumed by a higher window (typically F5 or F9/ctrl+t for refresh or new page at the main gui level) will no longer be duplicated
• fixed a shutdown issue with network job controls that could break a clean shutdown in some circumstances
• if the user attempts to create more than 128 pages, the client will now instead complain with a popup message. Due to OS-based gui handle limits, more than this many pages increasingly risks a crash
• if the client has more than 128 pages both open and waiting in the undo menu, it will destroy the 'closed' ones

• version 264

• converted the page of images downloader to the new network engine
• converted the thread watcher downloader to the new network engine
• rejiggered page of images and thread watcher management panels to better separate the two loops they work on
• rejiggered some more downloader layout stuff in prep for a new control to better display url cache and overall import status--downloader-global pause button is now always at the top, for instance
• removed 'waiting politely' indicator from download pages that use the new network engine
• network job controls now have a simple border
• network job controls will show current bytes progress and speed in a more intelligent and useful way--no longer spamming the initial 0KB/s, for instance, and flashing more useful summary information when downloading many small files
• network job controls now display their current bytes progress and speed on a separate right-aligned text beside the main status text (to stop the speed text jumping around so much)
• network job controls will not show the current download speed until some bytes are actually read
• wrote a first version of 'services->review bandwidth' review frame, which lists all network contexts with data in the past month
• wrote an independant and live-updating review frame for indivdual network contexts, launched from the review bandwidth frame, which shows current and historical bandwidth use for that context
• the main gui status bar now reports current bandwidth usage! updated once a second
• database->backup process now has some nicer gui, better and more reliable workflow, improved file copy notification, and now correctly obeys a popup-cancel command
• started the new 'migrate database' gui under the database menu. it currently shows live usage and will gain some verb buttons to atomically alter things hopefully next week
• moved from a strict begin/commit db model to a save/release system that only commits changes to disk at most every ten seconds or so. small 'write' transactions will now often have no disk-sync overhead and so are super fast, particularly when they come in a large batch.
• simplified database connection, disconnection, begin, commit, and rollback code significantly
• the url table in the client database is now non-unique (i.e. multiple files can have the same url). this may lead to undesired 'already in db' or 'deleted' statuses in some downloaders as they check for url status before starting their downloads. if this change turns out to a be a big pain in the future for pixiv manga or whatever, it may be revisited.
• fixed a bug in deviant art parser that sometimes meant nsfw urls could not be found
• fixed an issue where progress gauges could sometimes get stuck in a pulsing state
• fixed a bug with the new optimised result building which sometimes occured when all the files in a batch had common ipfs service membership
• improved support for unusual videos
• improved accuracy of duration calculation for unusual videos
• improved database optimisation after initial and ongoing repository processing
• a 'wake subscriptions' event now occurs even on 'manage subscriptions' cancel (meaning temporarily-dialog-paused subs will always continue after the dialog is closed)
• added a careful manual commit to stop the possibility of the definition-desync some users noticed after a power-loss during a big repository sync commit
• removed some redundant old db compatibility code
• permitted sqlite installations that allow in-memory temporary tables to use them (but not for vacuums, which tend to be a bit big for this stuff)
• cleaned up some redundant timer id code
• misc improvements
• misc refactoring

• version 263

• greatly improved how gui sessions are loaded--now the page tabs are loaded instantly, but the thumbnails are loaded in the background. session loaded should be significantly less laggy and buggy
• the issue of pages sometimes initially sizing at zero size (which could be caused by minimising the client while a session load was ongoing) should hence be better/fixed completely!
• gui sessions will now load their files in the exact order in which they were saved--behaviour that I think was not previously always reliable
• more general network code work and polishing
• added and improved unit tests for network code
• improved how short-time-delta data bandwidth is reported
• improved how short-time-delta data bandwidth is tested
• wrote a networkjobcontrol to display and control the new network job object
• tumblr parser now produces 68.-less urls
• tumblr parser now produces https urls
• cleaned up tumblr parser a little
• url caches will clip existing tumblr urls of the 68.-subdomain and convert to the new raw format, clearing out dupes along the way
• url caches will convert existing tumblr and pixiv urls to https, clearing out dupes along the way
• the pixiv parser now deals with missing creator/title tags without errors
• extended the specific file domain tag cache to also store deleted mappings, resulting in much faster request building for clients with large numbers of deleted mappings
• improved some downloader page queue text display timing
• added support for more types of mp4 file
• improved how some memory maintenance calls work
• improved how hydrus datacaches track their recently-used-data fifo list
• pages now regularly clear out spare thumbnail canvas bmps
• pages now regularly clear out cached thumbnail canvas bmps when they are not the currently viewed page
• import caches, when asked for url membership, will now test both the http and https versions the url
• maybe improved how 'open in file browser' works in windows
• fixed the 'recount video frames' advanced thumbnail menu entry, which wasn't working with the new ffmpeg wrapper
• moved some bloaty master hash data out of client.db and into client.master.db
• pubsub profile will no longer profile 'message', as it just makes for spam
• reduced some serverside pubsub spam
• reduced some significant clientside pubsub spam that I think was smashing the gui event loop at inconvenient moments
• improved some client shutdown object sync code
• fixed an issue where some duplicate maintenance popups would not clear themselves up properly if interrupted mid-job
• cleaned up some http-https conversion and comparison code
• fixed some status-setting code that meant thumbnail pages were sometimes setting status after they were replaced and scheduled for deletion
• misc improvements

• version 262

• added apng support!
• sessions will now append from the default insertion index (e.g. right of current page)
• sessions will now load in the correct order if the default insertion index is to the left of the current page!
• fixed an issue with custom 'favorites' gelbooru parsers
• manage parents/siblings dialogs now load their larger datasets on a thread, after initialising
• sped up how some default thumbnails are generated on client boot
• sped up how resized thumbnails are generated
• refactored ffmpeg video parsing to be simpler and easier to maintain
• improved some ffmpeg parsing to be much faster
• ffmpeg can now parse video that reports no duration
• fixed some more bad video framerate parsing
• the 4chan/8chan thread watcher will now always generate https urls (even if you enter an http thread url)
• pixiv will now produce https urls
• the known url file status system will now check both http and https versions of an url it is given
• refactored some core elements to simplify common controller requests
• started on some database migration help and ui--should be finished next week
• flushed out network engine a little
• more refactoring and simplification of new network engine object coupling
• flushed out bandwidth manager
• misc logic tweaks in bandwidth management
• wrote a network session manager
• wrote a new class for identifying all future 'network contexts'
• employed NetworkContext everywhere in the new engine, massively simplifying several things
• wrote the first batch of tests for my new network engine and its network jobs
• wrote tests for the bandwidth rules object, including covering the new short time delta support
• improved reliability and speed of bandwidth unit tests
• wrote tests for the new domain-based bandwidth manager
• misc improvements
• more misc improvements

• version 261

• wrote a new manage dialog for urls
• added 'manage known urls' to media right-click menus!
• double-left-clicking on any video animation will 'open externally'!
• added an option to options->gui to set where new pages will appear by default--either far left/right or left/right of current page
• the tumblr parser now produces '_raw' urls when the post was posted in 2013 or later
• created a new 'number' subtag cache that will be populated on update
• created a new 'tag as number' system predicate that can search for, say, all 'page:' tags > 200
• bandwidth management now tracks requests and num_bytes more sensibly
• bandwidth tracking objects can now better handle bandwidth usage and rule application in short intervals (i.e. sub-5-second) (however, the current networking engine cannot yet use this information accurately)
• wrote unit test for bandwidth tracker, including for the new short interval timing
• fleshed out the new network engine
• fleshed out the new network job
• thread watchers can now have a time delta that includes days (giving a max check period of 360 days, wew)
• adminside mapping petition processing now has adaptive max total petition weight--lower file count range petitions will have much higher max total permitted weight
• added a new 'callto' debug reporting mode that reports on current thread pool jobs
• improved calltothread pre-spawning checks to reduce outside chance of deadlock in busy periods
• the advanced review services repository panel buttons are now hidden unless in advanced mode
• eliminated some animation buffer looping redundancy
• fixed a little animation next-frame prediction code
• fixed up some '1 minutes'-type time_delta->string conversion
• fixed up label on time delta control button
• fixed some shutdown thread interactions
• fixed an issue where sometimes empty tags could be entered into the manage tags panel
• added some pydeadobject error handling during client shutdown
• refactored some db multi-tag->file search code, cleaned up wildcard searching
• misc string cleanup
• misc dialog cleanup

• version 260

• fixed video parsing when the video metadata includes random non-utf-friendly garbage
• fixed video parsing when ffmpeg reports no fps at all
• improved video frame counting accuracy
• thumbnail waterfall process now adapts to the current speed of the cache and can hence use significantly less overhead
• the thumbnail fading process now adapts to the current rendering speed, delivering smooth fade when CPU is available but otherwise skipping frames and more reliably filling in thumbnails within a quarter of a second
• canvases now draw their thumbnails with slightly less overhead
• increased database synchronous pragma to FULL to better safeguard against power loss during multiple-db commit--we'll see if it slows things down too much, and maybe add an option about it
• cleaned up and improved some client gui pubsubs
• made pubsub profile mode far less popup-spammy
• if a profile takes less than 20ms, it now won't be fully printed to the log
• tweaked some more server object maintenance code
• added 'changelog' link to help menu
• updated lz4 library and fixed some old deprecated calls
• misc serverside pubsub cleanup
• misc fixes
• misc refactoring
• misc code cleaning

• version 259

• planned out new networking engine and started the principal objects
• renamed the 'exact match' duplicate status to 'same quality', to reduce confusion on what is appropriate for this status
• the duplicate filter, on hitting the delete key, now offers the option of deleting both files
• the duplicate system now combines duplicate status setting and the consequent batch of content updates into the same database transaction, speeding things up
• the duplicate system now batches multiple duplicate status setting into a single transaction, massively speeding up large filters or thumbnail status set actions
• misc duplicate help tweaks
• you can now edit the default duplicate merge options from the new thumbnail duplicate menu
• the duplicates page's jobs are less demanding on gui time and take better breaks if something else happens
• renamed the new dupe system predicate to 'system:num duplicate relationships' to clarify what it searches
• for normal queries, current and pending mappings are now fetched from a faster mappings cache. you should see faster result building across the board, particularly on fresh boots or otherwise slow-disk-access systems
• added a prototype 'advanced mode' (defaulting to off, so experienced users will want to turn it on) under the help menu that will enable menu items that are often not helpful to new users. I will add more things to this in future, suggestions welcome
• the new thumbnail menu dupe relationship set stuff is now considered advanced
• thumbnail menu find similar files is now considered advanced
• the thumbnail menu copy hash entries are now considered advanced
• advanced content update buttons (on manage tags and review services) are now considered advanced
• added an advanced mode 'open file location' entry to the thumbnail share menu, which will open the file in your OS's file explorer (not available on Linux)
• added an advanced mode 'correct video frame count' thumbnail menu entry that will force-apply last week's more accurate video frame counter to correct and videos that render too fast and then cut off
• fixed many entries on the media viewer menus, which were being blocked by an over-eager 'can continue' test and hence silently failing
• fixed the youtube downloader on Linux and OS X--both now use youtube-dl
• fixed an issue where the GetLighterDarkerColour function was producing very bright alternates to very dark colours (meaning dark dupe filters were having their bright background text rendered unreadable)
• improved video frame number parsing accuracy
• improved the accurate version of video frame number parsing accuracy, particularly for longer videos
• the network engine now reports 5xx http status codes as ServerException to better contextualise to the user what went wrong
• adminside mapping petitions are now sub-ordered by tag
• adminside sibling/parent petitions are now ordered by the 'older' tag and sub-ordered by the 'newer' tag
• censorship taglists are now roughly sorted
• reduced default shutdown work max time to five minutes
• improved how subprocesses are started
• misc cleanup
• misc refactoring

• version 258

• added a duplicate entry to the thumbnail right-click menu
• the new duplicate menu will now attempt to fetch known duplicate counts for the focused file for the current file domain (if the db is locked, it will say so rather than block)
• the new duplicate menu's counts in the thumbnail menu can be clicked to show those files in a new page
• the new duplicate menu allows you to set the four main dupe statuses (with default merge options), or a customised verson of the same, or set unknown/potential, or delete dupe status completely, for all the possible pairs in the current selection
• added tentative support for variable framerate files, manually counting up their frames and displaying them with an averaged constant framerate in the client renderer. this now permits a number of webms that were previously 'mime unsupported'. let me know how it works for you!
• cleaned some server locking code, hopefully fixing the ssl handshake issue (which is actually a deadlocking issue)
• export file dialogs will remember the last value of 'export tags to .txt files?', and clicking that checkbox will always launch the dialog for editing (rather than clearing if there are some set)
• fixed the advanced content update (as used by service-wide update), which was not able to differentiate namespaces since the recent service changes. I apologise to anyone affected by this--I will add some unit tests to make sure it doesn't happen again
• reworked and cleaned some canvas event handling code
• the canvas will now accept mouse wheel events even when it does not have focus
• the canvas frame will no longer accept keyboard events (and hence activate fullscreen_switch) if the mouse is over a flash window
• hydrus servers now respond to /robots.txt with a 'disallow all'
• servers will now provide mappings petitions to admins in groups of similar size, rather than mixing petitions of weight 5000 with a hundred of weight 1
• fixed the back/skip buttons in the archive/delete filter's top hover window
• the file path tagging dialog's namespace and regular regexes now support parenthesis groups, and in that case will take each of those submatches instead of the wider match
• hence regexes with groups will now compile in that dialog
• you can now 'new page here' on the final (rightmost) page tab
• fixed an issue where media-viewer-launched manage tags frames would yield focus to the main gui if they had opened a sub-dialog while they were open
• the tag censorship edit panel now has a 'help here -->' label in blue text
• file imports will fail properly when the final temp_path->client_files file copy operation fails (due, for instance, to the destination being on a removable medium that was recently disconnected)
• fixed serverside upnp maintenance, which was silently failing early, and also updated it to the new server object code
• wrote a new checklistbox dialog that handles the underlying data explicitly rather than messing around with external text->data maps
• replaced all the old string checklistbox dialog instances with the new one
• subscriptions will now correctly clear their 'recent error' status on a successful sync
• refactored and cleaned how basic mediaresult objects store and consult simple file information
• locationsmanager objects now handle inbox status and the related content update events that affect it
• fixed a bug in HydrusTagArchive when attempting to figure out hash type from an existing hash
• removed the obsolete servertoclientcontentupdatepackage object
• misc duplicate code cleanup
• misc serverside service code improvements
• misc small fixes
• misc cleanup

• version 257

• the duplicate filter will now maintain zoom on files with the same ratio
• split the duplicate merge options into separate tag/rating controls--you may see some duplicate service entries, but these will be cleaned on your next shutdown
• duplicate merge options now allow syncing 'archive' status
• duplicate merge options now allow 'delete both files', which you may find use for in custom actions
• created a tag censorship object to handle and action a rich tag censorship ruleset
• tag duplicate merge options now have and use this tag censorship object to filter which tags are merged, with an initial value of 'let everything through'
• wrote a tag censorship edit panel and tied it into the duplicate action edit panel so these new tag censorship objects can be edited
• added an optimisation to the duplicate status setting code--if two files are better/worse, they are inherantly duplicates and so 'not dupe' and 'alternate' relationships apply to both equally and can be duplicated
• fixed and culled and normalised the 'this has more tags' dupe filter statements to be more accurate and useful
• added 'this has a larger filesize' type-statements to the dupe filter
• created a 'system:duplicate relationships' predicate that can find files based on how many duplicate relationships of a particular type they have
• cleaned up some misc duplicate filter code and added some tooltips to the top hover window dupe action buttons
• added 'move to left/right end' to main gui page tab right-click menu
• added 'new page' and 'new page here' to main gui page tab right-click menu
• you can now right-click for a menu from empty tab space on the main gui
• the main gui statusbar now updates more efficiently when under heavy refresh load
• the main gui statusbar now shows db read/write/commit status and sets the current db job summary as its tooltip--if you experience persistent hangs, please hover over the statusbar and report what you see!
• export tags to .txts checkbox will now default to 'all services on' when checked
• fixed the thread watcher, which was accidentally disabling its text input early
• 'similar files' searches launched from the thumbnail menu will now default to 'my files' file domain rather than 'all local files'
• downloader pages will now correctly sort their files on initialisation
• refactored and generally cleaned up some collect and sort code
• fixed some unlikely-but-possible collect/sort bugs
• fixed some bad layout in the top-right hover window that was making it grow unreasonably tall when many urls were shown
• on the different download import pages, the progress gauge that shows file download progress will now reset back to 0 as soon as the file download is complete
• fixed a problem where video imports with unicode characters in their path were failing to mime-parse
• improved the file import status update pipeline to better deal with large transactions (like skipping/deleting/retrying a thousand rows at once). all these big transactions should lag the gui far less
• improved some misc import status cache code
• made first step in a big size rewrite job that will size many elements according to local system font size rather than specific pixel values
• hydrus servers now explicitly default to TLSv1.2--we'll see if that clears up some of the handshake timeout problems we have recently seen
• cleaned up a bunch of possible pydeadobjecterrors when the new review services panel is closed
• improved and rescheduled gelbooru redirect url purge
• added a catch-and-recovery to hydrus network session initialisation, which may sometimes receive invalid data after a service deletion
• added similar catches to tag parent/sibling initialisation, which apparently can be vulnerable to a similar invalid data problem
• I think I cleaned up some more Linux ClientToScreen console errors
• refactored and cleaned some frame size event responsibility
• refactored and cleaned the panel and controls that display file import status
• did a little more menu code cleanup
• misc cleanup
• misc fixes

• version 256

• the duplicate filter now loads new pairs off the gui thread. it will display 'loading pairs...' during this time
• media viewers of all kinds are now more comfortable displaying no media (when this occurs, it is usually a frame or two during startup/shutdown)
• the duplicate filter now responds to any media_viewer_browser navigation commands (like view_next) with a media switch action
• you can now alter the duplicate filter's background lighten/darken switch intensity from its top hover window's cog icon
• fixed a bug in the new dupe pair selection algorithm that was preventing pairs from being presented as groups
• the duplicate filter will now speed up workflow by automatically skipping pairs when you have previously chosen to delete one of the files in the current batch
• auto-skipped pairs _should_ be auto-reverse-skipped on a 'go back' action
• added a |< 'go back' index navigation button to the duplicate filter top hover window
• the duplicate filter now displays several 'this file has larger resolution'-type statements about the currently viewed file. it lists them on the top hover window and in the background details text underneath
• the duplicate filter _roughly_ attempts to put the better file of the two first. this will always be indexed 'A'
• the duplicate filter now shows done/total batch progress in its index string--not sure how clear/helpful this ultimately is, so may need to revisit
• an unusual bug where Linux would spam the 'All pairs have been filtered!' duplicate filter message over and over and then crash _should_ be fixed--the filter no longer waits for that message to be OKed before closing itself
• drag-and-dropping text onto the client will now a) open a url import page if none already exist and b) put the dropped text into the input box of the first open url import page (and focus it, so you can quickly hit enter)! This works when dragging text links from browsers, as well
• you can now 'append' gui sessions, which will just append that session's tabs to whatever is already open--say, if you have several 'favourites' pages you want to be able to quickly load up without having to break your existing workflow
• ipfs services now have a 'check daemon' button on their review services panel which will test the daemon is running and accessible and report its version
• fixed the 'test address' button for ipfs services on their manage services panel
• the client can now automatically download files it wants and knows are on an ipfs service
• middle-click on an 'all known files' domain thumbnail will now correctly start a download (as long as a specific remote file service is known)
• the multihash prefix option is reinstated on ipfs manage services panels
• the gelbooru parser now discovers the correct page url to associate with its files
• wrote some redirect fetching code to fix the gelbooru bad urls issue
• discovered a quicker fix for the gelbooru issue--the redirect location is the garbage in the original url in base64
• all downloader/subscription url caches will purge any old gelbooru 'redirect.php' urls on update
• fixed an issue where 'previously deleted' gallery/thread imports were returning 'fail'
• fixed a problem that was causing some redundant laggy work in adminside petition processing
• thread watchers will now remember their file and tag import options through a session save even when no thread url has yet been entered
• fixed an issue where media 'removed' from a media viewer view of a collection resulted in the entire collection being removed at the thumbnail level
• fixed an issue where media deleted from a media viewer view of a collection resulted in the media not being correctly removed from selection tags
• tag, namespace, and wildcard searches on a specific file domain (i.e. other than 'all known files') now take advantage of an optimisation in the autocomplete cache and so run significantly faster
• fixed a hover window coordinate calculation issue after minimising the media viewer on some platforms
• removed some 'all files failed to download' spam that could sometimes occur
• misc fixes

• version 255

• the duplicate filter now supports shift+left-click to drag, like the archive/delete filter (this remains hardcoded for now)
• if a pair in the dupe filter has the same resolution, they will now maintain zoom and pan when switching back and forth (I might increase this to work for same ratio as well, let me know how it works in the real world)
• the duplicate filter will show a lighter/darker background colour as you scroll the current pair
• the way lighter/darker and alternate lighter/darker colours are calculated is now centralised and should be more reliable in edge cases
• improved the dupe filter pair selection algorithm--it now chooses pairs more reliably under edge case conditions and prioritises decision-groups that have high potential decision value. it should also run a little faster
• increased the dupe filter batch size to 250, let's see if it causes any problems
• the close button on the dupe filter's top hover window now works
• fixed the duplicate filter disappearing pairs that were skipped (meaning you could not go back to revisit them)
• fixed a frequent deadobject error when the dupe filter closes
• the shutdown 'maintenance due' test is less sensitive to dupe search tree rebalancing, which typically only takes half a second
• the archive/delete filter now uses the new shortcuts system for both keyboard and mouse input
• the archive/delete filter now intercepts archive or delete commands from different sources ('media' shortcuts, top hover frame button presses) more reliably and converts them into filter actions (hence moving on to the next file)
• you can now move pages one to the left or right from their menu!
• on the top-right hover window (and the background underneath), ratings are now on the top, and hence will always be in the same location as you scroll through your media regardless of known urls, remote location, or inbox status
• autocomplete searches for tags with apostrophes, quote marks, braces, brackets and paretheses should be more reliable
• urls are now associated with files through the same 'content' pipeline as tags and ratings and so on
• gui-level media is now aware of the 'new url content update' event and will update and redraw itself appropriately
• fixed deviant art nsfw parsing, but it might not hold for long. proper fix here is to wait for the downloader overhaul
• networking engine now uses the 'requests' module's CA .pem (which the overhaul will be moving to anyway), which should reduce the frequency of ssl verify failures (gelbooru on the relatively new 'Let's Encrypt' CA had this problem for many users)
• fixed the networking engine redirect parsing for gelbooru's unusual location header. unfortunately, gelbooru is still giving mickey-mouse garbage redirect urls from its main thumbnail pages, for which in this engine there is no immediate fix
• neighbouring .txt tags will now be properly cleaned and sibling-collapsed in the path tagging dialog
• neighbouring .txt tags will now be properly sibling-collapsed in the import folder workflow
• making a media viewer borderless fullscreen and back will now recenter the media (previously, the current drag delta was not reset, so this frequently put media off-screen)
• fixed a serious issue where the media viewer could lock the client up on opening with a video if its gui options set 'remember size' to false
• tag import options objects will cleanse themselves of missing services on options save/client shutdown
• manage tags now defaults to cross-referencing 'my files' on the 'local tags' domain, where 'remote' (i.e. deleted) files' tags are not useful
• reduced memory use when importing large pngs with transparency
• improved adminside petition processing gui reporting
• servers will now cap the size of their mapping petitions so as not to ovewhelm the admin processing them (they now won't be both >20 tags and >1000 total row weight)
• the media viewer's manage tags frame now listens for content updates from outside, so if you alter an in-view file's tags (such as with a shortcut key), the manage tags dialog will update as it happens
• moved a number of the buttons on the top hover windows to the new unified internal command engine (which the new shortcut system also uses)
• disk cache maintenance uses fewer resources but now occurs in the foreground (and should hence more reliably maintain the cache)
• critical repository service id lookup errors will now automatically reset the repository's processing cache and better inform the user of what has happened. if you see this, please let me know the details and how this error fired in the real world
• updated and reinstated the ipfs service panel in review services
• updated some ipfs service code for the new service system
• misc refactoring and cleanup
• more cleanup and deletion of redundant old pubsub command code
• updated ubuntu build machine to 17.04 and opencv 3.2

• version 254

• the duplicate processing page is now on the new page picker, under the new menu 'special'
• improved the dupe pair selection algorithm--it should be faster for everyone
• added help icon to dupe processing page--it can launch of a couple of message boxes with simple help, or you can open some html help
• wrote some dupe processing html help!
• the dupe processing page now refreshes its numbers when the dupe filter closes
• the archive, inbox, delete, and undelete buttons on the duplicate filter's top hover frame now work
• delete and shift+delete keys now work to delete/undelete a file in the duplicate filter
• the mouse cursor now hides/shows on halt and new movement on the duplicate filter
• you can now set 'content' shortcut actions (adding tags or ratings) to shortcuts in the 'media' reserved shortcut set (which are always on in media contexts), although they don't yet work in the thumbnail view yet
• added a help button to the manage shortcuts panel
• the edit shortcuts dialog will now now allow you to name a custom shortcut set to one of the reserved names and will explain the problem in a message box
• import pages update more efficiently during periods of busy cpu
• import pages will use less idle cpu time generally
• import pages will adaptively use less cpu time when they are in the undo deletion queue or have no files to import
• the page of images downloader will spam a little less idle time
• the canvas background details and the top-right hover window will now only show 10 urls max
• added a help button to the tag import options panel to better explain the namespace selection and explicit tags
• added 'gui report mode' to the help menu, which will report key and mouse shortcut events and and matched commands
• the adminside petition panel now sorts multiple petitions by number of files descending
• misc improvements
• misc help cleanup
• misc fixes

• version 253

• created a new object to hold tag and rating merging and 'worse file' deletion options
• wrote a dialog to edit this new object
• established some simple defaults for this object for different duplicate status actions
• the cog icon on the duplicates filter now lets you edit these defaults
• the 'custom' duplicate filter action now works--first by asking you what status you want to set, and then by throwing up the new merge options dialog to tune it to whatever you like
• wrote comprehensive unit tests for the new object
• fixed the super slow dupe filter launch time problem
• added a 'known urls' submenu to thumbnail and browser canvas right-click menu that lists all known urls for a file with an option to launch or copy the url
• added known urls' hosts to the top-right canvas details background, just below where known file repos are listed
• added the same known urls' hosts as clickable hyperlinks to the ratings hover window that pops over that top-right area
• added 'delete from deleted files' action to the local tags's service-wide update panel. it will limit the deletion to mappings that are currently on files that have been previously completely physically deleted from the program
• fixed namespace filtering on service-wide update panel
• the hentai foundry downloader broke, so the update code will pause all hf subs. the solution is not trivial (it is part of the downloader overhaul), but I will try to fix it soon
• debuted a new question-mark help button to better explain .txt tag importing on the manual import tagging dialog and the manage import folders dialog
• fixed a small potential error due to bad parsing in the 'page of images' downloader
• fixed a typo bug that stopped the 'delete shortcuts set' action working in the manage shortcuts dialog
• I may have fixed an issue where the server was sometimes not shutting cleanly with a keyboardinterrupt
• fixed the media embed button not reliably updating its thumbnail
• fixed an issue where a dummy animation bar was displaying on embed buttons that showed static images that included transparency
• the serious db missing tag and hash states will now not throw an error but will inform/spam the user (and hence not prohibit a boot)
• attempting to open a second manage tags frame from the media viewer will now instead put the focus on the first (previously, multiple manage tags frames could be made)
• misc db code cleanup that should result in faster result building in certain situations
• misc improvements
• misc layout fixes

• version 252

• the duplicate filter now processes pairs in batches and hence supports 'back' actions to revisit decisions. you will be prompted every fifty or so pairs to commit and checkpoint your progress
• the duplicate filter now presents related pairs together, rather than picking at random
• fixed a bug in duplicate filter shortcuts initialisation
• simplified duplicate filter default shortcuts (these will be overwritten on update) to only use simple left- and right-click for 'this is better' and 'alternates', since those are by far the most common actions. middle-click now goes back, like the archive/delete filter
• converted old 'main shortcuts' system to the new shortcuts system, also splitting it up into 'media', 'main_gui', 'media_viewer', and 'media_viewer_browser' constituent parts that will be applied in different contexts
• because the change is so significant, all clients will have their old 'options' shortcuts reset to the new default--I expect to expand shortcuts further in the next few weeks, so this default-overwriting will likely happen again, so you will likely wish to wait before recustomising your basic shortcuts
• completely eliminated the old main shortcuts system--all references now bodge with the new system to varying neatness
• as the old main shortcuts system no longer exists, the 'shortcuts' page on file->options is gone--all shortcuts are now managed through file->shortcuts, which is a completely revamped version of the custom filter shortcuts editing dialog
• selecting shortcut commands is significantly simpler for the reserved shortcut sets
• all media viewers with hover windows now have a 'keyboard' shortcuts icon button on their top hover window--it links to the manage shortcuts dialog as well as the current active custom shortcut sets and default custom shortcut sets
• as the above system supercedes the old custom filter system, custom filters are completely removed from the program! your existing custom shortcut sets will survive, but you probably want to purge them of all the redundant junk they still have
• all the media canvases use the new shortcuts system
• the canvas frame uses the new shortcuts system
• the thumbnails canvas uses the new shortcuts system
• the main gui uses the new shortcuts system
• some other misc places use the new shortcuts system
• most simple shortcut command actions have been renamed to be more readable
• some shortcut actions, like zoom stuff, is no longer hardcoded!
• mouse shortcuts are still not widely supported!
• the shortcut command edit dialog now throws a veto-driven error message if you try to ok on an invalid command (a blank services choice or action string, that sort of thing)
• ctrl+r is now 'remove_files_from_view' by default. the old 'show_hide_splitters' is now ctrl+shift+r by default
• the preview canvas, if focused, now responds to many normal media viewer shortcuts (content stuff like archive/manage tags, and canvas-specific like frame back/forth, zooming and panning)
• the new shortcut system now interprets double-clicks of any mouse button to be a second single click
• the new shortcut system correctly 'flips' ratings on and off, rather than always 'setting' to the chosen value. optional 'set only' support will come in the near future
• more shortcuts will correctly and reliably propagate to canvases when any part of a hover frame has focus
• fixed a conflict between the new shortcut system and taglists, which were no longer accepting otherwise interesting keys, like enter
• created a shortcuts manager cache that deals with a bunch of the shortcut workflow centrally
• completely eliminated the old accelerator table/menu command system for the main gui window
• lots of misc shortcut-related work
• fixed wildcard file search predicates that are on a specific file service and have no namespace (like 'mar*')
• certain routines that can cause mass refreshing of the menubar (like import folders) will now not spam (and often queue this spam up and hang the gui) the menu so much. the menubar will now always collapse multple overlapping refresh calls to reduce cpu load
• fixed a focus-None issue in the new hover window focus detection code
• improved some more focus detection and comparison logic--quick rating-scrolling in the media viewer should be less janked
• improved 'touch' drag event detection and improved media canvas cursor hide/show logic in general
• fixed '&' display in notebook page names
• fixed '&' display in some common dialogs' text
• wrote a new statictext class that deals with '&' better and in future will autowrap and maybe some other stuff and then switched most of the simple instances of wx.statictext over
• updated some out-of-date server help r.e. admin service initialisation
• converted a little of the help on custom filter->custom shortcuts. I'll do more in future
• wrote a simple checkboxlist dialog
• misc cleanup

• version 251

• started shortcut overhaul by updating shortcut storage and underlying objects
• shortcuts will now support multiple modifiers (ctrl, alt, shift) in some places
• the regular shortcut entry control now supports mouse events through a radiobox (although only the duplicate filter will capture and deal with mouse events on the new system!)
• on db creation or update, a new 'duplicate_filter' shortcut set will be generated that includes mouse shortcuts and new duplicate_filter commands
• the duplicate filter now obeys this shortcut set
• duplicate filter shortcut edit cog menu now works, and the active shortcut set will update if edited
• wrote simple duplicate pair status update db code
• added dupe filtering optimisation--given A > B, anything else better than A will be set to be better than B, and anything worse than B will be set to be worse than A
• added dupe filtering optimisation--after any duplicate status change affecting A and B, any 'same file' siblings of A and B will receive the same relationship
• the 'process now' button on review services is now gated by a yes/no dialog that better explains what's about to happen
• the repository buttons on review services will disable when they can't do anything
• repository update processing will now cancel mid-job much faster
• reintroduced the 'service-wide update' button to tag services in review services
• reintroduced the 'clear trash' button to the trash file service in review services
• reintroduced num_files rating service reporting in review services
• fixed mouse scroll wheel events from the ratings hover window not being correctly processed by the main media canvas
• added some 'touch' event detection to try to better deal with media dragging through a touchscreen
• import folders will now 'action' their import paths as they go, rather than only at the end of their import run
• the new taglists are now better at remembering their selection through a content change
• opening a file or path from a non-windows client should now create a non-child process for open/xdg-open calls that block (so closing the client should not then close the child process movie player application or whatever)
• added unit tests for the new shortcut object
• added unit tests for the new application command object
• fixed unit tests for the updated shortcuts object
• misc cleanup

• version 250

• improved 'file storage locations' help page descriptive text, including adding statements for actual current storage percentages
• improved boot missing file storage error handling
• wrote a 'repair file storage' dialog panel to manually fix missing folders during the boot phase
• added some bells and whistles to the repair file storage dialog. it now shouldn't be possible to boot the client with an invalid file store--let me know how this works for you
• added 'apply all siblings to all services' to options->tags
• whenever combined-service siblings are consulted, local siblings will now always have precedence (so, you can now definitely overwrite any ptr siblings you don't like at the local level). this stuff remains a mess, however. more work is needed
• fixed searching for 'like' ratings
• rating predicates will now render their value as 'like'/'dislike' or '3/5' rather than the underlying float
• fixed a variety of predicate string rendering errors when 'show namespace' is false
• fixed delete key not firing on autocomplete text entry controls
• the autocomplete entry's text box will now take home/end/up/down events when there is only one result in the dropdown, when navigation of that list is not important (previously, this was only true when there were no results)
• fixed an issue where invalid typed 'system:' tags were not being correctly cleaned
• improved tag cleaning code
• fixed fetching autocomplete results for small and specific searches (e.g. putting in 'a' will return results just for 'a'
• fixed small and specific searches to return namespaced instances as well (e.g. putting in '8' can now return 'chapter:8'
• fixed an issue with showing collected media in the preview window
• fixed the file import status button not laying out correctly on edit subscription dialog
• cleaned up some focus checking code
• harmonised all initial key event processing
• cleaned some key event processing
• adjusted some server daemon timings to prefer post-boot update generation catchup over db maintenance
• corrected a bunch of slightly mistaken unit test code
• added some 'like' and 'numerical' ratings search unit tests
• the advanced content update dialog now uses the new sizing system (so the go! button should no longer wander out of frame!)
• reduced what I believe is excess fuzzy padding on the new sizing system
• optimised a common client db request, speeding up a number of tag operations
• gui now asks for confirmation when you make to delete a gui session
• improved some out of date unit tests

Changelog 200-249

• version 249

• reintroduced shape and colour options to edit ratings service panels
• reintroduced num_stars and allow_zero options to edit numerical ratings service panels
• the export phrase--defaulting to '{hash}'--will now persist through export dialogs. it is saved whenever you click update or export
• removed the unintended 'counts' that were appearing after related tags
• fixed 'remove' action in custom filters
• hitting the delete key on manage tags taglist now will always remove
• added a BUGFIX option to not verify regular https traffic on the old networking engine for those non-Windows users who are getting SSL verify errors
• refactored the top hover frame more, making it more flexible
• created a new top hover frame for the duplicates filter
• duplicates filter now also supports tag and ratings hover frames
• duplicates filter now reports A or B as file index
• added some placeholder buttons to the duplicates filter for actions and 'cog' customisation
• reduced duplicate search shutdown logspam
• improved duplicate search job cleanup
• improved mime detection of thumbnail regeneration
• improved some shutdown error handling
• improved numerical rating search accuracy
• improved some hdd import error handling
• improved some unicode error handling
• improved reliability of some database transaction processing
• fixed animationbar having problems with single-frame videos
• fixed namespace colour taglist not updating colour correctly
• stopped the namespace colour taglist from deleting default namespace colours
• made the new class of button more compact--not sure if I like it
• refactored some db initialisation to avoid future transaction problems during special updates
• tidied up some last incomplete taglist code from last week
• did a little prep for some future shortucts overhaul
• did some more menu code updating
• added a simple subscription save/load unit test
• moved outdated server unit tests forward
• fleshed out some server certificate generation
• some other misc v245 catchup work
• cleaned up some pydeadobjecterrors caused by downloaders reporting progress to destroyed windows

• version 248

• fixed two more issues with recent update code!
• re-applied a more rigorously tested 'invalid subtag' replacement for any user that has experienced incomplete update code in the past few weeks
• added some tag rendering options to options->tags that affect most user-facing, non-editing windows:
• added custom namespace-subtag connector (i.e. instead of ':')
• added 'show namespaces' checkbox (i.e. to keep the colour but lose the namespace, like how most boorus display tags)
• created a basic framework for the duplicate filter--it doesn't do anything yet, but you can launch it from the duplicates page to check it out
• updated 'reset processing cache' action and added it to review services repository pages
• review services now always starts on 'my files'
• review services will try to stay looking at the currently selected service through a refresh_services event
• fixed some thread-gui refresh interaction in review services
• all listctrl-attached delete/remove buttons will now ask if you are sure before they go
• reenabled db->maintenance->clear orphans, which it turns out I had already fixed by accident
• serverside services will lock themselves more efficiently while generating updates, ensuring 'server busy' responses can always return promptly
• fixed tag manage parents a/c entry to remove tags when they already exist in the box (previously, the a/c would only ever add, so a double-click was required to remove)
• tuned adminside petition processing so it approves/denies in reasonable-sized chunks
• refactored all the taglist gui code to focus on the underlying tag rather than the string representation
• plenty of misc taglist cleanup besides
• started a unit test suite for listboxes and wrote a thorough test for namespace colour listbox
• misc canvas refactoring

• version 247

• fixed a problem with deleting more than 256 files at once
• furthermore, deleting from thumbnail view or after a filter will split delete jobs into chunks of 64 files at a time to reduce gui hang from deleting many tag-heavy files
• rewrote canvas media container code to recycle containers, embed windows, animation bars, static image windows, and animation windows. scrolling through all kinds of media is less flickery (less 'grey box' window initialise flicker) and scrolling through static images should be completely flickerless! William Gibson slideshow speed works again!
• this flickerless static image transition will be particularly useful in the forthcoming duplicate image filter!
• got adminside petition processing working again
• petition counts and fetching is now split by content_type and status
• the approve/deny colour hint is more obvious on the petition panel
• petitions now process off the main gui thread and throw up a popup message
• added 'check all' and 'check none' buttons to petition panel
• several serverside petition processing fixes
• generalised and improved dynamic menu check item initialisation and inversion support
• moved the 'get tags even if file already in db' option into a cog button on regular downloader pages
• added a default menu option for 'get tags even if file already in db' to the same cog button
• added this cog button to the edit subscription panel as well
• fixed exporting tags to Hydrus Tag Archives
• fixed exporting 'all known tags' to HTAs
• cleaned some HTA and related code
• fixed namespace based tag censorship
• fixed autocomplete not filtering out current/pending counts if they are set to 'excluded'
• fixed generation of non-expiring new accounts
• fixed some v245->v246 tag improvement code that was replacing invalid tags with the incorrect namespace
• patched a problem with '-:' dirty tag in the v245->v246 update code--I'm not sure what it was doing, but it catches the unusual problem and puts it in the 'invalid tags' category, so let me know if you get trouble with this in future
• wrote unit tests for bytesdictionary and shortcuts serialisable objects
• harmonised and improved how separators are appended to menus
• cleaned up some client db index creation
• cleaned up some client tuple stripping
• misc pylint warnings cleanup
• misc fixes

• version 246

• fixed a critical bug in serverside content deserialisating that meant servers were not processing most client-submitted data properly
• fixed a critical bug in 'all known tags' autocomplete regeneration--please run database->regen->a/c cache when it is convenient and let me know if your numbers are still off
• fixed the pre-v238 update problem for good by abandoning update attempts and advising users to try v238 first
• clientside invalid tags will now be collapsed like with the server last week. if a tag is invalid (typically something with an extra space, like "series: blah"), the update code will attempt to replace existing mappings with the collapsed valid version. some unusual cases may remain--they will be replaced with 'invalid namespace "series "' and similar. Please remove and replace these when convenient and contact me if there are way too many for you to deal with
• duplicates pages now have a file domain for the filtering section, and they remember this domain through session loads
• this file domain is accurate--counting potential duplicates and fetching pairs for 'show some pairs' only from those domains. the issue of remote files appearing should be gone!
• there is now only one 'idle' entry in the duplicates page cog menu--it combines the three previous into one
• fixed numerous irregularities across the wildcard code. all search input now has an implicit '*' on the end unless you put a '*' anywhere else, in which case it acts exactly as you enter it, with a non-* beginning matching beginning of string, whitespace, or colon, and non-* end matching end of string or whitespace
• autocomplete now searches namespace, so entering 'char' will load up all the 'character:' tags along with 'series:di gi charat'. this can lag to hell and back, so it may either need some work or be optional in the future. feedback would be appreciated
• typing 'namespace:' will include all the series tags below the special optimised 'namespace:*anything*' tag
• autocomplete searches recognise an explicit '*' no matter where it is in the entry text. typing 'a*' will load up all the a tags and present a 'a*' wildcard option
• quickly entering a wildcard entry will now submit the correct wildcard predicate ('rather than a literal 'hel*' or whatever tag)
• review services panel now reports total mappings info on tag services
• review services panel now reports total files info on file services
• manage services's listctrl is now type | name | deletable and initially sorts by type. the strings used for hydrus service types are also improved
• manage serverside services (called by server admins to manage their services) have fixed setnondupe port and name on edit service events
• new popup messages will now also appear if there were previously no popup messages to display if the current focus is on a child on_top frame, such as review services (you'll now see the processing popup appear when you click 'process now' on review services)
• the popup message manager now initialises its display window with a single message that is quickly dismissed. this helps set up some variables in a safe environment so they don't have to be generated later when the gui might be minimised or otherwise unusual
• hid hydrus update files from 'all local files' searches
• added 'media_view' entries for hydrus update files, just in case they are still visible in some unusual contexts (and they may be again in a future update anyway)
• fixed 'recent tags' being returned from the database out of order
• by default, 'recent tags' is now on for new users
• 'get tags even if file already in db' now defaults to False
• file import status now allows a 'delete' action below the 'skip' action
• file import status right-click event processing is more sane
• fixed the new raw width/height sort choices, which were accidentally swapped
• cleaned the media sort code generally
• cleared out some redundant rows that are in some users' client_files_locations
• namespaced predicates are no longer count-merged with their namespaceless versions in 'write' autocomplete dropdowns
• 'unknown' accounts should now properly resync after clientside service change
• improved how registration keys are checked serverside when fetching access keys
• fixed a v244 update problem when unexected additional tag parent/sibling petitions rows exist
• improved my free space test code and applied it to the old v243->v244 free space test (it'll now test free space on your temporary path and report problems appropriately)
• to improve log privacy and cleanliness, and to make it easier to report profiles, db/pubsub profiles now write to a separate log file named appropriately and labelled with the process's start time
• profiles are more concise and formatted a little neater
• across the program, three-period ... ellipses are now replaced with their single character unicode … counterpart (except to the console, where any instance of the unicode ellipsis will now be converted back to ...)
• cleaned up some log printing code
• cleaned up some experimental static serialisation code, still thinking if I like it or not
• started on some proper unit tests for hydrus serialisable objects
• fixed and otherwise updated a heap of unit test code to account for the v245 changes
• cleaned up a bunch of old database table join code
• started some databse-query-tuple-stripping code cleaning
• deleted more old unused code
• misc timing improvements
• misc code cleanup experimentation
• misc cleanup

• version 245

• fixed a v244 update problem for clients updating from<v238< li=""></v238<>
• some misc stuff:
• if you start editing many subscriptions, cancelling a single dialog will break the chain of loading new dialogs
• reduced some redundancy in regular client file import
• improved how the dialog for selection from a list of strings works
• if the client ever merges one directory to another (such as in external file locations migration) and any files fail to merge, the source will no longer be deleted
• created new flexible bandwidth tracking and ruling objects
• updated how repository updates work, splitting the old explicit and self-contained content update package system to a new implicit definitions/content split system and an improved one-step-sync metadata propagation
• updating now takes approximately 23% as much bandwidth as before
• update files are now stored in client_files and server_files like any other file (client_updates and server_updates folders will be deleted on update)
• the server will print update generation info to its log
• unfortunately, updates cannot be converted, so a complete resync of update files is required. the smaller update size and better bandwidth controls should mitigate the problem somewhat
• the server has been compacted across all content types--its mappings db file should shrink about 12%
• due to new service-specific ids, server.master.db should increase in size, typically about 50%
• harmonised how GET and POST/response args are built and parsed across the network
• server administration initialisation is now simpler, done with 'init' registration key
• leading and trailing spaces are now removed from both the namespace and subtag components of namespaced tags, meaning 'title: blah' will be collapsed to 'title:blah'. tag repositories will clean their existing tags on update
• invalid serverside tags will be replaced with valid placeholders on server update
• refactored, harmonised and simplified some server request code
• Eris makes a triumphant return to the root welcome page ('/' request of any service) with improved self-description text as well
• updated how bandwidth is tracked and overseen
• all requests now consume bandwidth
• fewer requests are actually constrained by bandwidth--at the moment, it is only update files and file/thumbnail downloads, as these represent the overwhelming majority of bandwidth consumption and are not at all critical to service operation--this may change in the future, but it suits our purposes for now
• bandwidth tracking code is more sane across the board
• the server's administration service now tracks server-wide bandwidth as it happens
• service and server-wide bandwidth rules are consulted as soon as the request begins
• improved some server response rendering
• permissions are more flexible and content-specific
• improved how file repositories check and process file requests
• created a gui control for managing bandwidth rules
• updated the serverside service object
• the serverside service object now contains bandwidth rules and tracking
• updated how the server deals with services on a db level
• refactored and cleaned a ton of server db code
• updated how services are edited over the network
• updated how the server associates its services with its http pipeline
• converted clientside server service management gui to the new panel system
• updated gui code for clientside server service management
• converted clientside server service management db code to the new system
• improved how bandwidth errors are reported
• updated the account type object
• the account type object now contains bandwidth rules and a more flexible permissions system
• updated how the server deals with account types on a db level
• converted clientside server account type management gui to the new panel system
• updated gui code for clientside server account type management
• serverside account types are now fetched from a cache, reducing memory sprawl
• updated the account object
• the account object now contains a bandwidth tracker
• updated how the server deals with accounts on a db level
• updated the clientside server object
• the clientside server object now contains bandwidth rules and a tracker and has improved error management, recovery, and reporting
• updated how the client deals with services on a db level
• improved how serverside bandwidth errors are caught
• all clientside services start with a 50MB/day bandwidth limit
• existing repositories will get a 250MB/month, 50MB/day default bandwidth limit, just to help us get over the hump--see the release post for info on how to get all the updates anyway
• updated manage services dialog extensively
• service account registration now occurs through a simple button on the normal clientside service edit panel
• all services can now be renamed
• updated the content processing pipeline
• prepared code for future merging of file and tag repositories
• added future support for pend-petitions for files and mappings and simple creation permissions for tag siblings and parents
• the clientside content processing pipeline now operates inside a single database transaction, reducing a great deal of previously redundant hard drive activity
• the serverside content processing pipeline now creates per-service timestamped content definitions
• the clientside content processing pipeline now maintains and appropriately consults a cache for server definitions
• pre-process disk cache is more intelligent
• updated review services to match the new service objects
• service review panels now show more information about error state and so on
• service review panels should now update as soon as services change
• you can now force sync repositories from review services
• you can now pause/resume repositories from review services
• you can now export repository update files from review services
• you can now import repository update files from the services menu
• repository thumbnail download now uses a cache for faster thumbnail ownership testing
• culled a lot of old code and experiments
• deleted all the old messaging depot code and db table cruft
• removed optimised petition processing
• serverside deleteorphans is temporarily disabled
• hydrus client-repository relationship no longer supports news
• removed the 'stats' admin service call--it will come back as a account review page
• clientside clear orphans is temporarily disabled
• clientside local file/thumbnail server is disabled for now
• custom service 'messages' for the root page are no longer available
• some things are not working yet--they will be back in soon
• content presentation on review services
• service reset, other advanced content service controls
• ipfs controls on review services
• local booru controls on review services
• most service-specific panels on manage services
• petition resolution
• serverside account modification, including banning

• version 244

• updated client database to compact ( namespace_id, tag_id ) pair into a single id for storage
• added some bells and whistles to the update code
• added a free space check and messagebox warning before the update
• updated db, service, and a/c cache creation code to reflect new schema
• updated absolutely everything else in the db to reflect the new schema
• for users with plenty of tags, the db should now be about 33% smaller!
• unified how unnamespaced tag searching counts are totalled
• unnamespaced tag searching counts are now totalled when the tags are fetched from the in-view ui media
• unified how tags are split into ( namespace, subtag ) across the program
• fixed deviantart gallery thumbnail parser
• fixed linux session load page key event handling bug
• os x can now support notebooks with zero pages open
• fixed an issue where os x was losing the first page of some session loads
• fixed some similar files shutdown work false positive calculation
• reduced server bandwidth check period from 24 hours to 1 hour
• improved calltothread scheduling under heavy load
• improved scheduling of how files are physically deleted
• numerous laggy temp_table replacement/cleanup
• more temp_table replacement
• misc efficiency improvements and general db code cleanup
• misc path code cleanup

• version 243

• updated more menu code to the new system
• finished updating all main gui menubar and thumbnail menu code
• all brackets, braces, parentheses, and single and double quotes are now ignored when matching search text to tag. inputting 'mercy' matches 'character:angela "mercy" ziegler' and entering '[[[intensifies]]]' matches '[intensifies]'
• '*' searches will now return no autocomplete results, rather than trying to load absolutely everything
• made wildcard searching significantly more permissable--it now implicitly applies a '*' to the beginning as well as the end of your search input
• the main gui's notebook pages will now be disabled while a session loads
• might have fixed a first-boot layout issue--otherwise added a layout entry to the debug menu for further testing of this problem
• improved progress layout on file import dialog
• created a more efficient way for threads to update gui elements
• the file import dialog uses this new update system and parses lots of small files much faster as a result
• the edit import folder dialog will now complain if you try to put in an import path that includes the install or db directory
• ffmpeg mime parsing now catches and reports audio formats without any hassle
• thread downloader no longer fetches file extension in filename tag 'filename:blah' instead of 'filename:blah.jpg'
• changed png default zoom rules to copy jpg--the unusual zoom as default was more confusing than helpful
• updated sqlite for windows--should be a decent bit faster
• deleted a lot of old code that'll never be needed again
• misc refactoring
• misc cleanup

• version 242

• optimised 'exact match' similar file queries to run a lot faster
• optimised similar file queries in general, particularly for larger cycle queries
• optimised hamming distance calculation, decreasing time by roughly 45%!
• the similar files tree maintenance idle job will not trigger while there are phashes still to regenerate (this was redundantly and annoyingly blatting the new dupes page as soon as phash regen was paused)
• removed similar files tree maintenance entry from db->maintain menu, as it can be done better from the new dupes page
• adjusted the duplicate search and file phash regen progress gauges to reflect total number of files in cache, not the current batch job
• all maintenance jobs on the duplicates search page will now save their progress (and free up a hanging gui) every 30 seconds
• the duplicates page's cog menu button now lets you put phash regen and tree rebalancing on the normal idle routine, defaulting both to off
• the cog menu can also put duplicate searching on idle time!
• added a very rough 'just show me some pairs!' button to the dupe page--it is pretty neat to finally see what is going on
• I may have reduced the memory use explosion some users are getting during file phash regen maintenance
• wrote an unclose_page action and added it to the shortcuts options panel--it undoes the last page close, if one exists. ctrl+u will be the default for new users, but existing users have to add it under options
• added ascending/descending sort choices for width, height, ratio, and num_pixels
• the client can no longer talk to old http hydrus network servers--everything is now https
• in prep for a later network version update, the client now supports gzipped network strings (which compress json a lot better than the old lz4 compression)
• fixed gif rendering in the Windows build--I forgot to update a build script dll patch for the new version of opencv
• the export file dialog's neighbouring .txt taglist file stuff now allows you to select a specific combination of tag services
• if an hdd import's original file is due to be deleted, any existing neighbouring taglist .txt file will now also be deleted
• the inter-thread messaging system has a new simple way of reporting download progress on an url
• the handful of things that create a downloading popup (like the youtube downloader) now use this new download reporting system
• sankaku seems to be 503-broke due to cloudflare protection--I have paused all existing sankaku subscriptions and removed the sankaku entry for new users (pending a future fix on my or their end)
• I've also removed danbooru for new users for now--someone can fix the long-running sample size file issue in the new downloader engine
• removed unnamespaced tag support from the hentai-foundry parser--maybe someone can try to fix that mess in the new downloader engine
• menubuttons can now handle boolean check menu items that are tied straight into hydrus's options
• menus launched from the newer frame and dialog code will now correctly display their help text on the main gui frame's statusbar! (at least on Windows! Linux and OS X remain borked!)
• fixed a unicode error parsing bug in the gallery downloader
• the server stop (or restart) command now correctly uses https!
• the server test code now works on https as appropriate
• fixed some misc server test code
• misc fixes
• misc cleanup
• misc layout cleanup

• version 241

• fixed the 'setnondupename' problem that was affecting 'add' actions on manage subscriptions, scripts, and import/export folders
• added some more tests to catch this problem automatically in future
• cleaned up some similar files phash regeneration logic
• cleaned up similar files maintenance code to deal with the new duplicates page
• wrote a similar files duplicate pair search maintenance routine
• activated file phash regen button on the new duplicates page
• activated branch rebalancing button on the new duplicates page
• activated duplicate search button on the new duplicates page
• search distance on the new duplicates page is now remembered between sessions
• improved the phash algorithm to use median instead of mean--it now gives fewer apparent false positives and negatives, but I think it may also be stricter in general
• the duplicate system now discards phashes for blank, flat colour images (this will be more useful when I reintroduce dupe checking for animations, which often start with a black frame)
• misc phash code cleanup
• all local jpegs and pngs will be scheduled for phash regeneration on update as their current phashes are legacies of several older versions of the algorithm
• debuted a cog menu button on the new duplicates page to refresh the page and reset found potential duplicate pairs--this cog should be making appearances elsewhere to add settings and reduce excess buttons
• improved some search logic that was refreshing too much info on an 'include current/pending tags' button press
• fixed pixiv login--for now!
• system:dimensions now catches an enter key event and passes it to the correct ok button, rather than always num_pixels
• fixed some bad http->https conversion when uploading files to file repo
• folder deletion will try to deal better with read-only nested files
• tag parent uploads will now go one at a time (rather than up to 100 as before) to reduce commit lag
• updated to python 2.7.13 for windows
• updated to OpenCV 3.2 for windows--this new version does not crash with the same files that 3.1 does, so I recommend windows users turn off 'load images with pil' under options->media if they have it set
• I think I improved some unicode error handling
• added LICENSE_PATH and harmonised various instances of default db dir creation to DEFAULT_DB_DIR, both in HydrusConstants
• misc code cleanup and bitmap button cleanup

• version 240

• improved how the client analyzes itself, reducing maintenance latency and also overall cpu usage. syncing a big repo will no longer introduce lingering large lag, and huge analyze jobs will run significantly less frequently
• the analyze cache will be reset on update, so you will have one big round of analyze the next time you maintain, and then you are good
• added data structures to support auto-discovery of duplicate files
• improved how some similar files maintenance occurs
• flushed out duplicate status reporting
• added a new page type, currently under pages->search pages->duplicates, to handle duplicate discovery and filtering
• created a gui skeleton for the new duplicates page
• started some handles and update code for the new duplicates page
• wrote a new txt file in the db dir about the new emergency extract scripts
• wrote an emergency extract script to migrate subscriptions to a new db
• wrote an emergency extract script to migrate options to a new db
• the trash clearing daemon now runs in the foreground, and a little of its code is improved
• the trash clearing daemon now makes a popup message when it does work
• the server's ssl keys are now set to read-only on Windows and user read-only only (i.e. chmod 400) on Linux and OS X on creation and update
• added a explicitly unicode popup message to the debug test
• fixed some network error catching code that was using Windows-only error codes
• converted more of the thumbnail right-click menu over to the new system
• improved some listctrl code
• misc cleanup

• version 239

• finished up similar files search data maintenance code
• similar files search data maintenance will now run during idle time
• similar files search data maintenance can be called from database->maintain menu
• the crowded database->maintenance menu is now split into maintain, regenerate, check
• improved the similar files tree generation code, speeding searches significantly
• wrote a new listctrl class to handle more complicated objects and also sort by underlying data
• the new listctrl now handles object name non-duplication
• cleaned a bunch of crap old listctrl code
• manage export folders now uses the new listctrl
• manage import folders now uses the new listctrl
• manage subs now uses the new listctrl
• manage scripts now uses the new listctrl
• options media viewer options now uses the new listctrl
• file import status panel now uses the new listctrl
• the new listctrl can now quickly fetch item index from the underlying object
• the file import status panel should now cope with extremely huge lists a bit better now!
• multiple parsing child nodes can now import from/export to clipboard as lists
• export folders now have names, so you can have multiple export folders pointing to the same path! existing export folders will get their path as their name, but this can be changed no prob
• cleaned import/export folder dialogs
• cleaned import/export folder dialog workflow
• several misc import/export folder improvements
• hydrus servers are now exclusively https with self-signed certificates
• hydrus servers now create server.crt and server.key in their db folders for SSL--these files will be backed up along with everything else on an admin backup command
• system:hash now ignores the file domain and any other predicate. it now returns very quickly, no matter the context
• improved system:hash search logic
• all the awkward choice dropdowns in system predicate panels are replaced with radioboxes
• improved system:rating panel grid layout
• wrote a better subclass of radiobox to handle more data
• moved first half of thumbnail menu to new menu system
• cleaned up a little thumbnail menu logic
• improved the different ways services are added to thumbnail menu
• thumbnail 'select' menu is logically cleaned up and allows for better file domain selection
• the thumbnail menu's copy files and copy hashes to clipboard will now send them ordered (they were previously pseudo-random)
• added 'paths' to the share->copy thumbnail menu for copying multiple files' paths. these are also ordered.
• if the popup message manager does not have any errors, it will no longer unhide (which can annoyingly raise the main gui window) when the gui window does not have focus
• removed some old redundant error reporting stuff in popups
• improved and quietened some some mime detection failure code, sped up mime failure loop in all cases
• massively simplified and atomised how new serialisable-object management panels can save their data
• the manage subscription dialog now saves in a single, faster transaction
• the manage script dialog now saves in a single, faster transaction
• reduced redundant index work from analyze jobs
• improved tag parse error handling
• fixed media removal rules when deleting from the 'all local files' domain
• polished and clarified some of the help's tag schema
• misc cleanup
• more misc cleanup
• misc refactoring
• more misc refactoring

• version 238

• added left/right equal ratio preference to the similar files search tree generation
• added high standard deviation preference to the similar files search tree generation
• added 'db profile mode' to the help menu, for briefer profiling in future
• when a similar files search occurs with db report mode on, it'll popup a nice bit of text saying how many cycles it took
• fixed a bug due to last week's 'all local files' change that meant archive/delete filtering would include any selected trashed files
• 'recent tags' will now only be populated by add/pend tag events, and only by those tags that were valid for that action
• moved the help menu to the new menu system
• added bitmapped menu item support to the new menu code
• fixed and cleaned up some of the new menu code
• improved a/c cache regen error reporting

• version 237

• added 'all local files' service that spans all local file domains
• improved trash service code
• trashed files now report their trashed timestamp in right-click menus
• trash views will sort oldest/newest by the trash timestamp
• renamed 'local files' to 'my files' to reduce initial confusion
• if you don't like 'my files', you can now rename the local files service under manage services!
• improved how some local file service metadata is stored
• cleaned and possibly fixed up some delete code
• cleaned up a bunch of misc service and file service code
• the client is more intelligent about what local files are -- you can now 'open externally' trashed files, for instance
• on multiple monitor systems, the new sizing system now bounds itself by the appropriate monitor's dimensions (previously, it was always consulting the primary, I think)
• if an expansion event causes a frame to grow off screen, the new sizing system will now attempt to move it up and left so it is completely visible
• fixed an important bug in the specific service autocomplete cache that was leading to several kinds of miscount--please regen your autocomplete cache at your convenience
• the client can now post additional messages on boot (you'll see one!)
• improved how errors with unusual characters are applied to failed import file status objects
• left double-click on the main gui greyspace now opens the new page chooser as well
• restored session pages now recover more gracefully from missing services
• shortened the 'stop after this many files' phrase, which was mis-sizing downloader panels
• changed several bits of old jank in the 'import options - files' collapsible. it is also now thinner
• misc gui improvements
• updated more of the menu system code
• the selection tags box now sizes its height more reasonably
• fixed 'check now' and 'reset cache' buttons on edit sub dialog
• subscriptions report better file import status on their popup
• misc cleanup

• version 236

• in prep for a network https upgrade, the client can now detect and escalate to https when making connections to hydrus services
• import/export to png and clipboard now supports multiple objects at once!
• rewrote the manage subscriptions dialog to work on the new panel system
• the new manage subscriptions dialog has a listctrl and a sub edit dialog
• the new manage subscriptions dialog has the same add/export/import/dupe/edit/delete buttons as the manage scripts dialog
• subscriptions are now importable/exportable, including en masse with the new multiple object import/export support!
• the new manage subscriptions dialog has retry failed/pause-resume/check now/reset buttons for easy mass subs management
• the edit subscription panel has a bit of a layout makeover
• the edit subscription panel now updates itself as its buttons are hit
• the edit subscription panel disables buttons that are not applicable
• subscriptions can now be renamed!
• cleaned some misc subscription code
• relabelled initial and periodic file limit in the subscription edit panel
• middle-clicking on the main gui's greyspace (e.g. to the right of the notebook tabs) will spawn the new page chooser!
• created a simple HydrusRatingArchive class--will do more with it in future
• added ffmpeg, python, and sqlite versions to the help->about window
• harmonised daemon code
• added a new class of daemon that will not fire while a session load is occuring
• subscriptions, import and export folders, and file repo downloads now use this new daemon
• cleaned the way background daemons check for idle
• expand/collapse panels now notify the new kind of toplevelwindow that a resize may be needed when they switch state
• time deltas (like on subs edit panel or a thread watcher) now render more concisely ('7 days' instead of '7 days 0 hours')
• serialisable object png export panel now has a width parameter
• fixed a bug where tags that begin with unicode digits were accidentally identifying as numbers for the purposes of sorting and throwing errors on convert fail
• the media viewer can handle some more unusual content update combinations--for instance, if it cannot figure out which media to show next, it will revert back to the first image rather than displaying an undefined null mess
• updated and cleaned a bunch of my old misc encryption code
• misc cleanup

• version 235

• finished first version of new faster dupe search--'system:similar to' now uses it
• finished faster dupe search tree creation
• finished faster dupe search tree search
• added faster dupe search tree leaf insertion
• added faster dupe search tree rebalancing maintenance
• added faster dupe search tree orphan deletion maintenance
• added faster dupe search leaf-regeneration-as-scheduled maintenance
• misc faster dupe search tree work
• new search tree will be created on db update
• fixed the 'add' buttons on the import files dialog
• fixed linux manage tags text input
• reduced incidence of linux media viewer post-manage tags key event swallowing
• fixed an issue where if the manage tags dialog was launched from the media viewer while a hover window was displaying, the main gui would often be raised when the manage tags frame was closed, annoyingly obscuring and de-focusing the underlying media viewer
• fixed manage tags action workflow when no valid tag choices exist
• entering a tag petition reason now has several common 'suggestions' for quick entry
• removed ffmpeg binary from default linux release (as basically every linux has a decent ffmpeg by default)
• fixed the incorrect ffmpeg version for os x
• added an option to options->media to prefer to use the system ffmpeg over any in the hydrus bin directory
• fixed linux arrow key navigation of the page picker dialog
• cleaned up the page picker dialog's gross old code
• the edit file lookup script panel now has a script management control on its test page to better report on and control tests
• scripts will new correctly attempt to decode unicode and other encoded text in responses
• if non-text data is accidentally fetched in the file lookup script panel's test page, it is now caught and a small summary of that event is printed to the text box instead
• added the same non-text catch to the edit link node panel
• improved how some script errors and status updates are handled so the new test window script management controls will see and report more useful information
• removed 'restart' option from linux for now--looks like a bug in pyinstaller
• improved how the new GET request read reattempt works
• cleaned up some spammy error code
• cleaned up spammy test code
• misc error handling improvements
• misc cleanup

• version 234

• the action choice workflow in the manage tags dialog now merges decisions for multiple tag entry events
• the button dialog that pops up on the new merge manage tags workflow has button tooltips to better describe the proposed action's tag and file combinations
• the manage tags dialog only requests one petition reason for an above merged multitag petition event
• perceptual hashes are now stored in the cache db (moved from the preferably leaner main db)
• the database now supports multiple perceptual hashes per file
• fleshed out the perceptual hash vptree generation and maintenance code
• added option for main gui title to options->gui--it even updates live
• added unicode path support when importing serialised pngs
• added unicode path support when exporting serialised pngs
• exporting a serialised png now reports success via the export button, which will temporarily relabel itself
• added discord links to help files and help->links menu
• when a GET network connection fails during the read phase due to an unexpected timeout, the request will be reattempted a couple of times, like failed connection initialisations currently are
• escape key now closes scrolledpanelframes (review services or import status frames)
• manage tags dialog and frame now closes due to escape key correctly
• fixed a size calculation bug that was not initially drawing scrollbars on manage options and any other listbook-containing scrolling panel when the screen is too small to show the whole dialog
• undo menu now works on the new menu system
• cleaned up some bad gui-thread interaction in the file import dialog
• fixed file->restart in the built release, including when the install path includes a space
• the trash service no longer records which files it has physically deleted, as this information is not used and is redundant compared to the existing local files' deleted record (existing records will be deleted on update)
• the subscription daemon will wait 90 seconds after boot before triggering--quitting the client before then will result in subs not being checked

• version 233

• made a plan for faster dupe search
• created skeleton of db tables for faster dupe search
• wrote out search algorithm for faster dupe search when the rest is ready
• updated search algorithm and skeleton to support a future multiframe (i.e. gif/video) similar files comparison
• the os x release now has 'client' as the main client executable
• the os x release now includes the server, under the 'server' executable
• added studio (128,0,0) and meta (0,0,0) default namespace colours
• the 'add all' button on the file lookup tag suggestions panel now will only ever add--it won't remove/rescind pend
• the password system now supports non-ascii input (be careful though, as even a subtle change in keyboard encoding that nonetheless may have the same visual characters will likely be considered a different password)
• fixed the 'nonetype has no dtype' rendering problem introduced by last week's 16-bit channel fix (this mostly affected static gifs)
• fixed non-null ratings changes in the numerical ratings dialog
• fixed culling and adding variables not initialising on the edit html formula panel
• fixed the htmlparser attribute fetcher to deal with both single value attrs ('id') and multiple value attrs ('class'). in the latter case, the many values will be joined up, as how they appear in html
• tweaked disk cache timings a little more to account for more scenarios--it now also reports itself to the shutdown splash screen

• version 232

• finished png object sharing system, created some gui for it
• switched the top half of the new png export to wx code, which can render with prettier system fonts
• you can now 'export' parsing scripts to png from manage parsing scripts for easy sharing
• you can also 'import' parsing script pngs in the same dialog!
• the file lookup tag suggestions panel now listens for media update events (so if you browse the media viewer with manage tags open, it'll keep up with the current file)
• added a 'add all' button to quickly add all tags from file lookup suggested tags panel
• created a 'script management' control for file lookup suggested tags panel and added some script text status updates for it (progress gauge doesn't do anything yet)
• added basic 'cancel' support to script management control--mid-http cancel support will come with the gauges
• added url support to thread notification objects, and hooked script code into it
• added a url menu button to script management control, so you can launch any parsed urls the script found in your browser
• html parsing formulas now support positive and negative front and end character culling
• html parsing formulas now support prepending and appending arbitrary text
• added some more pretty text to the html formula summary to explain when they do the new culling or adding
• script 404s and other network errors are now caught neatly and reported
• link node 404s and other network errors are now reported and recovered from
• fixed favourite file lookup script selection on initialisation
• fixed parsing veto test to include membership of search string in result, not just exact match
• fixed html formula tag attribute fetching
• updated os x release to python 2.7.10
• fixed os x about window
• fixed v231 linux release image rendering crash
• updated linux release to wx 3.0.2.0
• updated linux release to opencv 3.1.0
• fixed rendering for 16-bit-per-channel images
• fixed a potential out-of-order settagboxfocus event issue on panel initialisation
• created a 'menubutton' control and switched existing code over
• created some 'nullipotent panel' dialog code to handle some new stuff
• misc fixes
• bunch of misc refactoring

• version 231

• added file lookup scripts suggested tags control and appropriate options panel
• added final bit of gui->filelookup script->content update pipeline conversion and tie-in code
• moved all the suggested tag options into a notebook
• added notebook layout for suggested tag control columns. it is now the default
• suggested tag columns now have a unified set width
• the parsing ui's add script and node buttons now spawn menus rather than the awkward listofstrings dialog
• added 'iqdb danbooru' file lookup script. it hangs the ui pretty bad atm, but that will change in future
• fixed a critical pyinstaller problem with the os x builds
• log files will now be appending with 'year-month', and will roll over to a new file as the month turns
• help->about dialog now has some library version information
• converted more menu stuff to the new system
• wrote the guts of a new png-based object sharing system
• dropdown choices are more resistant to missing init and invalid defaults
• tweaked some disk cache maintenance timing
• fixed a multi-version update bug regarding external thumbnail paths not being initalised
• misc cleanup
• misc cleanup, improvements

• version 230

• all multiline text ctrls now support ctrl+a for select all, wew
• added dynamic menu_item tracking to new menu id system
• added submenu tracking to new menu id system
• added comprehensive menu, submenu, and menu item destruction to new menu id system.
• 'pending' menu now works on the new menu id system--is likely a major cause of the menu id bugs
• added 'info' tabs to the script/nodes/formula parsing gui panels
• added a 'veto' content type for discovering undesired redirects
• added special veto support to parsing engine
• expanded the content node panel to handle and display multiple content types
• added 'link' parsing node panel
• expanded 'link' panel test page and tied in secondary layer example url and data forwarding to children
• fixed the parsing node children control not forwarding example data up to nodes on edit
• file lookup url is now explicitly stored in the file lookup script object
• file lookup url is now passed to child parsing dialogs for test example data purposes
• expanded internal 'requests' interface a little--it now also always includes the 'user-agent' request header as the typical 'hydrus/version_number'
• wrote script GET/POST engine gubbins and tied it into the respective test page
• wrote link GET engine gubbins and tied it into the respective test page
• 'person' namespace added to default namespace colours--it is RGB ( 0, 128, 0 ), a darker version of 'character' green

• version 229

• edit html formula dialog can now be launched from edit content node panel
• edit html tag rule dialog can now be launched from edit formula panel
• edit formula panel test now works
• edit content node panel test now works
• edit script panel test now works
• html formula now filters out missing content rather than complaining
• misc parsing work
• fixed some listctrl resize columns to be more helpful
• attempting to size a resizing listctrl column will now move the resizing column to the last column, which will avoid the flickering scrollbar sperg-out of previous
• attempts to resize the final column of a listctrl while it is the resizing column will be silently vetoed, which will avoid more resize event flickering
• frame location options panel now allows for negative position coordinates (for multi-monitor setups)
• all the new resizing panels now handle certain size updates in a more unified and reliable way
• fixed an issue with hover windows' mouse position check breaking during media window init/shutdown edge cases
• avc1 mp4 support added
• ftypFACE mp4 support added
• ftypdash mp4 support added
• mpegts mpeg support added
• to help migration, paths are now only stored as portable (i.e. relative to the base path) if they are beneath the base path (previously, this was true for any path on the same partition, permitting PITA '../../my_files' portable paths)
• to help migration, portable paths are now relative to db_dir, not installation base_dir (existing client_files_locations, ideal client_files/thumbs_locations, export_path, and HTA paths will be updated)
• started work on cleaning up button code across the program
• created a better way to bind menu events that shouldn't suffer the id overflow bug
• started work on replacing the old id_to_action_cache menu mess with the new system
• too-many redirect exceptions are now richer and caught more sensibly
• circular redirects are caught as soon as they occur
• reduced sqlite cache size (with the new disk cache, this is now less important)
• client.py is now included in the source and windows frozen releases

• version 228

• added support for RIFF .avi files
• fleshed out parsing ui, made progress on node editing, moved edit/test panels to notebook pages to reduce clutter
• the video renderer now initialises off the gui thread, which should reduce some video browsing chunkiness
• trying to save a session that would overwrite another (from the same name) now throws up a yes/no warning
• reintroduced '--no_daemons' and '--no_wal' to the new command line argument parser
• you can now set the default tag service for new search pages under the file->options->tags page
• since it can now be anywhere, added 'db directory' to the file->open menu
• fixed a build capitalisation issue that was making the windows frozen exe release crash when trying to run from source
• the 'too many redirects' error message is expanded to print the url called and its redirect request
• errors from trying to delete files in use by another process will no longer make popups--they'll just write a note in the log
• improved how some errors are written to the log
• misc fixes

• version 227

• both the client and server now use standard command line input and will produce a proper help with a -h switch (although the 'windowed' frozen client executable can't print back to console)
• both the client and server now support a -d or --db_dir to set a db directory outside of the install directory. you can now run multiple clients off the same install!
• critical boot error reporting is more reliable
• if the admin port is in use on server boot, the error is handled better and the program will quit more easily
• fixed mime parsing for some unusual webms
• added support for even more unusual webms
• added support for SonyPSP mp4s
• fixed system:hash when searching for a non-existent non-sha256 hash
• fixed layout of the 'dismiss all' popup button
• preview windows will no longer update in the background if the 'hide preview window' option is set
• misc fixes
• misc cleanup

• version 226

• fleshed out a bunch of parsing script engine and gui framework and added an 'under construction' entry to the services menu. feel free to play around with it!
• fixed autocomplete same-service sibling count merging
• sped up how autocomplete counts are put together
• results will build faster due to an improvement in how tag rows are fetched
• results will build faster due to an improvement in how tag strings are fetched
• incidence taglist sorting now secondary sorts by lexicographic in the 'correct' direction (a-z for desc, z-a for asc)
• all listctrls now support ctrl+a to select all
• cleaned up some superfluous and possibly buggy canvas zoom calc when using the open externally button
• the zoom_switch event will now reset the drag coordinates (recentering the image) if the resultant zoom fits into the canvas frame
• the mouse cursor hides itself more quickly on media drag on Windows (so warppointer jitter is hidden)
• the popup message manager will be careful about switching from hide to show while the parent gui is minimised on windows, which should reduce the grey box problem
• the popup message manager will specifically re-layout on a transition from hidden to shown to stop the 'crushed into a corner' layout problem that would sometimes happen here
• popup message manager will now consider a minimsed gui as unfocused for the purposes the of the focus BUGFIX option
• added a new delayed popup message item to the debug menu for testing minimised popup creation
• vacuum maintenance period option is moved, reset to larger default of 30 days
• 'last session' will now always be updated on client close, even if it is not the default session

• version 225

• system:numtags is now much faster when applied to 'local files' or a file repository
• system:numtags now correctly counts same-tag-different-namespace tags as distinct (for instance, [page:1, chapter:1] was previously being counted as only one)
• refactored fast db integer iterable access code into a context manager
• media result building is faster
• added namespace-grouped incidental tag sorting
• fixed the OpenCV image loader for monochrome images
• fixed a thread interaction issue when drawing popup messages
• popup messages should no longer flicker while static
• popup debug test is richer
• fixed editing of some taglists such as in explicit tags and import tag editing
• SSL EOF errors are now caught by the networking engine

• version 224

• rewrote the static image rendering and caching pipeline -- images are now resized on the fly, and only the master image is cached
• all image rendering and zooming is faster and consumes less memory
• image rendering now obeys the zoom quality options in the 'media' options page!
• static image zooming will take advantage of OpenCL (video card acceleration) wherever available
• added an option to the 'media' panel to allow fast but potentially unstable opencv image loading
• fixed the tumblr gallery downloader (tumblr put an extra character in their API response, wew)
• the debug code profiler now prints more information
• cleaned and updated reducing lag help page to match new profiler
• 'recent' suggested tags are now sorted by recency
• the thread watcher now checks url history as well as md5 match to compensate for cloudflare optimisation making for unreliable api and causing dupe downloads
• misc fixes

• version 223

• fixed the popup message manager's bad minimise recovery
• added an option to automatically 'hide' the popup message manager on main gui minimise (which has patchy multiplat support--some mindow managers don't do this automatically, some do but break when you force it)
• added an option to automatically hide the popup message manager on main gui dofocus, which may help with window managers that minimise to the system tray, which wx cannot detect
• added a 'make some popups' test job to the help->debug menu
• siblings will now present in a more service-specific way
• many gui elements that display tags are now aware of their tag service for the purpose of collapsing siblings
• simplified a whole load of the siblings code
• in some conditions, media will load faster
• autocomplete tag censorship will now also apply on in a service-specific way in 'all known tags' queries
• another remote connection reset exception is now properly caught by the network engine
• the embed button will now draw the correct background colour behind transparent thumbnails
• the animation scanbar will no longer scan if it has not previously experienced a mouse_down event--it could sometimes inherit this status from a previous media filter or embed reveal, resulting in undesired instant scan and undefined animation canvas behaviour
• advanced content update panel will no longer list 'copy' as an action if 'local tags' is the only tag service
• fixed a bug when performing a 'go!' action in the advanced content update panel when 'local tags' is the only tag service

• version 222

• created a 'raw url' downloader page that just downloads urls and tries to import the result. it has a 'paste urls' button to make mass import of a list of urls easy
• fixed an options update bug when updating to v221 from any version before v220
• added support for 'ftypqt' quicktime (usually .mov) video
• embed button now uses system gui colours
• embed button now puts the thumbnail of the media, if one exists, behind the 'play' button
• sped up an inefficient existing mapping check that was slowing new pending mappings for popular tags
• 'last session' will no longer be listed on the gui session delete menu
• cleaned up the main gui's initialisation events--a sizing bug often triggered after system reboot may be fixed
• popup messages are initialised in a safer way
• popup messages are dismissed in a safer way
• popup messages will hide/restore themselves more reliably when the main gui window is minimised/restored
• the pending popup message queue is now regularly purged of already deleted messages
• new popup messages will no longer raise the main gui window to the top
• subscription http errors during the gallery sync phase are now caught and handled gracefully, with exact error text written quietly to the log
• network timeouts during successful response read are caught and converted to a hydrus network exception that will be caught and handled more reliably up the chain
• the client's upnp daemon will now silence upnp mapping errors that are due to the router being too busy or full or any other unknown errors. a simple statement about the error and an instruction to explore the problem with the manual upnp manager will be written to the log
• finished flexgrid refactoring
• the new automatic flexgrid creation detects subsizers and lines them up more accurately with standard controls
• wrapped the different sections of the 'speed and memory' options panel into staticboxes
• wrapped the misc crap up top the 'tags' options panel into a neater staticbox
• taglists with unusual tags will copy them more reliably and present fewer invalid menu options
• misc layout fixes

• version 221

• fixed the 8chan thread parser for the new sha256 file urls. legacy links should still work!
• created a 'x recent tags' suggested tags control and all the db and so on to go with it
• recent tags defaults to off--turn it on under options->tags
• replacing the previously basic zoom_in_to_canvas, the zoom options now support 100%, max regular zoom, or canvas fit for scaling up and down and for the media and preview windows!
• cleaned up some garbage zoom code
• refactored and harmonised how default zooms are calculated
• harmonised and improved how canvas zooms are set
• zoom_switch now works for flash
• if it looks like the last instance of the client did not shut down cleanly, a new client will now present a dialog to choose whether to attempt opening the default session or just a blank page.
• all listctrls across the program have better secondary sort
• added an isatty() stub to the HydrusLogger for any unusual error handlers wanting to call it (youtube parser pafy was doing this)
• removed some accidental (123) counts from some predicate menu and clipboard rendering
• refactored a bunch of redundant flexgrid creation code
• did a bunch more flexgrid refactoring
• refactored some ugly global variables
• misc mvc cleanup
• misc cleanup

• version 220

• fixed collection selection show action lookup for the preview window
• files added through the normal import dialog or import folders will be not be fed into the import pipeline if it looks like they are already in use by another process
• (hence setting an import folder to a browser download destination should be more reliable)
• rewrote some critical phash generation code that was running extremely slow when importing large pngs with transparency
• all listctrls that have an edit button now also trigger the edit event on item activate (e.g. double-click/enter key on an item)
• added zoom in/out/switch to edit shortcut action dropdowns, can't promise it works everywhere
• 'stopping' a gallery page search will now correctly make sure the search queue is unpaused so it can move on to any subsequent search
• the way excess messages are added to the popup message manager is quicker, improving gui responsivity on message spam
• fixed a potential race condition when a file import occurs at the same time as client_files rebalancing
• fixed a potential race condition when a check_file_integrity db maintenance occurs at the same time as client_files rebalancing
• the client will no longer suppress some core debug stuff
• fixed an invalid event handler on canvas close bug
• fixed a long-time wx locale issue
• fixed a misc main gui parent assert issue
• fixed some invalid non-wx-thread calllater calls
• cleaned up some inartful showmessage calls
• cleaned up some datacache init code
• fixed some shutdown event handling
• generally improved and fixed in some cases how threads signal job status
• improved some thread-interaction timings
• improved how files and thumbnails are deleted
• file imports add their files to the client_files structure in a more sensible way
• misc fixes
• misc layout fixes
• misc cleanup

• version 219

• wrote framework for per-mime zoom options--mimes now have separate show actions for the media viewer and the preview window and zoom in to fit
• added 'half/double zoom' to limit a mime's zoom to only 25%, 50%, 100%, 200%, 400% and so on (png defaults to this)
• added placeholders for zoom in/out algorithm quality, but they don't do anything yet
• added a listctrl to edit all this to options->media
• added a scrappy zoom values option to options->media
• cleaned up a bunch of zoom code
• added video/mpeg support
• fixed some mime layout stuff
• duplicate tag predicates across the 'all known tags' space are merged into more accurate (3-5)-type counts
• refactored and simplified how predicate counts are merged
• if launched from the manage tags dialog, the advanced content update dialog now correctly filters hta imports to only the files the manage tags dialog was launched with
• the advanced content update dialog has a bit of better text to explain this
• in prep for a complete rewrite of the image rendering pipeline, merged the fullscreen and preview image caches and cleaned up some related code
• added resume recovery to the v215->v216 update code for users for whom this update was interrupted
• directory creation is improved across the program
• fixed the optimised merge-move file code, which was forgetting to clean up the source when no move was needed
• the optmised merge-move directory code will now be much quicker in some situations
• all file moves across the program should be less stupid-error prone, and repeating/resuming many maintenance or update tasks that require a lot of moves will be much faster
• manual exports, which use copy rather than move, should also be more sensible and faster when repeated
• improved database exception rendering, which in some cases was being truncated
• misc cleanup
• misc small updates to help

• version 218

• related tags control now updates on new media events (i.e. if the parent media viewer scrolls to different media)
• related tags also initialises with suggestions and refreshes the list on media change
• related tags now properly filter out dupe suggestions caused by siblings
• went over all the gui colours in the client, cleared some redundancies and replaced all the static whites and blacks with appropriate system defaults or chosen custom colours
• full size thumbnails can now also be stored in a different location, like resized thumbs, under options->file storage locations
• fixed false positive shutdown maintenance checking due to vacuums projected to take longer than the given maintenance time
• fixed false positive shutdown maintenance checking due to sqlite_stat1 creeping in
• pasting tags or otherwise editing many in at once in the manage tags frame will immediately commit all the changes in one db transaction, rather than one for each tag laggily in turn
• removed 'import tags' from file menu--this advanced stuff is now done in and better left to the advanced content update dialog
• restoring from an old db backup with a simpler db structure will no longer create conflicts
• deleting currently rendering files from the trash should be slightly more reliable
• fixed some crash error reporting
• misc cleanup

• version 217

• fixed some high-res video streaming thread scheduling problems with the new video renderer
• fixed a cause of huge memory bloat with greatly upscaled videos
• to improve seek response time, streaming buffer for the video renderer has a much smaller cap
• renderer throttling calculations are more sensible and reliable
• the video renderer discards frames to save time if they happen to still be in its buffer
• the video scanbar now displays the current frame buffer around the caret!
• video canvas now recycles the same frame blit bitmap to save a little time
• wrote a prototype related-tags suggestion 'service' for the suggested tags control
• you can turn it on and set some options for it at options->tags, feedback would be appreciated
• munged increasingly complicated components of the suggested tags control into a clean and proper self-hiding panel
• fixed a very important bug that was failing to filter visible thumbnail fetch on mass select and thus massively slowing down the client on large ctrl+a-like operations
• open externally button now shows the media's thumbnail, if it has one
• open externally and embed buttons now use hand cursor
• the simple path tagging dialog panel now cuts off .jpg extensions from filenames on filename parse
• if the string component of a generated file export path already ends in the correct .jpg extension, a second will not be added
• ipfs unpin will no longer break if the file was already unpinned
• the hydrus server now gives filename (for a file save as dialog) correctly on a content-disposition header (this affects the client's local booru as well)
• the secondary sort can now be a namespace or rating sort
• fixed some potential init problems with some dropdown controls
• an edge case object-missing cache retrieval bug is fixed
• updated openssl on os x, which might have fixed some problems
• updated python on windows, which updated openssl and a bunch of other stuff
• updated sqlite on windows
• updated linux dev machine to ubuntu 16.04, so a variety of packaged libraries are updated
• fixed auto server setup if the client is launched from a windows cmd window
• misc cleanup

• version 216

• video rendering pipeline rewritten to be much smoother
• canvas video render timing rewritten to be more frame accurate and smoother
• video rendering pipeline will deal with 100% CPU rendering bottlenecks much better
• videocontainers will rush to render towards clicked cursor and then slow down
• videocontainers will stop rendering as soon as they are replaced on zoom
• videocontainers will stop rendering as soon as their parent canvas is closed
• added a video buffer size option to options->speed and memory
• client_files subfolders are now split into f, t, and r groups for files, full size thumbnails, and resized thumbnails
• added an option to override resized thumbnail storage location under options->file storage locations
• added secondary sort option to options->sort/collect
• fixed a bug where automatically pended tags could exist on top of current tags, forming redundant (1) (+1) situations
• fixed a bug when sorting by namespace and a respective namespaced tag has a sibling that de-namespaces it
• relatedly, namespace sorting will now filter correctly by collapsed siblings when those sibling pairs' namespaces differ
• and collecting will now collect correctly with sibling pairs where the pairs' namespaces differ
• simplified how the server caches its account sessions
• improved how the server refreshes specific account sessions
• the server now refreshes cached session accounts immediately when an admin actions an account type
• neatened a little server db account fetching code
• fixed file repository petition denial
• added a better catch for upnp addmapping error when the mapping already exists as a port forward
• the media canvas will no longer accept any mouse or key event if a menu is open anywhere in the program (this circumstance is difficult to pull off, but does cause a crash)

• version 215

• you can now set different suggested favourite tags for different tag services
• added an option to tags option panel to set fixed suggested tags control width
• file import status now works on the new window sizing system, and when launched from main gui uses file_import_status frame key
• file import status is now a generic apply/cancel dialog when launched from import folder or subscription dialogs
• these file import status dialogs are reenabled for linux and os x, as they work properly now!
• very large images will render faster and more beautifully on a freshly launched fullscreen media window
• all zooming in will be higher quality, particularly for sharper images
• better zooming in code also works for very large animations/video
• started a framework for better html parsing
• updated sankaku downloader to support 'medium' namespace
• analyze calls no longer analyze sqlite_stat1, which isn't needed and sometimes gives awkward maintenance timings
• fixed server's deleteorphans, which wasn't working with the new thumbnail names
• improved speed and weight of server's deleteorphans
• fixed a missing foreign key in file_petitions
• improved thread worker creation and job processing scheduling
• fixed os x media viewer window float_on_parent status
• default gui session now defaults to 'last session'
• gui recovers from missing default session on boot more gracefully
• crash log code improvements now applied to all boot scripts
• misc cleanup

• version 214

• wrote a new resizing dialog 'edit' class for simple resizing dialogs
• added comprehensive frame size and position options for the new system to the gui options panel
• moved review services frame to the new sizing system
• windows that initialise maximised will correctly return to their last remembered size and position on a restore event
• maximising a window by dragging it to the top edge of the screen should remember last position as the initial drag start position more reliably
• positioning code is a little safer
• fixed some missing recalculation of best/min size for wx.notebooks after page change
• fixed missing recalculation of scrolledpanel's virtualsize after child wx.notebook's page change
• fixed bad parentage for file import status frames
• hid file import status button in manage import folders and subscriptions dialogs for non-Windows, as this is very broken, and the parentage fix wasn't enough
• improved fuzzy padding on size calculations
• refactored and cleaned and harmonised a bunch of the new window resizing code
• suggested tags - favourites tag entry in options is now a live autocomplete dropdown
• cleaned suggested tags - favourites layout in options
• suggested tags listbox now sets its width to exactly fit its tags
• improved workflow logic of removing/petitioning siblings and parents (shouldn't get stuck in loops as much now)
• tag listboxes will update when tag siblings change
• the 'auto-replace siblings' state on manage tags will no longer incorrectly apply to removal actions
• import status caches now display errors in a more straightforward way
• errors sent to import status caches are now also printed to the log
• simplified how database exceptions are caught and displayed
• database exceptions now preserve the original exception type
• fixed db-side traceback line spacing in database exceptions
• improved general database exception rendering
• fixed imports for videos with negative start time
• deleting videos from the trash that are currently rendering should be more reliable
• crash.log now goes to the db dir, unless that isn't determined yet or is unwritable, in which case the traceback goes to console

• version 213

• created a new tag listbox for the new suggested tags control
• added a extremely basic and prototype suggested tags control for favourite tags to options->tags and manage tags dialog
• off-screen position detection is now more lenient to account for window managers that position slightly off screen to cut off border
• new dialog sizing code adds a little more fuzzy padding on all min size calcs, reducing superfluous scrollbars
• hydrus's custom listbook now informs parent scrollable panels of need to recalc scrollbars on page change
• the new flexible dialog and frame now catch these events and will expand their size if they need to and have the space
• the client options dialog now works on the new sizing system--it will automatically resize when you change pages
• some frame resize save event code is improved
• the media viewer is no longer aggressive about claiming focus
• stacked manage tags panel buttons to allow for thinner dialog
• misc dialog cleanup
• corrected some bad logic that was excluding transparent gifs from the new thumbnail handling
• all gifs and pngs will have their thumbnails regenerated on both the client and server, taking advantage of the recent thumbnail improvements
• fixed a bug if you click cancel when choosing an hta path to sync a tag service with on manage tags dialog
• fixed a bug where noneditable services (local tags and local booru) could sometimes become editable on manage services dialog. the edit buttons should now disable correctly on init, and there's an additional check in the add code in case of future problems
• any superfluous local files/booru services will be deleted on update

• version 212

• wrote a neat flexible system for recording and restoring a variety of window size and position information
• moved the manage tags dialog to the new size and position system
• moved the main gui and the media viewer to the new size and position system
• the manage tags window launched from a media viewer is now a non-modal frame, allowing interation with the underlying media viewer while floating on top of it
• the new manage tags frame commits changes immediately!
• review services and file import status will now stay on top of the main gui
• fixed some bad advanced content update dialog launch code
• the hydrus tag archives in hta syncs can now be anywhere, not just in the client_archives folder
• whole bunch of hta sync cleanup
• paletted images are now dequantized to RGB(A) before thumbnail generation, enhancing scale quality
• 'LA' (greyscale+transparency) images will now render with correct transparency
• png files will now always generate png thumbnails (previously it was just ones with transparency)
• trivial options changes are less frequently saved
• rule34@booru.org now parses namespaced tags
• fixed a bug in thumbnail resize error recovery code
• fixed a bug in file repository petitions
• updated to Pillow 3.2.0
• misc cleanup

• version 211

• added options for disk cache init and maintenance to 'speed and memory' page of options
• abstracted the manage tags dialog into a general purpose 'apply' dialog and a scrollable manage tags panel
• the 'auto-replace entered siblings' checkbox now remembers how it was last set
• uncertain sibling tag counts will now appear like 'tag (1,234-1,456)', showing the range of possible results they represent
• fixed a bug where siblings would be collapsed inside the tag manager object if a specific tag search domain was set
• altering the search text on an autocomplete from 'tag' to '-tag' or vice versa should graphically update more reliably, and an exact-matching predicate will be promoted to the top of the list correctly either way
• calls to upnpc will now kill the process and raise an exception if it hangs for more than thirty seconds
• misc improvements and cleanup

• version 210

• manage tags dialog now launches much faster when thousands of thumbs are selected
• manage tags dialog now has a prototype 'do this for all' checkbox to make mass selected-tag editing a bit quicker
• polished the new disk cache turbo mode, added free memory limits and more options
• a bit of disk cache population now runs on client boot, just to kick things off
• a full disk cache population now runs before repo sync
• the client now keeps about 200MB of itself in the disk cache during idle maintenance
• the update files in client_updates are now stored compressed
• the log files and 'already running' files are now stored in the db folder, meaning hydrus shouldn't have to write to anything but db dir and temp dir
• 'already running' files now delete themselves on program close if they can
• the log directory is no longer needed
• if there is only one booru to select, it will be auto-selected
• client failed exit is less likely to hang, more likely to log error
• client won't bother to save lest session regularly or on exit if this is not the default startup session
• hotfixed an auto-vacuum bug
• added a catch for any future vacuum mess-ups to save the db from collapse
• added other catches for other close/reinit db calls
• misc cleanup

• version 209

• the advanced content update dialog can now be !aunched from manage tags, and will only affect the files the dialog was launched for
• gave the advanced content update dialog a bit of a makeover
• system:ratio now supports 'taller than' and 'wider than' operators
• merged client thumbnails into the dynamic file directory structure
• the client files manager now manages thumbnail paths exclusively
• thumbnail import responsibility moved to the client files manager
• thumbnail resizing responsibility moved to the client files manager
• orphan clearing and thumbnail regeneration maintenance tasks moved to client files manager
• 'regenerate thumbnails' now has an 'only do missing' mode
• thumbnail regen on resize is faster
• copy file to clipboard responsibility moved to the client files manager
• media file and thumbnail deletion responsibility moved to the client files manager
• if a full size thumbnail is missing but regenerated ok, the client files manager will now only produce one, better error message
• cleaned some bad error detection and reporting code
• cleaned some regular 'thumbnail size has changed' code
• fixed and cleaned a mix of file and thumbnail path code
• client files rebalancing and recovery will now efficiently move-merge rather than copy-merge
• fixed some bad portable->absolute path normalisation
• improved missing external storage location error reporting
• server thumbnails are similarly merged into server_files
• to keep things simple, the client no longer keeps track of number of local vs. remote thumbnails for file repositories
• perceptual hashes are now calculated from original file, not thumbnail, and only for jpegs and pngs (videos had too many black frames etc...)
• the vacuum maintenance popup will now only appear if a vacuum can be completed in the time available
• slideshows will now yield to any open popup menu, not just those launched from their own canvas
• wrote a prototype turbo mode and put it on the debug menu
• misc cleanup

• version 208

• split the path tagging dialog's complex widgets into simple/advanced panels
• added filename and directory 1, 2, and 3 easy tagging, with namespaces to path tagging dialog
• fixed up some buggy 'tags just for selected files' behaviour
• ipfs directories now support a note
• the ipfs directory note can be edited from review services
• added 'all namespaced tags' and 'all unnamespaced tags' to the advanced content update/service-wide operation dialog
• added 'clear trash' button to trash panel on review services
• separate timestamps for file repo upload and ipfs pin will be displayed on single-thumb right-click menu
• the correct file domain timestamp will be used for newest/oldest sort
• db integrity check now scans all attached database files
• broke up mappings petitions in the server db into more efficient 'deleted' and 'petitioned' parts
• cleared out some static refs in prep for external db location
• simplified a bunch of file mirroring code and improved error handling of directory mirroring
• fixed a rare thumbnail resolution calculation bug
• misc smaller fixes and cleanup

• version 207

• finished ipfs directory creation
• ipfs review services pane now shows current directory shares
• cleaned up some content event responsibility in ipfs service code
• fixed a file download bug when using ipfs 0.4.2
• wrote some more ipfs help, for directories
• popup messages can now hold arbitrary copy-to-clipboard text content
• fixed the new listbook changes for Linux, which was having trouble tracking the client_data inside the sorted listbox
• fixed an issue with the custom filter management dialog possibly duping the 'default' entry on ok, making impossible to again launch
• offloaded tag parent generation responsibility in the a/c write tag pipeline
• added a checkbox to manage tags dialog that will control if siblings are auto-replaced on entry
• all the clunky 'add preferred sibling' workflow is removed from the manage tags dialog
• fixed incidence tag sort order not initialising properly in the options dialog (and transferring to lexi (namespace) every time)
• fixed too-eager quit during shutdown db analyze
• fixed the db restorebackup call, which wasn't closing the gui correctly
• the manage services dialog will now automatically re-check 'allow zero' on numerical ratings with only one star
• misc cleanup

• version 206

• if the currently focused thumbnail is removed from view, the earliest non-selected thumbnail will be remembered as a 'ghost' focus if the user then presses the arrow keys to continue to navigate
• if the user presses an arrow key to navigate the thumbs when there is no known focus thumb, the first will be assumed
• the listbook now supports duplicate display names
• services of the same type can now have the same name
• cleaned some listbook code
• inbox or custom filters will now render first-file animations correctly on initialisation
• fixed the 'change' event on the new timedeltabutton, which was not updating the thread checker on dialog ok
• subscriptions will now naturally terminate their gallery sync on a 404, rather than dumping out with an error
• fixed some false positive 'paths are different' testing in default directory mirroring code (used in db backup, restore) that was slowing these operations down
• if database files have the same file and modified timestamp, they won't be re-copied in a backup or restore
• client external storage location recovery/rebalancing will now skip over files with the same size and modified date
• if you backup the client and have external file storage locations, a popup will remind you to back those locations up manually
• did a quick pass over some help stuff, and added a 'can the client manage files from their origial locations' bit to the faq
• created a simple no-reward patreon, wrote some help about it, and added links in the usual places
• empty 'read' a/c dropdowns will be more careful about idly refreshing their system preds, hopefully leading to less accidental gui hang on large jobs
• main gui menus will be more careful about refreshing themselves when called, hopefully leading to less accidental gui hang on large jobs
• text snippets are stored inside the client db in a better way
• added flat service directory structure storage support to the client db
• fixed a mapping petition rescind sql typo
• fixed the v198->v199 server update code
• misc cleanup

• version 205

• fixed v201->v202 update code, which v204 retroactively broke
• wrote a new maintenance routine to regenerate the ac cache if it ends up miscounting or not generating correctly
• fixed specific ac_cache generation code, which was double-counting because of recent file-add optimisation
• client db maintenance now vacuums attached databases according to their own vacuum clocks
• client will more aggressively vacuum and analyze if the client has been idle for an hour
• client vacuum now prints success statements, with duration, to the log
• analyze now produces an auto-dismissing vacuum-like popup message
• the 'is maintenance due' check on shutdown now accounts for the new vacuum logic and includes analyze needs as well
• shutdown maintenance will use more of its allotted time if needed
• tag archive sync now always filters to local files on initial sync
• if you import a tag archive with sha256 hash, you are now given the option to filter to local files or import everything
• the occasionally crashtastic opencv.imread call is now completely replaced with PIL, let's see if it's too slow
• the new-but-now-redundant 'disable opencv for static images' option is removed as a result
• fixed a redirection location parsing bug that was affecting sankaku subs with >1000 initial files
• fixed some bad analyze timestamp tracking code
• fixed advanced content update code, which still had some mapping service_ids floating around

• version 204

• current, deleted, pending, and petitioned mappings are now stored on service-separated dynamic tables
• any tag service deletion or reset should now only take a few seconds
• fixed a physical file deletion bug that seems to have affected (some?) versions of linux and os x
• reintroduced a revamped 'clear orphans' database->maintenance routine for the client, which will delete or move orphaned files and delete orphaned thumbnails. run this if you are on os x or linux
• improved misc file deletion code
• fixed some multiplatform path conversion, and generally improved how paths are stored and retrieved
• the network application of a parent tag to all files with the child tag now happens serverside, and only on petitioner-approval of a tag parent relationship. it still occurs locally to the uploader, but is now wrapped up in the parent commit
• fixed some numtags search logic
• fixed an issue when mixing system:age with any tag-based predicate
• media_results db calls no longer require a file domain
• 'hide inbox and archive preds if either has no files' now defaults to off
• added an option to disable OpenCV for gif rendering
• the selection tags panel now initialises with the correct tag domain (it was previously always starting as 'all known tags', which is not always true for session pages)

• version 203

• thumbnail resize now happens on the fly--feel free to change it as often as you like, it takes no time at all
• added 'lexicographic (grouped by namespace)' tag sorting to the selection tags box
• added an option to disable OpenCV for static images under the media options page
• panning with the shortcut keys now pans by a twelfth of the media size or canvas size, whichever is smaller
• cleared the analyze timestamp cache for both client and server, which will force a reanalyze of the new db files on the next db maintenance run
• improved some misc analyze code--primary keys are analyzed again
• added an explicit analyze command to the client's database maintenance menu
• the server now analyzes attached database files
• improved an unhelpful server thumbnail error message
• added an invalid NULL check-and-skip to hash cross-referencing
• fixed some invalid a/c write dropdown search domain initialisation
• fixed some borked zoom calculation code that was sometimes lagging the media viewer and leading to 100% zoomed images being sent unneccessarily and then being nastily scaled down
• fixed a file import error if a synced tag archive is missing
• updated sqlite for windows
• misc cleanup

• version 202

• fixed a problem with the v198->v199 update step
• added a bad error catch in the vacuum step in v200->v201
• added some rollback exception handling for unusual situations
• the autocomplete caches under the client_cache subdirectory are dynamically folded into a single client.caches.db file, and that file is folded into the main db journal like the other attached dbs
• ac_cache file and tag processing lag and edge-case autocomplete miscounting should be removed as a result
• cache generation is optimised for empty services
• specific file_caches' add and delete file commands now modify their tables directly, saving a whole load of time that was previously done superfluously filtering
• the specific ac_caches' add mappings command is a bit more logically efficient
• increased cache size for each attached database
• folded all 'attached' db update code into the main update_db routine
• folded all 'attached' db init code into the main create_db routine
• made the vacuum checking routine safer
• reduced analyze log spam
• analyze maintenance breaks its larger jobs up better
• analyze mainenance now analyzes external databases
• the db connection now refreshes every half hour, to regularly clear out journal files
• the hydrus db assumes it generally has exclusive control over its db, so it no longer wraps its read-only requests in transactions
• session load now happens off the main gui thread, and media generation is broken into 256-file chunks, which reduces startup db and gui lag for large sessions
• the youtube video downloader now lists a webm option if youtube provides one (they seem to offer 360p webms on everything(?) now)
• the timedeltacontrol (where you set the period for the thread checker and import folders and so on) is replaced by the more compact timedeltabutton, which will launch a dialog with the old control when clicked
• the server will no longer revisit old updates and cull since-deleted content, as this was proving cpu-burdensome and not helpful to future plans for update dissemination, which will rely on static update files
• misc cleanup of some server db stuff
• content update processing will not spam menu updates throughout, but only notify once at the end, which should reduce idle gui hang due to db access choke
• the autocomplete tag entry will not refresh system pred menu during idle, which should reduce idle gui hang due to db access choke
• shutdown repo sync will now report the update summary text and will not spam the intervening statuses to the log
• moved timestamp out of the serviceless media_result and into the locations_manager, to reflect the database's knowledge of each current service having a different timestamp
• local and trash timestamps are generated from the locations_manager now, and non-local file service timestamps will soon follow (e.g. 'pinned to ipfs_service 3 days ago')
• if the timestamp is unknown for the current service context, it will not be displayed
• file repositories will now only sync thumbnails when their updates are completely synced (to stop 404s from since-deleted files)
• fixed a ( 0, 0 ) resize event bug that was sometimes causing borked media containers on media viewer shutdown
• syncing to a sha256 tag archive will still import the data as requested, but a popup note will explain that as everything will be imported, further syncing is pointless., and the sync will not be saved
• fixed a bug in hta export if you click cancel on the hash_type choosing dialogs
• misc cleanup

• version 201

• exported hash and tag master tables to external database files for both client and server
• the client's 'mappings' table is split into 'current_mappings' and 'pending_mappings', saving about 20% of space and speed for almost all mappings operations
• removed superfluous rowid column from client mappings tables, saving about another 20% of space
• removed superfluous rowid column from server mappings tables, saving about 20% of space
• to save time, service mapping counts now only aggregate current mappings, not current and pending
• exporting to tag archive now only exports current_mappings, not pending
• 'all known files' queries will more cleverly discern their current/pending search domain
• tag count queries will more cleverly discern their current/pending search domain
• cleaned up a lot of mapping-related query code, and any ugly status leftovers from current/pending union
• num_namespaces is no longer tracked for tag services
• unified clientside service deletion code, fixing several orphan-generation issues
• db update will remove all mappings orphans created by a recent tag service deletion
• some db settings init is harmonised into one location
• fixed up some external db settings init
• db connection settings are now applied to all attached db files
• db vacuum code is harmonised into one location
• vacuum cleans up after itself better
• db vacuum is now preceded by a hard drive free space check
• db vacuum can now predict how long it will take
• if a vacuum fails for an unexpected reason, the db will now recover better
• the db recovers from failed backups better
• the server db will vacuum all its databases on a backup, except any that it thinks will take total vacuum time to more than five minutes
• server backup will report roughly how long it took, clientside
• significantly sped up system-tag-only 'all known files' queries for tag services that have many mappings
• significantly sped up system:numtags for =0 or >0 when tag services have many mappings
• the status bar has some improved grammar, reporting overall common mime as appropriate
• the status bar reports single-thumbnail info text when one file is selected
• fixed an image cache key type-matching bug
• data cache now tracks objects with dynamic memory usage
• the static image rendering-cache relationship has been slightly dejanked
• added a couple of checkboxes to the options->gui page to hide/show thumbnail title banner and page indicator
• the manage tags censorship dialog now defaults to the 'all known tags' service on init
• slideshows will not progress if the current media is either a static image that has yet to render or an unpaused animation that has yet to run through once
• service content package status messages now summarise their final commit status better
• this nice new line is printed to the log as a record of repo sync rows/s speed
• the client will shut down faster if big jobs are running

• version 200

• added 'censor tag' to tag right-click menu
• cleaned tag censorship a bit and moved from the gui down to the db layer
• fixed tag censorship for multiple services
• sped up tag censorship a bit
• improved some tag censorship logic for unusual namespace situations
• tag censorship now applies to siblings and parents, even in their management dialogs!
• fixed the missing fit zoom when scrolling through zooms and when flipping between 100% and fit zoom
• cleaned some canvas init code, which was loading bad initial zoom values due to other recent changes
• restored standard static image zoom mode for cv-supported images to high, inter_area quality
• cleaned up some image/animation container init code
• cleaned up some image cache image tracking code
• fixed ipfs pin, which was conflicting with the new ac caches
• extracted client deleted_mappings to their own db table
• prepped mappings processing to work with status-separate mappings tables--it is also a lot cleaner now
• because of the deleted_mappings extraction, a bunch of mappings membership tests are much simpler now
• upped the db cache size to a semi-ludicrous 150MB, which nonetheless seems to improve processing speed
• improved how hydrus databases manage their db file locations
• improved how the client and server databases backup/restore
• improved some non-local thumbnail fetching code
• converted some old YAML stuff to JSON
• improved big job polite pausing code
• cleaned up some clientside imports, and set the client to only import its server code if it needs to boot a local service
• when an import folder needs to rename a file it is about to move because of a conflict, it will now add autoincrementing numbers, starting at 0, and before the file extension
• added some missing shortcut actions (pan_x and set_media_focus) to the shortcut input entry dialog
• tags and siblings that match the current WRITE autocomplete tags entry will no longer have doubled parents!
• improved how system:limit is applied to searches

Changelog 150-199

• version 199

• added 'all known files' autocomplete caches, so manage tags dialogs' autocompletes should now always appear quickly
• the now caches will be created on startup, which may take a few minutes
• deleted the old tag autocomplete cache
• wrote a faster initialisation routine for all known files caches
• fixed an ac_cache cleanup typo
• the client's (usually gigantic) mappings table is temporarily extracted to an external db file, which should have numerous benefits and perhaps a few drawbacks for now
• the update will take a few more minutes, and will need lots of disk space, to extract the mappings
• the server has a very similar external mappings table, so it will also need a while to update, but future admin backup calls will be a lot quicker, due to a much smaller vacuum
• greatly optimised a mass-membership calculation in tag processing
• reduced some db check-spam timer that was adding up idle cpu time because of recent ac_cache additions
• fixed predicate parent sorting
• if swfrender fails to generate a swf thumbnail within 60 seconds, the client will now dump out (rather than hanging indefinitely)
• fixed some string-selection dialog layout rules
• optimised some 'all known files' search queries
• subscriptions will now append new urls in oldest->newest order, roughly preserving that order in the internal url cache
• the 8chan thread watcher now accepts .json API urls directly
• the repository processing routine will handle server_busy errors more gracefully
• cleaned and refactored most media viewer 'canvas' code, dejankifying a whole bunch of related stuff and generally improving the object hierarchy
• cleaned up and unified a lot of logic and responsibility for idle checking and shutdown maintenance stop time for some big jobs
• misc cleanup
• misc dialog cleanup

• version 198

• ipfs download now parses directory multihashes
• ipfs directory trees are thrown up on a new checkbox tree dialog for selection/filtering
• created a new download_urls (i.e. multiple urls in one popup) popup call, which the new ipfs directory downloader uses
• improved a very inefficient line of sql in the mappings fetch stage of query result building that was also used in the manage tags dialog. on my dev machine, fetching 256 files' mappings dropped from 2.5s to 9ms!
• regular gallery pages and 'page of images' pages now have a 'paste input' button to mass-add newline-separated queries from the clipboard
• fixed a typo in new txt_tags import folder code
• fixed bad autocomplete predicate sorting
• fixed some sibling autocomplete search_text matching logic
• some byte-based value/range presentation will now correctly convert to bytes, rather than regular decimal numbers
• improved linesep splitting code across the program
• fixed get/setlabel calls across the program to get/setlabeltext, so ampersand characters are handled properly
• wrote an 'all known files' autocomplete cache db and prepped a bunch of code for it--it should be easy to finish next week
• added a new regex practise link to the regex dialogs
• os x app release should display better on retina displays
• misc cleanup

• version 197

• on client boot, autocomplete caches for specific file/tag service cross-references are now initialised and populated. progress is shown on the splash window
• on client boot, surplus autocomplete caches are deleted
• on service add, new autocomplete caches are created
• on file add/delete, autocomplete caches are updated
• on mappings pend/add/rescind pend/delete, autocomplete caches are updated
• the new autocomplete caches are consulted for all non-'all known tags' queries
• the old autocomplete cache no longer stores counts for specific file services, and the remaining associated maintenance calls are deleted
• databases now start their own mainloops
• databases now wait for their mainloops to finish prepping any large caches before they return to the controller
• the client database waits for autocomplete caches to finish before it finishes its own mainloop
• the padding around flash and the animation bar are included more accurately in some media zoom calculations, which should eliminate some general zoom jankiness and accidental 100% flash zoom coincidences that filled up the whole canvas
• fixed some clientside server boot error spam when local server or booru had no port set
• account refreshes that fail due to a network error will spam less to the log
• fixed .txt unicode tag parsing from import folders, which was not decoding at the correct step
• administrator immediate repository syncs now sync thumbnail downloads if needed
• service thumbnail sync will no longer superfluously check the presence of thumbnails whose files are local
• if a tag entered into the manage tags dialog has a sibling that already exists for all files, then a new 'ignore, as the sibling already exists' choice will appear
• fixed an overcounting bug in 'selection tags' when importing and adding tags at the same time
• fixed a typo in repository sync status text that was overcounting total number of updates by one
• fixed youtube downloader, which broke with the new library on my new dev machine
• the way that tags and predicates are filtered against a tag autocomplete text entry is now much faster
• bumped up the default content update chunk threshold from 100 rows to 5,000, which seems to be speeding up processing significantly, with a cost to recovery latency--see how it works for you

• version 196

• fixed the 8chan thread watcher for boards that host content on media.8ch.net
• improved the thread watcher url check logic so it won't lag with the new fix
• cleaned up the ac generation code a little
• 'all known tags' ac counts are now summed from all the known tag services rather than calculated directly (a <= indicator for when these cases overlap will be forthcoming). this speeds up file add/delete, service reset, a/c fetch time, and general tag processing, and reduces the size of the db
• ac generation code now deals with 'is the entry text an exact match or not?' better
• ac generation code will now no longer produce non-exact-match siblings on an exact match search
• ac generation code will no longer save half complete search text into the db as new tags
• on update, the a/c cache and its helper table 'existing tags' will be cleaned of a lot of orphans, which may take a few minutes
• fixed some bad unicode path parsing when importing files in some OSes, I think!
• fixed some bad read autocomplete sibling substitution
• fixed a bug where autocomplete predicate lists would not update if the new list was merely a reorder (which can happen in some unusual sibling cases)
• fixed the tumblr parser for the subtly new API
• import folders now support loading tags from neighbouring .txt files--check the dialog to set up which tag services you would like to import to
• the ipfs file downloader now queries DAG object links, determines if the given multihash is a directory or other complicated object, and if so politely dumps out (handling of directory downloads is forthcoming)
• some db code is cleaned up
• prepared db code for some future subclasses
• wrote most of the new ac cache db
• misc cleanup
• added some browser addon links to the ipfs help

• version 195

• wrote up a v1.0 ipfs help page
• added ipfs services to 'system:file service'
• added a 'multihash prefix' option to ipfs services in 'services->manage services'--it will prefix the given text to multihashes copied to the clipboard (e.g. you could use 'http://127.0.0.1:8080/ipfs/')
• fixed a bad repo sync processing bug that was incorrectly assuming packages were completely processed when they often were not
• all repos will have their processing timestamps reset on update, forcing a (fast) reprocess of everything they have to cover the previously missed content
• fixed 8chan OP image parsing, which was sometimes failing due to absent md5s in the json
• widened the splash screen a little more so repo sync text can fit a bit better
• a couple of splash status texts are shuffled around
• updated to sqlite 3.11.0 for windows
• if sqlite3 reports at least version 3.11.0, the db will stick to WAL for large transactions (lke vacuum and content processing), since these are fast now
• 'namespace:*anything*' will now only appear if the a/c input has no 'subtag' component., e.g. 'character:'
• import and export folders now print simple summaries to the log if they do some work
• tidied up the 'pages' menu
• the various hardcoded 'delete' key events are now also triggered by a backspace on os x
• added a rough 'copy known urls' to the regular thumbnail menu. this is prototype, let me know how you would like this information managed and displayed in future
• in prep for the new cache layer, the autocomplete dropdowns (and hence search domains of all contexts) no longer support 'all known files' cross referenced with 'all known tags'
• misc cleanup

• version 194

• ipfs pins and unpins can now be queued up like file repository pending and petitioned, through the regular thumbnail right-click menu, which also reports some/all ipfs pinned selection status
• this ipfs action queue is similarly summarised and commited at the normal service 'pending' menu
• ipfs's 'pinned', 'to pin', and 'to unpin' statuses are displayed on thumbnails with ipfs-specific icons
• you can copy the focussed file's ipfs multihash or all the selected files' ipfs multihashes from the thumbnail menu's share->copy->ipfs multihash
• added a .txt tag parser to the 'path tagging' import dialog--it will parse the same sort of txt files the export dialog produces
• the client's new 'requests' network code is harmonised, generally improved, and now produces hydrus-compatible exceptions
• updated help re the local server and boorus now defaulting to off
• db can now remember service-specific filenames (e.g. ipfs multihashes)
• cleaned up some overly complicated and confused thumbnail menu code
• the pending menu now specifies what it is about to do more plainly

• version 193

• the client's local server and local booru can be turned off from their respective management panels, and from now on, the client will initialise with them this way.
• if the local server or the local booru are not running, their copy/share commands won't appear in right-click menus
• the welcome dialog is now a simpler popup message
• incidence sorted tag lists are now sub-sorted by a-z lexicographic
• pasting many tags that have siblings to the manage tags dialog will ask you if you want to always preference the sibling, saving time
• added a 'clear deleted file records' button to the local file service on the review services window
• idle mode now cannot naturally engage within the first two minutes since client boot
• the autocomplete search logic will not count namespace characters in the autocomplete character threshold, so typing 'character:a' will not typically trigger a (very laggy) full search
• putting a '*' anywhere in an autocomplete search_text will force a full search, ignoring the a/c character threshold
• moved some specific 'give gui time to catch up' pause code to the generalised pause/cancel code that a lot of stuff uses, so big jobs should generally be a bit more polite
• split the daemon class into two--one for big jobs that remains polite, and another for small jobs that triggers regardless of what else is going on. this should increase responsivity for a number of scenarios
• fixed some bad wal failure detection and hence no-wal file creation on some instances of db cursor reinit (usually after service modification). because of now many superfluous no-wal files, existing no-wal files will be deleted on db update
• some external storage location errors are improved
• some internal and external storage location init is improved.
• if an error is detected in the external storage location manager, it will not attempt to rebalance again until the client is rebooted
• improved some upnp error catching
• cleaned up some misc shutdown thread-gui interaction error spam
• did some prep work on a future rewrite of daemon jobs pipeline
• split up some mixed file/data/404 'stuff was missing' exception code

• version 192

• added a 'check on ok' button to the manage subscriptions dialog's subscription panel
• check file integrity now prints missing paths to the log
• fixed a typo that was breaking some repo sync download resumes
• if an external storage location does not exist on client boot, a popup will say so
• resetting a service is now much faster when the service is large
• some repo processing stuff is a bit faster
• improved some image rendering error handling
• some thumbnail errors are recovered from in a better way
• improved db update bit rot resilience
• improved the vacuum disk space error popup with better explanation of the problem and solution
• misc improvements
• moved to win 10 dev machine, so several windows libraries are a bit newer
• updated to opencv 3.1.0 on windows

• version 191

• added ipfs service type
• added simple ipfs service gui to manage and review services windows
• added simple ipfs download and import to pages menu
• split confused service data object into a bunch of more flexible and readable subclasses
• raw url downloads are now powered by 'requests' rather than my hacked http engine, and work a hell of a lot better. more to follow
• raw url downloads now pause and cancel correctly
• optimised some a/c cache clearing in mapping update processing
• optimised content update object -> db yield logistics
• added no-daemons command line switch to disable all daemons
• added no-wal command line switch to disable WAL journalling for the db
• if the db fails to read a new WAL-journalled db file it has just created, it will delete the file and attempt to recreate the db, never trying WAL again
• changed the way the db writes big repo syncs to disk so that the job can be abandoned much more quickly
• improved local file not found error to state the directory or exact path that was the problem
• fixed some socket-level error reporting (I think timeout was part of this)
• the petition approval and denial client-to-server network commit is broken up into smaller steps to reduce server lag and the overall likelihood of timeouts
• subscriptions will no longer add files to their 'successful files' button when the file's url seemed new but it actually was previously deleted
• subscriptions will now identify themselves with an additional popup message when they encounter critical errors during sync
• editing a clientside service will force a reset of the appropriate session cache, so if you change access key, the account changes will be reflected immediately
• the import files dialog will no longer spam error popups on uninteresting or empty files--its 'parsing complete' statement is more verbose instead
• the 1.5s gap between parsing complete and the buttons being enabled is also removed. this was supposed to be helpful, to give you a moment to check the files were as expected, but in the end it was just annoying
• fixed a typo in some thumbnail error catching and regeneration
• improved my network code syncing, so linux and os x releases are cleared out of many old files
• misc code cleanup

• version 190

• fixed some hashing recalculation in collections that meant they reported no files in many cases
• fixed some hashing recalculation when 'remove files' is called
• improved the way the client db stores file information and service->file mappings
• idle processing jobs will now explicitly wake up as soon as the client naturally switches from not idle to idle
• the minimum allowed value for the 'max cpu %' control in the maintenance and processing options panel is now 5%
• the maintenance and processing panel is rewritten and laid out a little clearer
• 'busy' is now 'system busy' on the status bar
• force idle and force unbusy are now merged into a new 'force idle' that sticks until you explicitly turn it off
• busy and idle states should now update immidiately after closing the manage options dialog
• improved exit code event order to be less rude to the OS
• improved exit code emergency event handling
• fixed a typo that was not appropriately skipping the 'do you want to run shutdown jobs?' dialog
• file storage folder rebalancing will now occur on shutdown maintenance
• the client now closes down more reliably if the db fails to boot
• the client now closes down more reliably if the gui fails to boot
• if a client vacuum fails, it'll now also raise the exact error sqlite gave
• fixed ctrl+a on the autocomplete dropdown (I think the Insert/IME support change broke it)
• the ways the 'read' tag autocomplete control talks to other gui elements is improved
• the tag autocompletes will now refresh their results lists on more internal variable changes
• the query page management controller manages fewer redundant variables
• updated sqlite for windows
• the client and server dbs will attempt to change to TRUNCATE journal mode if WAL causes a read disk i/o error
• misc code cleanup

• version 189

• split the big analyze db calls into individual table/index calls and moved them from update code to the normal maintenance routines
• on vacuum, both the client and server dbs will now bump their page size up to 4096 if they are running on windows (server vacuum is triggered by running a backup)
• vacuum should be a slightly faster operation for both the client and server
• boosted the db cache significantly--we'll see if it makes much difference for things.
• the way the selection tags control updates its taglist on increases to its media cache is massively sped up. An update on a 5,000-thumbnail-strong page now typically works in 3ms instead of ~250ms. large import pages should stream new results much more quickly now
• sped up some slow hash calculation code that was lagging a variety of large operations
• some hash caching responsibility is moved about to make it available for the add_media_results comparison, which now typically works in sub-millisecond time (was about 16ms before)
• some sorted media list index recalculation now works faster
• some internal media object hashing is now cached, so sorted list index regeneration is a bit faster
• some medialist file counting is now superfast
• wrote a new pauser object to break big jobs up more conveniently and reduce gui choking
• the repo processing db call now uses this pauser
• some copy and mirror directory functions now use this pauser
• backup and restore code for the client now skips re-copying files if they share the same last modified date and file size
• backup code for the server now skips re-copying files if they share the same last modified date and file size
• http cannotsendrequest and badstatusline errors will now provoke two reattempts before being raised
• socket error 10013 (no access permission, usually due to a firewall) is caught and a nicer error produced
• socket error 10054 (remote host reset connection) is caught, and the connection is reattempted twice before being raised
• the old giphy API is gone, so I have removed giphy
• forced shutdown due to system exit/logoff is handled better
• pubsub-related shutdown exceptions are now caught and silenced
• an unusual shutdown exception is now caught
• fixed a copy subtag menu typo
• cleaned some misc hydrus path code
• tags that begin with a colon (like ':)' ) will now render correctly in the media canvas background
• some misc code cleanup
• dropped flvlib since ffmpeg parses flv metadata better anyway

• version 188

• if you have custom filters set up, they will now be listed in the normal thumbnail right-click menu, which will expand for them
• 'refresh account' button's event no longer waits for service response on the gui thread
• 'refresh account' button on review services will now re-enable if the call fails
• added 'hide inbox and archive predicates' checkbox to options->default file system predicates. not sure if this is the best place for it!
• system:hash can now query md5, sha1, and sha256
• existing pages with system:hash will break on update. since this is such a rare predicate, I assume this is not a problem!
• improved db hash cross-reference flexibility
• any pausable popups will now explicitly change their text to 'paused' when they are paused
• upload pending popup now has a title, so it should pause a little more sanely
• regenerate thumbnails popup now has a title, so it should pause a little more sanely
• cleaned up some job pause status logic
• upload pending now works on a stream, so gigantic uploads will cause fewer hitches and overall problems
• upload pending code is cleaned and simplified in several ways, with better error handling and progress reporting
• synctotagarchive is now split into a thousand tiny separate jobs and runs on a separate non-db thread. it will no longer lock the db the whole time
• synctotagarchive popup should be a lot more responsive and will specifically give the gui some time to catch up after every chunk of data processed
• synctotagarchive will update pending tags count (and gui stuff will show the pending tags) as it goes
• synctotagarchive is now a pausable and cancellable operation!
• large repository sync jobs should be a little less laggy on final commit
• system:duration now displays its value as a properly formatted milliseconds time delta, rather than just a big int
• profiles now flush to the log immediately
• some cpu-heavy daemon recheck periods have been extended to reduce potential hangs on slower computers
• some repository sync messaging wording and logic is improved (it should report what it is doing more reliably when you are cpu-busy)
• reordered instances of 'waiting politely', so download text/gauge updates should occur before the wait does
• some misc 'waiting politely' logic is improved
• pixiv ugoira is detected and a better error (that will be ignored by the new sub error throttling code, for instance) is produced
• 'uninteresting mime' errors have nicer import status note text
• fixed the 'thumbnail grid background' custom colour option

• version 187

• file->restart now works for linux
• the 'call to' thread pool now has a delayed start, which fixes a shutdown hang in linux (and perhaps the other platforms, in one way or another), when the user hits 'forget it' on the 'already running' boot dialog
• removed some console error junk on abandoned client boot
• if a requested full-size thumbnail is missing, the client will now attempt to regenerate it
• if a requested full-size thumbnail does not render (for instance, during a request to generate a resized thumbnail), the client will now attempt to regenerate it
• some thumbnail fetching and generation logic is slightly improved
• you can now drag animated media around when the inital drag click begins over the animation canvas
• ditched some ultimately unhelpful mouse warping when the cursor was near the edge of the media viewer when a drag began
• possibly fixed a strange tags_fts4 mirroring problem that has hit some users for a reason I do not understand
• import tag options panels will no longer order their tag services randomly
• fixed a bug where import folders would not save changes to their explicit tags
• fixed a bug where under some circumstances, the gallery importer was attempting to display previously deleted files on the screen, causing errors.
• subscriptions will now pause a few seconds after a file import error, which should solve a problem with client hanging when lots and lots of subscription file errors occur one after another
• if subscriptions encounter 5 file import errors, they will pause their sync and try continuing later
• pixiv manga error handling is improved
• bad mime error handling is improved for all importers
• fixed an important bug in the import status cache where error texts were not being stored as unicode, so error texts with non-unicode-compatible characters were causing exceptions when the session was being saved to the db or the import status frame was being loaded
• existing sessions' import status caches should automatically convert their 'notes' to unicode on update
• removed booru db fetch spam when booru download page has no work to do
• fixed a typo in regenerate thumbnails popup message
• fixed some bad event handling for the gallery page's file limit control
• updated some regex defaults to have better multiplatform support and handle a wider array of valid characters

• version 186

• Windows release is now 64-bit!
• some libraries got updated for windows, to varying minor effects
• BeautifulSoup updated to latest version for everything
• fixed a warning from newer versions of BeautifulSoup by explicitly using lxml
• updated the running_from_source help file to what is currently needed
• ffmpeg.exe is also 64-bit
• wrote a "reducing program lag" page for the help
• added program restart to file menu, doesn't work for linux yet
• improved restart code (it was used in restore database, and for some users would not close the parent process down properly)
• in the media viewer, page up/down, arrow keys and mouse scroll no longer unhide the mouse
• in the media viewer under Windows, dragging no longer unhides the mouse
• the implicit delay in the downloader-importers is reduced, so redundant files should always stream in a super fast now
• export files dialog will now auto-create missing destination directories
• export files dialog will now work off the gui thread (to stop gui hangups on big jobs) and will report its export status through the export button, which will be disabled until the job is finished
• changed the new tab/shift+tab shortcuts because they break panel control navigation (I want to make these editable soon):
• 'fetch autocomplete results now' shortcut is now Ctrl+Space (Raw Control for OS X, not command)
• 'IME mode toggle' is now Insert
• cleaned up how control is handled cross-platform--in general, 'control' is always control for windows and linux, and 'control' is command for os x. this may be incorrect.
• json dump errors now report more information about the specific object and its serialisable contents that are causing the problem
• clipped sibling/parent reason entry dialog message when lots of pairs are added
• fixed a bit of bad logic in sibling/parent dialog->content-update workflow
• fixed the thumbnail canvas not refreshing on a canvas shrink caused by expansion of the management panel that doesn't change the thumb column count
• fixed emergency boot error reporting for when HydrusData won't load
• added some unicode conversion unit tests

• version 185

• right clicking on a tag/predicate in the 'selection tags' box or the 'search' list of active predicates provides new intelligent menu options to discard/require/permit/exclude the selected tags from the current search. try it out!
• this system also works for namespace and wildcard and system predicates
• 'predicate a' and its inverse '-predicate a' are no longer considered the same!
• namespace and wildcard predicates will now remove their inverts if they are added to the list of active predicates
• the way inverse predicates are generated and compared is generalised and improved
• fixed 'open a new search page for tag x', which might have been broken?
• quick-entering a tag in the 'read' tag autocomplete entry will now always replace with a sibling if one exists
• quick-entering a tag in the 'write' tag autocomplete entry will now not replace with a sibling--you will get the option of what you want to do
• when this happens, the preferred sibling is labelled on the 'what do you want to do?' popup in the manage tags dialog
• also when this happens, the affected file count for add/pend the preferred sibling should now be correct
• fixed client support for tags that begin with ':', such as many emoticons. these should now be parsed correctly from websites and keyboard input, keep their leading colon through network and db conversions, and display correctly (for the most part!), despite a little bit of magic to make them work
• created a new TimeDelta control to handle finer tuning of optional periods
• improved timedelta->pretty string code to support >60s times
• manage subscriptions dialog now supports day and hour period, with min period of four hours
• manage export folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately
• manage import folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately
• the thread watcher now supports hour/minute/seconds check period
• folded 'search' and 'download' menus into 'view' and renamed it 'pages'
• 'sessions' moved from 'file' to the new 'pages'
• moved 'admin' menu to 'services->administrate services'
• added 'forced system:limit' to options->speed and memory panel
• if several system:limits exist in a search, the minimum will be used
• added a checkbox to options->downloading to replace the traffic light waiting politely indicator with text
• subscriptions with failed files will say 'x urls in cache, y failed' on their dialog panel
• at the gui level, media that superfluously receives a 'pend tag' instruction for a tag that is already current will discard the instruction (this improves the accuracy of the pending tag count during and after the manage tag dialog)
• and the same thing for 'petition tag' when the tag isn't already current
• any accidentally added 'namespace:'-type tags will be deleted on update
• fixed a bad merge in the manage tag siblings dialog's autocomplete dropdown lists
• the thread watcher can now deal with urls with a #54951495 post anchor at the end
• refactored some controller manager code
• removed some useless old code
• cleaned some misc code
• improved some bad old orphan deletion code
• deleted some old code
• finished off some old media result streaming code
• refactored a bunch of search stuff from clientdata to clientsearch
• rewrote some subs gui/help text to be a bit clearer, and added a help link to the dialog

• version 184

• added external client_files storage!
• you can add external client_files folders in options->file storage locations, further giving them weight
• a new daemon will incrementally rebalance your files (and recover orphaned subfolders!) over your different storage locations
• you can also force a full rebalance from the new database->maintenance->rebalance file storage
• simplified the maintenance and processing panel
• the maintenance and processing panel controls now appropriately support being set to 'none'
• the 'run jobs on idle?' question is now explicit on the maintenance and processing panel
• deselecting 'run jobs on idle/shutdown' will now disable subordinate controls
• hitting tab on the autocomplete control now triggers an immediate 'fetch results' call
• added a checkbox to options->speed and memory to completely disable automatic autocomplete results fetching (i.e. if you want to manually control a/c result-fetching only with this new tab shortcut)
• the new less-laggy autocomplete results fetch won't trigger if the latest query is shorter than the cached query (i.e. you won't get lag when hitting backspace a bunch of times on autocomplete)
• hitting shift+tab on the autocomplete control now disables all other key event capture, letting you enter IME without your up/down/enter presses controlling the dropdown list of results
• 'waiting politely' time is now indicated with a small 'traffic light'-type circle control on all downloader pages
• fixed some occasional 'I'll go sit in the top-left corner of the screen and not fix my position' hover window behaviour in Linux
• fixed sibling predicate collapse for the 'read' autocomplete dropdown for database results
• refactored how predicates are collapsed and sorted to be a bit more sensible
• fixed maintenance of local booru data use even if local booru does not receive any requests
• improved some service/content update error handling in repository sync
• the system:rating value-entry dialog no longer lists its rating services in random order
• 'open externally' will no longer show for non-local thumbnails' right-click menus
• fixed audio/pdf thumbnail display for local booru, although they will be full size for now
• harmonised some hex-prefix folder generation code
• improved some file and directory copying code
• improved upload pending popup message cleanup
• added help_dir to hydrusconstants
• miscellaneous refactoring
• misc cleanup
• extracted server-specific services and server resources from the common import path to the server import path
• a bit more misc server refactoring
• updated a couple of bits of help

• version 183

• added swf thumbnail support--it works ok for most swfs!
• thumbs for existing swf files will generate on db update for both client and server
• the server will also generate thumbnails for video on update, which it seems it was not doing before
• rewrote some hash comparisons in thumbnail downloading and thumbnail counting code to be a lot faster and memory efficient
• fixed thumbnail count invalidation after service sync
• in certain cases, fetching autocomplete tag results from the db will be slightly less laggy
• if the autocomplete dropdown has cached or otherwise quickly fetchable results, it will refilter for current input as you type, ignoring the normal character delay (i.e. typing through autocomplete is far less laggy now)
• fixed a couple of autocomplete tag-parsing bugs for tags with more than one colon character
• fixed some key event selection bugs in the autocomplete taglist when there are no results in the list
• if there are no results in the autocomplete taglist, the typically caret-moving key events up/down/home/end will move the caret (rather than being uselessly passed on to the empty taglist)
• all profiling now goes through a single location
• profiling now also prints extensive information on explicit code callers
• fixed db profile mode printing display for the new logger
• added new pubsub_profile_mode (AKA log-killer)
• all menu popup display and explicit memory cleanup is done through a single location
• hover windows will now not show whenever a menu is open
• hover windows will now not hide whenever a menu is open (useful when trying to go 'copy tag' off the hover taglist's menu!)
• a keyerror bug when force_idle_mode was hit right at program start is fixed
• unexpected serverside client disconnections are now caught and further request processing is cancelled
• middle clicking on an 'empty' system predicate in the a/c dropdown now will throw up the dialog to enter a value and then pass that filled in system pred onto the new search page
• duplicate parents (which can occur with overlapping grandparents) are now collapsed
• the upload pending popup message cancels properly after 'server busy' events
• whenever the client's services are iterated at the view-level, it will now be in random order, to stop sync bottlenecks choking other services (and anything else like this that might have been occuring)
• fixed a bug with double-clicking a tag selection on the the tag censorship taglist
• fixed the too-fast frame timings on some videos opencv had a problem parsing
• loading many video types will now be just a little faster
• remote files will no longer present the right-click menu option to share->copy->file
• hitting a shortcut for manage ratings from the media viewer canvas will no longer work if you have no ratings services
• the base scripts (client.pyw, server.py and test.py) now have a shebang line to facilitate non-Windows use
• laid groundwork for external client_files locations
• plenty of misc cleaning and refactoring

• version 182

• all printing to the log should now be unicode safe
• some other, miscellaneous file-write locations should now be unicode safe
• cleaned up some now-redundant unicode-bytestring conversion error handling
• the animation scanbar will now draw a little darker while the animation is paused
• the client now uses the same new log object as the server, so all logged data will be copied to the terminal, and all logged statements will be timestamped
• the client and server's boot and exit statements are more harmonised
• server backup is more log-verbose
• server backup now makes a simple copy to 'server_backup' folder--no more _backup nonsense
• all serverside requests will now print a line in the form 'PORT METHOD PATH HTTP_CODE in TIME TOOK' to the log
• page up and down now work (again?) for the thumbnail view. adding shift also works for selecting a page at a time
• improved some animation painting logic
• improved some rating control painting logic
• improved splash screen painting logic
• improved all other, misc painting logic--many things should flicker less or render just a little quicker
• ratings hover window will no longer re-layout (causing flicker) on ratings set
• improved some media canvas painting logic to de-jaggify some zooming
• in the manage options dialog, tag-related gui options are moved from the gui panel to the new tags panel
• added 'apply all parents to all services' option to the tags panel
• the delete key now removes active search predicates and tags in the manage tags dialog
• fixed webm link parsing for rule34hentai.net
• if you attempt to petition multiple tags, you will now be presented with a dialog asking you if you want to use the same reason for all the petitions
• edit import folder dialog now has scrollbars and will resize itself based on your monitor size
• if the static ffmpeg executable is absent from the bin folder, the client will now attempt to just call 'ffmpeg' in the normal system path
• fixed some 'None media' data calculation bugs when media viewer closed during very fast slideshow
• fixed a 'None media' mouse position bug when media viewer closed during very fast slideshow
• all wx timers will explicitly stop on exceptions (which should reduce some types of error spam)
• refactored client hydrus network session manager to a better position
• wrote a better wma/wmv determining test using ffmpeg
• refactored cv2 (OpenCV) out of the server's import tree
• fixed some server boot crash error handling
• cleaned some autocomplete matches-compiling code
• deletepath and recyclepath will no longer throw an error if the path does not exist
• server_messages folder is no longer referenced in the code--if you have it, feel free to delete it
• memory cleans up a little faster after gui page deletion

• version 181

• fixed a potential bug in the server's db, very important you update this week if running on Windows
• improved some thumbnail page calculation logic
• improved some thumbnail page drawing logic
• fixed broken vertical resize thumbnail grid issue
• fixed some broken 'thumbnails have changed dimensions' event layout recalculation
• fixed an idiotic typo bug that was making cached thumbnail page bmps taller or shorter than needed
• thumbnail fading and other general thumbnail redraws should be a decent bit faster now
• thumbnails have handed off all bmp storage responsibility to the thumbnail cache, which should mean greatly reduced memory use when browsing very large results
• thumbnails will now only quick-draw if they have been seen before on the same page, which should reduced some page-refresh waterfall jankiness
• thumbnail media panel double buffering seems to be working better
• improved some general thumbnail drawing code
• the media viewer background taglist will now show petitioned tags
• the media viewer hover taglist should now look exactly like the background taglist (sorted lexicographically, no more counts, and now shows petitioned)
• the 'selection tags' taglist will now show petitioned tags for 'all known tags' queries
• the 'selection tags' taglist and the media viewer background and hover taglists will now count collapsed sibling tags correctly
• all taglists now support ctrl+a to select all
• improved some generic taglist value-setting code
• parents should now only expand for their specific tag services
• siblings, parents, and the pending count should now be correctly recalculated upon repository update processing
• added 'copy sources' to the import status window's right-click menu so you can copy bad urls or whatever to the clipboard
• added 'skip' to the import status window's right-click menu so you can skip urls you don't want
• added default booru for rule34hentai.net
• under certain circumstances, services with duplicate names could be created--any services like this will be renamed on update, and the loophole where this could happen is now closed
• improved a locale number formatting call to ask for unicode
• improved newgrounds parsing--mp4/wmv files should now work
• the server logger has been formalised into a proper class
• fixed some bad static image initialisation in the canvas code that was causing error spam
• fixed default import tag options for hentai foundry and pixiv
• copy->(bmp|path|local url) will now only appear as an option if the currently focussed media is local
• fixed some gallery identifier print code
• frames and dialogs will now clip their initial size to the current monitor's resolution--better scrollbar support for cramped windows will follow
• updated ffmpeg static builds
• updated sqlite command line exes
• put off sqlite windows dll update because of critical server db bug
• misc code cleanup and redaction

• version 180

• middle click on any taglist now does 'open a new search page for selection'
• moved mouse idle checking code to the wx thread, which seems to have fixed a serious X11 crash
• memory gunk should now be painted over when transparent animations are resized on linux
• fixed some page up/down index calculation on the autocomplete dropdown
• system predicates will remove themselves from the active predicate list silently again
• system predicates will work again in queries that include tags!
• greatly improved my string/unicode conversion code
• dropped many inefficient unicode conversions from db code
• improved the sql-grammar of some specifically efficient db code
• removed incorrect unicode conversions from url/network request code
• removed unicode conversion code from network response code
• all directory iteration should now occur in unicode
• all inputted paths should now be converted to unicode
• replaced every single ugly raw path concatenation with a better method
• improved some bitmap placeholder initialisation that was causing Linux terminal spam
• added an option to always hide the preview window to options->gui
• the preview window will not accept setmedia calls if it is hidden
• if the preview window is actively hidden due to an unsplitting event, it will null its current media
• fixed a shutdown calculation that was under-remembering the height of the preview window by four pixels every time
• improved thumbnail waterfall code--it should be a little faster. less subject to lag when taxed, and waterfall with more genuine randomness
• improved some thumbnail page drawing and clean/dirty conversion code
• when pages are dirtied, any to-be-waterfalled thumbnails are cancelled, making superfast scrolling less crashtastic
• new dirty thumbnail pages are now generated as needed, rather than precached
• subscriptions will now correctly do a polite wait after fetching tags for redundant files
• splash screen will now explicitly raise itself to the front on init
• the export tag txts function of the export dialog now exports the tag txt to [filename].jpg.txt, rather than [filename].txt
• the server now stores its updates in service specific subdirectories, like the client
• some importers will better report when they are waiting politely
• OS X (and anything else with OpenCV 3.0.0) now uses OpenCV to render gifs by default, falling back to PIL on error (like only Windows used to)
• 'open externally' from thumbnails now defocusses the preview window
• refactored away a bunch of bad global controller references
• added some shutdownexceptions
• silenced shutdownexception terminal spam
• reworded the 'maintenace and processing' options panel a little to explain things better
• improved how the base_dir is figured out on program boot
• improved serverside 'should I clean dirty updates?' check code
• deleted some old code
• split path-specific functions off from HydrusFileHandling to HydrusPaths and de-looped my import tree appropriately
• general refactoring

• version 179

• all tag listboxes support multiple selection
• all tag listboxes manipulate their data as sets of tags rather than individual tags
• all tag listboxes report to their callbacks as sets of tags
• many dialogs and other windows that use tag listboxes now deal with tags as sets
• double clicking the media viewer's tag hover window now launches the manage tags dialog
• the media viewer's tag hover window now includes (+1) counts like the manage tags dialog--harmonising the underlying canvas tag list will follow
• lots of tag/sibling/parent code has been iterated over
• ditched some misc redundant code and needlessly tightly coupled object relationships
• some bizarro tag sibling dialog code has been rewritten
• the autocomplete dropdown for 'writing' will now not expand search results to include parents when the receiving control isn't interested in them or it is otherwise not appropriate
• when the autocomplete dropdown does have parents, they will be selected in a group with their child
• the autocomplete dropdown for writing now broadcasts 'I'm done with tagging, close the dialog pls' in a better way
• 'activating' the sibling or parent dialogs' listcontrols (usually by hitting enter) now processes all rows that are selected, not just the first
• manage siblings and parents dialogs can now take multiple initialising tags from the taglist right-click menu
• removed some redundant listbox code
• cleaned up and improved some tag list event processing
• cleaned up some taglist right-click menu code
• added system-wide mouse idle test to idle calculation. you can set this in files->options->maintenance and processing, and it defaults to ten minutes
• import folders now support tag import options' explicit tags for multiple tags to any tag service
• existing import folders will be updated to the new version, and if a local tag exists, it will be intserted into the new import tag options
• fixed a hentai foundry page parsing bug
• the deviant art downloader can now download >1024 pixel width versions of images via the download button (and a bit of cookie magic)
• gallery page queries that 404 (like for a non-existent username) will now report 'Gallery 404' rather than spamming the gallery's whole custom 404 html page to the status box
• fixed a bad layout flag that meant some namespace checkboxes in tag import options could remain hidden during first panel expand until first mouseover
• the import status frame now initialises its status text properly
• fixed last week's num_tags optimisations, which accidentally broke num_tags < x for x > 1
• the client should work on both OpenCV 2.4.x and 3.0.0 (thankfully, the only difference for our purposes turned out to be some static variable renaming)
• the Windows and OS X releases now come with OpenCV 3.0.0
• increased the max period of import and export folders to 30 days
• the launchfile/directory thread is now a daemon, so the (some flavours of Linux) client can shutdown even if an externally launched file/dir remains open
• misc cleanup

• version 178

• import tag options now supports 'explicit tags', which will be added to all files imported
• the default import tag options system supports explicit tags, and will propagate them to new import tag options objects
• the gelbooru 503 Forbidden problem is fixed
• all gallery downloaders now send the page_url as http Referer header when fetching their image_urls
• pixiv parsing now grabs the correct tags for the image! previously, it was fetching the user's most common tags
• for now pixiv manga (which is not yet supported), will be skipped over in the gallery parsing step
• fixed a thumbnail drag-and-drop calculation that meant the DnD event would not start if the initial mouse movement went in an up-left direction
• bad animated gif frame timing detection is improved
• the fallback animated gif frame timing is scaled back to a more reasonable 12fps
• fixed a critical bug in the manage ratings dialog that meant any NULL or MIXED numerical rating controls were mistakenly creating 'set rating to 0.0' content updates
• fixed an obselete call with the new Pillow library that broke video thumbnail generation
• added an explicit file import test to my test suite (previously, this was a manual test, and I did not notice the failed video files last week)
• fixed a critical os x crash after navigating media with the manage tags dialog launched from the media viewer
• improved speed and memory use of num_tags queries a little bit
• reordered the predicate filters in the main query code so that num_tags happens right at the end, where it will run fastest
• removed an unnecessary filter from the main query code
• removed an accidental doubling of the simple predicate filters from the main query code where the file service domain is specified
• improved an important db index and made sure the db uses it everywhere it can. it should speed up most non-'all known tags' queries
• harmonised how all the new serialisable JSON objects are sent to and read from the db to a handful of simple general functions
• externally opening a file is offloaded to a temporary thread to avoid gui lockups on OS flavours that have delayed 'open this file pls' calls
• increased the commonly used read/write 64k block size to 256k, let's see if it reduces some file fragging
• clientside hydrus bandwidth reporting is a bit faster and simpler
• general code cleanup
• general refactoring
• a long-term refactoring job that aimed to better decouple the client, server, and hydrus parts is now completely done
• delineated the layout of the gallery and page of images management panels a little
• updated several links in the help and the program to https
• the petition content checkbox list is now significantly taller
• cleaned up the static folder a little bit
• silenced an unhelpful webm last frame error

• version 177

• the old YAML "default advanced tag options" is converted entirely to the new JSON "default import tag options"
• created a new JSON object to store new options objects
• updated the gui to handle new default import tag options objects
• existing default advanced tag options data will be converted to the new system on update
• the petition management panel now presents similar petitions from the same user together, with the relevant data rows in a checkbox list, so janitors can approve/deny many similar petitions all at once
• the petetion management page no longer auto-loads file results for a petition--you can now see specific results for each sub-petition by double-clicking on them
• the server-to-client petition object is converted from yaml to json
• the new petition object now stores all the different 'content' rows for the same service/account/reason/action
• small content rows are now stored and transmitted in their own object
• the network now supports POST JSON requests
• account modifications now work through JSON
• the account identifier object is converted from yaml to json
• in the path tagging dialog, regexes that do not compile will now say so, reporting their short regex error statement
• fixed a bug that could sometimes occur when cancelling an ongoing gallery query
• cancelling an ongoing gallery query is now reflected in the gui much faster
• repositories will not burn CPU cleaning their updates if there are still pending file/mapping petitions to process
• repositories will clean updates in smaller bursts to reduce the contiguous server_busy time
• fixed drag and drop thumbnail ordering, which actually wasn't working but was accidentally randomly sorted when I tested it!
• a certain kind of error during vacuum that can mean two different things is now caught, and automatic vacuums then suspended while the user figures out what is going on
• revised some v176 update code that wasn't parsing some downloader pages correctly--now, if a downloader page cannot be updated, it will be discarded
• updated to Pillow 3.0.0 on all platforms and fixed some obselete image calls
• general code cleanup
• general code refactoring

• version 176

• the gallery downloader is updated to the new system
• the gallery downloader code is generally a bit better all around
• the gallery downloader will now remember its urls and file and tag options and paused status and so on throughout a reboot
• the gallery downloader supports 'get tags even if file already in db' option
• the gallery downloader management panel is updated for the new object. like the other panel updates, it is a little more minimalist
• the gallery downloader management panel now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever
• existing gallery downloader pages in gui sessions will semi-convert on the db update
• subscriptions are updated to the new system
• the subscription code is generally a bit better all around
• subscriptions will no longer dump out on a failed file!
• subscriptions can now recognise when files are uploaded during a page-walk (which shuffles everything up by one, of course), and recover without assuming it has synced
• subscriptions can now be set an initial file limit and a periodic file limit separately
• the manage subscriptions dialog is updated for the new object
• the manage subscriptions dialog now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever
• existing subscriptions will convert on the db update
• subscription popup now shows per-file download progress
• boorus that use number of thumbs to advance their gallery page indices can now resume
• the thread watcher now obeys 'exclude deleted' status more reliably
• the 'page of images' downloader now obeys 'exclude deleted' status more reliably
• subscriptions now attempt to recover their sync status when a new file is uploaded during sync
• fixed a bug where the thread watcher was not applying the filename namespace to new files, only redundant ones
• the old advanced_import_options dictionaries are all updated to the new import_file_options object
• improved subscription-dialog-was-opened-while-subs-were-running detection and reaction
• the way galleries are identified behind the scenes is now much simpler and more flexible
• for now, hentai foundry downloads and subs will no longer support the specific hf filter--they will be unfiltered and sorted newest first, so they work like all the other galleries. I may reintroduce 'sort by popularity' if there is demand for it
• a bit of importer status update refactoring and cleanup
• fixed some db vacuum maintenance timing that was interfering with gui reporting, so the 'currently vacuuming' popup never appeared until the vacuum was complete
• fixed a possible bug for import folders that have a local tag set
• splash screen is no longer stay_on_top--if there is demand, I will add an option
• fixed a bug in the page deleted detection code
• added page hidden detection code, replaced some ambiguous pause code with it
• improved pause/play code throughout the program
• corrected some bad x/y status texts
• for simplicity, the import_file db command now always returns ( status, hash )
• for simplicity, the import_file db command no longer takes service_keys_to_tags
• plenty of general code cleanup

• version 175

• the 'regex shortcuts' button in the path tagging dialog now permits the managing of favourite regex phrases
• added 'start animations this % in' options to options->media
• animations will no longer clear to white while scanning beyond their immediate frame cache
• modifier-click (like shift-click) will now drag animations without altering their pause/play status
• the splash screen is a little larger, to better fit db update text in
• the splash screen's text is split into two lines, and shutdown repository processing is a bit more verbose
• fixed some incorrect dialog text in manage sibling and parent dialogs
• rewrote sibling and parent pair update code to deal with unusual situations more gracefully
• improved how the new downloader engine saves error notes
• the import review frame now lets you copy the selected entries' notes to the clipboard
• video files will no longer fail to import in windows because of unicode characters
• recycling files with unicode characters on Linux will work more reliably, and when it breaks, it will fail more gracefully
• the client and server can now detect and recover from a rare error where a database file is created but never initialised with the appropriate tables
• the tag/file upload popup message now dismisses itself after a few seconds, rather than pointlessly lingering
• wx event ids used in one-shot menus will be recycled, which should alleviate the 'after a long time, some menu selections stop doing anything' problem
• some code cleanup

• version 174

• 'server stop' and 'server restart' now work!
• server will now log everything to console as well as server.log
• server log entries will always be prefixed by the time they were made
• server shutdown will close services more neatly
• server shutdown cannot be triggered multiple times with multiple keyboard interrupts
• server initialisation and shutdown is more verbose
• server now accepts POST '/shutdown' requests from localhost
• fixed a server shutdown bug
• mousewheel events that hit the command and ratings hover windows will be propagated to the media viewer (e.g. if you click a rating, you can now scroll immediately to move to the next file without having to refocus the media viewer window)
• fixed a bug where zooming out would go immediately to 1% if the media's exact fit zoom was already in the list of default zooms
• added 'check db integrity' to database maintenance menu
• multiple drag-and-dropped thumbnails will now be listed inside the drag-and-drop object in their hydrus thumbnail order (was previously random)
• animations will now pause/play correctly during inbox filtering
• most client errors now cause a 1 second wait on behalf of the caller, which should bottleneck error spam when it happens
• fixed some buggy display and rescind behaviour with pre-dialog-existing petitions in the manage parents and siblings dialogs
• the manage parents and siblings dialogs will now not attempt to remove/petition/rescind a pair if the entered list of pairs has entirely new entries
• the manage parents and siblings dialogs will now attempt to maintain their sort order after you change some pairs
• generally cleaned some parent and sibling dialog code
• added option to options->files and trash for whether to use your OS's recycle bin
• system:mime text now specifies specific mime group if appropriate, or otherwise lists every mime if >1 selected ('system:mime is specified' is dropped)
• hydrus repositories will sometimes catch and recover from zero-length update files
• hydrus repositories will catch processing parsing errors and pause when they happen
• the new hdd, page of images, and thread watcher import daemons will tidy themselves up if their accompanying pages are completely deleted
• improved how daemons shut down
• improved client shutdown logic and speed
• some misc code cleaning
• more general code cleanup

• version 173

• converted import folders to the new system
• import folders now have names, so you can have several pointing to the same location
• import folders can filter by mime
• import folders now have separate actions for successful, redundant, deleted, and failed files
• these actions can be 'leave alone', 'delete' or 'move to location'
• import folders now support import file options
• import folders can now be individually paused
• import folders' popups are individually optional
• import folder edit dialog provides a path cache button, like new download pages, to let you review the paths it is remembering
• existing import folders will be roughly converted to the new system on update, but they will be in a paused state, so you can check they are what you want
• created new mime selection control
• mime selection control added to mime system predicate selection dialog, so you can now search by mime more specifically
• updated some mime system pred code to deal with this change
• a single non-dragging left-click on an animation now pause/plays!
• the animation scanbar will now only resume playing the video/flash after a scanning click/drag if it was playing before the drag began.
• added a 'polite wait' option to options->downloading. this is temporary until I write a better clientside bandwidth throttling system
• added a 'remove/petition all tags' button to the manage tags dialog to make mistakes easier to clean up
• added 'confirm send to trash' to options->gui
• added 'confirm archive/inbox' to options->gui
• added a simple 'export tags in .txts?' button to the export dialog. In future, I will expand this to do service-specific exports and also add it to regular export folders
• fixed some invalid name error handling in the manage dialogs
• fixed a bug that sometimes occured at the end of an admin immediate sync
• fixed a rare shutdown-order error in video rendering
• fixed an animation bar initialisation error that was causing error spam on some videos/flash
• general import code cleanup
• cleared up admin-side petition wording and presentation
• improved the threaded workflow and error reporting of the admin-side petition page
• the thread watcher now reports its number of next seconds until the next check in properly formatted time

• version 172

• the server no longer needs wx!
• the server now runs from command line
• the server is now .py, not .pyw!
• run 'server help' for more info
• improved how the controllers start
• improved how the controllers stop
• improved controller boot error handling
• improved how pages close
• general controller code cleanup
• general server code cleanup
• fixed the always_on_shutdown (without asking) option, which was asking anyway
• removed a debug statement in isalreadyrunning code, whoops!
• improved isalreadyrunning detection. it should work breddy gud now
• added gallery file limit to new 'downloading' options page, which folds in the old thread checker options as well
• added option to always embed the autocomplete dropdown window (rather than having it a floating window), which is now default on for Linux and OS X
• manage services now supports two kinds of service reset for repositories
• the service reset buttons now only fire on dialog ok
• service reset will try, as cpu allows, to update its progress in a message popup window
• administrators now have a 'sync now' button on the review services window that lets you catch up immediately to the service without having to wait for the normal update time (this will burn cpu time serverside, so be careful!)
• fixed a bug when searching boorus with unicode-16 characters
• the client updates directory is neater
• 'system busy' status is now shown on the status bar
• 'force unbusy' added to debug menu
• invalid characters in export filenames will now be replaced by underscores
• fixed a bug where rating services' cached file counts were not decrementing on de-ratings
• rating services' cached file counts are reset on update
• hover windows will pop in over video again, but will not if the mouse is _near_ the animation bar
• searching for ':' in the autocomplete dropdown will no longer search the db for every single tag jej
• improved some server_busy logic
• improved some server shutdown logic

• version 171

• improved isalreadyrunning code
• fixed isalreadyrunning code for the Linux frozen build
• the client's isalreadyrunning splash check will respond more quickly if you choose to wait
• repository processing sync will no longer update the frontend, which approx doubles its speed
• repository processing sync now only occurs when the client is idle, so it won't slow your browsing down
• if you catch the client while it is processing, doing pretty much any action will make it quit out nice and quick
• repository processing sync now occurs at a lower db level, meaning less laggy popup ui update and overall faster processing time
• repository popup disappears after a few seconds once it is done--if you leave your client on all the time, you probably won't see it again.
• split 'maintenance and memory' options panel into 'maintenance and processing' and 'speed and memory'
• added 'cpu busy' option to options->maintenance and processing
• added 'run stuff on shutdown' option to options->maintenance and processing to set whether pending db maintenance and repo processing can happen on shutdown
• added 'max minutes on shutdown' option to options->maintenance and processing to limit how much shutdown processing can be done in one sitting
• repositories can now sync on shutdown--they will report through the splash window
• shutdown maintenance is checked and run in a better way
• moved to a two-stage shutdown procedure
• daemons are better kept track of, and they talk to the controller in a better way
• daemons shut down in a better way
• improved a bunch of idle/busy/just woke from sleep logic
• improved how the controller maintains the db and memory
• improved force_idle mode in the client
• idle calculation is more accurate, and resets on more user-driven events
• improved responsivity for processing popup
• fixed a serious bug that was counting some unprocessed content as done when the service processing window was cancelled
• misc code improvements
• misc maintenance code improvements
• fixed a bug with parentless center-positioned dialogs not sizing themselves properly
• removed delete_orphans timespan param option
• fixed bugs selecting and media-viewering collected media
• fixed a missing thumbnail selection bug
• fixed some os x dialog interaction and z-order problems when the splash screen is open
• os x splash screen is no longer stay_on_top
• fixed the 110.00% display bug
• removed wx from pubsub code
• pubsubs now route through the controller
• reduced some pubsub overhead, we'll see if it smooths out some of the gritty stuff
• misc thread worker pool is now managed by and routed through controller
• some other thread-wx interaction is now routed through clientcontroller
• db now interacts with controller directly, not through wx
• all other indirect references to controller go through hydrusglobals, not wx
• refactored everything wx away from everything hydrus/server except controller code
• about window now says the client's network version

• version 170

• added media viewer mime options to file->options->media page
• you can now set animated mimes to start in a paused state
• you can now set the client to 'open externally' any media from a thumbnail activation
• reworked the way the media viewer generalises and displays its media to obey these new options
• improved some misc zoom code
• flv video is now rendered natively by default
• audio now defaults to an 'open externally' button
• renamed the nebulous 'url' download page option to 'page of images'
• moved the 'page of images' download page to the new system--it'll now remember its state through a restart, has a the detailed import status button, all that stuff
• 'page of images' download page now supports checkboxes to customise which types of links it will search for
• you can now close and rename page tabs from a right-click menu!
• deviant art parser now works for mature images
• improved deviant art tag and image page parsing
• added 'remove filtered files' option to file->options->files and trash
• improved UPnP dialog error reporting for when external IP cannot be parsed and for when external IP is reported as 0.0.0.0
• added an external ip/host override option to file->options->connection page
• whenever the client or server deletes a regular file or directory, it will now send it to the recycle bin
• files stored in the db are no longer read-only
• existing files stored in the db will attempt to be set read&write on update
• wrapped undo manager access in a lock, which may stop index bugs when a lot of stuff is going on
• fixed a bug in download progress->progress gauge hook when content length was absent
• removed previously entered zero-length namespaced tags (like 'wallpaper:') from client and server
• moved 'client/server already running' check from db access test to more reliable process list review
• improved 'client is already running' mini-dialog
• added cpu percent utilisation check (if any cores >50%, dump out) to idle check
• if the 'canvas zoom' value exactly matches a default zoom, it will not be inserted into the list of zooms to scroll through
• the 'canvas zoom' will be displayed with hundreds of a percent accuracy
• the new thread and hdd import pages will now obey page-global pause/resume events
• the new thread and hdd import pages now have the 'sort by' dropdown
• the new thread import page supports set_search_focus event
• added import tag options info to 'getting started - tags' help page
• relabelled the path tagging button on the import files dialog to something that made more sense
• might have patched an unusual subscription error reporting bug
• updated sqlite on windows--db should be a bit faster
• updated pillow on all platforms
• compiled a new python for my os x build with newer openssl library, which should fix some ssl problems people were having
• updated some deprecated directory parsing code
• general code cleanup

• version 169

• revised all the help files
• updated out of date screenshots in the help files
• added some more screenshots to the help's index
• cleaned out unused help images
• fixed some missing service error catching in the custom filter setup dialog
• fixed random reordering of paths from import files selection dialog
• logged errors should now be prefixed by the time they occured
• errors should now be more reliably printed to the log
• fixed a problem with e621 queries that included '/'
• improved some server temp file error handling
• fixed a typo that was throwing errors when trash settings were set to 'no limit'

• version 168

• thread watcher is moved to the new system--it will remember its previous state
• thread watcher has more compact, flexible gui layout
• thread watcher now supports file import status button, so failures can be reattempted
• thread watcher check now button has improved logic
• import tag options and import file options gui controls now plug into the new system
• thread watcher will now remember its tag and file options
• thread watcher will accept changes to its tag and file options after it is started
• fixed a bug in import tag options deserialisation
• hdd and thread watcher static texts will flicker less
• improved some network transfer gui reporting code
• send to trash, delete from trash, and undelete now all have different icons in the command hover window
• added open externally button to command hover window
• opening an animated gif or movie externally from the media viewer will pause it in the client
• share->copy->image added to media viewer for static images
• fixed a typo bug in copy 'image' to clipboard
• fixed a typo in external ip discovery that was causing errors for local booru external link generation
• improved external ip discovery error recovery
• in prep for the de-wxing of the server, refactored a lot of code so the server and client code don't import each other at any point
• if an attempt to close the client is aborted (e.g. because of an active import page), the exit splash screen will now destroy itself

• version 167

• created gui control to show file import status
• added a button to launch this control, wrapped in its own frame, to the hdd import page
• you can now drag and drop thumbnails out of the client for quick export!
• the client will catch internal drag and drops, and won't interpret that as an import request
• dragging files onto the client will now always show the 'copy' mouse icon
• some thumbnail generation code is a bit quicker--resized thumbs are now always generated on their first viewing
• if you use 200x200 thumbnails, the client won't bother to generate resized thumbs, saving space and time
• the text entry dialog will now check its button status on a non-keyboard related text-change (e.g. selecting paste from the right-click menu)
• the popup created by downloading a raw url or a youtube video will finish itself properly when the import attempt is completed, removing the pause and cancel buttons and making the popup dismissable
• the splash screen will now stay on top
• the password entry dialog will also stay on top
• you can now copy the md5, sha1, or sha512 hashes of local files from the normal share->copy->hash menu
• fixed an overly broad deleted/redundant test on md5/url pre-import status checks
• options to remember size of manage tags and position of both manage tags and ratings dialogs added to file->options->gui
• updated some of the help

• version 166

• created new object to hold hdd import information
• created new object to hold generalised import file status
• moved hdd management controller to the new system
• moved hdd management gui to the new system
• hdd imports will now remember their import and pause status through a session change
• misc import code improvements
• hydrus client is getting out of the zip business--zips are no longer parsed for import nor able to be created for export
• import and export code is simpler
• repository file downloading daemon will now throw up a small auto-dismissing popup when it downloads files
• repository file downloading daemon will respond to new downloads much quicker than previously
• if a repsitory is in the process of backing up when it receives a request, it will now return a 503 'server temporarily busy' error (rather than timing out)
• the client can now catch and handle these 503 errors gracefully
• backing up will no longer block the client's gui, and it will no longer timeout if the operation takes more than ten minutes
• some networking code is a little simpler
• added 'select local files' and 'select trash' to thumbnail right click menu
• 'cancel download' added to file repository submenu of thumbnail right click menu
• some buggy listbox resize behaviour (scrollbars not disappearing and occasional layout and drawing update fail) has been fixed

• version 165

• added a db table to track when a file was sent to trash
• added an option for whether trashed files are removed from view
• added a max age to trashed files
• added a max size to the trash
• added a daemon to maintain the trash
• improved some generic deleted shortcut logic, so delete key will present option to delete from trash if appropriate
• shift+delete will now undelete
• misc trash code improvements
• thumbnails deleted from the trash will accurately remove their inbox status icons
• images downloaded from file repositories will accurately add inbox status icons
• reduced chance of problems when trying to delete media from the trash when it is currently being rendered
• further reduced this chance
• removed redundant undelete code, folded it into add/delete code
• the media viewer will now remove files deleted from the trash in all cases, even when launched from a file repository search
• significantly improved how animations figure out when to put frames on screen. these timings should be much more accurate, slowing down only if your CPU can't keep up
• 8chan thread watcher now parses all files from multiple file posts
• if a booru provides a link starting with 'Download PNG', which refers to the original source png, that will be preferred over the jpg (konachan and yande.re, which run Moebooru, do this)
• booru parser is now a little more liberal in parsing for the image link
• yande.re booru support is added by default
• fixed some local/remote state code, which was breaking file repository searches in several ways
• improved error handling in repository file download daemon
• cleaned up manage options dialog code
• reduced min size of the media caches in manage options dialog
• moved thumbnail size to 'maintenance and memory'
• added better error handling to repositories that cannot find an update file during processing
• repositories that have such errors will pause to give the user a chance to figure out a solution
• misc code improvements
• fixed a bug where file repository downloads were not importing if the file had once been deleted
• dropped the clever indices from last week--sqlite had too much trouble with them. I will reform my db tables next week to get around the whole issue. for now, I have reintroduced the bulky index and forced sqlite to use it where appropriate
• added a test for collecting pending tag data
• tags in the form 'text:', which would sometimes slip through when typing quickly, are now not processed
• improved tag cleaning error reporting
• improved when special wildcard and namespace predicates are provided
• namespace predicates now display as 'namespace:*anything*'
• fixed a bug when launching the media viewer from collected results
• fixed a command hover window layout bug that was putting namespace text in incorrect places
• fixed a bug that was causing the new client-wide processing phase option not to work because, ironically, its calculation was out of phase
• review services will now state how many updates have been processed
• review services will specify if an update is imminently due, rather than saying the repo is fully synched
• fixed a review services layout bug that was misaligning text after an account or data usage update
• fixed a bug in system:similar_to serialisation, so pages with this predicate can now be saved to a session
• fixed the same bug for system:hash
• vacuum should be a bit politer about waiting for its popup message to appear
• database maintenance won't run while a repository is synchronising

• version 164

• rewrote the drawing code for the listbox that displays tags in various ways to be a lot faster and more memory efficient
• updated one new client mapping index that wasn't working quite as I wanted it to something more clever
• db will be a little smaller and mappings stuff will be even faster
• merged the two ratings system predicate input panels, so you can now select like/dislike and numerical ratings system predicates at the same time
• fixed booru download page serialisation, which means they will save to sessions
• prototyped trash service
• locally deleted files will now be sent to trash
• locally deleted files will not be removed from the existing search
• files can be permanently deleted from trash, which will also immediately physically delete them from your hdd
• files can be restored from trash back to the local file service
• inbox state is now more separate from the local file service, so it will be remembered through a visit to the trash
• improved delete code all around
• general inbox/archive db code improvements
• misc content update pipeline improvements
• optimised mass-adding of files to a service (for instance, when (un)deleting a whole bunch of files!)
• delete orphans daemon is removed--it will be replaced by a more thorough single-shot hdd/db purge like 'check file integrity'
• files are not yet automatically removed from the trash--I will add that next week.
• updated db access info in db folder
• added sqlite command line executable to db folder for all platforms
• bit of code cleaning
• cleaned up some gui error reporting
• might have fixed a service cache bug in the db that was causing double bandwidth reports and possible looping sync behaviour

• version 163

• reconfigured some important mapping indices in the client db to reduce search time for many common tag operations
• the new indices have also sped up tag processing significantly
• added an automatic db optimiser analyzer run whenever the db updates
• pixiv now downloads the largest version of an image again
• pixiv tag parsing improved
• added support for some unusual mp4 types
• the noquery media panel is removed. pages with no query status will now show a normal thumbnail panel with 0 files, rather than 'None' files.
• check file integrity will now report number of missing files and incorrect files separately
• check file integrity can now take a folder to move incorrect files, rather than deleting them
• when you try to pend a tag with a sibling in the manage tags dialog, you will be prompted with the chance to pend the sibling instead
• the rating system predicate dialog now allows for easier (and multiple!) selection of like rating parameters
• the rating system predicate dialog now allows for easier (and multiple!) selection of numerical rating parameters
• the rating system predicate dialog now offers correct predicates for 'zero allowed' numerical rating services
• some bmp icon code cleanup
• changed pause and cancel buttons to bitmap buttons
• cleaned up a lot of pause and cancel code
• now, when popups are paused, they will hide their details
• popups that are cancelled by a dialog change (for instance when subscriptions are changed while subs are processing) will now dismiss themselves after a few seconds
• during repository sync, http failures are recovered from more gracefully, and any pending processing will continue
• the repository update downloading and processing loops will wait on the db and gui more efficiently
• fixed a bug in the way the manage export folders dialog was testing existence of subdirectories
• general code cleanup
• bit more code cleanup
• some good daemon refactoring

• version 162

• on update, all previous gui sessions are deleted!
• on update, all export folders are deleted!
• made an important efficiency improvement to the new serialisation protocol--network version is incremented as a result
• finished extracting management data from management panel
• management data is wrapped in management controller, a new JSONable class
• pages collapsed to a single class
• page and management panel instantiation completely rewritten for the new system
• created new JSONable guisession class for better session management
• db is updated to store all this stuff in JSON rather than YAML
• session loading is much less bloaty for clients with many sessions
• all page types are now serialisable and hence addable to a session
• moved predicates to JSON
• moved filesearchcontext to JSON
• autocomplete dropdowns now remember their tag service, include current, include pending and synchronised status over sessions
• export folders now remember their file and tag services, and also include current and pending status
• some general improvements to export folder code
• cleanup of export folder dialog code
• added a test for export folders
• some general refactoring
• some general cleaning
• significant refactoring of predicate variables
• fixed a bug in multi-version update from before v154
• if you do not have any ratings services, the 'manage->tags/ratings' submenu will now just be 'manage tags'
• changed some thumbnail menu entries to say 'selected files' instead of 'all files', which is more accurate, see if you like it

• version 161

• updated windows python, sqlite and ffmpeg
• added linux ffmpeg binary to the executable build
• updated os x ffmpeg binary
• animation is generally working for linux and os x
• fixed some linux/os x ffmpeg calls
• fixed webm import for linux/os x
• webms work great
• moved to rendering gifs with PIL by default on linux and os x, which has bad palette support but renders more reliably
• the animation scanbar position caret is now visible on linux and os x
• fixed some timing/resume issues the animation scanbar on os x
• the old mediactrl video embed is removed
• mp4, wmv, mkv are added to native rendering control, no audio yet (just like webm)
• flv and flash will now have a little vertical padding on max zoom to make vertical hover windows easier to access
• changed database to a faster synchronisation mode
• tuned update processing daemon for more accurate time calculations and faster baseline speed, even if that knocks gui latency a bit
• fixed rows/s being thrown wildly off by long pauses
• made update processing daemon more reactive
• improved some of the flow and generally cleaned the update processing daemon
• added processing phase option to regular delay to processing of updates (this is useful if you run multiple clients and don't want them to process at the same time)
• fixed e621 tag parsing
• completely rewrote the focus engine behind the autocomplete dropdown
• fixed a bunch of autocomplete dropdown's generally buggy behaviour for linux and os x
• fixed autocomplete dropdown's display, sizing, and positioning for linux and os x
• fixed autocomplete dropdown selecting system predicates with the keyboard
• fixed autocomplete dropdown show status in linux with multiple pages open
• improved all subprocess calls, removing interim shell step and parameterising passed arguments
• improved how files and directories are launched in windows
• upnp is fixed for linux
• fixed hover window archive/inbox/delete icon buttons, which were spamming their commands to all open media viewers, not just their own
• added a simple raised border to hover windows to better delineate them from the canvas background
• updated the hover window size and position code to be a bit more reliable (still seems to bug out a bit on linux)
• tags hover window should now expand sideways more reliably
• it is now not possible to create nested export folders
• searching for numtags < x will now include files with no tags
• fixed manage boorus dialog OK in linux
• fixed booru selection mini-dialog in linux
• generally improved restoring database code
• fixed restoring a database when you have tag archives in existing database
• fixed a unicode encoding error when converting certain jobs to text (this was throwing errors in deviant art downloads/subscriptions--we'll see what was actually going on now, whether this is an error not being formatted right or something else)
• started some great rewrite of management panel gui code
• rearranged the download panel gui hierarchy slightly
• improved collapsible panel collapse/expand layout code
• removed code and database table for the old numerical ratings filter
• some general code refactoring and cleanup
• improved client upnp daemon timing
• harmonised how the client and server check requests for bandwidth-tracking eligibility

• version 160

• added options for http, socks4, socks5 proxying
• improved some network-related errors
• ratings services can have custom border and fill colours for their various states
• ratings services can now also be squares or stars instead of circles
• numerical ratings services can disallow zero ratings
• fixed the JSON parsing error that broke the thread checker
• shrank the width of the thread checker to make it a little less ugly
• the autocomplete tag search dropdowns will now accept and search with quickly entered text
• this new system will substitute siblings in the manage tags dialog
• fixed rows/s average calculation
• rows/s is more accurate
• content update popup string update is less laggy
• content update popup now shows content rows, rather than content parts
• removed update 'taking a break' component, as it was not doing the job I wanted it to
• db debug profile mode can now be turned off lol
• fixed an error from middle-clicking greyspace in the linux notebook tab area
• general code cleanup
• some string conversion code cleanup
• fixed a missing canvas bmp error with flash/flv embed buttons

• version 159

• split previously monolithic repository updates into smaller pieces
• added service_update server calls
• extended content_update server calls to support sub-indices
• sped up some content update preprocessing code
• improved some content update preprocessing code
• radically reduced serverside memory usage while generating updates
• added iterator splitters to make sure any single update row cannot be too large
• thanks to iterator splitter, updates should process through the client more smoothly
• added timespan splitter to make sure any single server update query cannot be too large
• content updates are resumable if broken part way through downloading the list of them
• the update popup will state how fast it is currently working in rows per second committed
• cleanup of a lot of update related code
• more cleanup of update related code
• improved serialisable protocol so it'll work better over a network
• made serialisable protocol much simpler
• fixed numerical rating system predicate dialog slider range
• fixed numerical rating system predicate dialog OK for valued predicates
• fixed numerical rating custom filter action dialog
• improved some network yaml error handling
• replaced 'export tags' thumbnail menu entry with a tag archives system
• replaced 'import metadata' file menu entry with a tag archives system
• disabled thread dumper and manage imageboards and manage 4chan pass dialogs for now, because dumping code is out of date and completely screwed up
• client will now not start in idle mode
• help debug menu has new 'force idle mode' entry
• idle mode is displayed on status bar
• simplified client main gui status bar display and code
• the flash and flv files' embed button now has a little border to delineate it from the canvas background
• fixed some clientside bandwidth tracking code
• removed some old networking code
• made some custom objects draw themselves more efficiently
• for now, manage tag parents and siblings dialogs will not show deleted rows. I will eventually add a 'show deleted' checkbox, like the manage tags dialog has
• some static image rendering is slightly less laggy, particularly when browsing large images
• fixed initial height of manage tags dialog launched from preview window
• changed the 'search area' vs 'preview' sash gravity so the preview area won't expand or shrink on resize--see if you like it
• autocomplete dropdown should now hide itself when focus goes to other programs
• autocomplete dropdown should now hide itself when focus goes to other hydrus frames in os x
• fleshed out new URLCache object to handle better gallery download url management
• refactored some file status tracking stuff to a better system
• refactored gallery page fetching to a better system
• refactored gallery url handling to a better system
• some redundant import checking should be much faster
• improved error handling when booru image page parsing fails to find image url
• refactored how tags are fetched for DA, tumblr, and giphy as part of the above overhaul
• misc code cleanup

• version 158

• subscriptions now have a 'get tags even if file already in db' checkbox, defaulting to false, which will significantly speed up subs with redundant files
• subscriptions will now more accurately obey initial limit
• gallery downloaders will now more accurately obey file limit
• merged some hentai foundry code into the general downloader
• cleaned and refactored a lot of import page initialisation code
• cleaned and refactored of lot of import management panel code
• cleaned and refactored of lot of import code
• created numerical rating control
• added numerical rating control to rating dialog
• added numerical rating control to rating hover window
• updated like and numerical rating service info for future support of custom colours and shapes
• fixed listbooks page deletion display
• fixed manage boorus saving changes to existing panels
• fixed manage imageboards saving changes to existing panels
• fixed manage imageboards saving remove imageboard
• improved and refactored listbook code, further separating active and proto pages
• improved some listbook name conflict error handling
• misc cleaning
• export folders and the export dialog will also export file attributes like access and modification time
• export folders will now overwrite destination files if they are a different size to the exportee
• export folders can now be set to 'synchronise', and so will delete any other files in the directory. existing export folders will behave exactly as before
• improved misc export and import folder code
• moved folder type tooltip to an explicit bit of text on export and import folder edit dialogs
• thumbnails can now display multiple hydrus file repository icons (current, pending, and petitioned) if multiple states apply
• hydrus file repository icons are gone from media viewer--now there is a text list
• hydrus file repository text list is added to hover window as well
• exclude_deleted option will no longer hide files from file repository search results--it was more confusing than helpful
• improved some error-prone logic in how advanced_import_options were being stored
• ratings hover window icon background colour is fixed
• imported bmps should now be converted to pngs with 1:1 colour (before, they were being collapsed to a 256-colour palette, which was showing ugly dither on complicated images)
• fixed a sometimes segfault crash on search refresh on OS X
• the preview window now supports a simple right-click menu
• added 'open installation folder' to file menu
• improved some media viewer precaching code, which should speed up some scrolling, particularly on the first scroll
• improved some layout code
• improved some file permissions code
• cleaned some canvas code

• version 157

• fixed a bug in listbook page initialisation and retrieval that was affecting many dialogs on OK
• some general dialog OK code cleanup
• fixed a media-navigation bug in managetags dialog
• fixed a serious OK bug in imageboards dialog
• created a new 'periodic' object to manage future subscriptions timing improvements
• started subscription YAML->JSON conversion
• stopped compressing json in the client db so it is human readable and thus easily editable
• subscriptions are no longer renamable, as this code was super buggy and could delete other subs
• tidied up the database menu
• a bit of misc cleanup
• in many cases where it was once pseudorandom, services are now reported in alphabetical order
• prototyped 'like' ratings control
• added new like ratings control to the background bmp of the media viewer
• added new like ratings control to the manage ratings dialog
• added new like ratings control to a new hover window in the top-right
• added basic additional icon support to new hover window
• fixed some misc new alignment bugs related to new ratings stuff
• like ratings controls on the hover window have tooltips
• fixed up some icon/rating display logic in the background bmp of the media viewer
• updated ratings dialog error handling

• version 156

• improved my build workflow in several ways, which should result in fewer orphan files and other build weirdnesses
• some bad path usage in initialisation icon resizing has been moved to better temp paths
• hitting page up or down on a manage tags dialog launched from the thumbnail grid no longer clears the current media
• improved file permissions code across the program
• fixed import folders daemon's test code for non-windows
• fixed up some temporary file code that wasn't cleaning up those files when the application was about to close
• fixed a newline parsing problem in copy/paste tags in the manage tags dialog
• added tag cleaning to pasting in the manage tags dialog
• added newline removal into standard tag cleaning process
• fixed a server db bug that was stopping some accounts from being created
• fixed some network session exception creation and catching
• new popup messages should no longer steal focus in most circumstances
• client should recover from serious popup message manager errors better
• hover windows will now only pop up if their media viewer is the currently focused frame
• hover windows will not hide until the mouse moves off them when flash or webm are underneath them
• os x will no longer vanish media in the media viewer on an action like archive or inbox
• fixed juddery media mouse dragging in linux
• improved the way listbooks work to avoid a problem with clientdata in wx linux
• export folder is gone as the default export location--now it is 'hydrus_export' under the current user's home dir
• updated windows ffmpeg to latest version
• fixed an important opencv dll conflict that was causing some gifs to render wrong in windows
• shift focus media logic improved--shift initial thumbnail is now last image selected
• shift selection will no longer deselect anything

• version 155

• fixed a frame seek error when looping long and/or large gifs with unusual palettes
• improved recovery when fps reads as 0 on videos
• added error detection and graceful recovery and reporting for missing ActiveX flash control
• removed a lot of old imported messaging code that was slowing down boot
• removed some other old library imports that are no longer needed
• gracefully silenced 'application shutdown' daemon errors (they were previously spamming to console)
• fixed a hover window display check that was sometimes spamming linux with edge-case errors
• slideshows will now pause while the right-click menu is open, stopping bizarre full program crashes occuring on many menu items while slideshow was in progress
• fixed a thread_id-sqlite-cache_initialisation problem that was breaking tag archive sync
• harmonised copy/paste tag protocol in manage tags dialog with 'copy all tags' of the tags list
• removed some old messaging code in client db creation

• version 154a

• fixed a bug in v154 update code when there was more than one set of shortcuts to convert
• fixed a faulty default value for num_pixels system predicate that was stopping options from opening
• fixed an error when video fps is 0

• version 154

• managed to completely break my linux and os x dev environments trying to update python--their release will come soon, once I've cleaned them up
• removed some old cv code
• updated to opencv 2.4.11 on windows
• fixed a bug in trying to upload a small number of tag petitions
• hover command buttons are now shrunk to exact fit size
• added zoom buttons to command button hover window
• added navigation buttons to command button hover window
• added fullscreen switch button to command button hover window
• added archive/inbox button to command button hover window
• added delete button to command button hover window
• added generic close window button to command button hover window
• added tooltips to all the command buttons
• cleaned some misc canvas code
• hover windows now fit better, without overlap
• moved media info strings to top of media canvas, so they hide behind hover header
• hover windows will now not cover webms or gifs when the mouse is over the media container
• hover command buttons now work for inbox filter, including back/skip buttons
• inbox/archive button feels and works correctly for the inbox filter
• updated manage options dialog's 'default file system predicates' mess to use the new cleaner predicate panels
• all file system predicate defaults have been reset to default
• added system:num_pixels to file system predicates default panel
• fixed the system:num_pixels predicate being broken on options save
• removed the ratings system predicates from file system predicates default panel
• fixed misc system predicate bugs
• moved yaml->json conversion forward:
• settled on object code
• added built-in compression to serialisation
• added db tables and access code to support it
• fixed a little layout mess in the edit custom filter action dialog
• created a rich 'shortcuts' class for storing shortcut->action information that can be easily expanded to handle mouse events as well
• new shortcuts class works on new json storage rather than yaml
• moved custom filter shortcut action storage and general handling to new shortcuts object
• moved old favourite_shortcuts tests to new code
• cleaned a lot of the custom filter dialog code
• rewrote the custom filter setup dialog to work more like other dialogs--now all changes are saved on ok, and save/save as/delete is replaced with add/delete
• auto-creation of 'previous' shortcuts is removed, as all changes are saved to shortcut sets anyway
• removed the ratings filters--the numerical filter may make a return in another form, your thoughts would be appreciated
• fixed a cache counting bug when archiving redundant files during import
• sped up file deletion a little bit
• added an inbox cache to reduce laggy inbox checking, particularly search result fetching on dbs with large inboxes
• fixed subscriptions that have no initial file limit, please reset your sub url caches to fix in these cases
• did a lot of server/client database merging and refactoring
• improved some database error handling
• fixed an error in file repository superbans
• optimised some critical db code
• sped up tag censorship filtering
• added a db profiling mode, accessed from the help menu, that will dump copious db profiling info to the log
• misc code cleaning
• improved some popup mesage print sync timing
• rewrote some server auto-setup code to deal with slowly-starting server

• version 153

• cut out some out of date stuff from help
• put a nice big red warning at the top of the 'running your own server' page
• added system:num_pixels for megapixel searching
• system predicates height, width, ratio, and the new num_pixels are now collapsed to a super-predicate, system:dimensions
• system:rating will now only show if you have some ratings services
• the system:rating dialog will now hide like or numerical ratings if you have no services of that kind
• complete refactorisation of system predicate dialog
• some misc system predicate bugs and bad panel event precedence fixed
• shortened all predicate spinctrls
• improved how system predicates are stored and used for a search
• moved some view menu items to the download menu and the new search menu
• all of the download gallery pages are added to the download menu
• added some explanatory text to the pixiv dialog
• the tag uploading process will more carefully prepare its subupdates so that mass allocations of the same tag to many files will be split into smaller chunks, avoiding connection timeouts in these cases
• hover windows will no longer hover over flash files
• tag hover frame will resize a little based on the canvas width
• the twenty pixel mouse warping when the mouse starts dragging close to the edge of the media viewer will no longer occur on flash files
• the canvas buffer for flash files is expanded to five pixels either side
• the hover tags window will now update itself when the media's tags change
• new hover window for details and commands, up top of the media viewer
• moved the old popout windows' buttons to the new hover window
• added hover window commands for the normal browser
• cleaned up some general canvas code
• index string will now display with a slash in all cases (it was a backslash for windows, for accidental reasons)
• file limit for subscriptions and download pages now has minimum value 1
• searching for number of tags while tag censorship has some namespaces will no longer throw an error
• moved boot and exit code responsibility from the splash frame to the controller
• improved the boot and exit code generally, including dragging feedback lag and error handling
• improved some thread error handling
• fattening service info won't trigger on client shutdown quite so much, which should speed up shutdown a little

• version 152

• added prototype hover frame for tags in media view
• hover frame shouldn't show when a dialog is open
• manage tags dialog launched from media viewer now has a delete button
• subscriptions now have an 'initial limit' variable, defaulting to 500, that limits the total number of files the subscription daemon will look for on the initial sync
• added a similar file limit spinctrl to gallery download pages
• updated layout of import files dialog
• cleaned up some ugly global variable scope stuff
• fixed initialisation of advanced import options in file import dialog
• made a good start to better object serialisation
• subscription and repository sync daemon jobs that stop due to a dialog-driven change during processing will now cancel themselves after five seconds
• fixed namespace (e.g. 'series:') tag censorship
• fixed the fullscreen switch bug that was breaking an (initially fullscreen)->(non fullscreen) media viewer
• fixed some search logic (some system predicates were not firing when there were no regular tags present)
• removed some artificial delays on daemon db access, let's see if it chokes anything
• harmonised a bunch of client and server controller code
• created a common controller class and merged a lot of the client and server controller code into it
• general code cleanup
• more general code cleanup

• version 151

• added a possible solution to the manage tags dialog next/previous buttons crash
• completely overhauled temporary file management throughout client and server
• removed old temp folder
• copy files to clipboard will now copy the database's exact file paths (rather than copying to a temp folder beforehand), making it a lot quicker
• the import folders daemon will no longer attempt to import (and hence spam errors about) zero-length files
• added one-time tag archive sync button to the 'perform a service-wide operation' dialog in review services. this lets you add or delete the archive's tags by namespace
• revised the tag archive sync code a little so it syncs to non-local files as well, if the hashes can be lined up
• added hash cross-referencing to archive sync code before maina processing step, often saving a huge amount of time
• fixed a parsing error with advanced content updates at the gui-end
• added GetName and IterateHashes to hydrus tag archives
• added default 'similar to' max hamming distance to file system predicate defaults in client options
• fixed up the default text display of the file service system predicate
• added some text to the sort/collect options panel to better explain what the big listbox is for
• fixed a popup numerical typo when uploading files to a repository
• fixed an utterly bizarre bug caused by the import files dialog that was screwing up subsequent dialogs' layout and size, apparently by wx voodoo
• refactored all the functions and classes of hydrusconstants to better locations
• misc other refactorisation
• misc cleaning
• renamed hydrusdownloading to clientdownloading
• changed all default { } and [ ] parameters to None. I don't think it mattered, but just in case for the future
• moved some ugly global variable stuff to better places, hoping to do more in future
• cleaned up many unused imports

• version 150

• added an 'idle' updating processing mode that churns through tags four times faster but makes gui laggy. it will kick in whenever you haven't done anything in thirty minutes
• improved some gallery downloader status texts
• gallery downloader will no longer wait 5 secs before fetching first page of urls
• fixed a critical race condition that had a chance of causing complete and unrecoverable gui freeze in the gallery downloader code
• fixed rendering of some bizarro pngs that CV couldn't understand
• fixed tag parent management for admin users
• fixed up some tag parent dialog code
• tag sibling dialog now accepts single reasons for multiple pairs, like the tag parent dialog
• improved error state on rendering video that reports resolution of 0x0
• removed looping error reporting on certain static image rendering errors
• fixed an error when some thumbs were 'collected' while fading in
• tagboxes now select on home/end and support numpad versions of page up/down
• fixed a radio button initialisation bug
• boosted the initial size of the 'set default advanced tag options' dialog so the collapsible pane is more visibile
• silenced a upnp error when external ip can't be parsed
• reduced some tag dialog previous/next button lag
• general code cleanup and refactorisation
• gui flag refactorisation and unification
• refactorisation of most non-constant stuff from constants files
• refactorisations of media mixins class in prep for a general rewrite of that code
• improved client initialisation of default options, boorus, and imageboards

Changelog 100-149

• version 149

• all listctrls with an attendant delete button will now support the same action with the delete key
• several normal tag boxes will support the delete key as well
• added a temporary service cache to the client db to speed up service fetching until I can figure out a yaml replacement
• cleared out inefficient tag and hash database fetching and generation code
• tag parent dialog will now ask for an action/reason for every ( children, parent ) pair rather than every ( child, parent ) pair
• fixed an error with collecting by rating
• fixed the 'fix' to the gui_colours db update problem
• the autocomplete dropdowns in the paths-to-tags dialog will no longer produce errors when tags are submitted
• improved accuracy of taglist scroll-to-selected scrolldown calculation
• collapsed some of the helter-skelter db code in prep for future improvements
• moved daemon code to separate files
• silenced server upnp errors
• all except: continue lines are banished!
• cleaned up some miscellaneous exception code
• cleaned some ugly behind the scenes a/c dropdown button code
• did some general code cleanup

• version 148

• rewrote thumbnail canvas and scrolling code from a crashtastic monolithic bmp to a lighter, faster, and more flexible page buffer
• all custom gui elements should be less flickery
• the manage tags dialog will now grow significantly taller if its parent window is also tall
• dialogs launched by the media viewer will initially position themselves according to that, rather than the main gui
• dialogs launched by controls will initially position themselves according the control's toplevelwindow, as originally intended
• thumbnail download (for file repositories) no longer happens silently in the background; it will now occur in the repository sync daemon, reporting its progress in the normal repo sync popup
• missing thumbnails are now replaced with the hydrus psi symbol silently, with a simple statement written to the log
• fixed a tiny typo in update error code that was reporting version wrong
• fixed a typo bug in gettagarchivetags
• fixed a typo that had broken namespace sorting
• numerous other single-line miscellaneous bug fixes
• fixed a bug with displaying media with size (0,0)
• fixed a bug with zooming in flash files
• improved some buggy tag selection logic that was sometimes desyncing indices between menu popup and selection
• tags will now stay selected even through changes to the tags list
• any attempt to close the autocomplete dropdown floating frame should now bump the close event up to the whole program
• linux release now includes source code alongside executables

• version 147

• fixed a problem when trying to do a multi-release update that contained the v146 update
• fixed the 'canvas not resizing after media removal' bug
• when an autocomplete appears in a dialog, the dropdown window will integrate into the dialog (rather than being a popup), allowing mouse interaction
• refitted the paths-to-tags dialog so the different static boxes and newly embedded A/C dropdowns fit better
• fixes to and cleanup of sibling tag retrieval code
• fixes to and cleanup of sibling tag filter code
• fixes to and cleanup of sibling tag search code
• fixes to and cleanup of sibling tag display code
• fixed a couple bugs in READ autocomplete tag search caching that was stopping namespaced queries searching properly if typed in manually rather than pasted
• fixed a similar bug in WRITE autocomplete tag search
• fixed a bug that was ignoring namespace entirely in WRITE autocomplete
• synchronised subscription popups' new file buttons with the text

• version 146

• manage tags and ratings dialogs will now OK on F3/F4, not CANCEL
• zoom switch (default shortcut 'z') will now work for images smaller than the media viewer's canvas
• in the media viewer, the canvas-fitting zoom value is now inserted into a media's scrollable zooms
• added 'tags box background' gui custom colour
• volume, chapter, and page tags will sort properly again in thumbnail view, including last week's improvements
• the thread watcher will no longer break its checking loop on non-404 http request or parse errors (i.e. the manual 'check now' button will still work after a misc error)
• to remove clutter, the preview window will no longer show tags and file information like the normal media viewer
• subscriptions will now show a live 'show files' button as they process
• the copy and 'open new page' right-click menu options on tags will now work on more controls
• cleaned quite a bit of tags box code
• cleaned up and unified a lot of miscellaneous canvas zoom and display code
• fixed an error when trying to upload a file petition without any accompanying file uploads

• version 145

• added custom gui colours for thumbnail backgrounds and borders, the autocomplete background, and media background and text
• added <- and -> arrows to manage tags dialog launched from navigable media viewer
• on empty input in the manage tags dialog, page up and page down work as shortcuts for the new <- and -> buttons
• fixed the media height calculation for animations, so when they are vertically scaled, the total height including scanbar won't overflow off screen
• allowed non-integer page, chapter, volume tags in display and sort calculations
• semi-integer tags will sort along with integer tags and string tags like so: 0 < 0a < 0b < 1 < 2 < 22
• improved the old tag/media sorting code
• removed loli and shota from hentai foundry filter options
• patched old db-stored predicates to attempt to convert to the new format when queried for _inclusive
• this _should_ have fixed the recent export folder problems
• created an 8chan board, and updated my various links, including in the client, to migrate from my old forum to this
• added a 'fit to canvas' checkbox in file->options->media that will zoom small images to the full size of the media viewer
• misc code improvements

• version 144

• files named 'Thumbs.db' will now be skipped in the import files dialog
• fixed wildcard searches, which last week's predicate rewrite broke
• fixed a typo that was showing namespace predicates as exclusive (-series:*) when they were actually inclusive (series:*) and vice versa
• added wildcard namespace searching for database autocomplete queries
• fixed database wildcard autocomplete searching when wildcard match is not the first word in a tag
• fixed database file searching when wildcard match is not the first word in a tag
• added a comprehensive suite of predicate-unicode conversion tests
• cleaned and improved some of the downloader code
• added five second per-gallery-page delay to subscription daemon
• added three second per-file-delay for regular gallery downloads and subscriptions, just to be polite to those web services
• added three second per-file-delay to the thread watcher for the same reason
• added a 'check now' button to the thread watcher
• fixed a problem that was sometimes causing subscriptions, when paused, to continually restart
• removed unnamespaced tag parsing from deviant art
• fixed creator parsing for deviant art, which was formerly cutting off the first character
• patched an account sync problem in the manage tags dialog
• in add tags by path dialog, tags are now sorted before being added to the file list
• in add tags by path dialog, the regex sections now generate tags for every match in the string, not just the first
• stopped collapsible panels resizing dialogs to minsize on collapse or expand
• added shortcut for 'open externally', default Ctrl+E
• moved 8chan to new 8ch.net domain (old domain still works)

• version 143

• when making a READ autocomplete tag query, instances of tags that only have a count in a single namespaced domain will no longer accumulate helper results in the non-namespaced domain i.e. no more 'blah (1)' 'title:blah (1)' dupes
• improved the way the above results are calculated
• pixiv artist downloading now correctly asks for numerical artist id
• reworded some of getting started with files help page to better explain multiple selection
• widened the splash screen a bit so db update messages have more space
• all frames and dialogs now have an explicit minimum size that is usually far smaller than their initial size
• the dialogs with listctrls (import files, manage custom filter actions, and so on) now start a little shorter in height
• fixed some bad sizer flags in dialogs with listctrls
• rebalanced content update throttling for quicker correction under heavy load
• moved the new 5 second break in the gallery parser to a position for quicker status updates
• the gallery parser will report total urls found when it is finished (even if that is zero) and wait a bit to let you read that
• fixed services->news to hide if you aren't connected to any repositories
• autocomplete boxes are now a very slight shade of blue, see if you like it
• fixed 'launch file externally' for linux and os x
• fixed 'launch directory externally' for linux and os x
• fixed 'set up server for me' for linux and os x, I think
• improved some general external-process-launching code
• cleaned and improved some of the predicate code

• version 142

• added wildcards to autocomplete results and file queries
• autocomplete results will match your wildcard _exactly_
• a new predicate type will appear at the top of wildcard queries; selecting it will search files with that wildcard
• the new wildcard predicate can be prepended with a minus sign to exclude from results just like normal tag and namespace queries
• in wildcard predicates, namespace and/or tag can contain wildcard characters
• added some wildcard help
• putting in '***' as an autocomplete query is now a pretty bad idea!
• fixed some logic in how tags are matched against unusual search input in the db
• sped up and cleaned how tags are matched against search input
• fixed some namespace logic in how tags are matched against search input
• below-character-threshold autocomplete queries will now return all applicable namespace results (e.g. putting in '1' will return [ 1, page:1, chapter:1, volume:1 ], not just [ 1 ])
• added 'open externally' to launch the file in default external program to thumbnail and media viewer menu
• added a five second delay between gallery-page fetches in the downloader to reduce chance of 429 errors (this was affecting big sankaku searches)
• added danbooru webm downloading
• fixed a typo in the thread watcher
• fixed a bit-rot bug that was stopping the 'like' ratings filter from launching
• fixed right click menu in custom filter

• version 141

• combined mappings are no longer calculated and stored
• recalculate combined mappings obviously removed from database menu
• combined mappings no longer have to be recalculated on a service deletion; the accompanying 'this could take a long time' warning dialog is gone as well
• combined mappings autocomplete counts are now calculated on-the-fly
• combined mappings tag queries are now performed on-the-fly
• combined mappings namespace queries are now performed on-the-fly
• combined mappings zero/non-zero tag count queries are now performed on-the-fly
• combined mappings regular tag count queries are now performed on-the-fly
• corrected some logic with regular tag count queries, I think
• autocomplete tag cache dirties and culls itself more efficiently on tag update
• autocomplete tag cache dirties and culls itself more efficiently on file import/delete
• removed a couple of non-useful AMP tests that were breaking due to a previous change in connection code
• improved how popup messages give the gui permission to update; hopefully the gui will lock up less when big jobs with popups are happening
• improved some misc timing logic
• improved some repo sync timing logic
• added simple emergency throttling to the repo sync daemon when the CPU/HDD is getting hammered
• improved some repo sync text number-grammar and timing
• added sankaku channel booru, including flash
• the booru downloading code that discovers image urls is more flexible
• improved my job pause/cancel logic so paused jobs, when cancelled/completed, will report themselves as no longer paused (this was affecting the downloader page, with paused jobs not dismissing themselves on a subsequent cancel)

• version 140

• if a repository or subscription popup message has nothing to report, it will dismiss itself
• fixed handling of text popup display when the object passed was not text
• delete orphans is now cancellable
• vacuum, deleted orphans, and upload pending popup messages will dismiss themselves an hour after they are done
• tightened the subscription final state popup message to just a title and a button
• removed much of the very expensive autocomplete tag cache maintenance code, which seems not to be worth the effort
• culled the autocomplete tag cache in prep for new maintenance cycle
• fixed a resize timing bug that was causing large images to scale in an ugly way when the media viewer was launched in a borderless state
• 'open selection in new page' will no longer default focus on the sort dropdown; it'll go to the media panel (this was causing scrolling confusion)
• fixed a non-updating display bug when resizing frames/dialogs with auto-resizing listctrls on linux
• cleaned up a wall-of-text error when closing the client immediately after deleting a tag service
• filled a gap in static text image object cleanup
• cleaned up some thumbnail waterfall/fade code
• filled several gaps in thumbnail object cleanup

• version 139

• cleaned up all the old popup message code, and fully integrated the new
• moved repo sync to the new popup messaging system
• moved normal errors and db errors to the new popup messaging system
• improved some error handling code
• reintroduced message printing
• improved subscriptions messaging
• added cancel button to:
• check file integrity
• export to tag archive
• added pause and cancel buttons to:
• repository sync
• subscription sync
• pending upload
• regenerate thumbnails
• improved how jobs' pausability and cancelability are spawned
• improved and harmonised a lot of pause and cancel and general shutdown-job-interaction logic
• pausable and cancellable popups can only be dimissed with right click once they are done or cancelled
• improved some more pause logic in the subscription and repository sync daemons
• improved popup pause responsivity
• added comprehensive cancelled/finished log statements for the newly pausable and cancellable operations
• 'just woke from sleep' calculation will no longer evaluate to true on application start
• deleted the old broken message log page, which I had forgotten even existed!
• fixed a bug stopping adding tag repositories in the manage services dialog
• fixed a bug that was hiding the 'name and credentials' panel in the manage services dialog
• improved some kinds of thumbnail error reporting
• sped up client boot by one second

• version 138

• created new 'maintenance and memory' options page in the manage options dialog, and moved some things over from 'files and memory', which is now 'files and thumbnails'
• added options for idle, vacuum, and delete orphans periods to the 'maintenance and memory' page in the manage options dialog
• fixed the incredibly annoying animated scanbar delayed-frame bug, where a click on the scanbar would not draw the new frame until the old frame's expected delay was complete. scanbar scanning is a lot smoother all around, now
• animations now show their current frame number in the animation scanbar
• status bar now shows inbox/archived counts on any selection
• reworked static image zoom code so they resize beautifully, without jaggies, at the cost of a bit of CPU and memory
• created new flexible message pathway
• moved most messages to new pathway
• improved a bunch of message and job_key related code
• btw: messages will no longer close themselves; their end state is now to report what happened until you dismiss them. if this turns out to be annoying, I'll change it
• fixed the pending menu not updating its count when files imported and added tags via archive sync
• improved the logic behind the 'computer just woke from sleep' calculation
• improved the accuracy of the 'client is currently idle' calculation
• improved database vacuum so the .db-wal file is flushed afterwards, for both client and server
• because of better vacuuming, server db backup no longer needs to create a bloated .db-wal backup
• removed a very common superfluous empty tag upload packet in normal tag uploads

• version 137

• eliminated a loophole in the tag-pending process that allowed zero-length subtags through during archive sync
• improved the way tags are cleaned and checked, no matter their source
• added zero-length subtag (e.g. 'character:') checking and exclusion to the server
• removed erroneously pending or current tags that have a zero-length subtag
• added 'copy bitmap' option on still image right-click menu
• harmonised a whole bunch of clipboard code
• added a check file integrity db function, with quick filename existence and thorough content hashing modes
• added clear service info cache debug function to database menu
• added regenerate combined mappings debug function to database menu
• added a primary key to the hydrus tag archive db to save some space on duplicate mappings
• added AddMappings to the hydrus tag archive to make mass mapping-adding a bit more efficient
• added DeleteTags to the hydrus tag archive
• added Get/DeleteMappings convenience synonyms for Get/DeleteTags for the hydrus tag archive
• added an 'export to hydrus tag archive' button in the advanced content update dialog
• renamed the db tag archive sync methods to better reflect what they actually do
• massively simplified how cursors are referenced inside the client and server databases
• cleaned up some db cursor code
• fixed a critical error in the database test code that was spawning a double db mainloop
• improved error handling when regenerating thumbnails
• fixed error handling when waterfalling thumbnails
• fixed a typo in export tags process
• improved the way the client flushes its buffered log data to disk
• added log buffer flushing to the server
• added a confirm dialog to advanced content update dialog's powerful 'go' button
• added custom titles to many co-opted yes/no dialog boxes where the default 'Are you sure?' didn't make sense
• cleaned up some misc code
• cleaned up some misc bitmap language

• version 136

• added tag archives
• added tag archive sync initialisation on all existing local files
• added tag archive sync maintenance on newly imported files
• added a new db directory, client_archives
• added tag archive sync options to tag services in manage services dialog
• added local tags service to manage services dialog
• added sha512 to local hash cache
• added tag archive hash_type guessing
• added a new dialog for selecting n arbitrary strings
• got testing to work on all platforms
• fixed the hydrus server for linux and os x; it now has a stay-alive frame rather than a taskbaricon
• improved some dialog testing code so it would work on all platforms
• fixed a deffered problem that was causing the server AMP test to hang on Linux and OS X
• neatened and harmonised a bit of common file and network streaming code
• improved some misc manage services dialog code
• fixed a critical bug that was meaning certain service changes were not being saved to the database, so were being forgotten on restart
• fixed some select-string code that wasn't taking sets of strings for a weird wx reason

• version 135

• added a menu option to any tag's right-click menu to open a new search page for that tag
• added a subscription cache to the client database to speed up subs checking
• added a filter to the message popup system so those annoying and 99% pointless PyDeadObjectErrors will not display. they will still be written to the log
• cleaned up a couple of temporary file deletion errors
• improved some more temp file deletion error handling
• fixed a bug that I think was stopping file repositories from being deleted
• improved the manage services db edit log
• fixed a bad comparison that was causing superfluous edit actions after a manage services dialog ok
• fixed a new bug related to displaying non text in the popup system
• added a 'just woke from sleep' check to all daemons, so CPU heavy stuff like repository sync will not initialise if you just woke your computer. the grace period lasts about ten minutes
• retuned the way the subscrption daemon initialises (it'll now wait two minutes after startup before firing)
• fixed a typo that was causing fatten service info to fire more often than it should
• added a yes/no warning to options ok when the thumbnail dimensions have been changed
• added a popup message when thumbnail dimensions have been changed to report on deletion progress
• added account testing to my server db testing suite
• improved the security of the registration_key->access_key transaction; it'll now generate a new access_key with every call

• version 134

• updated to wx 3.0.1.1
• fixed a critical media scrolling bug due to the wx update
• improved some bad media scrolling code, sped things up a bit
• fixed 'top' and 'bottom' media scrolling events
• fixed a typo that meant the default fullscreen media browsing shortcuts were ctrl+ appended rather than working on their own
• overly-verbose errors and other text popups are now cropped to 1KB for gui display, with a notice. the full message will be printed to the log as usual
• improved how severe boot crashes are reported
• fixed a bit of text-reporting code that wasn't handling non-text very well
• improved handling of a weird popup message manager error
• fixed an occasional overhasty cleanup error in the checkimportfolders daemon
• added options for default values for the thread watcher's number of times to check and check period
• fixed the thread watcher complaining about closing when the checking was finished
• optimised some id generating code to stop spamming the id cache, which I think was overloading after a while and causing weird PyAssertionErrors
• neatened autocomplete dropdown service storage and menu id generation
• improved menu id generation for tagbox
• fixed opening new petition page from view menu
• added a 'this might take ages' warning yes/no dialog when trying to delete a tag service
• added a little popup message info to report on progress when deleting a tag service
• added some server db testing
• fixed an error when double-clicking a tag in a page without search predicates
• updated some help links from mediafire to the new github releases page
• fixed a typo bug in server db's account flushing code

• version 133

• reworked the add file process to correct file repository file counts
• made add file process to calculate inbox and local thumbnail count more efficiently
• fixed a critical bug in server db creation that was stopping file repository service from working
• added go/exit fullscreen options to fullscreen right click menu as alternative for default shortcut 'f'
• improved fullscreen position, size, maximised, and borderless state memory
• removed old 'fullscreen_borderless' option in local options
• thread checker now supports 8chan urls
• thread checker now has 'number of checks still to do' spin control so you have more options to fire and forget
• improved how the thread checker constructs and passes url information around
• improved some thread checker timing logic which _might_ have been causing problems
• improved some thread checker error reporting

• version 132

• merged two complicated serverside account tables into two simpler tables
• with this action, was able to clean out a lot of rubbish old server account code
• made it so accounts can only be on one service. existing (admin) accounts that straddled sevices will have new access keys printed in a text file in the base installation directory on server update
• rewrote the account object to be simpler and easier to maintain
• swapped the old rubbish 'account_id' identifier in the account object for the much better 'account_key' identifier
• harmonised some conflicting account-related variable names
• refined the way the unknown account is stored and identified
• split serverside account verification and identification into two separate paths, to reduce chance of security problems
• reworked account identifiers (a general purpose account identifying object that is used in admin-server interactions) and their associated db functions to be more secure and reliable
• simplified account data use checking
• simplified and harmonised the way used bytes and used requests are stored and retrieved in the account and account type
• with guarantee of account_key uniqueness across entire server, I have simplified session code in several places
• updated help to reflect the new relationship between access keys and account keys
• added 'copy account key' button to review services, which will now be the thing for users to use if they need an admin to modify their account
• fixed serverside credential verification for non-instantiated (still have a registration key) access keys
• added a bit of explaining text to the 'waiting' autocomplete state
• fixed a typo that caused errors when deleting files from a file repo

• version 131

• removed tag service precedence and its various expensive and overly complicated effects
• recalculated combined tag mappings using new simple union based merge
• cleaned updatemappings code, and I think fixed some tiny count logic bugs
• refactored eight overly complicated sub-functions of updatemappings down to two simpler ones
• fixed up a mention of tag service precedence in help
• deleted a bunch of old client update code
• added adaptive throttling to tag updates, so they'll speed up and slow down based on current CPU load
• refined a bunch of the parent and sibling code, given the new changes
• updated and cobbled back together a host of finicky parent and sibling and regular tag test code, given the new changes
• fixed a bit of logic in the censorship code that was causing overly broad namespaced matches
• fixed error when selecting a specific tag service in a normal search
• deleting a service no longer throws an extraneous error after it is complete
• fixed danbooru large pngs, which were being resized in a way that broke some of the downloader's url prediction code and caused 404s
• fixed a bug in the input custom filter action dialog
• fixed a misc clientserviceidentfier bug in custom filter initialisation
• fixed setting numerical ratings from a custom filter (I think) and I think fixed many other custom filter operations
• fixed several text-entry-dialog-launched-from-dialog bugs
• fixed, I really think for real this time, the serverside account desync bug. it was in session init, where I had not previously looked
• improved some popup messages' text
• improved the way some exceptions are reported
• cleaned some error reporting code
• silenced the more-annoying-than-useful account sync failure popups
• fixed and cleaned server start/stop/restart code a bit more
• reduced regular CPU hit from resize thumbnail daemon and made it more polite in starting

• version 130

• fixed youtube downloads
• fixed remote services not showing in OS X manage services dialog
• updated 'running from source' help page with some new info for Linux
• cleaned a little more code
• fixed bug that was stopping new/modified serverside services booting immediately
• improved the server-client service key translation code on serverside service modification
• fixed test credentials button in manage services dialog for unsaved services
• fixed bandwidth tracking errors for unsaved services
• deleted some old unused server options code
• improved server services start/stop code a bit more
• massively rebalanced the autocomplete maintenance routine after a tag service deletion or tag service precedence change
• made a mess of some of the deleted tag update processing code, I'll fix it next week

• version 129

• fixed a typo in launching the ratings dialog
• did some important code cleanup re a common ambigious variable name
• improved serverside account tracking in a way that will help future user management clientside too
• improved serverside session code a bit more
• improved general serverside login
• improved serverside access key verification
• aligned server db update code with the client's only-update-up-to-50-versions rule
• fixed some buggy repository test code
• fixed some buggy local booru test code

• version 128

• fixed right click on thumbnail error
• improved assumed permission defaults on tag dialogs and right click download/upload actions (i.e. tag dialog's entry A/C will no longer disappear when you desync from tag service)
• removed account desync on network version mismatch, to stop this problem anyway
• fixed tag parents not showing in autocomplete dropdowns
• added sqlite3 command line tool to db folder for convenience
• fixed a typo in manage server's services dialog
• some misc code cleanup
• added default advanced tag options pane to manage options dialog
• added default advanced tag options to manage subscriptions dialog
• fixed booru namespaces in manage subscriptions dialog to be booru-specific
• added default advanced tag options to normal download pages

• version 127

• finished service_identifier rewrite, phew
• added an exception catch for full program crashes; big errors will now be written to crash.log
• added a shutdown check to make sure popup message manager closes properly when certain weird errors occur
• 'files_service_identifiers_cdpp' is now renamed to the vastly less insane 'locations_manager'
• removed db update code older than 50 versions
• a new error message will display if trying to update more than 50 versions in one go
• many other small fixes I didn't keep track of

• version 126

• restored a dll that I thought was no longer needed, but was actually doing some weirder gif rendering
• added 'remove' to fullscreen menu
• harmonised thumbnail and fullscreen right click menus a bit more
• added pause button to popup messages for repo update and subscription processing
• moved service_identifier switchover forward
• moved all service fetching to streamlined and non-laggy manager
• changed client options to store default tag repository in a better way
• changed subscriptions to store their advanced tag options in a better way
• fixed a 'missing service' bug in advanced tag options
• remade idle calculation into a much better gui-based rather than db-based test
• fiddled more with maintenance timing, hopefully for the good
• new screenshots in the help index pages!
• improved how auto repo and server setup work and report their status
• the client's UPnP daemon will no longer spam errors if your IGD doesn't support UPnP
• fixed a bad db call in server's UPnP daemon

• version 125

• moved client splash screen and client boot to application event loop (i.e. your mouse won't hourglass over it now)
• splash screen is now draggable
• added splash screen for client shutdown
• cleaned up shutdown procedure
• added detailed A/C timing options
• created a new text entry dialog that won't allow ok on empty text
• hence fixed a startup bug when a session was created with no name (and likely others!)
• further tweaked idle calculation to improve maintenance routine
• revised a bunch of the maintenance timer logic
• fixed some bugs in the maintenance timer logic
• fixed a daemon error reporting typo
• fixed another way session data usage tracking could split, server-side
• brushed some more dust off the server-side session code
• improved some bad image processing nomenclature
• took first steps in service_identifier replacement rewrite

• version 124

• fixed some more broken gifs (those with funky transparency, I think)
• fixed a critical bug in the subscriptions dialog that was causing subscriptions to be deleted
• reintroduced old locale code, with minor improvement, to fix number grouping (123,456,789) issue
• added a short pre-activation wait to all daemons to reduce more wake-from-sleep problems like the UPnP issue last week

• version 123

• rewrote the old gif rendering code to the new system, so gifs with variable framerates now work again!
• fixed a bug that was stopping the first zoom of an image from showing if it took more than a frame to render
• webms are now zoomable beyond frame border
• webms will now respond to keypresses and mouseclicks when cursor is inside their border
• added service select to import metadata
• if repo sync is paused, either globally or individually, repo will no longer try to regularly sync their accounts
• fixed individual repository pause sync
• I think I fixed a serverside session synchronisation bug that was causing too-small data usage reports
• improved complicated tag action descriptions in manage tag dialog
• hastened initialisation of subscriptions daemon
• hastened initialisation of subscriptions dialog
• neatened all object declarations
• fixed a menu bug in OS X
• added put a little delay in the upnp daemon, hopefully to stop the error popups after waking from sleep
• updated an old hacky bit of locale code that was breaking certain platforms of wx 2.9.5.0
• improved animation and other rendering timings on non-windows platforms
• improved media focus on non-windows platforms

• version 122

• fixed server upnp daemon to use correct port, I think
• added client upnp daemon for local booru; it'll update upnp mappings as soon as they are changed
• added bandwidth tracking for local booru; it'll even magically update before your eyes in review services
• bandwidth limits are tested, so exceeding data usage will result in 403 errors until the next month
• moved local booru css to an external file any user can edit
• added 'open share in new page' button to local booru review services
• fixed local booru display for video, flash, audio, pdf and miscellaneous
• fixed local booru render resolution for all media
• fixed local booru thumbnails for video, flash, audio, pdf and miscellaneous
• added caching headers to static file requests to reduce browser-hydrus bandwidth
• fixed local booru page text's newline formatting
• added local booru unit tests

• version 121

• first version of local booru

• version 120

• improved quality of downsized animation rendering
• sped up downsized animation rendering in this case
• neatened animation code
• fixed a mid-animation resize parameter bug
• I think I fixed an animation scan bug that was sometimes giving the wrong frames
• added thumbnails for all video formats
• fixed an off-by-one framecount bug for certain videos, and retroactively fixed counts for existing videos
• fixed a couple harmless width/height numpy.shape switcharounds
• cleaned up some file parsing code
• added a pixiv unit test
• fixed a bug when servers were returning unusual relative redirect urls (gelbooru)
• semi-fixed a bug when servers were returning unescaped redirect urls (gelbooru)

• version 119

• fixed an overzealous error in v118 update code
• new direct ffmpeg video decoder for webm
• new system takes advantage of multi-core processors, increasing render speed
• improved animated frame timings
• fixed a number of gifs that were causing the "None has no 'shape' attribute" error
• sped up memory and subprocess cleanup after video playback
• added some new ffmpeg instructions to the running from source help page
• added client and server startup and shutdown statements to the log
• improved some idle time logic, which should improve maintenance timings
• fixed a critical idle timing bug that was adding 1s to many actions, including file queries!
• fixed a pubsub testing bug that was preventing daemons from quickly closing after a unit test

• version 118

• improved animated frame buffer size calculation, saving memory and improving smaller animation performance
• improved quality of static image thumbnail generation
• fixed webm mouse event propagation
• updated and improved perceptual hash code

• version 117

• fixed the gif rendering error that was causing threading lockup
• fixed a bug in the animated drawing code that was causing +5ms duration on every frame
• improved frame duration checking code
• lowered quality of animated frame resizing to improve effective framerate for previews
• fixed a problem with pixiv session management
• fixed a problem with pixiv tag subscriptions
• neatened an error with pixiv manga download failures
• corrected reviewservices hyperlinkctrl initialisation
• improved the wording for the 'port already bound' error message on client startup
• improved hydrus network session error handling
• cleaned up some old gui messaging code
• some general image/video code nomenclature cleanup
• changed http connection maintenance daemon to be snappier on client shutdown
• subscription minimum check period is now 1 time unit, not 0!

• version 116

• gifs now render their frames as needed, rather than building a giant memory-hogging cache of all of them
• webms now have thumbnails!
• refactored important hydrus threading code to a separate module
• added new 'call-to' worker threads for snappier job-thread behaviour
• fixed a typo that was stopping new repositories being entered in the manage repositories dialog

• version 115

• moved review and manage services windows to a local/remote dichotomy
• moved local booru service code generally forward
• added local booru service to db init/update
• added booru to review services
• added booru to manage services
• neatened some misc client code
• improved static/animated image rendering code in general
• added some cv stuff to gif rendering code
• cleaned up gif rendering code
• sped up animated gif rendering by about 5-10 times!
• added some cv stuff to static image rendering code
• cleaned up static image rendering code
• sped up static image rendering up to 4 times!
• image import should generally be faster
• pulled animated images off the image cache, reducing memory usage
• added some clever compression that cuts image cache memory usage in half!

• version 114

• gif rendering seems to be fixed in all cases! hooray!
• fixed 'pop from empty list' popup error spam in the new cache clearing system
• fixed weird behaviour on right-clicking 'dismiss all' popups button
• added a unit test for perceptual hash generation, with an eye to moving the delicate code from cv to cv2
• updated opencv to 2.4.9
• updated pyinstaller, so frozen releases may be a bit more stable!
• moved webm and mkv info parsing over to cv, which allows num_frames

• version 113

• added mkv+webm support!
• fixed some system:mime selection stuff
• fixed a missing db call in the server upnp daemon
• fixed upnp dialog's edit button
• cleaned up upnp dialog's post-button refreshing logic
• fixed getupnpmappings to deal with >10 mappings
• fixed fullscreen_switch for custom filter
• fixed a late canvas drawing exception when fullscreen window was closing due to being media-empty
• rolled back to wx 2.9.5 for windows again. not worth the trouble, right now
• hydrus caches should now clean up anything older than twenty minutes
• 'last session' will now be saved every five minutes, rather than only on a successful close

• version 112

• A/C dropdown is fixed on linux!
• wrote new UPnP library based on miniupnpc multiplat executables
• integrated new UPnP library into existing client code
• upnp management dialog should now work for os x and linux
• integrated new UPnP library into existing server code
• added UPnP unit tests
• fixed a bug when upnp executable path had whitespace
• renamed 'add, remove or edit services' to 'manage services'
• fixed some review services frame text display issues for restricted non-repository services
• optimised review services frame initialisation db requests
• added a database maintenance routine that should help review services launch a bit faster on average
• improved manage services layout a little bit
• removed initialise server button when it is invalid
• updated 4chan thread url parser to handle new 4chan thread urls
• redirected 4chan thread downloader to new api address
• fixed an error reporting problem in repository sync
• fixed my linux dev environment, so linux executable is back
• changed wx requirement to 2.9 on linux, due to bugs in linux 3.0.0.0 release
• fixed a typo in repository deletion
• added repository deletion to unit tests
• added num_base and num_step to file path regex dialog
• fixed popup display of certain exceptions, including file parsing exceptions
• cleaned up os x and linux build scripts, added static&help sync

• version 111

• fixed booru searches that include unusual characters like '&'
• added pause synchronisation to individual repositories in the manage services dialog
• fixed gui_sessions with predicates (again!)
• added a unit test for gui_session storage, shouldn't get problems with that again
• fixed a getcountlesscopy predicates bug
• first step in multi-platform upnp support is DONE, phew
• harmonised subscription file import errors with normal hdd import errors. they nshould be a bit clearer
• corrected how popup text width is enforced
• added a warning to the reset cache button in the manage subscriptions dialog
• os x autocomplete dropdown seems to be magically fixed, not sure how or when that happened
• started HydrusBooru
• fixed some problems with twisted startup/shutdown in unit testing

• version 110

• fixed a variable overwriting issue in the subscription daemon error reporting code that was causing error spam
• fixed more actual and potential instances of this error elsewhere
• fixed a bug in the import file error system for non-local files
• fixed url parsing for urls lacking 'http://' or 'https://'
• fixed hentai foundry image parsing for two different cases
• fixed client gui-initiated db backup to also backup client_updates folder
• A/C dropdown now shows current and pending counts separately!
• fixed display of uncaught database exceptions
• new version of sqlite for windows, might speed some things up a bit
• upgraded a bunch of other libraries, look forward to mysterious new bugs!
• I _think_ some transparency support is improved
• moved messaging forward
• dropped processed_mappings experiment
• removed a bloated mapping index, hopefully speeding a couple things up
• fixed an erroneous error message in hydrus network session manager

• version 109

• started processed_mappings table. for now, it mirrors normal mappings table
• improved manage tags dialog logic
• fixed only_add issue for parent tags
• fixed an important tags box surplus tags display bug when focus repository was changed
• fixed petitioned tag counts in tag boxes, as long as 'all known tags' is not selected
• fixed occasional zero tag count in tags box
• added petitioned tags to normal media fetch, not sure why they were missing
• fixed a random error when trying to refresh a 'open selection in new page' page
• improved help to reflect new manage tags dialog
• cleaned up an annoying spam-text-to-console issue when running test.py
• reworked the way the server clean/create update daemons interface with the database
• improved how updates are stored and fetched in the server db--no longer needs a db hit to fetch
• harmonised client db's update naming convention to the server's
• improved server db to update the same new-ish way client db does it

• version 108

• added 'database->delete orphan files' for manual firing of this maintenance routine
• improved redirect location parsing--should fix the booru 'Could not connect to None!' errors
• fixed 4chan thread downloader's erroneous 'still working' warning when trying to close while paused
• added multiple additions to manage tag parents dialog
• added multiple additions to manage tag siblings dialog
• generic improvements to both dialogs
• manage tags dialog now shows all the selection's tags combined, not just the tag intersection
• multiple improvements to manage tags dialog
• reworked some tagsbox classes to be more flexible
• improved dialogsetupexport's use of tagsbox

• version 107

• converted 'namespace blacklists' to more general 'tag censorship'
• tag censorship now accepts unnamespaced tag bans
• tag censorship now accepts all namespaced tags bans
• tag censorship now permits bans for all tag services
• fixed several small bugs carried over from the namespace blacklists code
• added a bit about tag censorship to help, under advanced.html
• fixed youtube downloading
• file import dialog now ignores dupe paths
• improved update status text in review services window
• fixed erroneous capital P on pending menu
• fixed shortcut input dialog's ok button
• fixed modify tags dialog's admin modify button
• added several bits of explanation text to advanced dialogs
• added more helpful timestamp to news dialog
• neatened export dialog's layout
• corrected some double-escaped backslashes in the export filename pattern button menu
• fixed modify accounts dialog's http request
• improved grammar in db pubsub code

• version 106

• download by raw url now sends the correct text to the popup on successful import
• first step in peer-to-peer repository sync is complete
• repository updates are now stored as local yaml, in the new client_updates folder
• repository reset keeps the downloaded updates, only does a reprocess
• all repository updates should process faster, especially if you have saved sessions
• a lot of misc gui lag should be reduced, especially if you have saved sessions
• the way the gui's menubar is updated is massively improved
• os x and linux menubar should be less spastic
• several improvements to daemon pubsub workflow

• version 105

• complete rewrite of message data handling and display
• hidden gauge messages will now display correctly
• fixed error-display bugs in the log page
• massively improved how database errors are generated and displayed
• improved and streamlined how other errors are generated and displayed
• improved the way all gui elements are deleted from memory
• fixed the frequent segfault in os x and linux when closing popup messages
• fixed the common segfault in os x and linux when refreshing a query
• fixed the occasional segfault in os x and linux when closing a page
• fixed pattern shortcut buttons
• the 'show deleted' checkbox on manage tags dialog makes a return
• neatened several checkboxes so clicking the label text clicks the checkbox

• version 104

• first version of export folders is ready
• cleaned up all ok and cancel dialog behaviour
• cleaned up some misc dialog button code
• fixed up some ugly db yaml pubsub stuff
• fixed a juddering 100% CPU loop in the subscriptions code
• subs and repo sync will now put up a popup gauge
• repo sync will now pause mid-update on repository pause
• services->pause->stuff will poke daemons to restart more reliably
• removed the service status area of the statusbar
• fixed 'subscriptions forgetting their tag options' bug
• added select invert, thanks to @fluffy_cub for contribution
• fixed a slight logic bug with display of select archive/inbox
• added better instructions on how to run in OS X from source, thanks to user steenuil for contribution

• version 103

• fixed a virtual size issue with expanding collapsible panes inside scrolling windows :S
• fixed images not showing on first zoom switch
• improved touch and feel of image/flash scanbar
• subscriptions dialog is now flattened
• reworked advanced_tag_options to be more efficient and flexible
• fixed a couple bugs in advanced_tag_options
• moved fake http unit tests to a fake version of the new networking system (so much better, now!)
• moved real http unit tests to new networking system (so much better, now!)
• fixed up a last couple network things
• I probably fixed some file upload code, as well
• cleared out old network code
• fixed a fake pubsub problem in unit tests
• made a single new db table for consolidating:
• 4chan_pass
• pixiv_account
• boorus
• favourite_custom_filter_actions
• gui_sessions
• imageboard_sites
• imageboards
• import_folders
• subscriptions
• couple small help edits
• fixed view->new booru download page
• started export folders

• version 102

• initial linux release
• rewrote running_from_source.html
• altered some service object and db stuff to look after null access keys and temp services a bit better
• key registration works on new network system
• all downloaders use new network system
• A/C timer tweaked a little more
• did some timer event grammar improvements
• improved button behaviour in import files dialog
• backing up a server no longer causes a timeout problem
• prototype export/import tag metadata is done
• rewrote the way animated gif frames are navigated and thrown to screen
• fixed an issue that was causing some dialogs to OK when escape was pressed or the close cross was clicked
• this above issue included the enter system predicate dialog

• version 101

• fixed another update problem for clients <v95
• uploading pending content should be less gui-blocking
• fixed numpad navigation of page chooser to work for qwerty rather than dvorak as default
• os x page picker now obeys arrow and numpad keys for navigation
• os x command key should be treated as ctrl now, for better cross-platform compatibility
• a couple more cmd/ctrl confusions cleared up
• os x menubar disappearing after fullscreen problem fixed
• added stateless http connection manager
• simplified how http works in hydrus
• http connections now used more efficiently
• better cookie parsing
• better http application shutdown handling
• some misc http bugs fixed
• new http error handling should limit occasional spam-errors
• moved all hydrus http connections to new manager
• moved many non-hydrus http connections to new manager

• version 100

• MVC import_controller system created
• download gauges lag reduced
• download cancel button desync bug fixed
• download error code harmonised
• download async code pushed to threads
• download job key checking and management improved
• download queue feed timer will obey page-hidden pause status
• download success and exception handling harmonised
• download on-close thread-signalling much improved
• download pages now take responsibility for download code factories
• download management panel code harmonised
• download management panels made much more MVC
• download error reporting is much improved
• download fail management is much improved
• download button response is much faster
• download progress display is split into the easier to read file/queue/builder trichotomy
• download display indices are less schizophrenic
• lots of misc improvements to download code and nomenclature
• thumbnail page's refresh and select menu items now only appear when appropriate
• thumbnail page's select menu will only show inbox/archive sub-items when appropriate
• thumbnail page's menu is less buggy generally
• db updates are now parcelised into each version, rather than one big job
• improved db version update notification
• a problem with updates from ~v70 to >v95 is fixed
• fixed a bug in popup message dismissal
• database exception formatting improved
• database exception display spam reduced
• database exception redundant traceback removed
• autocomplete character-search-delay time extended from 150ms to 250ms
• async wx window destroy code improved in certain places
• improved non-wx-thread popup error reporting
• some other bugfix, grammar and nomenclature stuff I can't remember

Changelog 50-99

• version 99

• added backup database menu option
• added restore database menu option
• gone back to wx 2.9.5 for now
• fullscreen scrolling should be a bit smoother now
• fixed sessions for 'open selection in new window'
• new regex shortcut for filename in the 'add tags before importing' dialog
• fixed dialogs cancel-closing on an unfocused 'enter'
• finally fixed the animation scanbar vertical sizing on preview screen
• animation scanbar now shows gif frame rendering progress more smoothly
• test.exe now included in windows release
• some frame parents code cleanup
• a tiny bit of gif code cleanup
• deleted some old server code
• cleaned up import pause/cancel code
• improved hydrus's thread-communication objects
• cleaned up downloader code and logic and gui display

• version 98

• update to wxpython 3.0
• you can now add tags when importing from a zip
• you can now download arbitrary urls from download->a raw url
• 'select files to import' dialog has massively improved file parsing, with:
• new gauge location
• queueable parse jobs
• paths that stream as they are parsed
• pause/cancel buttons
• new 'check credentials' button in manage services dialog
• new access_key_verification server request
• fixed a very important bug that affected normal tags when adding parents or siblings
• fixed a bug where old pending parents would not be remembered when new same-child parents were later pended
• fixed a similar bug where rescinding parent pends would screw things up
• pending counts will new correctly update on a sibling or parent pend rescind
• parent tags no longer (incorrectly) remove when:
• removing 'just this file' child tags in 'add tags before importing' dialog
• removing child tags in custom filter
• some youtube videos were not downloading due to a redirect issue. it is improved, possibly fixed completely
• collapsible panes, like 'advanced import options', will refit dialogs on collapse/expand
• fixed an bug in the way certain server errors were handled
• fixed a serverside bug for GET account when using account_id as identifier
• fixed 'modify petitioner' button in petition pages
• fixed a bug that was preventing clearing of temp path on client startup
• cleaned up some credentials code
• some general code grammar improvement
• reworded some help
• reworked account identifiers
• os x
• now has an app icon, woop
• os x page closing logic improved
• collections checkbox dropdown seems to work, now
• fixed segfault on popup message close
• thumbs loading from initial session seems to draw properly, sometimes
• fullscreen no longer causes segfault whenever you zoom or scroll

• version 97

• initial os x test build
• collect typo fixed
• 'remove all mappings with specific namespace' added to advanced content update dialog

• version 96

• went over _all_ my help (about 200KB of text!), cutting down waffly paragraphs, rewritting bits that were stiff or unclear, updating screenshots, and adding several new sections people suggested
• did a complete rewrite of how the client stores services in its database, streamlining everything
• several bugs exposed by the services rewrite fixed
• several moments of weirdness exposed by the services rewrite made more sane
• new version of sqlite (3.8.2)
• new version of python (2.7.6)
• cleaned up a debug print statement that was spamming gauge numbers to the log
• when loaded from a session, hdd import pages will now hide their processing panel
• fixed namespace sorting to manage numbers again
• collapsed the many different download gallery page classes to a single one
• started an important rewrite of how downloaders are going to work in future
• improved testing framework's pubsub testing support
• subs sync daemon now restarts correctly after a dialog-driven pause
• parents now obey siblings!
• siblings and parents are updated on forget pending
• got AMP message between persistent and temporary peers working

• version 95

• pages launched from a session will no longer run their query on startup; they will remember their thumbnails
• import pages added to gui sessions, although they do not yet remember their progress
• the way session pages are started is improved
• advanced content update dialog simplified and rearranged
• advanced content update db code written for copy operation
• advanced content update db code written for delete operation (local service only)
• advanced content update db code written for delete deleted operation ( local service only)
• made first version of IMMessage objects
• added network version checking to hydrus's AMP
• better AMP error catching and reporting added
• AMP tests moved further forward
• tightened up AMP network code, fixed last bugs
• cleaned up a huge pile of 'expiry/expires/expiration' semantic mess
• same for some mapping_ids/mappings_ids stuff
• removed predicate knowledge from media display classes, which was just a mess, really
• corrected a weird param order in sessions
• general code cleanup
• several old instances of tuple-list yaml compromise removed
• fixed a session bug in the testing code because of last week's manager management change
• fixed a hydrus session bug for accounts with no expiry
• fixed a very important server-side network version checking bug
• added client-side network version checking

• version 94

• reorganised the 'add tags from path' dialogs so the regex buttons made a little more sense
• when deleting a file, the current focus will pre-defocus
• 'confirm close client' dialog will now auto-yes after 15 seconds
• fixed a bug when POST hydrus queries couldn't return any data
• 'failed to update repository' message made simpler
• added tuples to pyyaml's safe_loader (which was more of a pain than I expected)
• manager management improved, with changes reflected across all the code
• session management prototype is done
• query pages can be added to sessions
• default startup session can be set in options
• sessions are managed with new submenu under file
• creating registration keys will show them in a new, more helpful frame
• same for initialising a server
• and for auto server setup
• auto server setup has cleaner error reporting
• new service admin accounts now have different access keys to the server admin account
• fixed up testing framework to test AMP a bit better
• IM session manager written
• server db support for new session manager added

• version 93

• can now undo/redo all tag operations except rescind petition
• can now undo/redo file pending, rescind pending and petition
• improved undo/redo description generation
• fixed a bug that was showing extraneous undo/redo actions that did nothing
• review services frame's permissions strings will size a little better
• new 'perform a service-wide update' button on review services frame
• started tag-specific dialog for this new button, and began db code that'll execute its stuff
• a little code reorganisation
• fixed a bug in the export dialog that was throwing errors when trying to overwrite read-only files
• moved a number of old dialog error messages to the new verbose popup system
• when accounts are made stale (as part of client version update), they will no longer lose their privileges
• fixed the log page to deal with (ignore) gauge messages
• fixed a db problem in my testing framework
• AMP IM support moved way forward
• IM manager first draft done

• version 92

• encrypted instant messaging framework started
• encrypted instant messaging test added
• encrypted instant messaging trust framework started
• started an AMP framework for encrypted instant messaging
• tag A/C exact match parameter added
• tag A/C exact match parameter test added
• tag A/C logic reorganised and improved
• tag A/C now gives exact matches for queries shorter than the A/C threshold
• tag A/C threshold default set to 2
• fixed an important architectural bug in pubsub
• generally cleaned up pubsub code
• 'popup messages sometimes not updating' problem is fixed
• a bit of initialisation cleanup to make startup behaviour more reliable

• version 91

• improved how accounts are identified in the server
• rewrote how serverside session data is managed
• added monthly data reset to session data
• fixed session update on account modification
• fixed a bug in accountidentifier generation from accounts
• fixed a bug in account-multi-session tracking
• server session manager unit test added
• reworked testing to take arguments to only run specific tests
• changed all file downloads to be more memory efficient, either by gui or subscription
• made download testing fail better
• added deviant art unit test
• fixed deviant art thumb parsing error
• fixed deviant art fullsize image parsing error
• undo system reworked in prep for first prototype
• undo/redo keyboard shortcuts (ctrl-z and -y) infrastructure added and db updated
• undo works for fullscreen
• a bunch of fullscreen keypress processing improved
• undo manager testing added
• added upnp option to all services
• added upnp daemon to maintain upnp serverside
• 'copy all tags' and 'copy all tags with counts' added to tagslists
• rejiggered a bit of the app startup, including db password check
• found a better error noise
• in fullscreen, the mouse cursor will no longer auto-hide when over an animation scanbar

• version 90

• client db unit tests added:
• 4chan_pass
• autocomplete_tags
• booru and boorus
• downloads
• favourite_custom_filter_actions
• imageboard
• import_folders
• md5_status
• media_results
• namespace_blacklists
• news
• pixiv_account
• improved services
• sessions
• shutdown_timestamps
• fixed modify account
• fixed modify account again!
• made account_info respond a lot faster with large tag counts
• neatened some client db code
• youtube 'file already deleted' error now reported correctly
• youtube general error reporting improved
• fixed multiple-computer-single-account data use reporting
• moved file parsing from nested dialogs to a single popup message gauge
• fixed an export folder path-display issue when an install moves location
• improved how some file downloaders process their files
• added prototype file download status bars to all download management panels
• fixed a bunch of tests that were cleaning up inelegantly

• version 89

• fixed blacklist manager for numtags
• fixed a sql typo for numtags
• delete from fullscreen menu fixed
• fixed a unicode error-handling issue
• sped up autocomplete tag fetch for large result sets
• improved the way autocomplete dropdown scrolls
• vastly improved efficiency of how all listboxes draw
• the server will no longer start if something is using the server admin port
• i.e. you can no longer run multiple copies of the same server by accident
• improved client and server port change (service shutdown and restart)
• improved testing framework to handle fake db reads better
• improved testing framework to manage client and server sessions
• several network request unit tests added:
• access_key
• account
• account_info
• account_types GET
• account_types POST
• file_repo file
• file_repo thumbnail
• init
• ip
• news
• petition
• registration_keys
• services GET
• services POST
• session_key
• stats
• update GET
• update POST
• fixed clientside permissions exception
• several db unit tests added:
• import_file
• system_predicates:
• age
• archive
• duration
• everything
• file_service
• hash
• height
• inbox
• local
• mime
• not_local
• num_tags
• num_words
• ratio
• similar_to
• size
• width
• limit
• fixed a bug in system:age?
• fixed a bug in system:duration<0
• fixed a bug in system:duration?0
• improved num_tags logic
• fixed system:num_tags<1
• fixed system:num_tags>1
• fixed a hash filter bug for system:similar_to
• fixed system:similar_to for unknown hashes
• fixed system:size Bytes
• updated linux.html and running_from_source.html in help

• version 88

• moved server daemon start to a better place
• cleaned up more server db code
• fixed getnumpetitions
• fixed getpetition
• reworked modifyservices and fixed server request
• reworked modifyservices dialog to work with new system
• server service options now managed from modifyservices dialog; old options dialogs deleted
• auto-repository setup restored and fixed
• harmonised edit_log actions all across the code
• added namespace blacklist manager
• added namespace blacklist dialog
• integrated namespace blacklist into:
• • search by tag
  • search by tag count
  • creating media results (thumbnails)
  • autocomplete tags read
  • autocomplete tags write
• fixed a problem with deleted tags not being applied correctly on initial thumbnail display
• started a server unit testing framework
• added 'running in linux' info to help

• version 87

• misc:
• • fixed system:untagged, which was doing numtags>0 by accident
• basics:
• • moved client local service to twisted
  • moved server to twisted
  • cleaned up a whole lot of unrelated server stuff I haven't touched in a while
  • fixed some misc typos
• details:
• • built twisted server framework to plug into hydrus
  • changed hydrus authorisation header
  • changed sessions to manage accounts, reducing server db load
  • reworked session cookie to be neater
  • cleaned up a bunch of server code
  • fixed how certain server errors were being printed to the log
  • reworked response_context to a body/path dichotomy to reduce cpu+memory on file requests
  • collapsed a bunch of GET POST requests to the same path
  • reworked server_service_identifiers to be port-independant
  • improved data use logging to support new session management system
  • moved session management to new system
  • moved error management to new system
  • integrated twisted thread into hydrus controller
  • reworked root and favicon
  • reworked local file and thumbnail requests
  • reworked restricted service requests
  • reworked admin service requests
  • reworked repository service requests
  • removed manage options query; it'll be rolled into manage services admin queries
  • updated serverdb to manage with new system, harmonising internal and external requests to one workflow
  • made server-side data use tracking simpler
  • added Content-Type header to most requests
  • improved how key registration, init, and account GET requests are handled client-side
  • harmonised how account_keys are created server-side
  • moved server management from serverdb to servercontroller, with better pubsub restart
  • moved several useful functions to the new serverconstants.py
  • fixed a max_age sessions issue

• version 86

• timeout on connections improved
• rewrote the objects behind the dumper
• the dumper should now select media properly again
• closed pages now timeout after an hour
• upgraded to new version of sqlite
• fixed client port bind detection and error reporting
• you can set the client's local server's port in client options
• local server port changes will happen when dialog changes
• unified the upload and youtube download popup messages into one popup
• fixed changing thumbnail size (stupid typo!)
• added 'database->regenerate all thumbnails' to fix certain thumbnail errors
• add a popup gauge for regen all thumbnails
• fixed yesno dialogs, which were showing a neutral 'Cancel' button rather than a red 'no'
• the upnp dialog's buttons now work, have fun with it!

• version 85

• updated to new version of wxpython
• several key_down events moved to char_hook
• rewrote collapsiblepane class to remove weird refit and scroll behaviour
• a problem with adding service via a registration key is fixed
• tag upload pending is granulated into separate queries, to reduce server and client lag
• uploads moved to an asynchronous popup message with a gauge, rather than the modal progressdialog
• reorganised file upload content_updates to be a bit smoother
• fixed a redirection bug that wasn't obeying schema changes such as http->https
• updated e621 url to https schema

• version 84

• switch fullscreen button added to fullscreen canvases
• messaging stuff is disabled for now
• gif scanbars now fill up as the frames render
• pngs with transparency are no longer drawn with black background after first viewing
• made a small change to listbooks to correct some gui-weirdness, particularly in local options
• autocomplete dropdown windows will now hide-and-reposition on parent scroll events
• autocomplete hide-and-reposition waits 250ms rather than 100ms, making it a little less flickery
• a graphical bug related to hitting end on a large search is fixed
• upnp framework started
• services->manage local upnp started
• new messagegauge added
• fixed mp4 import
• youtube url->formats added
• youtube format chooser dialog added
• youtube downloader thread added
• youtube gauge popup added
• youtube error handling improved
• youtube num_bytes_done added
• youtube unknown total_num_bytes handled
• youtube cancel button added
• youtube right click dismiss throws up a yes/no dialog to decide whether to cancel the download
• started manage upnp dialog
• closed page undo added
• closed pages will be paused and quiet
• pause import folders option added
• fixed an invalid index drawing bug after removing certain media
• thumbnail waterfall improved
• tags are limited to 1024 characters

• version 83

• sort by longest fixed for files with no duration
• reverse sort by rating fixed for files with no rating
• sort by largest, newest fixed for files with unknown size or timestamp
• sort by unknown/absent values sorts more accurately
• search by num_tags and min_num_tags fixed
• remove media thumbnail update bug fixed
• currently viewable indices screwed up after a remove media bug fixed
• remove all media entire black screen bug fixed
• delete media redraw optimised
• several message print unicode/raw byte errors fixed
• scrollbar position calculation improved, so 'black rectangle on scrolled thumbnail canvas when num_cols changed after resize' bug should be fixed
• 'tiny black lines on small slider drag' bug fixed
• moved a bunch of right_up events to right_down, see what you think
• popup message cleanup is better on shutdown
• pubsub typeerrors handled more gracefully
• improved how options init and update works
• client will now remember the restored size, restored position, maximised state, and screen the gui was last left at
• client will remember all the same details for fullscreen, separately
• these sizes _should_ rescue from offscreen if you disconnect/reposition a non-primary display
• shift+home/end now scrolls and selects, rather than only buggily scrolling
• I fixed a content_update bug re petitions
• 'show in new page' pages will no longer show search subbox
• added hentai foundry tests
• fixed hentai foundry title and creator tag parsing in certain cases
• fixed some related html unicode issues
• initial value on tag siblings dialog fixed
• tag siblings and parent dialogs now start sorted by the right column (parent/older sibling)
• setting a default collect will no longer cause a nasty crash!
• setting an orphaned default collect will no longer cause problems
• deselect thumbnails has improved focussed media management
• can now select inbox or archive from right click menu

• version 82

• a bug where slow search results would sometimes appear after search predicates were removed has been fixed
• a lot of autocomplete gui- and db-blocking reorganisation
• searches are now entirely asynchronous to gui thread
• searches are split into two granular phases, and are cancellable during processing
• simplified system predicate storage
• consolidated all system predicate filtering to initial db search
• huge improvements to how the thumbnail canvas is sized and extended and drawn to
• numerous fixes and improvements to how thumbnails are drawn to screen
• reworked how collect works
• and sort
• the raw cpu time behind sorting in muuuuuuuch faster
• clarified my listening media class
• added another optimisation to canvas resizing
• big improvement to canvas redrawing and refreshing
• important fix in how client figured out what to draw when clicking in whitespace
• thumbnail fade is much smoother
• thumbnails should now generally draw a little smoother
• selectall and selectnone's thumbnail fade is less cpu intensive
• cleared up a number of collect related selection and index bugs
• fixed a media identifier issue
• all import-related append and drawing is muuuuch faster
• flicker bug on small appends fixed
• fixed a page up/down drawing bug on the main thumbnail canvas view
• fixed an unknown-timestamp thumbnail right click issue
• made my internal media data storage system a lot simpler
• new sortedlist class to make some media stuff easier to manage
• reworked how new media is added to a page of thumbs (usually import-append, but the new system supports collect- and sort-sympathetic insertion):
• -calculating combined tags on the left can be up to ten times faster
• -integrating new files into the sort is much quicker
• -integrating new files into the collect is much quicker
• -improved how new thumbnails are decided to be drawn
• trying to archive or inbox more than one file, you'll get a yes/no dialog to confirm
• started db testing framework
• db tracebacks improved
• slimmed down content_update data-side processing
• a couple small code fixes
• archive and inbox no longer remove from the search if the opposite system predicate is set
• system:rating fixed
• fixed a bug where you could middle-click-download files that had no known source
• couple of bugs with system:ratio fixed
• system:ratio string representation is 16:9, rather than 1.777779
• gui thread is more intelligent in telling non-gui threads that it is busy

• version 81

• mp4 added
• mp4 mime search added
• wma added
• wma mime search added
• wmv added
• wmv mime search added
• changed to video and audio icons, rather than one for each mime
• popup messages now dismiss on RIGHT_UP rather than RIGHT_DOWN
• moved many more messagebox errors to the new popup error system
• cleaned up clientgui's superfluous error handling
• rewrote error redirection to new popup system
• more error overhaul
• print statements overhaul
• better db updated statement, through popup system
• traceback show/hide button added to new popup error box
• copy button added to new popup error box
• added a 'x more messages/dismiss all' popup when there are more than ten messages to display
• review services refresh account button will disable until the server replies
• file system predicate dialogs have better initial focus
• streamlined server file import and improved respective data usage calc
• made internal options storage a lot more efficient
• cleaned up a little cache code
• export to zip improved to new file handling system
• upload to repo improved to new file handling system
• some db write priority stuff switched around and renamed
• streamlined a little client db workflow
• neatened client db exception handling
• db updated message is now a popup
• removed some orphaned methods in client db
• client file and thumbnail server requests made a bit faster and neater
• removed db loop from export and dump file fetch
• neatened some client and server file 404 exception paths
• further cleaned up the way file paths are calculated and fetched
• further cleaned db in relation to files, and removed the old mime cache, for both server and client
• removed orphaned db export infrastructure
• add_thumbnails in client made simpler
• improved http response parsing
• refactored a bunch of http response processing
• new exception display system added for caught exceptions
• fixed a shutdown hang on certain delete orphans failure

• version 80

• manage tags and ratings dialogs have their initial focus corrected
• custom filter now obeys tag parents for tag actions
• fixed an annoying as hell multiple-page shared-thumbnail visual-selection-status bug
• added e621 test
• fixed e621 tag parsing, including new species namespace
• might have fixed a missing thumbnail error
• fixed system:hash
• fixed some repo file downloading stuff
• fixed some general db fetching file info stuff
• made x files imported from y message a little slimmer
• fixed minsize for the main gui frame
• added better initialsize for the fullscreen frame
• putting a thread url into the thread_id part of the dumper will auto-convert to the thread_id
• added dumper's multipart form data generation test
• added generic multipart form data generation test
• fixed dumper unicode issue
• error reporting in dumper is a little better
• streamlined how mime is calced
• massively improved how mp3, flac and ogg are parsed and validated
• massively improved the internal import files workflow to use less memory and hdd reads and writes, particularly for video
• similarly improved server-side import workflow
• the way temp dir works is improved
• sped up server file/thumbnail fetch
• reworked how thumbnails are fetched in a couple places
• started rework of how options is stored
• wrote mp4 properties parser, but no mp4 just yet

• version 79

• popup messages will now report whenever a subscription or import folder successfully imports some files, with a button to show them in a new search
• popup messages now wrap
• slightly better error popup
• many more errors reported through error popup
• old logging system switched over to new messaging system
• popup message manager will only show ten messages at once, now
• selectfromlistofstrings now supports enter key to select
• completely reworked how dialog cancel works; absolutely all dialogs should now close with escape key
• reworked how a bunch of dialogs do ok
• a huge amount of dialog refactoring
• made ok button initial focus of all dialogs
• couple of bugs in service options dialog fixed
• rejiggered some button names and focus behaviour a bit more
• some classname refactoring
• muchly improved string->unicode handling
• improved timestamp generation
• fixed a clientserviceidentifier->text bug
• reworked how namespace cache in tagsmanager object is calculated
• improved instantiation of noneablespinctrls
• fixed a bug in the thumbnails download daemon
• changed the way daemons wait for the db, much to the better
• moved daemons out of the db object
• rejiggered writedaemon synchrony so exceptions work
• fiddled around with some help links
• default, non-maximised size of client is a little more comfortable
• custom filter now has a popup that'll let you change the custom actions mid-filter
• server now uses new synchronous logging system
• fixed an options save bug for server
• updated server diagram in help
• added test for dialog selectfromlistofstrings
• added test for dialogyesno
• made a framework for testing that requires network stuff
• made a newgrounds test
• fixed newgrounds swf parsing
• made a framework for testing that requires file reads and writes
• fixed a graceful-exception bug in mime parsing
• added test for synchronous import_folders
• added test for delete import_folders
• wrote a test for importfolders daemon
• import folders no longer delete or reattempt failed imports; they'll just ignore them
• import folders are deleted on update, since old objects are obselete
• import folders won't try to do zips any more; they'll just ignore them
• rejiggered how import folders does its path parsing to remove mime calc cpu usage

• version 78

• expanded parents testing with a namespace example
• made getgstvcp in tags manager only return numbers for vcp
• refactored ManageDialogs to their own file
• refactored Exceptions to their own file
• improved my testing framework so it can do wx gui elements
• added test for dialogchoosenewservicemethod, fixed a typo
• added test for dialogfinishfiltering
• added test for dialogfinishratingfiltering
• those two finish filter dialogs are a little simpler, now
• set up a system so dialogs are a bit simpler, in terms of button event processing
• added test for dialogfirststart
• added 'add parent to tag' on tag right click menu
• added 'add sibling to tag' on tag right click menu
• made manage parents/siblings dialog focus selection more intelligent
• 'audio - any' mime search fixed
• reworked audio embed container significantly
• added embed button
• all noisy mimes are now protected by embed button
• you can now rescind pending file uploads or petitions from thumbnail right click menu
• fixed a redundant bit of code in process content updates
• in modify account dialog, set account expiry to 'does not expire' fixed
• logging now happens a little more synchronously for client
• popup messages prototype done
• popup message for normal text
• popup message for some errors
• fixed a small redundancy bug in data cache
• fixed a localisation bug when converting numbers (1234) to formatted version (1,234, 1.234, 1 234)
• improved the code behind 'open selection in new page' in two ways

• version 77

• tag parents manager unit tests done
• completely rewrote parents pair generation
• rewrote parents pair retrieval to be much faster
• parents manager is much more resistant to loops
• in two ways!
• loop detection fixed in manage parents dialog, as well
• fixed how deleted parents are applied to combined service identifier
• fixed how deleted parents are applied to specific service identifier
• tag manager unit tests done, and everything was working ok!
• shuffled objects around a bit, unified some tag code
• tag manager merge unit tests done, and everything was working ok!
• tag manager merge code simplified and result rewritten to a new, simpler class
• small bug in how preview canvas fetches tags fixed
• finished off import folders daemon
• added import folders gui
• added flac support
• added ogg support
• added a secondary layer of mime detection for greater accuracy
• added a tertiary layer of mime detection for greater mp3 accuracy!
• fixed an offset bug in mime detection
• simplified and unified client/server file info parsing
• import error handling improved
• neatened how special thumbs are stored, refreshed, and retrieved
• neatened how the bmp->png conversion happens
• improved hash->mime caching for both client and server, to speed up http retrieval time for the new filetypes
• better mime fetch in dumper, too
• improved debugging messages for a bunch of my custom objects

• version 76

• made unit testing framework
• tag siblings object now tested properly
• hydrusdownloading functions now tested properly
• added simple mp3 support to client, with external launch or embed
• added search by mp3 mime
• added mp3 to server
• higher precedence deleted tag siblings now overrule lower existing pairs
• tag sibling closed loop pairs ( a, a ) are skipped
• neatened predicate sibling matching in two ways
• improved tag autocomplete matching logic
• skip logic improved for like ratings filter
• skip logic improved and button added for numerical ratings filter
• did a little work on reducing memory usage
• reordered data cache's memory purge, which will make huge images load better
• fixed a zoom bug with pdf buttons
• added some select all, select none and refresh to thumbnail right click menus
• reorganised menus a bit anyway
• fullscreen tags on the left are now sibling-collapsed

• version 75

• fullscreenpopup window added
• fullscreenpopup window can be dragged about
• fullscreenpopupfilterinbox
• fullscreenpopupfilterlike
• fullscreenpopupfilternumerical
• accuracy slider moved to popup
• compare same image until done added to popup
• keep on left/random/right added to popup
• 'don't ratings filter this' added to popup
• that annoying as hell thumbnail selection drawing state bug is fixed
• display of flash with only one frame fixed
• downloaded tags now work again, sorry for the disruption!
• deleted tags will now always show in manage tags dialog
• fixed the various problems that were stopping DELETED->PENDING->CURRENT working for tags
• fixed the children can only have one parent bug
• fixed a deleted tag parents bug
• a little counting logic improved in the special deleted_pending case
• fixed a typo re uploading file to a repo
• main guis statusbar now has a little 'db locked' indicator

• version 74

• flash scanbar added
• made new mediacontainer class for media canvas
• extracted scanbar to separate class
• made animation frame tracking and control more sensible overall
• previous_frame/next_frame shortcut now works for flash as well
• fixed age predicate for > hours
• fixed an important 'all known tags' A/C count cache bug when multiple services have the same tags for a file
• if you have resolve_petitions permission for a tag service, tag siblings and parents petitions need no reason
• slight change in how un-namespaced totals are calculated in A/C read
• collection now obeys siblings
• thumbnail upper text now obeys siblings
• fullscreen text now obeys siblings
• bit of siblings code cleanup
• a little changeup to how showselectioninnewpage collects its media
• /asp/, /gd/, /lgbt/, /vr/ and /wsg/ added to default dumper support
• trying out making the server non-daemon, to see how that manages

• version 73

• every file operation from fullscreen was broke due to a single-character typo that got propagated
• fixed local ratings, which were broken five different ways
• ditched the deleted_pending concept, which was actually making things more complicated
• possible fix to review services opening on bottom level
• dumper no longer says it is still dumping on page close if it is done dumping
• you can now scroll the dumper's upper comment box
• moved site links over to github
• fixed a typo re downloading a file that is already in the db and adding tags at the same time
• local tags delete fixed
• tag parents now apply globally, like siblings
• thumbnail resizer daemon is less verbose and more helpful on IOErrors
• fixed namespace colours in manage tags dialog
• open selection in a new page now works for single-file selections
• upload tags progress bar misalign fixed
• namespaced counts no longer total up in A/C writes
• manage siblings dialog now catches collisions when you put in the old, not the pair, and asks you what you want to do
• age predicate now supports hours
• while search is not synchronised, A/C will now query db when it might have queried the current media
• think I fixed a sometimes-not-initially-showing-first-frame-of-gifs problem
• downloaders now run parsed tags through siblings and parents managers
• maaaybe fixed a cache counts error
• wrote a little more sibling and parent help, and updated schema

• version 72

• remote tag siblings!
• remote tag parents!
• simplified update nomenclature
• streamlined client-to-server update objects and processing
• streamlined server-to-client update objects and processing
• did a bunch of work on content_updates
• did some work on service_updates, too
• improved client-side petition handling
• a bunch of changes to how uploadpending works
• the way pending downloads are stored is improved
• mappings tables are collapsed server-side
• file tables are collapsed server-side
• moved tag siblings and parents to content_updates
• moved file upload/download to content_updates
• reworked tag content_updates to be more streamlined
• addfile/tagrepositoryupdate is made entirely content_updates, now
• simplification in the way deleted content is stored
• repo update processing string is muchly improved
• repo update processing is smoother!
• a slight change to default behind-the-scenes searches
• improved how the server creates updates on service init
• improvements to service_identifiers_to_statuses_to_tags sub-init and related parts of tags_manager
• some misc nomenclature improvements
• fixed a denypetition bug for file repositories
• fixed optimised petition handling in server
• fixed posting tags in dumper
• fixed a problem with getunknownaccount
• server now analyses on vacuum, like client
• fixed a lolbug in thumbnail resizer daemon that was continually resizing thumbs
• fixed a bug in cache counts, which should fix the 'pending menu disappeared' problem
• corrected num_archive count in file repos
• downloading files is simpler and faster; download menu is simpler
• fixed server initialisation
• updated to new sqlite version

• version 71

• collapsed the four mappings tables into two tables
• merged the two active_mappings tables into the mappings table
• made a great number of changes to how mappings and active_mappings are stored and processed throughout
• 'active' and 'null' nomenclature is now 'combined'; null service_ids are now just ints
• improved deletepending so it isn't so tough on the a/c cache
• tags regex dialog entries 'for all files' and 'just for this file' was all broke
• A/C read now bumps the exact match of the entry to the top of the list, if its count is non-zero
• fixed A/C for weird-character queries, like '['
• the dumper now makes success/error noises as appropriate
• you can turn these noises off in the new sound tab in file->options
• screwed around with garbage collection while checking mimetypes during pre-import
• adding a tag parent will spam-add the actual parent tags to every child instance for the appropriate service
• updated db diagrams
• revised my sibling chain collapsing algorithm
• locked db on init dialog message is improved a little
• system:limit added to combined file service searches
• num pending menu counts now split into with (pending/petitioned)
• corrected a server db index oversight
• newgrounds title tag fixed

• version 70

• tag parents db stuff
• tag parents manager
• tag parents predicate and matches indented expansion

• and drawing it!

• tag parents top result reordering
• tag parents top result insertion
• tag parents actually doing what they do
• polished off tag parent help
• made a full-search table to speed up tag A/C requests
• added tag full-text search
• reworked how tag and file services add and reset, to reduce A/C time
• reworked recalc active mappings to be more beautiful, if not faster
• fade animation timer improved in several ways
• thumbnail fetch made much smoother
• newgrounds artist downloading for games and movies added
• newgrounds subs added
• download panel input now highlights on init
• 4chan filename tag added
• the setfocus on filter close is neater
• age phrase is now 'imported [time] ago'
• age is now shown in fullscreen
• can now copy file from fullscreen, from menu or shortcut
• 4chan pass authentication improved
• fixed a couple tag service precedence sync bugs

• version 69

• first dialog for tag siblings
• tag siblings db table
• tag siblings manager in controller
• tag siblings display in tagsboxcpp
• tag siblings display in tagsboxflat
• tag siblings display in tagsboxmanage
• tag siblings display in tagsboxactiveonly, via A/C dropdowns, for both read and write
• top result sibling switcheroo in A/C write
• A/C db fetch now does siblings too
• db tag search does siblings
• siblings help, with some nice mini-charts
• I fixed collections, which were typo-broke; I'll make sure it doesn't happen again
• more granular subs error handling, meaning individual file failures won't crash an entire sub
• individual subs can now be paused
• e621 fixed for real this time
• A/C improvement that slows tag updates a little but should stop A/C lag after an update
• as a result, trying out dropping the CPU-intensive fatten_ac_cache maintenance call
• A/C read will now update system preds every time you click on it, so inbox/archive counts will stay accurate
• A/C "all known files + tags" will no longer show the mega-laggy total file count
• a list -> tuple convenience fix in sanelistctrl
• if you don't have any pixiv credentials set up, you will now no longer get the option to start downloading pixiv stuff
• fixed a tiny typo in the thumbnail resizer that made it wait far more politely than was intended
• slight change to ratio system pred that fixes some lockups, sometimes, I think
• fullscreen flash and video will get a pixel of whitespace on the right

• version 68

• fullscreen view now takes addmediaresult
• export to zip!
• import from zip, with tag regex stuff too!
• export to encrypted zip!
• import from encrypted zip!
• import encryption help in advanced.html
• import process code processes mimes less jankily
• delete after successful import checkbox added
• import regex tags dialog now has # namespace
• import regex tags dialog collects its info a bit more intelligently
• new confirm exit client option
• subs will save their progress every 20 files downloaded, so pause/restart is less punishing
• simplified thumbnail cache retrieval
• some thumbnail waterfall and animation tuning, let me know what you think
• e621 fixed
• danbooru fixed
• reworked booru gallery page num calc, which was completely screwed
• DA fixed for most cases
• booru thumb parsing made a _little_ more permissive
• server now uses the same new file storage system as the client
• download cancel buttons have a little better feel
• new thumbnail prefetch thread makes thumbs load a load faster
• fixed a little rubbish naming semantics in ClientGUI
• reorganised and streamlined much of the encryption code
• updated a bunch of help screenshots

• version 67

• subscription db access improved
• subscription 'delete subs seemingly at random' bug fixed
• some nice subs help
• subs and repos, if changed during a pause, will restart with new changes
• subs and repos will automatically pause while their respective dialogs are open
• new namespace | regex listctrl in regex dialog, instead of old sctvcp rubbish
• new /aa/aa...0 file storage system for client
• fixed deleteorphans mime issue
• export and copy files now export writeable files, not read-only
• rejiggered daemon db access, improving maintenance reliablity
• dumper and 'show in new page' pages now process content updates correctly

• version 66

• subscriptions done! works for all normal download types
• pause repo sync, pause subs sync
• all download sites moved to the new system
• lots of small changes to how download code works
• pixiv tags is renamed to pixiv tag, since it only does one!
• fixed DA
• fixed DA again!
• fixed DA tag parsing for all the diff types of username re http://help.deviantart.com/106/
• fixed pixiv, somewhat
• fixed giphy
• fixed boorus for gallery_advance_num > 1
• boorus reset in db to default
• fixed copy files
• fixed getmime
• a silly parent assignment for A/C meant they were not closing with pages
• made the cpu burn for the thumbnail resizer a bit more polite on hdd
• repo sync daemon and subs sync daemon combined
• some timing adjustments in sync daemon
• subscription_type constant -> site_download_type
• fixed temp folder being cleared on startup (a read-only issue)
• advancedoptions classes now support setinfo

• version 65

• added subscriptions dialog
• added prototype subscription daemon, but not yet activated it
• added downloader classes
• began reorganisation of nearly all download code
• moved parsing around
• moved advancedhttpconnection around
• moved serviceupdate around
• moved some content_update stuff around
• moved some constants around
• files in client_files now have extensions
• files in client_files are now read only
• pdf launch is simplified as a result of this new ext stuff
• fixed find similar images; it was just a typo
• also fixed initial predicates string display
• rejiggered the thumbnail resizer again; due to my stupidity, it was causing lag
• thumbnail resizer burns a little more cpu when there are >10,000 or >100,000 thumbnails to render
• it also now does thumbs in random order, for a couple of good reasons
• upgraded sqlite, hopefully some queries will run faster?
• upgraded to python 2.7.4. please report any weird errors

• version 64

• got rid of system:not_uploaded_to - now system:file service, which is a lot more powerful
• some display changes to system predicates to make them a bit more human-readable
• added export dialog
• made a couple small changes to help about the export dialog
• collect by rating
• sort by rating now works for collections, using an estimate (only accurate when also collecting by that rating)
• built a custom radiobox class and added it to ratings dialog so it has radio buttons again
• new ratings filter for local_ratings_like services. works just like normal inbox filter
• added a little info in help about this new filter
• new session manager for pixiv and hentai foundry, reducing number of session inits the client has to do
• new init welcoming window for first boot
• new politer error message for when db is locked on boot
• shift+arrow keys now pan in fullscreen mode. you can change them in options as normal, if you like
• resize thumbnails daemon is now part of maintenance thread stuff. hopefully less blocking now
• inbox pages will remove media when you archive stuff, again. this is just a bit I forgot to update from last week's predicate overhaul
• fixed and changed the way hitting the change tag service button works
• I fixed previews and fullscreen cache estimate in local options too!
• fixed giphy so it works with their new api
• rejiggered the update notification order to be a bit more accurate
• updated auto-setup code and help to use new no-ip domain rather than old raw ip
• v64 will also update the old ip to the new no-ip domain automatically
• bit of variable renaming
• cleaned up a couple menu memory leaks
• updated a little of future.html

• version 63

• added include/exclude namespace predicate
• added intelligent delay to A/C to smooth out db requests
• rejiggered the A/C code so that spamming/pasting several characters and then deleting some will re-search for matches as appropriate
• cut size down to just B, KB, MB, GB
• maybe I fixed system:ratio in certain cases!
• some general predicates display syntax changes
• (archive/inbox) and (local/not local) mutual exclusivity with new system
• system:num_words
• right click -> open selection in new page
• upgraded from PIL to Pillow, to zero noticeable effect lol
• had another look at dodgy animated gif transparency; couldn't figure it out
• CMYK jpegs images now supported! interlaced pngs simply not supported with PIL/Pillow yet
• numpad delete added as a valid shortcut key
• removed an old buggy deleteorphans line
• new thumbnail resize daemon to fill out the thumbnails directory in the background
• collections with only one file will now be converted to singletons
• screwed around with locale settings. not sure if this will break things!
• manage tags and ratings now in fullscreen rmb menu
• on downloads, ( url -> hash ) pairs are stored in db, even if the file was already deleted or already in db
• a few additions to help
• reworked the spinctrls system preds to not have 100 as default max
• changed a bunch more spinctrls, including memory sizes, in local options
• and a statictext that wasn't showing number of thumbs for cache size
• bugfix in ratings filter

• version 62

• remade a/c dropdown to a better, less buggy class
• reworked a little of the a/c dropdown focus logic
• reworked a little of the a/c dropdown show/hide logic
• a 'sometimes hangs on "gui" on startup after crash' bug is fixed
• can copy from listboxes with new right click menu
• dumper page will protest if you try to close it while it is still dumping
• import page won't protest if you try to close it while paused
• num_words will show on pdf right click
• fixed the network version mismatch exception, exactly one week late lol
• couple of tiny menu memory leaks removed
• couple of misc gui tidies
• custom filter favourites save and delete buttons will grey out as appropriate
• bit of general code cleanup
• we nested classes now
• moved contentupdate class from cc to hc, in prep for repo update rewrite and contentupdate gen

• version 61

• session db tables
• session manager objects
• session net code done
• session failure recovery
• registration key table
• registration key requests
• some registration help added
• harmonisation of many http request names
• extracted uploadpending code from db transaction, making it a lot more polite and architecturally correct
• pixiv now parses the ascii creator name as well as the japanese counterpart. No option to choose between them yet!
• improved processcontentupdate to support more actions, making db a little more serial
• home and end now scroll to focussed media
• some http timeout tweaks, to stop the 'can't connect to server' problems on complicated queries
• a typo in modify account types was fixed
• manage tags dialog shouldn't fritz out quite so much when a service doesn't have post_data permisisons

• version 60

• pixiv account management dialog
• pixiv gallery parse
• pixiv page tag parse, including creator and title
• pixiv user download
• pixiv tag download
• collapsed the page chooser a little to fit pixiv in comfortably
• pdf support!
• pdf launching via preview/fullscreen button
• pdf header parsing with num_words rough estimate
• system:mime=application/pdf
• system:mime=application
• custom filter favourites done! with default, previous, save, save as and delete
• some small changes to temp folder logic
• vastly improved scroll-thumbnail-prefetch logic. important bits all rewritten and made sane. data and gui rows line up better, with fewer bugs
• vastly improved thumbnail fade. now attempts to render at 60fps, and calculates alpha much more efficiently
• thumbnails won't fade if they are off-screen; they'll just draw in one frame
• fixed thumbnail last row drawing problem (again!)
• due to daemon-db-spam, db maintenance check was only firing on pc wake from sleep, now it fires properly every twenty idle mins
• maintenance timer won't fire directly after waking from a sleep
• deleteorphans copes with alien files a bit more gracefully
• A/C appearing on top is mostly fixed
• some general A/C weirdness is fixed/made better
• reworked some of the thumbnail selection and scrolling code to line up better
• thumbnail window will stop spamming 'scroll to focus' every time something small changes. it should only do it on fullscreen close and non-ctrl or -shift mouse/key events now
• some general focus code was looked at
• ratings filter now supports the fullscreen_switch shortcut
• just a couple changes to help to include the new 'f' fullscreen_switch shortcut
• 10s timeout on http queries
• http redirect code is improved
• http cookie handling code is improved
• made hf session establishment a bit cleverer

• version 59

• new vastly improved 'collect by' dropdown
• fullscreen 'f' shortcut
• fullscreen default option
• added tumblr prototype parser, with tags
• server init and repository init typos are fixed
• a misaligned sizer was fixed
• a bunch of spinctrls no longer init at <= 100
• the problem with the last thumbnail rows sometimes not drawing properly should be fixed
• A/C lag calculates a little more intelligently
• manage ratings for collections now works

• version 58

• new custom staticbox sizer/panel put in about fifty places
• big layout changes in management panels
• I made a new, better set of sizer flags for nested sizers
• testing out several new borders
• several other small layout changes, lining things up
• a few colour changes with the new classes
• advanced options class is better, more streamlined
• advanced options class is less flickery, I think
• A/C spazzes out less when dragged about
• A/C queries db less annoyingly, especially when you type fast
• the thumbnail display draws its whitespace a little better, filling in the right-hand and bottom-end gaps
• importing thumbnails now add to the last row one thumb at a time (rather than at one row at a time) once the scrollbar appears
• shift + f7 now sends things back to inbox, like a reverse archive.
• this works from thumbnail menu
• and in fullscreen
• new menu item 'remove these' for thumbnails, anywhere, including dumper!
• 'tags just for this file' now works a looooot better! it unions all selected tags, and detects deselections better
• giphy downloader added
• giphy tags added, thanks to @fluffy_cub
• copying file now _also_ copies paths to clipboard; depending on where you paste, you'll get the files or the text appropriately
• new shortcut (default ctrl+m) for 'focus media'
• focus search now works for manage tags and tags regex dialogs
• think I worked out the last of the up/down arrow cursor shortcut problems
• a ton more code cleaning
• hentai foundry artist not found 404 error detected
• in regex dialog, tags are cleaned before display on screen
• 4chan ban is now recognised, and thread dumps are hence suspended
• when ordering tags by incidence, disputes are now resolved by lexicographic
• fullscreen init is a little faster for the first image
• select imageboard dialog now toggles tree with double click
• select imageboard dialog is also taller

• version 57

• added quick vs accurate slider for numerical ratings filter
• new slider will remember where you left it last
• ratings filter internal comparison logic is more sane
• 'already rated' will now show the specific rating beside it
• ratings filter (hopefully!) chooses better comparisons, in about four different and new ways
• you can now ratings filter just one file
• ratings filter code is neatened a whole lot, especially on my horrible variable names
• I may have fixed an equality ratings filter bug
• internal rating data -> pretty text for display is more flexible and accurate
• going back several steps on ratings filter will no longer forget to re-rate already rated files
• delete from dumper no longer screws with the indices
• dumper is more stable, less likely to bail on the whole dump if something odd happens
• dumper should recognise duplicate file errors and recover the fail
• if the user has idled for the last twenty minutes, the db checks to see whether it can optimise itself
• fixed flickering on fullscreen drag
• fixed flickering onn fullscreen image init, meaning smoother mousewheel/whatever scrolling
• fixed a graphical glitch in manage boorus dialog, and a bunch of others (removed all staticboxsizers)
• fixed a similar glitch in manage ratings dialog - it now has dropdowns; I'll probably make it radio buttons again soon
• fullscreen page navigation (previous, next, first, last) shortcuts are now customisable
• custom filter now supports these new page nav shortcuts
• can now do home/end and page up/down in shortcuts
• shortcuts listctrls now sort by action by default
• I improved the my custom listctrl's code generally
• little less ugly code, here and there
• fixed a whole ton of regular little bugs

• version 56

• new existing tag sort dropdown
• an option to set default existing tag sort
• dumper error logic improved
• dumper now does random sort properly!
• double buffering hilariousness
• some reordering of the local options dialog
• made a 'shortcut' entry object
• made a parent frame object
• collection thumbnails will redraw if the sort changes
• add media and add thumbnail data and graphics logic is improved
• 4chan pass now authenticates for an entire year
• bit more in help about the autocomplete tag entry
• tag petition processing speed is a loooot faster

• version 55

• custom filter prototype ready!
• added some info on custom filter to help (in advanced.html)
• you can now set a default tag service in options
• tag rexeg dialog's listctrl refreshes without scrolling up to the top every time
• tag for this file in tag regex dialog applies to all selected, not just the first
• scrolling in autocomplete entry will scroll services in manage tags and tags regex dialog
• ctrl+scrolling in autocomplete entry will scroll through results
• complete rewrite of image prefetch calculation
• couple of content-update display bugs in ratings filter fixed
• in manage ratings dialog, sliders will preset to best guess at current score
• dumper should be fixed
• resized active predicates box (much smaller)
• a deprecated call updated
• added some forum extensions

• version 54

• several new flash key and mouse event bugs fixed - now: mouse in = to flash | mouse out = to frame
• the problem with scrollable management windows not drawing things on scroll is fixed; staticboxsizers now replaced with "- title -"
• ratings filter now accepts several shortcut keys, directing to whichever window the mouse is over
• the ratings filter windows will redraw themselves appropriately on content and service updates, e.g. changing tags with F3
• ratings filter will choose the files to compare to more intelligently, both from the db and the currently-rated pool
• the way services are identified behind the scenes is entirely overhauled and improved
• the review service dialog is better as a result
• all dialogs will now cancel-exit on pressing escape
• started fade in and out for thumbnails
• up cursor arrow in filter now skips
• the tags regex dialog is refitted so it will refit on small screens better
• dumper now dumps in its own thread (no longer locks up the gui while uploading)
• the calculation of num_frames for flvs was off by a thousand, lol
• synchro/wait shortcut typo fixed
• inbox and other icons moved to background in fullscreen
• fullscreen makes less db queries r.e. ratings
• a couple managementpanel resize issues corrected
• a couple drawing/flickering issues corrected
• some updates to help
• I made a forum to my site, and added links in the client and help

• version 53

• several terrible ratings filter bugs fixed
• ratings filter can now rate files internally
• ratings filter frame now has a statusbar
• system:rating:service=uncertain added
• upgraded to wx2.9, which included a ton of bug fixes
• sort by namespace wasn't working due to an odd typo
• collect by logic changed a little so that 'none' values are collected
• the shutdown lag from v52 is corrected

• version 52

• ratings filter prototype is done
• bit of help docs
• more intelligent cpu burn on shutdown
• dumper can manage flood time error, and recover from it
• whole load of work done on mappings optimisation, meaning mappings updates, uploading mappings and approving petitions work about 100x faster!
• forgetting pending uploads now throws up an ok/cancel dialog
• had another look at overwriting deleted tags

• version 51

• dumper deals with failed uploads better, although there is more work to do.
• flv support for client and file repo!
• reworked tag services' permissions calculations on thumbnail menu
• new fatten A/C cache on shutdown
• system:untagged works again for null tag repo ('all known tags')
• advanced import options now default no limit for size and resolution
• ratings service now have a little info on the review services frame
• new shortcut keys (ctrl-b and -n) for animated gif previous/next frame in fullscreen
• added thoughts on plurals to tag schema help page
• hentai foundry downloads now interleave pictures and scraps
• removed db locks on file/thumbnail reads
• you can open the page picker frame with the mouse by going view->pick a new page
• the copy->whatever thumbnail menu is more concise
• system:hash now works for null file repos ('all known files')
• there is now a title: regex on the tag regex dialog
• servers now store updates in a folder, not inside their dbs.
• couple misc changes to help pages

• version 50

• 4chan pass added
• captcha timings in dump are better
• dumper's file ordering is less insane
• some general updates to regex tag dialog
• new box on regex tag dialog for tags for single files
• regex tag dialog now supports arrow key up and down to change repo
• entirely new and vastly improved listctrl class
• collect by is now a thousand times more complicated, but also much more powerful
• the top-left media-flicker display bug in fullscreen is fixed
• some logging improvements
• some tag cleaning improvements
• thumbnails have less cluttered tag headers
• some small changes to help
• fullscreen precache doesn't do gifs any more, which should make things smoother
• got rid of the awful view pending uploads dialog. I'll improve it later
• started remote ratings, but its not done yet

Changelog <49

• version 49

• local rating like service
• local rating numerical service
• ratings dialog
• ratings drawn on preview and fullscreen canvas
• search by rating
• system:rating in options
• sort by rating
• a ratings help page added
• custom shortcuts
• overhauled shortcuts code
• a new, comprehensive regex menu in the regex tag dialog
• extensive new hentai foundry search and sort options
• a new option regarding the search style of tag A/C entries
• thu dumper can now dump with tags
• regex tag dialog selects tag A/C on page change
• regex tag dialog's button now now shows in import dialog even if there are no remote tag dialogs with POST privilege
• A/C is slightly faster in all cases
• preview canvas now updates on a content update
• numtags = 0 and numtags > 0 now work for pending tags
• shift- or Ctrl-Click no longer deselects
• listbook renaming and resizing bug fixed
• collections thumbnail is fixed
• sometimes, an empty tag, '', could be entered into pending and screw a lot up
• sometimes, a null tag could be entered into pending and screw a lot up
• advanced tag options has better checkbox defaults
• some help screenshots updated
• the db's processing is slightly rebalanced
• the db's startup is slightly rebalanced
• a misc gui bug was fixed
• a misc gui typo was fixed

• version 48

• New copy path and copy url menu entries on thumbnail right-click menu
• Gif scrollbar
• Hentai Foundry parsing errors fixed
• Hentai Foundry 404 errors fixed
• All the html parsers are rewritten using BeautifulSoap
• Flash now zooms full, but not fit
• Filter is now F12
• Help is updated in a number of ways
• Up/Down on empty input to change repository in manage tags dialog is fixed
• The servers now support OPTIONS requests
• Several status typos fixed
• I started local ratings services

• version 47

• Hentai Foundry interface is done, for both artist and tag searches
• A/C gui reorganisation
• A/C file and repo buttons added
• A/C cache count alterations to match those buttons
• A/C tag processing is cleaned up
• Search and thumbnail-action improved to support null file service or non-null tag service
• system:numtags=0 and system:numtags>0 optimised to be much faster
• Ctrl+I now switches searching immediately/waiting
• z now switches between zoom full and zoom fit in fullscreen view
• advanced import options now has exclude already deleted files checkbox
• Thumbnails choose the way they access the db more intelligently
• ManagementPanel colour is changed, hmmm
• Some DB cleanup
• Some general code cleanup
• Some download logic improved, including timer
• Some download status improved
• Download 404 notices better
• DeviantArt now initially shows 'artist username' in its searchbox
• Some code nomenclature improved
• I've made a changelog.html, compling all changes together.

• version 46

• DA parsing CLEAR
• rule34@paheal parsing CLEAR
• tbib (another booru) parsing CLEAR
• Canvas rearranging
• Preview now shows tags
• The autocomplete tag write dialog will promote existing exact matches to the top of the list, even if it is not the most popular.
• Autocomplete lists now scroll with the mouse.
• A/C scrolling is a little more accurate.
• Pressing up or down on an empty A/C entry will change tag repository.
• A sledgehammer has been applied to the python garbage collector.
• The code that navigates boorus is better.
• The import queuing/pausing/cancelling system is better.
• Booru tag-processing is better.
• The import-queue gui is slightly better sized on small screens.
• The new local tags service now shows up in advanced tag options.
• Even more flicker is removed from the main thumbnail view, this time when clicking on whitespace. (it was a stupid bug)
• Serverside db-backup now includes the WAL file.
• Manually adding an admin service is fixed.

• version 45

• Local tags: DONE
• Thread watcher: DONE
• Fullscreen now shows tags
• Preview shows some more file + tag info
• Listbooks load their panels on demand.
• Local options dialog is now a listbook.
• Some incorrect 'sample image' downloads from danbooru are fixed. Danbooru's inner workings remain a 404-cluttered mess though.
• Some remaining utf-8 html parse errors (where a limited content-type header is given) are cleared up.
• Another connection problem (a certain case of the server closing the connection) is now properly recovered from.
• Some flicker in the thumbnail view is reduced, but my slow laptop still shows some, so I need to revisit this.
• Autocomplete tag boxes now have little tooltips better describing their search domains.

• version 44

• All database triggers are removed and replaced with more efficient inline code.
• Cached number of files, namespaces and tags for a tag repo are kept updated, so review services should rarely take very long to load.
• Cached number of A/C counts (current and pending) are kept updated, so A/C results should rarely take very long to load.
• The way A/C cache searches are done is improved.
• The way A/C cache entries are calculated is vastly improved.
• The way several db SELECT DISTINCT * and SELECT COUNT( DISTINCT * ) queries are performed is improved.
• The custom httpconnection object is improved to better deal with server-side connection closes. (Cannot connect to server problem)
• The custom httpconnection object does http redirections more intelligently.
• The custom httpconnection object now parses non-ascii content-type (e.g. utf-8) properly, according to server-side header. The tag "pokémon" will now parse correctly.
• e621 and rule34@booru.org are fixed. (just a couple parsing-variable changes)
• If some tag parsing entries are checked on a booru download page, the tags will be downloaded, parsed and applied even to files already in the database.
• The mouse cursor will remain visible in the fullscreen browser while the right click menu is open.
• I corrected some which->that grammar in a couple dialogs.
• Auto-repo-setup (from the help menu) was broken in a very stupid way, but I fixed it. I apologise for not noticing.
• Thumbnails now show creator and title up top. They are a bit crushed, so I may rearrange them.

• version 43

• Updated import + booru code
• New booru buttons
• Cut tag parse bug fixed
• Download preview instead of real file bug fixed (I think)
• Fixed the 'thumbnails not appearing on import' bug
• Import flickers less
• Media Classes rewrite is done, although I'm not totally happy. Have to think about it.
• A problem with pending tags applying over existing tags no flushing correctly is fixed.
• A/C cache is better, but still a little slow on slow computers.
• system:duration=0 is fixed.

• version 42

• Autocomplete tag cache DONE. Please tell me if you encounter excessive slowdown or incorrect counts.
• Autocomplete tag logic improved in about five ways. Tags should appear more intuitively and with more accurate counts.
• Copy and paste buttons in manage tag dialog to facilitate spamming many tags from one file to another.
• Imageboard dump is fixed!
• URL download got fixed last week and I forgot to say!
• A peculiar pseudo-foreign-key bug with the active_pending_mappings table is sorted. (leading to more accurate queries and counts)
• Since display of large queries is so much faster, the 'woah! 12,000 query results' warning is gone.
• Fullscreen precache timings are rebalanced, hopefully for the better!
• The importing gui classes are a little smoother, and thumbs should load one by one, not row by row, once the page height is exceeded. The scrollbars should be less janky.
• Page transitions are smoother.
• Importing from URL will now load those urls that are already inside the db (hella fast, too), just like importing from hdd.
• The double-update-download bug is finally fixed! Now you'll download update number 258 only once, and not twice in a row. (I hope!) It never mattered much, beyond a little CPU, but it annoyed the hell out of me. Working out what was wrong took ages. (Long story short: Python's Priority Queues are not parallel FIFOs!)
• There was a weird bug in pending tags sometimes going to higher tag-precedence services, rather than the one chosen in the manage tags dialog.
• There was a typo in reset service.
• More content update unification in the db, making things much neater behind the scenes.
• I found a potential hang in the db-shutdown code.
• I found another potential hang in the db-shutdown code!
• The number of messages in your inbox is now displayed on the statusbar.
• Bold, Italic and Underline now work with Ctrl+B/I/U in the message compose panel.
• The 'don't receive messages from Anonymous' checkbox in Add, Remove and Edit Services for message depots now does something.
• system:unread in message search now has a count.
• A draft will remember its saved changes when you click on another conversation and back again.
• You can now save a draft with zero recipients.

• version 41

• Rich Text Control with Toolbar in messaging.
• Message unread/read status now syncs to message depot.
• Can set defaults for file system predicates.
• Animated Text class written.
• system:unread tag in messaging.
• Import bug fixed.
• Messaging search logic improved.
• Message depot default check time down to 180, and daemon thread improved to manage with that.
• Application startup is more stable.
• Some AddMessage logic improved.
• Drag+Drop service fixed and improved.
• Tags added while importing will now turn up in autocomplete.
• I unified more of the client's content-update db calls. Just making stuff neater.
• I massively simplified the autocomplete tag code in preparation for another bash at doing autocomplete cache. A/C is far too slow atm, imo.
• Some A/C counts are still off, particularly when there are a mix of current and pending. I couldn't figure out what is causing it, but I'll look at it more as I do the cache.
• In add, remove and edit serivces, you can now specify if you want to receive messages from Anonymous or not. Unfortunately, because I forgot to implement it in DoMessageQuery, this checkbox doesn't actually do anything yet! Whoops!

• version 40

• Inbox/Archive for messaging works. The inbox icon is back, and F7 to archive as usual.
• Unread/Read for messaging works. Click on your status next to a message panel to change.
• All message participants are shown in the conversation listctrl.
• The message page will open with 'system:inbox' active, ordered by newest message.
• I fixed conversation listctrl sorting in a couple ways.
• Some bugs in the db->gui update reporting system are fixed, so the compose window and draft panel will close properly on a message-send, and the conversation listctrl will swiftly update unread counts and inbox status and so on.
• The draft panel's save button wasn't updating sometimes when it should have been.
• html links in messages will show in the main frame's statusbar and launch in your browser when clicked.
• I reworked how many system tags are searched in file queries to make them muuuuch faster. Now, any time you go system:hash, :size, :mime, :age, :width, :height, :duration, it (should) be significantly faster than before.
• Ever have a png file that had an ok thumbnail but a black preview/fullscreen? It was probably an 'LA' image, or 'greyscale with an alpha channel'. Display of LA images is now fixed.
• Streamlined the way file 'archive' and 'delete' commands are sent throughout the application, with an eye to eventually adding all other user-commands to the same system (to make it simpler, and also to do undo-redo).
• I've updated the help docs, including the section on how messaging actually works.
• You can now enter system tags with enter/return from the keyboard, not just double-clicking with the mouse.

• version 39

• A whole bunch of stuff I didn't write down that generally made messages work beyond sending "Hello World".
• Unicode messages now work.
• Newlines in messages now display properly.
• Fixed contact saving and drag-and-drop to the manage contacts dialog.
• You can now add a contact using a contact address (assuming their service is v39).
• A bug in contact renaming is fixed.
• My contact (hydrus admin) is now added automatically.
• Messages are orphan-deleted properly server-side.
• Message depots you control now get an entry under the admin menu.
• The conversation listctrl is now properly sorted.
• The HtmlWindow that shows message body is now resized to its minimum needed height via magic that I ain't gotta explain.
• Messages that fail to send can be retried by clicking on the failed.
• Some draft save/send button enable/disable logic was updated.
• I switched around some server-side account logic.
• You can't zoom a flash window bigger than the screen.
• I've made the mouse appear in fullscreen when you move it, and this includes while you drag. See how you feel about it. I'll make options to change behaviour if people care.
• Better zip structure. (No more CWD rubbish)
• New tar.gz release with only the code and help. Code is 1.2MB, help is 14MB lol. I'll improve the structure of this release next week. I threw it together by hand today, but I'll improve its structure next week.

• version 38

• messaging v1.0

• version 37

• messaging prototype

• version 36

• animated gif fixes
• new page chooser
• svcp->cstvcp in fullscreen too
• new import progress
• just start the server on this computer, please
• I think I have scrolling to the current selection after an archive fixed now.
• Started an experimental simpler way to send commands to the db, which may open the way for undo-redo.
• Capitalisation option in file->options->gui. It only does the menubar for now, but I will expand it.
• Custom sorts/collects. Under file->options. Check it out, play with it if you care about it.
• Some program initialisation error handling is cleaned up.
• Can now no longer start a server if the admin port it wants to use is occupied. (can't run two servers at once on the same port)
• I broke dump with a typo in a previous release, I think! It is fixed now.

• version 35

• Slight speedup of review services frame loadtime.
• SVCP now replaced with SCTVCP and CSTVCP for now, with more planned next week.
• Repo sync daemon adds db jobs less manically, now, locking up the gui less.
• Repo sync daemon reports successful updates to the log.
• Tag petition review page has a visual bug fixed.
• ThumbnailMediaPanel manages its underlying bmp canvas more intelligently.
• Some ugly page-closing code was cleaned up.
• There is a 'regex examples' popout window on the regex-filename-tag import dialog, with copy-pastable examples of python regexes that'll work for common filename-parsing situations.
• Home and End keys now select the first/last thumbnail as well as scrolling.

• version 34

• DB consolidation, and foreign keys and triggers with it. The db is less prone to orphan ids and miscounts.
• For long-time users, all orphan ids should have all been deleted, reducing weirdness.
• Total size of local service is now tracked accurately.
• Archive while importing should work a bit better now.
• Update logic should be more reliable, as should all service_updates.
• Archive count for file repos should be less wrong.
• Fixed two (!) typos in system:age predicate code.
• File upload is fixed (not sure if it was ever broken in release, but I fixed the stupid thing I did anyway)
• DB (and server) errors should report better now, revealing inner-db code in the traceback. (and printing it to the appropriate log)
• Improved the mime_to_enum object for image/jpeg, image/jpg, image/jpe, image/wtfaretheygoingtocalljpegnow.
• HTTP connections can deal with some esoteric commands, like 301, without throwing a fit. All http goes through the same code.
• Danbooru should now be in your booru list, and it _should_ work.
• Pausing/Cancelling a booru gallery parse should now work.
• 4chan dumper now understands captchas being wrong and the thread not existing; in the former case it'll let you put in a new captcha and try again, in the latter it'll stop the whole dump.
• Some general code neatness improvements.

• version 33

• Boorus are working for users who started last week
• The server can now manage many more simultaneous requests without bugging out.
• The early freeze/import freeze due to simultaneous access in the client is fixed.
• The client's file+thumbnail server (details in help's advanced.html) works again.
• Client DB now stores md5 and sha1 for all local files.
• Client DB has a larger cache size.
• The freeze caused when importing thousands of files at once with tags is fixed.
• The yaml imageboard import (when you drag and drop a .yaml file onto the imageboards dialog) now adds form fields correctly! If you had problems here, check out install_path/static/default_imageboards.yaml to reset them!
• The autocomplete counts should be more accurate, particularly when there are a mix of current and pending tags in the db.
• The autocomplete in the regex-path dialog will now search the db correctly (it was misconfigured).
• Autocomplete is also just a _little_ faster
• File imports are now low priority, so gui activity should still happen nice and quick even while an import is occurring.
• The review services frame will now show how long until the next update will be checked (see picture above). In case of error, it will show how long until it tries again.
• The review services panel and manage services panel are of a better default size, and they use my new listbook class exclusively, rather than the old notebook.
• I've added some tooltips on not-obvious widgets.
• Content Updates have vastly improved processing speed on a thumbnail view (x100 in some cases!). Tag updates particularly will not be so slow to process if you have a large search open.
• The import gauge will now show the correct range according to the current queue size, not always the default 100.
• Active Import Pages will throw up a yes/no dialog if you try to close them.
• Some beautifying of code syntax. '\n' is now os.linesep, which means exported stuff should show up better in notepad.
• Some changes in network protocol, although now I think of them, I realise I forgot to update the network version. They aren't important for most users, so w/e.

• version 32

• client db serialisation DONE
• Cleaned up a dirty-bmp canvas init issue.
• Resized some dlg stuff that was too tall
• Cleaned up a bunch of unrelated db code
• Tightened the pubsub system so it integrates better with wx.

• version 31

• booru prototype
• Fixed some problem with last error timestamp
• Fixed a bug in repo bandwidth tracking
• Fixed URL registration with the client db.
• Made downloading URLs asynchronous again
• Made repo synchronisations not block so much, and not for nearly as long. You should be able to do queries and click on thumbs (albeit at a slower rate) while the client is downloading and processing updates.

• version 30

• Thumbs are exported, and, along with perceptual hashes, now treated as a global pool, reducing downloads and speeding queries.
• Thumbnail resize is now asynchronous, done as each thumbnail is requested. Changing the thumbnail size in the options dialog happens real quick, and if you don't like a particular size, you can change it back just as fast.
• Temporary DB locking mechanisms ensure serialisation, less inefficient locking.
• ImportPage superclass works a lot better, in a lot less code. DB import code is now even simpler, and much faster, especially for redundant files. The new import page will also tell you how many files were successful/failed/already deleted/already in the db.
• Fetching images for fullscreen or preview is fully asynchronous, and will no longer block the gui thread.
• Some query logic is improved.
• You can now import and export sites/imageboards from the manage imageboards dialog. It'll export to a .yaml file, which is easy to understand with a decent text editor. Just drag and drop onto the dialog to import.
• Selecting thumbs has a little less preview-flicker now.
• The new tagboxes now render numbers prettily, so 4123 -> 4,123.
• (+) namespace:tag now colours properly.
• The background for selected tags is now an appropriate colour.
• You can now choose the colours of any namespace you like in the normal options dialog.
• Clicking in whitespace in the new tagbox deselects, rather than select( 0 )s.
• Spacebar, PageDown, Return and all the other special command characters are now propagated to flash in fullscreen view, just as long as the mouse is over the canvas.

• version 29

• Big changes to the db's file storage, as above.
• database->vacuum is reinstated, database->reindex is retired. I recommend you run a vacuum once you are updated; it'll only take a minute now.
• Rollback and initialisation bugs are fixed.
• There's a splash window now that can offer client-boot feedback.
• The way scrolling-to-focus is calculated is improved.
• Pretty much everywhere you see tags will now show coloured tags! I finished up my own tagbox class, which for now will show tags coloured in the default booru scheme. It also loads in a few milliseconds, rather than the ~0.5s of the default listbox. It scrolls faster, too. I'll have user-customisable colours next release.
• Files with invalid resolution, size and mime are now filtered out before dumping to an imageboard.
• A number of misc other bugs were fixed.

• version 28

• I fixed the 'searching immediately' bug. It was a stupid typo as usual.
• Post-Filter scrolling should be a bit more reliable.
• Some more log events.
• I did a lot of work on my new tags box, although the actualy gui elements will have to be finished for v29.
• Manage Imageboards wasn't saving properly. It is now.
• My new progressdialog happens in several places. It is a little neater than the one in wx; just better for my purposes.
• I reworked some of the transaction stuff in the db so that I can atomically queue pubsub events. (this means file import are less juddery)
• After becoming very annoyed with wx's listbook/labelbook/whateverbook, I wrote my own. It's all over the place in my dialogs now. It is tighter, more reliable, and can scroll.
• I reworked some of the way sets interacted.
• There was an infinite recursion bug sometimes, when uploading files to a file repo.

• version 27

• Cleared out some old unused library imports, so the client is slightly smaller.
• Had a cyclical import (!). Luckily, python is forgiving, but I have still fixed it.
• Collection thumbnails will now update with their new pending tags correctly.
• Some wx.Panel vs wx.Window issues and nomenclature resolved.
• Full Page decouple. Now the window to the left is a seperate, much neater class. It and the media/preview windows talk to each other in a clever way that means I can make new kinds of page much faster. It scrolls, too.
• Thumbnails resize will tell the thumbnail window just to redraw itself, not refresh the entire query.
• 'system:local' and 'system:not local' are now mutually exclusive (will swap each other out, rather than overlap), like inbox and archive.
• A little neatening of the system predicate code.
• 'system:similar_to' RMB menu entry fixed for some cases.
• There is a new log window. Go view->new log page to see it. It doesn't show much right now, but you can see synchro successes and errors pop up as they happen.
• There is now a popup button on the import dialog and the URL downloader page that does 'advanced import options'. You can now tell these import processes to archive everything and/or ignore files below a certain size or resolution. By default, the client will not import anything <5KB or <50x50px. You can set no limit.
• Flash windows will now own all keystrokes as long as the mouse cursor is over the flash. If you are browsing/filtering, just move your mouse over or off the flash window to make sure your return/page down/spacebar/escape/whatever is interpreted as you want.
• The URL downloader will not download the same file twice in a row, so you can add a thread URL several times in a row, and it'll only download the newest files. The behaviour of this is affected by the 'exclude deleted files' checkbox in the client's options. I'll implement thread watching a bit later.

• version 26

• The double unnamespaced count bug ( 'system:blah (70)', 'blah (140)' ) in autocomplete should be gone.
• Some rejiggering of how the statusbar updates means active import windows can't spam the statusbar when they don't have focus.
• The count of pending tags is more accurate and uniform. When it says (+3), it means for three files, not in three cases.
• Tag counts overall are better. When it says 'diaper vore (17)', you should be getting 17 files if you enter that search.
• GUI tag repo precedence is slightly more accurate, compared to what the db thinks it should be.
• I nailed down why system:mime wasn't working sometimes, and now it should work all the time.
• I discovered some problems with repo-side bandwidth tracking. I fixed a couple bits, but have notes to look at it more.
• In v25, if system:limit was greater than the search population, it threw an error! Whoops! Fixed now.
• The post-filtering thumbnail problem is totally fixed, as are a couple other instances when thumbs would sometimes not draw.
• The fullscreen change-image-jittering-around problem is fixed, I think.
• Fullscreen archive works again. Not sure why/when it broke.
• Importing with archive checkbox ticked sometimes didn't archive, lol.
• A bunch of decoupling of gui elements, making for much neater code and less gui blocking.
• Decoupling of import from hdd as well.
• Query is now asynchronous! If you do a massive query on a slow computer, you won't lose the gui! You can go check other tabs, whatever, and the query will process in the background, rendering on the page you started it when it is done.
• A number of cpu-heavy numbers are now cleverly cached, so system predicate fetch happens much faster, especially on 100,000+ image dbs.
• Some 'collect' code is sped up and made generally less bonkers.
• I wrote a couple new classes for my most common gui elements.
• There is now an option in file->options to change the number of characters before autocomplete will go search the db for tags! If you have a slow computer and a huge db, you can set it to 2, 3 or even more chars. Default is 1.
• Dialogs that ask you questions will now usually center on their parent window.
• The current selection tagsbox won't show ( 'blah (12)', 'blah (+3)' ), it'll combine into 'blah (12) (+3) (-1)'. With the + meaning pending and the - meaning petitioned.
• Autocomplete now respects which of the include_current and include_pending buttons are green! It'll only search current/pending as you wish! And the numbers will now be accurate!
• My new question dialog with the yes/no buttons hooks the Escape key to mean no.
• I don't use it much yet, but I have a new message dialog too.

• version 25

• Thumbnails should load just a bit quicker.
• Made some dict + set code neater
• A new end-of-filtering dialog makes the choices a bit more obvious
• When you exit from fullscreen/filter view, the main thumbnail view it was based on should select (and scroll to!) where you were. But filtering is less reliable, since you may have just deleted the last thing you were looking at!
• Better scrolling in thumbnail view
• Scrolling in thumbnail view will follow the current selection, so the focussed media will always be visible.
• Double right-clicking in filter now deletes instead of keeps (typo)
• Slightly less flickering when processing repo updates
• Made some form code neater
• A lot of the thumbnail gui code has been decoupled and pubsub'd. It is all neater behind the scenes, meaning I'll be able to extend it easier in future.
• I completely overhauled how my pubsub works. (pubsub is a messaging system inside the code). It is more flexible and easier for me to work with.
• system:mime was not working in v23, not sure why. It's fixed now.
• Some small speedup + neatening in db queries.
• You can now scroll to the end of a thumbnail page by pressing End. WARNING: If you have like 10,000 thumbs in the current pane, it is going to have to load all those.
• In the AutocompleteTagWrite object, which is the tag entry box in the manage tags dialog, there was a stupid bug that made many requests more expensive. It's fixed.
• Autocomplete now works off one character's input.
• I've swapped a bunch of expensive objects with easier staticbitmaps for now
• some A/C ordering changes

• version 24

• imageboard dumper prototype
• some db update stuff

• version 23

• A typo was disallowing petition rescinds in certain cases.
• I've rearranged some of the RMB menu separators to reduce filter/archive misclicks.
• Tag pending upload has more accurate GUI response.
• You can now find similar images from RMB menu.
• Petition approvals are mirrored on an admin's client.
• The gui now accepts that admins can overrule deleted content.
• Couple tiny URL downloader gui bugs fixed

• version 22

• system:similar_to
• Some backend namespace stuff
• Couple of gui presentation fixes
• I removed the clunky inbox and archive buttons, see how you feel about it.
• I tried to do ultimatelistctrl to have coloured tags (like a *booru), but it was too buggy and slow for the client's purposes, so I'll probably have to write my own control.
• system:mime exists, now. Not many mimes, yet, but have a play with it, see if you like it.

• version 21

• On opening the URL downloader, the textbox input focuses.
• The URL downloader has some better update texts. It pauses nicely, etc…
• The URL downloader now orders urls in the order they are parsed. I forgot I was throwing the links in a set, losing all order!
• You won't see dupes in the URL downloaders' mediaresultswindow.
• Accounts should now correctly refresh on version update.
• When accounts fail to sync, it dumps the error in the log.

• version 20

• The preview pane should be a little less schizophrenic.
• Some code cleaning up behind the scenes.
• Thumbnail download optimisation if they already exist locally.
• Icons will show/hide in fullscreen with mouse movement, just like 21/123.
• Weird archiving bug on import is fixed. It was display only.
• Manage tags dialog now works in fullscreen, lol.
• system:not_uploaded_to: now works
• services->review services now loads muuuuuch faster (after the first time)
• There was a typo in setting up initial server's accounts. Email me if you can't make new admins.
• Importing is more streamlined, with more shared code.
• Importing with tags dialog happens more naturally in the import process. You can now do it from drag and drop imports!
• I reordered the default collection dropdown. Your default may be changed, so check your options.
• You can now fetch an account by a deleted file (hash).
• Thumbnails' tags now obey tag precedence for deleted stuff. I hope.
• Modify Account Dialog now does its stuff.
• Collections thumbnails now show how many files are in the collection, just next to the icon.
• I've added to and edited the help extensively, replacing screenshots and updating the text.

• version 19

• drop librarium nomenclature
• More intelligent prefetch in fullscreen - things should load muuuuuch faster
• A little more intelligent sum() calcs in autocomplete - non-namespaced tags now sum all their namespaced versions as well
• More rigorous client db transactions, just to make things more pretty and reliable.
• Moved db files to /db, logs to /logs
• Made inroads to getting review services to load faster, but version 20 will have more here.
• I've updated the frame view so the file info is draw on top of the image. It's a hack for now, inside the image-frame, but it is the only solution for wx's annoying 'no overlapping' policy.

• version 18

• Some misc gui image info bugs fixed
• Review services dialog is less pubsub buggy
• Fullscreen image prefetch does more and hopefully more intelligently
• Entering a blank tag to quit tag dialog was broke when I updated autocomplete textctrls. I've fixed it.
• Petition page is fixed, so I can finally clear out the tag petitions!
• F10 was broke when the lib had deleted files lol! Fixed it.
• Review services is more responsive to repo permissions updates.
• Flash files are now correctly put in the inbox on import.
• Synchronise Repositories daemon no longer takes ten gui-blocking seconds to process every update! This was an annoying slowdown, but I figured out what it was and reduced it to <1s with a quick patch. The problem will return with private tag repos, but I'll deal with that when I come to it.
• Synchronise Accounts daemon has some better pubsub so it fires more intelligently. This in turn affects several other daemons, which should all be more responsive; e.g. when your permissions change, your pending thumbs/files should start downloading.
• Improved delete orphans logic for the fiftieth time! It is great now, and won't ever bug you in the gui; it runs invisibly on shutdown and usually only takes a few secs to do its job.
• Some of the gui updates triggered by the daemons are optimised so they block the gui less. Multiple stuff is bundled into single updates, those updates are less buggy, and the gui catches them better. Resetting the public tag repo will no longer lock up a slow system.
• A few versions ago I changed the way repos were registered with the gui, and I've fixed some of the resultant daemon-related errors. This mostly affects admins, which mostly means me!
• Download Thumbnails daemon was getting pretty slow for large dbs (100k+ files) because of an inefficient db query. This has been cleared up.
• I banged my head against the wall for a couple of hours getting nested notebooks to work, and I am now a better person for it. Review and Manage Services dialogs feature better panel management as a result. Their panels are also much richer, reporting loads more data about the respective service. You can now see how many mappings the public tag repo has without having to delve into sqlite db browser.
• I have unified panel and frame background colouring away from wx's bonkers idea of what things should be coloured. It should mean that everything is properly grey, but please tell me if your gui turns all black or transparent or whatever.
• 21457 is now 21,457 in pretty much all cases where it is appropriate.
• Fixed a bug with wx gauges not liking large ranges because of signed short or whatever underlying var the C code was using to store it. (That should have been done at the wx-python level, but whatever.)
• 'localhost' resolves to 127.0.0.1 when editing a service, mainly because windows freaks out at the former, deciding to take a couple seconds longer to make any raw connection.
• Dialogs and frames have better, multi-monitor-friendly positioning.
• Flash files are loaded from a temp folder beneath the install dir, rather than lib's server, for various reasons. This temp dir also manages file copy/paste operations. I may move this back to the OS's approved temp folder, but we'll see.

• version 17

• Tag negation "-tag" works again
• You can now cancel a filter on the first image
• Some weird filenames that were breaking the new import page no longer do
• Autocomplete (either when searching for tags or adding new ones in) will now show the number of files tagged with that tag after the tag e.g. here
• Manage tag dialog now pops up slightly to the upper left, rather than the center (default) of the screen, so you can see what you are tagging. I may make this more intelligent in future, so it remembers where you last left it.
• I changed a couple small things in help regarding the new import.

• version 16

• Numtags system predicate now takes into account which of current/pending tags buttons you have active.
• Thumbnail downloads fire more reliably.
• Downloading a file was causing funky thumbnail draw, which seems to be fixed.
• Delete Orphans daemon only fires on application close now, so post-filter db lag is gone.
• Daemons now use events, not conditions! lololol, I'm an idiot who can't scroll down a library doc. For the longest time I was like "I wish 'threading' had events!" and it does.
• Some account checks were missing in tag upload, but are now there.
• A gigantic db rewrite that altered pretty much every table and query.
• Calculation of tags for current selection now support collections, and is fast.
• Was deleting a repo broke in 15? If it was, sorry! Didn't catch it, but it's fixed with the new system now.
• Pubsub can do more, and is a little less 'argh, I think it works like that', and more OOP
• The way tags are held by a thumb can support multiple tag servers (but only does public tag repo for now)
• Repos are now a subclass of 'service'. I'll do more on this in the next version, including some nomenclature changes in the gui.
• Imports are now asynchronous! There's a new import page that'll open on any import. It's a little jittery, but it all works. If you import 3000 tiny files, your gui may lock up for a few secs since it'll be able to import faster than it updates the gui.

• version 15

• manage tags dialog focus bug hotfix

• version 14

• Selection's tags box now shows collections' singletons' tags
• Autocomplete now sources from the current thumbnails whenever it can (faster than db), and does so intelligently
• Couple of small display bugs fixed
• Some delete logic approved
• Repos do some more efficient inserts on POST content
• Petition optimisation, for auto-soft-approval before update gen
• Dialog parents, dialog parents everywhere
• DB cleanup of a couple of old methods
• Autocomplete Write (when adding tags in manage tags dialog) now sources from the entire network's tags (not just for the files on your hard drive) when you type enough letters to stump it. i.e. If you have no files tagged 'evangelion', but you do have a file tagged 'eve', it'll show 'eve' if you type 'ev', but if you type 'eva', it'll requery the db more widely, and give you 'evangelion'.
• system:ratio for searching for wallpapers or whatever. Combine with system:width>x for extra utility!
• You can now hit F3 (manage tags) in fullscreen, whether browsing or filtering.

• version 13

• a couple of cpu-intensive DAEMONS schedule themselves better. (less slowdown after a big filter)
• image rendering prefetch improved
• smoother image rendering to screen (especially when zooming), less flickering
• petition approval has smoother, less buggy workflow
• new hashes and tags are added to the db muuuuuuuuch more efficiently. most public tag repo updates happen in a second or two, rather than 30+
• some of the internal commands are streamlined together into a single class
• ctrl-s fixed
• clear pending file repos pubsub fixed
• ctrl-r now shows/hides splitters
• thumbnail delete logic is better
• select logic is better
• deleted_remote_files bug fixed
• headache-inducing weakref repositories __eq__ bug in pubsub fixed
• 'with' keyword all over the place now. turns out wx has a bunch of objects with support it (including dialogs!)
• gridsizers are now all flexgridsizers
• the list of tags on the current selection now has a weight attached like 'flower (23)' to show how many of the current selection have that tag. it doesn't support collections' weight yet.
• revised some ambiguous repo error messages
• you can now have a main window with no pages open
• right click menus on a selection are vastly improved (lots more info), but still a way to go.
• on the import files dialog, you can now specify to auto-archive rather than send to inbox
• thumbnails now know where they are uploaded/pending/petitioned to and will show that info with different icons
• escape key now deselects
• manage repos dialog is completely revised. comments appreciated.

• version 12

• You can now tell lib to search 'current tags' only, 'pending tags' only, or both.
• Tag search is more efficient.
• Search in general is more efficient.
• Search code is more beautiful and decoupled.
• Tags are stored in memory more beautifully, and a similar class holds repositories for thumbs also.
• Age predicate is fixed (twice!)
• Server code is more beautiful. DB-Server interface is decoupled into a mixin class to make it easier to understand and manage.
• Icons for pending and petitioned files, and for 'currently downloading'.
• Prefetch fixed, so nearby fullscreen images load just in case you want to see them.
• Fixed a redundant pending tags bug
• Fixed a swf tag petitions bug
• On boot, temp folder is cleared.
• The 'current tags' window is now union on the current selection, not intersect. If nothing is selected, it shows all the tags of the current selection. In a later version, I'll order this list by count-weight.
• All (I think) tag updates are synchronous-ish with the gui. As updates come in from the public repo, thumbs will update themselves appropriately. Same goes for repositories.
• Gui-code tidy-up
• On deleting a local file, preview clears.

• version 11

• swf support, lol
• You can now backup repositories from the admin menu. Info in help.
• Under the help menu, a quick way to set up the beta repositories' credentials.
• Account sync was full of bugs, and is now less so.
• When you go fullscreen, the preview window should go blank, which should save a little cpu time when you are looking at gifs and your ears when you are looking at swfs.
• Auto-complete is muuuuuch faster than it was. Also more correct. Still more work to do, though.
• Librarium hosts an http server on 45874, which supports GET /file and /thumbnail just like a file repo. Utterly prototype, but you can test it right in your browser if you like. Go http://127.0.0.1:45874/file?hash=whatever when lib is running.
• Ctrl-T opens a new local page/tab, Ctrl-W closes the currently active
• Image display in preview and fullscreen has been revamped, and is now both a little buggy and much better.
• Bandwidth tracking both server- and client-side should be more reliable.
• Delete lag (the slowdown after you boot/delete something) should be reduced.
• Query lag is significantly reduced, and new operators for predicates have been added. You can now go ≈ on a bunch of values like size and width and duration to get 'approximately equal to', which actually means plus or minus about five percent.
• A stupid error on file upload is fixed. If you've been having problems seeing the files you uploaded in preview, try resetting the respective file repository.
• A few daemon bug fixes.

• version 10

• hotfix for tag updates

• version 9

• Pressing a cursor key when no thumb is focussed no longer throws an exception.
• Hitting enter on a blank input in the manage tags dialog now quickly oks the dialog.
• Annoying splitter exception on import is gone. It was a one-line fix, because I was being stupid.
• All searches should be a little faster. Mostly effects searches with 1000+ results.
• Modal dialogs (in particular) should be just a little more responsive now. Less stuff is being dumped on the wx gui thread, and all db handling is more parallel.
• Slow petition approval fixed, made almost instant.
• Image rendering has been revamped. It should be less stuttery now, but there is still more work to do.
• Some pending tag update logic was fixed.
• If the focused thumb goes from 'not local' to 'local', its preview will load. (its on my to-do to do the opposite: clearing preview on delete in remote search.
• DB daemons rewritten completely, so now a whole bunch of stuff flushes more efficiently, more reliably, and with less gui-blocking. Things happen smoother now.
• Can now create repository database backups via librarium. Sending the command (via the admin menu) tells the repo to lock and copy its .db files to .db.backup. I'll update help to explain this more.
• Can now copy files from within librarium to the clipboard! Just select from the right-click menu or press Ctrl-C, and your selection will be copied to clipboard. You can then paste to any directory, for quick export to anywhere. The export command remains for exporting thousands of files.

• version 8

• Fixed a missing transaction on repo db update.
• Export path can now be on other hard drives. You'll probably want to keep it on the same drive if you have lib on a usb stick, though.
• Fixed a really stupid '<' for '>' typo that was causing a ton of gui blocking and repository slowdown.
• File and thumbnail downloads are much snappier.
• One person had a very odd error while trying to open manage repos. I don't understand what actually caused the traceback to happen, but I think I fixed it nonetheless.
• inbox/archive buttons above the searchbox are now mutually exclusive.
• Fixed the version number in lib's about window.
• Fixed the preview window so it remembers its last size properly. Also reduced its min size so you can make it tiny.
• Fixed the problem people were having with admin rights. See this post if you were having problems.
• The RegisterRepoRequest that was filling people's logs on upload should be fixed now. Turns out I didn't understand how wx.CallAfter works during modal dialogs! I threw in a horrible fix, but it works. I'll revisit it.

• version 7

• db changes are done
• fixed a bug with posting tag petitions
• when typing tags in, autocomplete now harvests from pending tags as well as current
• the regex dialog's tag entry is now autocomplete
• leading/following whitespace removal for tags, and multi-whitespace -> single
• a little server-side pubsub neatening up
• when importing via dialog, there is now a checkbox to ignore subdirectories
• Added a quick readme.txt in the base_dir for people who choose to extract rather than install. (Everyone should check out the help at least once)

• version 3->6 (never released)

• after much feedback, completely rewrite db
• general improvements to gui

• version 2

• general fixes and improvements

• version 1

• first true metadata beta
• shit actually works

• versions 0.91 -> 0.99

• move to python and wx, and an application- rather than browser-based model
• building on concepts
• scrapping p2p, turning to client-server

• early versions of beta

• made db, playing around with LAMP
• an initial p2p framework started