the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select
my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'
when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.
I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control
the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically
fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed
if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'
.
misc:
the client now gives a once-per-boot warning popup if your session size exceeds 500k. for those who cannot reduce session size conveniently, this popup can be turned off under _options->gui pages_
when the file import options prohibit a file due to filesize or resolution etc.., it should now always record that as an 'ignored' result rather than an 'error'
fixed an unusual error popup in thread watcher display that could occur during session load. this problem seems to have been around for a long time, but it required a watcher in a previously saved and still valid 'wait a bit' error state and was only vulnerable for a few milliseconds, so it hadn't come up before. in any case, it is fixed
subscriptions with small 'first run' file limits now work better: if you create a subscription with a fairly small 'first run' file limit (this typically matters when the number is smaller than one of the site's gallery page's worth of results), subsequent normal checks with larger file limits will be more aggressive about noticing that they 'caught up' to that small initial sync (previously, they would sometimes incorrectly think the site just got some files tagged out of order and bump right past that initial 'already in db' batch and keep going until they hit their own file limit)
.
advanced string processing:
added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!
the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it
the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly
the new String Slicer edit UI similarly gets all the strings at that stage of processing
future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage
.
server/client api core improvements:
I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416
the client now understands 416 ("can't do that requested range m8") errors
I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly
.
boring:
fixed up a handful of typo-borked unit tests
fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao
started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful
updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance
updated sqlite dll on the Windows release to 3.34.1
when parents are hidden in the edit/write taglists (e.g. in manage tags), there is now a '(n parents)' suffix
thread watchers that are DEAD or 404 but still have files downloading now report 'working' status until that is done
search terms with ';' like 'steins;gate' should now work in downloaders. sorry for the trouble!
fixed an issue where un-ideal tags were sometimes becoming non-searchable when they were entirely replaced in manage tags with their 'ideal' siblings (i.e. their autocomplete count went to 0). this was due to overzealous deletion in the new tag definitions cache not filtering out sibling/parent chain members. a small routine will run on update to resynchronise affected tags
fixed an issue when loading up files in the main 'import files' dialog where a critical error (as opposed to a nice 'couldn't figure it out, sorry') in mime detection would cause the whole job to hang
that main 'import files' dialog now counts 'missing' files separately in the error count
fixed tags not updating on the filename tagging dialog when double-clicking to remove from the simple taglists
fixed the sort on the manage tags dialog's suggestion taglists--they now preserve their original sort, rather than alphabetising once sibling/parent data is populated
the manage parser test panel now catches all network errors. error data back from the server is presented better, and the traceback is now viewable in a special new button on the network job control
the edit shortcut set dialog now gives a veto text popup if you try to ok with a shortcut set twice (previously, it would ok and merge down to one command randomly). support for multiple commands per shortcut will come in future
entering alt+number in the shortcut entry dialog on windows will no longer spam some errors about 'null character'
fixed a 'this object is too huge' check in the database, which mostly affects gui sessions with millions of objects, to check against 1 billion bytes max size rather than 1GB, as here https://sqlite.org/limits.html (issue #816)
fixed the 8chan.moe parser, which was pulling hash incorrectly. it should now save more bandwidth
updated the e621 parser to pull their (new?) 'lore' tags, which all end in _(lore) and refer to canonical gender and some spice. they come with the 'lore' namespace for now, we'll see how it works out. a user reports these are useful for blacklists
.
new string processing sort step (for advanced users):
string processing objects have a new processing step: String Sorter
this sorter can sort the whole list of strings, either strict lexicographic or 'human sort' that does numbers properly, asc/desc
it can also take a regex for the sort 'key', so you can sample just the number or name you want for sort purposes
content parsers no longer have the 'sort formulae results' controls. any content parser with existing sort has been converted so its string processing object has a String Sort step appended
the string processing UI is still built around single string processing, so the test UI here is essentially non-functional, but you can see the sort happen in the formula test parse panel
I will add a String Slicer in future to sample the list of strings, so you'll be able to grab the top item etc...
.
boring code cleanup:
refactored the refresh call in filename tagging dialog to nicer Qt signals
the add/remove taglists on the simple panel are also moved to Qt signals
and so are 'filter' taglists
fixed some typos in new help text
removed a 'needs restart' string in 'gui pages' options that no longer does
fixed 'unusual character' collapse logic for short text inputs in tag autocomplete lookups. in human, this means typing 'a' now correctly gives you the tag '/a/' and _vice versa_ (issue #799)
to make this work, an old database subtag map cache is revived this week in a more efficient form. if you sync with the PTR, it will take a couple minutes to update. the regen routine is also added to the database->regen menu, in case it ever desynchronises in future
absent an override referral url, api-linked url fetches now use the original url as referrer. previously they were sending no referrer. this fixes watching spicy boards on 8chan.moe
updated a 'get all this stuff' database routine to report more info, and a handful of supermassive jobs (mostly db maintenance regen) now report x/y progress with y, rather than just a nebulous increasing x
fixed an odd bug in a common UI text-clearing call that was causing real text not to show up for a while after the clear. this was most apparent in the downloader highlight panels, where status text on file/gallery/network status could sometimes stay blank until a change
the manage tags dialog's "there are several things you can do" button box when you enter tags in complicated situations is now clearer. there are several sorts of intro text on the dialog, the button labels are clearer, and button tooltips have more action information
fixed the tumblr downloader! sorry for the trouble here, I hadn't realised the situation from some reports. if you have tumblr subs, please go into them and set to 'try again' any recent urls that say 'Found 0 new URLs.'
.
taglists:
you can now right-click any edit/write taglist (like those across the manage tags dialog) and choose to hide/show the implied parents that now hang underneath tags
you can set whether this defaults to hide or show, separately for the regular taglists and the autocomplete results dropdown, under options->tags
the taglist now sorts lexicographically using sibling tag data where available. I had expected to make options here to use storage or ideal tag, but once I tried it out, using the ideal all the time felt proper to me, so let's see how it goes
fixed the routine that removes mutually exclusive predicates (e.g. system:inbox/archive) when adding to the active search predicates taglist. this fixes the 'exclude xxx from search' menu action and other add/swap actions (issue #815)
gave the taglist right-click menu another quick pass. since there are all sorts of actions that may or not appear, and menu items can get pretty wide with tag text, I am trying out an intentionally short and thin top-level menu of 'verbs' that is quick to navigate with your mouse, and then tuck longer and taller stuff in secondary menus
.
boring code cleanup:
cleaned and unified a bunch of the new taglist sibling and parents display logic and other legacy variables. it now basically all derives from one storage/display state, so behaviour across the program should be more unified. this may cause confusion in some more advanced dialogs, so let me know anywhere it looks weird
the 'favourites' autocomplete tab in 'edit/write' a/c dropdowns now show siblings and parents for the current display service
the tag suggestions favourites dropdowns and taglists in the options now show siblings/parents according to the current service
the 'url class precedence' routine, which tests more 'specific' url classes first when trying to match an url, has a subtle logic change--now, url classes are first considered more 'specific' according to number of path components and parameters that have no default. this stops an url class with multiple optional parameters overriding another with a single fixed parameter (this is what affected the tumblr downloader above). the specific (descending) sort key is now (required components, total components, required parameters, total parameters, len normalised example url)
refactored client object serialisation access routines to a new db module
refactored database transaction code and status tracking to a separate object
refactored some more tag definition routines to the master tag module
fixed a bug in the new taglist backend that would sometimes error out in a paint event(!) on display initialisation or data changes for some clients
improved the taglist 'tag' vs 'copyable string' copy/select/action menu logic. e.g. 'namespace:*' is copyable, but it is not a tag
thread watchers now skip/clean up unactioned check log entries (this usually happens when a check is due during network traffic paused, queueing the job, and then the client shuts down). if you noticed some odd perpetually 'pending' checkers in last week's status overhaul, this was the issue, and they should clean up. this was always harmless, just revealed with new status code
thread watchers now record serious network error detail in the check log
thread watchers are quicker about notifying UI on checker log changes
thread watchers now report 'time delta' as their simple status when waiting to check, rather that 'checking in (time delta)'. let's see if that fits better in the columns
fixed an issue where several dialogs with multi-column lists would reset their 'last column' size to the minimum three characters on the next load if they did not receive certain size events while they were open. you should just have to fix any broken dialogs once and you'll be good again
I believe I also improved/fixed the issue of dialogs with multi-column lists sometimes shrinking by a few pixels every open/close
the 'we just woke from sleep' detection is now more aggressive. it should now detect a wake after sleeps as short as 60 seconds (down from 5 mins). let's see if we get any false positives during maintenance or other busy periods
if you have a complicated database (one stored across multiple locations), the 'database' menu now has a label in place of the simple database's 'backup/restore' commands
improved the 'directory is writeable-to' check used in the program. on windows, due to some python tempfile weirdness, this was actually hanging on Program Files.
improved the related 'is db dir writeable-to' test in the boot script. if you try to run the program on a custom non-writeable db directory, the crash error should now be nicer, and running a straight client.exe installed to 'Program Files' should now auto-place your db in your user folder, no complaints, like the macOS App
corrected 'writable' typo to 'writeable' across the program lmao
fixed the new header links in the FAQ file, which I accidentally messed up
started work on updating neighbouring .txt tag sidecar export. it isn't ready yet, but it will add tag filters and tag display type to sidecar export with easier expansions in future, and fold it nicely into Export Folders
improved some log-off detection + clean shutdown code, but I do not yet have nice multiplatform support
.
filetypes:
the stacked expand/collapse checkbox widget that lets you select filetypes now always starts collapsed. also, some 'partially clicked' logic is improved when you click through filetype group
application/clip (clip studio paint) files are now supported! thanks to a user for helping out here
just a side note: I looked into animated webp support this week, but it turns out decoding support is rarer than encoding. my normal and fairly new FFMPEG can't reliably render subsequent frames or figure out duration, nor can PIL or OpenCV. I think we will simply have to wait for an update on one of their ends
.
boring db cleanup:
wrote a local hashes cache to store hashes for all the files on your disk, much like the tag one. this should speed up all normal searches and other common file lookups in the db
the raw storage mapping tables are spun off to their own module
basic file info and inbox is spun off to its own module
improved and sped up some inboxing file count calculations
cleaned up some more misc file metadata and inbox code
when siblings display with the '(will display as xxx)' suffix, this text is now coloured by the correct namespace!
parents now show in 'manage tags dialog' taglists! they show up just like in a write/edit tag autocomplete results list
the tag right-click menu has had a pass. 'copy' is now at the top, the 'siblings and parents' menu is split into 'siblings' and 'parents' with counts on the top menu label and the submenus for each merged, and the 'open in new page' commands are tucked into an 'open' submenu. the menu is typically much tighter than before
when you hit 'select files with these tags' from a taglist, the thumbgrid now takes keyboard focus if you want to hit F7 or whatever
custom tag presentation (_options->tag presentation_, when you set to always hide namespaces or use custom namespace separator in read/search views) is more reliable across the program. it isn't perfect yet, but I'll keep working
a heap of taglist code has been cleaned up. some weird logical issues should be better
now the code is nicer to work with, I am interested in feedback on how to further improve display and workflows here
.
the rest:
added two mirrors for nitter, whose main site is failing due to load. I added them randomly from the page here: https://github.com/zedeus/nitter/wiki/Instances . if you have nitter subs, please move their download source to one of the mirrors or set up your own url classes to other mirror addresses. thanks to a user for providing other parser fixes here
gallery download pages now show the 'stop' character in the small file column when the files are done
gallery download pages now report their 'working' status without flicker, and they report 'pending' when waiting for a download slot (this situation is a legacy hardcoded bottleneck that has been confusing)
thread watchers also now have the concept of 'pending', and also report when they are next checking
improved the new grouped status sort on gallery downloader and watcher pages. the ascending order is now DONE, working, pending, checking later (for watchers), paused
the network request delay after a system resume is now editable under the new options->system panel. default is 15 seconds
the 'wait on files too' option is moved from 'files and trash' to this panel
when the 'just woke' status is active, you now get a little popup with a cancel button to override it
'open similar-looking files' thumbnail menu entry is moved up from file relationships to the 'open' menu
the duplicate filter right-hand hover window no longer has both 'previous' and 'next' buttons, since they both act as 'flip', and the merged button is moved down, made bigger, and has a new icon
added 'view next' to the duplicate filter shortcut set, so you can set a custom 'flip between pair' mapping just for that filter
thanks to a user helping me out, I was able to figure out a set of lookups in the sibling/parent system that were performing unacceptably slow for some users. this was due to common older versions of sqlite that could not optimise a join with a multi-index OR expression. these queries are now simpler and should perform well for all clients. if your autocomplete results from a search page with thumbs were achingly slow, let me know how they work now!
the hydrus url normalisation code now treats '+' more carefully. search queries like 6+girls should now work correctly on their own on sites where '+' is used as a tag separator. they no longer have to be mixed with other tags to work
.
small/specific stuff:
the similar files maintenance search on shutdown now reports file progress every 10 files and initialises on 0. it also has faster startup time in all cases
when a service is deleted, all currently open file pages will check their current file and tag domains and update to nicer defaults if they were pointed at the now-missing services
improved missing service error handling for file searches in general--this can still hit an export folder pointed at a missing service
improved missing service error handling for tag autocomplete searches, just in case there are still some holes here
fixed a couple small things in the running from source help and added a bit about Visual Studio Build Tools on Windows
PyOpenSSL is now optional. it is only needed to generate the crt/key files for https hosting. if you try to boot the server or run the client api in https without the files and without the module available to generate new ones, you now get a nice error. the availability of this library is now in the client's about window
the mpv player will no longer throw ugly errors when you try to seek on a file that its API interface cannot support
loading a file in the media viewer no longer waits on the file system lock on the main thread (it was, very briefly), so the UI won't hang if you click a thumb just after waking up or while a big file job is going on
the 'just woke' code is a little cleaner all around
the user-made downloader repository link is now more obvious on Lain's import dialog
an old hardcoded url class sorting preference that meant gallery urls would be matched against urls before post, and post before file, is now eliminated. url classes are now just preferenced by number of path components, then how many parameters, then by example url length, with higher numbers matching first (the aim is that the more 'specific' and complicated a url class, the earlier it should attempt to match)
updated some of the labelling in manage tag siblings and parents
when you search autocomplete tags with short inputs, they do not currently give all 'collapsed' matching results, so an input of 'a' or '/a/' does not give the '/a/' tag. this is an artifact of the new search cache. after looking at the new code, there is no way I can currently provide these results efficiently. I tested the best I could figure out, but it would have added 20-200ms lag on all PTR searches, so instead I have made a plan to resurrect an old cache in a more efficient way. please bear with me on this problem
tag searches that only include unusual characters like ? or & are now supported without having to lead the query with an asterisk. they will be slower than normal text search
fixed a bug in the 'add tags before import' dialog for local imports where deleting a 'quick namespace' was not updating the tag list above
.
windows clean install:
I moved to a new windows dev machine this week and a bunch of libraries were updated. I do not believe the update on Windows _needs_ a clean install this week, as a new dll conflict actually hits the coincidentally now-optional PyOpenSSL, but it is worth doing if you want to start using the Client API soon, and it has been a while, so let's be nice and clean. if you extract the release on Windows, please check out this guide: https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#clean_installs
the Windows installer has been updated to remove many old files. it should now do clever clean installs every week, you have nothing to worry about!™
.
boring db breakup:
the local tags cache, which caches tags for your commonly-accessed hard drive files, is now spun off to its own module
on invalid tag repair, the new master tags module and local tags cache are now better about forgetting broken tags
the main service store is spun off to its own module. several instances of service creation, deletion, update and basic fetching are merged and cleaned here. should improve a couple of logical edge cases with update and reset
.
boring taglist changes:
taglists no longer manage text and predicates, but a generalised item class that now handles all text/tag/predicate generation
taglist items can occupy more than one row. all position index calculations are now separate from logical index calculations in selection, sizing, sorting, display, and navigation
all taglist items can present multiple colours per row, like OR predicates
items are responsible for sibling and parent presentation, decoupling a heap of list responsibility mess
tag filter and tag colour lists are now a separate type handled by their own item types
subordinate parent predicates (as previously shown just in write/edit autocomplete result lists) are now part of multi-row items. previously they were 'quiet' rows with special rules that hung beneath the real result. some related selection/publish logic is a bit cleaner now
string tag items are now aware of their parents and so can present them just like autocomplete results in write/edit contexts
the main taglist content update routines have significantly reduced overhead. the various expansions this week add some, so we'll see how this all shakes out
the asynchronous sibling/parent update routine that populates sibling and parent data for certain lists is smarter and saves more work when data is cached
old borked out selection/hitting-skipping code that jumped over labels and parents is now removed
'show siblings and parents' behaviour is more unified now. basically they don't show in read/search, but do in write/edit
a heap of bad old taglist code has been deleted or cleaned up
fixed another ghost pending tags bug. this may have been new or there since the display cache started, I am not sure, but it shouldn't happen again. it was occuring when a pending tag was being committed to 'current' and another tag in its sibling group already existed as a current tag for that file. the pending tag and its count would not clear for non-'all known files' domains, causing ghosts to appear in search pages but not typically manage tags. you may have noticed ghost tags hanging around after a pending commit--this was it. the true problem here was in the 'rescind pending' action that occurs just before an add/commit. a new unit test tests for this situation both for two non-ideal tags being pend-merged, and a non-ideal tag being pend-merged into the existing ideal
wrote a routine to regenerate _pending_ tag storage and autocomplete counts from scratch for the combined and specific display tag caches. this job is special in that it regens tags instantly without having to reset sibling/parent sync. you can run the job from the database->regen menu. this is the start of 'fix just this tag' maintenance ability
the pending regen routines will occur on update. it shouldn't take long at all, unless you have five million tags pending, where it could be a couple minutes
.
autocomplete shortcuts:
there is a new shortcut set under _file->shortcuts_ just for tag autocomplete shortcuts. any 'switch searching immediately' shortcut previously on 'main gui' will be migrated over
the tag autocomplete input text box is now plugged into the new shortcut system and uses this set
migrated previously hardcoded autocomplete shortcuts to the shortcut system (defaults):
- force search now, for when you have automatic searching turned off (ctrl+space)
- enable IME-friendly mode (insert)
- if input empty, move left/right a tab (left/right arrow)
- if input empty, move left/right a service page (up/down arrow)
- if input empty and on media viewer, move to previous/next media (page up/down)
misc improvements to my shortcut handler
misc shortcut code cleanup
.
the rest:
I fixed a bad example url in the new gelbooru file page parser that was sometimes leading to a link to the gallery url class. this was an artifact of an old experiment with md5-search parsing, now fixed with newer redirection tech. the updated parser is folded into update, and if you ended up with the incorrect link, it should be detected, dissolved, and re-linked with the file page parser
thanks to a user report, wrote a new url class for 420chan's newer thread url format
sorting a gallery downloader or thread watcher multi-column list by 'status' should now group 'done' and 'paused' items separately
fixed a bug in the /add_tags/add_tags Client API call when checking some petitioned tags data types. cleaned all that code, it was ugly (issue #788)
added unit tests for /add_tags/add_tags to test the service_names_to_actions_to_tags parameter better and repository actions, including petitioning with and without specified reason
.
code refactoring:
finally addressing the near-1MB ClientDB file, I have started a framework to break the db into separate modules with their own creation/repair/work responsibilities. this will make the file easier to work on, maintain, update, and test. this week starts off simple, with the master definitions being peeled off into hashes, tags, urls, and texts submodules
cleaned some misc code around here, including a bunch of related decoupling
ClientDB.py is now in its own 'db' module as well. the db will further fracture and this module will gain more files in future
the boot code in the launch scripts is now migrated to the 'hydrus' directory, with the actual launch scripts now doing nicer __main__ checks to not launch the program if you want to play around with importing hydrus. more work to come here
finished the help's header linking job--all headers across the help are now #fragment links
thanks to help from Codexx at 8chan.moe, the old 8kun board is completely migrated and archived at 8chan.moe /hydrus/. going forward I will be maintaining a Hydrus Network General there on /t/ for merged release posts, Q&A, and Bug Reports. the plan is that whenever it fills up, it will be moved to the /hydrus/ archive. the links across the program and help are updated, please let me know if I missed any. Endchan /hydrus/ remains as a bunker
fixed a bug where subtag entries in the new tag fast search cache were being deleted for all namespaces when a single namespaced version was went to count 0. it meant some autocomplete results were not appearing, often after some sibling changes. a new 'repopulate' job has been added to the database regenerate menu to fix this efficiently if something like it happens again. this routine will be run on update to fix all users, it shouldn't take long (issue #785)
fixed a bug where the new network objects would throw an error on save when a 'dirty' object was quickly deleted. I think this was typically sessions that only have ephemeral session cookies being created in the final five minutes of the program and then being cleared during program exit
when an archive/delete filter finishes, it now fires off all its changes in one go. previously they would go in ~64-file chunks over the next few hundred milliseconds. this will add a small amount of 'refresh lag', delaying page refreshes etc..., on bigger filter jobs for some clients, but it will guarantee that if you hit F5 real quick after finishing filtering on a processing page with non-random sort, you won't see the same files again at the top, only for them to be swiftly archived/deleted as you watch. trash file performance is much better these days, let me know how this goes for you if you do megafilters
the tag import options whitelist now checks subtags of parsed tags. if you add 'samus aran' to the whitelist, but the site delivers 'character:samus aran', this now passes the whitelist
thanks to a user's submission, the gelbooru 0.2.5 post parser is updated and should get tags again, for those users who stopped getting them last week--however, I never experienced this myself, so please let me know if you still have trouble. there could be something more complicated going on here
updated the gelbooru 0.2.x gallery parser to handle an alternate form of gelbooru pools--we did not figure out why different users are being given different markup, it wasn't as simple as being logged in or not, but there is a difference for some. this parser is folded in on update, so the gelb pool downloader should be fixed for users who had trouble with it
also updated the gelbooru pool gallery url class to infer next page url, as in the alternate form the next page is difficult to parse
the 'clear all closed pages' command under the 'undo' menu now asks for yes/no confirmation
added a 'callto' profile mode, which will be very useful in diagnosing GPU lag in future. the 'callto' jobs are little off-main-thread things like image rendering and async panel preparation. should help us figure out where big download pages etc... are eating up CPU
the different profile modes in the debug menu now all show popup messages, but only when their job exceeds the particular profile's interesting time, usually 3-20ms. this should reduce spam
the 'this session' bandwidth tracker on the status bar is now a special tracker that only includes data from boot. previously, it was using the 'global' tracker, which after certain time intervals (four minutes, three hours, three days), will compress bandwidth history into larger time windows to save space. if one of these windows covered time before the client started, it could spookily report a little bandwidth used on a client started with network traffic paused
bandwidth data usage in times shorter than the last ten seconds (which are smoothed out to avoid bumps) now also get the 'don't get bandwidth from the future on motherboards that had a briefly crazy system clock' fix from last week
fixed some disengaged database tuning that was leading to worse cancel times on certain jobs
updated a whole bunch of the help so section headings are links with nice #fragment/anchor ids, making it easy to link other users to a particular section. I will continue this work, and future help will follow this new format
fixed some bad character encodings in the changelog document, siblings help, and tagging schema help. these should now be utf-8 valid
.
object load improvements:
the client now detects serialisable (saveable) objects that were generated in a future version format your client does not yet support. this mostly affects downloader objects like parsers, where you might import an object a user in a much newer version of the client made. for instance, this week some users imported a fixed gelbooru parser in an older client, which was then saved and double-updated later on, and that caused other problems down the line. downloader imports deal with this situation cleanly, but otherwise it mostly makes a popup notifying you of the problem and asking to contact me. there are about 170 places in the program where objects are deserialised and I am not ready to make this a fullblown error until I know more about people's IRL situations. let's hope this is not widespread. if you run into this, please let me know!
if you were running an older client and manually imported the updated gelbooru parser that was going around, and then you got errors about 'md5', hex' or 'additional_info' something, it _should_ be automatically fixed on update. you should be able to update from previous to ~422, see it in network->downloader components->manage parsers, and it should just work. many users will have the entry overwritten anyway in the above gelb update I am rolling in. if any of this does still give you trouble, please delete and re-import the affected object(s)
importing one of these future-versioned serialised objects using the import/export buttons on a multi-column list, either clipboard, json, or png, will cleanly discard future objects with a non-spammy notification
the Lain drag-and-drop easy downloader import does the same
the parser 'show what this can parse in nice text' routine now fails gracefully
multi-column lists now handle a situation where either the display or sort data for a row cannot be generated. a single error popup per list will be generated so as not to spam, bad sorts will be put at the top, and 'unable to render' will occupy all display cells
.
network server stuff:
fixed being able to delete an account type in the server admin menu
the way accounts are checked for permissions serverside now works how the client api does it, unified into a neater system that checks before the job starts
did some misc server code cleanup, and clientside, prepped for restoring account modification and future improvements
I fixed the new tag cache's slow tag autocomplete when in 'all known files' domain (which is usually in the manage tags dialog). what was taking about 2.5 seconds in 424 should now take about 58ms!!! for technical details, I was foolishly performing the pre-search exact match lookup (where exactly what you type appears before the full results fetch) on the new quick-text search tables, but it turns out this is unoptimised and was wasting a ton of CPU once the table got big. sorry for the trouble here--this was driving me nuts IRL. I have now fleshed out my dev machine's test client with many more millions of tag mappings so I can test these scales better in future before they go live
internal autocomplete count fetches for single tags now have less overhead, which should add up for various rapid small checks across the program, mostly for tag processing, where the client frequently consults current counts on single tags for pre-processing analysis
autocomplete count fetch requests for zero tags (lol) are also dealt with more efficiently
thanks to the new tag definition cache, the 'num tags' service info cache is now updated and regenerated more efficiently. this speeds up all tag processing a couple percent
tag update now quickly filters out redundant data before the main processing job. it is now significantly faster to process tag mappings that already exist--e.g. when a downloaded file pends tags that already exist, or repo processing gives you tags you already have, or you are filling in content gaps in reprocessing
tag processing is now more efficient when checking against membership in the display cache, which greatly speeds up processing on services with many siblings and parents. thank you to the users who have contributed profiles and other feedback regarding slower processing speeds since the display cache was added
various tag filtering and display membership tests are now shunted to the top of the mappings update routine, reducing much other overhead, especially when the mappings being added are redundant
.
tag logic fixes:
I explored the 'ghost tag' issue, where sometimes committing a pending tag still leaves a pending record. this has been happening in the new display system when two pending tags that imply the same tag through siblings or parents are committed at the same time. I fixed a previous instance of this, but more remained. I replicated the problem through a unit test, rewrote several update loops to remain in sync when needed, and have fixed potential ghost tag instances in the specific and 'all known files' domains, for 'add', 'pend', 'delete', and 'rescind pend' actions
also tested and fixed are possible instances where both a tag and its implication tag are pend-committed at the same time, not just two that imply a shared other
furthermore, in a complex counting issue, storage autocomplete count updates are no longer deferred when updating mappings--they are 'interleaved' into mappings updates so counts are always synchronised to tables. this unfortunately adds some processing overhead back in, but as a number of newer cache calculations rely on autocomplete numbers, this change improves counting and pre-processing logic
fixed a 'commit pending to current' counting bug in the new autocomplete update routine for 'all known files' domain
while display tag logic is working increasingly ok and fast, most clients will have some miscounts and ghost tags here and there. I have yet to write efficient correction maintenance routines for particular files or tags, but this is planned and will come. at the moment, you just have the nuclear 'regen' maintenance calls, which are no good for little problems
.
network object breakup:
the network session and bandwidth managers, which store your cookies and bandwidth history for all the different network contexts, are no longer monolithic objects. on updates to individual network contexts (which happens all the time during network activity), only the particular updated session or bandwidth tracker now needs to be saved to the database. this reduces CPU and UI lag on heavy clients. basically the same thing as the subscriptions breakup last year, but all behind the scenes
your existing managers will be converted on update. all existing login and bandwidth log data should be preserved
sessions will now keep delayed cookie changes that occured in the final network request before client exit
we won't go too crazy yet, but session and bandwidth data is now synced to the database every 5 minutes, instead of 10, so if the client crashes, you only lose 5 mins of login/bandwidth data
some session clearing logic is improved
the bandwidth manager no longer considers future bandwidth in tests. if your computer clock goes haywire and your client records bandwidth in the future, it shouldn't bosh you _so much_ now
.
the rest:
the 'system:number of tags' query now has greatly improved cancelability, even on gigantic result domains
fixed a bad example in the client api help that mislabeled 'request_new_permissions' as 'request_access_permissions' (issue #780)
the 'check and repair db' boot routine now runs _after_ version checks, so if you accidentally install a version behind, you now get the 'weird version m8' warning before the db goes bananas about missing tables or similar
added some methods and optimised some access in Hydrus Tag Archives
if you delete all the rules from a default bandwidth ruleset, it no longer disappears momentarily in the edit UI
updated the python mpv bindings to 0.5.2 on windows, although the underlying dll is the same. this seems to fix at least one set of dll load problems. also updated is macOS, but not Linux (yet), because it broke there, hooray
as 2020 ended, I attempted but failed to tune fast search for all kinds of clients, big and small and simple and complex. unable to guarantee decent speeds with just code, I have redesigned the tag text search cache. rather than checking the gigantic master table for all namespace and subtag lookups, the client can now zoom in on a small fast cache limited to the current search context, so doing a clever lookup on 'my tags' will no longer be hampered by having PTR beside it, and doing a solid lookup on the PTR or 'all known tags' will no longer be accidentally hampered by an optimisation for another situation
the 424 update will take some time to generate the new caches for your existing data. if you don't sync with the PTR, it should be a few seconds. if you do sync, it will be about ten minutes on an SSD (seems about 30,000 definitions a second), and somewhat longer on an HDD. it will count up the tags as it goes, and on the PTR there will be a bit of deletion work, then one or two counts up to perhaps a million, and then one big count up to about 16 million.
in my initial tests, this cache adds about 1-2% additional processing time to mass tag changes, but a wide variety of tag lookups and file searches are now significantly faster, have much nicer worst-case lag spikes, and should cancel quicker. these are best in any specific tag domain, although 'all known tags' should still be much better. a future expansion of the tag cache is planned to finally address clean and accurate 'all known tags' searches
summary; all these should be faster and cancel faster:
autocomplete searches for 'subtag*' (most normal searches) are optimised
autocomplete searches for 'namespace:*' are optimised, including when the namespace itself is a wildcard
autocomplete searches for wildcards with an asterisk in the middle of the subtag are optimised
autocomplete searches for wildcards with an asterisk at the beginning of the subtag are optimised (but this is still generally the slowest query)
autocomplete searches for namespace and subtag wildcard combinations are optimised, with either or both as a wildcard of any type
autocomplete searches for '*' are optimised
tag file searches without a namespace (i.e. in file search, with any namespace) are optimised
namespace file searches are optimised, including when the namespace is a wildcard
wildcard file searches are optimised, for all the classes of wildcard above
'tag as number' file searches are optimised
'has ><= x namespace tags' file searches are optimised for speed, including when the namespace is a wildcard, but still have bad cancelability on large domains. I'll work on this more
.
other tag cache info:
the 'tag text search cache' regeneration routine under the _database->regenerate_ menu is replaced with a service specific routine for the new cache
on boot, if the client sees any of the new cache tables are missing, it notifies you and regenerates the affected subsection of the cache
an old method of performing complex wildcard searches was using surplus data and has been eliminated. these searches are now also computationally cheaper beyond the other domain-based optimisations this week
I have identified the next bottleneck in the tag search pipeline and have a plan to speed all the above up even further, which can all be done in code
thanks to user feedback, I have also identified other wasteful overhead in tag processing. I'll keep working!
while the planned 'all known tags' cache will be useful since most file searches are in this domain, it will be a bit of work, so I will first let this new lookup cache breathe for a bit. 'all known tags' will not be nearly as big as the 'all known files/combined file' caches that have hit us with so much CPU recently. I expect it to increase the client.caches.db size by about 5%
unified all increments or decrements to autocomplete count caches, no matter the service domain, to one location
unified how autocomplete counts are fetched across different service domains
optimised specific and combined autocomplete count cache update overhead for new, existing, and deleted tags
optimised display autocomplete count cache updates for tags with multiple siblings or parents
optimised the 'local tags cache', which does fast tag text fetching for local files, when new tags or files are added/removed from the 'all local files' domain. this now occurs in the same unified autocomplete count update process. it now also caches pending tags that have no current count
merged 'exact match' autocomplete tag searching code into generalised wildcard search
misc autocomplete and other tag code cleanup and harmonisation
ditched some old mass UNION queries that were not cancelling well
.
the rest:
when you paste queries into a sub, the summary 'these were/were not added' dialog now always appears, and if you paste empty whitespace, it now says so
the manage siblings/parents dialogs now specify which services apply which siblings, whether they are fully synced, the current display tag sync maintenance settings, and ultimately whether you can expect changes to apply quickly after dialog ok
when a text entry dialog comes with suggestion buttons, it now focuses the text box by default. sorry for the trouble here! (issue #765)
updated a couple petition reason suggestions in manage tags and parents
added a shortcut to 'main window' to refresh _manage tags'_ related tags suggestions with 'thorough' duration. in future, these dialog-specific actions will be moved out of 'main window', these have just been a 'temporary' patch
updated the 'running from source' and 'install' help with some new numbers and info about mpv, and updated the 'server' help with a document helpfully provided by a user explaining that the server does not do what many new users think
sped up 'has tags' file searches in certain situations, mostly when there are few if any other search predicates
the default e621 parser now pulls meta tags, thank you to a user for providing this
the default nitter timeline url classes are updated, thank you to a user for providing this
the new little hook that takes 'file:///' off of paths pasted into the filename tagging path text now also normalises the path, so if you are on Windows, the URI's slashes will be Windows-corrected to backlashes. it also now removes wrapping quotes
the hydrus logger again correctly restores stdout and stderr after it is closed on program exit (this was disabled for some reason, but fingers crossed it seems fine now!)
an issue where automatically started duplicate potentials file search could not cancel when shutdown 'stop work' button was clicked or where idle maintenance mode turned off should be fixed
the shutdown maintenance work for the first client shutdown now has a little text saying it is just some quick initialisation work
for hopefully the last and completely final time, I think I fixed the invalid tag repair function for certain sorts of tags applied to currently local files
improved the way a job thread was pulling new jobs (issue #750)
the 'fetch results as you type' and 'do-not-autocomplete character threshold' options are moved from _options->speed and memory_ to _tags->manage tag display and search_. they are now service specific!
getting the raw '*' autocomplete is now hugely faster when both file and tag domains are specific (i.e. not 'all known xxx')
getting the raw '*' autocomplete is now hugely faster in 'all known tags' domain. this is likely still bonkers on any decent sized client that syncs with the PTR, but if you have a small client that once synced with the PTR, this is now snappy
the cancelability of 'namespace:*' and 'match namespaces from normal search' searches should be improved
'namespace:*' queries are now much faster in some situations, particularly when searching in a specific tag domain (typically this happens in manage tags dialog) or a small-file client, but is still pretty slow for clients with many files, and I think some scenarios are still bananas. I am not happy here and have started a plan to improve my service domain caches to deal with several ongoing problems with slow namespace and subtag lookup in different situations
fixed an issue with advanced autocomplete result matching where a previously cached 'character:sam' result could match 'char:sam' search text
some misc code cleanup and UI label improvements in autocomplete
.
the rest:
the siblings & parents tag menu, which proved a little tall after all, is now compressed to group siblings, parents, and children by the shared services that hold them. it takes less space, and odd exceptions should be easy to spot
this menu also no longer has the 'this is the ideal tag' line
added 'sort pages by name a-z/z-a' to page right-click menu and tucked the sorts into a submenu
the parsing test panel now shows up to 64KB of what you pulled (previously 1KB)
the parsing test panel now shows json in prettier indented form
when the parsing test panel is told to fetch a URL that is neither HTML or JSON, this is now caught more safely and tested against permitted file types. if it was really a jpeg, it will now say 'looks like a jpeg' and disable parse testing. if the data type could not be figured out, it tries to throw the mess into view and permits parse testing, in case this is some weird javascript or something that you'll want to pre-parse convert
the dreaded null-character is now eliminated in all cases when text is decoded from a site, even if the site has invalid unicode or no encoding can be found (e.g. if it is truly a jpeg or something and we just end up wanting to throw a preview of that mess into UI)
the 'enter example path here' input on import folders' filename tagging options edit panel now uses placeholder text and auto-removes 'file:///' URL prefixes (e.g. if your paste happens to add them)
the 'fix invalid tags' routine now updates the tag row in the local tags cache, so users who found some broken tags were not updating should now be sorted
added --db_cache_size launch parameter, and added some text to the launch_parameters help about it. by default, hydrus permits 200MB per file, which means a megaclient under persistent heavy load might want 800MB. users with megamemory but slow drives might want to play with this, let me know what you find
fixed the search code for various 'total' autocomplete searches like '*' and 'namespace:*', which were broken around v419's optimised regular tag lookups. these search types also have a round of their own search optimisations and improved cancel latency. I am sorry for the trouble here
expanded the database autocomplete fetch unit tests to handle these total lookups so I do not accidentally kill them due to typo/ignorance again
updated the autocomplete result cache object to consult a search's advanced search options (as under _tags->manage tag display and search_) to test whether a search cache for 'char' or 'character:' is able to serve results for a later 'character:samus' input
optimised file and tag search code for cases where someone might somehow sneak an unoptimised raw '*:subtag' or 'namespace:*' search text in
updated and expanded the autocomplete result cache unit tests to handle the new tested options and the various 'total' tests, so they aren't disabled by accident again
cancelling a autocomplete query with a gigantic number of results should now cancel much quicker when you have a lot of siblings
the single-tag right-click menu now shows siblings and parents info for every service, and will work on taglists in the 'all known tags' domain. clicking on any item will copy it to clipboard. this might result in megatall submenus, but we'll see. tall seems easier to use than nested per-service for now
the more primitive 'siblings' submenu on the taglist 'copy' right-click menu is now removed
right-click should no longer raise an error on esoteric taglists (such as tag filters and namespace colours). you might get some funky copy strings, which is sort of fun too
the copy string for the special namespace predicate ('namespace:*anything*') is now 'namespace:*', making it easier to copy/paste this across pages
.
misc:
the thumbnail right-click 'copy/open known urls by url class' commands now exclude those urls that match a more specific url class (e.g. /post/123456 vs /post/123456/image.jpg)
miniupnpc is no longer bundled in the official builds. this executable is only used by a few advanced users and was a regular cause of anti-virus false positives, so I have decided new users will have to install it manually going forward.
the client now looks for miniupnpc in more places, including the system path. when missing, its error popups have better explanation, pointing users to a new readme in the bin directory
UPnP errors now have more explanation for 'No IGD UPnP Device' errortext
the database's boot-repair function now ensures indices are created for: non-sha256 hashes, sibling and parent lookups, storage tag cache, and display tag cache. some users may be missing indices here for unknown update logic or hard drive damage reasons, and this should speed them right back up. the boot-repair function now broadcasts 'checking database for faults' to the splash, which you will see if it needs some time to work
the duplicates page once again correctly updates the potential pairs count in the 'filter' tab when potential search finishes or filtering finishes
added the --boot_debug launch switch, which for now prints additional splash screen texts to the log
the global pixmaps object is no longer initialised in client model boot, but now on first request
fixed type of --db_synchronous_override launch parameter, which was throwing type errors
updated the client file readwrite lock logic and brushed up its unit tests
improved the error when the client database is asked for the id of an invalid tag that collapses to zero characters
the qss stylesheet directory is now mapped to the static dir in a way that will follow static directory redirects
.
downloaders and parsing (advanced):
started on better network redirection tech. if a post or gallery URL is 3XX redirected, hydrus now recognises this, and if the redirected url is the same type and parseable, the new url and parser are swapped in. if a gallery url is redirected to a non-gallery url, it will create a new file import object for that URL and say so in its gallery log note. this tentatively solves the 'booru redirects one-file gallery pages to post url' problem, but the whole thing is held together by prayer. I now have a plan to rejigger my pipelines to deal with this situation better, ultimately I will likely expose and log all redirects so we can always see better what is going on behind the scenes
added 'unicode escape characters' and 'html entities' string converter encode/decode types. the former does '\u0394'-to-'Δ"', and the latter does '&'-to-'&'
improved my string converter unit tests and added the above to them
in the parsing system, decoding from 'hex' or 'base64' is no longer needed for a 'file hash' content type. these string conversions are now no-ops and can be deleted. they converted to a non-string type, an artifact of the old way python 2 used to handle unicode, and were a sore thumb for a long time in the python 3 parsing system. 'file hash' content types now have a 'hex'/'base64' dropdown, and do decoding to raw bytes at a layer above string parsing. on update, existing file hash content parsers will default to hex and attempt to figure out if they were a base64 (however if the hex fails, base64 will be attempted as well anyway, so it is not critically important here if this update detection is imperfect). the 'hex' and 'base64' _encode_ types remain as they are still used in file lookup script hash initialisation, but they will likely be replaced similarly in future. hex or base64 conversion will return in a purely string-based form as technically needed in future
updated the make-a-downloader help and some screenshots regarding the new hash decoding
when the json parsing formula is told to get the 'json' of a parsed node, this no longer encodes unicode with escape characters (\u0394 etc...)
duplicating or importing nested gallery url generators now refreshes all internal reference ids, which should reduce the liklihood of accidentally linking with related but differently named existing GUGs
importing GUGs or NGUGs through Lain easy import does the same, ensuring the new objects 'seem' fresh to a client and should not incorrectly link up with renamed versions of related NGUGs or GUGs
added unit tests for hex and base64 string converter encoding
thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation
fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again
fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices
prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start
when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)
the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later
the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on
fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist
I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search
when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast
fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later
reduced overhead of tag filtering, which should improve display speed of taglist for very large pages
parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster
.
duplicates search improvements:
potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going
the search distance in duplicates pages is now synchronised across all pages--when one updates, they all do
all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often
misc cleanup for duplicates page
.
database modes:
a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
added the '--db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY
ensured --db_synchronous_override was hooked up correctly
the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code
fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true
db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_
.
technical database nonsense:
PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn
hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done
hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see
in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations
when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour
after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%
the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated
improved launch switch code generally
boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values
the client and server no longer disconnect from the db to check whether it is possible to vacuum databases
fixed the bad position indexing when drag-selecting taglists that were scrolled down. this also caused some weird selection when scrolled and clicks included a little mouse movement. sorry for the trouble!
ctrl+drag-select now deselects!
fetching tag autocomplete results when you have thumbnails and 'searching immediately' on, which has been way too slow recently, now cancels much faster. in some large page situations, it was adding multi-second lag on the first character-press. it also runs faster overall
hydrus should now deal better with invalid tags that contain the null character (there is one we know about on the PTR, from a decode of botched Shift JIS, which could crash the client from too many errors during critical paint periods). when a tag like this turns up in a taglist, thumbnail, or canvas background, it now renders as an appropriate 'invalid tag' string, and a one-time 'woah, bad tag, run fix tags now' popup appears
regular tag cleaning now looks for and removes null characters, so all new sources of these bad tags should now be eliminated
_database->check and repair->fix invalid tags_ now fixes tags with the null character. it also fixes tags so broken that after cleaning they have no subtag left. it also now forces a full media tag reload when it is done for all media
the 'regen storage mappings', 'regen display mappings', and repopulate from cache' database routines now have an additional step where you can order them only to work on one tag service, so regenning or repopulating local tags, which usually takes a couple seconds, doesn't need to wait two hours for the PTR to go as well
added some menu help to the 'profile modes' debug menu, and gave 'reducing program lag' help page a pass
fixed virtual display regeneration on service delete
.
parents and siblings:
fixed situations where some grandparent and sibling relationships would not appear in the virtual system. it was a bug when certain links of a multi-part display 'chain' were updated at different times. when repopulating chain data, the sibling and parent update routines now correctly chase their complete chains both when wiping ideal data and repopulating from raw data, hitting all levels of the chain, ensuring to go back up and down chains when there are multiple grandsiblings/children/parents, and chasing parents where one or both members have better siblings. thank you to the several users who reported and helped figure out this problem, which was not simple to reproduce (issue #725)
your ideal display data will be regenerated on update, which should not take more than a couple of seconds. it will likely correct some siblings and add some grandparents to be filled in by the siblings/parents sync. my PTR test environment went up from about 189,000 display rows to 192,000
while sibling and parent lookup is more thorough (and hence more expensive), I also optimised many parts of lookup week. I believe tag display sync and tag processing will be much faster for tags with simple sibling and parent relationships, and slightly slower for tags with complex relationships and many instances to files on your drive. as always, let me know what sort of processing speeds and lag you get, and if you know how to make a db profile, please send them in when it gets bad
when a 'write' autocomplete results list includes parent expansion rows (as in _manage tags_), parents now show duplicated and properly for all the tags that have them, including siblings and other children/grandchildren (previously, a parent label could only exist once in a list, which meant parents were ending up hanging off the last valid tag for which they applied)
'write' autocompletes now show results that exactly match the text entry, and all their siblings, when they do not have count but do have sibling or parent data. so, if you type in 'samus aran', and it has a sibling to 'character:samus aran', but 'samus aran' doesn't actually have count, you now get it and all siblings anyway. this may need tuning, but it solves a persistent and annoying lookup and quick-sibling-access problem in _manage tags_
copying tags and their indented parents now removes the parent indent whitespace
tag sync display now takes way longer breaks (now 30 seconds, was 2.5) between 'normal' background work periods. this thing was hammering people far harder than needed and could clog up db write/commit time and nobble UI responsitivity when big bumps collided
the tag display maintenance manager now also tries to detect when many siblings or parents are streaming in (from a migration or a repository process with a heap of data), and pauses work while that continues
greatly sped up mass imports of sibling and parent data, either from tag migration or big dialog pastes. what was 40 rows/s should now be about 1,000 rows/s
fixed the database menu's 'regenerate tag parents lookup cache', which wasn't hooked up
.
boring changes:
gave tag parents and siblings update, regen, and chain fetch a full pass, correcting bad queries to fix the above, fixing raw pair chain level navigation and parent-sibling idealisation, and optimised these lookups as well
fixed some tag_id vs ideal_tag_id nomenclature (and related bugs) in tag parents cache
optimised 'all known tags' autocomplete count fetching a little. tag autocomplete and search should be a bit faster in this domain
reduced display sync pre-processing overhead by about 30% with a better random pair sampling routine
reduced the overhead of my now very commonly used single integer memory table select optimisation. this now recycles tables after use, which reduces overhead about 50% in small number scenarios. all features of the database will enjoy this speed improvement, particularly small repetitive tag lookup jobs (such as the new display sync and repository tag processing)
reduced overhead on some sibling chain lookup code
reduced overhead on the sibling lookup used by manage tag dialog taglist
reduced overhead on some parent chain lookup code
tiny optimisation on single sibling chain lookup
sped up the ancient OG single tag->tag-id fetching routine, seems to work about twice as fast now
more misc optimisations, mostly list/set/dict comprehension rewriting to reduce overhead, across virtual sibling and parent code
added a full combined siblings and parents unit tests for the main missing parent chain link problem solved this week
added a full combined siblings and parents unit test for large real world data added in multiple pieces
'a file identifier was missing!' critical errors now print a stack trace to the log for further debugging info
updated the 'help my db is broke.txt' document with a couple new comments
you can now set the default value for any editable system predicate. a star button beside each panel lets you set or clear the custom default
all editable system predicate panels now put 'recent' predicate buttons up top, for the five most entered predicates of the respective types. this is a little jank and grows pretty tall with multi-pred-type panels, but let me know what you think
all tag lists now support drag-selection!
taglists now have 'open a new OR page' menu entry when more than one tag is selected
when taglists can change the current search, they now have an 'add an OR to current search' menu entry when more than one tag is selected
OR Predicates are now editable! they launch their own little autocomplete input that is a little jank because you can technically make nested ORs, but it works!
system:rating is now editable! it launches the whole stack every time. the stack alignment is messed up though :/
invertible predicates (inbox/archive, tag/-tag, etc...) now flip on double-click only if you have one selected. if you have more than one selected, they appear as invertible buttons along with the rest of the edit UI
the active search predicates taglist now has an 'edit search terms' menu entry, if you find shift+double-click a pain
when you shift+double-click on more than one tag to add them to the current search, this is now added as an OR
similarly if you shift+middle-click on more than one tag, the new page is now an OR
when editing predicates, edited predicates now stay selected
shift+clicking on an already selected tag no longer adds any new selections (i.e. shift+click filling-in). this should make it nicer to do shift+double-click on selections. furthermore, the 'last clicked' focus ghost (from which a shift+click selection cascade starts) on tag lists is now cleared on edits or removes, which should reduce some other crazy/annoying select behaviour here
the list of active search predicates now correctly initialises sorted
entering hex hashes into system:hash or :similar_to now has unified hash parsing, auto-removes 'md5:'-style prefixes, and presents detailed error information when a hash is too long or short
.
faster and snappier file and tag searching:
searching for files by complicated wildcard (i.e. a search phrase that includes an asterisk in a non-rightmost character position) is now greatly optimised when the tag does not start with an asterisk (e.g. 'sm*l' is now much faster, '*all' is still hellmode), and now cancels (due to hitting the stop button or changing the query before results come in) much faster thanks to a new unified results fetching and cancel-checking routine
rewrote my autocomplete tag search to use the new namespace and subtag lookup code from the virtual siblings and parents system, unifying lookup logic and benefitting from the same new complicated wildcard optimisation and fast-cancel tech
autocomplete tag count aggregation (a later step, after the initial lookup) benefits from a little faster cancel tech
all file queries based on tag, wildcard, namespace, tag count, and tag existence now use the new fast-cancel tech. if you put in a 'has >4 tags' query and it is taking ages, changing the query or just hitting the 'stop' button should now free up the db pretty fast
related tags suggestions also gets the cancel tech and is now more timing precise for tags with either huge or tiny count
.
client api:
the /get_files/file_metadata call now returns a service_names_to_statuses_to_displayed_tags structure, which reflects the sibling-collapsed and parent-added tags, as displayed to the user in UI. the help is updated to reflect this
the client api version is now 15
.
the rest:
fixed an issue where regenerating the tag definition search cache would not tidy up the 'I am busy' modal dialog once it was done, resulting in a soft lock
fixed another upnp error handling bug, this time in the upnp daemon
updated Qt to 5.15.2 on Windows and Linux builds. this should fix the unusual button clicking area problem for some custom styles
.
boring specific code changes:
wrote widgets to edit invertible preds and OR preds
pulled the messy rating code out of the rating system predicate ui code to their own widgets
wrote some special predicate ui definitions and initialisation handling for OR preds and grouped 'multiple' preds (for ratings)
refactored search and predicate ui code to a new 'search' module
refactored collect and sort widgets away from search code
almost all system predicates are now editable if you shift+double-click them! you can also edit several at once in the same dialog
if you double-click on any predicate type that is not editable but does have an inverse version (e.g. archive/inbox, has audio/no audio, and tag/-tag), the inverse version(s) will be swapped in
all legacy custom system predicate defaults are eliminated this week. the panels now show a fixed default on launch, and will get a flexible favourites system in future, along with 'recently entered' quick buttons
restored the 'show system:everything' and 'hide archive/inbox' options, which were inadvertantly hidden when file system predicate defaults were hidden, to the new _options->search_ panel
fixed the borked list height for the file viewing statistics system pred panel checkbox lists
fixed an issue where namespace:anything predicates would not propagate to new pages on 'open page with these tags' commands
.
boring code specifics:
updated almost all the system predicate panels to take arbitrary initialisation values, and wrote a 'can I edit this' test for all predicate types to help some finnicky which-panel-and-pred-to-use issues
wrote some new filtering code and a little UI to handle editing of system preds
cleaned up some of the taglist item activation code
the hydrus network version is increased this week from 18 to 19. clients and servers can only talk to each other when they are on the same version, so please update your clients if you wish to keep talking to the PTR, and your own servers if you have a home network setup or similar. if a server and client are on different versions, you will get a polite error when they next try to talk, and sync will be paused
added 'run all export folders now' shortcut command to 'main window' shortcut set
added shortcuts to the 'main gui' shortcut set for navigating the currently selected page. you can move left, right, to the leftmost on the current row, or to the rightmost. the left and right will cycle up a page of pages layer when at left/rightmost boundaries, letting you iterate through all pages in a depth-first manner
updated the default newgrounds parser to deal with artists with more than 60/70 items in one art gallery (essentially, some clever 'next page' fetching now occurs to get older info that in your browser is drawn in as you scroll down). if you have some subscriptions for artists where you know this is true, try doing a full reset on them
added realbooru to the hydrus defaults. they also apparently just switched away from a gelbooru 0.2.x site, so if you have a gelbooru parser with a realbooru example URL, I remove that example URL
updated the page initial media load routine to my new async job
updated the imported file presentation page-publish routine to my new async job
when drag and drop or import file presentation now wants to add files to a page that is not yet fully loaded (rare, but possible for large sessions), that page now remembers the files it should add and appends them once load is done. these files-to-be-added are also preserved through a session save, if the client is closed before this long-loading page is initialised
updated windows mpv, the reported api version is now 1.09
updated windows ffmpeg to 4.3.1
updated windows release to sqlite 3.33.0
updated windows opencv to 4.4.0
just a little thing--I took the source links out of the release post. anyone running from source is probably pulling straight from the github repo anyway
cleaned up some misc inelegant string code
misc other cleanup
.
macOS shortcuts:
the client's shortcut system now detects macOS-specific 'scroll start/end' states, and will not spam scrolls or errors when these states are held
the client's shortcut system now attempts to detect artificial trackpad scroll/wheel events, and adapts the relative speed of scroll event generation according to the respective trackpad velocity. let me know how the hell this works for you in media viewer etc... (issue #710)
the client's shortcut system now detects Control and Command as separate and reliable modifiers in macOS, with correct shortcut string rendering (issue #717)
.
upnp:
fixed the awful typos in the upnp add-mapping error handling I changed last week. I am sorry for this!
improved the async mappings and external ip fetch routines in upnp dialog. closing the dialog while a job is going on should now be completely ok
upnp dialog add, edit, and delete actions are now async (they won't hang the UI while they work)!
all the upnp async jobs should now disable the main list controls while they work
fixed the 'edit' action on upnp dialog to correctly remove old and existing mappings depending on what was edited
when adding a mapping for an (external_port,protocol) that is already mapped, the upnp dialog now asks if you want to overwrite, rather than just failing with a notification
after an async action in upnp dialog, and a mappings refresh triggered, the cached external IP should now be properly restored to the status area
pulled parsing code out of upnp code and wrote some proper unit tests for this so stupid typo errors should not happen again
.
parsing:
subsidiary page parser separation formulae that throw an exception will now be ignored, as if they parsed nothing. in the weird case that you might receive json or html, you can now create subsidiary parsers for both types, and the one that fails will do so gracefully and silently
URL Classes now have a key->value 'header override' value. any time one of these URLs is hit, these headers are added!!! be careful with this, but it may solve some tough problems. also, sorry, the URL Class UI is becoming a hellstack, I need to break it into tabs or similar
.
client api:
added documentation for the new add_files commands, delete_files, undelete_files, archive_files, and unarchive_files
the new siblings and parents taglist menus now copy just the actual tag when you click, excluding the 'ideal/child/parent:' prefixes
added a checkbox to _options->files and trash_ that allows you to automatically prefix hashes copied to clipboard with their hash type in a booru-lookup friendly manner, such as "md5:2496dabcbd69e3c56a5d8caabb7acde5"
the media viewer now remembers if it was previously maximised when you set it to un-fullscreen (before, it would always restore-window-ise)
fixed the 'test address' button in _manage services_ for hydrus administration services
improved the 'add upnp mapping' error handling to better catch 'already mapped' error, with separate errors for redundant, already-on-but-wrong-port, and already-on-another-computer
improved error handling when saving objects to the database, particularly for encoding or giganto-size-session errors
rewrote my tag sibling lookup unit tests to deal with more situations
wrote similar fairly comprehensive tag parent lookup unit tests
.
new downloaders:
rolling in a user-created thread watcher for warosu. it may be CloudFlare hampered depending on your situation
rolling in a prolikewoah thread watcher
rolling in a smuglo.li thread watcher
.
multi-column lists:
spent a bunch of time cleaning out how I calculate multi-column list preferred initial width/preferred current width/minimum width, and made the final column more flexible in its resizing. instances of dialog suddenly getting gigantic because of a final column that wants to size itself at 1,000px should be completely gone, and lists that are shrunk due to non-last-column resizing will now adapt to this situation and not try to flex back to total initial width.
multi-column lists now have horizontal scrollbars again for those situations where the parent window is thinner than their (now better calculated) minimum size
improved the multi-column list num_rows height calculation, it should have less empty space at the bottom for lists that grow as items are entered into them (such as in the download pages)
.
manage tags megajob speedup:
sped up manage tags final application step when entering many tags for many thousands of files at once
optimised UI-side per-file tag cache (re)generation, reducing overhead and surplus work
granularised UI-side per-file tag cache (re)generation based on the four current tag display contexts--now, if a system (e.g. manage tags dialog) only needs storage tags, the different display tags do not need to be regenerated
optimised all tag filtering, which is also used in UI-side tag cache regen
overall, giganto manage tag dialog jobs should now be faster in several ways. on my dev machine, adding 6 tags to 10k reasonably tagged files went down from 52s to 4.8. even larger jobs will still need a lump of CPU time, but they should scale more efficiently (what was previously O( num_tag_changes x num_total_mappings ) is now O( num_total_mappings ), and better at that)
when a huge number of tags is added at once in the manage tags dialog, 'recent tags' is now populated more carefully
in _options->gui pages_ you can now set the main window's page tab alignment to top/left/right/bottom (previously it was just top/left). this property now updates for all page of pages on options ok, it no longer needs client restart (issue #642)
the maintenance task that migrates tag display from the current values to the ideal application now works in significantly smaller steps. big lag from adding hundreds of childen to one parent (or similar siblings) should now be radically reduced
rejiggered some layout in the new tag display dialogs
added green/red texts to the new tag display dialogs to talk about when sync can work atm and how fast to expect changes to apply
reordered the new tag 'siblings/parents info' right-click menu so the dynamic 'has x siblings/parents' submenus are on the bottom
added basic client api calls for /add_files/..., delete_files, undelete_files, archive_files, and unarchive_files. they take 'hash' and 'hashes' parameters. I am throwing these out at the end of the week, so they don't have documentation or proper unit tests, but feel free to play with them (issue #393)
sped up some UI refresh on content update for very large sessions
sped up right-click tag/file menu any/all select actions on very large file sessions
tl;dr: you don't have to do anything. if you haven't heard of a tag parent before, no worries. the database should work better now
.
top level:
parents are now completely virtual! this means that when you add a tag parent, the tags that 'fill in' to make it show do not really exist in storage, only in a computed cache. if you decide to undo the parent, the implications are recalculated and the virtual tags disappear, with no permanent changes made. also, petitioning a parent will 'preview' the delete, just as siblings now does
siblings and parents are now unified, and the logic is improved. all parents apply to all siblings, so no more worries about retro-active filling-in. the siblings and parents code is now basically 'nice'. this was a lot of quite complicated work, and it solves a number of lingering issues from the original prototypes I made several years ago. I will still do some smaller work and little fixes I am sure in the near future, but the 'big' siblings and parents work is done!
like with the recent siblings change, the client no longer needs to do the 'loading parent tags' step when booting--everything is now handled at the db level
like with the recent siblings change, you can now edit which services apply their parents to which service, now under _services->manage where siblings and parents apply_
in the _manage tags_ dialog (and some other places), tags with parent implications now show a '(x parents)' after their label, much like the 'will display as' sibling suffix. I do not like this, but I ran out of time. I hope to add a more advanced actual listing of virtual tags with a nice 'ghostly' colour or similar in future
right-clicking on a tag in a specific tag domain now shows a 'siblings and parents' submenu with detailed info on all known siblings and parents in that domain
'tag' menu entries are moved from the top 'services' menu to a new 'tags' menu. 'pending', when available, is also moved right
the process of changing siblings or parents, or which services apply where, is no longer a CPU-laggy process! actual changes, however, may not appear immediately. a maintenance task now tracks what is currently applied and what is 'ideal', and slowly migrates to the ideal in the background in little chunks. in most situations, the changes are very quick, but if you are behind due to big recent changes, they may be delayed. you can manage when this maintenance runs and see the current status under _tags->siblings/parents sync_. this is an entirely new thing, so feedback on IRL work would be appreciated--there may be some kinds of siblings or parents that cause a whole bunch of annoying lag
the PTR has a lot of non-virtual parents that were hard-added in older versions over the years. most are fine, but some are like the 'shadow'->'shadow the hedgehog' debacle. now the source of the problem is fundamentally solved, this problem will reduce over time. with luck, before the end of the year, no more will be added at all, and thanks to the janitors, the worst offenders should be chipped away
during all this work, a bug with tag siblings and parents repository processing has been revealed (some users do not 'get' all siblings/parents for some reason). now the system is nice and undoable, this will be more easily addressed in coming weeks, with automatic retroactive fixes rolling out to all clients
.
boring details:
like with siblings, wrote a parents structure object that constructs the parents tree without loops more simply and reliably. it populates a new parents quick-lookup table in the database, for which a full suite of lookup and maintenance methods are written
parents and siblings virtual tag presentation is now unified into a single 'display' (i.e. vs 'storage') system with a more granular tag implication algebra (essentially 0-n rows of 'if A is in storage, show B in display' for every tag) that can calculate new and updated display tags and counts without having to do the expensive 'clear-and-regen' that 408-413 used
wrote functions to quickly add or remove a display implication to the 'all known files' or specific file service tag display cache
migrated all the combined and specific tag display cache update code (add/remove files, add/remove mappings, add/remove sibling/parent, add/remove sibling/parent application, and misc regen maintenance calls) to use the implication system instead of the sibling 'ideal' system (basically moving from 1->1 to 1->n)
completely rewrote the complicated 'all known files' cache 'with tags' and 'with and without tags' lookup routines to use much less overhead in general and to use a single, albeit complicated, count-based query that carefully chooses whether to select the 'with tags' and 'without tags' portions using tags or files where available as the primary selector based on existing autocomplete count data
replaced all usage of the old ui-side 'tag parents manager' object. as parents pop in virtually and do not need to be bundled intentionally to various content updates, this was mostly just clearing now-surplus code, but for instance in 'write' autocomplete searches, the parents that appear below search results are now generated at the db level on first search, rather than looked-up live in UI time
the parents and siblings lookup tables are now split into two views: what the display cache currently holds, and what it ideally should hold. when adding new sibling or parent data, only the fast ideal table is changed
a new complicated maintenance function now takes actual and ideal data for a particular unsynced tag, hashes out the implication changes needed to effect a migration, and performs it
a new maintenance manager and accompanying db code now track and manage calls to migrate actual to ideal display presentation, and to update UI afterwards
as tag display changes are now more frequent, I have made the routine that refreshes tag UI after sibling/parent changes more efficient. tag display now only refreshes for files that have the affected tags in a particular change
wrote the UI panel and dialog to show and hurry up current sync status, and all the background hooks for that
added 'tag parents lookup' entry to the database 'regenerate' menu. this routine and the 'siblings' variant are now very quick thanks to the new actual/ideal maintenance system
updated my sibling unit tests to deal with the new actual->ideal syncing
improved the speed of mappings cache updates when deleting files
deleted all the old combined/specific 'regen chain' code and the sibling-based 'get sole/any tagged files' search code
optimised and generally cleaned a bunch of the new cache code, particularly cutting out overhead for unusual/small situations
fixed a counting bug with 'all known files' tag counts when rescinding pending tags
fixed a bug in the siblings display code where deleting or pend-rescinding all of the multiple tags that have the same ideal sibling in the same transaction (e.g. if both A and B sibling to C, and a file has both A and B, and you remove them in one manage tags dialog apply) would not remove the current/pending ideal (issue #571)
the 'add_siblings_and_parents' parameter on /add_tags/add_tags client api command is now obsolete! the help is updated to reflect this
cleaned up just a bunch of db/ui/tag code mate
.
the rest:
fixed an issue where long-running 'similar files' search was not cleaning its memory use properly as the job was going on, resulting in out-of-mem errors on very large clients (issue #669)
thanks to user submission, rolling in a fix for the default pixiv tag search downloader
cloudscraper updated to 1.2.48
removed surplus executables from linux and macOS builds (win32 upnpc exe was causing anti-virus false positive on mac lmao)
added 'sort by number of files in collection' file sort type. it obviously only does anything interesting if you are collecting by something
when you enter a tag from a manage tags suggested tags column with a double-click, the tag input box is now immediately focused. entering it with a keyboard action does not move the focus
wrote a new routine for the 'check and repair' database menu that scans for and fixes invalid tags. this might be some system:tag that snuck in, superfluous unicode whitespace, or some weird website encoding that results in null characters, or any other old tag that has since become invalid. tag translations are written to the log
added an experimental 'post_index' CONTEXT VARIABLE to subsidiary page parsers--whenever a non-vetoed post has pursuable URLs, this value is incremented by one. this is an attempt to generate a # 0,1,2,3 series. feedback on this would be appreciated, so I can formalise and document it
added 'no_proxy' option for the options->connection page. it uses comma-separated host/domains, just like for curl or the NO_PROXY environment variable. it defaults to 127.0.0.1. in future, options will be added to auto-inherit proxy info from environment variables
fixed an error when subscriptions try to publish to a page name when a 'page of pages' already has that name
activated some old 'clean url' parsing tech I wrote but never plugged in that helps parsing urls from source fields on sites that start with non-url gubbins
fixed the v411->v412 update step to account for a tags table that has duplicate entries (this shouldn't ever happen, but it seems some legacy bug or storage conversion indicent may have caused this for some users). if a unique constraint error is raised, the update step now gives a little message box and does dedupe work
fixed an issue where the 'will display as' tag was rendering without namespace when 'hide namespace in normal views' was on
fixed a recent character encoding routine that was supposed to filter out null characters
fixed some UPnP error reporting
_may_ have fixed an odd and seemingly rare 'paintevent' issue when expanding the popup toaster from collapsed state--it may also have been a qt bug, and fixed in the new qt:
updated qt to 5.15.1 for windows and linux builds. it fixes a couple of odd issues like 'unclicking' to select a menu item (issue #296)
added session save to holistic ui test suite
misc code cleanup
.
client api:
wrote a client test for the help menu so I can test some basic functions holistically, hoping to stop some recent typo bugs from happening again
did a couple of hotfixes for v412 to deal with some client api url pending bugs. the links in the 412 release now point to new fixed builds
fixed an issue setting additional tags via the client api when the respective service's tag import options are not set to get anything
fixed a 500 error with /add_tags/add_tags when a tags parameter is an empty list
fixed the /manage_pages/get_page_info client api help to show the 'page_info' key in the example response
added Hydrus Web, https://github.com/floogulinc/hydrus-web, to the Client API page. It allows you to access your client from any web browser
added Anime Boxes, https://www.animebox.es/, to the Client API page. This booru-browsing application can now browse hydrus!
the /add_urls/add_url command's 'service_names_to_tags' parameter now correctly acts like 'additional' tags, and is no longer filtered by any tag import options that may apply. that old name still works, but the more specific synonym 'service_names_to_additional_tags' is now supported and recommended (issue #456)
the /add_urls/add_url command now takes a 'filterable_tags' parameter, which will be merged with any parsed tags and will be filtered in the same per-service way according to the current tag import options.
the client api help is updated to talk about this, and the client api version is now 14
updated client api help to talk about http/https
.
the rest:
the 407->408 update step now opens a yes/no dialog before it happens to talk about the big amount of CPU and HDD work coming up. it offers the previous 'full' version that takes all the work, and a 'lite' version that applies no siblings and is much cheaper. if you have been waiting on a PTR-syncing HDD client, this should let you update in significantly less time. there is still some copy work in lite mode, but it should not be such a killer
the 'manage where tag siblings apply' dialog now has big red warning text talking about the current large CPU/HDD involved in very big changes
a bunch of file-location loading and searching across the program has the opportunity to run very slightly faster, particularly on large systems. update will take a few seconds to make these new indices
namespace and subtag tag searches and other cross-references now have the opportunity to run faster. update will take another couple of minutes to drop and remake new indices
gave tag and wildcard search a complete pass, fixing and bettering my recent optimisations, and compressing the core tag search optimisation code to one location. thank you for the feedback everyone, and sorry for the recent trouble as we have migrated to the new sibling and optimisation systems
gave untagged/has_tags/has_count searches a similar pass, mostly fixing up namespace filtering
gave the new siblings code a similar pass, ensuring a couple of fetches always run the fast way
gave url search and fetch code a similar pass, accounting better for domain cross-referencing and file cross-referencing
fixed a typo bug when approving/denying repository file and mapping petitions
fixed a bug when right-clicking a selection of multiple tags that shares a single subtag (e.g. 'samus aran' and 'character:samus aran')
thanks to some nice examples of unusual videos that were reported as 1,000fps, I improved my fallback ffmpeg metadata parsing to deal with weird situations more cleverly. some ~1,000fps files now reparse correctly to sensible values, but some either really produce 1000 updates a second due to malformation or bad creation, or are just handled that way due to a bug in ffmpeg that we will have to wait for a fix for
the hydrus jpeg mime type is now the correct image/jpeg, not image/jpg, thanks to users for noticing this (issue #646)
searching for similar files now requires up to 10,000x less sqlite query initiation overhead for large queries. the replacement system has overhead of its own, but it should be faster overall
improved error handling when a database cannot connect due to file system issues
the edit subscription(s) panels should be better about disabling the ui while heavy jobs, like large subscription resets, are running
the edit subscription(s) panels now do not allow an 'apply' if a big job is currently disabling the ui
cancelling a manage subscriptions call when missing query logs were detected no longer causes a little error
if a long-running asynchronous subscription job lasts beyond its parent's life, it now handles errors better
.
boring details:
improved a pre-optimisation decision tool for tag search that consults the autocomplete cache for expected end counts in order to make a better decision. it now handles subtag searches and multiple namespace/subtag searches such as for wildcards
wrote fast tag lookup tools for subtag and multiple namespace/subtag
fixed some bad simple tag search optimisation code, which was doing things in the wrong order!
optimised simple tag search optimisations when doing subtag searches
polished simple tag search code a bit more
added brief comments to all the new cross joins to reinforce their intention
greatly simplified the multiple namespace/subtag search used by wildcards
fixed and extended tag unit tests for blacklist, filterable, additional, service application, overwrite deleted filterable, and overwrite deleted additional
added a unit test for tag whitelist
extended the whole 'external tags' pipeline to discriminate between filterable and additional external tags, and cleaned up several parts of the related code
moved the edit subscription panel asynchronous info fetch code to my new async job object
cleaned up one last ugly 'fetch query log containers' async call in edit subscriptions panel
moved the edit subscription(s) panels asynchronous log container code to my new async job object
fixed the 'system:(like/dislike) rating = x' search predicate string, which was saying 'unknown' rather than 'like/dislike' in several cases
fixed a 'current_count' error in the new file search optimisation code for tag searches where the tag did not exist for any files in the domain (i.e. autocomplete count=0). thank you to users for helpful reports here
fixed the recent file search optimisation code to handle 'system:time imported' when it was mixed with tags or search predicates that would pre-populate the query file pool with file domain cross-referenced files. sorry for the trouble!
the forced delay overhead for table analysis is reduced from 0.1s to 0.02s. whenever many mostly empty tables need to be analyzed (like on first boot shutdown, when it is usually 100+ tables), it now zips by
.
siblings/tag improvements:
typing a shorthand sibling like 'lotr' into an 'all known tags' 'read' autocomplete - like on a default search page - now reliably discovers and matches text entry to ideal sibling results like 'series:lord of the rings'. this was previously buggy and unreliable--it now allows the match using better db knowledge, even when the merged 'all known tags' services involved disagree on siblings
when typing tags into a 'searching immediately' page that has media, the autocomplete count results that only refer to that media will now match shorthand sibling inputs to the ideal result. media-populated tag search now takes a little bump of extra CPU to fetch results (they are now passed through the db to get nice siblings info), so it is also cached for the duration of your typing (previously, the counts were re-computed on every new keystroke, so this should be significantly smoother to work with on large pages even if that first keystroke takes a moment to give results)
when typing into a 'write' autocomplete, like in manage tags, the process that promotes the entry text and known siblings to that entry and a potential ideal sibling result to the top of the list is now more sane. it now also only uses results with nonzero count. we'll see how this last change works out IRL
when typing into a 'read' or 'write' autocomplete, the pre-search tag insert no longer has sibling insertion/swapping. it was unreliable before, with weird sibling-swapping in the short moment before real results returned. if you have slow results and often quick-add tags into search pages or manage tags, let me know how this works for you
the 'additional tags' tag input dialog off the tag import options edit panel now shows the 'will display as' label
the 'favourite', 'file lookup', and 'recent' tag suggestion panels now show the 'will display as' label
the 'related' suggestion panel, which works on a slightly different system, now shows the 'will display as' label
the 'tag suggestions' options panel's 'favourite tags' edit lookup and list now displays 'will display as' labels and correctly finds service-specific siblings in its results (e.g. you type 'lotr', it also finds 'series:lord of the rings')
all autocomplete tag filtering should be just that little bit faster as you type
filtering cached autocomplete results based on subsequent search text is now faster
autocomplete inputs should no longer return 'ghost' results that have no current/pending count when one of the 'include current/pending' buttons is deactivated
the new database autocomplete predicate generation routine now checks for 'cancel search' signals, saving CPU time as you type
the slow 'regen chains' maintenance tasks now process sibling chains in random order, smoothing out the 500/100,000 progress label, which previously took about 80% of time on the first 20% of ids due to IRL tag distribution
.
the last UI-side siblings work is cleared:
the UI-side tag siblings cache is no longer used. the sometimes multi-second 'loading tag siblings' step of boot no longer happens!
media autocomplete fetches are now asynchronously populated with siblings data via the db
the exact-match and sibling 'insert' predicates at the top of pre-load and post-load read and write autocompletes now rely exclusively on db data for sibling matching
taglists now present 'will display as' labels asynchronously and are better about updating those labels when the list's underlying tag service changes
the taglist right-click menu that shows siblings to copy now fetches that submenu's contents asynchronously from the database
the test panel on a blacklisting tag filter now asynchronously fetches tag siblings to test against from the database
the actual blacklist tag filter test now fetches tag siblings to test against from the database
reworked my custom tag listbox to handle asynchronous text decoration, and unified sibling decoration for media taglists and string taglists
updated my old async updater class to be more flexible for different job types, and cleaned the code that already used it
wrote a simple class for one-shot async jobs
wrote a simple db lookup for UI-side tag sibling chain members
wrote a simple db lookup for UI-side tag ideal siblings
bunch of misc sibling, db, and ui work and cleanup to make all this work
fixed a bug in the new file service filtering code that was stopping file upload commands to file repositories or ipfs services from sticking
fixed an issue with the export files dialog auto-close-when-done function
I think I fixed a possible bug in the boot file location repair/recovery dialog sometimes not saving corrected paths on unusual file systems
file migration cancel button and shut off timer should work a bit more reliably, more to come here
copying subscription quality csv info to clipboard no longer does nice human numbers (you now get 1234, not csv-breaking 1,234)!
may have fixed a very rare 'or predicate' error when opening a dialog with a 'read' autocomplete input, like export folder or file maintenance jobs dialogs
all pages are better about dealing with missing (i.e. recently deleted) services on load, and autocompletes also
error handling from servers with strange character encodings should be better about dealing with null characters
cleaned up the combined display regen chain code
deleted some obselete db code
.
optimisation review:
after more profiling, and thanks to additional input from users, I have done another round of optimisation for the new caches. using a new technique, more than just mappings are sped up this week - a number of queries that were prone to lag spikes should now have much more reliable speed and also be faster when hammered often
.
join and analyse db optimisations:
these are mostly forcing table join orders, which reduces lag spikes, and reducing some related pre-query analysis overhead, which speeds things up more the faster your drive is (up to double processing speed on an ssd). they will affect different clients to different extents, but if your 'related tags' were taking more than a second to load, it should be sorted this week. systems affected:
archiving files
fetching 'related' suggested tags
tag siblings regen/update in about ten places
all mappings processing
additional mappings processing for add/delete, pend/rescind_pend
importing or deleting files that have tags
loading medias' tags for the first time or on regen
loading any media for the first time
num notes searches
similar files search tree maintenance
many general file hash lookups
many general tag lookups
.
other optimisations:
mappings processing
sibling processing
wildcard tag searches, with and without namespaces, particularly when mixed with other search terms
'tag as number' searches, with and without namespaces, particularly when mixed with other search terms
searching for tags when mixed with other search terms
has notes/no notes
searching files on 'all known files' with general file metadata system predicates (like size, filetype)
url class, url domain, and url regex file searches, particularly when mixed with other search terms
num tag file searches when mixed with other search terms
has/not has tags file searches when mixed with other search terms
sped up specific display chain regen significantly, with similar separate current/pending optimisations as last week's for combined
converted specific display cache overall regen to use a copy followed by the new chain regen rather than additive file import
sped up combined display chain regen a little bit
the splash window now updates itself with less UI overhead, so spammy updates (like the new tag regen code) use a little less CPU and fewer UI context switches
the slowest of the new sibling regen & update code has received a full optimisation pass. some sections take 10% less time, some 90%, and one critical query takes 99% less time. overall, several big jobs work much faster, and ptr processing, which slowed significantly for many users, should be back up to a good speed. uploading pending tags (which tend to be for local files) should be much faster in particular. let's do another round of IRL observation and profiling this week, and I'll keep at it
the various 'display' regeneration routines now provide more progress status text, drilling down to the x/y siblings being collapse-counted, or number of files added to a cache, and generally all tag sibling regen got a status update polish pass
optimised the way tag sibling application is set--now, only the tag siblings that are changed need to have their counts regenerated. hence, if you just apply (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes
.
the rest:
fixed the annoying issue with media viewer mouseovers stealing focus/activation from the manage tags dialog. this can now only happen if current focus is on a hover window. sorry for the delay!
updated manage tag parents dialog to state the pairs being petitioned on the 'petition reason entry' dialog
updated manage tag parents and siblings dialogs to have appropriate 'reason' suggestions for petitions (previously, they were inheriting the same suggestions as for add)
ipfs network jobs now have a minimum 'reply' connection timeout of two hours (so giganto directory pushes won't throw an error). connection timeout remains the same, so if the server is hanging on that, it'll still notice
fixed the 'test address' button on the IPFS manage services panel
petitioning an IPFS file when there is no IPFS multihash entry in the db no longer causes an error. now, in this case, the file entry is removed with no change made.
when pending to or petitioning from a file service, a quick filter is now applied to discard invalid files (i.e. (not) already in the service). any weird logical holes where this might occur should now be fixed
export folders now catch and report missing file errors more nicely
export folders now remember the last error they encountered and report that in the edit export folders dialog
.
boring tag siblings optimisations:
optimised the tag manager generation routine to use any common file domains for fast cache lookup for any subset of the files available, rather than falling back to 'all known files' domain when there is no single common file domain
optimised the new 'all known files' display autocomplete cache to use similar faster specific files cache lookups when available
optimised how the 'all known files' display cache regenerates tag sibling chains. it now takes a shortcut when given non-sibling tags and tags where all but one sibling member have zero count, and it can count current and pending counts separately according to the most efficient counting method (e.g. most pre-display pending counts are 0 across the board, so even if current count is a million, the pending count can often be assumed without lookup overhead). furthermore, the 'clever' count has better query planning and less non-sqlite data overhead, and with experimental data is now chosen more carefully. what was previously a 22s job on a test database now takes 5s
deduplicated how new mappings are filtered to all the specific cache domains, significantly reducing overhead
massively optimised a critical - and the slowest - part of the new 'combined' cache that handles add/pend mappings pre-insert presence testing, speeding up the core query about 100x!
reduced some overhead when doing file service_id normalisation in repository processing
split up specific chain regen into groups to reduce memory usage
optimised specific display tag cache 'add file' updates, and thereby basic cache regeneration, to be just a little faster for files that have multiple sibling tags
all predicates made in the database are now populated with ideal and chain sibling information, and this is used for '(will display as xxx)' labels and autocomplete tag search filtering (e.g. you type in 'lotr', it matches an autocomplete result of 'lord of the rings'). there are still some ui-made predicates to figure out, so the old system remains as a fallback
related tags lookup is a tiny bit faster and now populates its predicates with ideal and chain sibling info at the db level
cleaned up some 'fetch related tags' code, might make it a bit faster for large tag counts
cleaned up the way some mapping tables are fetched
unified table/table_name nomenclature in the db code
updated an old data->ui status presentation method (it typically does stuff like "regenning some stuff: 500/10,000"), to not hog so much UI time and not yield worker threads so often when new statuses are coming in real fast
tl;dr: siblings are faster and better now, you don't have to do anything. some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon
wrote the first version of a 'tag display' cache, which stores not your tags as they are, but how they appear after display rules such as siblings, parents, and filtering are applied, meaning this data need not be calculated every time on thumbnail load. this week marks the first concrete step forward in an improvement of siblings and parents storage, and begins with just siblings. all siblings and front-end tags work should be generally faster and more accurate
part one is for tag domains cross-referenced with file domains. it maintains virtual sibling-collapsed mappings and autocomplete counts through mappings added, deleted, pended or pend-rescinded, files added/deleted, and siblings added/removed
part two is for tag domains on the 'all known files' domain (i.e. no file domain). it maintains virtual sibling-collapsed autocomplete counts through mappings added, deleted, pended or pend-rescinded, and siblings added/removed
both parts also support full table drop/regen (under the new database->regenerate->tag display mappings cache) for when my logic inevitably miscounts something. the existing regen 'tag mappings cache'/'tag siblings lookup' commands also regen the display mappings cache, since it relies on them
when tag siblings on a repository are petitioned to be deleted, they are now instantly discounted from tag sibling application (previously, they had to be uploaded and committed to count, now both pending and petitioned offer a quick preview of outcome)
the display cache supports the tag sibling service application rules under the 'services menu', and regen when that changes, so you can now turn siblings on and off, and apply them across services. as a result, the old 'apply all siblings to all services' option is now gone! as parents will undergo a similar change soon, and the siblings changes this week may lead to some undesired parents in the interim, the 'apply all parents to all services' option is also gone
tag autocomplete counts in the form (x-y) due to siblings are eliminated. it will still do it when combining the same merged tag across different services, or when an unnamespaced tag includes how many potential namespaced will also be found
the following search types now obey tag sibling application rules accurately: number of tags search, namespace:anything search, wildcard search, tag search (on a per-tag-domain basis, previously it was globally hacked to all siblings), tag-as-number search. for instance, if you search series:anything, a file that has 'metroid' tag-siblinged to 'series:metroid' will now correctly appear.
the above search types are now exact to how the tag displays. if you have for files that are tagged 'samus' on either tag service A or B, and service B has a sibling for that to 'character:samus aran', searching for 'samus' gets the results in A, 'character:samus aran' gets the results in B. previously it was an expensive logical mish-mash of 'sure, try and get everything behind the scenes'. now it searches what you see
searching for files in the advanced 'all known files' domain currently has no sibling support for the above search types. autocomplete counts should be good, and the results that come up should have the correct tag display, but the actual results are calculated based on storage tags. getting this to work without doubling the size of the db is tricky, so it will have to be ongoing work
all tag siblings are now completely virtual. this means that when a tag comes in via a downloader or other means, it will not be automatically coerced to its ideal sibling in the actual db storage tables (the true tags you see in manage tags dialog), but remain as it is. there is no change in sibling appearance in normal operation--it still _displays_ and searches as its ideal sibling. the same will happen to parents in the coming months, and in the interim period, parents no longer apply across siblings. as siblings can come and go from anywhere, they are now divorced from actual stored tag mappings. in a similar way, the manage tags dialog no longer supports the 'hard-replace siblings and parents' command, nor the 'auto-replace with sibling' command. this may be jarring to workflows and preferences, so please bear with me and let me know what feels particularly bad. and please don't worry too much about parents not always being added in the meantime--I hope to do the same transition for them in four of five weeks, and all gaps will be filled in. also, in the coming weeks, I expect to improve manual tagging workflow by indent-grouping edit-view siblings together (ditching the old 'will display as' text) for easier review and selection, a bit like parents. actual 'hard' siblings and parents that do always get irreversibly renamed/added in storage will come in the future as a separate system
the 'add_siblings_and_parents' client api parameter no longer adds siblings, and soon will be retired completely
I had wanted to completely eliminate the old UI level siblings manager this week, but there are still some systems, mostly tag autocomplete work, that need it and are tricky to swap. I stripped it down, at least, and reduced its update delay to 2 seconds. therefore, the 'loading tag siblings' step of boot still occurs, albiet a _little_ faster. I hope to have it gone soon
this is some complicated code affecting core systems. almost everything 'siblings' is now different. there are likely to be laggy parts, awkward new workflow, and possibly some update or miscount bugs as I iron it out. the good news, now they are all virtual, is that problems are undoable. please report any issues, and I will work on it as I keep pushing on this and on parents
please expect your client.caches.db file to expand in size about 10-30% or so this week. the update itself will take a few minutes as the improved tag lookups and new caches are regenerated from empty
.
boring mostly db optimisation list:
after some thought, moved those new options for tag sibling application down to the db. previously, they were stored in an UI object for convenience, but since everything is going down to the db, it is worth doing it properly down there. thus they reset this week to the default
I also removed that complicated 'all known tags' page in the tag sibling application options--it wasn't doing enough to justify itself
tag siblings lookup cache now obeys the tag sibling application rules and regenerates the appropriate cache when the options change
tightened up the db tag siblings lookup cache and wrote more tools for it. it had a couple of blind spots for getting all siblings in a chain. also optimised the lookup for en-masse tag operations
tightened up my tag sibling structure builder object, which was not eliminating loops but collapsing them to (generally harmless, but not desireable) (A,A) pairs
extended mappings and siblings lookup caches to perform various sorts of tag sibling collapse filtering, to determine files that do or do not have another tag mapping on a tag sibling chain
optimised the existing mappings cache in several ways
optimised cross-domain file cache mappings filtering, and cleaned the code
optimised autocomplete count fetching from the mappings cache, particularly for large result sets
optimised how the combined autocomplete count generates from nothing
optimised how tags are loaded for search results (thumbnails)
optimesed basic tag search
greatly optimised how the mappings caches request cross-domain file cache filtering
broke up the rescind_pending/add mappings job into simpler separate parts, which was needed for accurate display cache counting. this may end up fixing the other weird pending miscount bug we had
the cached 'display' tags are now loaded with regular media results, not generated on the fly on first request (unless in the advanced 'all known files' domain, where it is done quickly on first load at the db level)
converted the db over to using its local sibling lookup cache for all sibling jobs
all data-level content updates to media result objects now occur in the database loop, reducing lag and eliminating a single UI event loop gap when the objects the UI relies on were desynchronised
optimised how the tag and hash id-to-definition cache maintains itself
cleaned up cache code generally
wrote a ton of unit tests to cover construction, tag, and tag sibling operations on the siblings and display caches
wrote a second optimised method for regenerating 'all known files' display cache autocomplete counts from nothing, which, when multiple siblings have wildly different counts (e.g. 50, 100, 200000), instead of counting them all, counts the smaller tags sans the largest, and adds this to the already pre-computed largest count
the old ui level siblings manager has been pared down to some final tools that will be trickier to replace
.
the rest:
fixed the stupid manage tag siblings dialog input/ok bug I introduced last week
fixed the pair preview label in manage tag siblings dialog when it asks to enter a reason for a remove petition
I believe I fixed the annoying recent bug where the top-right hover window would sometimes not position itself correctly on a window size/move until the top hover was shown once
fixed a bug where the 'do you want to do shutdown work?' dialog was not abandoning shutdown if cancelled (rather than yes/no)
updated the 'has free space to do db transaction?' checker, which needs to test device partitions, to do two sweeps--first only fast local devices, then potentially mega laggy network discovery if the mount point is not found (hydev was wondering why it was suddenly taking nine seconds to close his test client!)
fixed another issue with double-clicking some addremove/queue listboxes when no edit button is set--now in this case they all delete on a double-click
fixed a little bad error handling on pending content upload. an error with petitioning certain IPFS uploads is not yet fixed
I am preparing for a new siblings database cache for v408. this will ultimately make siblings (and parents) faster, more accurate, more powerful, and simple to undo. I have decided, as part of it, to make siblings and parents completely virtual (i.e. the tags won't exist for real, they'll be implied). better tools to manage hard-replace siblings and parents will come later, as trying to support both situations at once has not been excellent
.
created options to hold per-service sibling and parent preferences, so you'll be able to set up '"my tags" siblings and then "ptr" siblings apply to "my tags"' or 'no parents apply to this service'
wrote UI for the sibling options under 'services->manage where tag siblings apply'. you can play with it if you like, and it saves values, but it is not plugged in yet and makes no changes
siblings logic is a little tighter. the db and gui side of siblings structure calculation is more unified, petitioned siblings are discounted properly on all generation, and the db side now resolves conflict decisions the same on every regen. the gui-side still runs on an older structure, but will be updated to exactly mirror the db
updated and unified how large numbers of raw tag siblings are fetched in the database. it also supports fast tag slicing, speeding up sibling cache maintenance. the siblings lookup cache now uses this method for regeneration and update calls
.
the rest:
tag right-click menu copying now supports all combinations of selected/all, tags/subtags, and no_count/with_counts where appropriate (issue #325)
if the media viewer is too thin for the top hover window to fit into its space, the top-right hover now drops down below it. I don't really like how this looks, and will probably instead figure out a flow layout so the toolbar buttons always fit, but at least they are now accessible (issue #388)
altered the above fix--if the top-right hover window can be shrunk to fit in the available space, it will now squeeze in, only bumping down if it can't
moving the mouse off an activated (e.g. clicked) hover window now instantly activates the main canvas. this should fix up some fast swallowed clicks and annoying click-to-activate issues with the center-right duplicates hover window, which does not hide (issue #384)
the duplicates hover window now positions correctly if its min size is too wide to fit in a thin media window
if you make changes to a parser or content parser, there is now a yes/no confirmation when trying to cancel the dialog
fixed an issue where 'queue' listboxes with no edit button would throw an error on double-click. now double-click in this case deletes
fixed a couple of timestamp convertions that were doing YYYY/MM/DD instead of the more ISO-nice YYYY-MM-DD. also, when in UTC, they'll correctly say UTC now instead of GMT (issue #369)
fixed some borked centered text layout on ratings dialog and import folder dialog
fixed the manage services dialog's wrong headers for type/name columns
added links in the official help to the new user-written simple help guide at https://github.com/Zweibach/text/blob/master/Hydrus/Hydrus%20Help%20Docs/00_tableOfContents.md
moved object tag and ratings code to a new client module, 'metadata', and pulled various ratings gui code into a new separate file
refactored some more manager code around to generally more sensible locations
did a bit more work chasing down the highlight-downloader ui deadlock, which unfortunately still exists
reduced the number of db hits some paged downloaders need, particularly on highlight and init
updated some test code to support cleverer db testing
updated mpv for windows build. api version is now 1.109. this fixes at least one weird linux vm audio driver issue
the manage subscriptions dialog now has a 'deduplicate' button. it is enabled whenever your subs of a particular downloader contain duplicate queries. it launches a semi-bananas but thorough 4-step process to ask if you want to do upper/lower-case deduplication, then which downloader, then which queries, then which master sub(s) to retain the queries
subscription dedupe within the same sub keeps the query with the most files
the manage subscriptions dialog also now has a 'lowercase' button that coerces all queries of the selected subs to lowercase
when pasting a list of queries into a subscription, the 'already in sub' test is now caseless. pasting "Samus_Aran" into a sub already with "samus_aran" will not add anything
.
misc:
url classes now have a checkbox to keep fragment data (e.g. "#kwGFb3xhA3k8B") during URL normalisation. this data is not sent to the server and is not useful for almost all sites, but for sites like Mega, it contains useful clientside javascript navigation or access info if you open the URL in your browser
fixed video resolution parsing for some unusual SAR files. this stretches a video slightly, usually when the vid was created or converted with older analog tech (e.g. NTSC)
fixed rating system predicate label for 'rated/not rated'
the issue where miscounts in pending upload data would persist, sometimes leading to an annoying 'pending (13)' style menu that would not clear without debug action, is now fixed in a cheap way. on any upload action, this cached count is reset. a fix for the actual unusual miscount situation will have to come later
the different in-memory manager objects now save changes at different time intervals--lightweight things like favourite searches still save not long after any change, but column widths, network sessions, and bandwidth use now save only every ten minutes
I _may_ have fixed an issue with favourite tags not sticking correctly or resetting when added en masse via the tag right-click menu
I believe I fixed a rare but permanent ui hang on highlighting a gallery or watcher when that same downloader was spamming through a largely 'already in db/previously deleted' list
copying tags 'with counts' now works correctly for simple tag views (previously, it only worked for 'predicate' views)
copying tags now preserves the tag order as in the list (previously, it did a human sort)
to stop status-sorted gallery and watcher list entries bouncing around so much, they now just say 'working' in their status column when they are working. the highlight panel still reports granular file/gallery info. galleries also now say a more solid DONE when complete, to spot them more easily
the gallery and watcher search/checking column now includes stop status in sort
fixed the dowloader link in the help to https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders
added that same link to the Lain dowloader import panel's help button
updated cloudscraper to 1.2.46
updated cloudscraper interfacing code to adapt for new reCaptcha->Captcha object names
.
boring code cleanup:
refactored downloader gui code to its own file
refactored network gui code to its own file
refactored service gui code to its own file
finished import reordering. now all files import in a cleaner order
further reworked all hydrus imports to be more breadth-first, loading core modules earlier and catching potential errors in nicer places
checkbox selection is now wrapped in the 'quick' dialog system, and all checkbox selections now use this single method
simplified and unified a variety of layout code, and fixing some odd layout expanding bugs
system:number of tags now supports namespaces, for example 'find files with two character tags'! (issue #280)
it also supports wildcard namespaces, as now do regular namespace search predicates. both run faster. "crea*:anything" is now possible
system:number of tags has been optimised, and in many cases is now ten to a hundred times faster
system:number of tags still does not support siblings, something I hope to start correcting as of v408
both tag existence (numtags =0 or greater than 0) and tag count database routines now respond quickly to 'cancel search' commands, so if you do run a slow query (a bare 'has creator tag' search on 'all known files' on the PTR, for instance), you can now back out quickly after the 'stop' button appears
note that 'system:number of character tags greater than 0' and '= 0' are equivalent to +/-character:anything, which will be swapped in if you enter these. also, +/-unnamespaced:anything can now appear
the program is a bit better about determining =0 and greater than 0 and less than 1 being 'none' and 'any but none', when it needs to determine optimisations and special labels
unfortunately, I am taking away the default value for system:num tags in the options page (edit: I am killing the whole panel now). this old ugly mess of stacked predicate edit panels works on ancient, difficult to update code, so I will retire it and replace it with a unified system that is easy to use, supports in-search system predicate editing, and keeps up with changes automatically
system:number of tags is now comfortable with redundancies--if you add >2 and >4, it now knows that >4 is the true lower bound (previously, the one used was random)
boring code changes here:
updated tag existence and tag count searches to take advantage of the tag cache when in a specific file domain (which is pretty much all the time), which should speed them up significantly
updated tag existence and tag count searches to more carefully plan their queries, speeding them up both in advantageous and difficult situations
cleaned up tag existence and tag count code significantly
updated all edit system predicate panels to return full predicate objects, a step towards decoupling them and allowing in-place system predicate editing
wrote a new number test object to hold and help with number range test values. num tags now uses it, and eventually all range predicates will too
the namespace existence search code ('anything' queries) is now folded into the new generalised tag existence search code
streamlined how the search context propagates through all database tag searching--now, most queries do not know or care about domain or current/pending status--they just iterate over n tables as determined by a specialised routine
added a handful of unit tests for the new namespace num tag searching
.
database repair:
the database menu has a new entry, 'repopulate truncated mappings tables', under the newly renamed 'check and repair' submenu, which will try its best to 'fix' a client.mappings.db file that has been truncated due to hard drive fault by repopulating from the local-file-only tag cache. do not run this unless you know you need to
the 'help my db is broke.txt' document has a full update pass. the language is clearer, common issues and questions are better addressed, two new recovery routines are added, a section on the stages after boot recovery (like the new repopulate job above) is added, and I added my stock 'now become a backup patrician' nag at the end
the debug routine to clear cached service info numbers is now moved to the 'regenerate' database menu. this thing fixes hanging incorrect 'pending' counts until I can fix it properly
.
the rest:
fixed an issue where when you pasted queries into a subscription, those that were already in the sub (and got the dialog saying so), were being added anyway! I believe this bug came in the last few weeks, after the data storage rewrite. please check your pasted-into subs for dupes
fixed tab double middle-click behaviour (so you can spam page close), which I thought I had fixed last week but actually messed up completely right at the end (issue #314)
cleaned up some more of the page tab event code--it was a mess all around. should all be on Qt now, no wx hacks
network jobs will no longer wait for and consume bandwidth start tokens while all network traffic is paused. all bandwidth competition now halts. (previously, they would continue to consume tokens according to current rules and then all rush to start as soon as traffic was resumed)
fixed some client booru/client api requests to correctly 404 on missing file results, rather than 500
cleaned up some file sort code and fixed the sort string conversion, which was rendering the opposite sort direction (asc/desc) in summary labels (e.g. on manage favourite searches)
cleaned up some ui layout stretching code, including some borked tag import options expand sizing
improved some button and padding layout definitions, and improved, slightly, the way the top-right media viewer hover window lays itself out and changes its size on media change
improved some review services layout. should be fewer weird heights and widths in unusual situations, and the new multi-column list fits better
the manage subs dialog now saves its changes to db more cleanly and atomically
updated the default derpibooru parser to pull species tags. ten points if you can guess what that is most of the time
all multi-column lists across the program now remember the widths of their columns when they are next recreated
the last column of any list is now universally the 'stretching' column, which should correctly initialise with its preferred/previous size, but also grows and shrinks with the parent window
while all lists retain their initial rows height, and those in the gallery and watcher management panels will continue to grow and shrink in a fixed way, all lists in dialog windows can now be shrunk down to four rows
the minimum size of any column is now much smaller, about three characters
all column headers now tooltip their name
lists should be better about sizing in non-100% OS UI scalings
the lists that are automatically sorted (e.g. the download pages, and manage subs) now remember the last sort you gave them
future plans, now within reach:
all lists will sort, sort arrows will appear on the header, and sort will be faster
columns will be rearrangeable
columns will be hide/showable, and initially hidden complicated columns will be available
there will be some, maybe optional, capability to have lists sync live, so if you edit one, the others do the same
num rows height memory, maybe--we'll see how the above shakes out first
boring code changes:
moved list code to a new sub-module
wrote a status object for column list current columns, widths, and sort, and plugged it into list code
wrote a manager to handle column statuses, and plugged it into the main controller and db
wrote definitions for every list (66 or so different lists!) and all their columns, and unified width, name, default sort, and future hideabality and default hide/show status to that one easy to edit and extend location
rewrote list column and sort initialisation to work off the new status object and added hooks for list sorting and column resizing to save new status back
rewrote every list column instantiation to use the new system
numerous misc column list code cleanup
.
the rest:
double middle-clicking on the page tab bar should now correctly close two tabs in a row (rather than opening the rename page dialog on the second)
entering an odd number of hex characters into system:hash no longer causes an error. this will be changed in future to properly highlight and explain badly pasted or incorrect-length hashes in future
the new red text for non-functional status texts in review services now properly re-colours itself between normal/red when an error or resolution occurs while the panel is open
hydrus now knows if it is running in the Haiku operating system and has preliminary platform specificity. if you are interested in helping to get hydrus running properly in Haiku, please join in with github issue #358
cleaned up a mix of smaller code, unused variables and imports and so on
shortcuts have a backend update this week. a bunch of hacky stuff is now cleaner behind the scenes, and the related UI has some cleanup as well
converted all 100-odd simple shortcut commands from hacky text ids to a proper enumerated id system, and across every single instance across the program
wrote nicer descriptive labels for all simple shortcuts. gone is 'focus_media_viewer', now is 'keyboard focus: to the media viewer'
if you have no like/dislike or numerical services, the respective application command edit panels now say so and do not allow an ok action
like/dislike rating sub-panels now start with 'like' checked
when a like/dislike or numerical rating sub-panel is set to 'remove', the action dropdown is set to 'set' (rather than flip) and is disabled, as is the numerical slider
application commands now state better "3/5" information about rating actions, rating than the underlying "0.6" float implementation
all application commands existing in shortcuts or elsewhere are updated to the new enumerated id system
refactored ApplicationCommand (the side of shortcuts that holds the actual action to be done) and its edit UI to new separate files
completely refactored the application command edit panel, pulling the simple/tag/rating sub-panels into their own decoupled classes, simplifying the tangle and permitting easier future expansion
rearranged some application command functions and contant definitions to more appropriate locations
improved how application commands are interrogated by the objects that process them
added plenty of type hinting around application command processing code
cleaned up a bunch of shortcut and application command code, including some wx->Qt updates as well
.
menu and UI cleanup:
removed an old wx hack that prohibited last-second ui updates. the exit splash screen now reports final db shutdown info
if a service or account is currently non-functional (e.g. all repositories are paused), the appropriate status text is now in red
if there is work to do the first time a duplicate page is opened or looked at, it now moves to the 'preparation' tab
doing a 'migrate database' file migration now temp-closes the migrate db dialog and hides the main gui while it goes on
brushed up the tag filter ui a bit--now only one of the tag_filter/blacklist test phrases only show up, in the appropriate context, and the test text input now supports multiple newline-separated tags (e.g. if you want to paste a bunch)
every panel on review services now has a refresh button to force an update
the 'clear trash' button on the trash review services panel is now disabled when there is nothing to clear
updated edit subscription panel to point to the main html help and brushed up that help to talk about file limits more, also the earlier downloader help has a little section to highlight subscriptions and their use
reworked the 'restore from db backup' command--it is now integrated into client shutdown proper, and reports its basic restore progress to the exit splash screen
reorganised the 'network' menu. manage subs is now up top, downloader submenus are now split better into high-level vs component-level, and login stuff is pulled to its own submenu
put 'network traffic' at the top of the network->pause menu
rearranged some of the 'gui' and 'gui pages' option pages and tucked everything into box sections for clarity
the search pause/play button on search page tag autocomplete now has a simpler 'search paused' label when paused. the code has a similar nomenclature change, and eventually this will turn into a simple pause/play icon button or similar
fixed some weirdness with floating autocomplete dropdowns sometimes not appearing on dialogs on first load
fixed some focus logic so set-focus calls on downloader pages should work again on the query input text box and elsewhere
unified all numerical rating->stars and stars->rating calculations across the program. this may have fixed some edge-case bugs
unified all rating string generation across the program
.
the rest:
the disk cache options under _options->speed and memory_ are now default off and force-set off for all users on update. as more users are on decent ssds where these options are of limited value (and sometimes negative value), I now only recommend them for users on HDDs
added two options for autocomplete results list height to 'gui pages' option page, under the new 'controls' section
fixed a critical issue where the client api could duplicate-add tags with url imports to multiple services. the potential service duplicate cascade order was pseudorandom and particular to a client. thanks to a user for figuring this issue out (issue #317)
added a 'tag whitelist' to downloader tag import options. its edit button is below the blacklist. when there are no tags in the list, it does nothing, but if tags are added, then files that do not have at least one of the given tags at the download source will not be imported. for instance, if you have a username-based downloader (where you can't add more tags to the query to filter serverside), and you only want their metroid content, you can now filter it simply hydrusside (issue #279)
if you are both in advanced mode and a mad lad, the basic blacklist tag filter now allows you to show the 'whitelist' and 'advanced' panels again, if you have a complicated blacklist to set
the local booru and client api now support the same https as the hydrus server, using self-signed certificates stored in the db directory. just set the checkbox in manage services and you should be good. self-signed certificates are free and will work on a server hosted off an IP address, but they are imperfect. they are also likely to require special permission to be accepted by the web browser or whatever you want to talk to the https service. however, if you host your client from a real DNS domain and have your own fully signed cert+key files, you can swap them in no problem
local booru and client api urls adjust scheme for the new option, and unified and cleaned up how booru share urls are generated internally
the way cert+key files are generated is moved from server code to common hydrus code
cleaned up how additional db files like certificate files and the mpv conf are managed for backup/restore operations
cleaned up some ancient http urls to https. mostly stuff like the regex tutorial links
when files are appended to a regular search page (e.g. from a subscription publish to an existing page, or from a mouse drag and drop), the search context will now pause. this is to stop accidental F5 or mass refresh signals wiping out the changed page
to break advanced-case gallery search loops, gallery url jobs now have a 'run' identity token. galleries pass their token down to 'next page' or 'sub-gallery' urls they generate, meaning all urls of a particular search run share the same url. gallery logs now ignore to-be-added urls that already exist for their token, terminating loops. new tokens are generated if a search is restarted or similar, meaning duplicate urls can exist in a gallery log, just not from the same starting point (issue #302)
improved simple gallery url deduplication in several stages of the downloader pipeline
when right-clicking on multiple thumbs, the info lines off the top menu item now list the files' combined viewtimes (this previously only showed when one file was selected)
fixed some error reporting problems with adding urls to import via the client api--some url class exceptions were being converted from 400 to 500 errors unintentionally
a new stylesheet, 'Hydracula', is added to the default install. check it out under options->style. thank you to a user for contributing this
subscriptions are better about calculating a 90 second forgiveness window for bandwidth rules. they should schedule and startup more effectively, and the edit subscriptions and single edit subscription panels should also no longer show bandwidth delays below the next 90s, which are often a technical situation of regular work breaks that are better ignored for the purposes of the dialog
went back up to pyinstaller 3.6 again on windows, as 3.5 caused its own Qt bindings dll problems. if you had trouble with 3.6 (401), let me know how this works for you, as there are additional dll-finding fixes included (issue #329)
fixed an issue where under some conditions, file save dialogs were only happy with filenames that already existed (issue #319)
fixed an issue with the 'client already running' system sometimes not closing the client process correctly when told to cancel the boot
bumped the 'space needed for vacuum' estimate up to 120% (was 100%) of estimated final file size, just to catch some edge cases
rolling out updated danbooru parsers that pull associable urls correctly, thank you to a user for this fix
rolling out an updated deviant art parser that finds some unusual file urls when other methods fail, thank you to another user for this fix (issue #295)
upgraded cloudscraper to 1.2.42
improved some type hinting
fixed up some unit tests for new command and rating data
in many situations--such as a search result that gives no results, or a search cancel, or a downloader page cleared of a highlight--pages will now report a special status text rather than '0 files', such as 'no results for this search' or 'search cancelled!' (issue #277)
new pages, and the first page of a loaded session, should now correctly publish their status text to the status bar immediately after initialisation, (previously blank until first change)
clicking the 'searching immediately' button while a search is ongoing now correctly cancels a search, cleaning up status and page and buttons, rather than just stopping current work immediately
added 'copy_xxx_hash' shortcuts to the media shortcut set for 'md5', 'sha1', and 'sha512'
when copying file hashes to clipboard, a popup appears for two seconds to verify what happened
when copying file hashes to clipboard, recovery from missing hashes is more graceful, with multiple error report states
the way the client shuts down is untangled. the order in which the gui, managers, threads, database are shut down is smoothed out, with better error handling and fewer potential logical holes
the 'should I do shutdown work?' dialog is now only presented in the clean shutdown pipeline
menu labels now elide at 128 characters, extended from 64 previously. hopefully this strikes a better balance between fixed texts we do want to read while still not letting long dynamic texts go nuts (issue #276)
gallery and watcher pages now have 'show file/gallery log' on their menus, which directly zoom in to the edit dialogs for the top-most selected query or watcher (issue #256)
when file maintenance is forced to run from the thumbnail menu or file maintenance job panel, it now provides x/y progress text and gauge based on total jobs, e.g. 1,234/10,000, rather than out of the 256-job batches (issue #264)
the simple downloader page now updates its pending jobs list more efficiently, and supports multiple selection, and presents a yes/no confirmation on delete
most lists with clipboard/png import/export buttons can now also do .json files. they also accept json files in a drag and drop. you can mix json and png files in a multi-file drag and drop
when selecting a parser for a url class in 'manage url class links', those parsers with example urls that match the url class are now separately listed at the top of the choice dialog
in the recent autocomplete rewrite, the hidden repository update file domain was accidentally exposed in the file domain button. after some testing, it actually works(!), but as this is an advanced topic, it is now hidden behind advanced mode
the way services are deleted or completely reset is now changed to what should be a significantly faster and smaller operation
the latest user-made nitter/twitter downloader is rolled in to the update. some little fixes and adds support for mobile.twitter.com url imports
fixed an issue where uninitialised repositories thought they were caught up
to reflect that it does nothing in this case, the mouse shortcut edit panel now disables the press/release choice on double-click or scroll
fixed file save dialogs not filling in the default filename properly
removed an old wx safety hack where new pages would silently not create while the client was minimised. this fixes issues with large session loading and subscriptions publishing files to page names that do not yet exist while the client is minimised
removed an old wx safety hack where some tag lists would not regen their current tag display while the client was minimised
in lieu of a future better bit of html subscription help that I link to from the subscription panel, the 'file limits' help button has temporarily briefer text so it doesn't make such a giant popup
moving back to pyinstaller 3.5 (from 3.6) for the windows build, which appears to fix some dll loading for some users (issue #244)
the windows and linux builds are updated to Qt 5.15 (from 5.13.2). it does not seem to have the odd problems 5.14 gave us. let me know if you have any trouble or if any weird graphical issues magically fix themselves
.
client api:
the /get_files/file_metadata call has a new true/false parameter, 'detailed_url_information', default false, that adds 'detailed_known_urls' structure to list the known urls results as in /add_urls/get_url_info. it has a help example and a unit test and everything (issue #235)
the client api version is now 13
.
boring cleanup details:
reshuffled the shutdown code. now the controller takes the lead, booting splash as appropriate and commanding gui to save and close, and then proceeds to other shutdown
fast and normal shutdown code is unified, just run differently
shutdown calls should now always be idempotent
a catch for some OS-level shutdown commands, normally user log-off, also hooks into the newer UI-free fast shutdown
SIGINT and SIGTERM also hook better into the new shutdown, and are thread safe
performing multiple SIGINTS on shutdown should no longer throw an error after the gui is deleted
more potential startup/shutdown errors are now caught and presented to the user and saved to log, with subsequent shutdown urgency accelerated afterwards
critical errors on a fast shutdown no longer present to the user--they just save to log
updated how an emergency shutdown state is tested
updated how a 'clean exit complete' state is set and tested
various unusual shutdown states now skip human interaction and jump straight to guaranteed fast shutdown
refactored splash window to its own file
wrote a new qlistwidget subclass to do some common data storage/retrieval/selection. it will eventually replace most lists across the program
the 'queue' list widget that has up/delete/down and add/edit buttons beside a list has nicer backend code and now initialises with its buttons correctly disabled due to no selection
the similar 'add/edit/delete' list widget is updated to use the nicer backend
some wx->Qt list hacks, which were themselves using borked old display-string-based indexing, are deleted
the repository download/process daemon has been moved to the newer job scheduler. it should start up and close out on program exit a bit more neatly
untangled some messy value-change radio button code in the shortcut edit panel
updated the way page status text propagates up from the thumbnail grid to the main gui to Qt signals instead of the old inefficient pubsub
all UI file hash clipboard copying code is now unified and improved
added a new subscription file publish debug test to help->debug->gui
refactored some client specific time delta rendering code out of core to client
as subs can now load more flexibly, previously hardcoded waits are now eliminated:
- the subscriptions manager now only waits three seconds after initial session load to boot (previously 15)
- the subscriptions manager now wakes as soon as the subscriptions dialog is ok'd or cancelled
- a timing calculation that would delay the work of a sub up to five or fifteen minutes if more queries would come due for sync in that time window (in order previously to batch to reduce read/write) is now eliminated--subs will now start as soon as any query is due. if you were ever confused why a query that seemed due did not boot after dialog ok or other wake-up event, this _should_ no longer happen
re-added the import/export/duplicate buttons to manage subs. export and dupe may need to do db work for a couple of seconds and will have a yes/no confirmation on larger jobs
the import button on manage subs accepts and converts the old 'legacy' subscription object format, including a copy/paste of the objects backed up to disk in the v400 update
fixed an issue where creating a subscription query and then deleting it in the same manage subs dialog session would result in surplus data being written to the db (which the next dialog launch would note and clear out)
an unusual error with pre-run domain checking, exposed by the new subscription code and e621 subs, where the gallery url has also recently changed, is now fixed
.
issue tracker:
the Github issue tracker (https://github.com/hydrusnetwork/hydrus/issues) is turned on again! it is now run by a team of volunteer users. the idea is going to be to try to merge duplicate feature suggestions with the proper platform and put some discussion and cognition and prioritisation into idea development before it gets to my desk, so I can be more focused and productive and so 95% of feature suggestions do not simply get banished to the shadow realm of the back of my todo
this is mostly intended for wishlist and other suggestions, as the tsunami was just getting too much for me to handle, but we'll see how it goes for things like bug reports as well. I'll still take any sort of report through my normal channels, if you are uncomfortable with github, or if you wish for me to forward an item to the issue tracker anonymously
the website, help documents, and hydrus help menu links have been updated regarding the issue tracker
.
the rest:
improved how the database 'update default downloader objects' job works, ensuring that new defaults are better at simply take the place of existing objects and do not break/reset existing url class to parser links
tightened up how automatic url class to parser linking works, eliminating some surplus and potentially bad data related to api links. furthermore, whenever the links between url classes and parsers update, existing surplus data, which may creep in when api links change, is now cleaned from the data structure
rolling out updated e621 url class and parser to deal with their alternate gallery url format
rolling out an updated derpibooru parser that will link to the new api class correctly
thanks to a user's submission, rolling out updated versions of the new default nitter parsers that pull creator:username tags
before every subprocess launch, and when waiting for all subprocess communication (e.g. to ffmpeg), now tests regularly for program shutdown. if an unusual situation develops where a subscription is doing a file import job while the OS is shutting down, and that system shut down would hang or is hanging on a 'ffmpeg can't be launched now' dialog, the hydrus client should now notice this and bomb out, rather than going for that never-running ffmpeg. this may not fix all instances of this issue, and further feedback on the client not closing down cleanly with the OS is welcome.
when adding a new path to the 'migrate database' panel, any symbolic links will be converted to canonical equivalents
added some location checks and appropriate errors when the database is doing file storage rebalancing
fixed an issue uploading swfs, video, or audio to the server when it is launched from a frozen executable build
the formerly monolithic subscription object is finally broken up into smaller pieces, reducing work and load lag and total db read/write for all actions
subscriptions work the same as before, no user input is required. they just work better now™
depending on the size and number of your subscriptions, the db update may take a minute or two this week. a backup of your old subscription objects will be created in your db directory, under a new 'legacy_subscriptions_backup' subdirectory
the manage subscriptions dialog should now open within a second (assuming subs are not currently running). it should save just as fast, only with a little lag if you decide to make significant changes or go into many queries' logs, which are now fetched on demand inside the dialog
when subscriptions run, they similarly only have to load the query they are currently working on. boot lag is now almost nothing, and total drive read/write data for a typical sub run is massively reduced
the 'total files in a sub' limits no longer apply. you can have a sub with a thousand queries and half a million urls if you like
basic subscription data is now held in memory at all times, opening up future fast access such as client api and general UI editing of subs. more work will happen here in coming weeks
if due to hard drive fault or other unusual situations some subscription file/gallery log data is missing from the db, a running sub will note this, pause the sub, and provide a popup error for the user. the manage subscription dialog will correct it on launch by resetting the affected queries with new empty data
similarly, if you launch the manage subs dialog and there is orphaned file/gallery log data in the db, this will be noticed, with the surplus data then backed up to the database directory and deleted from the database proper
subscription queries can now handle domain and bandwidth tests for downloaders that host files/posts on a different domain to the gallery search step
if subs are running when manage subs is booted, long delays while waiting for them to pause are less likely
some subscription 'should run?' tests are improved for odd situations such as subs that have no queries or all DEAD queries
improved some error handling in merge/separate code
the 'show/copy quality info' buttons now work off the main thread, disabling the sub edit dialog while they work
updated a little of the subs help
.
boring actual code changes for subs:
wrote a query log container object to store bulky file and gallery log info
wrote a query header object to store options and cache log summary info
wrote a file cache status object to summarise important info so check timings and similar can be decided upon without needing to load a log
the new cache is now used across the program for all file import summary presentation
wrote a new subscription object to hold the new query headers and load logs as needed
updated subscription management to deal with the new subscription objects. it now also keeps them in memory all the time
wrote a fail-safe update from the old subscription objects to the new, which also saves a backup to disk, just in case of unforeseen problems in the near future
updated the subscription ui code to deal with all the new objects
updated the subscription ui to deal with asynchronous log fetching as needed
cleaned up some file import status code
moved old subscription code to a new legacy file
refactored subscription ui code to a new file
refactored and improved sub sync code
misc subscription cleanup
misc subscription ui cleanup
added type hints to multiple subscription locations
improved how missing serialisable object errors are handled at the db level
.
client api:
the client api now delivers 'is_inbox', 'is_local', 'is_trashed' for 'GET /get_files/file_metadata'
the client api's Access-Control-Allow-Headers CORS header is now '*', allowing all
client api version is now 12
.
downloaders:
twitter retired their old api on the 1st of June, and there is unfortunately no good hydrus solution for the new one. however thanks to a user's efforts, a nice new parser for nitter, a twitter wrapper, is added in today's update. please play with it--it has three downloaders, one for a user's media, one for retweets, and one for both together--and adjust your twitter subscriptions to use the new downloader as needed. the twitter downloader is no longer included for new hydrus users
thanks to a user's submission, fixed the md5 hash fetching for default danbooru parsers
derpibooru gallery searching _should_ be fixed to use their current api
.
the rest:
when the client exits or gets a 'modal' maintenance popup window, all currently playing media windows will now pause
regrettably, due to some content merging issues that are too complicated to improve at the moment, the dupe filter will no longer show the files of processed pairs in the duplicate filter more than once per batch. you won't get a series of AB, AC, AD any more. this will return in future
the weird bug where double-clicking the topmost recent tags suggestion would actually remove the top two items _should_ be fixed. general selection-setting on this column should also be improved
middle-clicking on a parent tag in a 'write' autocomplete dropdown no longer launches a page with that invalid parent 'label' tag included--it just does the base tag. the same is true of label tags (such as 'loading...') and namespace tags
when changing 'expand parents on autocomplete' in the cog button on manage tags, the respective autocomplete now changes whether it displays parents
this is slightly complicated: a tag 'write' context (like manage tags) now presents its autocomplete tags (filtering, siblings, parents) according to the tag service of the parent panel, not the current tag service of the autocomplete. so, if you are on 'my tags' panel and switch to 'all known tags' for the a/c, you will no longer get 'all known tags' siblings and parents and so on presented if 'my tags' is not set to take them. this was sometimes causing confusion when a list showed a parent but the underlying panel did not add it on tag entry
to reduce blacklist confusion, when you launch the edit blacklist dialog from an edit tag import options panel, now only the 'blacklist' tab shows, the summary text is blacklist-specific, and the top intro message is improved. a separate 'whitelist' filter will be added in the near future to allow downloading of files only if they have certain tags
'hard-replace siblings and parents' in _manage tags_ should now correctly remove bad siblings when they are currently pending
network->downloaders->manage downloader and url display now has a checkbox to make the media viewer top-right hover show unmatched urls
the '... elide page tab names' option now applies instantly on options dialog ok to all pages
added 'copy_bmp_or_file_if_not_bmpable' shortcut command to media set. it tries copy_bmp first, then copy_file if not a static image
fixed some edit tag filter layout to stop long intro messages making it super wide
fixed an issue where tag filters could accept non-whitespace-stripped entries and entries with uppercase characters
fixed a display typo where the 'clear orphan files' maintenance job, when set to delete orphans, was accidentally reporting (total number of thumbnails)/(number of files to delete) text in the file delete step instead of the correct (num_done/num_to_do)
clarified the 'reset repository' commands in review services
when launching an external program, the child process's environment's PATH is reset to what it was at hydrus boot (removing hydrus base dir)
when launching an external program from the frozen build, if some Qt/SSL specific PATH variables have been set to hydrus subdirectories by pyinstaller or otherwise, they are now removed. (this hopefully fixes issues launching some Qt programs as external file launchers)
added a separate requirements.txt for python 3.8, which can't handle PySide2 5.13.0
updated help->about to deal better with missing mpv
updated windows mpv to 2020-05-31 build, api version is now 1.108
No Comments