wrote ui to review and even edit session cookies by network context. it is still a bit rough but will help with future development.
added a 'open_selection_in_new_page' shortcut to the 'media' shortcut set that will work on the thumbnail view
added a 'export_files' shortcut to the 'media' shortcut set that will work on the thumbnail view
fudged manage siblings logic to not do the borked 'hey, that sibling already exists' as soon as you type the old sibling--it will now auto-petition any existing siblings when you click 'add' with a good automated petition reason that makes sense to the janitor
manage siblings now also only shows rows appropriate to the current selection like parents does. it gets a new 'notes' column to specify conflicts that will be auto-petitioned as above
manage siblings and parents now have a laggy 'show all pairs' checkbox to let you quickly review everything like you used to
fixed a database-level bug that meant petitioned and pending the same left-hand sibling tag (like petition a->b, pend a->c) would sometimes not both save together
when you middle-click or right-click->open a new page on a selection of tags/search predicates, the new page will now be named after the tags
if subscriptions hit their periodic file limit, they will now give a little popup message describing what happened and possible causes and actions the user can take
added a 'make a modal popup in five seconds' action to help->debug
modal popups will now hide/show other child frames (like review services) rather than minimise/restore, as this latter action can raise the entire progam to the front
added a 'make a parentless text control dialog' debug entry to test some key event catching
added a 'layout all tlws' debug entry to help->debug that'll hopefully help figure out some child window sizing/position issues
added a 'shortcut report mode', which will report caught shortcut keys and their matched commands, if any
fixed a little shortcut catching bug in the main gui
finished adding the new bytes control
updated ffmpeg for windows
improved sankaku default bandwidth rules to stop a subscription bandwidth rules mismatch that could sometimes make for subscription delays--existing users may like to add a 2GB/day rule for sankakucomplex.com
improved some service and account error handling to better propagate the exact problem up the exception handling chain
the clientside 'update A actually had hash B' repository sync error is now dealt with in a less severe way, and the bad update file is saved to disk with a request for it to be forwarded to hydrus dev for further investigation
file parsing is now more resistant to invalid negative values for properties like width, height, and duration
improved some focus code that may have been affecting linux stability
wrote a new 'bytescontrol' and related noneablebytescontrol that allows for a uniform way to choose a bytes size value with a wider range--and deployed it all over the place
hitting a mapped shortcut for 'manage file notes' should now ok the 'manage file notes' dialog as well as open it
the manage file notes text control will now start with its caret at the end of the document
manage siblings/parents now state how many pairs they have
as an experiment, manage parents now only lists pertinent information in its listctrl. this is hacky but potentially a big improvement to the workflow here, feedback would be appreciated
added a new menu entry, 'network->pause->all new network traffic', which will indefinitely pause any new network jobs. this value will persist through a restart
'pause subscriptions' is moved from 'services' to 'network'
'pause import/export folders' is moved to 'file' and all the import/export folder stuff is sent down to its own submenu
in lieu of proper session inspection gui, added some debug 'reset login' entries to the network menu for pixiv and hf
if a page has files but none are selected, it will now say the total size in the status bar
the edit bandwidth rules control and its subsidiary dialog use some saner and more user-friendly layout and presentation
the previous search distance on the 'review bandwidth usage' frame is now remembered
fixed some bad logic where a 'copy_bmp' event could trigger despite the current media being None or a non-static image
you can no longer open multiple copies of the subscriptions or import/export folder manage dialogs if you hit the menu entry multiple times while the first is waiting for the jobs to finish
like import folders and subscriptions, manage export folders now waits for currently running export folders to quit before opening
export folders run lighter and quit faster on client shutdown
the domain manager will give a better error if a URL submitted to it lacks a schema (the http or https part)
fixed a bunch of unicode error handling
fixed an issue with similar files metadata orphans prohibiting new file imports
finished up a simple shared shortcut processing object and replaced most old temp duplicate shortcut code with it
finished a prototype 'file notes' system. thumbnails and media viewer canvas now support 'manage->file notes' in their right-click menus. this launches a simple text box which will save its contents to db
added 'manage_file_notes' shortcut to the 'media' shortcut set
tag summary generators now have a simple show/hide checkbox and (for thumbnails) custom colours for background and text including alpha channel!
fixed a variety of timing and display logic related to subscription query DEAD vs next check time calculation
all currently dead subscription queries will be revived on update, just in case they were formerly set dead by accident
the 'fetch tags even if url known and file already in db' option is moved from the download/subscription panel's cog icon to tag import options
cleaned up tag import options layout, controls, internal workflow, and help button
added 'select all/none' buttons to tag import options panels with multiple namespaces
if a subscription is blocked by bandwidth, the manage subscriptions dialog will display that in its 'recent error/delay' column
the edit subscription dialog will show similar bandwidth blocking info, on a per-query basis, under a new 'recent delays' column
the review bandwidth usage panel will no longer show some unusual results by default that you can see with 'show all' hit anyway
the review bandwidth usage panel will show the usage at the current search distance in a new column
the review bandiwdth usage panel will show number of requests after data usage. this might be info-overload, so I might alter the syntax or roll it back entirely
fixed an issue with hentai foundry parser pulling images placed in the image description area instead of main image. this particularly affected the artist 'teku'
tags for deviant art and tumblr and thread watchers, which were formerly stored in volatile session memory--meaning half-completed import queues were losing their tags through a program restart--are now saved to the new import object directly
removed all the old volatile session memory patch code
added the new import object through a larger part of the parsing pipeline
deleted the old remains of the giphy parser--if it comes back, it'll all be rewritten in the new system
harmonised some other import pipeline code to the new system
added a new 'management and preview panels' submenu to the 'pages' menu
added an option to control 'save sash positions on close' to this menu
added an entry to force-save the current sash positions to this menu
added an entry to 'restore' the currently saved sash positions to all pages to this menu (this is useful if your window resizes real small and all your pages get crushed up)
rejiggered how URL Classes are matched with URLs to make sure some Post URLs are not lost (this was affecting Hentai Foundry Post URLs, which were sometimes not displaying in the media viewer despite matching)
fixed an issue where the duplicate filter page's jobs would not trigger an update after a job finished
fixed an outside chance of a crash after running a duplicate filter page job
improved how strings are coerced to unicode--now the preferred system encoding will be tried before utf-16, which should improve support for characters in various non-unicode sources (like neighbouring .txt files)
fixed an issue with the client's local booru and flash files (and some other file fetching and mime reporting is a bit faster and neater overall)
the options should be more reliable about redrawing all thumbnail banner summaries on an option ok now
the options->media->media zooms option will now remove any <=0.0 values when it saves
fixed up some old test code
improved how some thread-to-gui update reporting code works
deleted some old network object code
converted manage subscriptions panel to an edit panel--a decoupling refactor I will likely ultimately make across the program
wrote a help page for content parsers
did the first half of a help page for page parsers
the 'allow decompression bombs' option is now moved to 'file import options'. it defaults to False
file import options now allow max size and max resolution rules. they default to None
file import options now allows a max gif size rule to deal with THE SFM COMMUNITY. it defaults to 32MB
file imports will give better quality errors if they fail due to file import option exclusion rules
file import fail errors will now also specify the incorrect amount--e.g. if a file is 64KB but the minimum limit is 256KB, it will state this in the error
wrote a bunch of unit tests to make sure the new complicated file import options object works good
finished the tag summary edit ui and added a new 'tag summaries' page to the options
you can set the thumbnail top, thumbnail bottom right, and media viewer top summaries here, and everything should link up right!
the old options to hide thumbnail top and thumbnail bottom right text are removed from the options->gui page
tag summaries now include pending tags again
tag summaries will order tags according to hydrus numeric tag sorting rules (0 < 0a < 0b < 1 < 9 < 10)
tag summaries will collapse namespace groups that all start with decimals into a "min-max" phrase instead of the whole list (so 'collecting' a chapter will list p1-23 for the page phrase instead of each number in turn)
fixed some potential error popups from the autocomplete dropdown when not in float mode (this hit non-windows users more often)
split the parser help pages up a bit more and updated the client ui html links
finished off the formulae help pages
updated to wx 4.0.1 on all platforms. it promises increased stability!
default tag import options are now moved to the 'importing' options panel, alongside the default file import options
the database will deal with and recover from rollback exceptions more reliably. rollback-loops should be fixed, but the original event will now also produce a request to restart the client (which will safely reset everything) anyway
improved how I create and layout the 'help for this panel -->' help buttons
the job scheduler now regularly checks for and clears out dead wx-related jobs (this saves a bit of memory)
cleaned up an old tag retrieval method that was wasting CPU. some larger pages with many thousands of tags should do tag calculation after selection and so on a bit faster
fixed a thread shutdown test that wasn't working until app shutdown time
improved some more thread shutdown code
deleted some old thread-inefficient calllater and wx-safe timer code
emergency server errors that occur during the response writing stage should be handled better by the client
emergency server errors are more robust in general serverside
improved the quality of the errors thrown by the new network engine's login system
improved some test timing and key press simulation code
fixed the runtimeerror popups that would come up on restore from minimise or main gui move after the complete destruction of a general search page
cleaned up some main gui move code generally, and removed a memory leak on the way
file queries can now cancel at multiple checkpoints during the first phase, saving a bunch of CPU time on certain large queries that are replaced mid-search
after a file query has been going three seconds, a little 'stop' button will appear beside the regular autocomplete input. clicking this will cancel the current query! it will stop when it next hits one of the checkpoints above
the floating autocomplete dropdown should be less flickery in some circumstances
dejanked some more file query code
added a 'clear orphan file records' entry to the database->maintain menu. this looks for and purges orphan file rows as you may have seen a notification about recently. this mostly affects the duplicate filter system
fixed up the delete file code to be a bit more robust--it should lead to fewer orphans in future
all the parsing edit panels have new layout: they no longer have info panels but instead a help button that points to the html help, and the edit and test panels are now beside each other rather than in notebook pages
harmonised a bunch of the parser ui test panel code, refactored how the results are stored
the test panel now presents a better 'preview' of what it contains (the actual text control has like 64KB text limit on some OSes and has unreliable text encoding rules, so using it as the raw container for the example data has lead to problems), and we now read and write the example data with a couple of new copy to/paste from clipboard buttons
wrote another new test panel for subsidiary page parsers that does the separation formula stuff a bit better. the test results now come back for all posts as well, rather than just the first
added a new 'deeply_nested_dialog' frame key to options->gui for the parsing ui to better lay out five or six nested dialogs in a nice 'topleft' way
the 'topleft' frame padding is reduced from 50 to 24 pixels to better fit in deeply nested dialogs
misc parsing ui improvements and little fixes
the manage url classes and manager parsers dialogs now have a better 'add defaults' button that allows you to just select the defaults you want (by name) from a checklistbox
wrote a parser for 420chan and added it to the defaults. it should automatically add and link up when you update
if the install_dir/db directory is not writable-to (e.g. you have installed the program to a protected location like "C:\Program Files"), the client and server will default to ~/Hydrus as the db directory
wrote a new 'TagSummaryGenerator' class that will do 'bunch of tags'->'nice summary string' conversions for the thumbnail banners and export filenames
substituted some static tagsummarygenerators to do thumbnail banners
did the same for the media viewer top-center namespace summary
started some edit ui for tagsummarygenerators--I'll have some proper customisable stuff in the near future
moved the background memory maintenance and misc daemons to the new job scheduler, reducing thread count and idle CPU some more
added a debug 'show scheduled jobs' entry for the new job scheduler system
decompression bomb failures will no longer count towards a subscription's fail count, so having a bunch of them won't abandon a sync
fixed and otherwise improved a potential crash condition when a thumbnail panel closes while a menu is popup'd on it
to forestall this program instability, the thumbnail window will no longer replace while its menu is open. the behaviour after this delayed window delivery is slightly borked, but it isn't a crash so I'm ok with it for now
removed some other jank from the thumbnail media panel swap code
non-cancellable modal popups will no longer have the 'close' button. trying to close them with the dialog's X button will still give the 'sorry lad, can't cancel' error
rating and file service system predicates for services that no longer exist will now render a neat 'unknown x system predicate' presentation string rather than throwing an error
searches in 'all known tags'/'specific tag domain' no longer provide system:untagged, wew
some delayed events are now posted in a more thread-safe way
fixed up some import folder logic--they now run 'look for new files' checks separate from 'import anything still in the queue', so they can now catch up on outstanding files more easily
the ten minute file-processing break is reverted--import folders now just 'save' every ten minutes, to forestall lost work on a crash
import folders now have an explicit 'check regularly' checkbox to control whether you want it to check regularly or only when you tell it to. the paused status now means 'do not ever work'
import folders now make a 'working' popup, like subscriptions do, that shows new file discovery progress and found file import progress. it has a cancel button that will stop the current job and can be hidden in the import folder's options. existing import folders will default to ON for this, so you'll likely see one of these right after you update
import folders can now publish their files to a new page! this effectively cuts out the button step. furthermore, if a page already exists with the import folder's name (e.g. a previous page the import folder created is still around), the import folder will publish the new files to _that_ page, updating it
file popup buttons with the same name will now 'merge'! multiple work cycles of a subscription or import folder will now just update the first button rather than spamming several
running import folders will respond faster to a client shutdown event
wrote some better controller-level thread management, including surplus thread deletion--idle CPU should be reduced on import-busy clients
wrote some code to deal with sub-second times a bit better in certain places
finished off a new scheduled job queue that collapses an old multiple-thread-when-idle system down into just one
many things are moved to the new job scheduling system--all the old calllater calls, and the popup message timers as well
completely removed the old wx timers the autocomplete input was using, as they were just too much of a hassle. any crashing these were causing is now gone--it all works on the simpler scheduled job queue now
the autocomplete dropdowns are better at judging when to engage their timers and so now use far less idle CPU
fixed a crash that could occur sometime after starting a duplicate filter maintenance task from the dupe filter page
fixed several unlikely but plausible crashes on the admin-side of petition processing
wrote a bunch of help for the url classes and new parsing system--it isn't finished yet, but then neither is the new system!
if the client fails to initialise the db, it will now try to present the error in a bit of screenshottable-gui before the program quits
improved thread watcher error handling when the given url is unwatchable
fixed the issue with options dialog not opening again after a save--I apologise for the inconvenience
the default system:time imported predicate (as set in options) is reset to the default '< 7 days'
fixed another potential crash with the manage tags dialog (in fact with any dialog that has an embedded autocomplete or any other autocomplete with a non-floating dropdown list, such as the import file path tagging dialog)
the sankaku user-agent has been updated to be a generic Firefox string "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0", which seems to work with their new rules. if your sank has been broken and you haven't touched the settings that do this stuff, your sank should be magically fixed on update
the default global hydrus user-agent has been updated to the more polite and compatible "Mozilla/5.0 (compatible; Hydrus Client)". if your previous global user-agent is currently 'hydrus client' (the old default), you'll be updated
import folders now deal with interruptions more skillfully and take a break and save their progress every ten minutes
the manage import folders dialog now waits properly for any currently running import folders to quit and save themselves before opening itself (like how the manage subs dialog does it)
import folders deal with some error states in a better way and are able to cancel out of their loops more reliably when an error occurs
added a new checkbox option to options->tags that lets your 'write' autocomplete inputs (ones where you can submit new tags, like in manage tags) select the first result with non-zero count by default. this means it will skip over the first result if it is just nonsense like 'boyp' that you typed to get the results
linked all the final behind the scenes stuff in the thread watcher together with the new parsing system. this stuff remains complicated and non-user-friendly, so please feel free to completely ignore it and the following batch of points:
-
wrote new url classes and parsers for 4chan and 8chan--they should all be automatically linked up on update
the new thread parsers will use the first part of the first post's comment if no subject is set
the domain manager can now cope with URL->API URL links
manage url class links now displays the current expected api pairs of URL class links and filters out what cannot be currently parsed as a result
manage url class links now only displays watchable urls in the bottom listctrl and permits parsers to be linked!
the domain manager can now try to link url classes with parsers based on example urls!
a button to fill in missing links is now available on the manage url class links panel
content parsers that produce urls now define different url types--file, post, and 'next gallery page'
content parsers can now parse a 'source timestamp' type
veto content parsers now use a StringMatch object to do their matching. existing veto content parsers will be updated and _should_ work about the same as before (unless the previous string was crazy in regex rules) using a regex match
html formulas now pick up the 'string' value more reliably in complicated or mangled html
html and json parsing formulas can now work without any rules, in which case they action the top level node (this is useful to replicate the json/html for subsidiary page parsing)
activated some of the new file import objects' features for the new parsing system--now, they store tags and known hashes as informed by what is parsed. this information is tested and used in import
the db can now test all the hash types--md5, sha1, sha256, sha512--during pre-import checking
the new parsing system now takes a temporary 'parsing context' that will eventually be able to receive and serve temporary variables but at the moment just holds a 'url' entry if the parser would like to use the url used to fetch the data anywhere
all the new parsing ui now has a button to edit the parsing context, and the example parsing context is saved to the topmost page parser object
wrote some url classes for 420chan.org--they will be added on update
fixed an issue with the 4chan thread url class
-
the standard new listctrl panel wrapper can now now provide a special 'export to pngs' for certain named objects that will quickly export one object per png
the 'import from png' dialog as launched from the standard new listctrl panel wrapper now allows multiple png selection
the default parsers (and now the url classes as well) are stored in install_dir/static/defaults. they are read from here when you create a db or ask to load from defaults, so feel free to play around with this
the manage services dialog now has an additional Yes/No warning/confirmation dialog if it expects to delete any services on dialog ok
page tabs escape mnemonic characters (typically '&', which in ui labelling rules makes the following character an underlined shortcut) more reliably
in an attempt to stop subs making so many separate small file popups, the subscription daemon will now only naturally check every four hours (previously one hour) and will wait up to ten minutes for bandwidth to free up before dumping out (previously 30 seconds). these values will be user-configurable in a future update
fixed import for some unusual apngs
corrected some more GMT/local time stuff with the new system:time imported 'date' mode
misc GMT/local time fixes (profile logs names themselves correctly, etc..)
improved how the thread watcher reports thread-check errors in unusual situations
fixed an issue generating network contexts when the initial domain is 'localhost'
improved some misc program stability
started a new job scheduler object to reduce idle thread count and CPU usage--will play around with it a bit more and see about integrating it in the coming weeks
extended system:age to support searching by fixed calendar date, e.g. "system:age > 2018/01/24"
'system:age' is also renamed to 'system:time imported' and presents nicer strings
I believe I have fixed the manage tags dialog crash
the manage tags dialog now loads related tags, file lookup script info, and recent tags off the main ui thread, meaning the dialog itself should launch much faster in almost all situations
cleaned up how a bunch of autocomplete<->manage tags events occur
'set_search_focus' works in the manage tags dialog again, drawing from the new shortcut system
fixed ctrl+scrollwheel on autocomplete entry, which now correctly scrolls the selection over the results again
cleaned up a bunch of the taglist scroll and selection code, particularly for tag children/parent selection events
fixed a potential crash bug initiated by en early exit from the export files dialog
fixed a potential crash bug initiated by an early petition processing page delete
fixed a potential crash bug initiated by an edit repository close during access key fetching
the thumbnail view should be a bit less jittery--the scrollbar should fade neater on mouseover, and maybe the thumbs are smoother as well (I turned off a now-superfluous double-buffering mode, please report any visual bugs)
added 'check_all_import_folders' to main_gui shortcut actions
menus off the main menubar that are out of date (this often happens to 'pages' after a session addition/delete) will now be disabled (greyed out) until they are back in sync
the duplicate filter now detects media it cannot display more carefully. if it runs across undisplayable media due to file orphan status, it will skip over it. if there is nothing else to display, it will ask the user to inform hydrus dev
added json support to the parsing engine!
wrote a new json parsing formula object
wrote a new 'compound' parsing formula object that combines multiple formulae together
wrote new panel for editing formulae with some cleaner button workflow
added multiple 'separated' content parsing solutions to page parsers and wrote all new gui to reflect that
expanded the new separated content parsing solutions to be full-blown nested sub-page-parsers, wew
put some more general work into the new parsing engine--the new page parser object now saves and serialises, so it can be imported and exported. I don't expect to change it much more, but don't rely on that! this is all still under construction
added 'file hash' content type to content parsers and improved ui support for when string conversion rules generate bytes data etc...
new parsing objects now render their parsable content correctly in the 'produces' column
misc cleanup of new and existing parsing dialogs
fixed the handling of escape/cancel events on the 'select from a list of strings' dialog
on update, users with a certain kind of orphaned file will be informed of the numbers and a request to inform hydrus dev
fixed exporting serialisable object to png, which was failing due to a bmp data change in the new wx
fixed a couple of places where I was switching GMT and local time around
deleted some old useless tag editing and commandevent code
expanded the permissible shortcut keys to all reasonable ASCII characters, such as comma and ampersand and the types of brackets
fixed a delayed crash sometimes caused by the file import frame (the one launched by dropping some files on the client or going file->import files)
the file import frame does its file parsing job in a generally more sensible way, particularly when new drops occur while parsing is ongoing
fixed a crash that could be initiated by the manage tags dialog setting the favourite tags late
fixed a crash that could be initiated by manage parents closing before results came in
fixed a rare possible crash that could be initiated by a review service panel delivering results late
when media pages load, they will be more thread-crash-polite to wx
wrote a new 'PageParser' object that will be the primary parser in the new downloader engine. this can deal with a variety of complicated parsing situations. it is still a work in progress, not yet ready for proper use
started new 'manage parsers' dialog for the new parsing system and arranged a bunch of things around that to make the eventual transition easier
edited some more old 'parsing scripts' ui to again make the transition easier. file lookup scripts will be converted to something new as we move to the new parsing system
fixed some bugs in contentparser instantiation
fixed a bug in the html tag rule edit dialog that wouldn't allow attribute pair editing
the names in the manage tags file lookup script dropdown are now sorted
removed some old redundant error-hiding code from the pubsub system
a common 'is my page closed' check is now a lot simpler and faster
fixed an edge shutdown bug where log messages were not able to be written to the log file
some media viewer objects delete themselves more safely
maybe fixed a rare menu refresh bug in linux
improved how hydrus deletes temp paths--if they are 'in use by another process', it will now put them aside and try again later. this error typically occurs when file mimetype parsing fails, and due to the error context hanging around, an open file handle is still in scope.
the client is better about recovering from missing tags in the db--in certain circumstances, it will now generate a 'unknown tag:random hex' as a simple placeholder that can be better used to investigate the error, rather than dumping out
downgraded os x opencv to 3.1.0 to hoyefully increase compatibility
content (tag and rating) shortcuts in the 'media' shortcut set now work on the thumbnail view!
tags in lexicophically sorted taglists will now sort numerically if their subtags start with decimals (e.g. page:3 will come before page:20 now)
trying to add/edit a subscription query that is a dupe will now give you a messagebox saying no, and pasting multiple queries will filter out the dupes and inform you (so, if you aren't sure if some of your list of ten artists to add is already in your big 'artist sub', just paste them anyway--it'll take out the dupes for you)
string converters now support regex substitution! (I expect to eventually integrate this into the filename tag parsing panel!)
'modal' popup messages (that create a new dialog and stop interaction with the program, like the one launched for repository processing) will now minimise non-main-gui frames while they work. these frames will be restored when the job is done (this should relieve the problem of this modal dialog sometimes letting other semi-stay_on_top frames obscure it while still prohibiting interaction, wew
simplified and sped up how importers decide whether to do file work or wait
made file importers do ui-polite pauses during the file import loop more intelligently--if the files make no big change to db or ui (e.g file was known already deleted because of its url), no wait will occur at all, meaning redundant lean queue-sections should work real quick
fixed an issue with multiply-nested submenus sometimes not deleting themselves neatly and resulting in highlight-related RuntimeErrors.
fixed an issue where the 'select from list' dialog could sometimes be ok'd without a valid entry--I believe this may be have been causing a delayed crash as well
added a 'ui timer profile' mode, which profiles some common unified ui update loops that draw animations and keep downloader ui synced and so on
the 'delete processed' file import status right-click menu entry will no longer show if it has the same count as the 'successful' line
the options->colours panel has some better layout and the namespace listbox asks a yes/no to delete if you double-click/enter a colour
dropping a raw file url onto the client will now automatically pend it to a raw url importer (previously, it would be put in the input box, but after rethinking how this could go wrong, I concluded it wasn't such a big deal). if you use this a lot, see how you like it, and let me know if you would want an option to revert to the old behaviour
updated domain manager so it tracks url class identity in a neater way (renaming them will no longer break links or display options)
improved some widget feedback of edit url class panel
simplified the manage url class links panel
url classes now normalise gallery and page urls differently (page urls strip all non-declared data, gallery urls just switch scheme). this may change in future
url classes can now convert a normalised url to an api url--for instance for informing the program that an imageboard thread url has an api version that'll be better to check
the domain manager can now determine arbitrary url parsability capability, and this is just starting to be integrated into url drag-and-drop--try dropping a known url class on the client, like a booru link: it'll now tell you it sees what it is but can't yet parse it!
the 4chan and 8chan thread url classes will reset to defaults again this week--replacing with url classes that convert to the api versions--and you'll get the new api url classes themselves
if missing, the primary tables in client.caches.db can now regenerate on boot. they aren't properly repopulated yet, but the client will boot and you'll get an error message telling you what to do next
improved .txt and clipboard tag import unicode support
removed leading BOM (byte order mark) from .txt and clipboard tag imports
improved some clipboard text fetching code
added a new 'hover window profile mode' to the debug menu--it will print the logic info behind any media viewer's hover window show/hide decision
tweaked the locale-setting code, which was stopping some users (on Win10 creator's update, it seems) from booting the client
added system:rating to 'all known files' query contexts (so you can now search for deleted files that have ratings)
the manage import folders dialog now has a redtext warning about subdirectories
some file importers deal with shutdown events more gracefully
lz4 is no longer required. if it is absent, or the version is too old, a simple statement will be printed to the log and it will not be used in either the client or server
updated to ffmpeg 3.4.1 on windows
updated opencv to 3.4 on windows and linux (had trouble with os x--will continue working on it next week)
fixed an issue where scrollbars were only appearing on taglists after a resize event
fixed the raw filename component of file drag and drop events from the client to external programs
fixed the tag lookup scripts
fixed some wx menuhighlight issues
improved some shutdown code
fixed the add/edit namespace colours options panel, which needed to be updated to deal with the new wx's better alpha channel reporting
fixed an issue when hitting 'open externally' on a media collection
fixed a crash on client shutdown whenever closed pages were in the undo menu
think I fixed another shutdown crash
fixed a rare issue with the collect by dropdown not being able to generate a string to display
misc wx fixes
added a 'importing' page to the options dialog, which now sets the default file import options for quiet and loud file import contexts
the old and no longer used 'exclude deleted files' option is now removed from the 'files and trash' panel
finished off default url matches for all downloaders that come with the program--these will be set on update, so if you have custom ones, please export them before you update so you can import them again!
improved how urls are matched and presented for the user in the media viewer
added a 'delete "successful" file imports from the queue' entry to the file import status button right-click menu--this only removes 'successful' and 'already in db', leaving anomalies
improved locale instantiation in the client and added locale strings to the help->about dialog
you can now set the page name prefix for a paused thread checker. it defaults to a unicode pause character: ?
thread watchers will no longer pause on a network error during a check--they now have a 'delay' system like subscriptions, and on a network error, they will delay four hours (unless you hit 'check now')
patched in some simple 'connection cutoff' network error handling, we'll see how it does
wrote in some more proper error handling for a specific connection cutoff error that is being produced
the similar files search tree regen code now clears out orphaned files. if you have had blank 'unknown' files appear in similar files searches, please run database->regen->similar files search tree
bitmap buttons on download pages will now update using less CPU and will flicker less
improved some video rendering error reporting
fixed the 'author name' regex favourite default, which had a superfluous asterisk. if you would like to fix it yourself, please try: [^\\]+(?=\s-)
added 'flush log' debug command
client.pyw now makes a safe 'system' ui error popup if it fails to boot
all builds now require the new wx, so if you run from source, please consult the help files for new info on figuring this out
did a ton of wx refactoring
merged a large number of timers
wrote a new wx-aware timer to replace some of the more awkward old timers that could not be neatly merged
replaced wx.calllaters (the new ones aren't always garbage collecting nicely) with the new custom timer object
misc wx deprecation refactoring
cleaned up some wx test code
fixed some crashing wx test code
fixed 'select from list of strings' object, which was no longer processing enter key properly
cleaned up a bunch of ui object interaction code
cleaned up a bunch of wx-related shutdown code
fixed an issue with the splash window not shutting down cleanly
fixed a shutdown db-gui status report crash
was unable to get flash embed windows working properly without edge-case crashing on windows, so they are disabled for now (you'll likely get an 'open externally' button instead)
animations now clean up their memory buffer faster in certain circumstances--all users who view animations in the preview window should notice significantly leaner 'idle' memory usage over time
fixed a very important inefficiency bug that meant select->inbox/archive on large pages with more than a handful of inbox files was taking extremely long periods of 100% CPU
wrote a faster way of fetching some media and their paths
copying files to clipboard is now much faster
initialising any large file drag-and-drop event is now much faster
certain redundant image pre-fetching will no longer be done
'open externally' panels will no longer draw a (slightly buggy and useless) animation bar if the underlying media would have had one
added a new 'menu profile mode' that will profile any menu click
cleaned up the debug menu
deleted some old options code that is no longer used
thumbnails can now be drag-and-dropped to other pages!
dragging and dropping thumbs to another page tab will remove them from the source and append them to the destination, maintaining file order!
DnDing thumbs to a 'page of pages' tab will put the files in the next lowest selected media page
DnDing thumbs to a blank notebook area (or a page of pages without a selected media page) will create a new page for the thumbs
holding down ctrl when you drop thumbnails will not remove them from the source
please forgive the thumbnail DnD cursor, which for now will be in the 'copy' state, despite the internal DnD being move by default
improved page tab drag and drop drop logic--dropping onto the page area itself will no longer send the page to the right-end of the current notebook
the 'file import options' object now supports three 'presentation' booleans--for new/already_in_inbox/already_in_archive files--so you can customise whether new thumbnails appear based on each state. page imports will by default show everything, while 'quieter' import queues like import folders and subscriptions will continue to just show only 'new' files in their files popup buttons. if you have a gui page with 10k+ items in its queue, try reducing the presentation to speed it up!
all existing import queues will be updated when they are next loaded--but please note that for simplicity and safety they will all initialise to the 'quiet' presentation method, so if you have ongoing download pages in any of your gui sessions (including thread watchers!), they will only add 'new' thumbnails unless you edit them. I apologise for the inconvenience
the regular hdd import now has a file import options button!
subscription query 'finished' file popups are now merged up to the subscription level--so, a sub with five queries that each get 20 new files in a run will now ultimately leave one popup with 100 files
file popups (as produced by subscriptions and a couple other places) now preserve their import order!
if a subscription with many queries runs out of bandwidth, it should now only give you one 'no more bandwidth to download files' message, rather than one for every outstanding query to sync
added a checkbox to turn on/off the new random subscription syncing to the options->downloading panel
the file import status button's menu now supports import/export of sources to clipboard/png! it _should_ also support unicode. be careful not to paste paths into a url cache, or urls from one gallery site to another, or you'll just get errors--this is prototype, so please keep like with like for now
the png import/export system now supports raw string payloads
the new listctrlpanel can now hang its buttons in multiple rows
the manage subscriptions panel now has an 'overwrite checker options' button to mass-set checker options
the manage subscriptions panel now has a 'select subs' button that selects subs based on a basic query text search
separating merged subscriptions now sets better new subscription names of 'old_sub_name: query_text'
saving a session from a page of pages with a custom name will no longer suggest a session name prepended by [USER]
doubled the subscription and downloader instance default bandwidth rules to 400 and 200 rqs/day
the 'load_images_with_pil' and 'disable_cv_for_gifs' options are now officially BUGFIX in the options--unless you know you need them on, turn them off!
added some safeguards to the new dialog-panel system's OK stuff, which sometimes catches a duplicate OK event
simplified how thread watcher assigns DEAD and 404 status--it should do it more reliably now
thread watchers now publish their page names (including updating dead/404 status) right after they check
the save session menu under pages now lists the exsting sessions for quick-save--if you select one of these, it will throw up a yes/no overwrite confirmation dialog
'appending' sessions now loads them in a new page of pages named as the session name!
right-clicking on a page of pages now gives you a 'save this page of pages to a session' option
right-clicking on the page tabs lets you append a session anywhere
'load' session is now 'clear and load', and it throws a yes/no dialog for confirmation
the gallery pending queries box now allows multiple selection with ctrl/shift clicks. it'll even move up/down correctly on non-contiguous selections!
the gallery query input is now a new combined textctrl-and-paste button control that will appear in other places across the program
the page of images input now also uses this new control
the page of images and raw urls queues will now ignore any input that does not start with 'http'
pasting a list of texts from the clipboard will now typically strip the pasted content of leading or trailing whitespace
some text and tag cleaning is a bit faster and neater
the file import status panel now supports 'open sources' on its right-click menu--it opens in your web browser or file explorer as appropriate. (linux can't do file explorer though)
thumbnails will no longer 'fade' when their pages are not visible, reducing CPU load in several contexts
thumbnail fade will use less CPU overhead on very fast computers
the timer loops that all import pages use to keep themselves updated are now harmonised into one loop at the top gui level (this should reduce some idle CPU overhead and improve some ui responsiveness)
content (tag and rating) shortcuts can now be set to 'flip on/off' or 'set'
the manage subscriptions dialog now waits for any currently syncing subs to cancel and save themselves before opening itself (keeping it all synced and rollback-proof)
added a 'watchable' url class type and wrote examples for 4chan/8chan thread urls
added 4chan/8chan file url classes to the examples, which if added will auto-hide them from the media viewer
fixed 'add parameter' in edit url match panel
simplified some stuff in the existing parsing system
added preliminary 'url type' support to the parsing system
the html formula parsing object can now return the full child html of the node it finds
the new url and html stuff doesn't do anything yet--but it will in the new downloader engine
added a bunch more booru entries to the url classes defaults. I will continue filling these out, hopefully setting comprehensive defaults next week
the advanced mode thumbnail right-click menu lets you reparse and regen thumbnails for any files! if you have rotated or stretched files, you can now fix them from the menu!
added new tools to let the client update media file metadata and thumbnails live!
the new reparse and regen should update file metadata and thumbnails live!
improved how some 'quiet' errors are printed to the log. they'll now have the full trace of the error as well as the stack
added a daemon_report_mode debug mode. it throws up a popup every time a daemon fires its callable
fixed an issue with the editstringtostringdictcontrol
fixed an issue with loading (and updating to a new version) import pages or import folders with some kinds of unicode paths
you can now set the 'woah, you should close some pages' warning value under options->gui. its default remains 165 pages
system:similar_to searches should be a lot faster on clients with many files
the manage subscriptions and edit subscription panels now both list their num_urls summary column as '22' if all done but '11/22' if not. this column also sorts based on percentage completion, then num total, then num done
the database migration dialog now uses the new listctrl--its buttons are now also clever, and will disable when they are invalid
misc fixes
fixed an index bug in the new listctrl after certain types of 'setdata' call
cleaned up some search code
some veto code cleanup
cleaned and harmonised how text is pulled from the clipboard
added 'network' main gui menu and moved network stuff from services to it
split the new domain manager stuff up into separate dialogs and menu entries on the new network menu
manage url classes dialog now lists url type in the listctrl
added url class links info (which will permit client-specific settings and downloader mappings for url classes) to the domain manager
wrote a 'url class links' dialog and added it to the new network menu (only the 'display on media viewer' part works atm)
the domain manager now filters urls on the media viewer depending on whether they have a url match and are set to display in the new links panel
updated the local booru service code to the new service system
the local booru's shares can be reviewed again under review services
the local booru's port and bandwidth settings can be set again under manage services
the different gui parts of the local booru are updated to new controls
fixed a local booru 404 reporting error
the edit subscription panel now has a 'paste queries' button that lets you add queries en masse
added 'manage_file_urls' to shortcuts system
added several 'get_similar_to_x' actions to the shortcuts system
the manage upnp dialog now initialises its mappings on another thread and fails better when UpnP mappings cannot be fetched
connection and readtimeout network exceptions are now recognised more specifically by the client. subscriptions will only delay for an hour on one of these exceptions
improved the resilience of the HF login code after wake from sleep (when networking is often not available for a couple of seconds)
like the recent subscription query randomisation update, subscriptions themselves are now synced in random order (this stops a subscription named 'aadvark' always getting first bite into available bandwidth)
fixed import for jpegs that have unparsable exif data
fixed a bug in 'next month' bandwidth estimate calculation when the month is December, wew
fixed some logic that was setting max page at 165 rather than the intended 200 and added a dialog that even lets you engage the debug override at 200 if you are insane
all audio mime detection and duration parsing is now done through ffmpeg. hsaudiotag is no longer needed to run the program
since the old listctrl sort crash bug is still hitting some people, I've disabled sort on the old listctrl class. feel free to try to sort any listctrls now, no matter your situation. I will continue replacing the old class with the new working class over time
updated another listctrl
a ton of misc controller options/manager access refactoring
cleared out some old code
moved Time controls to their own file and added velocity and checker options stuff as well
wrote a new edit panel for single controls and updated the time delta button to use it
fixed subscription queries turning dead on the initial sync
all dead subscription queries have been set to check again in case they can revive
added query file velocity to edit subscription panel
subscription network contexts now reflect the new multiple subscription query system, and are named "sub_name: query_text". as every query now counts as its own separate subscription network context, this will stop query-heavy subscriptions from throttling so much on bandwidth limits
finished URLMatch object, which matches and normalises URLs into certain 'classes' like 'gelbooru post url'
expanded some URLMatch subdomain options
fixed some test logic in URLMatch
finished the last of the EditURLMatchPanel
split the 'manage network rules' dialog into two panels--it now has a 'url classes' tab
wrote a panel for managing url matches
added export/import/duplicate buttons to EditURLMatchesPanel
wrote some URLMatches for hentai-foundry as an initial test of the system and added a temp button to add them--please check them out to see how it all works
all, invert, inbox, and archive (and none, lol) thumbnail 'select' menu items now have counts
invert is now at the bottom
the thumbnail select menu now has local/remote entries if applicable (this typically is only true in 'all known files' file domain)
added png/clipboard export/import/duplicate code to the generic new listctrl button wrapper panel, which will save a bunch of time as the png/clipboard sharing system expands
added a human-facing serialisable name to all objects on the new serialisation system and tied the new import/export code into it for png presentation
the edit import folder dialog will now complain (but not veto) on an ok event if any of the entered paths do not exist
if you attempt to manually run an import folder while import folders are globally paused, you'll get a little popup telling you so
added an experimental 'thumbnail fill' setting to options->gui. it zooms the existing thumbnails so they fill the whole thumb space. feedback on this from those who would be interested in a prettier system would be appreciated
added 'paste tags' buttons to filename tagging options panel
the paths/urls in the file import cache are now their own object that holds the creation/modified/source times and current status and note. this object can also hold prospective urls, tags, and hashes for future use
a bunch of file import actions are faster
all the different importers now use this new file import object
fixed a screen position calculation in the new drag and drop filtering code that was accidentally including too many possible drop candidates on drops in the top-left corner of the main gui (if you had trouble moving tabs to the left, this should be it fixed!)
fixed a problem display volume/chapter/page tags that included unicode characters in thumbnail banners and media viewers
fixed a rare media display bug in the dupe filter
fixed some 'C++ part of panel has been deleted' bugs in review services if the frame is shut down before delayed db info is fetched
cleaned up some more 'C++ deleted' errors in import files selection dialog
fixed the network context custom header panel 'add' action, which wasn't saving the value of the panel
fixed a bunch of bugs in the newish QueueListBox class
SynchroniseRepositories daemon will be better about quitting early on application shutdown
cleaned up some pending pretty timestamp grammar
when the client cannot clean up a temporary file, it will print more error information
added pylzma to the 'running from source' library recommendations. this is not required, but if available it adds ZWS flash support
subscription popups show a bit more info about their individual queries
subscription popups hide the sub-query info in text and file buttons if it is the same as the subscription name
subscriptions now work through their queries' downloading phases in random order (so a heavy query named 'aardvark' won't repeatedly choke the sub's other queries if bandwidth is tight)
if a subscription query dies, a notification popup will now be made
changed a logical edge case that may have been killing some subscription queries after the first sync
re-enabled pixiv artist search, as it still uses the old gallery style
added tentative support for 'ZWS' flash files
if there is more than one import folder, the 'check import folder now' submenu now has a 'check all' entry
TimeDelta controls that have the 'days' entry (as in thread/subscription checker options panels) now have a max period of 10 years for those _really_ slow threads
fixed an issue where the 'quick namespaces' section of the new filename tagging options panel was not initialising correctly when it loaded data from a previously serialised import folder (this was the "unhashable type: 'list'" bug in the edit import folder panel)
the filename tagging dialog now works on the new panel system. it also has an entry under options->gui: 'local_import_filename_tagging'
polished StringMatch object
polished EditStringMatchPanel
html formulas now have a string match object
improved parsing error system
miscellaneous improvements to the parsing system
misc fixes to StringMatch and parsing system
the main gui will no longer interpret text drops on child frames as url drops. this fix just quashes the drop--it doesn't allow the initially desired drop on the child frame to occur. I will keep thinking about this
the dupe filter's 'no media to display' text now says 'Looking for more pairs--please wait'
the 'remove' media action is now labelled 'remove from view' to reduce 'is it a delete?' confusion
fixed an icon initialisation order problem that meant pre-boot errors were unable to display
fixed a graceful-shutdown-on-boot-error bug
the custom 'listboxtags' controls now post an event on their contents changing
the tag censorship panel now updates its status text when its listboxes change due to a mouse double-click event
fixed file import 'source time' column sometimes showing '3e-05 years' kind of garbage that was due to a missed float->int conversion
cleaned up how some page names are thrown around, which should stop some stray [USER] prefixes slipping though
the adminside petition management checklistbox will now escape mnemonic characters (typically '&') and hence display them correctly on sibling pairs and so on
wrote a 'help my client will not boot.txt' that explains the new debug builds and some other common fixes to try
'all known files' file search domain is now hidden behind advanced mode
fixed another instance of flashwin causing a failed boot
rolled back to an older version of pyinstaller that does not seem to have the embedded flash window problems windows users experienced
added an error handler to wx.lib.flashwin import--if it fails to import, the client will print the error to the log and thereafter show 'open externally' buttons instead of the embedded flash window
created a 'filename tagging options' object to handle the instance-non-specific auto-tagging parts of the filename tagging dialog
moved all the appropriate tag generation code to this new object
extracted the simple and advanced panels in the filename tagging dialog to a separate panel
wrote a new wrapper panel to edit filename tagging options objects directly
cleaned up some tag generation code--it'll now all be siblinged/parented/censored the same way
import folders now use the filename tagging options object, harmonising this tag parsing code
edit import folder gui now has an add/edit/delete listctrl to manage the new tag_service->filename_tagging_object relationship (this replaces the old .txt management button and summary text)
finished the StringConverter class in the new parsing engine. it applies a series of transformations to text
wrote a panel to edit string converters
wrote a panel to edit string converters' individual transformations
updated html formulas to use string converters instead of the old cull_and_add system
html formula edit panels can now edit their new string converters
file lookup scripts now 'stringconvert' their file identifier strings--this should allow the 'md5:md5_hash' fix we've been talking about (i.e. by first encoding to hex and then prepending 'md5:')
the help->about window now shows the client's default temp directory
thumbnail regeneration--particularly full-size regen and resized png gen--should be much faster in the client
the debug exes are now included in the windows build
the debug exes are now included in the non-windows builds
the test exe is no longer included in the windows install (can't remember it ever being useful, and it is like 10MB now)
unfortunately, a recentish change to how pixiv serves gallery page results has broken the hydrus pixiv downloader. this will have to wait for the downloader overhaul, and even then, it might be _slightly_ tricky. pixiv downloader entries are now hidden, and existing pixiv subscriptions will be paused on update
the thumbnail select submenu now clarifies 'all (all in x)' if all are in the inbox or archive
you can now archive/delete filter from the thumbnail menu on a single file
lowered the http session inactivity timeout even more, to 45 minutes
fixed a couple of instances of the new subscription dialogs being unable to mass-retry failures
ffmpeg parsing errors now print more info about broken file path
some daemons will be snappier about shutting down on application shutdown
took out the sometimes invalid 'booting' phrase in the during disk cache population status report
the client will now warn you about page instability at 165 pages after one user reported instability at 175. 200 is still the strict limit.
downloader pages and subscriptions will fail more gracefully if their downloaders cannot be loaded (e.g. they used a since-deleted booru)
fixed listctrl panel button enabled/disabled status when the child listctrl starts empty
the new listctrl can now select data in a cleverer way
subscriptions now use the new thread checker system to set their check periods!
improved some check timing calculations to deal with subscription timestamps
existing subscriptions will update
subscriptions can now see any 'delay' reason on their edit page and can scrub it there as well
subscriptions no longer track 'last error'. everything goes through the delay system and keeps track of reason
added 'last new file time' to the subscription listctrls
subscriptions now generate separate 'file button' popups for each query, and file button popups now include their name in the button label
the subscription listctrl is now breddy wide. I will be adding some way of hiding columns (with persistent memory) on the new listctrl at some point in the future, so please bear with it for now
manage subs now has a wrapper panel and buttons that disable when invalid
tumblr simple 'video' posts should now be supported! turns out they are just mp4s
in the client, static image thumbnails are now generated with opencv
files that give truncated image errors should now import ok. they'll render with errors. if cv is up to date and working ok, they should have good thumbnails as well (otherwise they will have all-black thumbs)
updated manage regex favourites dialog to the new panel system
regex favourites are now editable under the new options->regex page
ongoing import or export folders will now stop working if their respective manage dialogs are opened
import folders can now stop working if the client shuts down during the initial file parse phase
export folders will now stop working immediately if the client shuts down
export folders can now stop working during the file copy stage
the splash screen is now a little wider and taller
the splash screen now has a subtext row for misc technical data
the boot and exit splash screens show a bit more data about several things
the repository sync daemons should be a bit snappier about responding to a client shutdown
the import files dialog is now a non-modal frame! (you can now continue to use the client while this frame parses files) you can even open up several at once!
the 'import file path tagging' dialog should now never be taller than its spawning screen and also slide up and left if it spawns with its bottom-right corner off-screen (this dialog is still on the old system, however, so the newer sizing stuff that does this stuff automatically is not yet available)
sliding up-and-left should now also occur on spawn for frames on the new system
the advanced 'process now' repository button will now disable itself if there is no work to do!
if simple downloads like gallery pages are larger than 100MB, the client will throw an error (this is a stopgap for an ongoing issue)
the 'maintenance and processing' page has a bit more warning text
the new listctrl wrapper panel now supports menu buttons
libpng's iccp terminal warnings from loading the flash/pdf/zip thumbnails should now be gone!
the client should now recognise EXIF file rotation and flipping in resolution discovery, thumbnail generation, and image rendering. existing files will be borked, so will have to either be manually deleted&reimported, or wait for a maintenance routine that will retroactively fix all these
videos with a non 1:1 sample aspect ratio (wew lad) should now import with correct resolutions
all 'tag import options' are now managed with the new button
deleted all old tag option management gui, including deleting ClientGUICollapsible entirely
with the simpler tag import options, increased max page count to 200
page of images downloader now has two pause separate buttons for queue and files processing
the page of images downloader should no longer too-quickly blank out its parser status at the end of queue processing
the default hydrus bandwidth rules now no longer has the choking 'requests per day' limitation that was messing up large mappings uploads. (it is now just 64MB/day)
your hydrus default bandwidth rules will reset to this on update
POST requests (which in for our purposes are always user-driven) no longer obey bandwidth rules
fixed a bug when sometimes hitting the file limit or hitting a siteside 404 in the gallery downloader
a rare bug due to empty string menu labels is now no longer possible. empty string labels, if they slip through, will be replaced with 'invalid label'
you can no longer choose the empty string as a gui session name
improved how network jobs keep track of all applicable network contexts vs the preferred session and login contexts
cleaned up a bunch of login code, improved verification testing
fleshed out domain login system
in the networking engine, the read timeout is now six times the connect timeout--we'll see if this reduces some hydrus and other read timeout problems we've seen
increased the server transaction period from 10s to 120s--we'll see if this helps reduce some spiky server POST lag or what
the default tag import options panel now edits on double-click, rather than deletes. it also yes/no dialogs on a delete action. this control is still trash, but it now works more like the rest of the program
moved the hydrus service object over to the new networking engine
hydrus services now appear in the bandwidth review panel
hydrus requests now obey larger network bandwidth rules (this mostly means 'global')
service bandwidth usage and rules are no longer managed from manage services--it is now under review bandwidth usage, like all other network contexts
updated some network engine stuff for misc hydrus states like 'server busy'
fixed a bug in the server code where the session key cookie had an invalid expiration timestamp--servers should update this week to make sure new clients can log in properly
hydrus network requests will force-set User-Agent as 'hydrus client/(network_version)' as the network version is used in the protocol to determine compatibility
fleshed out the hydrus specific network job, giving it the various bandwidth tracking and version checking responsibilities the service object used to have
moved session cookie decay (only matters for Hentai Foundry atm) to the new session manager (was previously hacked into the login manager)
moved some hydrus response parsing stuff around, added content type awareness to the new network job
updated several unusual hydrus 'static/test' requests used to test credentials and fetch access key and so on to the new system
added a special 'test service' service to better accomodate these requests
wrote a static login script for hydrus services
polished up login management system overall
import folders now support a 'check now' state, like subscriptions, that will cause them to check immediately
import folders can be 'check now'ed from the file menu under a new submenu! if you would like to have a 'manual' import folder, try pausing it and just running it from this menu!
added an optimisation to the file search algorithm to search ratings queries super fast when they lack tag or file system preds to otherwise speed them up
updated the booru presets that now support https to be https
added a 'restore defaults' button to the manage boorus dialog--you can restore specifics or all of them
optimised how fading thumbnails are blitted to screen, which may provide a huge performance boost for high-res/small-thumb clients
pages that have been renamed by the user will no longer be rename-overwritten by any auto-renaming system (currently just thread watchers, I think, but this will expand in future). unfortunately, this is not retroactive--only pages renamed from now on will be aware that they were user-renamed
in advanced mode, the pages menu now states how many pages are currently open
the 'page of images' downloader will now say '(x already in queue)' when it reports how many urls were found, if any were already in the queue. (this should clear up some confusion where it would previously say '0 new urls' even when it found some stuff)
the gallery downloader will do essentially the same, but on a per-page basis. the text is a little crushed, so I may revisit this
fixed an issue with the manage services dialog not being able to rename dupe-named services on edit subdialog ok
the manage services dialog now uses the new listctrl! the listed services are no longer a horrific unsorted mess!
fixed the file import status button not showing on raw url downloader or import folder edit dialog
improved how some directory tree merging code deals with read_only files in the source
maybe fixed some unusual selection behaviour with the booru selection popup dialog. it now has a real ok button, rather than a mickey-mouse hidden one
completely deleted the old networking engine!
cleaned out some unused imports from the networking code and related entries from running_from_source.html
changed up and fixed some odd bugs in how how repositories test some error/isfunctional stuff vs regular paused status
hydrus network contexts now have a prettier name
ip address-based network contexts will no longer spam the bandwidth tracker with their useless subdomains
network jobs that are unable to attempt validation or login will now error immediately rather than waiting indefinitely
fixed up a bunch of test code that was broken due to the mismatch of network engines
broke some other test code due to the network engine transfer!
the client is more resilient about broken 'twisted' installs, and _should_ be able to boot without it. this may or may not apply to the built release--more work can be done here
fixed the tumblr raw url converter to now point to the data.tumblr.com domain
added a hardcoded ssl verify exception for data.tumblr.com, which has an incorrectly defined ssl cert (at least for public-facing interactions), wew
all existing db urls and file import cache urls for media.tumblr.com will be updated to data.tumblr.com on update! (everything _should_ just magically work again)
fixed apng import, which the decompressionbomb detection code was not handling correctly
collapsed the different instatiations of the 'file import status' button down to one class
the file import status button now has a right-click menu that supports 'retry failures' and 'delete processed', if applicable
misc import status cache cleanup and refactoring
you can now edit or completely turn off the [404] and [DEAD] thread watcher page name prefixes under options->downloading
thread watchers should more reliably keep 404 status
'open selection in a new page' now preserves file order!
'view this file's duplicates' now sorts the files!
options->gui now has an option to change how often 'last session' is saved
'last session' will no longer autosave to the database if there are no changes
tags exported to neighbouring .txt files are now correctly sibling-collapsed
tags imported or exported via neighbouring .txt files are now correctly tag censored
the manage tags dialog will now protest with a yes/no dialog if you attempt to cancel it with uncommitted changes
the manage parents and siblings dialogs will now protest with a yes/no on an ok event if there are 'uncommitted' pairs in the lower boxes (e.g. if you forgot to click the 'add' button)
fixed an issue that would sometimes stop old sessions from loading properly
the duplicates page now does its maintenance jobs in modal popups!
attempting to apply a duplicate status to more than 100 pairs now throws up a warning yes/no dialog
the manage urls dialog now has copy/paste buttons
added a (somewhat debug) option to disable the mouse hide&anchor behaviour on slow Windows canvas drags to options->media
added a 'regen all phashes' command to the database regen menu
the disk cache options in help now have a help button to explain good values for ssd vs hdd
the edit bandwidth rules dialog now uses the new listctrl
expanded the domain manager into a legit serialisable object that holds data and saves itself on changes. to begin with, it supports custom http headers for particular on network contexts
the global User-Agent (which now defaults to 'hydrus client') is set through this new manager, as attached to the 'global' network context
wrote a basic panel to edit custom http headers under services->manage network rules (this panel will be fleshed out with more in future--please ignore if you are not an advanced user)
wrote a panel to edit an individual custom http header
wrote a panel to edit/select network contexts. this will get a bunch more use elsewhere as the overhaul continues
flushed out and tested the new popup yes/no button system
sanakucomplex.com has a User-Agent entry in the new custom header system--on the first sank request, you will be presented with a yes/no popup asking if it is ok to use.
if you ok the user-agent, sankaku now seems to work again!
sankakucomplex.com network context will get some specific conservative bandwidth rules (80 rqs per 7m, 1 rq per 4s) on update
added an option to options->gui to reverse the page tab shift behaviour
'don't scroll down on key navigation event if thumb is at least this % visible' value is now 75%. it is also editable under options->gui
the hydrus icon used on frames is now the non-transparent version that shows up better on dark backgrounds
the standard hydrus.ico used in web favicons and the executable builds is now also non-transparent (this should also propagate up to OS shortcuts and taskbar icons wherever the frame icon is not inherited)
file export drag and drop events will now defocus the currently focused media on successful drop (e.g. if you drag a video to an external media player, it will stop rendering clientside), just like for open externally
database analyze maintenance should be more reliable with respect to fresh repository syncs
reduced default impermanent session timeout in new networking engine to 60m (should fix/reduce some hentai foundry 503s)
the new thread watcher object will no longer produce check periods shorter than the time since the latest file. this effectively throttles checking on threads that were posting very fast but have since suddenly stopped completely
thread watchers now parse their thread subject and place this in the left management panel
thread watchers now name their pages based on the thread subject, if one exists
an option to permit or deny thread watchers renaming their pages is now under options->downloading
dead and 404 threads now disable their checker pause button--to attempt to revive, hit 'check now'
thread watchers now preface their page name with [DEAD] or [404] when appropriate
misc thread watcher code improvements
added basic import support for zip, rar, and 7z files. they get no useful metadata (yet) and have a default 'archive' thumbnail
the client will now by default detect and not import decompression bombs before they blat your computer. an option to allow them nonetheless is under options->media
the server will now not parse or accept decompression bomb uploads in POST requests
added a 'refresh all pages' entry to page of pages's right-click menu
added 'send this page down to a new page of pages' to page right-click menu
added 'send all pages to the right to a new page of pages' to page right-click menu
fixed a page of pages drag and drop issue when dropping the last page of a notebook onto the same notebook tab
fixed some index calculation problems when DnDing page tabs to the right on the same notebook
sending a refresh event to a 'show selection in a new page' page (which has no search predicates and so cannot 'refresh' its search) will now trigger a sort event (like importers got last week)
thumbnails at the bottom of the current view but are at least 90% in view will no longer scroll into view when selected
click events will no longer scroll thumbnails that are semi-out of view into view
improved how all 'wait until the client ain't so busy' checks work. importers that have a whole slew of 'already in db' to catch up on should now not clog the gui so much
similarly, under ideal conditions where nothing is busy, importers will iterate over their files more quickly
the network engine now has a 'verification' loop that doesn't do anything yet, and a stub domain engine is generated to be consulted in this
wrote some verification code, extended popup messages to support yes/no questions
polished some domain engine code
fixed an issue where file repositories were not recording deleted files in certain cases
all file repositories will be reset on update
the date entries on the review bandwidth bar chart now have leading zeroes on 0-9 months to ensure the sort correctly (this month's 2017-10 entry was sorting before 2017-8, wew!)
the migrate database dialog now shows approximate total thumbnail size
gave the migrate database help a quick pass
gave the 'help my db is broke.txt' file a quick pass
if you hold shift down while dropping a page tab, the client will not 'chase' that page to show it (try it out!)
the gui will be more snappy about dealing with drag-and-drop drops (of types file import, page tab rearrange, and url drop), returning the mouse to normal state instantly on drop and dealing with the event in a subsequent action
dropping url text on the client will specifically inform the source program not to delete the dragged text (that the operation was copy, not move), if it is interested in hearing that
page drag-and-drops should transition a little less flickerily
all file import status objects can now track 'source time', typically to represent upload time
file imports now populate 'source time' based on the earliest of creation/modified time! (this will be used later to parse as a tag)
thread watchers now populate 'source time' based on post time!
finished a watcher options object for the new thread checker system
wrote a panel to edit watcher options
converted the thread object to the new watcher system
thread watchers now have two pause buttons--one for the file queue, one for the checker
compressed thread watcher ui layout
converted the thread left-panel ui and options->downloading page to reflect the new watcher system
improved the watcher options to generate better timings for fresh threads
cleaned up some thread watcher check time code
the total/selected mime summary on the status bar is a little prettier and will now report by individual mime sometimes
generating the total/select mime summaries are now faster on pages with >1,000 files (it'll just use 'file')
a 'refresh' action on an import page now triggers a sort event
added 'flip_darkmode' shortcut command to 'main_gui' and 'media_viewer' shortcut sets
added copy_file/path/hash/bmp actions to the 'media' shortcut set, removed the hardcoded ctrl+c for copy_file. I added ctrl+c to the defaults, but existing users will have to re-add it manually if they want it!
simplified some page ui update code
import pages will no longer update their left-panel ui (which uses a bit of cpu) when they are not in view
polished some new string and url matching code the domain engine will be using
wrote a panel to edit string match objects
wrote a new panel to handle simple ordered lists of data in a better way
wrote most of a panel to edit url match objects
misc domain manager work
fixed an issue with the old listctrl where object name de-duplication was sometimes not permitting (1)-type names to be cleaned up
fixed some inelegant time duration->text conversions
improved complete process shutdown reliability when some downloads are waiting on bandwidth on shutdown
the help menu now has an easy on/off check entry for the darkmode colourset
changing any individual colour or the entire colourset will now immediately refresh almost all custom-coloured controls with the new colour
added a BUGFIX option to options->gui to permanently fix all discord file drag-and-drop events (as long as they contain <= 10 files and total < 50MB)
you can now set specific 'open externally' launch paths on a per-mime basis under options->files and trash
improved error reporting on a bad file launch
improved the network engine to recover from and reattempt in-progress response read errors (previously reported as ReadTimeout)
fixed the 'scroll to focused thumbnail' calculations on key events (when hitting up/down arrow key on thumbnail grid, the page wasn't scrolling correctly as needed)
known urls no longer display with the scheme (http or https) in the media viewer top-right summary
known urls in the media viewer top-right summary now tooltip their full url
wrote a new button for editing tag import options. it has a good summary tooltip. it is only in the manage import folder dialog atm, but I will replicate it across the rest of the program in the coming weeks
in some situations, the file import status window will list some timestamp note info for 'already in db' and 'deleted' statuses. see if you like it
the new listctrl will generally give its data to other consumers in ui sorted order (this fixes some stuff like 'copy sources' in the file import status window, which was copying them in random order)
manage tag parents now uses the new listctrl (and is hence now safely sortable)
manage tag sibings now uses the new listctrl (and is hence now safely sortable)
some behind-the-scenes of manage parents/siblings is a bit neater
improved some thumbnail internal media structures to fetch specific media based on hash much much faster, particularly for pages with 10,000+ thumbs. this should speed up large imports and other content update events that can result in thumb redraws
fixed an issue where dismissing a popup message could spawn the entire result of the queue, ignoring the 'show 10 max' rule
completely finished the menu rewrite! all menus now work on the new system
deleted a ton of old and now obselete menu event processing code
some boot/shutdown terminal printing should be more reliable
converted the gui colour options to the new options system
added a 'darkmode' colour set and extended the colour options page to support it!
added a secret discord-compatible drag and drop mode. start a file transfer drag and drop with ctrl held down to initiate. keep ctrl held down, as this is secretly a 'move' drag and drop, and you want to keep it a 'copy' one. it should work for discord dnd, at least to the web version. I have an idea on how to do this better, but feedback on this would be appreciated.
converted 'import options - files' collapsible panel to a button that launches a dialog. this reduces total gui object count when many importers are open and reduces overall ui lag and limits. the button shows a summary of its current options as its tooltip. furthermore, all of this sometimes confusing nomenclature is now uniformly presented as 'file import options'
added a BUGFIX option to gui options page to force hover windows to display at all times
improved last week's gelbooru 404 fix--page urls generated pre-v272 should now be more reliable
in a further attempt to improve workspace support, popup messages will no longer update in any way if the mouse is not on the same display as the main gui
fixed an issue where dupe import urls could be queued up in a file import queue if they came in the same batch. this was essentially harmless but lead to some mixed x/y progress counts and row indices
fixed an issue where url drops were not filling in the url entry when a new raw url page had to be created on a currently selected page of pages. this may have affected a couple of other page spawning situations
may have fixed an issue with the mouse shortcut-setting button, where for some users scroll events were not registering
manage subs dialog now has the new help button up top
manage subs dialog's sub verbs are now wrapping into a single menu button that I can expand more easily in future
sped up thumbnail access a bit on certain well-defragged hard disk drives
made thumbnail fading a little smoother in some situations
rendered images that aren't otherwise pushing any memory limits will now stay cached for less time (10 mins) while thumbnails will hang around for longer (24 hours)--this was previously 20 mins for both. you should now see less thumbnail page 'refreshing' after returning to the client after inactivity
sketched out basic classes for login and domain engines
sketched out basic class for url matching
sketched out basic classes for matching and transforming strings in certain login/domain situations
updated the manage export folders dialog's listctrl to the new control
updated the last media viewer menu code to the new system
removed some old proxy code that was sometimes intruding on the new networking engine. proper proxy support will come in a later version
finished moving the last misc network consumers to the new networking engine
greatly simplified and harmonised some of the new network job response processing
the gelbooru 404 issue should be fixed!
added a parsing patch for rule34hentai.net's mp4 links, which was causing 'link not found' failures on the hydrus parsing end. please tell your R34 subs to retry their fails
wrote a patch (actually updated an existing patch that had fallen to bitrot) for the danbooru booru (and extended it to anything else with 'Running Danbooru' in its html) to try to fetch the full-size image if one exists. this may not last long, so I do not recommend regular users rely on it, but feel free to play with it until I have a better fix in the new downloader engine
you can now add 'thread watcher' to the default import tag options. it has 'filename' and explicit tags options. unlike gallery downloaders, thread watchers will not inherit from the 'default' import tag options entry--you need to set the specific entry
added regular number key support to the new shortcut system
fixed the borked splash screen drag coordinate calculation
heavy session loads should be more polite to CPUs and window managers
pages should generally spawn with their proper name (rather than a flicker of 'page') no matter their spawning context
figured a way to make os x page tab drag-and-drop start on a single left-mouse-down event
in an effort to improve popup display on systems with virtual desktops, the popup message parent window will now not go from hidden to shown while the mouse is on a display screen other than the main gui
fixed a wacky bug where clicking on a page tab, then on its sort dropdown, then on a different tab, would invoke the Deep Ones to commence a page tab drag and drop event
fixed an index issue when unclosing some pages in certain orders
increased pre-processing disk cache population time
tweaked regular memory maintenance disk cache population timings
fixed shortcut-driven tag petition events when the tag has a sibling (e.g. if you have sibling pair (a->b) and a shortcut (key->flip a), hitting 'key' will now correctly flip a (appearing as b) on and off in the media viewer)
fixed some ugly layout sizing calculations on the path tagging dialog
updated the path tagging listctrl to the new control
updated the quick namespaces listctrl to the new control
updated the regexbutton's menu to the new menu system
fixed some typo errors with the regexbutton's entries!
removed the old gallery downloader unit tests, as these operated on the old network and downloader engine--new, granular tests will be reintroduced for the new parsing components as they are written
fixed an issue where the notebook 'motion' event was being consumed, disallowing the regular OS 'highlighting' on tab mouseover
fixed page tab drag and drop crash for Linux
fixed page tab drag and drop coordinate calculation for OS X
page tab drag and drop should now work for all platforms
new 'page of pages' pages now start with a blank file search page
fixed several issues with row display in the new listctrl
added a # index column to the file import status window, so you can resort the 'to be imported' order reliably
fixed an issue where the 'delete_file' application command was not firing on an appropriately mapped shortcut
fixed an issue where automatic bandwidth override would only fire if its page was in view. it now occurs at all times
fixed an issue where the file import status frame, as launched from manage subscriptions, would sometimes be non-responsive (typically for Linux)
fixed the 'pattern shortcuts' button on the export files dialog, which was misbehaving for certain choices for some window managers
fixed an unusual error where middle-clicking to create a new page while a menu was open would lock the application from receiving new key/mouse events until program focus was lost, wew lad
updated some old network code to the new engine
added some delay to network connection error reattempts so that instant fails due to local network disconnect (e.g. while computer is waking from sleep) do not consume all the reattempts too quickly
greatly simplified low-time-delta bandwidth testing and reporting, improving essential accuracy and overall feel
gallery download errors will now print additional info to the log--let's see if we can gather more info on the gelbooru 404s
wrote a patch for users of 2.X opencv who were running from source, who were unable to boot
due to hardware failure, had to make a new Linux build computer. some libraries are updated, but build _should_ be the same--let me know if not
added page tab drag and drop (windows only, have to iron out critical bugs for linux/os x).
dragging onto the middle of a normal tab will put the source tab there
dragging onto the edge of a tab will try to insert the tab there
dragging onto the middle of a page of pages tab will insert the tab onto the end of that page's list
dragging a page up a notebook level will work ok
dragging a notebook into itself will do nothing and not crash the client :^)
new pages that come from the main gui level (such as from pages menu or file import) now open in the deepest open notebook (previously, they would always appear in the top row)
fixed some misc page of pages bugs
fixed a bandwidth calculation that meant 1s time delta rules were working at 50% capacity (e.g. 1rq per 1s rule for domains were actually running at 1rq per 2s)
improved a bandwidth estimate calculation that was cutting out early in some situations for large time deltas
tag manager's page up/down shortcuts no longer mistakenly navigate the archive/delete filter
added a network timeout option to the 'connection' options page
reduced some hover window show/hide flicker when the media viewer is fullscreen and the OS has a taskbar that pops in for non-fullscreen windows (this mostly affected Linux)
fixed a long-time issue with yes/no dialog layout
improved some rendering of EXIF-rotated and -flipped jpegs. I expect to add more support for this (and retroactive image metadata-parsing to figure out correct reversed resolution--atm rotated images remain stretched) in the coming weeks
fixed an error when cancelling the booru-picker dialog from the page chooser dialog
if no files or all files are selected, the 'invert' select choice will no longer be shown (in this case, it redundantly does the same job as 'all' and 'none')
fixed an issue where setting namespace sort as default would persist through a client reboot
fixed an issue where force idle debug mode was not waking sleeping daemons
increased frequency of mappings processing reporting
wrote an exception and the needed maintenance code for the 'repairfilesystem' dialog to allow a proceed action if the only remaining incorrect paths are thumbnail paths--in this case, the client will create empty prefix folders and prompt the user to regen thumbnails
export filenames are now clipped to make <255 total characters in path
removed the old 'processing phase' option, which was no longer in use
removed the proxy settings from the 'connection' options page--the new engine does not use this old system, but if there is demand, a more flexible system will return
moved all page management (session load/save, new/close page, page navigation, page name maintenance, etc...) code from the main gui to the new PagesNotebook object
expanded the session object to hold nested page information
added a 'page of pages' page to the 'special' new page entry!
numerous other gui-notebook page-related event fixes and improvements
figured out cross-platform menu and other mouse event support for nested notebooks, but there may still be holes--please let me know if your new pages ever appear in the wrong tier!
ways to move pages up and down rows will come in the coming weeks!
main gui status bar now shows total bandwidth this session as well as current speed
added a cog button to network job controls that allow for manual- and auto-bandwidth-override
added a 'blocked?' column to the review bandwidth panel to quickly see which network contexts are currently available to do work
added a button to reset 'default' and 'global' bandwidth rules to review bandwidth panel
merged the network request start test and consumption code into one transaction, stopping some accidental overconsumption when the engine was under heavy load
did some logic work to make sure unusual network context rule/usage situations are visible on the review bandwidth panel for editing
network jobs now report more information as they get ready to start, including while they are held up waiting for a download slot
added a simple 'network profile mode' to the debug menu that atm just prints a summary of new jobs
fixed an error that could sometimes be a crash when 'review services' was opened while no pages were open in the main gui
fixed the pixiv login test button with a hacky workaround--I will make extensive proper login testing gui when I move to the login engine
fixed an issue in the youtube downloader
if the bandwidth or session managers are missing on boot, empty defaults will be created in their stead
bumped the max page limit up to 150--I expect to increase it more in the coming months as I rejigger how some gui stuff is laid out
renamed 'sort by age' to 'time imported'
fixed and improved some test code
cleaned up some shutdown code
should have fixed a rare unicode conversion issue when printing to log
split the sort dropdown into two, splitting the sort type and sort order
file sorting works more intelligently behind the scenes in several ways
added 'sort by number of tags' to sort options
session pages now remember their sort status!
sessions also more reliably remember their actual thumbnail order for all pages (typically, this matters for importers atm)
network job controls now report an estimate of how long they will have to wait for bandwidth
subscriptions will now show some 'no more bandwidth to download files' text if they have to stop because of bandwidth rules
subscription 'should I start/continue' testing now has a little padding to forestall some potential unexpected long delays in operation due to edge cases
the edit subscription dialog now uses the new file import status control
fixed an issue with hentai foundry filters not applying (they added some categories since the downloader last worked, breaking the POST form), hence hiding most non-vanilla results
finished the 'database->migrate database' dialog and its help and removed the 'under construction' labels. the new help page is also now linked from the standard index.html
fixed an issue where pages would sometimes never initialise (due to being queued up after an infinite job like the network engine's mainloop!)
improved how all long-job threads are spawned
improved some more thread scheduling logic that meant some long-term jobs could be stacked undesirably
expanded how the new listctrl updates and deletes its data
reduced flicker on the new listctrl update events
the file import status window now uses the new listctrl
the manage subscriptions dialog now uses the new listctrl
collections now track their tags more accurately
moved the 'delete original files after success' checkbox up to reduce misclicks on ok'ing the file import dialog and added a bit of red warning text whenever it is on
sped up some behind-the-scenes content processing for large pages
improved how the program cleans some things up during exit
the issue with some clients not clearing their exit splash screen until a mouseover event should be fixed
at 120 open pages, the client will inform the user about the approaching max page limit
by default, import folders no longer delete anything. default is to ignore original files in all cases
drag-and-dropping a 4chan or 8chan url onto the client will now automatically open a thread watcher for that url
fixed an issue where web domain or subscription network contexts that included unicode characters in their context data were unable to serialise and save to the db, causing error spam when the bandwidth manager attempted to save itself
subscriptions will now show a network job control in their popup as they do network work
subscriptions will cancel more reliably during gallery parse
subscriptions now have some 'delay' logic that will stop them sometimes restarting as soon as they are cancelled or otherwise have to stop mid-work
subscriptions will now tolerate up to five 'already in cache' urls per page parse until it considers the page 'already seen'. this is to catch the odd additional late insert and avoid the problem of a page updating and shuffling everything up one as the subscription walks through pages
maintenance modal popup messages should no longer appear if their jobs are very quick
queued up modal popup messages will no longer flicker their dialogs on a de-minimise
should have fixed an issue where modal popup messages could sometimes error out on a close attempt, locking the client's whole gui and requiring a force quit
fixed an issue where the modal popup message dialog was OKing on a close attempt of a non-cancellable job, despite presenting user text indicating otherwise
did a bunch of data work on pages
page tab names will now be clipped to 20 characters by default
pages now show (num_files) after their name by default, although you can set this for only import pages or turn it off completely
options for these new behaviours are in options->gui--and page names will update immediately on dialog ok
subscription and import folder 'show files' events (and a couple of other misc occasions) will now launch their page tab with the sub/folder name rather than a flat 'files'
wrote a new listctrl class with custom sort code that does not suffer from the 'crash when sorted while many pages open' bug. it also handles data in a simpler way for hydrus
review bandwidth listctrl now uses this new class and should now not crash your client. I will replace all the other listctrls with the new class over the coming weeks
temporarily, no old listctrls will auto-sort themselves (as this causes a crash for many users)! if you do not suffer from the crash, please sort them yourselves for now
reduced some large-scale gui import lag:
reduced content processing CPU load on clients with many thumbnails open
massively reduced content processing CPU load on non-current pages
offloaded newly-imported file thumbnail detection and generation to a non-gui thread
migrate database dialog now lets you move the whole database and all portable locations, which requires a client shutdown
the raw url downloader will no longer have a problem with pasted url lists that include empty newlines
all the downloaders/importers now sleep on cleverer event objects so they will burn less idle cpu and wake as soon as they have new work to do
fixed and otherwise improved some gallery downloader timing logic
hdd import pages now use the file import status control
the thread watcher's controls now try to wrap in a single sizer. it uses less space, but might sperg out, let me know if it is a problem. I'll replace the whole watcher timing system in the new engine anyway
fixed a gelbooru (and possibly others) booru parsing bug that meant half of the pages in the gallery walk were being skipped (e.g. 'mogudan' was producing ~350 files when there were actually 690 in the list)
fixed some dialog panel layout scrollbar-cutoff in sevaral Linux places and perhaps elsewhere
cleaned debug menu a little and added a save 'last session' entry
fixed an issue where some kinds of media would error on notification of new url association
clients will now save small transaction progress reliably within ten seconds no matter how idle they are
deleted the old 'gui capitalisation' option--I never got around to expanding it beyond a handful of menu labels, it was always too much to work on for too little reward
improved support for certain broken videos--these will import ok, but full rendering might be borked to different degrees, so let me know how it goes.
cleared out some old content processing code
cleared out some old unused db data
removed all old 'waiting politely' download settings and gui code
converted gallery downloaders to the new network engine
greatly simplified how gallery downloaders report network activity and converted them to show the new network job control as well
subscriptions also work on the new engine but will not show network gui yet
hacked hentai foundry and pixiv login to use the new network engine
successful logins to hf or pixiv now print to the log
the new network engine now clears temporary session cookies after 90 mins of inactivity
gallery downloaders and subscriptions now use the new 'downloader instance' and subscription bandwidth rules. by default, this means downloaders will do small bursts every five minutes and that subscriptions will do at most 256MB per day
subscriptions' bandwidth use is now listed by name in the review bandwidth panel
subscriptions use a new bandwidth test to determine if they should start or continue based on current bandwidth limits. it should mean subs do a good bit of work and then stop when they are supposed to without ever waiting on bandwidth more than 30s or so
thread watcher and page of images now ignore bandwidth limits when doing their 'page' fetching part
gallery page fetching will ignore bandwidth rules (in order to stop gallery walk desyncs from having to wait a long time)--it will fetch one page per five seconds
the thread watcher and page of images importers now work on their files and page-checking simultaneously--also, the page of images will process its page queue at any time, not only when the current file queue is finished
the gallery downloader page now works on its files and gallery page parsing simultaneously!
wrote a 'file import status' control to better wrap up the import summary, progress gauge, and file import status button into one panel
thread watchers now use the new file status control
'page of images' downloaders now use the new file status control
the gallery downloaders now use the new file status control
all network jobs will now retry up to four times on the BadStatusLine ConnectionError, which seems to be a TLS (https) negotiation timeout/remote termination
all requests on the new network engine will now timeout after ten seconds
they will also retry on generic timeout errors
popup messages can now be shown in 'modal' mode as a dialog that prohibits interaction with the rest of the client.
these will not boot while the client is minimised
database maintenance routines will now all publish messages like this
'migrate database' panel now publishes a modal message when it does a file rebalance
rewrote the controller-side pubsub pipeline to respond faster and consume fewer program resources, particularly for the client
import pages now update themselves in a less spammy way behind the scenes, meaning more active pages can be open at once without them stepping on each other and clogging things up
simplified how importers set their status information in several ways
reduced a swath of pubsub spam related to content updates
improved how spammy small jobs are written to the profile log
reduced text flicker on all download pages
more misc pubsub improvements
reduced some gui update/refresh spam on hidden pages
cleaned up a bunch of database->gui message reporting and cleanup code
added close other/left/right pages to tab right-click menu
the top-right media hover window will no longer refit-flicker on a ratings change
wrote a new panel wrapper for listctrls that handles the underlying row of buttons in a neater way and automatically disables them if they are nullipotent (mostly, this means greying out 'delete' buttons when nothing is selected)
several listctrls use this new panel
when it is not strictly necessary, videos that are >30MB will no longer use the CPU-expensive manual frame count parsing
a problem where newly reloaded thread watchers could sometimes stick in a 1/1 initialisation state _should_ be fixed
fixed gelbooru url parsing (they stopped using the janky redirect.php urls)
fixed an issue that meant ipfs pin was erroring when trying to show gui-side
'page of images' downloaders now protest if they are told to close while working
some small layout and status text fixes
adminside petition processing now has a 'flip selected' box to flip checked status of all selected contents
adminside petition processing contents chechklistbox now supports multiple selection
the bandwidth engine now recognises individual thread watcher threads as a network context that can inherit default bandwidth rules
tweaked default bandwidth rules and reset existing rules to this new default
review all bandwidth frame now has a time delta button to choose how the network contexts are filtered
review all bandwidth frame now updates itself every 20 seconds or so
review all bandwidth frame now has a 'delete history' button
review all bandwidth frame now shows if services have specific rules
review all bandwidth frame now has an 'edit default rules' button that lets you select and set rules for default network contexts
review network context bandwidth frame now has a bar chart to show historical usage!
bar chart is optional based on matplotlib availability
review network context bandwidth frame now lists current bandwidth rules and current usage. it says whether these are default or specific rules
review network context bandwidth frame now has a button to edit/clear specific rules
rows of bandwidth rules and current usage, where presented in ui, are now ordered in ascending time delta
misc bandwidth code improvements
client file imports are now bundled into their own job object that generates cpu-expensive file metadata outside of the main file and database locks. file imports are now much less laggy and should generally block the feel of the ui much less
removed the database 'rebalance files' menu entry
removed the 'client files location' page from options
db client_files rebalance will no longer occur in idle or shutdown time
(this stuff is now handled in the migrate database dialog)
'migrate database' now uses a dialog, meaning you cannot interact with the rest of the program while it is open
migrate database now has file location editing verbs--add, remove, +/- weight, rebalance_now. thumbnail location and portable db migration will be added next week
flushed out the backup guide in the getting started help, including to reflect the new internal process
the client now saves the 'last session' gui session before running a database backup
the shutdown maintenance yes/no dialog will now auto-no after 15 seconds
gave status bar tabs a bit more space for their text (some window managers were cutting them off)
tumblr api lookups are now https
tumblr files uploaded pre-2013 will no longer receive the 68. subdomain stripping, as they are not supported at the media.tumblr.com domain (much like 'raw' urls)
pages will now not 'start' their download queues or thread checkers or whatever data checking loops they have until their initial media results are loaded
key events started from an autocomplete entry but consumed by a higher window (typically F5 or F9/ctrl+t for refresh or new page at the main gui level) will no longer be duplicated
fixed a shutdown issue with network job controls that could break a clean shutdown in some circumstances
if the user attempts to create more than 128 pages, the client will now instead complain with a popup message. Due to OS-based gui handle limits, more than this many pages increasingly risks a crash
if the client has more than 128 pages both open and waiting in the undo menu, it will destroy the 'closed' ones
converted the page of images downloader to the new network engine
converted the thread watcher downloader to the new network engine
rejiggered page of images and thread watcher management panels to better separate the two loops they work on
rejiggered some more downloader layout stuff in prep for a new control to better display url cache and overall import status--downloader-global pause button is now always at the top, for instance
removed 'waiting politely' indicator from download pages that use the new network engine
network job controls now have a simple border
network job controls will show current bytes progress and speed in a more intelligent and useful way--no longer spamming the initial 0KB/s, for instance, and flashing more useful summary information when downloading many small files
network job controls now display their current bytes progress and speed on a separate right-aligned text beside the main status text (to stop the speed text jumping around so much)
network job controls will not show the current download speed until some bytes are actually read
wrote a first version of 'services->review bandwidth' review frame, which lists all network contexts with data in the past month
wrote an independant and live-updating review frame for indivdual network contexts, launched from the review bandwidth frame, which shows current and historical bandwidth use for that context
the main gui status bar now reports current bandwidth usage! updated once a second
database->backup process now has some nicer gui, better and more reliable workflow, improved file copy notification, and now correctly obeys a popup-cancel command
started the new 'migrate database' gui under the database menu. it currently shows live usage and will gain some verb buttons to atomically alter things hopefully next week
moved from a strict begin/commit db model to a save/release system that only commits changes to disk at most every ten seconds or so. small 'write' transactions will now often have no disk-sync overhead and so are super fast, particularly when they come in a large batch.
simplified database connection, disconnection, begin, commit, and rollback code significantly
the url table in the client database is now non-unique (i.e. multiple files can have the same url). this may lead to undesired 'already in db' or 'deleted' statuses in some downloaders as they check for url status before starting their downloads. if this change turns out to a be a big pain in the future for pixiv manga or whatever, it may be revisited.
fixed a bug in deviant art parser that sometimes meant nsfw urls could not be found
fixed an issue where progress gauges could sometimes get stuck in a pulsing state
fixed a bug with the new optimised result building which sometimes occured when all the files in a batch had common ipfs service membership
improved support for unusual videos
improved accuracy of duration calculation for unusual videos
improved database optimisation after initial and ongoing repository processing
a 'wake subscriptions' event now occurs even on 'manage subscriptions' cancel (meaning temporarily-dialog-paused subs will always continue after the dialog is closed)
added a careful manual commit to stop the possibility of the definition-desync some users noticed after a power-loss during a big repository sync commit
removed some redundant old db compatibility code
permitted sqlite installations that allow in-memory temporary tables to use them (but not for vacuums, which tend to be a bit big for this stuff)
greatly improved how gui sessions are loaded--now the page tabs are loaded instantly, but the thumbnails are loaded in the background. session loaded should be significantly less laggy and buggy
the issue of pages sometimes initially sizing at zero size (which could be caused by minimising the client while a session load was ongoing) should hence be better/fixed completely!
gui sessions will now load their files in the exact order in which they were saved--behaviour that I think was not previously always reliable
more general network code work and polishing
added and improved unit tests for network code
improved how short-time-delta data bandwidth is reported
improved how short-time-delta data bandwidth is tested
wrote a networkjobcontrol to display and control the new network job object
tumblr parser now produces 68.-less urls
tumblr parser now produces https urls
cleaned up tumblr parser a little
url caches will clip existing tumblr urls of the 68.-subdomain and convert to the new raw format, clearing out dupes along the way
url caches will convert existing tumblr and pixiv urls to https, clearing out dupes along the way
the pixiv parser now deals with missing creator/title tags without errors
extended the specific file domain tag cache to also store deleted mappings, resulting in much faster request building for clients with large numbers of deleted mappings
improved some downloader page queue text display timing
added support for more types of mp4 file
improved how some memory maintenance calls work
improved how hydrus datacaches track their recently-used-data fifo list
pages now regularly clear out spare thumbnail canvas bmps
pages now regularly clear out cached thumbnail canvas bmps when they are not the currently viewed page
import caches, when asked for url membership, will now test both the http and https versions the url
maybe improved how 'open in file browser' works in windows
fixed the 'recount video frames' advanced thumbnail menu entry, which wasn't working with the new ffmpeg wrapper
moved some bloaty master hash data out of client.db and into client.master.db
pubsub profile will no longer profile 'message', as it just makes for spam
reduced some serverside pubsub spam
reduced some significant clientside pubsub spam that I think was smashing the gui event loop at inconvenient moments
improved some client shutdown object sync code
fixed an issue where some duplicate maintenance popups would not clear themselves up properly if interrupted mid-job
cleaned up some http-https conversion and comparison code
fixed some status-setting code that meant thumbnail pages were sometimes setting status after they were replaced and scheduled for deletion
added 'manage known urls' to media right-click menus!
double-left-clicking on any video animation will 'open externally'!
added an option to options->gui to set where new pages will appear by default--either far left/right or left/right of current page
the tumblr parser now produces '_raw' urls when the post was posted in 2013 or later
created a new 'number' subtag cache that will be populated on update
created a new 'tag as number' system predicate that can search for, say, all 'page:' tags > 200
bandwidth management now tracks requests and num_bytes more sensibly
bandwidth tracking objects can now better handle bandwidth usage and rule application in short intervals (i.e. sub-5-second) (however, the current networking engine cannot yet use this information accurately)
wrote unit test for bandwidth tracker, including for the new short interval timing
fleshed out the new network engine
fleshed out the new network job
thread watchers can now have a time delta that includes days (giving a max check period of 360 days, wew)
adminside mapping petition processing now has adaptive max total petition weight--lower file count range petitions will have much higher max total permitted weight
added a new 'callto' debug reporting mode that reports on current thread pool jobs
improved calltothread pre-spawning checks to reduce outside chance of deadlock in busy periods
the advanced review services repository panel buttons are now hidden unless in advanced mode
eliminated some animation buffer looping redundancy
fixed a little animation next-frame prediction code
fixed up some '1 minutes'-type time_delta->string conversion
fixed up label on time delta control button
fixed some shutdown thread interactions
fixed an issue where sometimes empty tags could be entered into the manage tags panel
added some pydeadobject error handling during client shutdown
refactored some db multi-tag->file search code, cleaned up wildcard searching
fixed video parsing when the video metadata includes random non-utf-friendly garbage
fixed video parsing when ffmpeg reports no fps at all
improved video frame counting accuracy
thumbnail waterfall process now adapts to the current speed of the cache and can hence use significantly less overhead
the thumbnail fading process now adapts to the current rendering speed, delivering smooth fade when CPU is available but otherwise skipping frames and more reliably filling in thumbnails within a quarter of a second
canvases now draw their thumbnails with slightly less overhead
increased database synchronous pragma to FULL to better safeguard against power loss during multiple-db commit--we'll see if it slows things down too much, and maybe add an option about it
cleaned up and improved some client gui pubsubs
made pubsub profile mode far less popup-spammy
if a profile takes less than 20ms, it now won't be fully printed to the log
tweaked some more server object maintenance code
added 'changelog' link to help menu
updated lz4 library and fixed some old deprecated calls
planned out new networking engine and started the principal objects
renamed the 'exact match' duplicate status to 'same quality', to reduce confusion on what is appropriate for this status
the duplicate filter, on hitting the delete key, now offers the option of deleting both files
the duplicate system now combines duplicate status setting and the consequent batch of content updates into the same database transaction, speeding things up
the duplicate system now batches multiple duplicate status setting into a single transaction, massively speeding up large filters or thumbnail status set actions
misc duplicate help tweaks
you can now edit the default duplicate merge options from the new thumbnail duplicate menu
the duplicates page's jobs are less demanding on gui time and take better breaks if something else happens
renamed the new dupe system predicate to 'system:num duplicate relationships' to clarify what it searches
for normal queries, current and pending mappings are now fetched from a faster mappings cache. you should see faster result building across the board, particularly on fresh boots or otherwise slow-disk-access systems
added a prototype 'advanced mode' (defaulting to off, so experienced users will want to turn it on) under the help menu that will enable menu items that are often not helpful to new users. I will add more things to this in future, suggestions welcome
the new thumbnail menu dupe relationship set stuff is now considered advanced
thumbnail menu find similar files is now considered advanced
the thumbnail menu copy hash entries are now considered advanced
advanced content update buttons (on manage tags and review services) are now considered advanced
added an advanced mode 'open file location' entry to the thumbnail share menu, which will open the file in your OS's file explorer (not available on Linux)
added an advanced mode 'correct video frame count' thumbnail menu entry that will force-apply last week's more accurate video frame counter to correct and videos that render too fast and then cut off
fixed many entries on the media viewer menus, which were being blocked by an over-eager 'can continue' test and hence silently failing
fixed the youtube downloader on Linux and OS X--both now use youtube-dl
fixed an issue where the GetLighterDarkerColour function was producing very bright alternates to very dark colours (meaning dark dupe filters were having their bright background text rendered unreadable)
improved video frame number parsing accuracy
improved the accurate version of video frame number parsing accuracy, particularly for longer videos
the network engine now reports 5xx http status codes as ServerException to better contextualise to the user what went wrong
adminside mapping petitions are now sub-ordered by tag
adminside sibling/parent petitions are now ordered by the 'older' tag and sub-ordered by the 'newer' tag
censorship taglists are now roughly sorted
reduced default shutdown work max time to five minutes
added a duplicate entry to the thumbnail right-click menu
the new duplicate menu will now attempt to fetch known duplicate counts for the focused file for the current file domain (if the db is locked, it will say so rather than block)
the new duplicate menu's counts in the thumbnail menu can be clicked to show those files in a new page
the new duplicate menu allows you to set the four main dupe statuses (with default merge options), or a customised verson of the same, or set unknown/potential, or delete dupe status completely, for all the possible pairs in the current selection
added tentative support for variable framerate files, manually counting up their frames and displaying them with an averaged constant framerate in the client renderer. this now permits a number of webms that were previously 'mime unsupported'. let me know how it works for you!
cleaned some server locking code, hopefully fixing the ssl handshake issue (which is actually a deadlocking issue)
export file dialogs will remember the last value of 'export tags to .txt files?', and clicking that checkbox will always launch the dialog for editing (rather than clearing if there are some set)
fixed the advanced content update (as used by service-wide update), which was not able to differentiate namespaces since the recent service changes. I apologise to anyone affected by this--I will add some unit tests to make sure it doesn't happen again
reworked and cleaned some canvas event handling code
the canvas will now accept mouse wheel events even when it does not have focus
the canvas frame will no longer accept keyboard events (and hence activate fullscreen_switch) if the mouse is over a flash window
hydrus servers now respond to /robots.txt with a 'disallow all'
servers will now provide mappings petitions to admins in groups of similar size, rather than mixing petitions of weight 5000 with a hundred of weight 1
fixed the back/skip buttons in the archive/delete filter's top hover window
the file path tagging dialog's namespace and regular regexes now support parenthesis groups, and in that case will take each of those submatches instead of the wider match
hence regexes with groups will now compile in that dialog
you can now 'new page here' on the final (rightmost) page tab
fixed an issue where media-viewer-launched manage tags frames would yield focus to the main gui if they had opened a sub-dialog while they were open
the tag censorship edit panel now has a 'help here -->' label in blue text
file imports will fail properly when the final temp_path->client_files file copy operation fails (due, for instance, to the destination being on a removable medium that was recently disconnected)
fixed serverside upnp maintenance, which was silently failing early, and also updated it to the new server object code
wrote a new checklistbox dialog that handles the underlying data explicitly rather than messing around with external text->data maps
replaced all the old string checklistbox dialog instances with the new one
subscriptions will now correctly clear their 'recent error' status on a successful sync
refactored and cleaned how basic mediaresult objects store and consult simple file information
locationsmanager objects now handle inbox status and the related content update events that affect it
fixed a bug in HydrusTagArchive when attempting to figure out hash type from an existing hash
removed the obsolete servertoclientcontentupdatepackage object
the duplicate filter will now maintain zoom on files with the same ratio
split the duplicate merge options into separate tag/rating controls--you may see some duplicate service entries, but these will be cleaned on your next shutdown
duplicate merge options now allow syncing 'archive' status
duplicate merge options now allow 'delete both files', which you may find use for in custom actions
created a tag censorship object to handle and action a rich tag censorship ruleset
tag duplicate merge options now have and use this tag censorship object to filter which tags are merged, with an initial value of 'let everything through'
wrote a tag censorship edit panel and tied it into the duplicate action edit panel so these new tag censorship objects can be edited
added an optimisation to the duplicate status setting code--if two files are better/worse, they are inherantly duplicates and so 'not dupe' and 'alternate' relationships apply to both equally and can be duplicated
fixed and culled and normalised the 'this has more tags' dupe filter statements to be more accurate and useful
added 'this has a larger filesize' type-statements to the dupe filter
created a 'system:duplicate relationships' predicate that can find files based on how many duplicate relationships of a particular type they have
cleaned up some misc duplicate filter code and added some tooltips to the top hover window dupe action buttons
added 'move to left/right end' to main gui page tab right-click menu
added 'new page' and 'new page here' to main gui page tab right-click menu
you can now right-click for a menu from empty tab space on the main gui
the main gui statusbar now updates more efficiently when under heavy refresh load
the main gui statusbar now shows db read/write/commit status and sets the current db job summary as its tooltip--if you experience persistent hangs, please hover over the statusbar and report what you see!
export tags to .txts checkbox will now default to 'all services on' when checked
fixed the thread watcher, which was accidentally disabling its text input early
'similar files' searches launched from the thumbnail menu will now default to 'my files' file domain rather than 'all local files'
downloader pages will now correctly sort their files on initialisation
refactored and generally cleaned up some collect and sort code
fixed some unlikely-but-possible collect/sort bugs
fixed some bad layout in the top-right hover window that was making it grow unreasonably tall when many urls were shown
on the different download import pages, the progress gauge that shows file download progress will now reset back to 0 as soon as the file download is complete
fixed a problem where video imports with unicode characters in their path were failing to mime-parse
improved the file import status update pipeline to better deal with large transactions (like skipping/deleting/retrying a thousand rows at once). all these big transactions should lag the gui far less
improved some misc import status cache code
made first step in a big size rewrite job that will size many elements according to local system font size rather than specific pixel values
hydrus servers now explicitly default to TLSv1.2--we'll see if that clears up some of the handshake timeout problems we have recently seen
cleaned up a bunch of possible pydeadobjecterrors when the new review services panel is closed
improved and rescheduled gelbooru redirect url purge
added a catch-and-recovery to hydrus network session initialisation, which may sometimes receive invalid data after a service deletion
added similar catches to tag parent/sibling initialisation, which apparently can be vulnerable to a similar invalid data problem
I think I cleaned up some more Linux ClientToScreen console errors
refactored and cleaned some frame size event responsibility
refactored and cleaned the panel and controls that display file import status
the duplicate filter now loads new pairs off the gui thread. it will display 'loading pairs...' during this time
media viewers of all kinds are now more comfortable displaying no media (when this occurs, it is usually a frame or two during startup/shutdown)
the duplicate filter now responds to any media_viewer_browser navigation commands (like view_next) with a media switch action
you can now alter the duplicate filter's background lighten/darken switch intensity from its top hover window's cog icon
fixed a bug in the new dupe pair selection algorithm that was preventing pairs from being presented as groups
the duplicate filter will now speed up workflow by automatically skipping pairs when you have previously chosen to delete one of the files in the current batch
auto-skipped pairs _should_ be auto-reverse-skipped on a 'go back' action
added a |< 'go back' index navigation button to the duplicate filter top hover window
the duplicate filter now displays several 'this file has larger resolution'-type statements about the currently viewed file. it lists them on the top hover window and in the background details text underneath
the duplicate filter _roughly_ attempts to put the better file of the two first. this will always be indexed 'A'
the duplicate filter now shows done/total batch progress in its index string--not sure how clear/helpful this ultimately is, so may need to revisit
an unusual bug where Linux would spam the 'All pairs have been filtered!' duplicate filter message over and over and then crash _should_ be fixed--the filter no longer waits for that message to be OKed before closing itself
drag-and-dropping text onto the client will now a) open a url import page if none already exist and b) put the dropped text into the input box of the first open url import page (and focus it, so you can quickly hit enter)! This works when dragging text links from browsers, as well
you can now 'append' gui sessions, which will just append that session's tabs to whatever is already open--say, if you have several 'favourites' pages you want to be able to quickly load up without having to break your existing workflow
ipfs services now have a 'check daemon' button on their review services panel which will test the daemon is running and accessible and report its version
fixed the 'test address' button for ipfs services on their manage services panel
the client can now automatically download files it wants and knows are on an ipfs service
middle-click on an 'all known files' domain thumbnail will now correctly start a download (as long as a specific remote file service is known)
the multihash prefix option is reinstated on ipfs manage services panels
the gelbooru parser now discovers the correct page url to associate with its files
wrote some redirect fetching code to fix the gelbooru bad urls issue
discovered a quicker fix for the gelbooru issue--the redirect location is the garbage in the original url in base64
all downloader/subscription url caches will purge any old gelbooru 'redirect.php' urls on update
fixed an issue where 'previously deleted' gallery/thread imports were returning 'fail'
fixed a problem that was causing some redundant laggy work in adminside petition processing
thread watchers will now remember their file and tag import options through a session save even when no thread url has yet been entered
fixed an issue where media 'removed' from a media viewer view of a collection resulted in the entire collection being removed at the thumbnail level
fixed an issue where media deleted from a media viewer view of a collection resulted in the media not being correctly removed from selection tags
tag, namespace, and wildcard searches on a specific file domain (i.e. other than 'all known files') now take advantage of an optimisation in the autocomplete cache and so run significantly faster
fixed a hover window coordinate calculation issue after minimising the media viewer on some platforms
removed some 'all files failed to download' spam that could sometimes occur
the duplicate filter now supports shift+left-click to drag, like the archive/delete filter (this remains hardcoded for now)
if a pair in the dupe filter has the same resolution, they will now maintain zoom and pan when switching back and forth (I might increase this to work for same ratio as well, let me know how it works in the real world)
the duplicate filter will show a lighter/darker background colour as you scroll the current pair
the way lighter/darker and alternate lighter/darker colours are calculated is now centralised and should be more reliable in edge cases
improved the dupe filter pair selection algorithm--it now chooses pairs more reliably under edge case conditions and prioritises decision-groups that have high potential decision value. it should also run a little faster
increased the dupe filter batch size to 250, let's see if it causes any problems
the close button on the dupe filter's top hover window now works
fixed the duplicate filter disappearing pairs that were skipped (meaning you could not go back to revisit them)
fixed a frequent deadobject error when the dupe filter closes
the shutdown 'maintenance due' test is less sensitive to dupe search tree rebalancing, which typically only takes half a second
the archive/delete filter now uses the new shortcuts system for both keyboard and mouse input
the archive/delete filter now intercepts archive or delete commands from different sources ('media' shortcuts, top hover frame button presses) more reliably and converts them into filter actions (hence moving on to the next file)
you can now move pages one to the left or right from their menu!
on the top-right hover window (and the background underneath), ratings are now on the top, and hence will always be in the same location as you scroll through your media regardless of known urls, remote location, or inbox status
autocomplete searches for tags with apostrophes, quote marks, braces, brackets and paretheses should be more reliable
urls are now associated with files through the same 'content' pipeline as tags and ratings and so on
gui-level media is now aware of the 'new url content update' event and will update and redraw itself appropriately
fixed deviant art nsfw parsing, but it might not hold for long. proper fix here is to wait for the downloader overhaul
networking engine now uses the 'requests' module's CA .pem (which the overhaul will be moving to anyway), which should reduce the frequency of ssl verify failures (gelbooru on the relatively new 'Let's Encrypt' CA had this problem for many users)
fixed the networking engine redirect parsing for gelbooru's unusual location header. unfortunately, gelbooru is still giving mickey-mouse garbage redirect urls from its main thumbnail pages, for which in this engine there is no immediate fix
neighbouring .txt tags will now be properly cleaned and sibling-collapsed in the path tagging dialog
neighbouring .txt tags will now be properly sibling-collapsed in the import folder workflow
making a media viewer borderless fullscreen and back will now recenter the media (previously, the current drag delta was not reset, so this frequently put media off-screen)
fixed a serious issue where the media viewer could lock the client up on opening with a video if its gui options set 'remember size' to false
tag import options objects will cleanse themselves of missing services on options save/client shutdown
manage tags now defaults to cross-referencing 'my files' on the 'local tags' domain, where 'remote' (i.e. deleted) files' tags are not useful
reduced memory use when importing large pngs with transparency
servers will now cap the size of their mapping petitions so as not to ovewhelm the admin processing them (they now won't be both >20 tags and >1000 total row weight)
the media viewer's manage tags frame now listens for content updates from outside, so if you alter an in-view file's tags (such as with a shortcut key), the manage tags dialog will update as it happens
moved a number of the buttons on the top hover windows to the new unified internal command engine (which the new shortcut system also uses)
disk cache maintenance uses fewer resources but now occurs in the foreground (and should hence more reliably maintain the cache)
critical repository service id lookup errors will now automatically reset the repository's processing cache and better inform the user of what has happened. if you see this, please let me know the details and how this error fired in the real world
updated and reinstated the ipfs service panel in review services
updated some ipfs service code for the new service system
misc refactoring and cleanup
more cleanup and deletion of redundant old pubsub command code
updated ubuntu build machine to 17.04 and opencv 3.2
the duplicate processing page is now on the new page picker, under the new menu 'special'
improved the dupe pair selection algorithm--it should be faster for everyone
added help icon to dupe processing page--it can launch of a couple of message boxes with simple help, or you can open some html help
wrote some dupe processing html help!
the dupe processing page now refreshes its numbers when the dupe filter closes
the archive, inbox, delete, and undelete buttons on the duplicate filter's top hover frame now work
delete and shift+delete keys now work to delete/undelete a file in the duplicate filter
the mouse cursor now hides/shows on halt and new movement on the duplicate filter
you can now set 'content' shortcut actions (adding tags or ratings) to shortcuts in the 'media' reserved shortcut set (which are always on in media contexts), although they don't yet work in the thumbnail view yet
added a help button to the manage shortcuts panel
the edit shortcuts dialog will now now allow you to name a custom shortcut set to one of the reserved names and will explain the problem in a message box
import pages update more efficiently during periods of busy cpu
import pages will use less idle cpu time generally
import pages will adaptively use less cpu time when they are in the undo deletion queue or have no files to import
the page of images downloader will spam a little less idle time
the canvas background details and the top-right hover window will now only show 10 urls max
added a help button to the tag import options panel to better explain the namespace selection and explicit tags
added 'gui report mode' to the help menu, which will report key and mouse shortcut events and and matched commands
the adminside petition panel now sorts multiple petitions by number of files descending
created a new object to hold tag and rating merging and 'worse file' deletion options
wrote a dialog to edit this new object
established some simple defaults for this object for different duplicate status actions
the cog icon on the duplicates filter now lets you edit these defaults
the 'custom' duplicate filter action now works--first by asking you what status you want to set, and then by throwing up the new merge options dialog to tune it to whatever you like
wrote comprehensive unit tests for the new object
fixed the super slow dupe filter launch time problem
added a 'known urls' submenu to thumbnail and browser canvas right-click menu that lists all known urls for a file with an option to launch or copy the url
added known urls' hosts to the top-right canvas details background, just below where known file repos are listed
added the same known urls' hosts as clickable hyperlinks to the ratings hover window that pops over that top-right area
added 'delete from deleted files' action to the local tags's service-wide update panel. it will limit the deletion to mappings that are currently on files that have been previously completely physically deleted from the program
fixed namespace filtering on service-wide update panel
the hentai foundry downloader broke, so the update code will pause all hf subs. the solution is not trivial (it is part of the downloader overhaul), but I will try to fix it soon
debuted a new question-mark help button to better explain .txt tag importing on the manual import tagging dialog and the manage import folders dialog
fixed a small potential error due to bad parsing in the 'page of images' downloader
fixed a typo bug that stopped the 'delete shortcuts set' action working in the manage shortcuts dialog
I may have fixed an issue where the server was sometimes not shutting cleanly with a keyboardinterrupt
fixed the media embed button not reliably updating its thumbnail
fixed an issue where a dummy animation bar was displaying on embed buttons that showed static images that included transparency
the serious db missing tag and hash states will now not throw an error but will inform/spam the user (and hence not prohibit a boot)
attempting to open a second manage tags frame from the media viewer will now instead put the focus on the first (previously, multiple manage tags frames could be made)
misc db code cleanup that should result in faster result building in certain situations
the duplicate filter now processes pairs in batches and hence supports 'back' actions to revisit decisions. you will be prompted every fifty or so pairs to commit and checkpoint your progress
the duplicate filter now presents related pairs together, rather than picking at random
fixed a bug in duplicate filter shortcuts initialisation
simplified duplicate filter default shortcuts (these will be overwritten on update) to only use simple left- and right-click for 'this is better' and 'alternates', since those are by far the most common actions. middle-click now goes back, like the archive/delete filter
converted old 'main shortcuts' system to the new shortcuts system, also splitting it up into 'media', 'main_gui', 'media_viewer', and 'media_viewer_browser' constituent parts that will be applied in different contexts
because the change is so significant, all clients will have their old 'options' shortcuts reset to the new default--I expect to expand shortcuts further in the next few weeks, so this default-overwriting will likely happen again, so you will likely wish to wait before recustomising your basic shortcuts
completely eliminated the old main shortcuts system--all references now bodge with the new system to varying neatness
as the old main shortcuts system no longer exists, the 'shortcuts' page on file->options is gone--all shortcuts are now managed through file->shortcuts, which is a completely revamped version of the custom filter shortcuts editing dialog
selecting shortcut commands is significantly simpler for the reserved shortcut sets
all media viewers with hover windows now have a 'keyboard' shortcuts icon button on their top hover window--it links to the manage shortcuts dialog as well as the current active custom shortcut sets and default custom shortcut sets
as the above system supercedes the old custom filter system, custom filters are completely removed from the program! your existing custom shortcut sets will survive, but you probably want to purge them of all the redundant junk they still have
all the media canvases use the new shortcuts system
the canvas frame uses the new shortcuts system
the thumbnails canvas uses the new shortcuts system
the main gui uses the new shortcuts system
some other misc places use the new shortcuts system
most simple shortcut command actions have been renamed to be more readable
some shortcut actions, like zoom stuff, is no longer hardcoded!
mouse shortcuts are still not widely supported!
the shortcut command edit dialog now throws a veto-driven error message if you try to ok on an invalid command (a blank services choice or action string, that sort of thing)
ctrl+r is now 'remove_files_from_view' by default. the old 'show_hide_splitters' is now ctrl+shift+r by default
the preview canvas, if focused, now responds to many normal media viewer shortcuts (content stuff like archive/manage tags, and canvas-specific like frame back/forth, zooming and panning)
the new shortcut system now interprets double-clicks of any mouse button to be a second single click
the new shortcut system correctly 'flips' ratings on and off, rather than always 'setting' to the chosen value. optional 'set only' support will come in the near future
more shortcuts will correctly and reliably propagate to canvases when any part of a hover frame has focus
fixed a conflict between the new shortcut system and taglists, which were no longer accepting otherwise interesting keys, like enter
created a shortcuts manager cache that deals with a bunch of the shortcut workflow centrally
completely eliminated the old accelerator table/menu command system for the main gui window
lots of misc shortcut-related work
fixed wildcard file search predicates that are on a specific file service and have no namespace (like 'mar*')
certain routines that can cause mass refreshing of the menubar (like import folders) will now not spam (and often queue this spam up and hang the gui) the menu so much. the menubar will now always collapse multple overlapping refresh calls to reduce cpu load
fixed a focus-None issue in the new hover window focus detection code
improved some more focus detection and comparison logic--quick rating-scrolling in the media viewer should be less janked
improved 'touch' drag event detection and improved media canvas cursor hide/show logic in general
fixed '&' display in notebook page names
fixed '&' display in some common dialogs' text
wrote a new statictext class that deals with '&' better and in future will autowrap and maybe some other stuff and then switched most of the simple instances of wx.statictext over
updated some out-of-date server help r.e. admin service initialisation
converted a little of the help on custom filter->custom shortcuts. I'll do more in future
started shortcut overhaul by updating shortcut storage and underlying objects
shortcuts will now support multiple modifiers (ctrl, alt, shift) in some places
the regular shortcut entry control now supports mouse events through a radiobox (although only the duplicate filter will capture and deal with mouse events on the new system!)
on db creation or update, a new 'duplicate_filter' shortcut set will be generated that includes mouse shortcuts and new duplicate_filter commands
the duplicate filter now obeys this shortcut set
duplicate filter shortcut edit cog menu now works, and the active shortcut set will update if edited
wrote simple duplicate pair status update db code
added dupe filtering optimisation--given A > B, anything else better than A will be set to be better than B, and anything worse than B will be set to be worse than A
added dupe filtering optimisation--after any duplicate status change affecting A and B, any 'same file' siblings of A and B will receive the same relationship
the 'process now' button on review services is now gated by a yes/no dialog that better explains what's about to happen
the repository buttons on review services will disable when they can't do anything
repository update processing will now cancel mid-job much faster
reintroduced the 'service-wide update' button to tag services in review services
reintroduced the 'clear trash' button to the trash file service in review services
reintroduced num_files rating service reporting in review services
fixed mouse scroll wheel events from the ratings hover window not being correctly processed by the main media canvas
added some 'touch' event detection to try to better deal with media dragging through a touchscreen
import folders will now 'action' their import paths as they go, rather than only at the end of their import run
the new taglists are now better at remembering their selection through a content change
opening a file or path from a non-windows client should now create a non-child process for open/xdg-open calls that block (so closing the client should not then close the child process movie player application or whatever)
added unit tests for the new shortcut object
added unit tests for the new application command object
improved 'file storage locations' help page descriptive text, including adding statements for actual current storage percentages
improved boot missing file storage error handling
wrote a 'repair file storage' dialog panel to manually fix missing folders during the boot phase
added some bells and whistles to the repair file storage dialog. it now shouldn't be possible to boot the client with an invalid file store--let me know how this works for you
added 'apply all siblings to all services' to options->tags
whenever combined-service siblings are consulted, local siblings will now always have precedence (so, you can now definitely overwrite any ptr siblings you don't like at the local level). this stuff remains a mess, however. more work is needed
fixed searching for 'like' ratings
rating predicates will now render their value as 'like'/'dislike' or '3/5' rather than the underlying float
fixed a variety of predicate string rendering errors when 'show namespace' is false
fixed delete key not firing on autocomplete text entry controls
the autocomplete entry's text box will now take home/end/up/down events when there is only one result in the dropdown, when navigation of that list is not important (previously, this was only true when there were no results)
fixed an issue where invalid typed 'system:' tags were not being correctly cleaned
improved tag cleaning code
fixed fetching autocomplete results for small and specific searches (e.g. putting in 'a' will return results just for 'a'
fixed small and specific searches to return namespaced instances as well (e.g. putting in '8' can now return 'chapter:8'
fixed an issue with showing collected media in the preview window
fixed the file import status button not laying out correctly on edit subscription dialog
cleaned up some focus checking code
harmonised all initial key event processing
cleaned some key event processing
adjusted some server daemon timings to prefer post-boot update generation catchup over db maintenance
corrected a bunch of slightly mistaken unit test code
added some 'like' and 'numerical' ratings search unit tests
the advanced content update dialog now uses the new sizing system (so the go! button should no longer wander out of frame!)
reduced what I believe is excess fuzzy padding on the new sizing system
optimised a common client db request, speeding up a number of tag operations
gui now asks for confirmation when you make to delete a gui session
No Comments