Blog | LordVan's Page / Bloghttps://blog.lordvan.com/blog/2024-03-28T05:19:05.261543+00:00If you were looking for something specific you probably got redirected here from an old link to my (now gone) drupal blog. I migrated all the pages & blog entries to this blog, so just use the search here to find what you were looking for.DBMail swap from LDAP to SQL auth2019-06-16T19:05:35.240583+00:002024-03-28T00:47:08.335738+00:00lordvanhttps://blog.lordvan.com/blog/author/lordvan/https://blog.lordvan.com/blog/dbmail-swap-from-ldap-to-sql-auth/<p>Due to my old ldap server (with samba 2.X) dying I wanted to decouple the emails from the rest of the accounts since only 3 users actually use the email anyway the whole setup with LDAP was complete overkill..</p>
<p>Fortunately swapping that was relatively simple! - more to come soon (when I get a chance to finish this blog entry)</p>Dbmail quickly get mailbox counts (including deleted emails) [postgresql]2018-03-01T07:31:32.529802+00:002024-03-28T05:19:05.261543+00:00lordvanhttps://blog.lordvan.com/blog/author/lordvan/https://blog.lordvan.com/blog/dbmail-quickly-get-mailbox-counts-including-deleted-emails-postgresql/<p>So while tracking down problems with performance with (webmail - ) clients I found out, that the problem is - at least partly - related to the sheer amount of IMAP folders (dbmail_mailboxes) a user had ( > 3500 !).</p>
<p>So we set out to merge those (old Archive) folders to cut down on the folder numbers (roundcube webmail for example seems to reload the folder list every login and also for *every* compose window, since the user can pick where to save the sent email).</p>
<p>The first thing was checking if I can do it automatically. That unfortunately got discarded quickly, because all the folders had been created manually and contained whitespaces, typos, differences in wording,..</p>
<p>So then the next step was to get a count of mails per folder (doing this through imap was slow and a pain. With a bit of help (FILTER) from someone in #postgresql I made this view:</p>
<pre>CREATE VIEW mailbox_mailcount AS<br/>SELECT mb.mailbox_idnr, usr.userid, mb.name, COUNT(*) FILTER (WHERE msg.deleted_flag = 1) AS MessagesDeleted, COUNT(*) FILTER (WHERE msg.deleted_flag = 0) AS Messages<br/>FROM dbmail_mailboxes AS mb<br/>INNER JOIN dbmail_users AS usr ON mb.owner_idnr = usr.user_idnr<br/>LEFT JOIN dbmail_messages AS msg ON mb.mailbox_idnr = msg.mailbox_idnr<br/>GROUP BY mb.mailbox_idnr, usr.user_idnr<br/>ORDER BY usr.userid, name;</pre>
<p>This makes it really easy to get the mailboxes for a user (not the internal userid, but the login) with a count of messages - and also marked as deleted messages. (handy for finding empty folders !!)</p>
<p>I also wrote some SQL to "move" (rename) some folders from one parent to another (it uses the view from above to (greatly) simplify the queries):</p>
<pre>BEGIN TRANSACTION;<br/><br/>CREATE TEMPORARY TABLE mb_rename<br/>(<br/>mailbox_idnr BIGINT UNIQUE not null,<br/>oldname VARCHAR(255) NOT NULL,<br/>newname VARCHAR(255) NOT NULL,<br/>mailcount BIGINT,<br/>ignorethis BOOLEAN DEFAULT FALSE<br/>);<br/><br/>INSERT INTO mb_rename<br/>SELECT mailbox_idnr, name, REPLACE(name, 'Archiv2012', 'Archiv') AS name_new, messages<br/>FROM mailbox_mailcount<br/>WHERE userid = '<insert your user>' AND name LIKE 'Archiv2012/%' <br/>AND Messages > 0;<br/>-- could use SIMILAR TO instead of LIKE to have regex matching<br/>-- ignore mailboxes, that are empty (or only contain deleted messages)<br/><br/>-- sometimes existing names were overlooked (they should be manually moved / merged)<br/>-- so we ignore ones already present<br/>UPDATE mb_rename SET ignorethis = TRUE<br/>FROM dbmail_mailboxes WHERE dbmail_mailboxes.name = mb_rename.newname;<br/><br/>-- all set so now just update the names<br/>UPDATE dbmail_mailboxes SET name = newname<br/>FROM mb_rename<br/>WHERE mb_rename.mailbox_idnr = dbmail_mailboxes.mailbox_idnr<br/>AND dbmail_mailboxes.name = oldname<br/>AND NOT mb_rename.ignorethis;<br/><br/>-- Check what was done here ! (also make a note of the ignored ones to sort manually)<br/><br/>-- don't forget to run this (commented in case i copy paste too fast):<br/>--COMMIT TRANSACTION;</pre>
<p>Now when I did run this i decided to 1) tell the users to close their email clients, but to be save i shut down dbmail-imapd for a minute (query ready,.. just stop, run query, check, start).</p>
<p>Also webmail clients like squirrelmail seem to cache the folder list, so logout and log back in was needed anyway.</p>
<p>Furthermore the <code>VIEW</code> created can of course easily be used to find all empty folders and clear them out with a single <code>DELETE</code> statement (be careful not to remove parent folders that happen to be empty while their child folders are not.. some clients get a bit confused :)</p>