Archive for the ‘Zimbra’ Category

Bulk delete etc. failing in Zimbra 8

Sunday, August 4th, 2013

Shortly after upgrading to Zimbra 8, I ran into annoying behavior when trying to delete large numbers of messages, both via IMAP and the web interface. If deleting the messages one-by-one (even in quick succession), then everything was fine, but if I selected several messages and deleted them all at once then it would fail with a “UID COPY failed” message in my IMAP client or a “An unknown error (zclient.UPLOAD_FAILED) has occurred” message in the Zimbra web interface.

Looking in the Zimbra mailboxd log file (/opt/zimbra/log/mailbox.log), I found that mailboxd was returning HTTP 503 Service Unavailable messages for these delete requests.

When using IMAP:

2013-08-04 10:13:09,188 WARN [ImapSSLServer-41] [;mid=5;ip=;oip=;via=;ua=Zimbra/8.0.4_GA_5737;] imap – UID COPY failed
com.zimbra.common.service.ServiceException: system failure: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
at com.zimbra.common.service.ServiceException.FAILURE(
at com.zimbra.client.ZMailbox.invoke(
at com.zimbra.client.ZMailbox.invoke(
at com.zimbra.client.ZMailbox.addMessage(
at com.zimbra.cs.service.mail.ItemActionHelper.executeRemote(
at com.zimbra.cs.service.mail.ItemActionHelper.schedule(
at com.zimbra.cs.service.mail.ItemActionHelper.COPY(
at com.zimbra.cs.imap.ImapHandler.doCOPY(
at com.zimbra.cs.imap.ImapHandler.executeRequest(
at com.zimbra.cs.imap.NioImapHandler.processRequest(
at com.zimbra.cs.imap.NioImapHandler.messageReceived(
at com.zimbra.cs.server.NioHandlerDispatcher.messageReceived(
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(
at com.zimbra.cs.server.NioLoggingFilter.messageReceived(
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$
Caused by: com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 503 Service Unavailable
Code:service.PROXY_ERROR Arg:(url, STR, “”)
at com.zimbra.common.service.ServiceException.PROXY_ERROR(
at com.zimbra.common.soap.SoapHttpTransport.invoke(
at com.zimbra.common.soap.SoapHttpTransport.invoke(
at com.zimbra.common.soap.SoapTransport.invoke(
at com.zimbra.client.ZMailbox.invoke(
… 28 more

When using the web interface:

2013-08-04 10:12:15,141 INFO [qtp820657724-13204:] [;mid=5;ip=;ua=ZimbraWebClient – SAF5.1 (Mac)/8.0.4_GA_5737;] SoapEngine – handler exception
com.zimbra.common.zclient.ZClientException: Attachment post failed, status=503
at com.zimbra.common.zclient.ZClientException.UPLOAD_FAILED(
at com.zimbra.client.ZMailbox.uploadContentAsStream(
at com.zimbra.client.ZMailbox.uploadContentAsStream(
at com.zimbra.client.ZMailbox.addMessage(
at com.zimbra.cs.service.mail.ItemActionHelper.executeRemote(
at com.zimbra.cs.service.mail.ItemActionHelper.schedule(
at com.zimbra.cs.service.mail.ItemActionHelper.COPY(
at com.zimbra.cs.service.mail.ItemAction.handleCommon(
at com.zimbra.cs.service.mail.ConvAction.handle(
at com.zimbra.soap.SoapEngine.dispatchRequest(
at com.zimbra.soap.DocumentHandler.proxyRequest(
at com.zimbra.soap.DocumentHandler.proxyRequest(
at com.zimbra.cs.service.mail.ItemAction.proxyRemoteItems(
at com.zimbra.cs.service.mail.ItemAction.handleCommon(
at com.zimbra.cs.service.mail.ConvAction.handle(
at com.zimbra.soap.SoapEngine.dispatchRequest(
at com.zimbra.soap.SoapEngine.dispatch(
at com.zimbra.soap.SoapEngine.dispatch(
at com.zimbra.soap.SoapServlet.doWork(
at com.zimbra.soap.SoapServlet.doPost(
at javax.servlet.http.HttpServlet.service(
at com.zimbra.cs.servlet.ZimbraServlet.service(
at javax.servlet.http.HttpServlet.service(
at org.eclipse.jetty.servlet.ServletHolder.handle(
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(
at org.eclipse.jetty.servlets.GzipFilter.doFilter(
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(
at org.eclipse.jetty.servlets.DoSFilter.doFilter(
at org.eclipse.jetty.servlets.DoSFilter.doFilter(
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(
at org.eclipse.jetty.servlet.ServletHandler.doHandle(
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
at org.eclipse.jetty.server.session.SessionHandler.doHandle(
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
at org.eclipse.jetty.servlet.ServletHandler.doScope(
at org.eclipse.jetty.server.session.SessionHandler.doScope(
at org.eclipse.jetty.server.handler.ContextHandler.doScope(
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
at org.eclipse.jetty.server.handler.HandlerCollection.handle(
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
at org.eclipse.jetty.server.handler.DebugHandler.handle(
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
at org.eclipse.jetty.server.Server.handle(
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
at org.eclipse.jetty.server.AbstractHttpConnection.content(
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(
at org.eclipse.jetty.http.HttpParser.parseNext(
at org.eclipse.jetty.http.HttpParser.parseAvailable(
at org.eclipse.jetty.server.AsyncHttpConnection.handle(
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
at org.eclipse.jetty.util.thread.QueuedThreadPool$

After a bit research, I found that Zimbra have implemented a “DoS filter” in the Zimbra 8, which is supposed to rate limit the number of requests from one IP address. You can see if you’re triggering this by locking for messages like this in /opt/zimbra/log/zmmailboxd.out:

2013-08-04 10:19:41.096:WARN:oejs.DoSFilter:DOS ALERT: ip=,session=null,user=null

In Zimbra 8.0.3 and later you can use zmprov to modify the settings for DoSFilter

  • zimbraHttpDosFilterDelayMillis can be used to configure whether you should reject any requests over the limit (set to -1) or impose a delay on the handling of the request (in miliseconds). By default it is set to -1 and so will reject any requests over the limit.
  • zimbraHttpDosFilterMaxRequestsPerSec sets the threshold (in requests per second) at which the DoSFilter feature is triggered. This is set to 30 by default, which seems rather low.
  • zimbraHttpThrottleSafeIPs allows you to whitelist IP addresses which you want to be exempt from the DoSFilter feature. The list needs to be comma seperated uf configuring multiple IP addresses. If you are running a multi-server setup, then the other servers are automatically whitelisted.

Because these settings affect mailboxd, you need to restart it in order for them to take effect.

zmprov mcf zimbraHttpDosFilterDelayMillis 500
zmprov mcf zimbraHttpDosFilterMaxRequestsPerSec 250
zmprov mcf zimbraHttpThrottleSafeIPs,
zmmailboxdctl restart

Disabling highlighting of objects in Zimbra web UI

Sunday, February 5th, 2012

Zimbra has a couple of Zimlets (plugins) that highlight parts of messages when viewed in the Zimbra Web Console such as dates, phone numbers and e-mails and turn them into special contextual links. Whilst this functionality is often quite useful, sometimes you just need to see the raw, unadulterated e-mail without your client interfering with the content – particularly if you are testing e-mail designs!

You can either disable Zimlets server wide through the admin console, or on a per-account basis using the preferences. In particular, look for the “Date”, “Email”, “Phone” and “URL Links” Zimlets as these are the four of the default Zimlets in Zimbra 7.x that are responsible for highlighting parts of messages and turning them into contextual links.

Zimbra mailbox import/export and migration of e-mail filter rules

Monday, June 27th, 2011

Zimbra has a fantastically useful built in system for exporting an entire mailbox, including the contents of the entire e-mail inbox, calendar, address book and briefcase ready to be imported on another Zimbra server either via the web interface or using zmmailbox from the command line. This makes migrating mailboxes between separate Zimbra installations incredibly easy.

You can export a chosen mailbox from the source Zimbra server with:

zmmailbox -z -m getRestURL “//?fmt=tgz” > /tmp/

And then import it into the destination server with:

zmmailbox -z -m postRestURL “//?fmt=tgz&resolve=reset” /tmp/

You need to make sure that the target account exists before attempting to import the archive on the destination server. Using the “reset” resolve method will ensure that everything is wiped from the target account before importing from the archive.

Simply replace “tgz” with “zip” in order to chose between the two archive formats when importing and exporting, making sure to use the right one on the import!

If you want to download a copy of an account from your browser, just visit the appropriate URL (e.g. where “user” is the account’s username) or use the nice Import/Export GUI in the Zimbra preferences tab, which also gives you the option to upload and import an archive.

The Zimbra preference interface to the export function also allows you to easily specify advanced settings such as date ranges, search filters or limiting the export to a certain data type such as calendar items.

The one problem with Zimbra’s import/export system is that user settings such as signatures and mail filters which are stored in an account’s LDAP attributes aren’t included in the exported data. It’s easy enough to manually move signatures between servers, but anything more than a couple of mail filters can be tedious to manually re-create.

Luckily, you can get the information you need from the zimbraMailSieveScript attribute for a chosen account using the zmprov command line utility:

zmprov ga zimbraMailSieveScript

This should give you something a copy of your mail filter rules in the sieve format, for example:

require [“fileinto”, “reject”, “tag”, “flag”];

# No Reply
if anyof (header :contains [“to”] “”) {
fileinto “Inbox/No Reply”;

You can then easily re-import this into LDAP on the destination server by placing single quotes around the result and using “zmprov ma”:

zmprov ma zimbraMailSieveScript ‘require [“fileinto”, “reject”, “tag”, “flag”];

# No Reply
if anyof (header :contains [“to”] “”) {
fileinto “Inbox/No Reply”;

You can of course apply the same technique to other account details if you wish, you just need to know the appropriate LDAP attribute, such as zimbraPrefMailSignatureHTML for your signature or zimbraPrefOutOfOfficeReply for your out of office auto reply.

Preventing backscatter on aliased domains in Zimbra

Tuesday, April 5th, 2011

By default, an aliased domain in Zimbra will accept all e-mail at SMTP time and then bounce a message later if it is unable to delivering it after carrying out the aliasing. This generates backscatter, which can be abused and even lead to your mail server appearing on some blacklists. Luckily, since Zimbra 5.0.12 there has been a way to fix this; just su to the zimbra user and run:

zmlocalconfig -e postfix_enable_smtpd_policyd=yes
zmprov mcf +zimbraMtaRestriction “check_policy_service unix:private/policy”
postfix stop
postfix start

Zimbra anti-spam improvements

Sunday, April 11th, 2010

The built in Zimbra anti-spam system is quite a neat bundle of Amavisd-new, SpamAssassin and ClamAV with some fancy automated ham/spam training based on messages being moved in and out of a “Junk” mailbox under each user’s account, but it lacks a few nice to have extra features. Luckily, it’s quite easy to enhance the Zimbra Amavisd and SpamAssassin with a new plugins such as DCC, Pyzor and Razor as well as enabling SPF record checking and turning on DSPAM.

Zimbra includes DSPAM as well, but doesn’t use it by default. You can change this quite simply by updating the Zimbra LDAP configuration with the following:

zmlocalconfig -e amavis_dspam_enabled=true

I’d recommend upgrading to 6.0.5 if you are going to use DSPAM as there are annoying bugs in earlier versions such as needing to chown the DSPAM folder as zmfixperms used to set the permissions incorrectly. There is also an updated version of DSPAM in Zimbra 6.0.5.
The beauty of DSPAM with Zimbra is that the zmtrainsa utility run nightly on the spam/ham mailboxes also trains DSPAM from the same messages.

Now I’m presuming that you don’t already have the RPMforge (formerly Dag Wieers) and Atomic Rocket Turtle yum repositories installed on your Zimbra server and that you’re using CentOS/Red Hat like I am. We will install these two repositories but restrict them to only provide the packages that we are interested in so that they don’t clash with each other or the base vendor repositories.

wget -q -O – | sh
rpm -Uvh rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm

Now you need to edit /etc/yum.repos.d/rpmforge.repo to add the line includepkgs=perl-Error perl-NetAddr-IP perl-version perl-Mail-SPF as well as /etc/yum.repos.d/atomic.repo to have includepkgs=dcc pyzor razor-agents under the [atomic] section
Now the packages we need are available through a normal yum install:

yum install dcc pyzor razor-agents perl-Mail-SPF

Now we just need to create a custom SpamAssassin configuration file to tweak the settings for the plugins that we just installed. To do this, go to /opt/zimbra/conf/spamassassin/ and create a new .cf file with the following:

loadplugin Mail::SpamAssassin::Plugin::DCC

score SPF_FAIL 10.000
score SPF_HELO_FAIL 10.000
score DCC_CHECK 4.000
score RAZOR2_CHECK 2.500
score PYZOR_CHECK 2.500

The Zimbra SpamAssassin configurations already load the Pyzor and Razor plugins if present, but don’t load DCC by default (even if it is present) as it isn’t open source. Rather than edit files that Zimbra will then reset on an upgrade, we create a new .cf file that does this as well as settings the scores given by DCC, Pyzor, Razor and SPF. You might want to tweak these depending on how much you trust each service/test or you might want to skip these lines altogether and leave the scores set as the SpamAssassin defaults.
Remember to chown the file to zimbra:zimbra and chmod it to 0444 to be in line with the other SpamAssassin .cf configuration files.

The last thing that you need to do is restart the Zimbra MTA and Amavisd-new so that it loads the new configuration.

su – zimbra
zmantispamctl reload

If you want to test your new SpamAssassin setup then run the following (test and Debug mode) on the GTUBE sample provided by SpamAssassin

/opt/zimbra/zimbramon/bin/spamassassin -D -t < gtube.txt

Like the EICAR signiture for anti-virus scanners, GTUBE is a signature for anti-spam systems that will always show as spam so you can easily test your anti-spam setup. Among others, you should see RAZOR2_CHECK, PYZOR_CHECK and DCC_CHECK flagged with their appropriate scores if everything is working properly.
You will need to test DSPAM in the same way as you would with SpamAssassin’s bayesian filtering as well as checking SPF failures manually by sending a message from a server not designated in the SPF records.

Detecting brute force attacks on Zimbra with zmauditswatch

Wednesday, April 7th, 2010

Zimbra 5.0.11 introduced the zmauditswatch script which notifies a specific e-mail address of a potential brute force attack if certain conditions are met. This is disabled by default and the documentation to enable it isn’t particularly clear, so here is a quick run through:

zmlocalconfig -e
zmlocalconfig -e zimbra_swatch_ipacct_threshold=10
zmlocalconfig -e zimbra_swatch_acct_threshold=15
zmlocalconfig -e zimbra_swatch_ip_threshold=20
zmlocalconfig -e zimbra_swatch_total_threshold=60
zmlocalconfig -e zimbra_swatch_threshold_seconds=60
touch /opt/zimbra/conf/auditswatchrc
touch /opt/zimbra/conf/
zmauditswatchctl start

Obviously use a relevant IP address and tweak the various thresholds appropriately to better suit your environment.

DomU kernel panic under Xen with R1Soft CDP agent

Sunday, March 28th, 2010

I ran into a weird problem on a freshly installed server today where one of the DomU virtual machines kept locking up with a kernel panic message:

BUG: soft lockup – CPU#1 stuck for 10s!

All of the other Xen DomU virtual machines running on the box were fine, so I was pretty sure it wasn’t a hardware bug. The only possible solutions that I could come up with were

  1. My somewhat aggressive stripping down of the Dom0 host
  2. Zimbra (as the Java process was always the one referenced as being stuck
  3. The kernel module for the R1Soft CDP backup agent that I had installed a few hours ago.

I’ve run Zimbra on a CentOS 5 Xen VM before (although not 5.4 admittedly), so didn’t think that would be the problem (anyway, user land software should never be able to cause a kernel panic – in theory at least ;-))
Again, the other Xen DomU virtual machines were fine, so hopefully the configuration on the Xen Dom0 host machine itself shouldn’t be causing the problem.

Just to be safe, I updated the kernel-xen package from the default 2.6.18-164.el5 to the latest 2.6.18-164.15.1 which was only released a few days ago and doesn’t appear to be on all of the mirrors yet!
With this done and the virtual machine restarted I had to run r1soft-cki to load an updated kernel module for the R1Soft CDP backup agent and the virtual machine has been stable for several hours now. Fingers crossed it stays that way!

Binding Zimbra to an IP

Monday, October 19th, 2009

Zimbra is a fantastic collaboration suite, but it suffers from one major flaw; it really likes to take over your entire server!
If you want to run Zimbra and your web site on the same server without moving Zimbra’s web UI to another port then you’ll have to convince it to bind to one IP address you can use the other one(s) as you please.
This is possible and actually quite easy but completely undocumented.

Before we begin, you will need a working Zimbra install. I’m not aware of a way to pre-configure this process, so you may have to stop your web server etc. whilst running as the installer will likely bail with port conflict errors.

First off, SSH in to your server and find out what Zimbra server(s) you have defined:

su – zimbra
zmprov gas

The su is important as it sets the various environment variables Zimbra needs to point at it’s home in /opt
Now take the server from this and modify the following commands with <server> and <ip> substituted appropriately.

zmprov ms <server> zimbraImapBindAddress <ip>
zmprov ms <server> zimbraImapSSLBindAddress <ip>
zmprov ms <server> zimbraPop3SSLBindAddress <ip>
zmprov ms <server> zimbraPop3BindAddress <ip>

This will bind the POP3 and IMAP Java process appropriately but we need to edit the Jetty config to do the same for the web and admin interfaces.
Throughout this guide I’m assuming that you are running a single server setup with everything on one machine, but it should be quite easy to tweak the IP addresses appropriately in a multi-server environment.

Zimbra uses the Jetty Java application server, so we need to tell Jetty to only bind to one specific IP. Open up /opt/zimbra/mailboxd/etc/ in your favourite text editor and look for lines starting

You need to add a “<Set name=”Host”><IP></Set> line to the <Arg> list for each of the HTTP and HTTPS connectors as well as the Admin connector. If you want, you can also edit the “Extension Port” connector which is only used for mail routing in multi-server environments.
For example:

<Call name=”addConnector”>
<New id=”ssl” class=””>
<Set name=”Port”>%%zimbraMailSSLPort%%</Set>
<Set name=”Host”></Set>

Now open up /opt/zimbra/mailboxd/etc/ and you should see a “param” named “zimbra.soap.url” with the value set to “http://localhost:7070/service/soap”. you need to change this to “http://<IP>/service/soap”. Notice that the :7070 port declaration is removed.
For example:


Now you need to do the same for /opt/zimbra/mailboxd/etc/

Finally, if you want to bind the Zimbra Postfix SMTP service to a particular IP then edit the first few lines of /opt/zimbra/postfix/conf/ that start with smtp, 465 and submission to have : prepended. For example: inet n – n – – smtpd inet n – n – – smtpd
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes inet n – n – – smtpd

Once all this is done, restart Zimbra using zmcontrol and use netstat to check that everything is bound to the right IP.
The only thing to remember with this is that you will have to edit each of these files every time you do an upgrade.