Bug 918705 - alpine locking failure (against procmail) leads to mail corruption
alpine locking failure (against procmail) leads to mail corruption
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Other
201502*
x86-64 Other
: P5 - None : Critical (vote)
: ---
Assigned To: Reinhard Max
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-02-19 18:12 UTC by Gerald Pfeifer
Modified: 2022-03-11 14:04 UTC (History)
5 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gerald Pfeifer 2015-02-19 18:12:38 UTC
This is something I started to see a bit ago, and since it now happened
twice today I figured I'd report it.

When I obtain mail via fetchmail, using procmail as the delivery agent,
occasionally I get corrupted mails like the following (in mbox format):

==== snip ====
From gp  Thu Feb 19 18:56:54 2015
Mime-Version: 1.0
Date: Thu, 19 Feb 2015 18:56:00 +0100
Subject: AW: Re: Your help needed: Fwd: Comment opportunity: Software
Status: RO
X-Status: 
X-Keywords:                 
X-UID: 9310

 definedstorage
Subject: AW: Re: Your help needed: Fwd: Comment opportunity: Software

 definedstorage
References: <54E302D90200006F00158C8B@suse.com>
         <54E2D72A0200002B000A1B55@suse.com> <54E339FA0200006F00158DBD@suse.com>
         <54E2D7DE0200002B000A1B68@suse.com>
         <alpine.LSU.2.11.1502191739290.2252@ghan.fvgr>
In-Reply-To: <alpine.LSU.2.11.1502191739290.2252@ghan.fvgr>
Message-ID: <54E623B70200006F001594A6@suse.com>
From: ...
To: "Gerald Pfeifer" <gp@suse.com>
==== snap ====

This does not happen daily, and I believe it happens especially when
I have been offline for several hours and a few dozen e-mails are 
delivered at once (though this specific case was not that).

The same configuration had been working for years without ever showing
this problem, and it looks like looking on the Alpine side (or procmail,
though less likely I guess, since it essentially only reads
  :0
  INBOX
)
Comment 1 Reinhard Max 2015-02-21 20:59:14 UTC
Do you have alpine-branding-openSUSE installed?
Comment 3 Reinhard Max 2015-02-21 21:14:49 UTC
BTW, I think a second colon in your procmail rule to ensure proper locking?

:0:
INBOX

BTW2, have you configured alpine to consider INBOX as an incoming folder?
AFAIK alpine doesn't do locking on folders that are not marked like that.

I use alpine together with procmail and a very versatile .procmailrc and I haven't seen problems like this even with NFS between the machine that delivers mail and the one that runs alpine. But I have to admit that I converted at least the busier ones of my incoming folders to maildir long time ago to avoid locking problems.
Comment 4 Gerald Pfeifer 2015-02-21 21:33:53 UTC
(In reply to Reinhard Max from comment #1)
> Do you have alpine-branding-openSUSE installed?

Yes. 

% rpm -qa | grep alpine
alpine-branding-openSUSE-0-5.1.noarch
alpine-debuginfo-2.11-7.2.x86_64
alpine-2.11-7.2.x86_64
Comment 5 Gerald Pfeifer 2015-02-21 22:27:17 UTC
(In reply to Reinhard Max from comment #3)
> BTW, I think a second colon in your procmail rule to ensure proper locking?
> 
> :0:
> INBOX

This is a good catch.  I thought I had that, but amidst some testing
(and due to procmailex not having it in many examples when I checked)
apparently lost it.  So, this is something I add now.

In addition I removed alpine-branding-openSUSE.

> BTW2, have you configured alpine to consider INBOX as an incoming folder?
> AFAIK alpine doesn't do locking on folders that are not marked like that.

I have all my folders listed as Incoming Folders, except for the inbox
due to the following in the Alpine help:

   ==== snip ====
                         OPTION: Incoming Folders

   This is a list of one or more folders other than INBOX that may receive 
   new messages. 
   ==== snap ====

And my inbox is set via "Inbox Path", thus should be covered.  If you
believe it may make a difference, I can add it, though.

> I use alpine together with procmail and a very versatile .procmailrc and I
> haven't seen problems like this even with NFS between the machine that
> delivers mail and the one that runs alpine. But I have to admit that I
> converted at least the busier ones of my incoming folders to maildir long
> time ago to avoid locking problems.

Yes, I have used similar combinations (incl. NFS) for close to 20 years
and never lost a single mail.  And my current setup (on my notebook, no
NFS) has been in place for several years.  That's why I was so surprised
to see these corrupted mails after not having changed anything.

Perhaps it was the missing colon and I got lucky for years (unless I 
did, as I am now not sure about, incidently remove it while trying to
debug this issue) and some other aspects on the system changed to
unhide this or the -branding settings or it's something else.

Let me try to reproduce the issue after these changes...
Comment 6 Reinhard Max 2015-04-02 14:26:44 UTC
Any news?
Comment 7 Gerald Pfeifer 2015-04-08 11:26:48 UTC
I had a single corruption since reporting this and adjusting my
procmail rules, so this is not rampant, but apparently still there
appears to be a bit of an issue.

A few days ago I updated to Alpine 2.20, let's see whether I run
into this with that version at all?
Comment 8 Gerald Pfeifer 2015-05-11 15:57:40 UTC
Unfortunately this is still happening.  I had at least three 
occurrences the last week, usually after I had been offline for
several (peak) hours and then received a lot of mails -- which 
is a strong indication that we are looking at locking issues.

For reference, here is the current procmail rule I am using:

  :0:
  INBOX

And here is one of those mails as it landed in my inbox:

  From gp  Mon May 11 11:47:44 2015
  Mime-Version: 1.0
  X-Mailer: GroupWise 2014 SP1
  Date: Mon, 11 May 2015 16:59:08 +0200
  Subject: RE: Panel at Summit (was: SUSE Overview session for MF sellers at
  Status: RO
  X-Status: 
  X-Keywords:                 
  X-UID: 9497

   Summit)
  Subject: RE: Panel at Summit (was: SUSE Overview session for MF sellers at

   Summit)
  References: <55411433020000380015F956@suse.com>
           <alpine.LSU.2.20.1504300133211.1897@ghan.fvgr>
           <55411B23020000380015F961@suse.com>
           <alpine.LSU.2.20.1505110152180.2744@ghan.fvgr>
  In-Reply-To: <alpine.LSU.2.20.1505110152180.2744@ghan.fvgr>
  Message-ID: <55506EC90200003800161308@suse.com>
  From: ...
  To: "Gerald Pfeifer" <gp@suse.com>

Observe how the multi-line Subject: header is duplicated, and the
four headers (Status, X-Status, X-Keywords, X-UID) that Alpine adds
are in between.
Comment 9 Reinhard Max 2015-05-13 12:59:21 UTC
Please turn on logging/debugging in procmail and alpine.

For procmail, add these two lines to your .procmailrc:

LOGFILE=$HOME/procmail.log
LOGABSTRACT=all

For alpine there is no option in the config file to enable debugging, it has to be started with the "-d debuglevel" switch, but I don't know which level (0..9) would be appropriate.

BTW, how and in what order do fetchmail, procmail and alpine get started when you return after some hours offline?
Comment 10 Eduardo Chappa 2015-05-23 17:09:53 UTC
(In reply to Reinhard Max from comment #3)
> BTW, I think a second colon in your procmail rule to ensure proper locking?
> 
> :0:
> INBOX
> 
> BTW2, have you configured alpine to consider INBOX as an incoming folder?
> AFAIK alpine doesn't do locking on folders that are not marked like that.
> 
> I use alpine together with procmail and a very versatile .procmailrc and I
> haven't seen problems like this even with NFS between the machine that
> delivers mail and the one that runs alpine. But I have to admit that I
> converted at least the busier ones of my incoming folders to maildir long
> time ago to avoid locking problems.

Max,

  Alpine has nothing to do with this issue, not in the way you are thinking of.
First, locking is an *access* issue, not a *configuration* issue. No matter how you configure Alpine locking will happen when appropriate.

  The issues of locking come from the internal c-client library that Alpine uses. What this means is that Alpine is an interface to that library, and so the same issues happen with other c-client applications (imapd, pop3d, etc.)

  Ok, having said that. Locking issues happen when two programs try to access the same folder and do not communicate with each other. So in this case, procmail and c-client do not work in compatible ways.

  Since I do not know what is the exact setup here, I would suggest to take a look at the following FAQ, which I transcribe here:

<quote>
http://www.washington.edu/imap/IMAP-FAQs/index.html#7.10

In order to update a mailbox in the default UNIX format, it is necessary to create a lock file to prevent the mailer from delivering mail while an update is in progress. Some systems use a directory protection of 775, requiring that all mail handling programs be setgid mail; or of 755, requiring that all mail handling programs be setuid root.

The IMAP toolkit does not run with any special privileges, and I plan to keep it that way. It is antithetical to the concept of a toolkit if users can't write their own programs to use it. Also, I've had enough bad experiences with security bugs while running privileged; the IMAP and POP servers have to be root when not logged in, in order to be able to log themselves in. I don't want to go any deeper down that slippery slope.

Directory protection 1777 is secure enough on most well-managed systems. If you can't trust your users with a 1777 mail spool (petty harassment is about the limit of the abuse exposure), then you have much worse problems then that.

If you absolutely insist upon requiring privileges to create a lock file, external file locking can be done via a setgid mail program named /etc/mlock (this is defined by LOCKPGM in the c-client Makefile). If the toolkit is unable to create a <...mailbox...>.lock file in the directory by itself, it will try to call mlock to do it. I do not recommend doing this for performance reasons.

A sample mlock program is included as part of imap-2006. We have tried to make this sample program secure, but it has not been thoroughly audited.
</quote>

The protection 1777 should exist in the the directory that contains the inbox (/var/mail or /var/spool/mail typically) and in /tmp (the latter is that c-client applications interact with each other well)

Does this help?

-- 
Eduardo
http://patches.freeiz.com/alpine/
Comment 11 Gerald Pfeifer 2015-12-12 02:32:53 UTC
Thank you for the detailed background, Eduardo and both you and Reinhard
looking into this.

I can report that I have not seen this issue happen again in the last
couple of months and assume this is due to one of two things:

 a) changes on the server side (Groupwise)
 b) changes on the openSUSE Factory side (alpine,...)

Let's assume whatever I was encountering got fixed somehow.

Should this should up again, in a different context possibly, here is
some more background I found: It appears this only happened around the
Subject header, and only when the Subject was long and spread over two
lines, that is "Subject: bla bla bal\n    and more of it\n".

Then it appears that some part of some software (and that is where it's
not clear where and how) did not cope with that second/continuation line
properly and the Status: and other headers that Alpine inserts where then
put there / inserted without shifting everything else properly).
Comment 12 Richard Biener 2017-10-24 07:28:16 UTC
I'm now experiencing this regularly multiple times a day.  This is with Leap 42.2 which has alpine-2.20-11.32.x86_64 and procmail-3.22-268.23.x86_64.

Somehow this is a more recent issue (since the last 10 days?) but neither
alpine nor procmail changed inbetween.

Note that the mboxen I use reside on NFS storage and I am running
4.4.87-18.29-default (so I guess it happens since the power outage
which triggered a reboot...).  Which means it somehow didn't happen
with 4.4.74-18.20 which should have been the kernel running before.

Documentation tells me that procmail uses fcntl style locking (in addition
to lockfiles?).

Yes, all affected mboxen are configured as incoming folders for alpine.

Usually apline closes the folders with an access error but I have now twice
seen corrupted display of mails from within alpine (second mail displayed
within first, spaced by what seems to be a block of zero bytes).
Comment 13 Reinhard Max 2022-03-11 13:49:21 UTC
@Gerad @Richard As this bug has been rotting for quite a while and alpine has seen several upgrades since, I'd like to hear if these corruptions are still happening to you?
Comment 14 Richard Biener 2022-03-11 13:51:49 UTC
(In reply to Reinhard Max from comment #13)
> @Gerad @Richard As this bug has been rotting for quite a while and alpine
> has seen several upgrades since, I'd like to hear if these corruptions are
> still happening to you?

I'm no longer using procmail but moved to imap & sieve, sorry.
Comment 15 Gerald Pfeifer 2022-03-11 14:02:41 UTC
I do not recall running into this issue (and I have the originally
reproducing situations still taking place such as a swath of e-mails
being locally delivered at once) for years (plural).

So from my perspective, whatever it was, it's gone.
Comment 16 Reinhard Max 2022-03-11 14:04:46 UTC
Thanks for the quick response, closing as FIXED, then.