Latin <-> utf problem in rt-mailgate

Palle_Girgensohn · December 8, 2003, 4:23pm

Hi!

I just got this problem with an email sent to RT, one text part in the
ISO-8859-1 charset, and two jpeg images.

Problem is, both jpegs are corrupt and the text in the message is presented
converted to utf-8 twice (i.e. four bytes, an ö is tranferred as “ÃÂ¥”).
See the attached image for info. Emails look the same, and only the text
that was delivered to RT as an attachment (well, the text part on en email
that had two images attached, really, but that is technically an attachment
too…).

In the database, the text is stored in the described (false) way, so this
must be en error in mail gateway.

Here is the tuple in the database. Perhaps the contenttype is bad?

Se bifogade bilder.
| content-type: text/plain; charset=“utf-8”; format=“flowed”
Content-Transfer-Encoding: 8bit
X-RT-Original-Encoding: ISO-8859-1
Content-Length: 485
| 51 | 2003-12-05 16:45:51
(1 row)

If you need it, I can send you the original email as well.

Setup: Perl 5.8.1, RT 3.0.7, postgresql-7.4, freebsd 4.9.

/Palle

Jesse_Vincent · December 9, 2003, 5:17am

Hi!

I just got this problem with an email sent to RT, one text part in the
ISO-8859-1 charset, and two jpeg images.

Problem is, both jpegs are corrupt and the text in the message is presented
converted to utf-8 twice (i.e. four bytes, an ö is tranferred as
“ÃÂ¥”). See the attached image for info. Emails look the same, and
only the text that was delivered to RT as an attachment (well, the text
part on en email that had two images attached, really, but that is
technically an attachment too…).

Palle,

I’d love to hear if you can still reproduce this issue with the latest
release of RT 3.0.8pre_

Best,
Jesse

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Jari_Lehtonen · December 9, 2003, 11:07am

I just got this problem with an email sent to RT, one text part in the
ISO-8859-1 charset, and two jpeg images.

Problem is, both jpegs are corrupt and the text in the message is
presented converted to utf-8 twice (i.e. four bytes, an ö is
tranferred as “Ã?Â¥”). See the attached image for info. Emails look the
same, and only the text that was delivered to RT as an attachment
(well, the text part on en email that had two images attached, really,
but that is technically an attachment too…).

Palle,

I’d love to hear if you can still reproduce this issue with the latest
release of RT 3.0.8pre_

I’m not Palle, but I can reproduce this issue with the latest 3.0.8pre2
from your site. I’ll send you an email that reproduces the case.

    Jari Lehtonen
    Unix & Network services
    Computing Center
    University of Turku, Finland

Palle_Girgensohn · December 9, 2003, 12:57pm

Hi,

Just as Jari, I’m sending you a mail the produces this problem, as a
separate email (Subject: Aktivitetsloggan placeras olika på Meny och
Lektion). I didn’t try 3.0.8pre2 just yet, since Jari still experinced
problems. I’ll await a response before trying a 3.0.8pre_.

Regards,
Palle–On tisdag, december 09, 2003 13.07.41 +0200 Jari Lehtonen jari@utu.fi wrote:

On tiistai 9. joulukuu 2003 00:17 -0500 Jesse Vincent jesse@bestpractical.com wrote:

I just got this problem with an email sent to RT, one text part in the
ISO-8859-1 charset, and two jpeg images.

Problem is, both jpegs are corrupt and the text in the message is
presented converted to utf-8 twice (i.e. four bytes, an ö is
tranferred as “Ã?Â¥”). See the attached image for info. Emails look the
same, and only the text that was delivered to RT as an attachment
(well, the text part on en email that had two images attached, really,
but that is technically an attachment too…).

Palle,

I’d love to hear if you can still reproduce this issue with the latest
release of RT 3.0.8pre_

I’m not Palle, but I can reproduce this issue with the latest 3.0.8pre2
from your site. I’ll send you an email that reproduces the case.

–
Jari Lehtonen
Unix & Network services
Computing Center
University of Turku, Finland

Jari_Lehtonen · December 9, 2003, 9:39pm

I’m not Palle, but I can reproduce this issue with the latest 3.0.8pre2
from your site. I’ll send you an email that reproduces the case.

I’ll have to take back this statement. I can’t reproduce the problem
reliably with 3.0.8pre2.

However, I had, during the tests earlier today, two cases of corrupted
characters with this version. First was an email sent as a new ticket to RT
from iPlanet Messenger Express (their webmail), and the other was a reply
from Pine to a case opened earlier. Later tests from iPlanet Messenger
Express, even a mail exactly like the earlier, corrupted one, went through
without problems.

So far, so good. Thanks for the effort to fix this, Jesse, it means a lot
to us Northern Europeans! Could you describe what exactly was the problem,
as I don’t see much that (IMHO) relates to this in the ChangeLog?

    Jari Lehtonen
    Unix & Network Services
    University of Turku, Computing Center

Jesse_Vincent · December 11, 2003, 4:27am

I’m not Palle, but I can reproduce this issue with the latest 3.0.8pre2
from your site. I’ll send you an email that reproduces the case.

I’ll have to take back this statement. I can’t reproduce the problem
reliably with 3.0.8pre2.

However, I had, during the tests earlier today, two cases of corrupted
characters with this version. First was an email sent as a new ticket to RT
from iPlanet Messenger Express (their webmail), and the other was a reply
from Pine to a case opened earlier. Later tests from iPlanet Messenger
Express, even a mail exactly like the earlier, corrupted one, went through
without problems.

If you can come up with anything plausible about how to make it happen
“somewhat consistently” with 3.0.8, I’d love to hear about it.

So far, so good. Thanks for the effort to fix this, Jesse, it means a lot
to us Northern Europeans! Could you describe what exactly was the problem,
as I don’t see much that (IMHO) relates to this in the ChangeLog?

IIRC, there were a couple places where we were double-encoding.

–
    Jari Lehtonen
    Unix & Network Services
    University of Turku, Computing Center
rt-users mailing list
rt-users@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Jari_Lehtonen · December 11, 2003, 12:08pm

If you can come up with anything plausible about how to make it happen
“somewhat consistently” with 3.0.8, I’d love to hear about it.

Still with 3.0.8pre2, I can now reproduce the problem.

An email sent from iPlanet Messenger Express, bounced to Requst Tracker
from Pine 4.58 (on Solaris 8), breaks the latin1 characters every time.
Bounced from Mulberry 3.1 on Mac OS X, the email is fine.

Relevant headers from the original email:

MIME-version: 1.0
X-Mailer: iPlanet Messenger Express 5.2 HotFix 1.13 (built Mar 7 2003)
Content-type: text/plain; charset=iso-8859-1
Content-language: fi
Content-transfer-encoding: quoted-printable
Content-disposition: inline
X-Accept-Language: fi

and from the bounced message:

MIME-version: 1.0
X-Mailer: iPlanet Messenger Express 5.2 HotFix 1.13 (built Mar 7 2003)
Content-type: text/plain; charset=iso-8859-1
Content-language: fi
Content-disposition: inline
X-Accept-Language: fi
…
ReSent-Message-ID: Pine.GSO.4.58.0312111347200.25311@alya
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by oban.utu.fi id
hBBBliLf050653

I’m going to bounce the same message to you, Jesse, first from Pine 4.58,
then from Mulberry 3.1. Mail subject will be “mooze-dsl” in both emails.

Environment (RT server):

OS: FreeBSD oban.utu.fi 5.1-RELEASE FreeBSD 5.1-RELEASE #0: Thu Jun 5
02:55:42 GMT 2003
root@wv1u.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC i386
Sendmail: 220 oban.utu.fi ESMTP Sendmail 8.12.9/8.12.9; Thu, 11 Dec 2003
13:59:26 +0200 (EET)
procmail: procmail v3.22 2001/09/10
perl: perl5 (revision 5.0 version 8 subversion 1) compiled on this box
from source taken straight from www.perl.org.
apache: Apache/1.3.29 (Unix) mod_perl/1.29 mod_ssl/2.8.16 OpenSSL/0.9.7a

postgresql: postgresql-7.3.4_1 compiled from the ports collection

  Jari Lehtonen
  Unix & Network services
  Computing Center
  University of Turku, Finland

Jesse_Vincent · December 17, 2003, 6:29pm

If you can come up with anything plausible about how to make it happen
“somewhat consistently” with 3.0.8, I’d love to hear about it.

Still with 3.0.8pre2, I can now reproduce the problem.

With the current version, which will soon be a release candidate, I
can’t reproduce the issue using the two messages you sent. Oh. because
they both got autoconverted to quoted printable on outoging. But from
your mail, it looks like the quoted-printable one (not the 8bit one) is
the one that had the issue. Hm. (And yeah, I saw your message about a
“trigger”. That’s not outside the realm of possibility)

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Aurel_Schwarzentrube · December 23, 2003, 10:32am

If you can come up with anything plausible about how to make it happen
“somewhat consistently” with 3.0.8, I’d love to hear about it.

We have the same problems as Jari, using 3.0.8rc1. I can send you a mail which
produces the error reliably, if you want.

Aurel