PHP-IMAP issue... i think.

Forums › IT Pro and developers › PHP-IMAP issue... i think.

chevrolux

4962 posts

Uber Geek
Inactive user

#216590 4-Jul-2017 13:42

Alright, so we run Vtiger CRM internally - the open source version. It's awesome for being able to muck around with, without the need to do any actual coding.

However, I have one feature I would really like to explore a bit more and I think it is the code that is either buggy or I am doing the wrong thing with it.

So we have a "Billing" module, it should receive an email (which we send in HTML format), read the body and extract the body, then convert this to a PDF.

I really don't know what I am looking at, but what I did was just go through the script (i know which one it is because there is a cron set to run) and add a bunch of "echo" statements to try and find where it stops working.

Script: https://pastebin.com/BBSSPTYL

From what I can see the whole script executes correctly (and I get the resulting email with PDF attached as I should), however the issue is the PDF is always blank.

So looking back through the echo output there is the lines:

if (strpos($mail_subject, '#8369') !== false) {
echo 'mail subject has hash #8369';
 echo ' ';
 $mailbody=imap_fetchbody($inbox,$email_number,2);
 echo 'raw email body input... '.$mailbody;
 echo ' ';
 $msub=$mail_subject;
 $x=explode("#",$msub);
 $accno=$x[1];
 $mailbody=quoted_printable_decode($mailbody);

Am i right in thinking that is the bit where the email body is being read?

And if i do this echo 'raw email body input... '.$mailbody; it should print the actual email body which should be a HTML file.

At the moment this echo statement just prints a blank line so I think that is why I am getting a blank PDF because there isn't actually there to be converted.

As I said, have no proper idea what I am looking at, but any nudge in the right direction would be awesome!

Edit: that code didn't format properly. fixed.

chevrolux

4962 posts

Uber Geek
Inactive user

#1812647 4-Jul-2017 15:38

Ok so I think it definitely revolves around this line....

 $mailbody=imap_fetchbody($inbox,$email_number,2)

When i changed that number at the end to a 1 I get this....

PCFET0NUWVBFIGh0bWw+PGh0bWw+PGhlYWQ+CTx0aXRsZT5JbnZvaWNlPC90aXRsZT4JPG1ldGEg
aHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRm
LTgiPjxtZXRhIGNoYXJzZXQ9IlVURi04Ij48bWV0YSBuYW1lPSJ2aWV3cG9ydCIgY29udGVudD0i
d2lkdGg9ZGV2aWNlLXdpZHRoLCBpbml0aWFsLXNjYWxlPTEuMCI+CTxzdHlsZSB0eXBlPSJ0ZXh0
L2NzcyI+QG1lZGlhIHByaW50ICAgICAgICB7ICAgICAgICAgICAgYm9keSAgICAgICAgICAgIHsg
ICAgICAgICAgICAgICAgZm9udC1zaXplOiAxMnB4OyAgICAgICAgICAgIH0gICAgICAgICAgICAg
ICAgICAgIHRhYmxlLmZvb3RlciBoMyAgICAgICAgICAgIHsgICAgICAgICAgICAgICAgZm9udC1z
aXplOiAxMnB4OyAgICAgICAgICAgIH0gICAgICAgICAgICAgICAgICAgIHRhYmxlLmZvb3RlciAg
ICAgICAgICAgIHsgICAgICAgICAgICAgICAgZm9udC1zaXplOiAxMHB4OyAgICAgICAgICAgIH0g
ICAgICAgIH0gICAgICAgICAgICAgICAgYm9keSAgICAgICAgeyAgICAgICAgICAgIGZvbnQtZmFt
aWx5OiBzYW5zLXNlcmlmOyAgICAgICAgICAgIGxpbmUtaGVpZ2h0OiAxLjVlbTsgICAgICAgICAg
ICBsZXR0ZXItc3BhY2luZzogMC4wMmVtOyAgICAgICAgfSAgICAgICAgICAgICAgICAuY29udGFp
bmVyICAgICAgICB7ICAgICAgICAgICAgbWF4LXdpZHRoOiA5NjBweDsgICAgICAgICAgICBtYXJn
aW46IGF1dG87ICAgICAgICAgICAgdGV4dC1hbGlnbjogbGVmdDsgICAgICAgIH0gICAgICAgICAg
ICAgICAgdGFibGUgICAgICAgIHsgICAgICAgICAgICB3aWR0aDogMTAwJTsgICAgICAgICAgICBi
b3JkZXItY29sbGFwc2U6IGNvbGxhcHNlOyAgICAgICAgfSAgICAgICAgICAgICAgICAudW5kZXJs
aW5lZCAgICAgICAgeyAgICAgICAgICAgIGJvcmRlci1ib3R0b206IDNweCBzb2xpZCAjNjY5OWZm
OyAgICAgICAgfSAgICAgICAgICAgICAgICB0YWJsZS5zaGFkZWQgdGhlYWQgICAgICAgIHsgICAg
ICAgICAgICBib3JkZXI6IG5vbmU7ICAgICAgICAgICAgY29sb3I6ICM2Njk5ZmY7ICAgICAgICAg
ICAgZm9udC13ZWlnaHQ6IGJvbGQ7ICAgICAgICB9ICAgICAgICAgICAgICAgIHRoLCB0ZCAgICAg
ICAgeyAgICAgICAgICAgIHBhZGRpbmc6IDAuNDNlbTsgICAgICAgICAgICB2ZXJ0aWNhbC1hbGln
bjogdG9wOyAgICAgICAgfSAgICAgICAgICAgICAgICAucmlnaHQtYWxpZ25lZCAgICAgICAgeyAg
ICAgICAgICAgIHRleHQtYWxpZ246IHJpZ2h0OyAgICAgICAgfSAgICAgICAgICAgICAgICAubGVm
dC1hbGlnbmVkICAgICAgICB7ICAgICAgICAgICAgdGV4dC1hbGlnbjogbGVmdDsgICAgICAgIH0g
ICAgICAgICAgICAgICAgLnNjaXNzb3JzICAgICAgICB7ICAgICAgICAgICAgcG9zaXRpb246IHJl
bGF0aXZlOyAgICAgICAgICAgIGZvbnQtc2l6ZTogM2VtOyAgICAgICAgICAgIHRvcDogLTAuNzRl
bTsgICAgICAgICAgICBsZWZ0OiAzZW07ICAgICAgICAgICAgY29sb3I6IGdyZXk7ICAgICAgICB9
ICAgICAgICAgICAgICAgIC5zaGFkZWQgICAgICAgIHsgICAgICAgICAgICBiYWNrZ3JvdW5kLWNv
bG9yOiAjRUVFOyAgICAgICAgfSAgICAgICAgICAgICAgICBoci5jdXQtaGVyZSAgICAgICAgeyAg
ICAgICAgICAgIG1hcmdpbi10b3A6IDVlbTsgICAgICAgICAgICBib3JkZXI6IG5vbmU7ICAgICAg
ICAgICAgYm9yZGVyLWJvdHRvbTogM3B4IGRvdHRlZCBncmV5OyAgICAgICAgfSAgICAgICAgICAg
ICAgICAjaW52b2ljZS1oZWFkZXIgICAgICAgIHsgICAgICAgICAgICBtYXJnaW4tYm90dG9tOiAz
ZW07ICAgICAgICB9ICAgICAgICAgICAgICAgICNpbnZvaWNlLWRldGFpbHMsICNpbnZvaWNlLXN1
bW1hcnkgLmNvbnRhY3QgICAgICAgIHsgICAgICAgICAgICB3aWR0aDogMzMuMyU7ICAgICAgICB9
ICAgICAgICAgICAgICAgIC5oaWdobGlnaHRlZC1yb3cgICAgICAgIHsgICAgICAgICAgICBiYWNr
Z3JvdW5kLWNvbG9yOiBkYXJrYmx1ZTsgICAgICAgICAgICBjb2xvcjogd2hpdGU7ICAgICAgICAg
ICAgZm9udC13ZWlnaHQ6IGxpZ2h0ZXI7ICAgICAgICB9ICAgICAgICAgICAgICAgICNpbnZvaWNl
LXJlY2lwaWVudCAgICAgICAgeyAgICAgICAgICAgIHdpZHRoOiA1MCU7ICAgICAgICB9ICAgICAg
ICAgICAgICAgIC50b3RhbCwgLnN1YnRvdGFsLCAuZ3N0ICAgICAgICB7ICAgICAgICAgICAgZm9u
dC13ZWlnaHQ6IGJvbGQ7ICAgICAgICB9ICAgICAgICAgICAgICAgIHRhYmxlLnNoYWRlZCB0ZC5s
YWJlbCAgICAgICAgeyAgICAgICAgICAgIGNvbG9yOiAjNjY5OWZmOyAgICAgICAgfSAgICAgICAg
ICAgICAgICAuZm9vdGVyICAgICAgICB7ICAgICAgICAgICAgZm9udC1zaXplOiAxMnB4OyAgICAg
ICAgfSAgICAgICAgICAgICAgICAuZm9vdGVyIGgzICAgICAgICB7ICAgICAgICAgICAgY29sb3I6
ICM2Njk5ZmY7ICAgICAgICAgICAgZm9udC1zaXplOiAxNHB4OyAgICAgICAgfSAgICAgICAgICAg
ICAgICB0ZC5zdW1tYXJ5ICAgICAgICB7ICAgICAgICAgICAgcGFkZGluZy1yaWdodDogNjBweDsg
ICAgICAgIH0gICAgICAgICAgICAgICAgLnNwYWNlciAgICAgICAgeyAgICAgICAgICAgIG1hcmdp
bi10b3A6IDYwcHg7ICAgICAgICB9CTwvc3R5bGU+PC9oZWFkPjxib2R5PjxkaXYgY2xhc3M9ImNv
bnRhaW5lciI+PHRhYmxlIGlkPSJpbnZvaWNlLWhlYWRlciI+CTx0Ym9keT4JCTx0ciBpZD0icm93
LWxvZ28iPgkJCTx0ZCBjb2xzcGFuPSIyIj48aW1nIGFsdD0iIiBoZWlnaHQ9IjQwIiBzcmM9Imh0
dHBzOi8vd2VicmVzLm5sYi5uei9sb2dvL2JpZ190cmFuc3BhcmVudC5wbmciIHdpZHRoPSI3OSIg
Lz48L3RkPgkJPC90cj4JCTx0ciBpZD0icm93LWhlYWRlci1kZXRhaWxzIj4JCQk8dGQgaWQ9Imlu
dm9pY2UtcmVjaXBpZW50Ij4JCQk8aDE+TmV4TGV2ZWwgQnVzaW5lc3MgU29sdXRpb25zIEx0ZC48
YnIgLz4JCQlUYXggSW52b2ljZS9TdGF0ZW1lbnQ8L2gxPgkJCTxhZGRyZXNzPjwvYWRkcmVzcz4J
CQk8L3RkPgkJCTx0ZCBpZD0iaW52b2ljZS1kZXRhaWxzIj4JCQk8dGFibGUgY2xhc3M9InNoYWRl
ZCI+CQkJCTx0aGVhZD4JCQkJCTx0cj4JCQkJCQk8dGQgY29sc3Bhbj0iMiI+PC90ZD4JCQkJCTwv

Is that in an encoded format? If yes, how do I convert it?!?

itxtme

2102 posts

Uber Geek

#1812654 4-Jul-2017 15:48

Try take a look at the function imap_fetchbody. The variable you have changed has a profound effect, and being able to understand the actual function will go a long way in seeing what it is doing/not doing. Not sure how much experience you havew with coding/php so appologies if teaching to suck eggs. There will be either "include"/ "include_once" / "requires" near the top of the page. That function will exist in one of the included files, or it is possible its within the page you are looking at.

darylblake

1162 posts

Uber Geek

Trusted

#1812660 4-Jul-2017 15:54

Yeh so you need to get the structure:

You need to get the charset too.

Check the structure. 3 is base64'd. And 4 is qprint.

You can convert the encoding to UTF-8 to get yourself started.
iconv('cp1251', 'UTF-8//IGNORE', $string);

EDIT: the part number will usually be 1.

chevrolux

4962 posts

Uber Geek
Inactive user

#1812673 4-Jul-2017 16:15

itxtme:

Try take a look at the function imap_fetchbody. The variable you have changed has a profound effect, and being able to understand the actual function will go a long way in seeing what it is doing/not doing. Not sure how much experience you havew with coding/php so appologies if teaching to suck eggs. There will be either "include"/ "include_once" / "requires" near the top of the page. That function will exist in one of the included files, or it is possible its within the page you are looking at.

That's good advice. And yea i need to be told exactly how to suck eggs because I have no idea.

chevrolux

4962 posts

Uber Geek
Inactive user

#1812678 4-Jul-2017 16:31

darylblake:

Yeh so you need to get the structure:

You need to get the charset too.

Check the structure. 3 is base64'd. And 4 is qprint.

function x($conn, $messageNumber, $part_number=1) {
$structure = \imap_fetchstructure($conn, $messageNumber);
$text = \imap_fetchbody($conn, $msg_number, $part_number);
if($structure->encoding == 3) {
return \imap_base64($text);
} else if($structure->encoding == 4) {
return \imap_qprint($text);
} else {
return $text;
}
}

You can convert the encoding to UTF-8 to get yourself started.
iconv('cp1251', 'UTF-8//IGNORE', $string);

EDIT: the part number will usually be 1.

Yep so I think I have figured out it is base64.

So I went and stuck this in directly after the imap_fetchbody...

$mailbody = iconv('cp1251', 'UTF-8//IGNORE', $mailbody);

But that seems to just return the same encoded string. I went and found an online base64 decoder, ran the output through that and I get the exact HTML page I expect to get. So go to googling on conversions... found base64_decode()

So i put in

$mailbody=base64_decode($mailbody);

But now the script hangs and I get a 504 timeout. So clearly that isn't the way to go about converting.

chevrolux

4962 posts

Uber Geek
Inactive user

#1812789 4-Jul-2017 18:29

this is why developers get the big bucks.. im ready to just slowly push a pen through my face.

So it seems all I need to do now is decode that body from base64. Tried this:


$mailbody = imap_fetchbody($inbox,$email_number,1);
 echo imap_base64($mailbody);

And sure enough it echoes the exact HTML output I expect. But then when I go

$mailbody = imap_base64($mailbody);

The script hangs again and I get a timeout error. Have also tried loading that imap_base64 in to a new variable and get the same result.

Clearly I'm doing something that is not allowed.

darylblake

1162 posts

Uber Geek

Trusted

#1813039 5-Jul-2017 08:53

chevrolux:

this is why developers get the big bucks.. im ready to just slowly push a pen through my face.

So it seems all I need to do now is decode that body from base64. Tried this:

$mailbody = imap_fetchbody($inbox,$email_number,1);
echo imap_base64($mailbody);

And sure enough it echoes the exact HTML output I expect. But then when I go

$mailbody = imap_base64($mailbody);

The script hangs again and I get a timeout error. Have also tried loading that imap_base64 in to a new variable and get the same result.

Clearly I'm doing something that is not allowed.

Put error_reporting(E_ALL);

At the start of your script, or at least just before what you are trying to debug.
It may also pay to wrap the code you are having trouble with inside the try block of something like this:

try {

//Your code goes here

} catch(Exception $e) {

echo $e->getMessage()." ".$e->getTraceAsString();

}

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

chevrolux

4962 posts

Uber Geek
Inactive user

#1813863 6-Jul-2017 14:33

I should have enabled that error logging right at the start.

Have now found out my issue is with TCPDF itself doing the conversion and the the actual HTML that I am throwing at it. The initial HTML is being extracted just fine and imap_base64 is totally sweet as I can have that load in to a file.

So will have a play with different PDF makers and see what I find. DomPDF seems to work but can't handle the tables and doesn't really understand much CSS.

darylblake

1162 posts

Uber Geek

Trusted

#1813875 6-Jul-2017 14:49

chevrolux:

I should have enabled that error logging right at the start.

Have now found out my issue is with TCPDF itself doing the conversion and the the actual HTML that I am throwing at it. The initial HTML is being extracted just fine and imap_base64 is totally sweet as I can have that load in to a file.

So will have a play with different PDF makers and see what I find. DomPDF seems to work but can't handle the tables and doesn't really understand much CSS.

DOM PDF is ok if the files are small and the HTML is not Malformed. Or else it will poop the bed. It chews on memory, and if there is a lot of elements (Especially unclosed) it will pop the max memory cap very quickly.

TCPDF is actually a wrapper for f PDf. But if you are not constructing proper HTML then that wont like it either. F PDF is actually AWESOME. It just doesn't go directly from HTML, hence why you were probably using TCPDF.

You can use WKHTMLTOPDF which uses a webkit engine, its WAY less fussy. But you will have to run a static copy on a seperate thread from the CLI. Which is not particularly ideal.

So if you don't have control or any guarantee that the HTML is properly formed, you are gonna have a hard time converting it to PDF.

Why dont you just get the attachment sent as a PDF initially? What are you trying to actually do?

chevrolux

4962 posts

Uber Geek
Inactive user

#1813894 6-Jul-2017 15:03

So the situation is we have billing system, that we can't completely control, but it sends all its invoices as emails (and there isn't a decent way of following up invoices). We can control the HTML template it uses and I have made sure this template is email friendly so it renders really well in the major email programs - slightly loathed to change it much to suit this application.

But I want the CRM to receive this email, store the invoice against the customer, and then send it to the customer with it attached - the main benefit being the tracking is really easy and I can make it do reminder emails etc.

The module in the CRM is doing a fine job of taking the PDF, storing it and notifying the customer. Its just the PDF creation itself is the issue at the moment.

WKHTMLTOPDF was mentioned to me by the guys that run the billing platform so am going to investigate.