Sam Pointer

I recently had cause to liase with the support team at a hosting company, who shall remain anonymous to protect the guilty.

I was having some problems with getting the SMTP server on the VM image they had there to properly accept email. As far as I knew I had everything in place:

• correct MX, A and SPF records for the domain published and properly propagated, confirmed by a ‘dig’ from an external server
• domain listed in the local domains (Cw) section of the Sendmail configuration
• Sendmail restarted to ensure everything had taken effect
• SMTP port 25 allowed in the local iptables configuration

Everything should have been working, and yet I was receiving the dreaded "Connection Refused" back from my sending SMTP server after the requisite time-out period.

From a local machine I attempted to connect to port 25 of the receiving SMTP server with netcat, reasoning that if I could get at least a TCP connection to the port I’d know that my iptables configuration was good and something must be up with my Sendmail configuration. No joy, connection refused.

So I was getting somewhere. In order to make sure it wasn’t just a local problem I confirmed this from another remote server. Same result. I then took advantage of the port monitoring facility of the HyperVM virtualization manager the host was using to see if I could connect to port 25 from there, my reasoning being that it would narrow the problem down to the hosting company’s infrastructure and rule out any sillies at my end. Sure enough, expected ports such as 80 for http and 22 for ssh were open and accepting connections, 25 wasn’t.

So, I lodged a support ticket with my findings, what I was trying to achieve and what I needed doing, namely opening up whatever part of their infrastrucuture was blocking port 25 to the VM in question. Being in the UK and the hosting company being in the states I left it at that and expected to come back the next day to find everything sorted.

Two hours later I got an email from the monitoring service on the VM. xinetd had failed, which has odd because xinetd virtually never fails.

Connecting to the machine for some investigation revealed that the Dovecot POP3 server was constantly trying and failing to bind to port 110, and the rapid respawing had caused xinetd to fail leaving a trail of pid files and subsystem lock files in it’s wake. This was quite confusing, as I was using a custom-compiled version of qpopper launched via xinetd to handle POP3 mail, and had never installed Dovecot.

All was revealed when an email appeared shortly afterwards from the support desk of the hosting company saying "I’ve installed the Dovecot pop3 email server, let me know how you get on now".

Fail number one.

The support guy at the other end had obviously not read the support ticket, which made no mention of pop3. He’d also assumed that I wasn’t running another pop3 server - a quick `netstat -a | grep -i listen` would have told him that I was.

So, I promptly told him all of this an reiterated my original request that, thankyou very much I know what’s going on (and here’s the evidence to prove it), and would you kindly remove the POP3 server and open port 25 please. Oh, and please uninstall Dovecot as I’ve already got a POP3 server and I don’t want a repeat of the xinetd failure.

I then receive another reply from a different technician saying sorry for the confusion, and he’d enabled SMTP Authentication within Sendmail, and could I please try again. Should things still not be working, could I please forward them my email address and password so they could configure their mail client the same way and attempt to send an email via my Sendmail instance.

Fail number two.

I’d never mentioned collecting email, sending email via the SMTP instance (which was configured only to relay from the local machine) or any requirement for authentication. I’d only asked for the port to be unblocked in their firewall.

This change had the compound effect of causing the ActionMailer portion of a Rails app that I was running to promptly fail, as it expected to relay via a local SMTP server without authenticating.

The helpdesk had now succeeded in knocking out the ability to collect email for the domain, caused an application sending email from the domain to fail, and still not solved the original problem.

Eventually after stating in explicit terms that they should undo everything they had done aside from keeping port 25 open the problems were fixed. It brought home to me a lesson that I had learnt long ago, and that these anonymous helpdesk technicians would do well to learn themselves.

First and most importantly, read what is in front of you. I had provided enough information in my original support ticket, backed up with sensible evidence of what was wrong, and made an explicit request for a single task to remedy this.

I have dealt with a wide number of financial institutions in my time. Some of these are large multi-national banks, others are small shops with barely a single IT literate person in the company. You learn to pitch your questions and your answers to the audience that you are addressing. For some people you have to explicitly spell everything out. For others you need only tell them what to do in a few concise sentences, and provide the technical detail (IP addresses, paths, passwords, whatever) that they could not know themselves and let them get on with it. The first group appreciate the hand-holding as they don’t have to ask questions and perhaps sheepishly asking for more help. The second group appreciate being treated as a professional and you getting out of their way to get the job in hand done. The real skill is learning how to pitch to the shades of gray in-between.

Being a specialized hosting company I addressed the helpdesk guys as professionals, and they resolutely showed that whilst they might have the technical cojones, they faled at good helpdesk interaction and judging how to pitch and receive information. It was obvious from the information I provided in the support ticket that I knew my stuff and had a simple request that they could have carried out and closed the ticket. If I had written, "hello there, I am not able to get my email. An error keeps popping up and it says refused, or something" then it would have well been in reason to go about installing a POP3 server and asking me to test it out. But they didn’t pick up on the subtle cues of content, syntax and context which would have enabled them to deduce what was going on.

In the technical consultancy field we should never forget who we’re talking to and what we’re talking about, and we should pitch our interactions as such. And like the test we’ve all done at school where the last instruction is to only write your name, always read what is in front of you.