After I'd finished writing last night's post, hung the washing out, brushed me teesh, got nekkid, etc. it was gone 1:30am by the time I got to bed. I think it was around 2am by the time I really started trying to get to sleep.
So I was pretty surprised that I woke up fairly well this morning and actually felt in a great mood, and even made it to work a little early :)
This mood continued throughout the morning and everything just seemed great :)
Until I read an e-mail from a client. E-mails were getting CC'd to someone in their office and they shouldn't have been. As I'd been delivered a few li'l updates to the client yesterday, I'd also delivered some other changes that had been made (not by me).
It appeared to be a silly little problem... Someone had probably just put in a hard-coded e-mail address to use to test with, and forgotten to take it out, or something.
Oh I wish it was that ;)
No, it turned out, once a CC address was used, all the subsequent e-mails that went out were CC'd to the same address.
Brown. Trousers. Time.
Can you imagine that in a business situation where you're sending around confidential information having a whole bunch of e-mails get CC'd to someone in a different company?
On the scale of embarassingly huge cockups, this sort of thing is pretty major. Honestly, I'd rather deliver an enhancement to find the thing just plain didn't run (which is very embarassing) than something like this.
"Fortunately", due to the way the program was set up, this "only" applied to e-mails that went out over the next 15 minutes or so.
Speaking of 15 minutes, that was roughly how long it took me to read the original report, find the problem and deliver a fix. It took about a bit more than another hour after that though for me to deliver a list of all the e-mails that had incorrectly gone out when they should not have done (In total, only eight e-mails had been sent that shouldn't have been - but it would only take one to potentially start Who Knows What Sort Of Crisis).
As I was on the phone to our client who was reading my e-mail explanation at the time, "Are you serious?" he asked. "Oh shit," he said.
I am glad that I work for a company where this sort of problem very rarely occurs, and I am glad that when it does happen, the entire company takes it very seriously.
I don't understand how this sort of thing seems to happen "all the time" with some IT companies. More than that, I don't understand how it is that they don't actually appear to care that it does.
Oh well. At least it got caught early on and with a little luck it won't cause any seriously sticky shit to go hitting the fan :)
Subscribe to:
Post Comments (Atom)
OMG! Sounds like you managed to pick up on and sort out the problem before it got too out of hand. Have there been repercussions for you?
ReplyDeleteMy fingers crossed for you hon.
Fortunately everything seems to be ok.
ReplyDeleteThe company is close by and I popped past and spoke to someone there before I came home last night.
He was thankful for the fact that as soon as the problem had been discovered we gave them all the information on exactly what had happened (they hadn't been aware - we could have kept our mouths closed). He had contacted someone at the company most of the e-mails had gone to and they said they'd delete them.
I know that years and years ago something happened like this to them (human error rather than technical - we introduced new technical features to help prevent the human error from occurring again), before the era of e-mail, and it caused major problems at the time that led to their losing a client I think.
It seems this time everyone has been lucky. Phew! :)