Email “Virus” Outage Incident Report

Tuesday, March 7th, 2017 | Jim Cheetham | Comments Off on Email “Virus” Outage Incident Report

Summary

On Thursday 2 March 2017, email to and from the Internet between approximately midnight and 7am was being incorrectly classified as containing a virus, and this caused some messages to be permanently lost. Inbound email was described as having been quarantined, but this was not correct; the original messages had not been preserved.

Between 7am and midday on the 2nd, the email service was effectively shut down for investigation and repair. By midday, all services had been restored. All email sent from 7am onwards would eventually be delivered normally.

Although not yet officially confirmed by the vendor, the cause of the problem was a corrupt or absent antivirus update to the edge email servers.

Timeline

Thursday 2 March 2017

  • Midnight to 1am : Inbound email is increasingly being marked as [PMX:VIRUS] and notification versions of the originals are being delivered to end-users.
  • 2:30am : outbound email is now being marked as infected, and is rejected (i.e. the senders are being notified that their messages are not being sent out).
  • 6:30am : The Information Security Office becomes aware of the issue, and halts all of the inbound and outbound email services in order to investigate.
  • 7:15am : Vendor documentation describes the error that is being seen, but the recommended fix does not work.
  • 8:10am : Our external support partner pro-actively contacts ISO to inform them that there is a current issue affecting multiple customers globally.
  • 8:30am : First ITS Service Notice published – updated with current information at 10:30, 11:30, 12:30 and 3:30
  • 10:00am : Announcement “Email delivery issues” emailed to all-depts@ and CITSP@
  • 10:20am : Outbound email services are restored, but only by disabling the normal antivirus checks. This is not a suitable choice for inbound email, however; this remains shut down.
  • 11:50am : Vendor supplies a working update to the antivirus; testing confirms that this fixes the problem properly.
  • 12:15pm : Inbound email services restored. All email sent to us since 6:30am will eventually be delivered normally.
  • 3:20pm : efforts to restore original copies of the incorrectly-marked inbound email are unsuccessful, and are halted. A further announcement “Re: Email delivery issues” is sent to all-depts@

Remediation

We will review the vendor’s incident report when this is published in order to identify any improvements we need in our configuration.

We will investigate the failed quarantine action that caused the mis-categorised email to have been lost.

We will discuss this incident within the context of Disaster Recovery and Business Continuity Plans, to see if any improvements need to be made to these.

Phone-to-email Spam

Wednesday, November 26th, 2014 | Jim Cheetham | Comments Off on Phone-to-email Spam

Can I send you an email?

In the last few months there has been a rise in “pretexting” phone calls from legitimate marketing organisations, probably in response to anti-spam legislation around the world.

The usual script is an unsolicited phone call from a real live human, asking for permission to use your email address in order to send you some marketing material, usually described as a “White Paper”.

The calls are often made over a low-quality connection (i.e. cheap VoIP) and come from non-native English speakers (a kind way to suggest “offshore call centres”). However, they do generally respond well to a polite “No thanks” as an answer, and to requests to not be called again in the future. If permission is given the eventual email usually represents a legitimate trading company of some sort.

All in all, no real problem.

I have a business opportunity for you!

However, we’re beginning to hear of the same approach being used by spammers, particularly of the advance-fee fraud variety. A small amount of research (i.e. get your name and job title from a website), a hacked VoIP system (which lets them call anywhere in the world for free), and a fresh email address from one of the big free webmail providers potentially gives these criminals a much more direct line to your mailbox and your attention.

This is a particular worry because it won’t be long before these techniques are used for distributing fresh malware – you receive a difficult-to-understand phone call from someone with urgent information to send to you, and a couple of minutes later in comes the email, along with a juicy PDF attachment. Would you resist the temptation to click? How can you tell the difference between an attack, and a real foreign student or academic trying to work with the University?

(Some of you reading this might suddenly realise that you already open too many attachments without stopping to check fully the source!)

What should you do?

The best defence if you are unsure would be to check with your colleagues, see what they think; to check with your IT support; or to ask the ITS Information Security Office for an opinion.

If you don’t have an opportunity to get a second opinion, you have a few technical opportunities to reduce the risk. Firstly, wait a while … come back to the message in a couple of hours time. If the message came out of hours, just don’t open it until you are back at work. Remember that just because it seems to be urgent to someone else it is not necessarily urgent to the University!

Instead of just opening the attachment, ask your anti-virus software to scan it. This is best combined with the “wait a while” approach – if this is a new malware sample (there are tens of thousands per day automatically created), give your AV software time to get an update from the vendor.

Finally, open the attachment in an unusual program. For example, malware PDFs often only successfully attack Adobe Acrobat Reader, so if you have a different PDF reader available you could use that. Instead of opening untrusted attachments with Microsoft Word, open it with a copy of Libre Office. If your job role means that you will be receiving unsolicited attachments regularly, get your IT support to help you install these alternatives.

Finally, if you are in any doubt, leave the file alone and refer the whole thing to the Information Security Office. We can check a lot more details to find out what is going on, and we really don’t mind being asked.

Down or Not?

Wednesday, October 10th, 2012 | Jim Cheetham | Comments Off on Down or Not?

Here, this looks like a good joke … downornot.com is down!

Except of course, it isn’t down …

Or perhaps it is …

So, is DownorNot.com down, or not?

The problem here is that the meaning of “it” in the phrase “is it up or down?” is different each time. For the human, “it” is the service provided by the website, and that service is certainly non-functional (it looks like the website’s usage of the Google AppEngine service has gone over quota; and the developer who wrote their website didn’t anticipate this error. In this case, we would also anticipate that the service behind the website is also broken at the moment, whereas often a service and a website are separate). For isitup.org, “it” seems to be the initial contact with the site’s webserver, which is responsible for handling the HTTP conversation. For downforeveryoneorjustforme.com, “it” is the same thing, the webserver and not the application, but they are going a couple of steps further, following the redirections and noticing that the final page is delivered with an error status … and in this case, the failure of the application is reflected in the webserver status code. It doesn’t have to be.

Let’s ask isitup.org again, but this time go straight to www.downornot.com instead of just downornot.com …

So, if you are monitoring or testing a service, make very very sure that you understand what the question “Is it up?” is supposed to mean — and this depends on who is asking. Then, make very sure that you understand how your monitoring tools work, what you are asking them to do, and how to interpret them. Obviously, this isn’t as easy as it sounds …

If you are not careful, this is what you end up with :-

The future of computer threat response

Thursday, August 23rd, 2012 | Jim Cheetham | Comments Off on The future of computer threat response

Dan Geer is a voice in the IT Security world that should be listened to, and he has co-authored a short but intense article in the IEEE S&P Cleartext column that addresses the reality of the rapidly-changing threats that we all face.

Stand Your Ground” has a few key messages, which I’ll try to summarise :-

  • Minimise the number of targets; stop adding new services without effective defences, remove old services. Use the savings to fund better security for what remains.
  • Distrust the internal network; distrust any service that is not continually verified. Defend against outbound traffic as well as inbound traffic.
  • Do not assume perfection is possible; plan for failure modes that reduce services sensibly. Reduce the time-to-repair with automation instead of extending the mean-time-between-failure.

For more reading and a less technical take on these ideas, here’s another article about Dan’s thoughts from Ben Tomhave, a GRC (Governance, Risk & Compliance) consultant.

Half a million Aussie credit card numbers fly to Eastern Europe

Wednesday, August 22nd, 2012 | Mark Bedford | Comments Off on Half a million Aussie credit card numbers fly to Eastern Europe

In an article from Wired thieves allegedly stole half a million Australian credit card numbers. The raid on an unnamed business in Australia used keystroke logging software on POS terminals then transferred the information back to Eastern Europe. The POS terminals had default passwords and stored transaction data insecurely. They were accessed using an unsecured Microsoft Remote Desktop connection. But the notable quote is

“The network was setup by some local suppliers who didn’t understand IT security,” Det. Sup. Marden told the magazine. “It was a disaster waiting to happen.”

As is often the case in many incidents, most of the layers of defence were turned off, left with default values, or just were not considered necessary. Had any one of the following been in place this particular incident could have been averted: changed the vendors default password, forced access to the terminals via a VPN, AV software on the workstations, IDS on outward network traffic, logging and monitoring of authentication services. Lastly, had the Aussie business performed an independent information security audit these would have been identified and the costly compromise could have been avoided.