Email “Virus” Outage Incident Report

Tuesday, March 7th, 2017 | Jim Cheetham | Comments Off on Email “Virus” Outage Incident Report

Summary

On Thursday 2 March 2017, email to and from the Internet between approximately midnight and 7am was being incorrectly classified as containing a virus, and this caused some messages to be permanently lost. Inbound email was described as having been quarantined, but this was not correct; the original messages had not been preserved.

Between 7am and midday on the 2nd, the email service was effectively shut down for investigation and repair. By midday, all services had been restored. All email sent from 7am onwards would eventually be delivered normally.

Although not yet officially confirmed by the vendor, the cause of the problem was a corrupt or absent antivirus update to the edge email servers.

Timeline

Thursday 2 March 2017

  • Midnight to 1am : Inbound email is increasingly being marked as [PMX:VIRUS] and notification versions of the originals are being delivered to end-users.
  • 2:30am : outbound email is now being marked as infected, and is rejected (i.e. the senders are being notified that their messages are not being sent out).
  • 6:30am : The Information Security Office becomes aware of the issue, and halts all of the inbound and outbound email services in order to investigate.
  • 7:15am : Vendor documentation describes the error that is being seen, but the recommended fix does not work.
  • 8:10am : Our external support partner pro-actively contacts ISO to inform them that there is a current issue affecting multiple customers globally.
  • 8:30am : First ITS Service Notice published – updated with current information at 10:30, 11:30, 12:30 and 3:30
  • 10:00am : Announcement “Email delivery issues” emailed to all-depts@ and CITSP@
  • 10:20am : Outbound email services are restored, but only by disabling the normal antivirus checks. This is not a suitable choice for inbound email, however; this remains shut down.
  • 11:50am : Vendor supplies a working update to the antivirus; testing confirms that this fixes the problem properly.
  • 12:15pm : Inbound email services restored. All email sent to us since 6:30am will eventually be delivered normally.
  • 3:20pm : efforts to restore original copies of the incorrectly-marked inbound email are unsuccessful, and are halted. A further announcement “Re: Email delivery issues” is sent to all-depts@

Remediation

We will review the vendor’s incident report when this is published in order to identify any improvements we need in our configuration.

We will investigate the failed quarantine action that caused the mis-categorised email to have been lost.

We will discuss this incident within the context of Disaster Recovery and Business Continuity Plans, to see if any improvements need to be made to these.

Who looks at your data? Evernote, for a start.

Thursday, December 15th, 2016 | Jim Cheetham | Comments Off on Who looks at your data? Evernote, for a start.

Evernote is a great app that helps you create and keep track of notes and synchronise them across the various devices you own – so you can take a photo from your phone, label it, and use it in a document on your PC simply.

It’s also a great example of a “Cloud” service; in order to get that photo from your phone to your PC, it is first copied up to Evernote’s servers, and then your PC copies it down again. You can also access your data directly from a web browser from any computer, if you need it immediately.

However, Evernote does not do anything to encrypt the copy of the data that they store on their own servers. They have a privacy policy to promise to be good, of course … but that’s just changed.

The latest privacy policy goes into effect in January 2017, and as well as the perfectly necessary exceptions for things like court orders and malware incidents, they have now added a clause that says that employees of Evernote will access your data “to maintain and improve the Service”. That’s a very imprecise and broad statement. How will your data be used to improve their service? What is their service? Is it “anything the company does” or only “synchronising your files”?

Here’s a set of articles and longer discussion of some of the issues around this :-

* http://arstechnica.com/tech-policy/2016/12/evernotes-new-privacy-policy-raises-eyebrows/
* http://www.forbes.com/sites/thomasbrewster/2016/12/14/worst-privacy-policy-evernote/
* https://techcrunch.com/2016/12/14/evernotes-new-privacy-policy-allows-employees-to-read-your-notes/

If you are storing data which you believe to be sensitive in any way, you need to be aware of these policies, and when they change. While Cloud-based services offer many conveniences and a low cost to get started, the long-term costs are sometimes unacceptably high.

Remember, “The Cloud” means nothing more or less than “Someone else’s computers”, and there is often no enforceable contract of any kind.

Update:

The CEO of Evernote is now clarifying that the wording of their Policy was misleading; he states that “Human beings don’t read notes without people’s permission. Full stop.”

So, does that mean that you’re all OK to carry on using Evernote, that you can relax and the emergency is over?

You tell me – it’s your data. If you need to control access to your data, and you’re not able to do this completely because “it’s in the cloud” (where the provider changes their terms, conditions, ownership and even physical location without consulting or informing you), then perhaps you should be doing things differently.

https://www.fastcompany.com/3066680/the-future-of-work/evernote-ceo-explains-why-he-reversed-its-new-privacy-policy-we-screwed-u

Checking SHA256 OpenSSH fingerprints

Wednesday, December 7th, 2016 | Jim Cheetham | Comments Off on Checking SHA256 OpenSSH fingerprints

Many people using recent versions of ssh are now seeing SHA256 fingerprints by default when connecting to a new server, and finding it difficult to verify the fingerprint because the server itself doesn’t seem to have the right versions to tell you!

For example, here’s the client trying to connect …

$ ssh galathilion
The authenticity of host 'galathilion (10.30.64.220)' can't be established.
RSA key fingerprint is SHA256:8DpA4frlTxKnZ5GJXkORq8QQlLn4eCx4nZf51g55vYc.

The correct thing to do here is to check this fingerprint, by connecting to the target server over something that isn’t ssh. Then you run the ssh-keygen command to see the fingerprint …

# ssh-keygen -lf /etc/ssh/ssh_host_rsa_key
2048 d3:c6:fa:83:03:f4:ed:44:a4:3e:80:e1:b1:7b:ca:42 /etc/ssh/ssh_host_rsa_key.pub (RSA)

But that’s the wrong format – the MD5 version of the fingerprint, not the SHA256 version. That’s probably because the server version of the openssh tools doesn’t support SHA256 at all. And you can’t work out what the SHA256 fingerprint will be if all you have is the MD5 fingerprint data.

No problem; you can just ask your client ssh to display the server’s fingerprint using the old MD5 presentation :-

$ ssh -o FingerprintHash=md5 galathilion
The authenticity of host 'galathilion (10.30.64.220)' can't be established.
RSA key fingerprint is MD5:d3:c6:fa:83:03:f4:ed:44:a4:3e:80:e1:b1:7b:ca:42.

So that works a treat, and you can validate the connection. Regardless of the scheme used to present the fingerprint to you, it’s the same server public key, so validating the MD5 presentation is the same as validating the SHA256 version.

As an alternative, you can use standard command-line tools to generate the SHA256 fingerprint on the server itself, even though openssh doesn’t do that for you.

# cat /etc/ssh/ssh_host_rsa_key.pub \
  | awk '{print $2}' | base64 -d | sha256sum -b \
  | awk '{print $1}' | xxd -r -p | base64
8DpA4frlTxKnZ5GJXkORq8QQlLn4eCx4nZf51g55vYc=

That mouthful produces the same output as the openssh tool.

Here’s a worked-through example of how this command chain works. I can reproduce the original machine’s data here, because this is a public key. Remember to carefully check what data you are publishing online!

# cat /etc/ssh/ssh_host_rsa_key.pub
 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3R3I0dJxyg61jKuAqY3wJ/gwHzzEVg73sVqqJnzzEGWEkpjEYsIBk1NWh/Ur2q9CnR1KPk8Av22fNgeQay6dm9FcGK7TImiD3ZZfZjfHPzwkcoXyQPuJHW9pT8rlktkihdpTNJqlHHibVUz481AntmptypGqPKdg22EjvjrHk5Q4Op/ahZjgkSoFPphH1gWZcCC2xSPi/mk6nu9DF4Jyr1dJq+hJMPuvQ10ozOpzhemUKD9dGoXIh9g78/+M9Y8/naOW+UxZAy8BGrcpjM27sLHU0K+qxLRFw36Xlgur2+lEiSVt0F2iPpbAiJug3hUQTx2K3gkMG36auVsgrWvK9Q==

This file has a single line, with two (or three) values space-separated. The second field is the Base64 representation of the public key itself, which we’re extracting using awk '{print $2}' (although we could have done this with other commands, such as cut -d' ' -f2). Once we have that field, we convert it from Base64 back into raw binary with base64 -d. Then we pass the binary key through sha256sum, which will produce two fields, a hex-encoded fingerprint and the filename (which is just ‘-‘ for standard input), and through awk again to select just the first field. xxd is used to convert the hex-encoded data back to binary again, and finally base64 gives us the same encoding that the openssh tools present.

Why bother with all that? Well, remember that the requirement to verify a server’s ssh fingerprint should not be carried out over ssh itself. I get my servers to write their ssh fingerprints into the /etc/issue file, and this is displayed on the server console by default along with the login prompt. So I can always validate the ssh keys using something that isn’t ssh …

Can my light bulbs DoS me?

Friday, October 7th, 2016 | Mark Borrie | Comments Off on Can my light bulbs DoS me?

Denial of Service (DoS) attacks are situations where an IT system is deliberately overwhelmed to a point where normal activity is no longer possible. A DoS attack usually comes from a single source. Where the attack comes from many sources we call this a Distributed DoS, or DDoS.

DDoS traffic is usually sent from many computers from around the world. These computers will have been hijacked and grouped together into a botnet which are then controlled by the bad player. These computers usually have been compromised because security patches have not been applied.

For some time InfoSec people have been wondering what impact the Internet of Things (IoT) will have on things like DDoS. The IoT generally refers to all those everyday objects that can now connect to the Internet. Think CCTV, toys, nappies, cars, door bells etc.

At the KiwiCon conference last year in Wellington, and at other conferences,  some of the presentations talk about the security of various IoT devices. Some of the findings were

  • Certain brands of car immobilisers could be activated by other people by attacking the website where the device was registered. This was particularly concerning for those vehicles that could have the fuel system shut down. Imagine suddenly running out of fuel while in the middle lane of a motorway doing 110 km/hr!
  • Home security devices being sold in NZ could be controlled by other parties.
  • Baby monitors can be listened into, and worse.
  • Barbie Dolls are relatively hard to take over.

So are the IoT really a problem? In late September a DDoS attack was launched against a well known security writer. The attack forced his website off line for a time by the huge volume of traffic sent to it. What is interesting is that most of the devices involved in the attack were on line cameras (it is estimated that about 1.5 million cameras were involved).

The security issues with the IoT may well turn out to be a bigger problem than Y2K. When preparing for Y2K it was possible to identify likely systems that needed fixing, and then update them. In the end a Y2K disaster was avoided since we understood how to fix the problems.

The problem with IoT is that we cant identify and/or fix most of the devices. Few manufacturers of an IoT device include options for efficiently getting updates onto the device. Almost no one will commit to providing support for any set time. For most devices, if there is a significant security issues with them, it will be a case of throwing them away.

So this comes back to the original question. Can my light bulbs DoS me?

Well, the current versions probably cannot launch a co-ordinated network attack, which is good. However, a bad player may well be able to take control of your light bulbs. Think about the result of all your lights coming on at 3 am. Perhaps we should call this a DoSl (Denial of Sleep).

What to do? Some of the functionality of some IoT devices is truly exciting. We are going to see more and more options out there. When buying these devices we need to start thinking about the impact if things go wrong. Ask the retailer about security updates. If updates are available ask for how long support will be provided.

Finally, be prepared to throw the device away. This may end up your only option.

Update on RansomWare

Friday, October 7th, 2016 | Mark Borrie | Comments Off on Update on RansomWare

In March this year I wrote about the upswing in ransomware attacks. Well since then we have seen even more attacks. Unfortunately some people have been caught out by the attackers and have had files encrypted.

Luckily everyone so far has been able to restore their data from backups and other sources.

The criminal gangs running these attacks are constantly looking for new ways to get results. Recently they used a flaw in certain types of Word docs. These were Word files with macros in them. Once we worked out what they were up to we started using our spam management system PureMessage to quarantine all these Word files with macros (these have a docm suffix). Only a few genuine files were quarantined and they were still available to the user.

During August we quarantined about 150 000 docm files.

Since then we have seen a decline in the use of docm files but a large increase in zip files. Zip files are a convenient way to bundle together a number of files in a compressed format that makes them easier to distribute.

During the first 12 days of September we quarantined about 1.5 million zip files. Almost all of these had some sort of malicious content.

Dealing with these ongoing attacks is a team effort and we all have a part to play. Remember if something looks suspicious then get someone to check it out.

User awareness videos

Tuesday, July 12th, 2016 | Mark Borrie | Comments Off on User awareness videos

Here are a couple of videos to help people become a bit more aware of social engineering risks. It would be interesting to hear from you as to which one you think is more effective.

http://click.email.sans.org/?qs=c5f96f16ca893d8d2f0d14de6b7a75772c968cabaa38a6908445ec57e78146b3c527b9f0ea796009

 

Here is another good one from a bank

 

What is Ransomware?

Monday, March 7th, 2016 | Mark Borrie | Comments Off on What is Ransomware?

In recent years another new term has emerged to describe yet more malicious software that attacks users. This one is called ransomware.

So what does ransomware do?

When a computer become infected with this software, all the files on the computer get encrypted. The user is then notified and offered an option of paying a ransom to get the secret decryption key in order to recover the files. If the user refuses to pay up all the encrypted files are lost.

There has been a large increase in ransomware attacks worldwide in recent months. The Information Security Office team is seeing large numbers of spam emails being intercepted here at Otago that are connect to ransomware attacks.

A recent attack

Many staff recently received an email claiming to be from a lawyer that suggested the user had breached copyright on some material. This spam was deliberately sent during the weekend so that users would not have the usual support channels available (Alarm bell #1). This email was sent to many other Universities.

An analysis from another institution of the email revealed that some interesting things.

  • The email had a zip file attached (Alarm bell #2)
  • The zip file attached to the email contained a pdf that had a script in it.(Alarm bell #3)
  • This script requested the user to install a special font in order to read the pdf (Alarm bell #4)
  • If the user (or their IT support person) finds the font and installs it then the ransomware is installed and immediately starts encrypting all the user’s files INCLUDING those on file shares.

Protecting yourself

Targeted spam attacks are getting more sophisticated. They use real companies and individual’s names. They are sent outside normal work hours, i.e. during weekends or holidays, or overnight. They often appear to be relevant to the target people, i.e. copyright issues for academics, or account information for financial staff.

Things to do (or not do)

  • Do not respond to unexpected emails outside work hours (It really isn’t that urgent)
  • Do not respond to requests to “take an action” (It truely is not that urgent)
  • Check with IT staff or colleagues if you get an unusual email. Chances are it will be a known attack, or it will alert staff of a new one under way
  • Be prepared. Make sure all your data files are properly backed up. Some of the ransomware attacks are now targeting backups as well as file shares so backups should not be accessible to the attack

For more information or assistance, contact the ITS ServiceDesk or the Information Security Office.

Malware – more than just a virus?

Thursday, March 3rd, 2016 | Mark Bedford | Comments Off on Malware – more than just a virus?

Seems that the term malware is causing confusion as the term itself covers a wide variety of malicious activity and is a contraction of the two words “malicious software”. It is generally used in the information security area to refer to software that is malicious in intent but does not cover unintentionally bad or faulty software.

There is a type of malware called spyware which is sometimes embedded in applications that appear useful but may have additional hidden functionality that gathers marketing information.

The SANS Ouch this month contains information describes it in more detail and provides some tips on ways to protect yourself.

Financial fraud phishing emails

Wednesday, February 17th, 2016 | Mark Borrie | Comments Off on Financial fraud phishing emails

The Information Security team has noted an increase in phishing emails that are targeting staff who may handle financial transactions. Initially these emails targeted senior staff and attempted to get fraudulent payments made by the University. A tertiary organisation up north fell victim to this and may be out of pocket to the tune of over $100k.

The phishing emails are now targeting departmental staff. The email will appear to come from another University staff member and attempt to establish further email communication. The email address will not be an @otago.ac.nz address. Eventually the target victim will be asked to set up a fraudulent financial payment. These emails are asking staff to work outside University financial processes by suggesting that there is some urgency in processing the request and hence bypass normal processes.

Staff who handle financial transactions are asked to be vigilant for these types of attacks. If you receive unusual requests to process payments then ensure that the following is undertaken
– Check with the apparent requester via another channel, i.e. if the request comes via email then give them a call to verify.
– At all times follow the University account processing systems. Contact FSD if you have questions.
– Report any attempts of this nature to the Information Security Office so that we can keep up to date with current attacks.

If you have any questions regarding this matter then please contact myself.

Thanks, Mark

P.S. A copy of this email has been posted on the Information Security Blog site for verification. See https://blogs.otago.ac.nz/infosec/2016/02/17/financial-fraud-phishing-emails/

-- 
Mark Borrie
Information Security Manager,
Information Technology Services, University of Otago,
Dunedin 9054, N.Z.
Ph +64 3 479-8395, Fax +64 3 479-8813
Email: mark.borrie@otago.ac.nz

Airline Boarding passes

Thursday, October 8th, 2015 | Mark Borrie | Comments Off on Airline Boarding passes

Ever wondered what is recorded on your airline boarding pass? Well someone has done some analysis of one and quickly turned up some interesting information. The bottom line is don’t throw them away, instead make sure you take them home and destroy them at the end of your travels.

See http://krebsonsecurity.com/2015/10/whats-in-a-boarding-pass-barcode-a-lot/ for an explanation.

 
 
 

Any views or opinion represented in this site belong solely to the authors and do not necessarily represent those of the University of Otago. Any view or opinion represented in the comments are personal and are those of the respective commentator/contributor to this site.