For the last 2.5 years, we've used a popular service called Zendesk to store, organize, and answer emails to Tumblr Support. We've learned that a security breach at Zendesk has affected Tumblr and two other companies. We are sending this notification to all email addresses that we believe may have been affected by this breach.
This has potentially exposed records of subject lines and, in some cases, email addresses of messages sent to Tumblr Support. While much of this information is innocuous, please take some time today to consider the following:
The subject lines of your emails to Tumblr Support may have included the address of your blog which could potentially allow your blog to be unwillingly associated with your email address.
Any other information included in the subject lines of emails you’ve sent to Tumblr Support may be exposed. We recommend you review any correspondence you've addressed to support@tumblr.com, abuse@tumblr.com, dmca@tumblr.com, legal@tumblr.com, enquiries@tumblr.com, or lawenforcement@tumblr.com.
Tumblr will never ask you for your password by email. Emails are easy to fake, and you should be suspicious of unexpected emails you receive.
Your safety is our highest priority. We're working with law enforcement and Zendesk to better understand this attack. Please monitor your email and Tumblr accounts for suspicious behavior, and notify us immediately if you have any concerns.
Nowadays, it's not enough to wipe the machine. You have to toss (or, at least, re-image) the hardware.
Modern servers have firmware running on internal devices, including Linux runnning on full ARM cores, that can be 1) imaged from the base OS, and 2) have full control over hardware or other important data.
... as well as ethernet controllers, any devices with PCI option ROMs (they can control boot), and quite a few other things.
This essentially means that you have a bunch of fixed, poorly documented, runtime reconfigurable and re-imageable hardware, with full access to data and system resources.
"As soon as we learned of the attack, we patched the vulnerability and closed the access that the hacker had."
Ok, so striking out so far. The machine is still running? With the same software (patched) and user accounts? How do you know only 3 users were exposed?
If you have a mid-sized Rails app, with say 2-3 developers working on it, a full time security engineer would probably be overkill. Anyone have any recommendations of services/consultancies to be able to tell "oh, someone is hacking us right now" or "our page which has stripe.js on it has been compromised"?
I'm hoping for automated tools, services to install on our servers, or security auditors who have an out of the box package.
Tinfoil Security (https://www.tinfoilsecurity.com/) has automated scanners and seems to keep up to date with all the vulnerabilities quickly, like in case of the rails vulnerability recently.
Actually, I've had really bad experiences with Detectify. Their results didn't provide anything useful that I couldn't have gotten from something like Nessus. They have a pretty nice design, but not much in the way of actual useful security.
I'd Highly recommend Nessus as well. OpenVAS is another that is alright. Most scanners seems to favor false positives though, so I wouldn't put a ton of weight behind their results.
But since the question was one of a live monitor that detects intrusion, I've never heard of such a thing. There's always the possibility of aliasing `mysqldump' or `pg_dump' to another command that emails your admins, or other manual commands that shouldn't be run throughout the course of the day. My personal boxes run such an email script anytime someone logs in as root, and emails the logfile anytime someone uses sudo. That won't help against SQLi, but might against RCE that's allowed someone to tunnel into your box.
But, in the long run, there's nothing that won't beat subscribing the the security lists of all the software you run to get immediate notice of any vulnerabilities, hiring a pen tester, and stopping every day to read the code you've written to discover what kind of edge cases might help an attacker compromise your system.
Dear throaway132,
I am one of the founders of Detectify and, like inkel above, I am very curious to hear your feedback regarding your experience using Detectify. Please respond publicly here or DM me at piotr#detectify.com. Looking forward to hearing from you soon!
If it was, at it occurred after the vulnerabilities were made public, they probably wouldn't say so as it would look pretty bad given the amount of advance warning they had.
No, but he's suggesting that you should've upgraded by now.
It would be difficult to defend such a position: "We didn't upgrade because, well, we didn't think it was a big deal"?
There's really no good answer to that question when the exploit has been public for so long (and widely covered in media, and by rails officially).
But there's a greater than 0% chance that zendesk had an API token for at least one of those services. That could easily allow a hacker to make authenticated requests to those services to gain user info. The fact that usernames and passwords weren't stored on zendesk doesn't mean much, if a hacker can gain full admin access to those other services through an admin token that might have been stored on zendesk.
I seriously doubt any company (especially the three listed) would give Zendesk admin access to their service. Why would such a thing be necessary, anyway?
I think he meant it the other way around. Having their API token would allow the attacker to have access to all of Twitter/Tumblr/Pinterest's information that's accessible via the Zendesk API.
From the perspective of a complete server administrator novice, are all of the mainstream "hacks" due to the complexity of these applications? For example, if I were to setup a basic, updated Ubuntu Server LAMP stack with a MySQL database, is this system vulnerable? I understand how to protect against XSS and SQL injection and how to hash and salt passwords properly, but where can I begin to learn about implementing basic, hard server security? Additionally, how can I hope to secure my web app if corporations with entire security departments are failing to secure theirs?
how can I hope to secure my web app if corporations with entire security departments are failing to secure theirs
Excellent question. The answer really is, if you are running a "web app" with any kind of sensitive information, you need to have a security expert configure and administer your systems, or become one yourself, and you need to stay on top of every update.
Now, there are basic things you can do that will eliminate most of the casual script kiddie attackers. Firewalls. Defense in depth. Keeping up with patches. Google or any good book on computer security will provide this information. But the sad truth is, if a smart, determined attacker has you in his sights, you will most likely lose. And even the automated tools are getting better and better.
I think we are starting to see a tipping point in the web. More and more high profile sites are getting compromised. We have learned that China has been inside US government, utility, and industrial systems for years. Right now I would not trust any sensitive, personal information to any website or cloud service. I think we are going to see some major, consequential attacks on government, banks, and other commercial entities in the coming years. At some point I believe we are going to need to rethink the cost/benefit equation of having everything connected to everything.
> Right now I would not trust any sensitive, personal information to any website or cloud service.
A friend of mine put together an open source project that uses cryptography to store your data in the cloud securely, so it's certainly possible. [1]
It's also possible to write complex software and not be vulnerable, though 99.999% of the time companies (start-ups and otherwise) seem more concerned with an MVP and new features than security. If you design a system from the ground up with security as a core feature, then you have a CHANCE of having a system that won't be vulnerable to script kiddies every other week. On top of that you need to be sure to protect against social engineering, but that's another discussion.
I don't even know if it's possible to use something like Rails (or Ruby, even) and be secure for the long term without having to deal with constant updates and patches. On the other hand, I HAVE used complex systems that were designed from the ground up to be secure and that simply NEVER turned out to have a security vulnerability after the first few releases. (Anything by DJB, for example. [2] Some of those tools have gone 15+ years with no vulnerabilities. Compare the constant sendmail or bind security exploits, numbering in the hundreds at this point, to DJB's qmail and djbdns.)
Until it's a priority, it's always going to be an afterthought, by definition. People will use Rails or the framework du jour, despite the fact that such frameworks are designed with the same "get it done and release ASAP" philosophy that most commercial sites are developed with, and then everyone wonders at security holes. Sigh.
> a basic, updated Ubuntu Server LAMP stack with a MySQL database
After just apt-get'ing that, the system is of course always vulnerable to the next as-of-yet-undisclosed 0day vulnerability in the base software.
As soon as you start adding useful scripts of the "P" variety (as in "LAMP") then you are at the mercy of those scripts not having any vulnerabilities.
If you wrote those scripts yourself, then yes, xss, sqli, rfi etc are all issues you need to consider. The OWASP pages could be a useful place to start getting a feeling of the most common pitfalls.
The typical LAMP-stack out of the box is nowadays relatively safe. They were a lot unsafer earlier.
So, how to keep safe:
1. The app itself:
If you have a small web app, you have an incredible advantage that makes your web app potentially way more secure than that of a big corporation: You write the code and you know the inside outs of the system. And you are probably the only one having passwords etc. to the system.
In a big corp, sometimes the interns write some code and they have no clue of web app security. From an attacker's perspective, one security hole is enough.
So, if you know about web app security, you're probably better off than any other big corp.
2. The server infrastructure:
If you worry about the server architecture, get a managed dedicated server. There's probably not really a guarantee that this system is set up perfectly, but there might be trustable hosters that know what they're doing.
- Keep everything up-to-date (apt-get update, apt-get upgrade).
- Don't install software you don't know or can't trust
- use SSH only with public key authentication, remap port 22 to something else
- use SFTP + SSL for your website
- don't include any third party JS-software that messes with your website. Personally, I don't even include Google Analytics. I self-host everything and install only stuff that is trustable.
- don't use the root user if not necessary
There's this "myth" going on that every system is hackable. This is only partly true. You can't do anything against 0-day hacks, but if you protected your web app against XSS, SQLI etc. and your software is uptodate, there's hardly any chance that you get hacked.
What makes every system "hackable" is the human factor. The secretary that gives out her password to some guy on the telephone who pretends to be the technical administrator of server X. Are you likely to get social engineered? Probably not.
The big picture here is that three customers' data was compromised -- customers in this context means entire platforms using Zendesk for support, not users. If the customers were, say, WePay, Box.net and OpenTable (random companies taken from their portfolio), this is potentially hundreds of thousands of users.
Also, it is a big deal because (as a former support person I know this), users often send in sensitive info with their support requests: SSNs, full credit card info with CVV data, date of birth (yes, sometimes all in the same message).
It doesn't sound like the investigation is complete, so we really don't know what was exposed. If this was a sophisticated intrusion, the attackers may have covered the evidence that a lot more was taken, but just didn't quite get it all cleaned up.
What a short-sighted comment. Dig a little further down. If this is their reponse to an issue that affects three customers it sends a message that every customer is important. Their reponse makes me, a potential customer, trust that they take these matters seriously.
Obviously you can also take the opinion that this should have never happened and question their competence and security. I personally weigh their response and transparency more than the issue itself, but it may seem easier since the impact to overall customers was relatively small.
Their response seems to have been handled well and may even generate some positive PR. That may change if it turns out to have been one of the recent Rails security flaws.
My thoughts exactly, lol. The headline is truly hilarious when you consider the typically quite spectacular number of compromised accounts reported in most other high-caliber incidents.
For the record, when I wrote the above comment, the headline of this thread was "Zendesk was hacked, 3 customers affected". It has been changed since, without due notice of course. I wonder why...
Now, my comment looks like I'm a nutjob and have a personal gripe with Zendesk or something.
That the three customers were Twitter, Pinterest and Tumblr didn't come out until later as well.
The original title still makes perfect sense. Maybe "3 customers" made sense to me because I'm already familiar with Zendesk, but it immediately struck me as a big deal.
Seconded. The sheer fact of the matter is that the state-of-the-art on offense is outpacing the state-of-the-art on defense. These guys didn't expose any password data, and evidently, not even the contents of the support cases (just the subject lines). That's small potatoes.
I'm curious why you picked tax filings as an example of information that would be catastrophic to leak.
Here in Finland tax information is public and it doesn't seem to be that bad. As you guess, yellow papers embarrasingly make yearly rankings of rich people and they lose some anonymity. But anybody that was in any way interested in their life likely knew that they were rich.
I don't say that this Finnish practice of having public tax information is a good one, but it just shows that it's not the end of world.
Leaked Tumblr support information that allows associating Tumblr blogs to email addresses can cause a similar level of agony that leaked tax information.
>I'm curious why you picked tax filings as an example of information that would be catastrophic to leak.
I'm assuming that the original poster is from U.S. One reason why tax filling are considered sensitive is that they include the individual(s) social security number (SSN). This number is meant to be private and, since it's often used as an identifier, it's frequently used for identity theft. See the wikipedia article below.
Just alike in many countries companies fillings, including for SMEs, are available to everyone.
It's very convenient before you start contracting for someone to see if they look like a legit business and to see if they do any kind of volume (and are in the business since a few years).
For the last 2.5 years, we've used a popular service called Zendesk to store, organize, and answer emails to Tumblr Support. We've learned that a security breach at Zendesk has affected Tumblr and two other companies. We are sending this notification to all email addresses that we believe may have been affected by this breach.
This has potentially exposed records of subject lines and, in some cases, email addresses of messages sent to Tumblr Support. While much of this information is innocuous, please take some time today to consider the following:
The subject lines of your emails to Tumblr Support may have included the address of your blog which could potentially allow your blog to be unwillingly associated with your email address. Any other information included in the subject lines of emails you’ve sent to Tumblr Support may be exposed. We recommend you review any correspondence you've addressed to support@tumblr.com, abuse@tumblr.com, dmca@tumblr.com, legal@tumblr.com, enquiries@tumblr.com, or lawenforcement@tumblr.com. Tumblr will never ask you for your password by email. Emails are easy to fake, and you should be suspicious of unexpected emails you receive.
Your safety is our highest priority. We're working with law enforcement and Zendesk to better understand this attack. Please monitor your email and Tumblr accounts for suspicious behavior, and notify us immediately if you have any concerns.