“Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.”
Dumb question but what's the difference between the two? If the underlying config is broken then DNS resolution would fail, and that's basically the only way resolution fails, no?
My speculation: 1st one - it just DNS fails and you can repeat later. second one - you need working DNS to update your DNS servers with new configuration endpoints where DynamoDB fetches its config (classical case of circular dependencies - i even managed get similar problem with two small dns servers...)
DNS is trivial to distribute if your backing storage is accessible and/or local to each resolver, so it's a reasonable distinction to make: It suggests someone has preferred consistency at a level where DNS doesn't really provide consistency (due to caching in resolvers along the path) anyway, over a system with fewer failure points.
I feel like even Amazon/AWS wouldn't be that dim, they surely have professionals who know how to build somewhat resilient distributed systems when DNS is involved :)
I doubt a circular dependency is the cause here (probably something even more basic). That being said, I could absolutely see how a circular dependency could accidentally creep in, especially as systems evolve over time.
Systems often start with minimal dependencies, and then over time you add a dependency on X for a limited use case as a convenience. Then over time, since it's already being used it gets added to other use cases until you eventually find out that it's a critical dependency.
I don't think that's necessarily true. The outage updates later identified failing network load balancers as the cause--I think DNS was just a symptom of the root cause
I suppose it's possible DNS broke health checks but it seems more likely to be the other way around imo
I don’t work for aws, but a different cloud provider so this is not a description of this incident, but an example of the kind of thing that can happen
One particular “dns” issue that caused an outage was actually a bug in software that monitors healthchecks.
It would actively monitor all servers for a particular service (by updating itself based on what was deployed) and update dns based on those checks.
So when the health check monitors failed, servers would get removed from dns within a few milliseconds.
Bug gets deployed to health check service. All of a sudden users can’t resolve dns names because everything is marked as unhealthy and removed from dns.
So not really a “dns” issue, but it looks like one to users
DNS strikes me as the kind of solution someone designed thinking “eh, this is good enough for now. We can work out some of the clunkiness when more organizations start using the Internet.” But it just ended up being pretty much the best approach indefinitely.
I actually think the design of DNS is really cool. I'm sure we could do better designing from a clean slate today, especially around security (designing with the assumption of an adversarial environment).
But DNS was designed in the 80s! It's actually a minor miracle it works as well as it does
This is just a reminder to take a backup of your cloud photos and store it locally (or other cloud provider). For example, with Google Takeout you can get a ZIP file with all your photos and videos from Google Photos.
I hate how Google Photos seems to no longer have the ability to auto sync to your computer. When it was integrated more with Google Drive you could do that, and then my daily automated backups would pull them. That way I had them in Google Photos, my computer's hard drive, and my off-site backups, all automatically. Now my only option is to periodically do a manual Takeout, as far as I can tell.
So... what do y'all actually do with your hundreds of thousands of photos? I just bulk delete everything older than a month when my phone is low on storage. Do you ever actually go back and look at any of the hundreds of thousands of pictures? When do you do this?
When you have kids, and when they turn into teenagers, you'll want to remember them as they were before.
;)
My daughter loves looking back at home videos of the family. So embarrassing, so beautiful, so many little things about their personalities that would otherwise be forgotten.
Just a few hours ago I was checking backups and stumbled into a nest of folders that had some more home videos in it, and of my son just being his quirky little kid self. Gives my life meaning, shows me I did stuff ok.
I have photos backed up on Google Photos for the last 10 years. I go back to find old photos all the time. There's different reasons, sometimes I'm just feeling nostalgic for a time in my life and want to see photos for them. Sometimes I want to find a specific photo of someone to send to them.
Here's a practical example of it coming in handy: back in university I moved to another city for an internship and took some framed photos I had on my walls. When I moved back into my parents' house when the internship was over, the nails were still in the walls and I wanted to hang them the same way I had before but couldn't remember which photo went where. So I went into my Google Photos, typed "photo frames," and was able to find pictures in my library with the framed photos in the background, and piece together which photo went where to get everything back to where it had been before.
All the time actually. I have monthly folders, automatically backed up by my phone to the NAS. Each month my wife and I sift through the photos and delete all duplicates and boring stuff and then we have around 150 photos per month that we sometime just have on as a slideshow.
Yearly, we create printed photo albums that we look at regularly and show to our kids.
But all this only works after lots and lots of hours of manual work filtering the photos.
Still, over the last 15 years or so for two people we have way more than 100k photos stored on the NAS.
I look at them a good bit. Having a good tool like Google Photos or similar makes it easy to jump back to the photos from that one trip a decade ago, that holiday together with the family no longer with us, the dreams I had for myself in highschool. I find it interesting to see the changes in my life objectively by looking at myself, the places, the people, and things around me at various points in my life. I find it helps me continue to keep perspective of my life in the same way some might keep a journal. It can help me jump back into the shoes of who I was 5 years ago, 10 years ago, 20 years ago, and reflect on who I am today and where I'm going.
If I don't have any real reference points, its hard to measure things. Its hard to improve things you cannot measure. And sometimes it just feels good to help remember the good times with people no longer with us.
The thing is also to delete photos you don't want to keep long term. Most of my photos are only synced when I am on my local wifi, so until then having a good hygiene of deleting badly taken photos right after taking them.
Having said that I have recently started (late december) to do a weekly printing session of my key moments of each weeks. Well there have been weeks in january and february I didn't print anything. I sometimes only have one or two photo per day/event but since I put the date on the notebook next to the photo and keep all other photos it helps me find back the whole event if I want to see them all. I am using smaller photo format than 10x15cm, mostly zink 2"x3" adhesive photos and kodak instant print 3"x3" from instant cameras that have a printer function.
I've been enjoying the iOS photos widget. Every day it surfaces new photos from the past, based on different themes. A nice 2 minute diversion each morning.
But you're right, otherwise I would never have time to dig back into old photos.
Fun party trick: look up old photos of people when you meet them after a long time and show it to them. Everyone is shocked/embarrassed. Extra points if you can pull up childhood photos of them.
I liked that the X-Files reboot kept the old credits. Mulder and Scully look like babies. Then the credits end and reality intrudes. Of course, I'd aged at the same rate (maybe faster) but it was still a shock.
I have no idea why you're getting downvoted. The icloud-photos-downloader is an amazing tool. I run it from a monthly cronjob to have a local backup in case icloud iexplodes.
Local photo library is a folder, disguised as a package. One of the folders store all the originals on the device. Copy that folder, and you're good to go. You can also get the full folder hierarchy, if you wish.
If you enable iCloud for the photo library, and even if you select store originals, I have found that Photos still offloads some pictures. It’s just not guaranteed.
I really wish Apple had a complete export button, similar to google takeout. Or at least a good CLI tool to get the original images one by one.
I rsync images out of Apple’s weird file structure that’s in the bundle. It’s horrible, but with ‘download originals’ enabled it works well. It’s particularly frustrating with large libraries as they occupy nearly all an SSD. For this reason I do it all on a VM that’s on a Synology.
I had a pretty big incident mid last year, which will turn into a dramatic blog post eventually.
I didn't lose anything I've yet noticed. Very happy with how well the backup strategy held up, although potentially quite lucky the second on-site backup was unaffected. Didn't have to dip into the complexities of the off-site chaos (which reminds me...).
One of the nice things about having an off-site backup is that, in addition to being offsite, it's presumably a completely different backup process so if the local backup wasn't working for some reason (and you didn't notice) you're probably still covered.
Having a spare hard drive that you occasionally sync up and drop off at a family member's house in addition to other backup strategies is indeed a decent idea.
However, I'm guessing that--if you're anything like me--you get lazy about refreshing it and, if something happens, you realize it's been a year since you did a fresh backup.
A cloud provider backup (as opposed to sync) is a good belt-and-suspenders cheap insurance backstop to local backups that you hopefully never need.
I have local Time Machine and Synology NAS but I also pay Backblaze a nominal amount. Companies were paying Iron Mountain large sums of money before there was a cloud.
You can also say: cloud servers are on fire and the house is on fire, have another backup someplace else. It's all probability.
The probability that your house will be on fire these days is low, and the probability that your computer/hard drive will get on fire is also low (since you will probably take it out, no one will leave his phone in a burning house right?).
We can say that the probability above is that low that it doesn't really affect anything. There are many other different parameters to add, and I use a bit of math here which doesn't always reflect on life, but you get the point.
please take a look and also note that Tabnine while being more secure has also continued to evolve. We also have a free option that we have stood behind for 3+ years.
https://www.tabnine.com/tabnine-vs-github-copilot
