[MUSIC]
>> This is Pragmatic, a weekly discussion show contemplating
the practical application of technology.
Exploring the real-world trade-offs,
we look at how great ideas are transformed
into products and services that can change our lives.
Nothing is as simple as it seems.
I'm Ben Alexander and my co-host is John Chidjie.
How you doing, John?
I'm doing very well, thank you. And how you doing, Ben?
Very well. It's a beautiful zero-degree morning in Ohio.
And it's been a stinking hot, horrible day here.
I have to unfortunately report. So, yeah, opposite ends of the spectrum, but then I guess we're on sort of opposite ends of the world.
Exactly.
You get that, I suppose. But anyway.
It all evens out during this show. It's a beautiful spring afternoon.
If only that was a yes, I wish that was the case but anyway.
All right, we have a lot of thank yous this episode so I'm going to sort of break it down
quickly into two pieces which is first of all, all the people on Twitter that have had
shout outs to us about enjoying the show which I really appreciate and so the thanks are
to Solid Mail. I can't actually find out who Solid Mail's name is, but it's a company account,
so thanks to them. Daniel Gnomes, Andrew Escobar, Matt Volk, Brandon Gribbon, and I'm going
to struggle with this one, apologies in advance. It's a Polish name. Radislaw, oh my goodness,
Pietrzewski. There you go. Apologies if I minced anybody's names there, but thank you
very much for that. Much appreciated. Also, however, just as importantly, some people
have written, been very nice to write some articles about the show, some blog posts saying
different things that they've liked about it, which I, again, really wholeheartedly
appreciate. And so thank you to Scott Wilsey. He included the list, the show in his list
suggested podcasts to listen to and we're in some pretty good company there. So, thank you for that.
There's a link in the notes for that. Thank you also to Flo and that Twitter handle is Flohei.
I'm not sure what their full name is, but they wrote a nice piece about the show on their site.
Again, link in the show notes. But also a very special thank you to Marco Armand who said some
wonderful things about the show on his site, Marco.org. Again, there's a link in the notes for
that. I've been following Marco's work for a bit over five years now, when Instapaper first came
out mid-2008 through when he was doing Build & Analyze podcast on 5x5, also his work with the
magazine, and most recently Accidental Tech podcast, and the soon-to-be-released and much-anticipated
overcast podcast app that he's working on. So, for me, it's a very different feeling
to have someone that you've followed and respected for so many years appreciate what you do.
So, again, very special thanks to Marco for listening and enjoying the show. So, thank
you.
Marco is the reason I'm drinking strong yet not disgustingly bitter coffee this morning.
So, this show wouldn't be happening otherwise.
So, there you go. So, okay, fair enough. If I drank coffee, I could reiterate but unfortunately,
I'm drinking water. So, anyhow. Cool. All right. So, the topic today that I'd like to
go through is something that we discussed on one of our previous shows and we said we
would get back to it which is what we store digitally, specifically digitally, where,
how and why. And I can't remember the exact details of why this came up, but I wanted
to break this down into two pieces. And the first piece is filing of information and filing,
I know, hey, I'm not so bad filing, isn't that exciting? But nevermind. It is, oddly,
it's one of those things I find fascinating because it's so hard to get it right. And
what we keep and on what will be the second piece. So, dive right into the first piece which is
filing. So, when you're filing anything of any kind, there's more than one way to skin a cat.
You can go with alphabetical system or a numerical system which is generally arbitrary.
You can go with the chronological system if you like. The date that something occurs,
just chronologically. In some cases, you might have other ways of bringing down like geographical
location. So, this information relates to this geographical location. So, in engineering
parlance, maybe it's this particular site is up in the mountains in this location. So,
you keep all of that stuff filed together. If in a more personal context, I took a bunch of videos,
I took them all at Six Flags Magic Mountain, well, I might have them all over in one folder
or subfolder or whatever. So, you know, geographical potentially and obviously also by subject. So,
you know, Ted's birthday or, you know, the graduation day or, you know, whatever you
might have photos, videos, let's say, you know, or documents, you know, building the new house
and so on and so forth. So, subject-based filing systems. However, the problem with filing systems
of any kind is whether or not in a day and age where we can search through things so easily
with indexing systems and powerful computers we're at a point where you can search for the
entire contents of your hard drive in not a hell of a lot of period of time really and when you've
got search power like Google to search the internet as well and you know if you apply that to a desktop
scenario you have spotlight in OSX for example, it's got to the point where it's almost become
quicker to search than it has been to file things away by any system really. The funny thing is
there was a study done by IBM, an IBM research paper, there's a link to this in the show notes,
it's a bit dry, but it's interesting. And what they did is they did a study of email refinding.
So, they had 364, I think it was, people that partook in the study. And the concept was they
wanted to see whether or not there was a time saving to be actually measurable by filing your
things in your email, in your inbox, into folders, subfolders, categories, however you want to choose
to file it. System not important exactly, but some kind of filing an organization versus someone who
was allowed to search and just dumped it all in one folder. And they found that it was marginally
faster to search, but when you included the amount of time that it took to file and organize in the
first place, search was the clear winner. And once I read this paper, I read this paper
a few years ago, when I read that, I immediately threw my hands up in the air. There's a part
of me that's been trained through years and years of just habit. Oh, I'll put this in
this folder or that folder. My email inbox is broken down into different projects that
I've worked on. My hard drive, like when I store things, my photos are in a directory
called photos. Videos are in videos and so on and so forth. And I break down videos by
by years and so on.
But why do I do that if I've got something
that's fully tech searchable like an email?
Because search is now essentially so much quicker.
I know when I'm doing it, what's the point?
Because they've shown that it doesn't actually
reduce that amount of time,
at least when it comes to email.
But the other concept that's interesting with filing
is the method by which you break it down.
And the problem is every method has its drawbacks.
So if you go alphabetical,
that assumes you have knowledge of the content
of the subject or what it is you're looking for alphabetically.
So the problem with alphabetical,
let's say you're looking through photos
and each of your photos, you've got a file name,
the file name is a description of what that photo is.
And if you say, you know, Ted's birthday party
or birthday party at Ted's,
Well, if you don't know what letter it starts with,
B, birthday party at Ted's is much, much higher
than Ted's birthday party,
which is T close to the bottom of the alphabet.
And that's the problem with alphabetical
is that it's arbitrary insofar as you have to know
what the expression that you're searching for,
the letter that it begins with,
or the item, the method by which it's been organized.
The problem with a numerical system, obviously,
is it depends on your recollection of when things happen.
Because in a numerical system,
typically you'll number things based on,
like for example, if I'm working on a quote,
the quotes may come in in a certain order.
So, oh, this will be quote 21, quote 22, quote 23.
And if you don't remember the actual,
what the number relates to,
and you don't remember that,
and sometimes it's not chronological.
So a tender will come in and I've got to work on it,
and I won't actually assign a number for a week.
Whereas one that came in technically afterwards
got assigned a number first
'cause it was quicker and easier to do, let's say.
So numerical systems, you've got to be careful
what sort of methodology you use
for dishing out the numbers
'cause it can end up being a bit arbitrary.
Geographical is all well and good,
depending on how close you want to go down in detail.
So I mean, if you want to say,
okay, it's geographically broken down.
So right now I've got a site and this site,
or this house with a certain size yard.
Okay, that's great, but I want to look at all of the photos
that were taken in this specific room.
It's like, okay, well, geographically,
I didn't go down to that level of detail.
So too bad, so sad.
You still got to go through something manually
and search manually.
Let's just say.
Chronologically again,
do you remember what date things happened?
Half the time we don't.
Human beings, we have a terrible memory with respect to time
our brains compress time and we perceive it in different ways. I was talking to my wife recently
about a medical test that I had done a few years ago and I said, "Oh, it must have been about five
or six years ago," and she pulled out some of the forms and x-rays and the date was two years ago.
And I was convinced it was five or six years ago and I'm like, "Oh, okay, okay then." But it's been
a busy two years, right? A lot happens and you tend to think it's been a lot longer than it
really has been. And yeah, that's the problem with perception of time. So, chronological systems
aren't perfect either, although they're probably one of the better options. So, honestly, the
best way is to break it down by subject. And a lot of people do that and they'll say, well,
this has to do with invoicing, this has to do with accounts receivable, this has to do
with whatever, when I file something. But the problem is, no matter how you slice it,
eventually, you will end up with a category called general or miscellaneous. And it's
It's maddening because I remember when I went to work for a company and I went when I went
in there, they said, "Oh, the jobs are all under this directory.
If you're looking for other information, it'll be under miscellaneous."
And I'm like, "Okay, that's a great filing system you guys have got there."
Because when I worked in Canada, I worked with someone who was particularly very, very
specific about her filing system and you did not mess with it.
put the wrong thing in the wrong slot, she got very upset. So, yeah, doc control people
and there are other people you don't cross. But anyway, so in terms of that, it's actually
a really cool XKCD comic. I don't often link to that but I do actually read this stuff
all the time. It's really hilarious. And so, the whole miscellaneous category thing, have
a look at that link to the XKCD comic. I'm not going to try and describe it because this
This is an audio show, not a visual show.
It's a Google suggested search here.
So, yep.
OK.
Yeah, it's yeah.
Is that's I don't want to say they can look.
That's what I'll say.
That's pretty much what it looks like in here.
So, OK.
There you go. So, yeah.
And I agree.
Yeah, because inevitably, no matter what system you choose, there will be a limitation.
So, the best filing systems will have multiple methods by which you can find the information
you're looking for. Now, in the old days, the bad old days where everything was manual,
you have a piece of paper, a physical folder, a filing cabinet, a row of filing cabinets in a room,
obviously, that was more difficult. But now, with everything being searchable,
do you really need to file it that way, especially when you've got multiple attributes
with relation to a file. So anyway, we'll get to that in a minute. One of the interesting things I
found when I was doing some of my research on this, I was looking for links specifically about
the IBM research paper and one of the ones that I came across in my searches was how Mac experts
organize their files. That's the title of the article by Lex Friedman on Macworld. And I
I occasionally read Macworld and this particular article
has interviews, one, two, four people.
So you've got Casey List, Federico Vatici,
John Siracusa and Katie Floyd.
And all of them had different ways
in which they organized their stuff.
They had their own, essentially their own filing systems,
digitally, of course, for their information.
And they were quoting all sorts of helper applications
like Quicksilver, LaunchBar, Alfred,
and one of them was even using Hazel.
So for some file renaming.
- I use Hazel a lot for that actually, yeah.
- Oh, cool.
Well, there you go. - I'm weird,
but yeah, for a lot of the episodes,
all these different shows, I have a little workflow,
so I drop certain things in a folder and things happen.
So it's nice. - Nice.
Well, there you go, that's nice.
And I'm glad that works for you.
See, I look at all of those things and think,
well, it's kind of, it's solving a,
How do I put this?
The organization of individual files and tools
in order to find them more easily,
it's something that is a very personal thing to solve.
But I guess where I'm going with this discussion
is to think of it from a point of view of,
is the effort put in in that workflow saving you time?
And if you need to go to some advanced tools
like Hazel, for example,
well, first of all, that puts you in the power user category.
I was hoping to sort of look at it more
from the average user point of view
and whether or not you should be bothering.
So we'll see, let's stack up the cards on each side
and see how it ends up.
So bear with me, hang in there.
Okay, so as I said before,
the great thing about digital storage is,
well, quite frankly, you have a time and date
of accessed, created, modified.
So for any file, you have those attributes
typically in most file systems,
as well as obviously the file name and the file type.
So that's a lot of information for you to index,
search, cross-reference by.
There's quite a bit there.
So a lot of people will take some of the attributes
and they'll replicate them and they'll say,
"Okay, well, I'm gonna embed the file creation date
or a chronological system."
And I'll put that at the beginning of the file name
or the end of the file name or at some point in it.
And what that'll do then is that'll say,
okay, well, now I've got it in the file name,
it's easy to search maybe, I don't know,
or it'll sort more easily in a large directory listing.
That's all well and good,
but I suppose the question is, isn't that duplication?
I mean, if your time and date access credit modified
isn't enough to give you that.
And I do realize if you clone a hard drive,
sometimes there are some operations where you clone restore,
it can mess that up.
And I guess that is an issue, I guess.
by putting that in the file name, then you sort of guard against that. But in any case,
if it was grouped by a subject that was more natural to search by, so let's say all of the
audio clips from this show, let's say, from Pragmatic, do you really need to know the date
that they were done or do you just need to know the episode numbers? Right, I need the episode
numbers and I need the time, but I like the timestamps. Sure. But yeah, I don't need the
the date doesn't matter at all. I just need them in chronological order relative to each
other. Right? Beyond that.
Yes, okay. Yeah, exactly. And it's that sort of thinking, but a lot of people would, I've
come across it, you use Hazel for, well, Federico's example, and I'm aware of one other person.
So I say all of the people that you Hazel that I know, that's two, now you use it, that's
three. There you go.
I use it for like two things though.
So statistically...
They move some stuff around.
Let's just say it's statistically questionable.
Okay, so we'll leave it there.
You could say it's anecdotal if you'd like.
But anyhow, in joke.
But the point is that that information is embedded in the file system information for
that file.
So, you don't have to re-duplicate it if you don't want to.
Right.
So, anyway, and that's one of the wonderful things about digital that is so much harder
when you've got an actual printed document.
So, one of the things about file types that I find interesting is that file types are
kind of a new level of granularity, I guess is the right word.
Because when you pick up a book, you pick up a book with two hands, you open it and
you hopefully open your eyes.
And as you start reading, you'll say, okay, I get that this is text.
I know how to read English, French, Spanish, German, whatever language might be written
in and hopefully, you know how to read it and then you'll say, okay, here's a picture,
it's a pretty picture or it's a chart or whatever it is.
But there's nothing on the cover of that book that tells you explicitly that this book has
a chart in it or this book has a whatever in it.
Yeah, it's just, it's a book.
You got to pick it up and open it to interrogate what's in it.
That sounded weird, didn't it? Interrogate. Anyway, you know what I mean.
It's the right word for it though.
I guess it is. I guess it is. But I just conjured up an image of someone like sitting across
a desk from a book yelling at the desk, yelling at the book, interrogating it. But you know
what I mean, right? Programming, sorry. Anyhow.
We will break you.
Yeah, something like that.
We will speak.
Oh dear. Okay. And if books start talking to you, then I'd suggest you should adjust your
communication. But anyway, unless it's an audio book, right. Okay, I need to get back
on track here for a second. Okay, so however, with a file, it'll have a type that is associated
with it. So it's a Microsoft Word document, unfortunately, or it's a pages document, which,
if you love Microsoft Word, you might say also, unfortunately, but irrespective, you
You might have a PNG file, which is obviously an image.
If you know your file formats or a GIF file, I'm not going to pronounce it because I might
get in trouble, a JPEG file, any of those other different, you know straight away, okay,
they're an image.
And you know that before you open it.
I suppose a book is more analogous to a Word document because in a Word document, it could
be a mix, it's a mix, could be a mixture of video, it could be a mixture of images as
well as text.
So, but then again, at the same time, the difficulty in the digital landscape is that
there are some file formats that will not open in all the same different kinds of software.
But the point is more that the type tells you a bit more about what is actually in it
than an actual book would.
So again, what I'm getting at, the point is the difference between the physical world
and the digital world.
So in the digital world, you don't have to deal with the issues of a book.
To understand what's in a book, you got to open the book.
Sometimes there'll be a synopsis at the front or an index or a table of contents.
You got to open the book, flip through it, see what you can see.
And even then, that might not be a good indication of if it's got what you're looking for in
there.
So realistically, the file type is critical and very, very, very useful.
So in terms of a filing system based on its file types, that's quite appealing and something
that's not as easily possible with physical pages.
So okay, I just want to quickly talk then about cataloging systems.
And you may think that's a bit odd to talk about, but the reason is that sometimes file
So, for example, if I've got Spotlight on,
I've got a Mac, I've got Spotlight built in.
I want to search on something, that's great for any volume
that's currently connected to that computer.
And there's a Spotlight index on that drive,
and it searches through the Spotlight index,
and it's all hunky-dory and relatively fast.
Hopefully anyway.
However, if you have a drive that's not connected,
it won't find it.
So a cataloging system sort of maintains an index, but it doesn't have to be connected to the system.
So for example, let's say you back up a bunch of information on an external hard drive or on a
bunch of CDs, DVDs or Blu-ray discs. Well, if you have cataloging software, you can actually
use that cataloging software to have a dedicated index or database of all the files that you've
got backed up somewhere. And I guess one of the reasons I bring it up is because when
you are looking at how you organize things, if you were to just take all the files you
ever wanted to keep, photos, videos, whatever, you could just whack them into a single directory
and then burn that across a bunch of Blu-ray discs if you wanted, or back it up to a single
external hard drive and put it on a shelf. And if you've got a catalog of that that's
on your computer live, you don't need to maintain index, there's no spotlight search for it
because you've got the software with the index on it, so the catalog on it. So you simply
search through the catalog and say, "Oh, okay, what I'm looking for is on this drive." And
yeah, you may say, "Well, why would you want to do that?" I guess the point in the case
of a hard drive is you don't really want to be running that hard drive all the time. So
you want to catalog it and leave it turned off on the shelf. You want it sitting cold,
you don't want it running hot all the time because if it's running spinning, then that's
its wear out mechanisms or some of its wear out mechanisms. So when you've got CDs optical discs
obviously when they're not in a drive you can't read them. So cataloging sort of is an extension
of the search capabilities for your data. So the thing that I find a lot of people do when they are
doing their filing systems is that they spend a lot of time building up this directory structure
and then they'll put it on a hard drive or they put it on burn to a bunch of
disks and then have no idea which it's on.
So, you know what I mean?
It's like...
Yeah, I'm thinking about, I'm looking at my portable hard drives right now and thinking what's on what.
Yeah, and how do you find it?
It's like, well, OK, so let's assume that because I do all my backups to Blu-ray disks and I've copped a lot of, you know, flack for that one.
But and I'll explain some of my reasons shortly.
but irrespective, if you want to call it a hard drive,
that's something not connected and powered on
or a bunch of flash drives,
or it's a bunch of optical media,
irrespective of the actual physical device
you're storing your information on as a backup,
if you are backing it up or cloning or whatever you're doing
you need to know what is on what device.
Because a device works for hard drive
but does it work for optical media?
You know what I mean?
So if you have 10 external hard drives that are sitting there cold with all your backups
on it for the last six, seven years with the data on them, how do you know which one's
got what on it?
And if you don't catalog it, how are you going to know?
You got to go through and search them manually.
And you know what?
I was insane for a, well, I was about to say I was insane for a short period of my life.
I'm not sure that's changed.
But anyway, it's something that I used to do.
I used to go through and I would go through a hundred or thereabouts CDs, DVDs and a few
Blu-ray discs in the early days until I found what I needed.
And it was crazy.
And then I got into a cataloging system.
And on the Mac, I use NeoFinder.
There's multiple examples out there.
And on Windows, if you are a Windows user, one I played with briefly was Jantibus CD.
freeware one I think from memory. There's quite a few different ones out there.
Not necessarily recommendations, do your own research of course, but
yeah there's not much to them. You pop the drive in, point it to the drive or the CD
and say you know back it up or you know like create a catalog of it and the better ones will
even give you a snapshot. So for example you got a bunch of photos on there it'll take a
very low resolution you know snapshot of the image you know just like four kilobytes or whatever.
like a contact sheet. Yeah, exactly. So you can look at it and search for images and have a flip
through them say oh that's the one I want and all you have to do is give it a name that you
cross-reference. So I've done the simple thing and simply had an arbitrary, I call it a numerical
filing system if you want for the actual blu-ray discs but it goes to A to Z and then AA to ZZ.
So, and all of it fits into a single,
you know, like 120 slot CD case,
essentially that I've had for the last,
geez, Canada, 13 years.
And, you know, one case got all my stuff in it.
You know, just general backups and stuff.
But anyway, I'm focusing too much on what I'm doing.
The suggestion is that a cataloging system is very helpful
because, and don't bury things in ISOs.
Yeah, a lot of geeks will say,
well, I'm gonna do a super duper clone
or a carbon copy clone of my hard drive.
That's fine, you know, but geez, if you're gonna do that,
that's for restoring purposes.
That's not for actually backing up your data.
So, you know, and this is the other thing.
- It's not very, yeah, it's not very convenient
to try to, I've tried it.
It doesn't work that well.
- No, it's terrible because what'll happen
you end up with this 80 gig, 150 gig image file,
depending on the size of your computer and so on.
And then you'll end up with two or three of them
and you'll say, okay, well, I was looking for this file.
I don't know which one it's on.
And let's say I've updated or reinstalled
my operating system in the meantime.
So my time machine backups, the continuities,
it could potentially be broken there.
If that's assuming I'm running a time machine backup,
mind you, if you're doing carbon copy,
cloning or yeah, super duper cloning,
then obviously I would expect you also be doing
time machine backups.
But I guess my point is that if you bury files in an ISO
and then you back up the ISO,
you've got no visibility of what's in it.
- Right.
- So yeah, there's something to be aware of.
So I guess that's the filing systems
that I just wanted to sort of touch on
the first part of this discussion. And I've sort of, or I've already started working on
the next bit, which is exactly what we keep and where. So, obviously, every now and then
I've sprinkled in this discussion, I'm talking about business stuff or professional stuff,
and that is going to be highly variable based on what you do. So, for example, in my line
of work, there's a lot of drawings, there's a lot of contract documents, there's a lot
of there's a lot of yeah, lots of different documents of different kinds. And obviously,
that's going to have some unique requirements that are specific to what I do. And I sort
of mentioned some of them before like quote numbers and so on, or even job numbers, whatever.
And that's not what I want to talk about. It's to our personal stuff because that's
what is going to be most useful to people. And one of the things that I'm trying to do
in this podcast is that people walk away from it with some ideas about things they could
have a, that they could try, that they might be able to apply, find useful and do something
with.
And this is one of those areas where I think that people need to think about it a bit more.
Well, and that's the thing, you know, with when you're working with other people, if
you're working at a company, um, or kind of any sort of big project, you need, I think
that, you know, the searching versus filing goes out the window a little bit because you
don't know if you're going to get hit by a bus.
Um, and I guess that same thing applies when you're dealing with family stuff too, which
we're probably going to get into. But yeah, I feel like I stress a lot.
This really is a perk. You know, these are,
if you try to apply the same sort of thinking and consideration that you would
in a, you know, in a corporate setting, right inside the enterprise,
you are going to probably have a hard time with your own stuff.
Absolutely. The rules that are set out in larger corporations,
my experience are very rigid.
You got to do what they say and they say, put on this drive.
This is our file structure and that's it.
And when you've got the whole departments, I've worked at companies where they've got a whole
department just simply called document control.
Right. And if you set a foot wrong, you will know.
They will find you and they will, yeah, they will do things to you you will not forget.
Well, we had just at the small, the design shop we ran and we had to be really, really,
I mean, exceedingly strict with people about what we're talking about at the time.
Yeah, a lot of some pretty wacky, weird, you know, filing systems and naming
structure and stuff like that. But he sent the wrong thing to the printer.
I mean, thousand many, many thousand dollar mistake.
And oh yeah, yeah.
That kind of that control, the document control is vital.
I mean, it was the most important thing.
It was the brakes on the car.
Absolutely. Yep.
So I guess, if anything, the message I want to get across
from the first half of this discussion is filing is overrated.
I'm sorry, but it just, it is for digital information.
And when I say that, I mean, for most digital information.
Between a standard file name, a file creation date,
and especially now with photos where they're being geo tagged,
a lot of photos are being geo tagged now.
People take them on smartphones, they've got GPS built in.
And even if they're within 100 meters or 200 meters, sorry,
three or 400 feet of your location,
it's good enough to know you're at Disneyland.
It's good enough, you know.
And yeah, that kind of geographical
is all the information's automatically built in for you.
You don't need to add it, you don't need to file it.
It's just, it's there.
So what are you filing it for?
And there's lots of software out there
that can read that information and organize things for you.
I'm not gonna extol the virtues of iPhoto specifically
'cause it's become somewhat of a lumbering dinosaur,
but, or even Aperture, which is also seems to be
sort of the neglected stepson or something of Apple.
But nevermind, the point is that there is software out there
that can do a good enough job of it for photos.
But anyway, irrespective of that,
the point is modifying file names
and putting emails into sub folders
and grouping things together and tagging things
since the latest thing in Mavericks.
Honestly, is it really worth the effort
based on everything that I've seen,
everything that I've experienced
and that the study that IBM did,
I have to think that search is simply quicker and easier.
And if you add a cataloging system
for your offline stored data, then you've got both.
You've got everything you could need.
And, you know, that's, I guess, that's sort of how I see it anyway.
However, that doesn't address the issue of what on earth you do with all this data.
Because the problem is we're accumulating so much data all the time now.
We're taking so much video and it's not just low res video now.
This is 1080p.
I mean, my iPhone has been, iPhone's been taking 1080p video now for a couple of years,
I think.
And you know, any before that 720p, these file sizes are not insignificant. The hard drive on my,
so I call it hard drive, the SSD on my MacBook Air is 256 gig. And you know, it's got a combination
of music and movies and TV shows and home videos and photos. And there's so much stuff on there
that's just personal stuff, memories of the kids and you know, places we've been on holidays and so
on. That's the majority in terms of size these days that you're going to find on any individual's
computer. It's not documents. Yeah, you'll have some official documents. I mean, yeah,
okay, I had to write a letter to the bank for some reason or maybe I've got some receipts
because sending soft copies of receipts has become quite popular these days and even bank
statements up to a point, some of them you can download. So, you can download your bank
statements and save them in soft copy. Keep that on a lock and key. But you know, you maybe do a
family budget. But all of that takes up hardly any space at all. When you think about it, it's really
only a few hundred kilobytes here and there for each of those files probably. If they're PDFs,
maybe it's a meg or two here and there. One photo at full res on an iPhone with a mixed background
of JPEG compression, you're looking at five meg for one photo. So that's where the actual problem
is in terms of the focus of storage.
And the thing is, you know, this is the, with the cliche of people, your house is on fire,
you run out of the house, the fire department's there, and they're asking why you're running back
in. You're going back in to get not your bank statements, not your computer, you're going to
get your photo albums. And so we have this, we're feeling it really acutely right now with, you know,
we have a young daughter, I mean, she's four months old, and the amount of video we've produced in the
in the past four months and photos and everything is insane.
- Oh yeah.
- I mean I think we drove a few smartphone sales
for people to get on our photo stream.
So, but your behavior changes so much
and this stuff is, you know, actually I get it now.
Right, didn't quite get why people were so concerned
over their iPhoto libraries or any of this before
but yeah, I just instantly kind of understand,
okay, I've got to work a system out for this.
- Absolutely.
And what I want to walk through
the different possible solutions.
Now, before I even begin,
I'm not here to talk about
time machine
versus
carbon copy, clone or super duper
versus an offsite storage system like Backblaze
or crash plan.
I'm not talking about any of that.
I'm talking about the media that you would choose locally
to back things up on.
And I'll tell you why.
First of all, I think that that discussion
has been well and truly covered previously
on other podcasts.
And the ultimate solution seems to be to pay money
to someone offsite, 'cause offsite backups are,
from a professional point of view,
that's exactly what I recommend my clients do is,
you want offsite backup
because you've got geographical separation.
If there's an incident at one location,
it's unlikely the other location
physically will be affected.
And it makes sense to have more than one
alternate physical location for supercritical information.
And that's pretty standard stuff.
But we're talking about people that,
let's assume for instance,
that they're not geeks that haven't got,
that they don't have files or they're like me
and they're stuck on ADSL1 behind a rim.
in the sticks, quote unquote.
I mean, I'm not really,
but unfortunately from a telecoms perspective,
I'm in the sticks.
So, you know, not everyone can use
an offsite storage system with an automated backup.
It's just not available.
And it's gonna be a very long time until that is available.
And even so, that has an ongoing cost.
Do I wanna sign up for ongoing costs?
Yeah, and then of course,
there'll be the people that say,
well, you know, I have lots of valuable data and so on.
And you know what, there's actually ways of handling that
without doing the offsite storage.
Some of them are more or less safe than others,
but in any case.
So I don't want to really talk about that.
I want to talk about locally.
And frankly, most people just do it locally.
That's one of this discussion is aimed at,
is people that are quote unquote normal.
Maybe they're not our listeners.
I don't know.
I would like to think that I'm not the only person out here
that is a geek that doesn't have offsite backup,
but maybe I'm wrong.
I guess we'll see.
So if you're going to store it locally,
your options are essentially three broad types
of technology.
You have magnetic storage, you have optical storage,
and you have a flash storage.
And we'll just quickly talk a little bit about,
you know, which are the pros and cons of them.
So magnetic broadly falls into two kinds of categories.
You've got the, essentially what is,
I would call a flexible media,
and you've also got the non-flexible or rigid media.
So rigid media being a hard drive,
flexible media being any kind of tape or floppy disk,
for those of you that remember floppy disks,
And it's funny, I showed one of my kids a floppy disk the other day.
I still had a five and a quarter inch disk in the house.
And I said, what on earth is that?
Wow.
And my son piped up and said, oh, I've seen a picture of that on some software we used at school.
And he was referring to the toolbar button.
Well, that's what that is.
It's an icon now.
I know.
This is funny.
Someone made this.
It's you could probably sell those as kind of kitschy 3D representations of icons that no one has any idea what they are anymore.
That's terribly sad, we're showing our age. But anyway, okay, irrespective of that, the
point is that flexible magnetic media has all sorts of problems. I mean, the media itself
can be physically damaged by being, having too much tension applied to it, it can stretch.
It doesn't like humidity, it doesn't like temperatures, it can delaminate as in the
magnetic, the particles can actually partially delaminate from the surface of the plastic
on the tape, it's just dodgy. And of course, the obvious one, which is you pass a strong
magnet nearby and it wipes all the data, flips all the bits if it's close enough and strong
enough, it will. The other one, of course, is that eventually magnetic tapes will depolarize
and they'll lose their data. Just naturally, they will. And the thing is, with most magnetic
media that's flexible, you're looking at about 10 to 20 years for their lifespan.
And there's a good link, there's a good Wikipedia entry that I've linked to on the show notes.
Please feel free to read that.
It goes into a lot more depth than I am just sort of skimming over the key points.
So having said that, 10 to 20 years, you can get professional grade backup tapes that are
guaranteed for 35 years.
And that's kind of obvious that such a market exists, because you've got people with massive
servers and they want to do tape backups.
And then, you know, people paid lots of money, well, people paid money to do all the backups
and everything and change the tapes and put them in fireproof safes in different locations
physically outside the building because they don't want sensitive information being transmitted
over the internet to, you know, crash plan or backblaze or whatever that could potentially
be intercepted and decrypted.
So you know, the good old-fashioned tape and physical media has a big role in that application.
the average person is not going to go and buy a tape backup system.
Yeah, they're just not. And if you look at floppy disks, I mean,
a 5.25 inch floppy was was what? 1.44 meg?
1.44 or 2.0.
Oh, no. 5.25? I don't even know.
5.25 was 1.2 meg was the high density.
And 720K was the...
Yeah, so it was just terrible.
It's just, you know, I mean, they can't store anything and they're dead.
So throw them out the window. You've really only got those those tapes.
So you know not necessarily the good idea for the average person. You're not going to buy
thousands of dollars worth of professional grade backup takes just to last you 35 years for backups
of your photos. So a lot of people these days are looking to hard drives and because hard drives
have come down a lot in price and they're saying well you know what hard drives are hard drives are
good you know because they're cheap and the problem with hard drives is well they're magnetic media
It's no different to a backup tape, except of course that the spinning platter itself that
stores the actual information is protected from the outside world that exists in an
isolated atmosphere, sealed away from dust and so on. And it's going to be far more reliable,
right? You would think. But the problem with hard drives is that notwithstanding the failure
mechanisms. So, for example, if it's turned off sitting on a shelf, you may think, well,
you know, it's not spinning, it's not going to have any of those wear out mechanisms. I say,
well, I keep saying wear out mechanisms, it's like a reliability lingo, right? It just means that
if my drive is spinning 100 million rotations and it's got a drive bearing, well, that bearing
might only be rated to half that. And statistically, you go beyond that, the further you go beyond that,
and more statistically likely you're going to have a failure.
So you say if I'm not spinning it, I plug the hard drive in, I do all my backups to it, I disconnect
it and I set it up on the shelf. It's now no longer rotating therefore that wearout mechanism
is no longer in play and that's true. However, there are other wearout mechanisms that are still
in play and that is these things have got power supplies in them. They've got point of use power
supplies or regulator chips, they've got different capacitors in there for voltage regulation and
smoothing and all that sort of stuff. And those individual devices have all got a limited lifespan.
You can't just leave it on the shelf. It'll eventually, a lot of capacitors will dry out
and you will get the whole thermal expansion and contraction will still happen unless of course
the hard drive is being stored in a temperature controlled environment. And most people don't
have that in their house. So it'll be sitting on a bookshelf, let's say. Or maybe if you're smart,
in a fireproof safe, let's say. Yeah, but it's still going to fluctuate. You're still going to
have differential expansion rates between the components on the circuit board as well as the
circuit board itself and even the casing of the hard drive. And all of those things, eventually,
you'll get issues with, you'll have bad connections and eventually it will just die.
And you just can't avoid that. No technology lasts forever. It just doesn't. And, you know,
those are WERAC mechanisms. So, let's just ignore them for the moment. Let's just ignore it. Let's
assume that we leave it on the shelf and everything's all hunky-dory. How long does the
actual information last? And when I was doing some research into this a while ago, I came across an
article on, of all things, on Reddit. And it was quoting someone who was doing the physical,
doing the, sorry, doing the physics behind the hard drives and they were aiming for
basically what he called a 10-year bit life, which is to say after 10 years the integrity
of a single bit cannot be guaranteed because statistically a cosmic ray might hit it
and obviously if a cosmic ray hits it it'll flip a bit amongst a few others in the local
vicinity most likely depending I guess and eventually inevitably the magnetic domains
will start to break down and you'll start to get data corruption so that's what they're aiming for
So even without wear-out mechanisms, let's assume they achieve that. Maybe they go better than that.
Maybe it makes it 15 years before you start getting significant data corruption. And obviously,
if you've done multiple backups, multiple hard drives, then statistically,
you may be better covered. I don't know. But irrespective, the point is that you're not going
to get huge amounts of guaranteed life out of these things, even if they're sitting cold on a
shelf. I mean it relative to magnetic media. So the idea of using hard drives as long-term
backup storage is actually a relatively new idea because previously hard drives were a lot more
expensive and tape drives were a lot cheaper for the same amount of storage. If I'm trying to store
you know 200 terabytes it was a lot cheaper to get you know magnetic tapes to do that lots and
lots of tapes than it was to do it to a bunch of hard drives. That's only changed in the last I
I don't know, maybe five, seven, eight years, something like that.
So, relatively recently.
Okay, so that's magnetic.
Optical, which is the one I am using.
It's not perfect.
And the thing to appreciate about optical media for those that don't realise is that there's two ways you can make optical media.
You can press it physically.
you can, in other words, the gullies, the mountains and the flats for the ones and zeros in your optical disc can be pressed into it physically.
In which case, you know, it's far more solid, reliable, long term proposition.
It's like a glass master.
You're actually, yeah, the machine.
Yeah.
Yeah, exactly.
It literally, they're pressed.
and yeah, there'll be a master and you'll basically get the material, the plastic will be slightly
warm, you'll press it a certain amount of force and let it sit to cool for a few seconds,
yeah, actually it's fractions of a second I would think, and then you would pop the optical
media off and off you go, you're done. The point is that that's far more reliable than if you burn
a disc yourself and if you're doing backups you're going to be burning the disc yourself you're not
going to be creating a master and pressing a hundred thousand copies of the thing you're
going to be doing it once and those ones are a completely different kind of technology and that
requires a higher powered laser to literally burn a hole in a layer of dye and the problem with that
is and there's more than one way of doing i just want to point out but irrespective the strength
of the laser and the age of the drive affects how effective it is and actually doing that
and certain drives tend to have more issues with alignment and jittering than others. So, for
example, if I burn a DVD or a Blu-ray disc on one drive, sometimes it can have problems reading it
on other drives. So, it's not exactly perfect. And the other problem is, of course, that exposure to
light, intense light and heat, can cause a breakdown of that layer that you're running
the information to optically. So most optical media out there generally, just the average stuff
off the shelf, you're looking at 10 to 25 years. Some of them will advertise as much as 100 years,
but whether or not you trust that or not, it's debatable. If you're really paranoid about backing
up data on physical media, I wouldn't trust that. But something interesting that they're working on,
on something called M-Wave.
And the concept is to have a 1000 year lifespan
for optical media.
And this is something that's,
there's 25 gig, single layer, I believe, Blu-ray.
And that was released about six months ago, rather pricey.
Don't have the price in front of me,
but obviously you get what you pay for.
But the whole idea is it's supposed to be tested and proven to last to 10 to 1000 years.
And the funny thing, though, about that is obviously there's other control conditions,
you know, like control temperature and, you know, being away from sunlight and so on.
But the problem with that is obviously is how do you qualify this?
How do you actually prove this?
Well, that's the problem.
I mean, who's going to be around a thousand years?
Well, that's true.
know why not just lie about that? Well, the problem we've got is accelerated life testing
is what they call it when you try and predict. And the way they do that is they run it through
thermal cycling, they run it through visual, like for example, an optical disc. I'm not
entirely sure, but we did ALT on electronics all the time at Nortel. And that's just what
you did. It was part of your robustness testing for reliability. You want to stress the living
crap out of the poor cards such that when they're out in the real world and they go
through 365 days of a year of thermal cycling from a cycling shift of say 50, 60 degrees
Fahrenheit in an average day, that's max to min, and then back again day in, day out for
extreme locations. Well, you know, that will, if you replicate that in a lab and you do
those same cyclings and you do one of those temperature cycles in the space of 10 minutes.
Right. But a thousand...
You can estimate.
Are we going to have spinning platter? I mean, are we going to be having... I guess what
I'm saying is we'll jump off any kind of platform well, well before a thousand years.
Absolutely right.
So, what's the point?
no, yeah, I mean, I'm assuming yes, there would be some way to
get it back off. But you're basically being nice to the
archaeologists who dig up the remnants of our civilization.
Well, yeah, exactly right. And even if they did dig it up,
would they have working functioning equipment that was
capable of reading it?
And know the file formats? And yeah.
Yeah, exactly. I mean, they need to know everything. Yeah, it's
what good is an optical disk if you don't have a working
computer? Because even if you can spin it up and get the data
off, how the hell do you know what the file system is? So, there's so many holes in that,
but I guess and... HFS+
Yes, we're not going to get into a Syracusian discussion about HFS+. He's done that, I think.
I think so. Well and truly done it. So, 35 years for
a professional grade backup tapes, you could even argue that's a stretch. So, the companies
that are selling that would need to continue to support that with current drivers for current
operating systems and what happens if they go out of business.
So the whole concept of having a really long lifespan on these things is somewhat misleading
and I went through an exercise recently and once I recently I mean the last 12 months
where I took all of my CDs, all of my three and a half inch, I still have three and a
I have a floppy disks.
And I went through and I imaged all of them
'cause I had a USB floppy drive and I imaged all of them
and I took them all and combined them onto
and compacted them down to a bunch of 25 gig Blu-ray disks.
So I now have everything that I've ever had for the last,
yeah, geez, some of it will be getting close to 20 years
all on a bunch of Blu-ray disks,
just in case I ever need it.
I mean, do I realistically,
Am I realistically going to need software for an operating system that doesn't exist anymore?
Well, I guess maybe they don't want to mess around, feel nostalgic, get an emulator out and have a fiddle with Windows 95 again.
I don't know, maybe. I did a Windows 98
VMware Fusion
virtual machine, but I couldn't get the audio to work properly.
Can you run Windows 98 in JavaScript yet?
I don't know.
No.
Anyway, okay, I want to get back on topic because we're almost at the end.
So what I've done is I've found a great website.
It's about CDs, okay, but it's in the show notes and it's a site called xLab and they've
got a nice little discussion over optical discs and life expectancy, longevity and
so on and it's really, I think it's really good.
It's not super, well, you know, there are parts of the game that are technical but it's
quite easy to read.
I'll just put it that way.
It's not too, doesn't have too much required prior knowledge.
So it's a nice one.
So check that one out if you want to read more about it.
The last one that occurred to me as I was doing all of this and preparing the notes
for this episode is Flash.
And the reason that I thought about Flash is that Flash is now
where hard drives were a decade ago.
It's reaching the point where you can now use a flash drive
as a backup medium for small quantities of data.
You can't justify if you've got a terabyte to backup,
a terabyte of SSD is going to cost you a hell of a lot of money, but you could do it. And if you sat
on the shelf, it's going to have a completely different set of wearout mechanisms because
you don't have to worry about the caps and everything drying out or anything like that.
It's a silicon-based solution. So, you plug it into any compliant USB port and it's going to
fire up no problem. We assume USB will be around another 20 years, of course, but maybe not.
I'm sure adapters will exist. So, the problem though with Flash is that I struggle to find
very much information on their actual estimated storage lifespan if it's sitting on a shelf.
Lots and lots of information about the number of read writes for if you've got SLC, MLC or
EMLC. There's a whole bunch of different technologies and we're going to do a show
about SSDs, not all SSDs being created equally at some point in the future but for the moment at
least, we'll save that discussion for then. For the moment at least, I just wanted to mention the
fact that it's all about read/write cycles to the majority of people. They think okay well I'm using
the SSD as a replacement hard drive on my laptop or my desktop whatever but yeah there's also
removable thumb drives you know and there's a dozen names for those you know thumb drives,
memory sticks, flash drives, USB drives, USB sticks.
Jump drives.
Jump drives, thank you.
My favorite jump drives.
I don't get jump drive either. I mean, what are you going to do, jump on it? Anyway, whatever.
You get it jumping off your computer.
It's, yeah, well, I guess the point is that those are slower speed, generally,
unless it's one of those Magnum Patriot ones that are meant to be high speed USB 3.
Anyway, again, a topic for another show.
But what you have essentially is a system that you would think you put that on a shelf,
it's got no moving parts, it has minimal non-silicon electronics, so the wearout mechanism should
be relatively benign.
So what have you got?
You've got potentially the perfect time capsule.
The one thing I did find when I was looking into this is essentially the charge in one
of the memory cells will leak out over time.
It's just a matter of how long and that's what I couldn't find out.
So if anyone has any information on that, I'd be very curious.
But at the moment, it's still a cost prohibitive thing.
The suggestion is of course that you keep it cooler because at higher temperatures,
that'll accelerate the leakage rate.
if you store your flash drive in a cool location that's probably going to give you a much better
result but then again you can argue the same is true of hard drives and optical drives optical
disks sorry so at the end of all that as i see it optical media is the better choice that said
it's there's not much in it between that and hard drives for local backup storage because frankly
Assuming you have an obsolescence problem, which I'm assuming we're going to end up having at some point in 15-20 years time,
presumably both your hard drive and your optical disks will have lasted most of that distance, at which point then you can simply go to the next big thing,
the next optical format, the next... Yeah, by that point, maybe everything will be SSDs and spinning platters will be gone. Who can say?
- I was just thinking as we're talking about this,
it seems like we're actually in the middle
of an obsolescence point right now
with the transition to devices,
and I mean, particularly on iOS with this,
you know, the inability to get to the file system.
If I, you know, let's say you've been using a Mac
and you decide, hey, I'm going iOS only,
now you've got a new set of problems, right?
I mean, it's not seamless and it's not quite clear to me how you would even make use of a lot of these systems.
Because there's really no great way to do it without a PC.
Well, that's true. And obviously, this whole discussion is focused on the PC situation and files and filing and all that sort of thing.
On iOS, and to a lesser extent, but it's still true on Android.
I mean, if you ran a Nexus 7 tablet,
I suppose some of the Android tablets
you can have removable media,
but the problem is then you've got to,
what are you gonna do with the removable media?
So you take an SD card out of an Android tablet,
well, then I guess the flash drive's sitting on the shelf
for your backup.
I guess it scares me a little bit
because not all services last forever.
And when services die, not all of them die gracefully.
So people that rely on them
to store all their personal information
and don't do some kind of backup,
whether it's local or not,
but do some kind of backup.
If they don't do some kind of backup,
I think they're clinically insane.
I mean, if you've got photos that you cannot get back
And you're not printing them.
People just don't print their photos much anymore.
They really don't.
What are you gonna do?
Let's say Apple goes out of business tomorrow.
And yeah, okay.
This is a parallel dimension here
because that's extremely--
- Well, yeah, but let's just say Apple screws up
their web services 'cause that's been known to happen.
- Okay, so statistically more likely
that they would have a catastrophic failure.
And let's say it's not everyone that gets screwed.
Let's say it's 1%, which is still a hefty number
of people that would be shafted
and all of your photo stream and all that is gone.
And somehow that propagates out
to all those people's devices
and all that data is just gone somehow.
Okay, there's a big, you know, what if,
but maybe it's not, maybe you've got your,
you don't take a huge number of photos
but they're still important to you.
They're backed up on photo stream,
but they're no longer on your camera roll
for whatever reason.
- Oh yeah.
- 'Cause you've had to delete them
to make room for new photos, let's say.
- Right, and what's very, very likely is,
you know, someone, people that aren't technical at all,
probably not people listening to the show,
but you know people who are,
who just got their first iPhone
and now they're taking tons of photos
and are any of us really totally sure
what's going on with PhotoStream?
I don't know.
- I'm not convinced.
- No.
And it's a big transition and yeah,
I mean, yes, in many ways, the model for these devices is better and easier to use, but people aren't used to it and they don't know how to think about it.
And yet, sure, it wouldn't surprise me if someone, you know, has gone a year or two without backing up at all.
And I know from my brother worked at the Apple store all the time, people coming in saying, hey, where's all my stuff?
Because they didn't do any backups and they lost they lost stuff.
They lost deeply personal stuff.
And that happens. And it doesn't require some big catastrophic failure.
No, I suppose that's true. Just don't have iCloud backup turned on, don't have photo stream turned
on for whatever reason and you drop your phone in a toilet and it's gone. So, I guess my issue,
I hear what you're saying but for the subset of people that are, I wonder what the Venn diagram
is of people that are creating lots of personal content, but are doing so solely on an iOS
device without any other companion, Mac or PC in their lives.
I do wonder what that Venn diagram would look like, because I suspect a lot of people that
are in that scenario are not as prolific.
It's but it's changing, I think.
Yeah, it's changing.
And there's a generational aspect to it, at least, you know,
anecdote here.
But I think I think that's pretty safe that people are.
But you're right as you start to start to produce a lot, you're going to run
into the good side of the relatively low amount of storage space on iOS
devices.
You're going to run into that limit and you're going to be forced to start
thinking about it.
Maybe it's because...
My issue isn't necessarily that it's the
reliance on an external system
You're putting so much faith in Apple and
whether that's justified or not isn't the point. Whether they're reliable or not isn't
the point. The point is that companies fail, things die
they're beyond your control. However, if you had
a Mac or a PC locally, you backed up your photos to a bunch of Blu-ray discs, kept them in a
fireproof safe in the house, which is what I do, by the way, and you update that, say, every 12
months. If it was a fire and I walked into this house, I'd leave the fireproof safe, I'd grab my
CD collection in one hand, I'd rip the time machine out in the other, and that's it. Oh,
yeah, and I'd probably grab the kids too. So, you know, priorities, you know, anyhow.
But the point is that it's one hand, two hand done. In fact, if I was really, you know,
I could probably fit them in one arm if I had to. It's no big deal. They're both very small
and relatively light. But those time capsules are a bit heavy, but still. So, you know, it's like
between the two, I've got everything I want, everything I need. Because anything that isn't
burnt to a disc is a time machine backup. Yeah. So, and that's my strategy. However, if people,
and I think people need to consider their strategy and maybe that's the final point to wrap up on
is if you are just operating on an iOS device and you're not thinking about it because if you're not
and something does go wrong and it's not a question of you can't treat I don't think you
should treat life as an if something happens you should be planning for when something goes wrong
because if you're not, I mean, you could be lucky. I've met people that have had, you know, what do
they call it a charmed life where not much seems to have gone wrong. But honestly, that's the
exception. What generally seems to happen is computers fail, people drop their phones in a
toilet or, geez, I said it again, but I mean, I'm sorry, it wasn't a phone, it was a pager and no,
it wasn't mine. Anyhow, never mind. The point is it survived, but if it's a phone, it wouldn't.
I mean, my wife had our orange juice spilt on her iPhone 3GS, which was a hand-me-down
for me because I'd upgraded to a 4S at that point.
It was one of those things that it was just unrecoverable.
Fortunately, she had good backups because, well, she's married to me and I'm a nut for
that sort of thing, I guess.
But in any case, for goodness sake, think about it.
And I would not focus so much on the organizing.
I would focus more on the storing and having some kind of plan.
And if it's not locally for you, that's fine.
If you want to use something like crash panel back plays and you've got a massive pipe going
into your house, a data pipe, then great.
Good for you.
And you don't mind the ongoing costs, then go for it.
Recommend that, sure.
But for a lot of other people, that's not the answer.
And if that's you, then strongly consider an external hard drive to back up to, catalog
it, or optical disks like I've done, and keep them safe.
And I think a lot of this came to the fore for a lot of people.
A lot of people I know went with the Mavericks update, the new OS.
I knew a few people that had really serious issues and,
you know, lost, lost data, that sort of thing. You know, and, and I think
for me, a big part of the equation is, is, is, yeah, it is think it's stopping,
stopping the mindset of thinking about it,
like some sort of single catastrophic event happening and more recognizing that,
you know, it's just entropy is,
is going to do its work and you're going to
have little catastrophic events all the time.
And, uh, and yeah, you know, and to not, not think about the thousand year CD,
right. And to think about, well, what's going to get me to my next, like,
I think about it like reviews, um, that you,
you need to kind of go through maybe on a yearly basis or, or, you know,
and, and do a, do a certain,
a certain roundup every year and do a different roundup every month and do a
different roundup every week and kind of have these sort of series of stages that
your data goes through. Um,
Because I think there's the other side of this whole thing is, I mean, there's this
search aspect helping us in terms of finding old stuff.
But when you're talking about video, we're talking about audio and you're talking about
photos, you know, we're still not there yet.
If we're generating just a huge amount of stuff all the time and we're spending time
and money and, you know, you know, resources in the form of extra drives or disks and stuff,
can track the stuff we don't need or want.
That's not great either. And all these things have a cause.
Like, you know, like the IBM paper you were talking about, you know, if we're,
if our system is so complicated or so thorough that we're
just saving everything. I don't, I don't know if that's ideal either.
Well, it's a good point that I didn't really cover. And it's true,
people will draw the line at a different place. But the truth is that when you look at storage
these days and the examples that I was giving, I guess the one that comes to mind the most
is video. And I mean, you'll sit there and you'll take 100 hours worth of video of Elizabeth
rolling around and smiling and making cute gurgling noises and all that sort of newborn
little bub stuff.
But then you think to yourself,
okay, well, am I gonna keep all 100 hours
or am I just gonna snip out the bits where,
okay, I've been watching it to have the hiccups
for the last five minutes.
I think I've got the gist of how cute that is
and I'll keep the funniest,
let's say 30 seconds to a minute or something.
I don't need to keep five minutes.
And yeah, so that's the whole idea
of we've got raw footage,
you edit the footage and you get a final product.
And you say the final product,
save the raw footage. And that kind of clean out, that's how I handle my videos is using the same
technique you describe, which is I read, I accumulate it for a year, I go through and I do
a cut for the whole year and I keep the good stuff and I'm like, okay, I got enough of this and I
won't keep that and then I'll burn the final result and ditch all the raw footage. And
that kind of concept applied across all of your information is a good way of thinking about what
what you should back up and when is it's just do you really need to keep it.
If you want to talk more about this, you can find John on Twitter at John Gigi.
It's the same on app.net and you should check out John's site,
tech distortion.com. If you'd like to send an email,
you can send it to john@techdistortion.com. I'm Ben Alexander,
and you can reach me on Twitter at Fiat Lux FM.
You can follow @PragmaticShow on Twitter to see show announcements and other
related materials. And we wanted to take a moment just to talk about this for a second.
We don't track very much about how this show does, but one thing we really do like to pay
attention to is the discussion and feedback. It's how we know we're doing a good job.
Absolutely. And we've had some really good feedback and always would like some more.
So with every episode, if you have just a spare moment, then we'd love any feedback
you'd like to give us and if that includes a review or writing on iTunes as well, that's
fine or just part of the continuing the discussion of something we've talked about on the show.
Yeah, because these are big topics and there's a lot of technical details and there's a lot
of edge cases and things that, you know, quite honestly, it's difficult for everyone to know
the whole story here.
So it's, it's, it's, it's, we also get a real kick out of it too.
It's nice to know that people are paying attention and it gives us a better idea of what we should
be focusing on.
Absolutely.
And about the edge cases, it's not possible to cover everything on a lot of these topics
in one show.
So obviously, if there are edge cases you want us to explore, we're more than happy
to look into them.
Just let us know.
[Music]
[ Music ]