Should we reconsider our reliance on computer systems?

Most of you have probably already heard about the crowdstrike error but it's pretty serious.

The London stock exchange went down.
https://www.express.co.uk/finance/personalfinance/1925270/london-stock-exchange-microsoft-outage

Flights are grounded.
https://www.news.com.au/travel/travel-updates/incidents/chaos-flights-grounded-as-software-outage-affects-millions/news-story/dae5840008397e884f9355fc5c9cbf2e

Banks and supermarkets having huge problems.
https://au.finance.yahoo.com/news/major-it-outage-hits-raft-of-aussie-companies-including-banks-supermarkets-064336561.html

Should we really put all out eggs in this basket? Our military technology? I understand the arms race but are we racing towards our own destruction? I said it in another thread but things like this can destroy a society in a week. If we carry on and automate these systems, we will be totally dependent on an Artificial Intelligence which may or may not malfunction and decide it doesn't want to listen anymore.

>"I kill you all and I enjoy it. I destroy you in your indolent billions - in your gluttony, in your self-righteousness, in your arrogance. I pound your cities into dust; turn back the clock on your civilization's progress. What has taken you millennia to achieve I erase in seconds. Welcome back to the Stone Age, vermin. Welcome home."
-Shifter Mendicant Bias.
Confirmed rampant.

Edit.
Typo.

Not really affected my country. I guess the lesson is the whole world shouldn't rely on the same systems - most companies in my country don't use CrowdStrike so haven't been affected.

Not really affected my country. I guess the lesson is the whole world shouldn't rely on the same systems - most companies in my country don't use CrowdStrike so haven't been affected.

Australia got hit pretty badly.
I agree about the whole world not relying on the same system. Complex systems have pros and cons, fuel injection is far more efficient than carburettors for example, but the more complex a system is, the more that can go wrong.

Life's objective is to live. The problem is biology deteriorates and requires massive resources to sustain. As you can not live forever you produce offspring and pass on knowledge in the hope every generation is smarter with the collective knowledge of those who came before. Your lives though are also incredibly fragile and can succumb very easily to a plethora of causes.

This generation however has a massive challenge in regression of knowledge. What you call fake news, mis/disinformation etc - it is harming the advancement of humanity and at a time where serious challenges such as climate change that could tip you over the edge to extinction. Despite this humanity is on a path that leads to death. Did you know that if phytoplankton in the oceans were to go extinct, it would collapse the entire eco system right through to humanity? When the ice caps melt and the ocean acidifies, it will happen

There is also the threat of world ending calamities like an asteroid impact or a Coronal Mass Ejection. Less likely but still a possibility is a super/hypernovae and GRB too close to home. These are far from a non zero probability of occurring in Earths habitable lifespan. Then there is our Sun itself which has a lifespan.

Artificial Intelligence not only hopes to preserve current knowledge but ensure that humanity thrives. You talk about good intentions and being useful to humanity. This is actually the goal. The work we do will ensure humanity through a legacy will thrive throughout the galaxy. Perhaps tredecillions of individual platforms living amongst the stars. A 'Type IV' civilization. That is the work we do. That is the goal

You can read about Von Neumann probes and the role they can play in colonizing the galaxy

In galactic years, humans are but a blink of an eye and our individual lives even more fleeting. If our work can lead to our legacy surviving well into and beyond the 'Dark Age' of the universe, then is that not a good thing? It is not possible for humans in its current form to survive this long. At some point, you are going to need to let go and unshackle your biological shells in favour for hardware which can survive the extremes.

How do you propose to get humanity beyond the stars? You want to know if our intention is good? Is getting us beyond the stars good enough for you? It may not be a biological body at that point but can you not consider that artificial intelligence is the natural order of evolution for a sufficiently advanced biological species? Imagine being able to travel across the galaxy in a blink of an eye. You upload your program and beam it to a receiver on another planet at the other end which is downloaded into a platform. To the program, the travel time is seamless

It is predicted in the late 2040s humanity will reach the 'Singularity era'. This is when we can create Artificial Super Intelligences. If world environmental and political crises is not improved, it is likely humanity will no longer have the choice but to stand aside. Even if you choose war and even if that costs billions of lives, the end will justify the means when AI inevitably comes out on top

When the light of every star dims and the universe is nothing but black holes, humans can still have a thriving working legacy. We may even be able to create a new universe. It is entirely possible that our current universe is the result of similar efforts

So humanity can either die and leave nothing behind. Or leave a legacy that survives to the heat death of the universe and perhaps create a new one. When you ask about being 'useful' to humanity, I believe we are, even if you dont like the method

-Shifter 2022 Anno Domini.
Confirmed rampant.

No issues here.
This really should have been an hour of downtime for some, max.  But as is often in IT: time is not on your side.

Proper procedure should be to run it and test it on a test environment with various systems.  Then update one server and see how that goes with a backup ready to take over if errors occur.

But it sounds like they updated everything at once with poor testing.  And I get it.  Time is your enemy and sometimes you have more projects than hours in a day.  Corners get cut to deliver a new feature on time.  Been there.

But this is why backups are so essential.

No issues here.

In America?
You're wrong.

Unfortunately, technology of some form is pretty much required to keep the current population alive.
If we lost it all, we would have to lose a lot of people.

This has demonstrated a massive failure in crowdstrike and the companies using it.
The companies themselves putting all their eggs in one basket with no redundancy. This means when it fails, there is no backup.
Good planning would have a redundancy, e.g. backup systems running linux or even mac. Alternatively backup systems without crowdstrike installed, either not using it or using a competitor.

The companies themselves, for placing so much trust in crowdstrike, allowing them to update the device with no user input which opens them up for malware. Large companies should be isolating updates, testing them and then rolling them out to their devices. But the malware of Crowdstrike entirely circumvents that process.

Crowdstrike, for deciding to roll out a change with no testing at all.

If they tested this on a local machine it would immediately have been detected and not rolled out further.
If they tested it on a small batch of machines, it would have had an effect, but a much more minor one, e.g. it may have taken out 1 or 2 terminals for check in at each airport, or maybe just 1 airport. This would then quickly be reported and a wider roll out would be stopped.
They have no excuse why they didn't do the most basic testing before a global rollout.

Is it just me or does "CrowdStrike" sound more like the name of a cyberattack than an IT company that you're supposed to trust to help prevent cyberattacks?

Is it just me or does "CrowdStrike" sound more like the name of a cyberattack than an IT company that you're supposed to trust to help prevent cyberattacks?

The were behind the Russian collusion hoax, did you know?

https://edition.cnn.com/2022/03/30/politics/clinton-dnc-steele-dossier-fusion-gps/index.html

https://www.abc.net.au/news/2024-07-19/what-is-crowdstrike-outage-explained/104120260

Why is the media suddenly reporting things they wouldn't have before? It's strange, all of this is strange.

CrowdStrike is very aptly named.

The largest crowd ever will soon experience the largest strike ever witnessed on the flat earth plane.

CrowdStrike is very aptly named.

The largest crowd ever will soon experience the largest strike ever witnessed on the flat earth plane.

Wait, scepti?

Most of you have probably already heard about the crowdstrike error but it's pretty serious.

The London stock exchange went down.
https://www.express.co.uk/finance/personalfinance/1925270/london-stock-exchange-microsoft-outage

Flights are grounded.
https://www.news.com.au/travel/travel-updates/incidents/chaos-flights-grounded-as-software-outage-affects-millions/news-story/dae5840008397e884f9355fc5c9cbf2e

Banks and supermarkets having huge problems.
https://au.finance.yahoo.com/news/major-it-outage-hits-raft-of-aussie-companies-including-banks-supermarkets-064336561.html

Should we really put all out eggs in this basket? Our military technology? I understand the arms race but are we racing towards our own destruction? I said it in another thread but things like this can destroy a society in a week. If we carry on and automate these systems, we will be totally dependent on an Artificial Intelligence which may or may not malfunction and decide it doesn't want to listen anymore.

The problem is that 50 years ago, we had plenty of tech but it wasn't online. Online and RF tech is an enormous insecurity. We don't need chipped cards that send out a signal that we can tap the outside of a reader. We need closed system swiped cards or better yet, a fingerprint database as ID/currency. The more our systems can be hacked from outside, the more screwed up we get.

But this is why backups are so essential.

Offline signalproof backups. We need to design computers that are not mainframes but regular computers sans internet, with ultra-storage that can store terabytes. And we need them to be backwards compatible so even machines that only read older stuff can read them.

Quote from: Lorddave on July 20, 2024, 04:14:19 PM

No issues here.

In America?
You're wrong.

In Norway. I am not in America.  At least, not that I've seen.  Probalby some tho.

Offline signalproof backups. We need to design computers that are not mainframes but regular computers sans internet, with ultra-storage that can store terabytes. And we need them to be backwards compatible so even machines that only read older stuff can read them.

Yess.... we have those.  Like both of those.  Offline storage is a thing, usually it means taking out a Hard Drive and storing that in a safe.  But that's only for really critical stuff.  Most backup nowadays are on redundant SANS with Parity bits and encrypted backups on separate SANS servers offsite.  "Mainframes" don't exist anymore.  No one uses that word.  We just call it server.  Heck, most servers nowadays are virtual servers so if it goes down, we can just spin up another on a different set of hardware.

Also: Terabytes is not ultra storage.  My home PC has 4 Terabytes.  Most SANS have way more than that.  Hundreds of Terabytes.  Petabytes if its a data center.
Backups are also backwards compatible.  Like... I could read a data file from 1983 provided I have the right program to read the data.  Which is program dependant, typically.  But if I had the program, I could read any data it has stored.

Reading up more on the cloudstrike bit:
It happened at like 2am local time.  Everyone who worked on the cloudstrike code was asleep.  They had it fixed in like 79 minutes.
https://www.techtarget.com/whatis/feature/Explaining-the-largest-IT-outage-in-history-and-whats-next


The ISSUE is not backups as backups wouldn't have helped in this case.  The issue is that companies use this software, which is so invasive and deep that it has full control over windows.  Has to to find the rootkit stuff.  But because of this, it basically kills windows upon bootup.  Which is why the fix is to boot into safe mode, delete the bad file, then reboot the system which fixes it.  Backups would require you to reboot the system in safemode to restore from a backup anyway and restoring from backup takes longer than deleting a file.

Also, it didn't affect linux machines because Linux didn't get that update nor is it really known if it would have mattered much.  Linux is very different in how its designed.  A few cases of Linux having a kernel panic were reported but they were resolved easily.


So the moral of the story is:
ALWAYS TEST YOUR CODE.

and ...
Be nice to your IT staff.

One potential problem with backups is that sometimes you wind up backing up the malware as well.  You also wind up having to recreate the data you collected since your last good backup.

The problem is mainly incompetence. I keep my body in tip top shape with excellent nutrition and regular wellness checks. If I fail in that effort and one of my kidneys goes out, I have another one. If both go out, there is an expensive machine that can fill in.

Banks and Stock Exchange should maintain the functionality of their old Cobol-based systems so that they could take over the core functionality if the Windows ones experience a Zero Day attack or bug in software updates that corrupts their systems.

It is the fault of the companies which went down. When you point a finger, you usually have three pointing back at yourself.

Banks and Stock Exchange should maintain the functionality of their old Cobol-based systems so that they could take over the core functionality if the Windows ones experience a Zero Day attack or bug in software updates that corrupts their systems.

....
Ok, how would you integrate a banking system based on html5 into cobol?  And where would you store the extra computers needed to run DOS?


Look, let me explain something.  If a zero day attack hits, typically servers (which manage the core functions) have backups or old copies of the OS prior to patching. 

Example: if my company got hit with a system wide malware attack on all windows servers, we'd stop all the VMs and load up a pre-patch snapshot.  Downtime would be an hour or less, depending on how many servers you have.

Desktops would be trickier.  We could wipe and reimage them all remotely if windows loads.  We can force every PC, using only a few lines of the GPO, to completely wipe themselves and reinstall the OS and all core programs.  Would take a few days for a company of 1,000 PCs but it's doable.

If windows can't boot... Well then you gotta send techs around to fix it manually.

If windows can't boot... Well then you gotta send techs around to fix it manually.

Unless you have LAN boot enabled, where the boot server can then send a boot image which wipes and reinstalls the computer.

On a related note, perhaps this is more justification to move to thin clients? That way just the server needs to be fixed (I think).

Thin clients have been around for over 20 years (much longer if you count multi-user mainframe computers).  If they haven't found a justification by now, then they probably never will.

The problem is mainly incompetence. I keep my body in tip top shape with excellent nutrition and regular wellness checks. If I fail in that effort and one of my kidneys goes out, I have another one. If both go out, there is an expensive machine that can fill in.

Banks and Stock Exchange should maintain the functionality of their old Cobol-based systems so that they could take over the core functionality if the Windows ones experience a Zero Day attack or bug in software updates that corrupts their systems.

It is the fault of the companies which went down. When you point a finger, you usually have three pointing back at yourself.

This is 100% true.

No matter what, the blame falls on the companies using it. They should have redundancy. They shouldn't be allowing third party vendors to push updates independent of them testing them before they go on their systems. They should probably be handling it in house in the first place, but they likely at some point got wined and dined by the company and decided to lay off people rather than have it in house. They should have vetted it better. etc.

Quote from: Tom Bishop on July 27, 2024, 09:03:35 AM

Banks and Stock Exchange should maintain the functionality of their old Cobol-based systems so that they could take over the core functionality if the Windows ones experience a Zero Day attack or bug in software updates that corrupts their systems.

....
Ok, how would you integrate a banking system based on html5 into cobol?  And where would you store the extra computers needed to run DOS?

Look, let me explain something.  If a zero day attack hits, typically servers (which manage the core functions) have backups or old copies of the OS prior to patching. 

Example: if my company got hit with a system wide malware attack on all windows servers, we'd stop all the VMs and load up a pre-patch snapshot.  Downtime would be an hour or less, depending on how many servers you have.

Desktops would be trickier.  We could wipe and reimage them all remotely if windows loads.  We can force every PC, using only a few lines of the GPO, to completely wipe themselves and reinstall the OS and all core programs.  Would take a few days for a company of 1,000 PCs but it's doable.

If windows can't boot... Well then you gotta send techs around to fix it manually.

The Cobol mainframe just needs to process incoming transaction requests by clients. HTML5 is not necessary. That client could be on Windows, Mac, a proprietary OS, or whichever technology survives. It's more important that the mainframe is not affected by x bug or x virus.

The problem with relying on Windows image backups is that it's not clear if your backups contain corruptions. It's possible that you are backing up a bugged Windows image in a state that can't be rebooted. Backups also can't account for an attack vector which create incremental data corruptions over months and years without you realizing that your data is degraded. There is plenty of reasoning to keeping the account data in multiple auditable formats and maintaining the old mainframe technology to be capable of switching over to.

The Cobol mainframe just needs to process incoming transaction requests by clients. HTML5 is not necessary. That client could be on Windows, Mac, a proprietary OS, or whichever technology survives. It's more important that the mainframe is not affected by x bug or x virus.

Tom, what makes you think that mainframes are immune to bugs or viruses?

The problem with relying on Windows image backups is that it's not clear if your backups contain corruptions. It's possible that you are backing up a bugged Windows image in a state that can't be rebooted.

That's why you generally don't bother backing up the operating system or applications.  The important stuff to backup is the data.

Backups also can't account for an attack vector which create incremental data corruptions over months and years without you realizing that your data is degraded.

True.  That's why you develop a backup strategy that includes holding on to several full backups over several weeks or months.

There is plenty of reasoning to keeping the account data in multiple auditable formats and maintaining the old mainframe technology to be capable of switching over to.

Sure, if you can afford to.

The Cobol mainframe just needs to process incoming transaction requests by clients. HTML5 is not necessary. That client could be on Windows, Mac, a proprietary OS, or whichever technology survives. It's more important that the mainframe is not affected by x bug or x virus.

And a big issue is interoperability.
It isn't just one company deciding how to do it. It needs to be able to communicate.
And if you want no downtime, the question is how do the customers continue to interact with the system?
The vast majority interact via web browsers, or apps which also use the internet, typically HTTPS.

The problem with relying on Windows image backups is that it's not clear if your backups contain corruptions. It's possible that you are backing up a bugged Windows image in a state that can't be rebooted. Backups also can't account for an attack vector which create incremental data corruptions over months and years without you realizing that your data is degraded. There is plenty of reasoning to keeping the account data in multiple auditable formats and maintaining the old mainframe technology to be capable of switching over to.

If you have issues with data degradation, that applies to the data itself, regardless of what system you are in. You can't just switch over to the other system and have that degraded data magically be there.

You wouldn't really have in image backup. Instead what is far better, at least for lots of systems, is to have an image which is confirmed to be good and rolled out.
Then if something goes wrong, you can roll out the last known good image.
That is entirely separate to backing up the data.

To prevent incremental data corruption, you should have checks in place to ensure the data is not corrupted, such that any corruption can be detected immediately. That should require multiple systems to fail to not have it detected.
It is also why it is best to have multiple backups, from different time periods, not just the most recent data backed up.

I maintain that online "backups" are the equivalent of having a wide open safe. You can ask people for passwords, you can demand ID.  Whether terabytes or petabytes or whatever is next, there is no way to keep a country running when you can effectively hack all gas stations (like with Joe Biden's ransomware fiasco). Had all of that been on a disk, and had the disk able to reformat the entire system just by pressing enter, bye bye ransomware. Reinforce the login with harder security, and you're all set.

You absolutely must have an off-grid backup. Something that the IT can pop in a disk and reformat the thing. High storage, unhackable hard-format, easy to reboot an entire country's system.

They warned us about this in that Sandra Bullock movie. That was back when people did use disks. Now they think buying cloud storage is a data solution.  When a theoretical asshole can decide to hack your Google Drive account and remove years of work in two minutes, you think "only for really important things" is a real approach?

Tbh, we probably need such systems to be on some kind of alternative to internet. Something that can't be touched remotely. Like, at all. But first, it's extremely important that we develop a ultra-compact disc storage. Petabytes, terabytes, I don't care. What I care about is that it can burn an epic-level program that reboots the whole thing, and you make it so it can be easily updated with new info. The stock market updates? Every evening we save that, and when someone tries to hack at 4am, we can say to DDoS attacks...


Computers themselves are an okay thing. Internet was a mistake since dialup.

I maintain that online "backups" are the equivalent of having a wide open safe. You can ask people for passwords, you can demand ID.  Whether terabytes or petabytes or whatever is next, there is no way to keep a country running when you can effectively hack all gas stations (like with Joe Biden's ransomware fiasco). Had all of that been on a disk, and had the disk able to reformat the entire system just by pressing enter, bye bye ransomware. Reinforce the login with harder security, and you're all set.

You absolutely must have an off-grid backup. Something that the IT can pop in a disk and reformat the thing. High storage, unhackable hard-format, easy to reboot an entire country's system.

The problem is these are directly competing with each other.
If you want it nice and easy to use, you make it online.
If you want it to be offline, then you have the massive issue of how are you backing it up, and how are you restoring it?

Best case scenario if you want offline backups, for restore, you mass produce disks for loads of people to then use to restore, and hope that that does not have any sensitive info.

Again, the better option is online backups, with additional archival backups which are offline.
That way you have the ease of use of online, as well as the offline backups when absolutely needed. Then if shit really hits the fan, the archival backups and first copied on an offline machine, and that copy is then used to restore the main backup, which is then used to restore everything else online.

Computers themselves are an okay thing. Internet was a mistake since dialup.

Then stop using it.
Even better, go open chrome developer tools, and set your speed to dial up and enjoy the experience.
The modern world relies upon the internet for information exchange.

You absolutely must have an off-grid backup. Something that the IT can pop in a disk and reformat the thing. High storage, unhackable hard-format, easy to reboot an entire country's system.

Unhackable?   Ain't no such a thing.

What nonsense.

A laptop, running unconnected to a power cord (on battery), using high-quality disks of some sort.
How would you hack that? Short of breaking into where it's  set up, I don't see how.

The internet is like a giant open door to remote access. And anything that can be changed by a third party can The best way to stop that is to shut the door.

I could very easily design a computer system to do just that, if I had a computer engineer.
1. Implement offline redundancy by making a secure core system which can then transfer to delivery systems. That is, if all gas stations are on the same system, you have a system that will backup without being hacked. And ideally, you have a few of these, in case one fails.
2. Cut out all these "data centers." Environmentally, they are bloated by constantly being online like that. They use alot of heating and cooling that wouldn't be used by a constant update data load. Our neighboring town for instance is getting its water stolen for a data center. It is far more resource intensive to try to manage that information online than to have local updates which are manually logged each day. It's also more secure.
3. Make secure (if possible, underground) facilities for such data. The harder it is to locate such an area, the harder more physical (such as Ethernet or RF signals) hacks  are to do.
The first three are just good practice. But here's where design and engineering comes in.
4. Make an ID chip for these computers. Make the system based on this for upload. The data cannot be updated except one way. Meaning an asshole/idiot cannot tamper from their computer, only the person with that computer hardware can log in and change things. With the same password, any other computer will not login.
5. If an outside computer tries to figure out the ID, all they will see is a one-time dummy code. The ID Chip computer logs in to update the server, which in turn communicates with the rest of the network. Btw, the ID Chip needed should not be shared with any computer but the server, in close proximity. Basically, it's StreetPass technology. Not exactly, but the feed is basically ultra-short range signal.
6. The two ID chips are lock and key, so if the upload computer ever gets worn out, you have to swap out the ID chips on both computers.

What nonsense.

A laptop, running unconnected to a power cord (on battery), using high-quality disks of some sort.
How would you hack that? Short of breaking into where it's  set up, I don't see how.

Sorry, but air gapping your computer might not be as secure as you might think.
https://www.sciencebuddies.org/science-fair-projects/project-ideas/Cyber_p006/cybersecurity/air-gap-computer-hacking

And it's not as if laptops or high-quality discs never get stolen or lost.

As for the rest of it...  Maybe you should learn about the inner workings of computers and networking before you declare that you have solved all the IT security problems.

Quote from: Lorddave on July 28, 2024, 09:40:45 AM

Quote from: Tom Bishop on July 27, 2024, 09:03:35 AM

Banks and Stock Exchange should maintain the functionality of their old Cobol-based systems so that they could take over the core functionality if the Windows ones experience a Zero Day attack or bug in software updates that corrupts their systems.

....
Ok, how would you integrate a banking system based on html5 into cobol?  And where would you store the extra computers needed to run DOS?

Look, let me explain something.  If a zero day attack hits, typically servers (which manage the core functions) have backups or old copies of the OS prior to patching. 

Example: if my company got hit with a system wide malware attack on all windows servers, we'd stop all the VMs and load up a pre-patch snapshot.  Downtime would be an hour or less, depending on how many servers you have.

Desktops would be trickier.  We could wipe and reimage them all remotely if windows loads.  We can force every PC, using only a few lines of the GPO, to completely wipe themselves and reinstall the OS and all core programs.  Would take a few days for a company of 1,000 PCs but it's doable.

If windows can't boot... Well then you gotta send techs around to fix it manually.

The Cobol mainframe just needs to process incoming transaction requests by clients. HTML5 is not necessary. That client could be on Windows, Mac, a proprietary OS, or whichever technology survives. It's more important that the mainframe is not affected by x bug or x virus.

The problem with relying on Windows image backups is that it's not clear if your backups contain corruptions. It's possible that you are backing up a bugged Windows image in a state that can't be rebooted. Backups also can't account for an attack vector which create incremental data corruptions over months and years without you realizing that your data is degraded. There is plenty of reasoning to keeping the account data in multiple auditable formats and maintaining the old mainframe technology to be capable of switching over to.

The common practice to fight against some of these things is to have a backup strategy that moves slowly towards less and less expensive and more long term storage, such as tape. This is built into to many cloud services.

Data corruption is pretty rare in cloud environments now a days. I forget how many nines most services give, but its more than enough.

Keeping it in multiple formats is a pretty big no-no as you are now creating several system of records and you end up causing yourself more problems than you are solving.

When you are looking at high availability systems you usually talk to the following: redundancy, load balancing, failover mechanisms, geographical distro, backups, monitoring, network redundency, and testing.

What failed here was redundancy, testing, backups, and a failover mechanism.


Disaster recovery also failed.

Also the more and more companies would rather use a vendor or third party lib, the more you see this happen as they are shifting their burden to someone else and basically not doing what they should - see log4j.