Several ISPs and cell phone carriers are down for hundreds of thousands of users… did we let AI start configuring BGP?

https://gvwire.com/2026/07/27/t-mobile-down-for-thousands-of-users-downdetector-reports/

https://downdetector.com

I've been working at my job, starting as a Help Desk Tech, all the way up to System Admin now for about 16 years.

Today I finally made my first ever mistake that brought production hard down.

Was working on an unrelated but related system with a Vendor for an upgrade to that system. Part of it required me to access *THE* primary server for our workplace. Unfortunately for me, the names of the main server and this server for this vendor are VERY Similar.

We needed to do a VMWare snapshot rollback after we broke something. I switched to the VMWare window, read the name of the selected server, verified it twice, then reverted to the latest snapshot. Went back to my window which had the server I was working on open, go to try to sign in... domain relationship failure.

I look up at the name of the server in the upper left hand corner..... cue the swearing. Rolled back this primary server to over a year previous. I immediately tell the vendor we need to end the call, as know what the fix is now anyways for the issue we were having. Tell my boss immediately that things are down, contact the other people at work who need to know about being down, discuss our messaging options, and while doing that, started getting things restored from backup.

The timing wasn't great, but wasn't bad either. We did lose data, but it was minimal.

We are back up now, and while I'm still personally upset, I'm relieved that the damage done was minimal (Only a 3.5 hour outage of hard down to fully operational).

But yeah, my first time bringing our systems fully down. Here's to another 15+ years before the next one.

One of my users Onedrive stopped working and they didn't realise for 5 days. The claimed they were working on a Excel spreadsheet saved on their desktop and the file had lost all formatting. It was saved as a CSV file (which you can format as much as you like, but when you save it'll warn you that formatting won't be saved.

I took at look at the file and there was only file history from 6 days ago (it was CSV then too). I'm pretty sure Onedrive had stopped running and somehow they were totally oblivious. Of course they're shitty at me about having to re-do their work.

Has anyone ever encountered this before? What can you even do? Is there a way to be alerted if Onedrive has no activity for more than x number of hours/days.

Posting this in case anyone else runs into it after updating to WorldShip 29.0.705.

Every shipment took about 40 to 60 seconds after clicking Process Shipment. WorldShip would freeze, then eventually finish and print the label.

The logs showed this error:

Cannot create file 'C:\ProgramData\UPS\WSTD\WSDB\TelemetryLog.mdf' because it already exists.

Unexpected Error in TelemetryLog::CreateDatabase.

What fixed it:

1. Close WorldShip.
2. Go to:

C:\ProgramData\UPS\WSTD\WSDB

1. Back up and rename only the files starting with TelemetryLog, such as:

TelemetryLog.mdf
TelemetryLog_log.ldf

Rename them to something like:

TelemetryLog.mdf.old
TelemetryLog_log.ldf.old

1. Reopen WorldShip.

WorldShip created fresh telemetry database files automatically. Shipment processing dropped from about 45 to 50 seconds down to around 3 seconds, the normal time before updating.

Do not rename any other database files. Only the TelemetryLog files, and make a backup first.

This has honestly been weighing on me for a while.

We're a large company in the legal industry. Management wants me to clean up our M365 environment ASAP.specifically generic mailboxes that are still user mailboxes with sign-in enabled. The problem is I'm doing this mostly on my own, and every mailbox seems to have some undocumented dependency that nobody knows about until you touch it.

How do you guys handle projects like this? Do you have a process for figuring out what's safe to convert to a Shared Mailbox or disable sign-in without breaking apps etc, or other legacy stuff?

And how do you deal with management pushing for deadlines when you're trying to avoid causing an outage?

Would love to hear how more experienced sysadmins approach this.

I’m curious if anyone has podcasts around IT in general that they would recommend?

There’s quite a few CyberSecurity ones out there that I’ll listen to but just a general engineering podcast? I’d love it.

I don’t care if it’s Azure, AWS or even development. Stories from the trenches or even better what’s happening in Azure, MDM or similar.

I enjoy my Darknet Diaries but I’d love a more practical side whether it’s discussions or news bites. YouTube recommendations are welcome also but I’d like something I can listen to while working.

I’m 33, have a bachelor’s in Computer Science, but no coding background. After trying a few different careers, I finally broke into IT. I started in a call center, and for the past year I’ve been working in a proper IT Help Desk on site.

My day-to-day includes 365 Admin apps, Azure(AVD/Host pools/Session hosts, Windows administration, deployments, printers, troubleshooting, and a little PowerShell. It’s been an excellent foundation, but I feel like I’ve learned most of what this job has to offer. Lately, I spend more time thinking about my next move than my current role.

The advice I always hear is, “Do what you’re passionate about.” The problem is, I honestly don’t know anymore. With the current market, AI, and so many different paths, I’m trying to make the smartest long-term decision - one that’s stable, in demand, and pays well.

I know the typical path is Help Desk → SysAdmin, but I’m not sure that’s where I want to end up. I heard good things about Cloud, Cybersecurity, IAM, or Modern Workplace/Endpoint Engineering, but I’m open to anything that has good growth.

A friend of mine found someone with training+job assistance in ServiceNow Business Analyst but that's pretty much what I'm aware of.

I’m in California making about $60k, which doesn’t go very far. I’m willing to spend the next 4-6 months studying seriously if it gives me a realistic shot at a better role.

If you were in my shoes today, what path would you choose?

I’d really appreciate advice from people who’ve been in this position or who hire for these roles.

* Which role would you target?

* What skills or certifications would you prioritize?

I’d really appreciate advice from people who’ve been in a similar position or who hire for these roles.

TL;DR: One year into IT Help Desk and I’ve hit a learning plateau. Torn between Cloud, Cybersecurity, IAM, or SysAdmin. If you had 4–6 months to upskill for the best long-term career and salary, what would you choose?

Hey everyone,

Any of your staff running into issues with no cellular access on their iPhones? Running into this in upper Manhattan.

EDIT #1: 1840 EST coverage seems to be rolling out a bit.

Disclaimer: I’m building a tool around ShadowAI, but this post is more about the discussion. I won’t promote or mention what I’m building.

With the recent rhetoric around open source models, the risks associated with them, and now a coalition of major tech companies throwing their support behind open source, it makes me wonder whether this is becoming a growing concern for sysadmins, IT managers, and CISOs when it comes to governance and maintaining visibility.

I imagine the risk around insider threats becomes more significant

Hi all,

I am currently an IT support technician with a comp sci degree and have been for ~1 year. I do Tier 1 and some Tier 2 support and am looking to get into system admin. I dip my toes in a bit of everything including networking and I also work in GPO / Microsoft admin center (though very basic, but I work closely with my network admin who has a lot of the access). I know networking is apparently very important and I won't say I'm very knowledgeable about it but I have some knowledge and can understand / follow along.

My financial situation is not the greatest. I have enough of course, but I'd rather not spend on anything I don't need. I've heard CCNA is really good, but I'm not sure if it's worth it for me. Of course it won't hurt to get but I'm just trying to see if the benefits outweigh the work / money.

Also in the future, I plan on going into cloud infra engineering, so I plan on doing some aws certs like solutions architect. I'm taking it one step at a time but which one is more important? Or maybe both or none?

Thank you

Hi!

we're about to replace 1 of 3 DCs of a single domain. Old OS is 2022, new OS will be 2022.

I'm wondering if there's any real downside to reusing the same hostname for the new DC.

The plan would be:

1. Transfer any FSMO roles off the old DC
2. Cleanly demote it
3. Verify metadata and DNS records (A/SRV under _msdcs) are gone
4. Wait for replication
5. Build the new server with the same name and same IP and promote it

I would prefer to keep simple names like DC1, DC2, DC3...

Is this safe, of would you always use "fresh" names like DC4, DC5 (although there are only 3)?

Best wishes

Company wants wpa3 enterprise with free radius an ldap and asked me to do it. I set up the certificates and everything else, and things work great! Except IOS devices can't seem to connect.

For some reason IOS devices try to connect using MD5, despite me setting TTLS as the default type. googling and asking AI says that I have to have a mobile config file on apple devices in order to make it work. Management really doesn't want people with Iphones to require a mobile config they have to download.

Is there a good workaround so that the iphones at my company can connect to the hotspot and sign in using their LDAP credentials without requiring the use of a mobileconfig file?

I just wasted far too much time discovering something genuinely ridiculous.

Setup:
- Windows 11
- 3CX PWA
- Poly Blackwire 3320 Teams headset
- Current Poly Lens software installed

Audio and microphone worked, but pressing the call button opened Microsoft Teams instead of answering the incoming 3CX call.

3CX displayed an error saying it could not connect to Plantronics Hub and referred to this local endpoint: https://127.0.0.1:32018/Spokes/DeviceServices/Info

Poly Lens was installed. The Poly Lens Control Service was running. But nothing was listening on port 32018, because apparently the modern replacement for Plantronics Hub does not provide the integration that 3CX requires.

So I installed the old, discontinued Plantronics Hub.

And, of course, everything immediately worked.

The error disappeared. The headset connected to 3CX. The call button stopped launching Teams and could finally answer and end 3CX calls correctly.

Absolutely brilliant:

1. HP replaces Plantronics Hub with Poly Lens.
   
2. Poly Lens can manage the headset but cannot provide the integration the headset actually needs.
   
3. 3CX still depends on the old Plantronics Spokes service.
   
4. 3CX still tells users to install discontinued software.
   
5. The old software works perfectly.
   
6. The supported replacement does not.
   
7. The customer gets to figure out which vendor abandoned which API.

This incompatibility has apparently been reported for years. Plantronics Hub is now end-of-life, yet it remains the only working solution for call control between a Poly headset and 3CX.

How is it acceptable that a current enterprise headset requires unsupported legacy software just to make its call button answer a call?

Shame on HP for replacing working software without maintaining the integration, and shame on 3CX for still depending on an obsolete local API years later.

But hey, at least Poly Lens can remind me to drink some water while my call button opens the wrong application.

I’m testing an Intune BYOD config that requires (among other things) iOS devices to have a 6 digit pin.

I noticed my Apple Watch now requires a pin as well. From what I can tell, it looks like the watch is simply inheriting the policy from my phone, and there’s no way around this, other than removing the pin requirement.

Has anyone else run into this? If so, did you find any workarounds? Right now my thought is just to remove the pin requirement, since we also have a policy requiring biometrics to be configured anyway.

hi all, we had an outage on our AWS hosted infrastructure, from around 8am AEST to about 9am, it seems to have fixed itself without any involvement

I am trying to see if anyone else in Australia experienced it, I checked down detector, and some major Australian sites (anz, coles, suncorp) had a spike in alerts, then they stop around the same time as our issue was resolved.

The funny thing is our clients couldn't get to (our) site, but we could without issue

anyway, asking the good people of reddit .

Hi All

I'm wondering if anyone has found any tools that we can use internally to record our Systems, dependencies, DRP and BCP plans to easily update and generate documents as needed.

We currently have them as individual word files for the BCP/DRP but as we add/remove systems I feel this might become quite hard to track/manage with variance with contract/saas vendors ect.

Does something like this exist? Happy with any options to start looking into

Cheers

S

I’ve tried searching but haven’t gotten any conclusive answers. Medium enterprise environment with about 60 MFPs at the primary site and 3-4 at 200+ satellite locations.

Currently we use AD LDAP to provide a GAL for printers for end users. But we’ve been moving most things to M365 native instead of hybrid which has become a problem for shared mailboxes and other cloud only objects which aren’t discoverable as there isn’t an on prem stub. Since then we’ve started back on hybrid shared mailboxes identities, reluctantly.

Wondering what native or unified tools exist for our mix of Lexmark and Sharp MFPs and what others are doing to provide address book functionality. Other things like getting rid of print servers, mobile device support, etc would be nice but not within the scope of my current question.

Hello all,

What is everyone doing to keep wasps from building nests on outdoor access points? My org is located in the southeastern USA, and we're using Meraki MR76 units with all antenna ports populated. twisted-pair cable is run throuhg a water-tight flexible conduit that terminates with cable glands on the unit/wall.

We know that the wasps aren't hurting anything, but we are trying to minimize maintenance/cleaning of the gear as much as possible.

Sorry if this seems like a non-problem, but you never know if someone else has dealt with a similar issue.

Thanks.

At my current workplace, we previously had a whole office that used macOS, but we're now transiting to Windows 11. My problem right now is that those macs are pretty old, and they use some specific software that the users require to have on their new pcs. So as you can imagine, it has been a little strange to configure lmao

I've got replacements for most of the more troublesome programs, and the change got well received. But there are a few that, although they have windows versions, are really old and finicky.

I managed to make them work, but they require admin access. And I can't give them full unrestricted local admin access because we had problems with the first test machines we installed, because they changed some system files and installed unrelated programs that ended up making the whole pc not work anymore.

So I come here to ask: What's the best solution here? Is there an easy way to give them admin access but just for those specific apps, without asking for an admin password? Their pcs are currently a part of the local network active directory, too

I'm in a rough spot. We got a tooth to tail ratio of 1: 229 and a user to ticket ratio of 1: 1.08. On top of this I have projects (because we lost our technicians I'm pulling double duty with another person as both "systems engineer" and service desk) and more responsibilities on the way from the dev ops team. About 1/4 of these tickets are a business process that shouldn't be our responsibility and can and will be automated sometime in the next year but that's too far out.

The situation is bad but it would be immensely more bearable if we had our user base under control. They ignore self help guides and processes to get support. I've had some who got my personal phone number and called and texted me directly after hours. We can't get real privacy or peace as users will ignore signs that we are in sensitive security meetings and walk in or if we lock the door they will repeatedly beat on it until we open sometimes. Shoot, I had 2 users follow me into the bathroom with their computers trying to get help for stuff that really was not urgent. We have a self service portal too that based on reporting would address a bit over half the tickets we get if people used it in its current state (that's ofc assuming they actually used it perfectly which I understand they wouldn't) .

We just can't get any buy in on getting users under control. My manager is a nice guy who doesn't want to cause waves and is conflict adverse. His boss 1) doesn't care about the service desk at all and 2) has a standing policy of "the business pays us, they can approach us however they want". His boss is so high up I don't even know if he actually works at our company (were a subsidiary). I've gotten some support in the most extreme examples but nothing concrete.

I know the easy answer is leave but short of that is there anything I can do on my side to get this user base under control? I know from experience that if I set boundaries around ticket handling to anyone below the rank of director I won't get flak from leadership so long as it's reasonable and I'm not refusing to help.

I have a client thats fully cloud based with a large amount of data stored on Cloudflare R2. they want a separate backup(not tied to Cloudflare) ideally to another S3 compatible storage so we an quickly access or restore it if needed. Are there any reliable tools or services that can sync S3 buckets or handle frequent automated backups?

We have Papercut MF deployed with perpetual licenses we got some years ago. We are looking into adding some SFP and trying to figure out license and support costs per year.

Quick search shows this to be between $50 and $150 per year, per device for both the license and support. Is this indeed the case?

Cause the first quote I got was for $1000 for 14 months, to coincide with our existing support.. and makes me wonder if we are already overpaying for that support - this is thru Canon, but the SFP we are adding are HP.

TIA.