Linux nerd and consultant. Sci-fi, comedy, and podcast author. Former Katsucon president, former roller derby bouncer. http://punkwalrus.net

  • 0 Posts
  • 34 Comments
Joined 1 year ago
cake
Cake day: June 22nd, 2023

help-circle

  • Probably HR (or the NCS equivalent) never told the right people. I am not saying this is actually what happened, but a lot of IT bemoan the fact they are never told some rando employee was fired because HR neglects to inform them. Sometimes it takes months to discover, and even with a 90 day password/login lockout, some halfway decent admin could get around this by secretly building a back door, and using the messed up communication and politics between departments to hide this. Even in the 1990s, I saw people put in “time bombs” in their code that “if such and such is not updated in 6 months, run destructo-script A.”

    But imagine someone like Kandula Nagaraju here. Worked in QA, probably did a great jobs with some skills, but had the personality of swallowing broken glass. He was terminated in October 2022 due to “poor work performance,” which could mean anything. “Not a team player.” Or maybe he really was an idiot: I mean, a smart person would have a conniption, but get employed elsewhere and then slam his former company at parties. “Those NCS folks didn’t know what they had with me!” But this guy was probably someone with some anger management issues, probably a jerk, and possibly stupid. He might have had revenge fantasies, and set up a small virtual server posing as a backup code mirror. But outside the audits, it allowed ssh from the outside, and hid it through a knockd daemon. Or maybe only launched ssh at certain hours before shutting it down again. Silently working away in a sea of virtual servers with little to no updated documentation. He gets in, has internal access, and runs a script with admin credentials because they don’t rotate their AWS keys/secrets quickly enough. Or didn’t even know he was let go.

    After Kandula’s contract was terminated and he arrived back in India, he used his laptop to gain unauthorised access to the system using the administrator login credentials. He did so on six occasions between Jan 6 and Jan 17, 2023.

    That’s embarrassing to the company. Not only did he get in, but SIX TIMES after he was let go. he probably knew what order to run the delete commands (like, say, an aws “terminate-instances” cli command from a primary node), and did so one by one, probably during hours with the least amount of supervision, where the first few alerts would take hours to get someone in the monitoring chain to wake an admin. Given his last day was in November, and he got back in January, the admins probably thought their 90 access credential rotation was “good enough,” but he got in on his 80th day or whatever.

    I know this because I have had to do triage when a former contractor did this to a company I worked for. But instead of wiping out instances, he opened a new set of cloud accounts from the master account, put them in an unmonitored region (in this case, Asia), and spun up thousands of instances to run bitcoin mining. Only because AWS notified us of “unusual traffic” were we made aware at all, and this guy knew his shit and covered his tracks very well. He did it at a speed that could have only been automated. Thankfully, AWS did not charge us the seven figure amount that this activity amassed in just three days.


  • I remember hearing that some Hollywood contracts require that if you sign up for some studio, you must make X amount of films. Big stars get to chose those films to some degree, but once in a while, they have to do “a stinker” to end the contract as “X amount of films done, okay?” or something. Contractual Obligation and all. This film feels like a dumping ground of a lot of those contractual obligation hires from the trailer alone.


  • I was burned afoul by a former admin who, instead of diagnosing why a mail service was failing, labeled a script as a /etc/cron.d file entry as “…” (three dots) which, unless you were careful, you’d never notice in an "ls " listing casually. The cron job ran a script with a similar name which he ran once every 5 minutes. It would launch the mail service, but simultaneous services were not allowed to run on the same box, so if it was running, nothing would happen, although this later explained hundreds of “[program] service is already running” errors in our logs. It was every 5 minutes because our solarwinds check would only notice if the service had been down for 5 minutes. The reason why the service was crashing was later fixed in a patch, but nobody knew about this little “helper” script for years.

    Until one day, we had a service failover from primary to backup. Normally, we had two mail servers servers behind a load balancer. It would serve only the IP that was reporting as up. Before, we manually disabled the other network port, but this time, that step was forgotten, so BOTH IPs were listening. We shut down the primary mail service, but after 5 minutes, it came back up. The mail software would sync all the mail from one server to the other (like primary to backup, or reversed, but one way only). With both up, the load balancer just sent traffic to a random one.

    So now, both IPs received and sent mail, along with web interface users could use. But now, with mail going to both, it created mass confusion, and the mailbox sync was copying from backup to primary. Mail would appear and disappear randomly, and if it disappeared, it was because backup was syncing to primary. It was slow, and the first people to notice were the scant IMAP customers over the next several days. Those customers were always complaining because they had old and cranky systems, and our weekend customer service just told them to wait until Monday. But then more and more POP3 customers started to notice, and after 5 days had passed, we figured out what had happened. And we only did Netbackups every week, so now thousands of legitimate emails were lost for good over 3000 customers. A lot of them were lawyers.

    Oof.


  • Having moderated forums back in the day, I can answer to some of that motivation.

    First, some people are just bullies. A sense of tribalism forms around bullies, who feel the need to act out and repeat the abuses they have endured. Hazing stems from this, too. Cruelty masked as “you should know better,” advice. Given too late.

    Some have a smug sense of superiority, and want to keep it that way. Less smart people means they stay king of the mountain. Others are scared their own lack of knowledge will cripple them if they don’t keep the potential competition down. Insecurities drown out any sense of empathy.

    Some people hate themselves so they punish others in retaliation. Like, trying to erase past cringe by making others hurt to even the score.

    A few are sick of “the same fucking newbie questions again and again and again,” but still hang out in newbie forums for some reason.


  • really just doesn’t do what I needed to do.

    This has been my experience, or sort of does what I want it to do, but I have to rethink what I need it to do instead of something really simple. Like a “new type of shared file system” that replaces NFS/Windows sharing. So instead of files in a standard file system one can manage with a file browser, it has “indexed” your files in such a way that the actual files are renamed into data chunks, and one “finds” files by their non-intuitive search engine that can’t do even basic search engine tricks like “AND/OR” searches, wildcards, and the results are hit and miss. “But it’s faster and more elegant!” So how do you restore from backup when the system fails? “When the system does whatnow?”

    Yeah, no thanks. I can recover files from a file system much easier than some proprietary encoded bullshit fronted with a bad search engine over a proprietary and buggy index.



  • I hate to be honest, but I used Amazon Prime a lot because:

    1. I cannot drive. Thus, getting to the store is difficult.
    2. I must bring in 3-4 items a week, so yeah, I save on shipping.
    3. Auto-subscriptions save a little.
    4. I have priced a lot of stuff over the years, and while Amazon is not always the best, the convenience is impressive.
    5. They have, multiple times, been incredibly helpful with customer service. Like above and beyond.
    6. COVID and nobody masks around here. I have an autoimmune condition, so it’s important that I not leave unless it’s a medical appointment or similar need.
    7. They just have stuff I can’t find anywhere. Yes, as some have said, caveat emptor, but that’s true for all the stores.

    I also save a shit ton of money. When I used to browse Walmart or Target, I used to buy a lot of shit I didn’t need. I don’t get as distracted with focused buying. I also order from Aliexpress if I can wait 30 days, and I have only been ripped off three times in several years, for a total of maybe $35.

    I’m not saying my way is better, and certainly not if it’s better for you, but it’s been a godsend to the house-bound.





  • “The simple act of coordinating human resource decisions with IT department actions, such as revoking account access for dismissed personnel, would significantly mitigate such risks.”

    HAHAHAHAH… ahh… yeah, like that’ll ever happen. How do I know my users have been deleted? When I find out by accident 90 days later. I have worked for several companies where HR doesn’t do shit for IT. I find out employees have been hired when they show up to my desk, asking for a login and laptop. I find out they were let go when the 90 day expiration report shows who got expired for not logging in for 90 days. “Jim’s admin password expired.” “Jim left in October.” I have worked for companies where the simplest of forms, a form generated by an email or a popup in some ticketing system only requires checking off a checkbox… nothing. Can’t be arsed.



  • More directly, “This is Molle. He came to Varberg in 2013 and has found his favorite place here at Hajen [“The shark?”] and can sit here for hours and watch all the people passing by. He loves to be petted but DO NOT FEED HIM as he has a family and lives in a house behind the shop.” This is the ICA supermarket chain in Varberg, I believe. I have never been there, so I am not sure what the shark reference is (and there’s even a shark logo in the lower right corner), and my Swedish ain’t that great.





  • Being poor. In college in the 90s, my lead sysadmin couldn’t afford Minix for this system we had, so we tried to compile Linux on it. Three days later, we still failed, and gave up, but this was kernel 0.93 or something, so it had a ways to go. But I learned so much from that experience without paying for a university course or something.

    Years later, I bought a copy of Red Hat 6 at a Costco. Windows 95/98 was big, I didn’t know how to pirate it, so I went back to Linux and it worked great on my “franken-puters” cobbled together from spare parts dumpster diving. Steep learning curve back then, though. Then I brought it to my workplace, went from UNIX admin to Linux admin, and soon I preferred it to Windows. Been my daily driver for decades, now.

    Am I an evangel? A little, but I find that “right tool for right job” is a better approach. Linux is great for everything, BUT a comprehensive system like MS Office AND Active Directory simply does not exist in FOSS space yet; everything is cobbled together and a kludge still trying to catch up.

    Obsessed? Kinda. I just assembled some ansible scripts to roll my own distro. Why? To see if I could.


  • I worked in a job with build scripts. Developers would list what they wanted in a drop-down menu on a website, with very few “fill in the blanks.” This would create a template, which was sanity-checked.

    One of the “fill in the blanks” was “home directory of user, if not default /home/username.” Some people filled it in, some didn’t. A lot of “users” might be apps with /home being “/opt/appname” “/var/www/html” or something. We checked to make sure that directory existed, if not, create, and set permissions. Easy peasy, all automated. Ran this lots of times.

    Then one day, the script failed. Borked the whole box. Sometimes the VM was corrupt, so delete VM and try again. Usually worked. But this time, the build kept failing. The box went down. Wasn’t even bootable. This happened several times with this one build. So we mounted the borked drive under a new VM and checked out the logs. Just like the dessert stage of Willy Wonka chewing gum, it always failed at the last stage: making /home directories.

    It would create them, then halt that it could not find bash. We looked for bash on the bad drive, and it was the usual /bin/bash shortcut to /usr/bin/bash and we were truly puzzled. I did a chroot to the drive and NOTHING worked. It just hung. That was the first clue.

    The second was looking through the build script (in bash, which we didn’t write) and checking the steps. Looked it the logs. Always died at creating some user named sapadm, the user for the HANA database. Eventually, I checked the configure file, and noticed it was the only user with the odd home directory “/usr/sap.” Then it hit me: the permissions.

    The script, thinking it was a home directory, did a chmod - R 755 for all directories and chmod - R 644 for all files! That meant, while creating home, it made everything under /usr not executable anymore! Holy shit, no wonder nothing worked! So we commented out that user in the config, ran the build again, and we were good! We created the sapadm by hand, and then later fixed the bug in the script.

    SANITIZE YOUR DATA. Or you might turn Violet Beauregarde into a blueberry.


  • I have found that it’s like having a junior programmer assistant. It’s great for “write me python code for opening an in file from a command line argument, reading the contents into a key/value dict array, then closing the file.” It’s terrible for “write me a python code for pulling data into a redis database.”

    I find it’s wrong 50% of the time for certain command line switches, Linux file structure, and aws cli.

    I find it’s terrible for advanced stuff like, “using aws cli and jq, take all volumes in a vpc, and display the volume id, volume size in gb, instance id it’s attached to, private IP address of the instance, whether is a gp3 or gp2, and the vpc id in a comma separated format, sorted by volume size.”

    Even worse at, “take all my gp2 volumes and make them gp3.”