Advice for Young SysAdmins

Posted On 2017-06-06 by dwirch
Tags: Tip Blog Tutorial 
Views: 330

I've been working in the technology field for nearly thirty years. It's been a long and winding road, with many twists and turns. I thought I would take a minute and share some tips with my brethren who might be just starting out.

Apologies for what turned out to be a long post. I wanted to impart some knowledge, maybe help out a few people, with what started out to be a simple bullet list of things to watch out for, and things to do. As you can see, it turned into a wall of text.

Documentation is a never ending story

Document everything. Perform your documentation as you are working an issue or project. Waiting until after the project is completed will simply not work. Inevitably, there will be steps that you missed, which may be key in reproducing a positive outcome.

Let's say a problem occurs, with which you have past experience. If you are on vacation, or otherwise unavailable, how are your teammates going to fix the problem in a timely manner if it is not documented? Your team will basically be stuck "re-discovering" the process to resolve an issue. Further, if a problem doesn't happen very often, you may need the documentation to remind yourself.

Points of Interest

  • Make habit of documenting things, daily.
  • Even if it is a directory of text files, have a repository of knowledge.
  • Don't forget about dependencies or things to watch out for!

Stay organized

You don't need a content management system to stay organized. Like I said previously, you could just have a directory full of text files for documentation. Or another directory for your script library. I've seen people use a single text file for documentation, with each entry having a date/time stamp. The idea is to have a location you can go to and find the information you need without searching through email and stickynotes to find it. If using text files is not your thing, try something like OneNote or Evernote. If you want or need to share information with others, move up to a CMS or similar. Wikimedia is free and open source, if you want to run your own Wikipedia.

Points of Interest

  • It's easier to find your info if you know where to look

Treat users with care, but skepticism

Don't believe a user's problem until you see it yourself. Question any reports carefully to ensure that what you think they're doing matches what they think they're doing - from the fundamentals upwards. Try using SOAP notes for both documentation and understanding users problems.

Trust the users if they report a problem. There may well be something wrong even if you haven't come across it yourself yet.

Points of Interest

  • Users will rarely describe their problems with a level of detail that is needed to solve their problems.
  • Ask for screenshots.
  • Eat your own dogfood. Use the same environment, machine(s), and network(s) as the users, so that you can witness problems that they might be having.
  • Provide or have available training for your users. Even it is a simple FAQ page with links, giving a user a bit of knowledge goes a long way towards empowering your users, and giving them the ability to resolve issues on their own.
  • Your users may not be technically inclinded. Don't assume that they have a basic understanding of operational security, IT processes, etc.

If you're doing it more than once, automate

If you find yourself performing the same task daily, weekly, or monthly, automate it. You'll magically get the time back for performing that task, and automation reduces the chance for errors for subsequent runs of the process or task. Maintenance tasks are prime for automation. Think about things like log pruning, backups, and stale object removal. How much time do you spend checking for and removing old log files? On many servers? Again, give yourself some time back and automate all this little stuff.

Be rigorous and do it right first time. Never be tempted to think, "That works, I'll clear up the mess later" - you won't get the chance. An extra hour spent doing it properly now will save confusion and distress later.

Points of Interest

  • Learn one scripting language well, and use it consistently. Using it consistently helps you to become an expert with the language.
  • Conversely, learn multiple scripting languages. Just because you are great with one language, doesn't mean there is another language that performs a particular task better or more easily. Choose the right tool for the job.
  • Build a library functions and short scripts. This will allow you to build apps and larger scripts more quickly and reliably.
  • Write scripts with scalability in mind. Just because you are only managing a small group now, doesn't mean you won't be administering a large enterprise later.

Generalize as much as possible

Keep your scripts as simple as possible. Ideally, you would have a library of functions or short scripts that can be included in other projects. For documentation, you should be able to hand it over to anyone, and have them follow it through to completion.

Points of Interest

  • Each function should perform one task, and do it well.
  • Reusable code is a good thing. Why re-invent things for each new process?
  • Documentation should be written with the mindset that the audience has minimum previous knowledge.

Patch and monitor

Performing patches and updates should be automated. With the multitude of tools available for this, you have no excuse for not patching your systems. Even it is simply creating a script to check for updates for an OS or product, it needs to be done. Also, verify that patching has taken place. Just because your script or process downloaded the patches, doesn't mean they were successfully installed.

Keep an eye on event logs and error reporting. Anything that occurs multiple times in a short period of time needs to be investigated.

Points of Interest

  • Patch your stuff. No excuses.
  • Watch your errors

Handle security in layers

Security doesn't end at the firewall. Don't leave privileged account passwords in text files. Implementing a password vault is quick and will make a big impact, both in making sure credentials are kept secure, but also serve as part of your documentation. Segment your networks so privileged systems don't co-exist with regular ones. Find the weak points, sandbox your web apps so they don't put the host server at risk. Monitor your firewall rules and IDS/IPS to make sure no unwanted traffic goes through. Make sure your anti-virus software is up to date and educate your users on how to behave in a secure way.

Points of Interest

  • Operational security is everyones responsibility.
  • Key concepts are:
    • Confidentiality
      In information security, confidentiality "is the property, that information is not made available or disclosed to unauthorized individuals, entities, or processes" (Excerpt ISO27000).
    • Integrity
      In information security, data integrity means maintaining and assuring the accuracy and completeness of data over its entire life-cycle. This means that data cannot be modified in an unauthorized or undetected manner. This is not the same thing as referential integrity in databases, although it can be viewed as a special case of consistency as understood in the classic ACID model of transaction processing. Information security systems typically provide message integrity in addition to data confidentiality.
    • Availability
      For any information system to serve its purpose, the information must be available when it is needed. This means that the computing systems used to store and process the information, the security controls used to protect it, and the communication channels used to access it must be functioning correctly. High availability systems aim to remain available at all times, preventing service disruptions due to power outages, hardware failures, and system upgrades. Ensuring availability also involves preventing denial-of-service attacks, such as a flood of incoming messages to the target system essentially forcing it to shut down.
    • Non-repudiation
      In law, non-repudiation implies one's intention to fulfill their obligations to a contract. It also implies that one party of a transaction cannot deny having received a transaction nor can the other party deny having sent a transaction.

Be prepared for the worst

Don't forget the backups. Don't ever forget the backups. The most depressing phrase you can hear in this line of work is, "What backups?" If you're restoring to fix a problem, make sure the problem hasn't migrated to the dumps as well.

Stay optimistic, but plan for the worst. This means doing proper backups using the 3-2-1 system, having three copies of any important data in two formats, making sure you always keep one copy off-site. Test your restore process, document that process and have a recovery plan that makes sense for your environment. Think up scenarios from software bugs to online attacks, physical breaches, power failures, flooding and fire, and find the best solution for them. People make mistakes, your procedures should keep those mistakes isolated.

Points of Interest

  • If you haven't done a test restore, the backup didn't happen. Test your backups by performing restores from time to time.
  • Even if you are in a small environment, it's always good to be thinking of disaster recovery. The business pays you to keep services running, and ensuring business continuity does nothing but make people happy.

Keep learning

Don't get set in your ways. Always strive to learn more, and keep a percentage of each year to learn new software, products, or get new certifications. Be ready to handle the next shiny thing or switch role at a moment's notice as your business evolves. Take advantage of the incredible amount of free resources from YouTube videos to the Microsoft Virtual Academy, recorded talks at USENIX, DefCon and more.

  • The world of technology is constantly changing. Keep your skills fresh.
  • The phrase "That's the way we've always done it." could merit a beating.

Don't change for change's sake

On the flip side of the Keep Learning point above, don't change just because someone saw some new hotness at a trade show. You might have an old service or application that has been faithfully performing its function for years, and it does it well. Leave it alone.

I found an interesting maxim somewhere, that has served me well:

New applications:make them obey the FIFO rule - Fit In or Fuck Off.

You don't want to end up maintaining separate, autonomous systems if possible. (FIFO sometimes goes for people too.)

Points of Interest

  • Save the money of implenting a new project by asking yourself, "Is this really needed?"
  • Feature creep can be an instigator of time delays and budget spend. Research a simpler solution before implementing an overly complex one.

Have fun

Don't get burned out. Be respectful to your users and colleagues, but learn to say no. Think about what is most important to you, and how you will think back on these days in 10 years. If you're not having fun where are you anymore, maybe find something else?

Points of Interest

  • Some suggestions for preventing burnout:
    • Vacation Time
    • Non-tech Hobby
    • Not Working
    • Travel (Conferences)
    • Pursue Certification
    • New Job
    • Sports

About the Author

dwirch has posted a total of 190 articles.

Comments On This Post

No comments on this post yet!

Do you have a thought relating to this post? You can post your comment here. If you have an unrelated question, you can use the Q&A section to ask it.

Or you can drop a note to the administrators if you're not sure where you should post.

Your IP address is:

Before you can post, you need to prove you are human. If you log in, this test goes away.

Code Links