Search Tools Links Login

Advice for Young SysAdmins


I've been working in the technology field since 1986, and it's been a long and winding road, with many twists and turns. I thought I would take a minute and share some tips with my brethren who might be just starting out.

Apologies for what turned out to be a long post. I wanted to impart some knowledge, maybe help out a few people, with what started out to be a simple bullet list of things to watch out for, and things to do. As you can see, it turned into a wall of text.

Documentation is a never ending story

Document everything. Perform your documentation as you are working an issue or project. Waiting until after the project is completed will simply not work. Inevitably, there will be steps that you missed, which may be key in reproducing a positive outcome.

Let's say a problem occurs, with which you have past experience. If you are on vacation, or otherwise unavailable, how are your teammates going to fix the problem in a timely manner if it is not documented? Your team will basically be stuck "re-discovering" the process to resolve an issue. Further, if a problem doesn't happen very often, you may need the documentation to remind yourself.

Points of Interest

Stay organized

You don't need a content management system to stay organized. Like I said previously, you could just have a directory full of text files for documentation. Or another directory for your script library. I've seen people use a single text file for documentation, with each entry having a date/time stamp. The idea is to have a location you can go to and find the information you need without searching through email and stickynotes to find it. If using text files is not your thing, try something like OneNote or Evernote. If you want or need to share information with others, move up to a CMS or similar. Wikimedia is free and open source, if you want to run your own Wikipedia.

Points of Interest

Treat users with care, but skepticism

Don't believe a user's problem until you see it yourself. Question any reports carefully to ensure that what you think they're doing matches what they think they're doing - from the fundamentals upwards. Try using SOAP notes for both documentation and understanding users problems.

Trust the users if they report a problem. There may well be something wrong even if you haven't come across it yourself yet.

Points of Interest

If you're doing it more than once, automate

If you find yourself performing the same task daily, weekly, or monthly, automate it. You'll magically get the time back for performing that task, and automation reduces the chance for errors for subsequent runs of the process or task. Maintenance tasks are prime for automation. Think about things like log pruning, backups, and stale object removal. How much time do you spend checking for and removing old log files? On many servers? Again, give yourself some time back and automate all this little stuff.

Be rigorous and do it right first time. Never be tempted to think, "That works, I'll clear up the mess later" - you won't get the chance. An extra hour spent doing it properly now will save confusion and distress later.

Points of Interest

Generalize as much as possible

Keep your scripts as simple as possible. Ideally, you would have a library of functions or short scripts that can be included in other projects. For documentation, you should be able to hand it over to anyone, and have them follow it through to completion.

Points of Interest

Patch and monitor

Performing patches and updates should be automated. With the multitude of tools available for this, you have no excuse for not patching your systems. Even it is simply creating a script to check for updates for an OS or product, it needs to be done. Also, verify that patching has taken place. Just because your script or process downloaded the patches, doesn't mean they were successfully installed.

Keep an eye on event logs and error reporting. Anything that occurs multiple times in a short period of time needs to be investigated.

Points of Interest

Handle security in layers

Security doesn't end at the firewall. Don't leave privileged account passwords in text files. Implementing a password vault is quick and will make a big impact, both in making sure credentials are kept secure, but also serve as part of your documentation. Segment your networks so privileged systems don't co-exist with regular ones. Find the weak points, sandbox your web apps so they don't put the host server at risk. Monitor your firewall rules and IDS/IPS to make sure no unwanted traffic goes through. Make sure your anti-virus software is up to date and educate your users on how to behave in a secure way.

Points of Interest

Be prepared for the worst

Don't forget the backups. Don't ever forget the backups. The most depressing phrase you can hear in this line of work is, "What backups?" If you're restoring to fix a problem, make sure the problem hasn't migrated to the dumps as well.

Stay optimistic, but plan for the worst. This means doing proper backups using the 3-2-1 system, having three copies of any important data in two formats, making sure you always keep one copy off-site. Test your restore process, document that process and have a recovery plan that makes sense for your environment. Think up scenarios from software bugs to online attacks, physical breaches, power failures, flooding and fire, and find the best solution for them. People make mistakes, your procedures should keep those mistakes isolated.

Points of Interest

Keep learning

Don't get set in your ways. Always strive to learn more, and keep a percentage of each year to learn new software, products, or get new certifications. Be ready to handle the next shiny thing or switch role at a moment's notice as your business evolves. Take advantage of the incredible amount of free resources from YouTube videos to the Microsoft Virtual Academy, recorded talks at USENIX, DefCon and more.

Don't change for change's sake

On the flip side of the Keep Learning point above, don't change just because someone saw some new hotness at a trade show. You might have an old service or application that has been faithfully performing its function for years, and it does it well. Leave it alone.

I found an interesting maxim somewhere, that has served me well:

New applications: make them obey the FIFO rule - Fit In or Fuck Off.

You don't want to end up maintaining separate, autonomous systems if possible. (FIFO sometimes goes for people too.)

Points of Interest

Have fun

Don't get burned out. Be respectful to your users and colleagues, but learn to say no. Think about what is most important to you, and how you will think back on these days in 10 years. If you're not having fun where are you anymore, maybe find something else?

Points of Interest

About this post

Posted: 2017-06-06
By: dwirch
Viewed: 971 times

Categories

Tip

Tutorials

Blog

Tech Career

Attachments

No attachments for this post


Loading Comments ...

Comments

No comments have been added for this post.

You must be logged in to make a comment.

ADODB.Connection error '800a0e79'

Operation is not allowed when the object is open.

/assets/inc/inc_footer.asp, line 37