fortypoundhead.com

How Pusher Killed the Labs

Posted On 2015-05-05 by dwirch
Keywords:
Tags: Blog Pusher Tip 
Views: 1945


This is the story of how 3 lab environments were rendered inoperable with a click of a mouse button. Well, there was some configuration involved, but it was completed with a single click.

In the time before time, I worked for company that didn't have or didn't want to release the cash for a "real" software distribution solution. At the time, Microsofts System Management Server 2.0 (SMS) was all the rage, and would've been a perfect solution for me to manage our lab environments, as well as production. So after fighting the good fight, I ended up writing my own solution called Pusher, which performs the task of deploying software to remote machines. Works pretty good, too. By the way, look elsewhere on this site, and you can download a copy for yourself.

The software was pretty raw in those early versions, and there were some potentially nasty behaviors in the software. The way it worked followed a simple flow:

  • Copy the bits to a distribution directory on the target computer.
  • Copy the installation script, usually a batch file, to the distribution directory on the target computer.
  • Create a scheduled task on the target computer, executing the installation script at a specific time.
  • Scheduled task executes at the specified time, installing the software.

Easy, right ?

The installation script (batch file) would usually contain something like:

cd\spdistro
Del *.*
msiexec /package MyPackage.MSI /quiet

This is pretty straightforward. First, we change to the correct directory. Next, we remove any existing files in the directory. Then we execute the installation with MSIExec. I had happily been using this method for quite awhile, and it definitely made my life easier. I could distribute packages to many target machines, with a minimum of effort. Worked pretty good.

After a period of time, other team members were hired, and it was time for me to take a vacation with the family. So I handed over the method, as well demonstrating a few times, to the new folks so they could perform pending deployments while I was gone. During these periods of instruction, I mentioned that they could use my current crop of batch files as an example to build their own batch files for the deployments. No problem.

Fast forward a couple days, and I am at the beach, enjoying the sun and the mighty Pacific ocean. While I did have my cell phone with me, reception was spotty in the area, which was fine with me. Towards the end of the day, heading back to town, my phone started blowing up, as we came back into cell range. I got a call within a few moments, from the PM for the project.

"Help! We've been deploying software with your tool, and every machine we have deployed to has become unresponsive!"

Oh boy.

Upon arrival back at the hotel, I reluctantly dialled in, and found that the machines were indeed unresponsive. I got one of my cohorts on the phone to see if I could get some insight as to what had happened. He mentioned that they had followed my procedure to deploy to packages, and rebooted the target as required by the installation. No big deal. Hmm. The plot thickens.

So I asked to see the batch file that was being used as a scheduled task for the deployment. Here is what I was sent:

Del *.*
msiexec /package MyPackage.MSI /quiet

Notice anything different? Yes, the line that changes the directory is missing. I will give you two guesses why this is bad, and the first one doesn't count.

When a scheduled task fires a batch file, a command prompt session is fired. The default directory for a command prompt on Windows XP (the target OS) is (fanfare here) C:\Windows\System32. I think you can guess what happened. When the scheduled task fired off, the first line of the batch file is Del *.*. This deleted every file that was not in use (locked) from C:\Windows\System32. Upon reboot at the end of the script, Windows couldn't boot.

I called the PM back, told him why the deployment had failed, and the lab environments would have to be completely re-deployed (they were virtual machines).

This particularly nasty "feature" in Pusher has been long since fixed, but taught some people some valuable lessons. For example, if you are going to copy someones code (the batch file), be sure to copy all the pertinent parts, and that you understand how it works. For me, I learned the value of providing more comprehensive documentation.

I did finish my vacation, though.


About the Author

dwirch has posted a total of 172 articles.

 


Comments On This Post

No comments on this post yet!


Do you have a thought relating to this post? You can post your comment here. If you have an unrelated question, you can use the Q&A section to ask it.

Or you can drop a note to the administrators if you're not sure where you should post.


Your IP address is:23.20.157.174

Before you can post, you need to prove you are human. If you log in, this test goes away.


Beat With A Stick List Returns: By request, BWASL has returned to the site. Free your rage by telling the world about something that really grinds your gears. Add your beating.



Recent Forum Posts

BWASL returns
dwirch posted on May 13, 2017 at about 15:24 in Site News

BWASL returns
dwirch posted on May 13, 2017 at about 8:46 in Site News

Job Spammer: Balashankar Bose Bose
dwirch posted on May 11, 2017 at about 10:05 in Spammers

Job Spammer: Bharti Jigyasi
dwirch posted on May 11, 2017 at about 7:58 in Spammers

List of Shady Characters
dwirch posted on April 25, 2017 at about 16:39 in Webmaster Stuff

Job Spammer: Bilal Uddin
dwirch posted on April 25, 2017 at about 11:00 in Spammers