How Pusher Killed the Labs
This is the story of how 3 lab environments were rendered inoperable with a click of a mouse button. Well, there was some configuration involved, but it was completed with a single click.
In the time before time, I worked for company that didn't have or didn't want to release the cash for a "real" software distribution solution. At the time, Microsofts System Management Server 2.0 (SMS) was all the rage, and would've been a perfect solution for me to manage our lab environments, as well as production. So after fighting the good fight, I ended up writing my own solution called Pusher, which performs the task of deploying software to remote machines. Works pretty good, too. By the way, look elsewhere on this site, and you can download a copy for yourself.
The software was pretty raw in those early versions, and there were some potentially nasty behaviors in the software. The way it worked followed a simple flow:
- Copy the bits to a distribution directory on the target computer.
- Copy the installation script, usually a batch file, to the distribution directory on the target computer.
- Create a scheduled task on the target computer, executing the installation script at a specific time.
- Scheduled task executes at the specified time, installing the software.
Easy, right ?
The installation script (batch file) would usually contain something like:
cd\spdistro
Del *.*
msiexec /package MyPackage.MSI /quiet
This is pretty straightforward. First, we change to the correct directory. Next, we remove any existing files in the directory. Then we execute the installation with MSIExec. I had happily been using this method for quite awhile, and it definitely made my life easier. I could distribute packages to many target machines, with a minimum of effort. Worked pretty good.
After a period of time, other team members were hired, and it was time for me to take a vacation with the family. So I handed over the method, as well demonstrating a few times, to the new folks so they could perform pending deployments while I was gone. During these periods of instruction, I mentioned that they could use my current crop of batch files as an example to build their own batch files for the deployments. No problem.
Fast forward a couple days, and I am at the beach, enjoying the sun and the mighty Pacific ocean. While I did have my cell phone with me, reception was spotty in the area, which was fine with me. Towards the end of the day, heading back to town, my phone started blowing up, as we came back into cell range. I got a call within a few moments, from the PM for the project.
"Help! We've been deploying software with your tool, and every machine we have deployed to has become unresponsive!"
Oh boy.
Upon arrival back at the hotel, I reluctantly dialled in, and found that the machines were indeed unresponsive. I got one of my cohorts on the phone to see if I could get some insight as to what had happened. He mentioned that they had followed my procedure to deploy to packages, and rebooted the target as required by the installation. No big deal. Hmm. The plot thickens.
So I asked to see the batch file that was being used as a scheduled task for the deployment. Here is what I was sent:
Del *.*
msiexec /package MyPackage.MSI /quiet
Notice anything different? Yes, the line that changes the directory is missing. I will give you two guesses why this is bad, and the first one doesn't count.
When a scheduled task fires a batch file, a command prompt session is fired. The default directory for a command prompt on Windows XP (the target OS) is (fanfare here) C:\Windows\System32. I think you can guess what happened. When the scheduled task fired off, the first line of the batch file is Del *.*. This deleted every file that was not in use (locked) from C:\Windows\System32. Upon reboot at the end of the script, Windows couldn't boot.
I called the PM back, told him why the deployment had failed, and the lab environments would have to be completely re-deployed (they were virtual machines).
This particularly nasty "feature" in Pusher has been long since fixed, but taught some people some valuable lessons. For example, if you are going to copy someones code (the batch file), be sure to copy all the pertinent parts, and that you understand how it works. For me, I learned the value of providing more comprehensive documentation.
I did finish my vacation, though.
Loading Comments ...
Comments
No comments have been added for this post.
You must be logged in to make a comment.