704578 - Install a version of the maintenance service on our Windows test slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(no longer active)]

In order for us to be able to test the maintenance service which is being added in bug 481815, we need a way to get system privileges.  I have a proposal here <https://wiki.mozilla.org/Silent_Update_OS_Dialogs/Automated_testing> about how we can do this.  I've talked to bbondy and bhearsum, and neither told me that it's a horrible idea, so I'm filing this bug.

bbondy said that he can provide bhearsum with the binaries and the install script necessary for this work.  Let me know if there is any way for me to help here.

Thanks!

Comment 1

[[e-mail] (:bhearsum)]

From e-mail:
Hi Ben,

You can find the maintenanceservice bins here:
http://people.mozilla.org/~bbondy/maintenanceservice.zip

The elevated script would need to have all those files in the same directory (I think after downloading them), and then execute:
maintenanceservice_installer.exe

We will probably have a new version later but this one should do for now.
Also, please let me know if there is a consistent directory where firefox is run from.
We need to set a registry key as well but it depends on the path where you are running firefox from.


--

I did a manual install of the service on an XP and W7 slave - both worked correctly. I'm going to look at scripting it now, and getting a scheduled task installed.

Comment 2

[[e-mail] (:bhearsum)]

Working out of https://github.com/bhearsum/opsi-package-sources/tree/updateservice for an OPSI package that will deploy the scheduled task on XP slaves. Windows 7 will use the same scripts, but has to be installed manually.

Comment 3

[[e-mail] (:bhearsum)]

I whipped together a script and OPSI package pretty quickly, but I'm having trouble getting the service to install correctly through OPSI. It does create a task, but not one capable of running, apparently. When I go to the properties of the task, I get told "Failure 0x80070534 no mapping between account names and security IDs was done". When I create the task with the exact same command, on the command line, it works fine at next reboot.

%$#%$#%$#%$#ing windows.

Comment 4

[(no longer active)]

I searched around a bit, and I came across this: https://groups.google.com/group/microsoft.public.windowsxp.general/msg/90be33d3c3a54321?hl=en&pli=1

Is the user who created the scheduled task the same one as the one who's supposed to run it (I assume cltbld)?

Can you try to see if you can get a built-in Windows app to run in the scheduled task without this error?

Feel free to ping me or Brian on IRC and we'll see if we can help.  :-)

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Ehsan Akhgari [:ehsan] from comment #4)
> I searched around a bit, and I came across this:
> https://groups.google.com/group/microsoft.public.windowsxp.general/msg/
> 90be33d3c3a54321?hl=en&pli=1

Thanks.

> Is the user who created the scheduled task the same one as the one who's
> supposed to run it (I assume cltbld)?

Nope -- this needs to be run as SYSTEM. In fact, we already _have_ a scheduled task for something else that runs as SYSTEM without issue.

> Can you try to see if you can get a built-in Windows app to run in the
> scheduled task without this error?

Good idea!

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #5)
> > Can you try to see if you can get a built-in Windows app to run in the
> > scheduled task without this error?
> 
> Good idea!

Interesting, I changed the scheduled task to run notepad.exe instead of this batch file, and had the same issue. The problem seems to be with the task being added through OPSI, which runs as SYSTEM, i think. Everything, including the mozilla service installation task, seems to work fine when done by hand.

Comment 7

[[e-mail] (:bhearsum)]

For fun, I just tried re-installing the _existing_ task through OPSI, and now _it_ doesn't work either. This makes me think that OPSI got changed in some way, or is otherwise broken. So, I'm giving up on deploying this through OPSI and I'll do it by hand. I want to do a bit more testing, then I'm going to throw the script up for review.

Also, I'm blocking deployment on bug 701266, which will be setting up a permanent place for these files. If we deploy before that is done, we'll have to redeploy after it *is* done.

Comment 8

[(no longer active)]

As I understand the article in comment 4, the user who's creating the scheduled task should be the user who's running it.  If the latter needs to be SYSTEM, the former needs to be SYSTEM too.

(Stupid, I know, no question about that!!!)

Comment 9

[[e-mail] (:bhearsum)]

Attachment: script to download and install maintenance service

I FINALLY managed to get the service installed on a win XP box through a scheduled task. Installing the service manually, as the administrator avoided the account name issue previously mentioned. After that, I found that the service STILL wasn't getting installed. By adding a 30 second sleep to the start of the script, I fixed that. I assume this is because ONSTART tasks are run before the network is started. So, here's the complete list of things that need to be done to the XP slaves for this bug, through an ssh session:
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
runas /user:administrator "schtasks /create /tn mozillamaintenance /tr \"c:\\windows\\system32\\cmd.exe /c \\\"c:\\installservice.bat\\\"\" /sc ONSTART /ru SYSTEM"

Once that's done, the machine will install the service automatically at next boot. Currently, this script is looking at https://build.mozilla.org/clobberer/updateservice.zip, and expects that zip file to have the required files unpacked to the current directory. The location of this needs to be updated when https://bugzilla.mozilla.org/show_bug.cgi?id=701266 is fixed. Once that's done, this is ready to deploy on XP.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: installation script, now windows 7 compatible

Got it working on Windows 7, too, with a minor fix to the installation script. Here's the commands used on Windows 7:
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
schtasks /create /tn mozillamaintenance /tr "c:\\windows\\system32\\cmd.exe /c \"c:\\installservice.bat\"" /sc ONSTART /ru SYSTEM

These must be run from a cmd.exe started with "Run as administrator" through VNC. No way to elevate privileges over ssh :(.

Comment 11

[[e-mail] (:bhearsum)]

In addition to the maintenance service, we need:
* The Mozilla Test CA (note, this is one specifically generated for code signing, _not_ this one https://wiki.mozilla.org/MozillaRootCertificate)
* Some registry keys

I'll be attaching both of those things here, along with deployment instructions for everything.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: registry keys

Comment 13

[[e-mail] (:bhearsum)]

Attachment: root certificate

Comment 14

[[e-mail] (:bhearsum)]

XP Install instructions for everything:
Maintenance Service (ssh):
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
runas /user:administrator "schtasks /create /tn mozillamaintenance /tr \"c:\\windows\\system32\\cmd.exe /c \\\"c:\\installservice.bat\\\"\" /sc ONSTART /ru SYSTEM"

Registry keys (ssh)
wget -O keys.reg --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577617
regedit /s keys.reg

Root Cert (VNC):
wget -O MozRoot.cert --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619
* Browse to download location
* Right click cert, choose "Install certificate
* Choose "Trusted Root Certificate Authorities" as the install location

Comment 15

[[e-mail] (:bhearsum)]

Install instructions for Windows 7 (all must be done over VNC, in a cmd.exe window that was started with "Run as Administrator"):
Maintenance Service:
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
schtasks /create /tn mozillamaintenance /tr "c:\\windows\\system32\\cmd.exe /c \"c:\\installservice.bat\"" /sc ONSTART /ru SYSTEM

Registry keys:
wget -O keys.reg --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577617
regedit /s keys.reg

Root Cert (VNC):
wget -O MozRoot.cert --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619
* Browse to download location
* Right click cert, choose "Install certificate
* Choose "Trusted Root Certificate Authorities" as the install location

Comment 16

[[e-mail] (:bhearsum)]

Small correct, MozRoot.crt instead of MozRoot.cert. XP instructions are now:
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
runas /user:administrator "schtasks /create /tn mozillamaintenance /tr \"c:\\windows\\system32\\cmd.exe /c \\\"c:\\installservice.bat\\\"\" /sc ONSTART /ru SYSTEM"
wget -O keys.reg --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577617
regedit /s keys.reg
wget -O MozRoot.cert --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619

* Browse to download location
* Right click cert, choose "Install certificate
* Choose "Trusted Root Certificate Authorities" as the install location

Windows 7 are:
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
schtasks /create /tn mozillamaintenance /tr "c:\\windows\\system32\\cmd.exe /c \"c:\\installservice.bat\"" /sc ONSTART /ru SYSTEM
wget -O keys.reg --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577617
regedit /s keys.reg
wget -O MozRoot.cert --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619
* Browse to download location
* Right click cert, choose "Install certificate
* Choose "Trusted Root Certificate Authorities" as the install location

Comment 17

[[e-mail] (:bhearsum)]

Ugh, forgot to update the wget command:
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
runas /user:administrator "schtasks /create /tn mozillamaintenance /tr \"c:\\windows\\system32\\cmd.exe /c \\\"c:\\installservice.bat\\\"\" /sc ONSTART /ru SYSTEM"
wget -O keys.reg --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577617
regedit /s keys.reg
wget -O MozRoot.crt --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619

* Browse to download location
* Right click cert, choose "Install certificate
* Choose "Trusted Root Certificate Authorities" as the install location

Windows 7 are:
cd c:\
wget -O installservice.bat --no-check-certificate http://people.mozilla.com/~bhearsum/installservice.bat
schtasks /create /tn mozillamaintenance /tr "c:\\windows\\system32\\cmd.exe /c \"c:\\installservice.bat\"" /sc ONSTART /ru SYSTEM
wget -O keys.reg --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577617
regedit /s keys.reg
wget -O MozRoot.crt --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619
* Browse to download location
* Right click cert, choose "Install certificate
* Choose "Trusted Root Certificate Authorities" as the install location

Comment 18

[[e-mail] (:bhearsum)]

XP machines (001 through 062) are all up to date with this except:
- 045, which is dead
- ref, which is offline. I'll update this one when it comes back up.

Catlee has already finished talos-r3-w7-002 through 014, and currently working on 015 through 023. In addition to those, 024 through 064 need to be updated, as well as talos-r3-w7-ref, t-r3-w764-ref, and t-r3-w764-001 through 005.

Comment 19

[[e-mail] (:bhearsum)]

Ref machines and all w764 machines are done now. Catlee finished with 015 through 023, nthomas is working on 025 through 035.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

talos-r3-w7-025 thru to 035 are done.

Comment 21

[[e-mail] (:bhearsum)]

OKAY. The following slaves are all done:
talos-r3-xp-ref, 001 through 062 except 045 (which is dead)
talos-r3-w7-ref, 001 through 064 except 024, 036, 053 (which are all down at the moment). I've left comments in their bugs to update them when they come back up.
t-r-w764-ref, 001 through 005

Additionally, Amy informed me that the r3-snow machines that are getting recycled will have the old image, so they need to be updated when they come up, before they hit the pool. They are:
talos-r3-w7-066
talos-r3-w7-067
talos-r3-w7-068
talos-r3-w7-069
talos-r3-w7-070
talos-r3-w7-071
talos-r3-w7-072
talos-r3-w7-073
talos-r3-w7-074
talos-r3-w7-075
talos-r3-w7-076
talos-r3-w7-077
talos-r3-w7-078
talos-r3-w7-079
talos-r3-xp-063
talos-r3-xp-064
talos-r3-xp-065
talos-r3-xp-066
talos-r3-xp-067
talos-r3-xp-068
talos-r3-xp-069
talos-r3-xp-070
talos-r3-xp-071
talos-r3-xp-072
talos-r3-xp-073
talos-r3-xp-074
talos-r3-xp-075

I put a comment in bug 683734 noting it, too.

Comment 22

[[e-mail] (:bhearsum)]

Attachment: final version of install script

Forgot to update this bug with the final version of install script used.

Comment 23

[[e-mail] (:bhearsum)]

All current production slaves are updated at this point. Slaves that are currently down have notes in their bugs to have this applied when they come back up. Same goes for the recycled r3-snow machines. We're done here now.

Comment 24

[Justin Wood (:Callek)]

Comment on attachment 577940 [details]
final version of install script

>@echo off
>
>SET workdir=c:\updateservice
>SET mozillabuild=d:\mozilla-build
>SET mozilladrive=d:
>IF NOT EXIST %mozillabuild%\nul (
>    SET mozillabuild=c:\mozilla-build
>    SET mozilladrive=c:
>)
>
...

>%mozilladrive%
>cd %workdir%
>

Note that in the case of a mozilla-build in d:\ you're not actually using the workdir, since if your current drive is d:\ a |cd c:\foo| will change the working dir when you switch to c: but will leave your cwd as whatever it was to begin with on d:\

And then the following code won't use that work directory for this. We do however get lucky and the remaining code there works fine at just about any directory, but you're not using the clean area you mean to.

Comment 25

[[e-mail] (:bhearsum)]

There's at least one issue that needs to be corrected here:
The registry keys deployed were based on the assumption that xpcshell tests run updater.exe out of c:\talos-slave\talos-data\firefox\updater.exe, which is _not_ the case. They actually run it out of c:\talos-slave\test\build\firefox\updater.exe, it seems. I managed to confused talos paths with unittests paths when looking at this.

We'll need to fix this. Ehsan also tells me that we may need a new version of the maintenance service, and the installservice.bat script will need an update to handle that. Ehsan, is 'maintenanceservice_installer.exe forceinstall' the correct syntax to overwrite the existing service?

Comment 26

[(no longer active)]

(In reply to Ben Hearsum [:bhearsum] from comment #25)
> There's at least one issue that needs to be corrected here:
> The registry keys deployed were based on the assumption that xpcshell tests
> run updater.exe out of c:\talos-slave\talos-data\firefox\updater.exe, which
> is _not_ the case. They actually run it out of
> c:\talos-slave\test\build\firefox\updater.exe, it seems. I managed to
> confused talos paths with unittests paths when looking at this.

Actually we ended up deciding that we will use the existing invalid hash as a "fallback".  This is necessary because I realized that every test uses its own app dir, which is what we create the hashes from, so we would need one hash per test if we were to use the real hashes for testing, and that is a pain.

> We'll need to fix this. Ehsan also tells me that we may need a new version
> of the maintenance service, and the installservice.bat script will need an
> update to handle that. Ehsan, is 'maintenanceservice_installer.exe
> forceinstall' the correct syntax to overwrite the existing service?

Let's hold off on this.  I have not confirmed that we need a new version of the service yet.  I will update you either late tonight or tomorrow.

Comment 27

[(no longer active)]

So, two things need to happen here.

1. A new version of the maintenance service needs to be deployed to the test machines.  I will attach a zip package containing the binaries.

2. The certificate for the fake CA needs to be installed in the cert root store for the Local Computer.  Here's the steps to do this:

a. Open mmc.exe from an admin cmd.exe shell.
b. File > Add/Remove Snap-in.
c. Double click Certificates.
d. Select "Computer account".  Hit Next until the wizard is done.
e. Go to "Trusted Root Certificate Authorities\Certificates", and import the CA cert.

Thanks!

Comment 28

[[e-mail] (:bhearsum)]

(In reply to Ehsan Akhgari [:ehsan] from comment #27)
> So, two things need to happen here.
> 
> 1. A new version of the maintenance service needs to be deployed to the test
> machines.  I will attach a zip package containing the binaries.
> 
> 2. The certificate for the fake CA needs to be installed in the cert root
> store for the Local Computer.  Here's the steps to do this:
> 
> a. Open mmc.exe from an admin cmd.exe shell.
> b. File > Add/Remove Snap-in.
> c. Double click Certificates.
> d. Select "Computer account".  Hit Next until the wizard is done.
> e. Go to "Trusted Root Certificate Authorities\Certificates", and import the
> CA cert.

We talked about 2) in the context of Windows 7 on Friday. Any idea if this applies to XP, too? What can I test to see if it does?

Comment 29

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #28)
> (In reply to Ehsan Akhgari [:ehsan] from comment #27)
> > So, two things need to happen here.
> > 
> > 1. A new version of the maintenance service needs to be deployed to the test
> > machines.  I will attach a zip package containing the binaries.
> > 
> > 2. The certificate for the fake CA needs to be installed in the cert root
> > store for the Local Computer.  Here's the steps to do this:
> > 
> > a. Open mmc.exe from an admin cmd.exe shell.
> > b. File > Add/Remove Snap-in.
> > c. Double click Certificates.
> > d. Select "Computer account".  Hit Next until the wizard is done.
> > e. Go to "Trusted Root Certificate Authorities\Certificates", and import the
> > CA cert.
> 
> We talked about 2) in the context of Windows 7 on Friday. Any idea if this
> applies to XP, too? What can I test to see if it does?

I just poked around mmc on XP and found that there's no "Computer account" in the snap-in wizard, so we'll need some sort of altered instructions here.

Comment 30

[(no longer active)]

Attachment: New version of the maintenance service

This is the version of the service which needs to be installed on our test machines.

Comment 31

[[e-mail] (:bhearsum)]

Attachment: moar bettar version of installservice.bat

My previous version tried to reference the %mozillabuild% variable before it was assigned, which means that we never actually slept before trying to wget - resulting in almost always failing to install. Need to deploy this updated version, as well as the updated version of the maintenance service.

So, we need to:
1) Put the new zip file on build.mozilla.org
2) For XP, install the new version of installservice.bat, which can be done over ssh.
3) For Windows 7, install the new version of the script. Here, this must be done through the GUI. Detailed instructions to be posted.

We should probably fix up the root certificates at the same time. I'll post full instructions for what needs to be done to both platforms ASAP.

Comment 32

[Rail Aliiev [:rail]]

Attachment: MMC shortcut to get to needed location

I created this file by saving MMC state after switching to Trusted Root Certificates

Comment 33

[[e-mail] (:bhearsum)]

Windows 7 instructions:
* Open cmd with "Run as Administrator"
* Run the following:
c:
cd c:\
wget -O installservice.bat --no-check-certificate https://bug704578.bugzilla.mozilla.org/attachment.cgi?id=579099
wget -O add_cert.msc https://bugzilla.mozilla.org/attachment.cgi?id=579119
wget -O MozRoot.crt --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619
* Browse to c:
* Double click add_cert.msc
* From menu: Action -> All Tasks -> Import...  launches Certificate Import Wizard
* Click Next
* Browse and use C:\MozRoot.crt
* Next, Next, Finish
* Close the MMC window
* Restart


XP instructions still to come.

Comment 34

[[e-mail] (:bhearsum)]

Ugh, make that:
Windows 7 instructions:
* Open cmd with "Run as Administrator"
* Run the following:
c:
cd c:\
wget -O installservice.bat --no-check-certificate https://bug704578.bugzilla.mozilla.org/attachment.cgi?id=579099
wget -O add_cert.msc --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=579119
wget -O MozRoot.crt --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=577619
start add_cert.msc
* From menu: Action -> All Tasks -> Import...  launches Certificate Import Wizard
* Click Next
* Browse and use C:\MozRoot.crt
* Next, Next, Finish
* Close the MMC window
* Restart

Comment 35

[(no longer active)]

(In reply to Ben Hearsum [:bhearsum] from comment #29)
> (In reply to Ben Hearsum [:bhearsum] from comment #28)
> > (In reply to Ehsan Akhgari [:ehsan] from comment #27)
> > > So, two things need to happen here.
> > > 
> > > 1. A new version of the maintenance service needs to be deployed to the test
> > > machines.  I will attach a zip package containing the binaries.
> > > 
> > > 2. The certificate for the fake CA needs to be installed in the cert root
> > > store for the Local Computer.  Here's the steps to do this:
> > > 
> > > a. Open mmc.exe from an admin cmd.exe shell.
> > > b. File > Add/Remove Snap-in.
> > > c. Double click Certificates.
> > > d. Select "Computer account".  Hit Next until the wizard is done.
> > > e. Go to "Trusted Root Certificate Authorities\Certificates", and import the
> > > CA cert.
> > 
> > We talked about 2) in the context of Windows 7 on Friday. Any idea if this
> > applies to XP, too? What can I test to see if it does?
> 
> I just poked around mmc on XP and found that there's no "Computer account"
> in the snap-in wizard, so we'll need some sort of altered instructions here.

Well, I managed to install the CA cert under the Computer account the same way that I did on 7, and I verified that by doing that and also using the new version of the service, the unit tests pass successfully on XP as well.

Comment 36

[Rail Aliiev [:rail]]

I updated installservice.bat on w7 and xp talos slaves (see exceptions below). I also installed Mozilla Fake CA certificate on these slaves.

For XP I used a different MMC state file (attachment incoming).

The following slaves still needs to be updated:

w7:
talos-r3-w7-011: bug 707234

xp:
talos-r3-xp{1,2,3}: loaned
talos-r3-xp-045: bug 661377

I also accidentally powered off talos-r3-xp-019 (instead of reboot) :(

If we are Ok with the installation, we also need to install everything on the ref machines and take a snapshot of them.

Comment 37

[Rail Aliiev [:rail]]

Attachment: XP MMC file

Comment 38

[[e-mail] (:bhearsum)]

Rail, you are incredible. Thanks for going through all of these.

Comment 39

[(no longer active)]

We are now getting green test runs on Elm.  :-)  So I'm gonna call this fixed!

Thanks a bunch for your help guys!

Comment 40

[[e-mail] (:bhearsum)]

I updated the ref machines (talos-r3-xp-ref, talos-r3-w7-ref, t-r3-w764-ref) and the 5 w764 machines actually doing tests.

(In reply to Rail Aliiev [:rail] from comment #36)
> w7:
> talos-r3-w7-011: bug 707234

Updated this.

> xp:
> talos-r3-xp-045: bug 661377

This one is still offline.

Also to be updated still: the recycled snow leopard slaves. This is tracked in bug 683734.