Sam Pointer

When I’m on a Windows box and I want to sync two directories (remote or local), I use this little one-liner:

@echo off
xcopy %1 %2 /M /E /Y /Z

The switches are:

• M: Copy only files with the archive bit set
• E: Copy directories and subdirectories, even empty ones
• Y: Supress prompting for overwrites
• Z: Copy in network restartable mode

The %1 and %2 obviously denote parameter arguments when called from a batch file, so when saved as ‘file_sync.bat’:

C:\>file_sync.bat c:\somedirectory \\someserver\sometargetdir

Will sync the local directory with the remote share. When files are modified or added, Windows sets the NTFS archive bit. When run again the script will only copy the new or amended files.

I’ve you’ve landed at this page I’ll bet my bottom dollar that more than one of the following is true, and some of these you don’t even know yet:

• You’re running Sun’s Glassfish or Sun Java System Application Server 9.1 (possibly the EE or Enterprise edition)
• You’re on some flavour of Windows Server
• You’re attempting an upgrade from a previous version - most likely 8.x to 9.1. This upgrade is more than likely an installation alongside an existing one, rather than an in over-the-top installation
• You have some kind of Mozilla product on the machine, and that product uses encryption in some way.

I recently attempted to install Sun Java System Application Server 9.1 EE (SJSAS) onto an aging box alongside an existing copy of 8.2. The reasoning behind this was the usual blather about back-out plans and minimising risk. I had real problems with the thing: no matter what I did it would always complain with the error:

CLI156  Could not start the domain domain1

The stack trace in the server.log looked something like this:

[#|2009-02-26T22:51:04.515+0100|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=10;_ThreadName=main;_RequestID=b3263e27-5cb1-4f58-8ec8-c9c2b55724a9;|java.lang.reflect.InvocationTargetException 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
        at java.lang.reflect.Method.invoke(Method.java:597) 
        at com.sun.enterprise.server.PELaunch.main(PELaunch.java:415) 
Caused by: java.lang.NoClassDefFoundError: sun/security/pkcs11/SunPKCS11 
        at java.lang.Class.forName0(Native Method) 
        at java.lang.Class.forName(Class.java:169) 
        at com.sun.enterprise.pluggable.PluggableFeatureFactoryBaseImpl.invoke(PluggableFeatureFactoryBaseImpl.java:84) 
        at $Proxy0.getSecuritySupport(Unknown Source) 
        at com.sun.enterprise.security.SecurityUtil.getSecuritySupport(SecurityUtil.java:364) 
        at com.sun.enterprise.security.SSLUtils.(SSLUtils.java:102)
    at com.sun.enterprise.security.SecurityLifecycle.onInitialization(SecurityLifecycle.java:101)
    at com.sun.enterprise.server.ApplicationServer.onInitialization(ApplicationServer.java:262)
    ...  

So, at this point I did what any self-respecting production engineer would do: I bunged SEC8001 into Google. Take a look at what the SJSAS 9.1 Error Message Reference says. Yup, you read it right: "Cause TBD, Solution TBD". Nice one Sun. Excellent.

I tried a few random scattershot things, banking on hunches from a decade of dealing with this kind of stuff:

• Check how the classpath was constructed
• Remove the JRE that some numpty had installed alongside the JDK
• Reinstall the thing using its own JDK rather than the system one
• Make sure there was nothing stupid in the system %PATH%
• Check the asenv.bat to ensure that everything seemed right and proper

It seemed that some digging was in order.

Taking a look at the stack trace, some investigation of PKCS11 seems to be the next step. I’ll save you the leg-work and just link straight to the Wikipedia page. So, we’re dealing with encryption - and so working again on a hunch I widened my search to include NSS - as we’re talking encryption, cross-platform, Sun and Java.

A few minutes of research lead me to this gem:

On Windows: Application Server fails to start due to NSS initialization failure. (6618442)

 

Description

In some cases, Application Server (enterprise profile) might not start on Windows when PKCS11 fails to initialize.

Solution

Remove softokn3.dll from the Windows system path.

 

What on earth is softokn3.dll? First hit on Google says this:

softokn3.dll is a NSS PKCS #11 Library from Mozilla Foundation belonging to Network Security Services

Things are starting to make sense now. Confident I’ve got this thing sussed, I renamed the copy of softokn3.dll I find it %SYSTEMROOT%\system32 and restart SJSAS. Nope.

When I search the file system I found 7 copies of this .dll. All of these were 300ish K, apart from the two installed with SJSAS 9.1which were 200-odd K. Checking the %PATH% it seems that these versions of softokn3.dll had made it into the search list before the intended 9.1 - remember automatic %PATH% additions are appended, and if you’re upgrading you will most likely have bits of previous installations still in there.

However, there’s one last trick to getting all of this to work. It’s not good enough to just rename and/or eliminate these existing copies of the .dll from your %PATH%. Your domain still will not start. They key issue is that they cannot be present during the installation. If they are present during the installation it seems to screw up the keystore(s) for the admin and master keys. No matter what you do post-installation, correct .dlls or not, you won’t easily be able to examine the contents of the keystore and you won’t be able to boot your domain. The domain upgrade wizard failing due to invalid credentials that you know are correct is another sign of this problem.

So, we’re down to:

1. Remove all previous 300-ish K copies of softokn3.dll from the %PATH% by renaming them all to .dlx and amending the path entries.
2. Uninstall and reinstall SJSAS 9.1 EE
3. Confirm the domain starts OK
4. Unpick the mess you’ve made of all of your other applications that rely on this .dll, including your "backout" 8.x instance, in my case.

You might have remember that I mentioned above that you could be running some kind of Mozilla product that has nothing to do with SJSAS/Glassfish, and still be experiencing this problem. Every version of Firefox I’ve ever come across includes this .dll at one version level or another, for example. If the Firefox installation root is in your path before SJSAS you’ll be for it too.

Even though I’ve fixed this, there’s something tempting me to run the Sysinternals FileMon/ListDLLS/Process Explorer on the thing to get a measure of what’s going on during the installation. If I get enough time in the next couple of days I’ll do so and post my results.

This post documents my attempt to get PAR and PAR::Packer running under ActiveState Perl on Windows XP SP3 in order to produce a native Windows executable of a Perl script. We use this to compile some custom cross-platform Nagios modules.

The first key thing is to start with ActiveState build 5.10.0.1004 or greater, as this fixes a YAML parsing problem that prevents CPAN modules from being installed. PAR::Packer isn’t in the ActiveState repository, so we’ll need to use CPAN to get some of the bits that we need.

• ActiveState Perl

Once that’s done its thing, you’ll need to grab a copy of Microsoft Nmake 1.5. Running the .exe will unpack nmake.exe, nmake.err and README.TXT into the root of C:. You will need to copy these three files into c:\perl\bin. Nmake is needed to build CPAN modules.

• Nmake 1.5

At this point I used the ActiveState Perl Package Manager to update all of the modules that had updates, also electing to install any packages required to satisfy dependencies.

Next, I used CPANPLUS to install PAR::Packer, as it’s not in the default ActiveState repository used by the Perl Package Manager. From a prompt:

cpanp
...
i PAR::Packer

confirming the installation of the module dependencies. At this point you might want to put the kettle on as there are a bunch of build/test tasks to run whilst all of the modules install.

It was here that the build failed for me. I realised I’d neglected to install MinGW to do the native binary compliation, unless you’ve got Microsoft Visual Studio 6.0 that is - in which case it is preferred.

• MinGW one-click installer

You will need to manually select to install the g++ component, along with the base install. Once this is installed, add C:\MinGW\bin (assuming you took the defaults) to your system %PATH%, close and reopen your command prompt to get the new value, and re-try the cpanp steps.

This should give you a completed installation. Just to check:

pp
...
Set up gcc environment - 3.4.5 (mingw-vista special r3)
C:\Perl\site\bin/pp: No input files specified

Ready to go.

pp -o myprogram.exe myprogram.pl