Wix and msi installers: downgrade a library during a msi upgrade

Here's a tricky one I came across at work:

If you look closely, with an upgrade of OurProduct, we downgrade a library. So what? you say. (Hint: try not to ever do a library version downgrade! It's painful.)

Well, it turns out, that with xcopy, this problem would easily be solved. Your forfeit all the other kinda cool things that msi can do, though. Not the path we'd like to go.

So, what to do?

Just to show the baseline: this is the component of the initial OurProduct:

    <Component Id="Newtonsoft.Json" Guid="SOMEGUID-42FD-44B5-8C93-C245F85DF84E">
      <File Source="Newtonsoft.Json.dll" KeyPath="yes" /> <!-- version 5.0 -->
    </Component>

How do we downgrade that?
Ok, easy, just pack in the new lib, take the old component guid, and do configure major upgrades just like this:

<MajorUpgrade
     DowngradeErrorMessage="A newer version of [ProductName] is already installed." 
     Schedule="afterInstallValidate" />

this is the default scheduling used by Wix now, it will remove the old product first (that's the 'RemoveExistingProducts' Action), and then install the new product. That means, it removes this Newtonsoft.Json lib before it puts in the version of the upgrade.

Try #1 - fail

We also pack in the new version

<!-- Try #1 -->
    <Component Id="Newtonsoft.Json" Guid="SOMEGUID-42FD-44B5-8C93-C245F85DF84E">
      <File Source="Newtonsoft.Json.dll" KeyPath="yes" /> <!-- version 4.5 -->
    </Component>

same guid, same thing, lower version.
And it doesn't work:
Action start 17:32:45: CostFinalize.
.....
MSI (c) (38:C0) [17:32:45:977]: Disallowing installation of component: {SOMEGUID-42FD-44B5-8C93-C245F85DF84E} since the same component with higher versioned keyfile exists
Action ended 17:32:45: CostFinalize. Return value 1.
What please? Why does it exist?

As you can see, the msi plans the installation of components in the CostFinalize Action, which is pretty early in the whole InstallExecute sequence. And it's even before the first possible scheduling location of the removal of the old product. CostFinalize runs before InstallValidate. Damn.

What did I have on the system, after this upgrade above finished? The file was gone: the upgrade says it won't install it, the previous version removes it, the upgrade will not install it again => the file is gone. Solution? Repair the new product. But then it'd be easier to uninstall the previous product, and then install the new version. I looked on StackOverflow for help: http://stackoverflow.com/questions/15138731/wix-major-upgrade-not-installing-all-files, but the first answer didn't work for me. The file was even left in the higher version (5.0).

Try #2 - fail again

Ok, let's give it a new guid, then the old component is removed

<!-- Try #2, new guid!! -->
    <Component Id="Newtonsoft.Json" Guid="SOMEGUID-DE0F-4CC9-BC16-2FF543B2FA90">
      <File Source="Newtonsoft.Json.dll" KeyPath="yes" /> <!-- Version 4.5 -->
    </Component>

Well, we will still get
Disallowing installation of component: {SOMEGUID-DE0F-4CC9-BC16-2FF543B2FA90} since the same component with higher versioned keyfile exists
What please? This component is supposed to be new!?!
What the windows installer does here is compare keyfile paths (which probably means it compares the path of the key resource). So it sees the resource already lying there, in a higher version, and doesn't want to overwrite it.

Try #3 - finally succeed

In the end, I found out that there's still a chance how you can do it, but it's not nice:

<!-- Try #3, with still another guid -->
    <Component Id="Newtonsoft.Json.Fix" Guid="SOMEGUID-A46A-4C3E-801A-6F79B16CD0B4">
      <File Source="CarryOn.dll" KeyPath="yes" />
      <File Source="Newtonsoft.Json.dll" />
    </Component>

This works. It's not my idea, I found it somewhere, kudos to whoever wrote it.
This will make the installer look for the CarryOn dll, which can be any other dll where you go for higher versions with upgrades. The old installer will remove the old, higher versioned, Newtonsoft.Json, and the new installer will place the new, lower versioned, library.

There seem to be other answers to this problem out there. I didn't try them. They seem to require InstallShield, some other wix control elements like RemoveFile, scheduling of the RemoveExistingPRoducts to before CostFinalize or they even manipulate the msi directly by convincing it that the lib it has actually has a higher version. The latter will probably just create problems later:
http://stackoverflow.com/questions/30377613/downgrade-file-in-majorupgrade (schedule after CostFinalize)
http://stackoverflow.com/questions/4227456/windows-installer-deletes-versioned-file-during-product-upgrade-instead-of-down (schedule after CostFinalize)
http://stackoverflow.com/questions/14122136/how-can-i-make-sure-a-downgraded-library-is-installed-by-my-msi (hack the msi database)
http://stackoverflow.com/questions/25311969/how-to-downgrade-a-third-party-file-in-a-wix-msi (spoiler: the answer does not work)

It seems, that this is a seldom occurring problem, which doesn't really concern those who are in charge of the msi installer itself. So, rather try to not run into this problem. If you do custom compiles of open source stuff, maybe even chose to take a lower version than the current release which you probably end up using some time later.