Upgrading to IBM MQ v10? Watch out for this MQ operator pitfall!

Upgrading to IBM MQ v10 with the MQ operator pitfall

Recently we've starting migrating customers from IBM MQ version 9.4 to version 10. Not only was it great to finally get many of the additional features added on top of 9.4.0.0 through CD releases in a new LTS, but it also added to lot of security enhancements. Most MQ migrations are very simple in nature, because it isn’t common for features to get deprecated, but we did face a very niche issue with one setup.

As mentioned, security was a big part of what sets MQ v10 apart from its predecessor as IBM decided to force some stricter security rules by default. If a customer kept up somewhat with the latest encryption requirements however, even those stricter requirements aren’t a big issue. The problem we had however, required a very specific set of circumstances to be met:

MQ is running on containers, managed through the MQ Operator.
The MQ web component needs to be enabled and used.
You need to have a custom certificate present on the MQ web.
You used a workaround for its keystore.

That last one is key. For those having set up IBM MQ on containers before operator version 3.5.0 with these requirements, you will have noticed that there was no real option available in the operator definition to upload its keystore with your own company private key and possibly trusted certificates. Sure, you could manually add the keystore to the pod or update the base image, but there was an easy workaround available that allowed you to manage it without anything custom.

That method was to just add the keystore to the queue manager KPI section as an additional key and MQ would ignore it for its own keystore, but it would still add the keystore on the system. From there, you could just reference it in the mqwebuser.xml and you had a working setup with your own certificates being presented through the console and rest API.

This system was still valid up until the latest 9.4 LTS, but when we updated to v10, the queue manager pods wouldn’t start, throwing the following error during pod creation and the queue manager wouldn’t start.

failed to cast private key to comparable type

We knew that MQ had upgraded requirements to its supported ciphers, so at first, we thought the ones we used didn’t comply with the new, stricter standards, but that wasn’t the case. After some investigation, it turned out that MQ no longer ignored the additional KPI entry and just added it, rather it tried to treat it is a proper queue manager key and because it was a JKS, it refused to validate it and in turn, wouldn’t start the queue manager.

Before: MQ Operator < 3.5.0 workaround

The web keystore (JKS) is added as an extra entry under spec.pki.keys. It gets mounted in the pod. It is NOT the queue manager's own key, it's only referenced by mqwebuser.xml.

apiVersion: mq.ibm.com/v1beta1
kind: QueueManager
metadata:
  name: example-qm
spec:
  version: "9.4.0.26-r1"
  license:
    accept: true
    license: L-APIG-BMJJBM
    use: Production
  web:
    enabled: true
  pki:
    keys:
      - name: qmkey
        secret:
          secretName: qm-tls-secret
          items:
            - tls.key
            - tls.crt
      # Workaround: web keystore piggy-backed into the PKI keys section
      - name: webkey
        secret:
          secretName: mq-web-keystore-secret
          items:
            - mqwebKeystore.jks

The Fix

The good news is that the fix is very simple and doesn’t feel like a hacky workaround anymore. Updates to the MQ Operator have been made and since version 3.5.0 you can add custom ConfigMaps and Secrets to the pod. Just remove the MQ web keystore from the main KPI section, add it as a file reference to the secret you were using before and update the mqwebuser.xml to point to the new location. Once done, your queue manager will only find its own key and provided it adheres to the MQ minimum requirements, your v10 queue manager will start and the MQ console will still have its own keystore with the keypair and trusted certificates if needed.

After: MQ Operator >= 3.5.0

The web keystore is removed from spec.pki.keys entirely. It is instead mounted as a file reference via spec.queueManager.files, mqwebuser.xml will also need to be updated to match the mountPath.

apiVersion: mq.ibm.com/v1beta1
kind: QueueManager
metadata:
  name: example-qm
spec:
  version: "10.0.0.5-r1"
  license:
    accept: true
    license: L-APIG-BMJJBM
    use: Production
  web:
    enabled: true
  pki:
    keys:
      - name: qmkey
        secret:
          secretName: qm-tls-secret
          items:
            - tls.key
            - tls.crt
    files:
      - secret:
          name: mq-web-keystore-secret
          items:
            - item: mqwebKeystore.jks
              mountPath: /reference/in/webuser/xml

←

back to Blogs