... and removing the bullet yourself.
Backstory
One part of a network I've been caring for uses Cisco WLC 3504's with 1852i APs (802.11ac Wave 2). Moving to new hardware also enabled me to revisit some options the old setup had previously needed to drag forward.
Compard to the previous 1602i's these newer APs supported more recent cryptography such as DTLS 1.2 which was introduced with AireOS 8.3.11[1|2]. DTLS is used to encrypt the CAPWAP tunnel between the WLC and the AP. By default only the mangement / control data is encrypted between the WLC and a CAPWAP AP.
After removing the last 1602i from the network I've bumped the minimum DTLS version to 1.2 on the WLC 3504's: config ap dtls-version dtls1.2
Changing the DTLS version requires a WLC reboot, hence this was done during a maintenance window. Everything came back up as expected and we moved forward.
However I forgot to check the firmware of the remaining spare devices in stock.
Spare devices on old releases
Then recently a office had get better Wi-Fi coverage and one of our spare devices got installed and I was tasked to finish up the network and wireless config. While adding a Cisco AP to a existing WLC is not really challenging I realized that the AP didn't join the AP within the expected time frame.
Checking WLC's logs I saw some DTLS error wich matched the MAC address of the freshly installed AP. Some search indicated that checking the clock on both the WLC and AP could be useful but both AP and WLC successfully synced to the same NTP server.
Licensing was also good, we definitely had enough licenses on the WLC for the AP to join. Connecting a console cable to the AP spilled DTLS errors to the CLI console log to.
Then it dawned on me that I left a note in our documentation that DTLS 1.2 was only supported with AireOS 8.3 - but some of the units in stock (even though bought in late 2018 / early 2019) came up with AireOS 8.2 which definitely didn't support DTLS 1.2 (!)
Usually an AP boots up, detects its WLC, establishes a CAPWAP tunnel and then checks for available firmware versions. If the CAPWAP tunnel cannot be established up, no firmware update via the WLC starts. Then I had to find a way to update the AP firmware to a release was based on AireOS 8.3.112 or newer.
Updating the firmware over TFTP
Ultimately a blog post from Piotr Wojciechowski pointed me to right direction.
I had to download a Lightweight AP Software image from Cisco and the do a manual TFTP update on the AP.
Chosing a firmware
While AireOS WLCs mention the version string in both the AP and WLC firmware as i.e. 8.5.161, the section for Lightweight AP Software only shows an IOS version string (15.3x in my case). Based on the release date I guessed a possibly good version. The image in this example turned out to be for AireOS 8.8 WLCs. As you will see later on, the only thing that mattered is that version is "modern enough".
Updating
Boot the AP then connect a serial console, ignoring the annoying DTLS errors popping up on the console:
! Log in using Cisco / Cisco on factory default 1852i's
! Enter privileged EXEC mode, the default password is also Cisco
!
! Download the new image from a TFTP server reachable by the AP
archive download-sw /reload tftp://172.16.10.1/ap1g4-k9w8-tar.153-3.JI5.tarNote: In my case my particular firmware required "/reload" but didn't need /overwrite as Piotr mentioned in his blog post.
If everything goes well, the AP downloads the image, reboots, updates the secondary partition and boots the updated partition. Once the firmware is new enough it will successfully establish a CAPWAP tunnel with the WLC, compare its current firmware with the one offered by the WLC.
If the firmware doesn't match it will reboot once again to come up with the matching firmware, this is why the exact version used to update the AP via TFTP doesn't matter that much.
Wrapping up
I am not the one able to tell the details between DTLS 1.0 and 1.2. I can RFCs (RFC4347 / RFC6347). DTLS 1.0 dates to 2006 and is based on TLS 1.1. DTLS 1.2 is based on TLS 1.2 and was standardized in 2012. Cisco added support in its wireless platform in 2017.
This article was written in 2020 so neither the protocol was really bleeding edge anymore at that point in time, nor the software using it. Bumping the minimum DTLS version seemed OK to me but as always given that outdated protocols should be retired when possible or when security requires it.
However: The devil was hidden in the details in the form of spare APs that came with a outdated factory firmware (now) unable to join a reasonably modern WLC.