The Great (caching) Purge

Standard
caching-purge

Earlier tonight, myself & a few others on the #osx-server channel in the macadmins.org Slack started to receive alerts from our macOS Servers running a Caching Service.

Sadly, it was not just the harmless “Caching service unavailable” alerts that you see at times.

So what happened?

Well, after getting the error. I checked my home server.

screen-shot-2016-09-29-at-9-15-50-pm

Yep nothing much to see there.. Next I checked the service.

Oh, nothing really to see there either..

screen-shot-2016-09-29-at-9-15-10-pm

Whaaat?

screen-shot-2016-09-29-at-9-16-16-pm

Ok, maybe the log helps clarifies:

ಠ_ಠ

Oh noes..

Sep 29 19:49:46 macmini AssetCache[216]: Request for establishment from https://lcdn-registration.apple.com/lcdn/session failed: HTTP response 400, body "SIGNATURE_VERIFICATION_FAILURE"
Sep 29 19:49:46 macmini AssetCache[216]: Registration rejected.
Sep 29 19:52:02 macmini AssetCache[216]: Removed all 247.93 GB in 14627 assets (including 99.14 GB in 13177 personal [icloud] assets) from the cache because registration failed
Sep 29 19:52:02 macmini AssetCache[216]: Server is suspended pending successful registration. Will re-register in 3300 seconds.
Sep 29 19:52:02 macmini AssetCache[216]: Property list invalid for format: 200 (property lists cannot contain NULL)
Sep 29 19:52:02 macmini AssetCache[216]: Posted alert of type CachingRegistrationRejectedAlert, subject "Caching service unavailable", message "The Caching service is unable to register. Users may continue to download content from other Caching servers or directly from Apple."
Sep 29 20:47:03 macmini AssetCache[216]: Request for establishment from https://lcdn-registration.apple.com/lcdn/session failed: HTTP response 400, body "SIGNATURE_VERIFICATION_FAILURE"
Sep 29 20:47:03 macmini AssetCache[216]: Registration rejected.
Sep 29 20:47:03 macmini AssetCache[216]: Server is suspended pending successful registration. Will re-register in 3300 seconds.
Sep 29 21:14:05 macmini AssetCache[216]: Caching Server shutting down (15)
Sep 29 21:14:05 macmini com.apple.xpc.launchd[1] (com.apple.AssetCache[216]): Service exited with abnormal code: 15

So, not only has the Caching service had an issue connecting to Apple, it’s also shut down & deleted all content from the cache.

What’s next?

At the time of writing, the Caching service cannot be re-enabled.

Attempts to do so, error like the below:

screen-shot-2016-09-29-at-9-47-31-pm

With the following in the Caching service log:

Sep 29 21:47:45 macmini AssetCache[16314]: Caching Server started
Sep 29 21:47:46 macmini AssetCache[16314]: Request for establishment from https://lcdn-registration.apple.com/lcdn/session failed: HTTP response 400, body "SIGNATURE_VERIFICATION_FAILURE"
Sep 29 21:47:46 macmini AssetCache[16314]: Registration rejected.
Sep 29 21:47:46 macmini AssetCache[16314]: Server is suspended pending successful registration. Will re-register in 3600 seconds.
Sep 29 21:47:48 macmini AssetCache[16314]: Caching Server shutting down (15)
Sep 29 21:47:48 macmini com.apple.xpc.launchd[1] (com.apple.AssetCache[16314]): Service exited with abnormal code: 15

This issue was not just affecting my home server, but several customers of ours as well as those folks whom mentioned it in the #osx-server channel in the macadmins.org Slack.

I’ll post an update, when there is one.

For now, this is all green.

I have logged a bug report, & posted on open radar.

Update

See: https://jamfnation.jamfsoftware.com/discussion.html?id=21469#respond