The Great (caching) Purge

Standard
caching-purge

Earlier tonight, myself & a few others on the #osx-server channel in the macadmins.org Slack started to receive alerts from our macOS Servers running a Caching Service.

Sadly, it was not just the harmless “Caching service unavailable” alerts that you see at times.

So what happened?

Well, after getting the error. I checked my home server.

screen-shot-2016-09-29-at-9-15-50-pm

Yep nothing much to see there.. Next I checked the service.

Oh, nothing really to see there either..

screen-shot-2016-09-29-at-9-15-10-pm

Whaaat?

screen-shot-2016-09-29-at-9-16-16-pm

Ok, maybe the log helps clarifies:

ಠ_ಠ

Oh noes..

Sep 29 19:49:46 macmini AssetCache[216]: Request for establishment from https://lcdn-registration.apple.com/lcdn/session failed: HTTP response 400, body "SIGNATURE_VERIFICATION_FAILURE"
Sep 29 19:49:46 macmini AssetCache[216]: Registration rejected.
Sep 29 19:52:02 macmini AssetCache[216]: Removed all 247.93 GB in 14627 assets (including 99.14 GB in 13177 personal [icloud] assets) from the cache because registration failed
Sep 29 19:52:02 macmini AssetCache[216]: Server is suspended pending successful registration. Will re-register in 3300 seconds.
Sep 29 19:52:02 macmini AssetCache[216]: Property list invalid for format: 200 (property lists cannot contain NULL)
Sep 29 19:52:02 macmini AssetCache[216]: Posted alert of type CachingRegistrationRejectedAlert, subject "Caching service unavailable", message "The Caching service is unable to register. Users may continue to download content from other Caching servers or directly from Apple."
Sep 29 20:47:03 macmini AssetCache[216]: Request for establishment from https://lcdn-registration.apple.com/lcdn/session failed: HTTP response 400, body "SIGNATURE_VERIFICATION_FAILURE"
Sep 29 20:47:03 macmini AssetCache[216]: Registration rejected.
Sep 29 20:47:03 macmini AssetCache[216]: Server is suspended pending successful registration. Will re-register in 3300 seconds.
Sep 29 21:14:05 macmini AssetCache[216]: Caching Server shutting down (15)
Sep 29 21:14:05 macmini com.apple.xpc.launchd[1] (com.apple.AssetCache[216]): Service exited with abnormal code: 15

So, not only has the Caching service had an issue connecting to Apple, it’s also shut down & deleted all content from the cache.

What’s next?

At the time of writing, the Caching service cannot be re-enabled.

Attempts to do so, error like the below:

screen-shot-2016-09-29-at-9-47-31-pm

With the following in the Caching service log:

Sep 29 21:47:45 macmini AssetCache[16314]: Caching Server started
Sep 29 21:47:46 macmini AssetCache[16314]: Request for establishment from https://lcdn-registration.apple.com/lcdn/session failed: HTTP response 400, body "SIGNATURE_VERIFICATION_FAILURE"
Sep 29 21:47:46 macmini AssetCache[16314]: Registration rejected.
Sep 29 21:47:46 macmini AssetCache[16314]: Server is suspended pending successful registration. Will re-register in 3600 seconds.
Sep 29 21:47:48 macmini AssetCache[16314]: Caching Server shutting down (15)
Sep 29 21:47:48 macmini com.apple.xpc.launchd[1] (com.apple.AssetCache[16314]): Service exited with abnormal code: 15

This issue was not just affecting my home server, but several customers of ours as well as those folks whom mentioned it in the #osx-server channel in the macadmins.org Slack.

I’ll post an update, when there is one.

For now, this is all green.

I have logged a bug report, & posted on open radar.

Update

See: https://jamfnation.jamfsoftware.com/discussion.html?id=21469#respond

7 thoughts on “The Great (caching) Purge”

  1. Mark Snowdon

    I noticed the same thing about an hour ago today. There was a server.app update available and once applied, the caching service started working again.

  2. Curt Laven

    Same here, all 5 of my caching servers are stopped and empty. LAME I received an automatic email at 1:50 PM CST from our primary caching server.

    “Caching service unavailable
    The Caching service is unable to register. Users may continue to download content from other Caching servers or directly from Apple”

  5. Ed Lund

    All,

    Yesterday afternoon, the caching server registration service at Apple had some difficulty.

    Caching Servers re-register themselves every 55 minutes with Apple. This is done so that if a server fails to re-register, Apple stops re-directing clients to the local caching server for app updates and downloads. When the registration service started having difficulty, some caching servers in the world were not able to re-register themselves and so the caching server service on those servers stopped. In some cases, for reasons I don’t know, the servers then purged their cache as well. I know mine here in my home office did.

    The good news: The registration service is now functioning normally again. It appears that for most people, the servers that were not able to re-register themselves kept trying and were able to resume without any intervention.

    I have had reports that some servers didn’t automatically restart the caching server service. That was the case with my server here in my home office. I suspect this has to do with me trying to manually restart it after I got notified that it stopped.

    Here’s your homework: Double check that your caching servers are indeed running as they should be.

    For the future: If you ever see a problem with your caching servers being able to register themselves, please contact AppleCare and let them know right away. The sooner AppleCare is aware that there is an issue, the sooner that our engineers can figure out what the problem is and get it resolved.

    —Pete


    Pete Markham, Systems Engineer
    Apple Education
    Apple Inc.
    (763) 218-0949
    http://www.apple.com/education

  6. Pingback: Apple 17.0.0.0/8 Services – Richard Purves

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.