A week after the recent updated Terms & Conditions, we saw DEP sync errors reappear in our datajar.mobi deployments.
Below is some detail on what has caused these new sync errors & how to resolve.
Errors
So, these errors appeared in the Jamf Pro UI as shown at the top of this post.
This the same same UI that was presented with the recent Terms & Conditions update.
However, logging into an affected deployments ABM/ASM did not offer any new Terms & Conditions, the logs were not showing the same errors & re-uploading the DEP token would either not work or not resolve the issue.
Discovery
Digging into the logs, each affected deployment showed the following in it’s Jamf Pro’s logs:
[ERROR] [duledPool-6] [eEnrollmentProgramRequest] - javax.net.ssl.SSLHandshakeException: Received fatal alert: decode_error
After reaching out to various Slack channels yesterday & today, it became clear that numerous folks were impacted by this issue & it was not just us or an issue with our hosting.
The discovery of this was compounded by the fact that you might only be affected for a short while before everything synced correctly &, with a recent Jamf Pro update, the sync changed to an automatic every 5 minute activity with no option to manually trigger a sync in the UI.
Once some people started to come forward about being affected, we all traced back the initial entry for the “decode_error” to 26/09/19.
This then assumed that the issue was upstream & not caused by any updates as the folks that were affected had all updated Jamf Pro & differing times & none on 26/09/19.
Java
Digging deeper, Jamf Cloud did not appear to be affected here.
Jamf Cloud uses Amazon Corretto for Java. Where all affected had been using OpenJDK, however as typing this some Amazon Corretto users found they were impacted by the issue
Presuming this was a Java issue, the logs messages, & reaching out to Jamf Support lead me to:
- https://github.com/structurizr/java/issues/119
- https://stackoverflow.com/questions/57103697/how-to-fix-sslhandshakeexception-received-fatal-alert-decode-error
- https://webtide.com/openjdk-11-and-tls-1-3-issues/
All of which pretty much advise to disable TLS 1.3.
The Solution
So, disabling TLS 1.3 is what’s needed.
As such, you’ll need to add the below to either CATALINA_OPTS or JAVA_OPTS on your Jamf Pro host.
-Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2"
This will likely be the file which you’ve set Tomcats memory, & if you have a clustered Jamf Pro setup only the Master Web Application will need this amendment as it’s the one which performs the DEP Device Monitor checks:
Once inplace, you’ll need to restart Tomcat.
After which the DEP Sync error should go from the UI, no “decode_error” should show in the logs & all should be well.
However, as of OpenJDK 11.0.5 there will be a fix in place to disable TLS 1.3 & therefore not need this change to the OPTS.
The issue here is, OpenJDK 11.0.5 is not scheduled for released until 15/10/19 & then any forks will need to add the change afterwards.
So, what happened?
I can only guess that some one or more hosts in the CDN that is used for DEP has a bad TLS config, & that this config went bad on 26/09/19.
Jamf have this logged as PI-007508 & have raised a ticket with Apple, so maybe something here will come to light down the line.
Many thanks to those that helped diagnose this on Slack & the folks at Jamf for their support with this issue.
How about having a setenv.sh with something like:
pagesize=`getconf PAGE_SIZE`
normalized=`echo “${pagesize}^2 / (1024^2) * .75” | bc`
rounded=`printf %.f “${normalized}”`
export CATALINA_OPTS=”$CATALINA_OPTS -Xmx${rounded}m”
This just hit us once again. I had to redo it recently, not sure what changed.
Hit us on 12/9/2019. We have 2 on-premise Jamf Pro servers which totally stopped syncing to ASM within 2 minutes of each other. Prior to this we didn’t have to do the fix in this blog post. Both of our Jamf servers are running on Red Hat Enterprise Linux 7.x. Both were running Jamf Pro 10.16.1. Upgrade to 10.17.1 didn’t fix it. We are using Corretto for Java and installed the latest version available (11.0.5) as another fix attempt and that didn’t work. We ended up needing to manually disable TLS 1.3 in JAVA as mentioned in this blog post. We had to edit /usr/local/jss/tomcat/bin/setenv.sh and add the line: export JAVA_OPTS=”$JAVA_OPTS -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2″
Thanks for this! This bit us the last couple days, and I had a heck of a time finding the file to modify. =)
For whatever reason, 11.0.5 didn’t fix this… this change seems to have worked though. Thanks again!
This also worked for us at Notre Dame. We had the same lost sync issue since 12/9.
Your clear directions to the file to modify were very much appreciated!
Thanks!
it also works for us, thanks so much!
Ben, add another to impacted list of Jamf Pro admins. Here’s a little more detail…
– On-prem/private AWS VPC
– Ubuntu 18.04.3 LTS
– openjdk 11.0.4 2019-07-16
Both our Prod (Jamf Pro v10.15.1-t1569637051) and Test (Jamf Pro v10.17.0-t1572881164) environments were impacted about the same time. I can confirm the following (per your fix) restored DEP sync in …
1. Edit (I used vim) /usr/local/jss/tomcat/bin/setenv.sh by adding the following at end of file…
export JAVA_OPTS=”$JAVA_OPTS -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2″
2. Save file.
3. Restart Tomcat
4. Verify DEP/Automated Device Enrollment no longer shows sync error.
5. Buy Ben a drink at earliest ocassion.
Thanks very much!
*occasion*
Well we had this one Windows Server 2016 – 1607.
Go to C:\Program Files\JSS\Tomcat\bin\tomcat8w.exe Java Tab or setenv.bat > JAVA_OPTS and then check if we have settings line like:
-Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2,TLSv1.3.
If it is not there, please add only:
-Djdk.tls.client.protocols=TLSv1.1,TLSv1.2
If we have TLSv1 or TLSv1.0 or TLSv1.3, we would need to remove it.
The restart the tomcat server.