This issue is caused due to 401 errors received in response to the authentication challenges sent between the BBIS and BBCRM web servers responsible for running the schedule.
Sometimes Load Balancers will not pass internal service requests because the Load Balancers are acting as a reverse proxy and internal NTLM requests will fail while external ones will succeed. The Load Balancer will likely require special configuration to enable the successful response of internal requests.
Following are some resources you can use to configure this option on a number of popular Load Balancers. Please note that Blackbaud is unable to assist in the implementation of suggestions from any of these links. For additional assistance in implementing any of these configuration changes, consult additional support resources for the Load Balancer vendor:
It is also possible to add Kerberos as a provider to IIS to negate issues with the Load Balancer's handling of NTLM authentication.
Prior fixes workarounds are still included this solution, below.
Each of the following items have addressed at least 1 reported occurrence of this issue. As of 1-30-16, item number 7 has addressed the largest number of reports.
- In IIS on the BBAppFx server(s), verify that Windows authentication is enabled for the WebUI folder (http://host/bbAppFX/webui/)
- Ensure that 'NT Authority\anonymous logon' is added as an application user, and that it belong to the CRM database roles "Blackbaud Built-In AppFx System User Role" and "Email Services Administrator".
- Check bbec.productconfig.xml under the \\AppPath\vroot\browser\brand\current folder and confirm "NoScheduleHeartbeat=true" does not exist as an AppSetting key
- Add the following AppSetting to the BBAppFx web.config on all CRM nodes: <add key="HeartbeatEnabled" value="True" />
- Ensure that Regional date/time settings are consistent on all machines in the rig (webserver(s) and database server)
- In some load balancer configurations/environments, authentication will be lost if a process response lands on a different node than the one from which it originated. One way to prevent this scenario is to pair CRM & BBIS nodes together by adding entries to the Hosts file for each node pair.
- On each CRM and BBIS node, apply registry changes provided by Microsoft in Knowledgebase article https://support.microsoft.com/en-us/kb/896861 (note that method 2 has addressed multiple reports).
- Manually invoke the endpoint to start the scheduled service. Contact support for more information on this method. A script can be created to automate this at a regular interval
- In SQL server, right-click the database and select "Properties". Select "Extended Properties". Verify 'bb_DisableScheduledProcesses' is set to 'False'
- In the same registry key referenced in step 6, set the following order for the BackConnectionHostNames entires:
- DSN name
- Domain machine name
- IP address
- Reboot the web server