The same job can be executed on multiple cluster nodes sequentially

After a job is run the job's next fire time is calculated and updated, or if it was a one-time job it is deleted. In the former case, although the property is updated in the database, this information is not reflected in the index of other cluster nodes until the cluster node is synced. Although this happens every five seconds, there is a time window in which the index of other cluster nodes contains job information that is out of sync with the database. Unless the index is updated before the query to get the next pending jobs is executed the query result may contain jobs based on the previous value of the next fire time property. In short: a cluster sync must be forced immediately prior to querying for next pending jobs. And even then, the next fire time property must double-checked after the lock on the trigger has been acquired since it can still change until it is locked. This logic is currently absent resulting in the same job sometimes being executed on multiple cluster nodes.