Data Synchronization

You can setup how data is synchronized to the SmartHub Mart and thereby control when data is to be available for reports based on the hub. In addition, you can launch a synchronization task directly by clicking the Sync Now button.

Note: The default and recommended behavior for synchronization is continuous data updates, where a service will check every five minutes for changes in the data, and a synchronization will be run in the event changes are found. This default behavior starts automatically when a survey is added to the hub, so if this arrangement covers your requirements then you do not need to make any changes to the setup. However there may be times when you do not want to wait until the task runs by itself, and then the Sync now button can be used.

1. Click Surveys in the SmartHub menu bar to open the Surveys page, then click on the survey you wish to set synchronization for to open its Manage Survey page.

2. Locate the Scheduling section and click the Sync Now button.

3. Alternatively, if you want to schedule a sync, click the drop down arrow next to the Sync Now button and click "Schedule sync".

Figure 1 - The Data sync drop down

After clicking this, the Data sync overlay opens:

• Not scheduled - the synchronization task is turned off.

• Continuous - data is automatically synchronized on a (near) continuous basis. The task is run every five minutes, however updates may take longer depending on the size of the data-set and the queue status.

• Future - set up the synchronization task to follow a recurrence pattern.

  When you select Future, a number of additional fields are displayed

  


Figure 2 - The additional fields

On completion click the Save button. The settings are saved and the overlay closes.

Important: The Data Synchronization itself runs as a regular task called “Hub Forsta Project Loader”. You will find it in the task system if you search by the ID of the survey or the hub ID in task management. If this task fails due to an issue with the data or the system, synchronization may stop.

Possible issues and causes are:

• If a handled exception occurs, the task system will automatically disable the task (switch from “Continuous” to “Off”). A handled exception could for example be a data inconsistency such as an illegal value.

• If an unhandled exception occurs, the task system will allow the task to start again a configured number of times (set to 5 times on our production servers). An unhandled exception could for example be a server connectivity issue, from which the system might recover in time for the next load. If however the task keeps failing, the task system will automatically disable the task when the configured limit has been exceeded (switch from “Continuous” to “Off”).

Notifications will be sent depending on user and/or task settings for notifications (see the User Settings and The Task Properties Page sections in the Authoring manual for further information).

If the task stops running, you will have to check the task log to see what the problem is and try to correct it, for example by removing an illegal value in the data. If you are not able to resolve the issue, you can contact support for assistance with details about the task that failed.

Once the issue has been resolved you can switch the Data Sync service on again by switching it back to the desired setting (for example Continuous), and a new hub loader task will be started.

Data synchronization is typically incremental, updating the Hub with changes in data since the last synchronization (new, remove, changed records), and unless there has been a large amount of changes or new data, it will not take long to process. However occasionally the hub loader will perform a full reload, which may take some time, especially on large datasets. A full reload is triggered under the following conditions:

• When the hub loader version is incremented. This is an internal version number in the system that is rarely changed. However there may sometimes be a bug fix that requires a full reload, or other large structural changes in hub loading.

• If the question type is changed for a question, for example from single to multi.

• If the variable type is changed for a single-question, for example from normal answer list to table lookup.

• If a variable that was previously removed from the survey is added back (either by deleting and re-adding questions in the survey or by changing the “exclude from reporting” property on a question).

• Adding, moving or deleting loops.

• If more than a specified number (currently 500.000) records are touched (updated) in the survey database, regardless of what has triggered these changes (data processing rules, data imports, web services updates, data editors or survey responses).

• If field widths are increased for open text questions.

• If total digits and decimals are changed for numeric questions.

• For combined surveys, a full reload will be run if the combined survey is changed.

While the hub loader runs, data from the last sync is fully available for the reporting applications so there is no down-time while the data is being synchronized. Once the load has completed, the hub will switch to the latest synchronized data.

Note: When a survey has a variable "invalidResp", records where this is set to 1 will not be included when the hub loader writes data to the SmartHub Mart. This can be used to flag as "invalid" specific responses that should not be included in reporting, for example if you suspect the survey respondent has just provided random answers.