Here are the best practices for the HTTP Connector to optimize the network traffic and prevent data loss.
Network traffic optimization
Metric data aggregation
If you need to send multiple metrics (with the same metric path), it is suggested to group them all into a single message.
All the metric values in the same message will share the same timestamp.
Moreover, if a metric does not change so often, you can decide to publish it only when its value changes.
Deferred publishing
A way to optimize the network traffic is to collect messages locally into the device, and publish them in a single connection session opened periodically (e.g., every hour).
When you are ready to publish data messages, you can send each one by one, or alternatively, you can send multiple using a JSON array in the payload. In the alternative, you can also use the CSV format for the payload.
Note that in this case, measures are not in real-time. To mitigate this, you can decide to publish in real-time only the important data, like alerts.
Data Loss Prevention
It could happen that the HTTP Connector may not be reachable for a limited time, and this can occur when:
The thing is offline (e.g., no GSM/Wi-Fi coverage)
There is a problem on the network where the thing is connected (e.g., Wi-Fi without Internet access).
The HTTP Connector service is rebooting for planned maintenance (e.g., new release, certificate update) or unplanned maintenance (e.g., security fix).
The HTTP Connector is down due to unexpected problems.
For these reasons, to create a robust system without any data loss, the publisher client running on the things must:
Automatically retry to publish (with an increasing delay) when a connection problem occurs.
During the OFFLINE period, locally store unpublished data, and then bulk publish when the client turns back ONLINE.
Timestamp Management
When publishing data, you can specify the timestamp at which the data was actually acquired on the thing, rather than the time of publication. This timestamp must be in the past.
Publishing historical data (backfilling) is supported; however, if a metric already contains more recent values, there is no guarantee that the older data will be fully processed.
In particular:
Derived metrics and events rely on an internal checkpoint that moves forward as data is processed.
If the new data points are older than this checkpoint, no derivation or event computation will be performed on them.Insights are computed daily on a 24-hour data set.
If you publish data older than one day, it will not be included in the insight computation.
Publishing data with a timestamp in the future is not recommended and may lead to unexpected results: the data will be stored, but events will not be computed.
The best practice is to publish data as soon as it becomes available on the thing, thereby keeping the data delay to a minimum.
If the device's internal clock is not reliable, we recommend omitting the timestamp (ts) from the message: in this case, the server will use the message receive time as the timestamp.
Certificate Management
Always remember that to open the connection, your client must have the Servitly CA installed and up-to-date.
For more details, refer to the SSL Certificates article.