Here are the best practices for MQTT to optimize the network traffic and prevent data loss.
Network traffic optimization
Keep-alive interval
MQTT libraries generally use the keep-alive interval to generate a periodic PING telling the server the client is still alive.
A wrong keep-alive interval may generate a waste of bandwidth; to prevent this, it should be sized according to the publishing frequency.
Generally, this value is set to a value double that of the average time between consecutive messages.
In case the LWT has been configured, and if the client uses a keep-alive interval of 5 minutes, Servitly will recognize the product is OFFLINE only after 5 minutes from the last received message.
Metric data aggregation
If you need to send multiple metrics (with the same metric path), it is suggested to group them all into a single message.
All the metric values in the same message will share the same timestamp.
Moreover, if a metric does not change so often, you can decide to publish it only when its value changes.
Deferred publishing
Another way to optimize the network traffic is to collect messages locally into the device, and publish them in a single connection session opened periodically (e.g., every hour). Note that in this case, measures are not in real-time.
To mitigate this, you can decide to publish in real-time only the important data, like alerts.
Data Loss Prevention
It could happen that the MQTT broker may not be reachable for a limited time, and this can occur when:
The thing is offline (e.g., no GSM/Wi-Fi coverage)
There is a problem on the network where the thing is connected (e.g., Wi-Fi without Internet access).
The MQTT broker service is rebooted for planned maintenance (e.g., new release, certificate update) or unplanned maintenance (e.g., security fix).
The MQTT broker is down due to unexpected problems.
For these reasons, to create a robust system without any data loss, the MQTT client running on the things must:
At least use a QoS (Quality of Service) of 1, allowing the thing to acknowledge the message has been received.
Automatically reconnect to the broker when a connection problem occurs. This can be done through the underlying MQTT library, which should support auto-reconnect or allow listening on connection events (e.g., onConnect, onDisconnect) and react to them.
During the OFFLINE period, locally store unpublished data; this is generally automatically performed by the underlying MQTT library, for instance, PAHO supports an in-memory or disk persistence. Data are stored on the client-side while OFFLINE, and then bulk published when the client turns back ONLINE.
Timestamp Management
When publishing data, you can specify the timestamp at which the data was actually acquired on the thing, rather than the time of publication. This timestamp must be in the past.
Publishing historical data (backfilling) is supported; however, if a metric already contains more recent values, there is no guarantee that the older data will be fully processed.
In particular:
Derived metrics and events rely on an internal checkpoint that moves forward as data is processed.
If the new data points are older than this checkpoint, no derivation or event computation will be performed on them.Insights are computed daily on a 24-hour data set.
If you publish data older than one day, it will not be included in the insight computation.
Publishing data with a timestamp in the future is not recommended and may lead to unexpected results: the data will be stored, but events will not be computed.
The best practice is to publish data as soon as it becomes available on the thing, thereby keeping the data delay to a minimum.
If the device's internal clock is not reliable, we recommend omitting the timestamp (ts) from the message: in this case, the server will use the message receive time as the timestamp.
Certificate Management
Always remember that to open the connection, your client must have the Servitly CA installed and up-to-date.
For more details, refer to the SSL Certificates article.