Implementation of sync sets

Hi all,

how are sync sets implemented in detail? Are they applied before the custom sql statement of the datasource? Or is the sql statement of the datasource somehow extend with a filter (filtering according to the sync sets) and the sync sets are therefore working concurrent with the sql statement of the datasource.

Thanks in advance.

1 Like

Maybe this helps you:
https://[[YOUR_INSTANCE]].onedata.de/apps/apps-docs/odml-documentation/AppBuilder/sync-sets/SyncSets.html#performance-implications-of-the-syncset-rework

It depends. If server-side filtering is enabled, then it is the approach via a filter that you describe. The filter is basically a join + filter operation. It joins a data request with a number of other data requests and filters for those rows that have a join partner. This works only if nothing else than data tables of type POSTGRESQL (no PostgreSQL connections (!)) is involved. This needs to be enabled explicitly in the App.
Otherwise, it is some client-side logic that is based on distinct value computation and the data requests involved are executed one after the other.
See https://internal.onedata.de/apps/apps-docs/odml-documentation/AppBuilder/sync-sets/SyncSets.html and
https://internal.onedata.de/apps/apps-docs/odml-documentation/AppBuilder/sync-sets/SyncSets.html#optimizing-performance-with-server-side-filtering

1 Like

Here an addition about the order of transformations.
The order how transformations are applied is as follows:
The more to the bottom the later it gets applied.

  • Builtin Queries (e.g. using a Virtual Datatable, besides that the standard SQL / Spark SQL)
  • Datasource Transformations/SQL
  • SyncSet Filters
  • Element Transformations/SQL (not if distinct values are collected and distinctValuesUseAllTransformations is not set)
  • User Filters (e.g. in tables or highcharts)
  • Row Filters (only in tables)
  • User Sorting (e.g. in tables)
2 Likes