Objects, queries and filters
midPoint works with tree-shaped data structures, just like hierarchical databases do. These structures are called prism objects.They are stored in various places, namely:
|midPoint repository||almost all midPoint objects||repository|
|remote resources||resource objects (accounts, groups, ...)||provisioning + connectors|
|embedded workflow engine||work items, process instances||workflow|
Each of these objects is conceptually a tree. Tree nodes are called prism items: containers, references and properties, while each item has one or more values: container values, reference values or property values. (See the page on Prism Data Structures). Some items are strictly single-valued, others may contain more values. The resource objects are currently a bit simplified: they consist of one single-valued container having a set of non-structured properties and their values - called attributes.
In the following we will discuss primarily queries targeted at repository objects. These notes apply to resource and workflow objects appropriately.
A query can be:
- an object query, i.e. the one that is targeted at one or more object classes. Examples of queries are e.g.
- "give me all users with a given name",
- "give me all the objects with a given name",
- "give me all focal objects with a given assignment" etc.
- a container value query. Such queries are targeted at container values. These are being introduced in the context of special objects, like certification campaigns and lookup tables, or even in cases when no parent object exists, like work items. Examples of queries:
- "give me all certification cases related to this reviewer, irrespective to in which campaign they are contained",
- "give me all rows for this particular lookup table that start with this prefix",
- "give me all work items allocated to particular user".
Each query consists of a filter and a paging instruction. The former says which records (objects, container values) are to be retrieved. The latter says how they are to be sorted and which ones have to be selected (e.g. those with numbers 100-199). Both of these components (filter, paging instruction) are optional.
A filter is either a primitive or complex one.
Primitive filters are these that do not contain other filters. They are of two categories:
|None filter||Passes no values, i.e. always evaluates to "false".|
|All filter||Passes all values, i.e. always evaluates to "true".|
Treated like nonexistent or invisible filter. For all fiters F1 and F2 the following holds:
F1 && Undefined = F1
F2 || Undefined = F2
These filters decide on value(s) of a given property, reference or container.
Generally, they are characterized by:
- a (left-side) item path, pointing to a property or a reference,
- a (right-side) constant value(s) or item path, pointing to value(s) to be used in the comparison,
- optionally a matching rule.
|Filter||Applicable left-side items for repo queries||Applicable left-side items for resource queries||Applicable right-side constant values||Applicable right-side path-pointed values||Description||Comments|
|Equal filter||property||property||single-valued (see note 2), nullable||single, nullable (see note 1)|
For non-null filter value: Accepts if one of property values is the same as filter value.
For null filter value: Accepts if property has no values. This option is not available for resource objects, due to ConnId limitations.
|Greater, Less filter||property||property||single, non-null||singleton||Accepts if one of property values is greater/greater-or-equal/less/less-or-equal in comparison to the filter value. For null-valued singleton items always returns false (TODO verify that).|
|Substring filter||property||property||single, non-null||-||Accepts if the filter value is a substring of one of the property values (optionally specifying if the property value should start or end with the filter value).|
|Ref filter||reference||-||single or multivalued (since 3.6), nullable||-|
For non-null filter values: Accepts if one of the reference values match the filter value (or one of filter values, if there are more than one).
For null filter values: Accepts if reference is empty.
"Matches" means that:
|Org filter||(applicable to object as a whole)||-||single, non-null (or null with isRoot flag)||-||Accepts if the object is direct child or any descendant (this is configurable) of the referenced org. Alternatively, passes if the object is the root of the tree. As of 3.7.1 it can check the relation as well (see a note below).||Although technically not a Value filter, this filter can be seen as a special case of Ref filter using parentOrgRef as the item to be tested, and with some advanced options (scope, isRoot).|
|InOid filter||(applicable to object/container value as a whole)||-||multivalued, non-null||-||Accepts if object OID (or ID for container values) is among filter values.||Similar; see note 4.|
- Resource and workflow filters do not support items on the right side of an operator. Only constant values may be present there.
- We should probably introduce special kind of Equal filter (named e.g. In filter) to implement the following comparisons:
- Equal filter: multivalued left-hand side (LHS) vs. single-valued right-hand-side (RHS)
- In filter: multivalued LHS vs. multivalued RHS ("non empty set intersection" semantics)
- Interestingly enough, there is an InFilter available now. It is implemented only when searching in the provisioning module. This filter is mapped to ICF EqualsFilter that provides set equality test. (So the filter name does not match its function.) It should probably be removed.
- Question is if we should treat querying by ID/OID in the same way as querying by property, i.e. via Equal filter (and, maybe, Greater filter and the like), where ID/OID would be treated as special kind of property. This would eliminate the need for InOid filter; but it might require deeper changes (e.g. there is no itemDefinition for ID/OID, etc). So, at least for midPoint 3.4, querying by ID/OID is done via InOid filter, not Equal filter.
- Ref filter and Org filter can specify a relation to be looked for. It is specified as a relation on the reference value passed to the filter. However, for historical reasons, the null relation value is treated differently:
- For Ref filter, null relation means default relation. If you need to check for any relation, you have to provide a value of q:any there.
- For Org filter, null relation means any relation. Of course, q:any can be used as well (and is recommended for clarity).
- The Org filter relation is supported only for the "directChildOf" and "childOf" queries. It is silently ignored for "parentOf" queries. It is interpreted as a relation of the last (lowest) reference in the path, i.e. if we are looking for a user that is a child of org O1 with the relation of manager, we are looking for a user that is a manager of an org O2, which is either O1 itself or is any of its descendants.
Complex filters do contain other filters. They are:
|And, Or, Not||Basic logical filters.|
|Type (type T, filter F)||Accepts iff the object is of type T and filter F passes.|
|Exists (item I, filter F)||Accepts iff there exists a value v of item I so that F(v) passes. This is useful e.g. to find an assignment with a given tenantRef and orgRef.|
And, Or and Not filters are quite self explanatory.
An example: Imagine that the original query asked for an ObjectType. Then it is possible to set up Type filter with type=UserType, filter=(name equals "xyz") to find only users with the name of "xyz":
First of all, how should be individual value filters evaluated?
- equal(name, 'xyz')
means "the value of object's name is xyz". Simple enough.
In a similar way,
- ref(assignment/tenantRef, oid1)
means "there is an assignment with a tenantRef pointing to oid1".
But what about this?
- and(ref(assignment/tenantRef, oid1), ref(assignment/orgRef, oid2))
This one could be interpreted in two ways:
- There should be an assignment $a that has $a/tenantRef = oid1 and $a/orgRef = oid2.
- There should be assignments $a1, $a2 (potentially being the same) such that $a1/tenantRef = oid1 and $a2/orgRef = oid2.
Up to and including midPoint 3.3.1, the query is interpreted in the first way (one assignment satisfying both conditions).
But the interpretation should be following:
- Each condition is interpreted separately.
- So ref(assignment/tenantRef, oid1) should be read as "There is an assignment/tenantRef that points to oid1".
- Therefore, the above complex filter should be interpreted in the second way: There should be assignments $a1, $a2 (potentially being the same) such that $a1/tenantRef = oid1 and $a2/orgRef = oid2.
If it's necessary to say that one particular value of an item (presumably container) satisfies a complex filter, we use Exists filter.
The above complex filter - if needed to be interpreted in the first way - should be written like this:
- exists ( assignment , and ( ref (tenantRef, oid1), ref (orgRef, oid2) ) )
Written in XML:
This feature is a part of midPoint 3.4 and above.
Differences in filter interpretation
There are actually four "query engines" that interpret filters and queries:
|repository||Interprets queries issued against repository objects.||almost all, except the ones described below|
|provisioning (connectors)||Interprets queries issued against resource objects, i.e. objects that reside on particular resources (AD, LDAP, CSV, ...).||ShadowType (some parts of them)|
|workflow engine||Interprets queries issued against work items, because they are kept in Activiti repository.||WorkItemType|
|in-memory evaluator||Interprets queries/filters issued against objects already loaded into memory. Typically used for authorization evaluation.||all|
These engines differ in capabilities and supported options. Due to historical reasons they might even interpret some filters in a slightly different way; this is unwanted and will be eventually fixed when discovered.
Let us summarize main differences here. Note that "ok" means "fully supported". "N/A" means "not applicable", i.e. not supported at all. The state is current as of midPoint 3.7.1.
General constraint for provisioning queries: It is not possible to mix both on-resource and repository items in one query, e.g. to query for both
Filters can be created using Java API (traditional or fluent one) or via XML.
Note that QueryBuilder can return either whole query when .build() is used, or just a filter - with .buildFilter().
None and Undefined filters are created in a similar way.
Just for completeness, the whole query looks like this:
and the corresponding Fluent Java API call is:
To be concise, we'll show only filters (no wrapping queries) in the following examples.
Fluent Java API:
Another example (we'll show only XML and fluent Java API from this point on):
Comparing item to another item:
Or a more complex example:
"Canonical" form is the following:
Semantics of individual 'or'-conditions is:
- resourceRef should contain: target OID = 'oid1', relation = (empty), and the type of target object (stored in the resourceRef!) can be any
- resourceRef should contain: target OID = 'oid1', relation = (empty), type of target (stored in the resourceRef!) must be 'ResourceType'
- resourceRef should contain: target OID = 'oid1', relation = 'test', and type of target (stored in the resourceRef!) must be 'ResourceType'
Currently, the reference target type, if used, must match exactly. So e.g. if the references uses RoleType, and the filter asks for AbstractRoleType, the value would not match.
It is suggested to avoid querying for target object type, if possible.
XML can be written also in alternative ways:
This one selects container values with ID 1, 2 or 3, having owner (object) with OID of "00000000-1111-2222-3333-777777777777".
An artificial example:
An example: Find all certification cases that have at least one missing response for a given reviewer.
So we are looking for a certification case, that has a decision D for which:
- D's reviewer is the given one,
- D's stage number is the same as case's stage number (because certification case contains decisions from all the stages),
- D's response is either null or 'noResponse'
It looks like this in XML:
And in Java:
This example returns all objects with a name starting with "C".
This example returns all objects with extension attribute "EndDate" (type of XMLGregorianCalendar), which is set since 31 Decenber last year to 01 January of this year.
Limiting the number of returned entries, offset etc. can be configured using paging. Following table shows paging options:
|Option||Possible values||Default value||Description|
|property path (e.g. "name")||arbitrary search||Property by which the results should be sorted. Only one property supported for now|
|ascending/descending||ascending||Direction of ordering (ascending or descending). Only valid if "orderBy" is specified.|
|any Integer||0||The index of the first returned entry, starting with zero. If 0 specified, resulting list will start with first entry. If 1 specified, resulting list will start with second entry.|
|any Integer||2147483647||The maximum number of entries returned. The operation may return specified number of number of entries or less.|
|property path (e.g. "name)||no grouping||Property by which the results should be grouped. Just one property for now.|
Following is the example for using paging in the query.
Using example above will return first 10 records ordered by name where administrativeStatus is set to enabled.
Special symbols in item paths ("..", "@", "#")
An example: Find all active certification cases for a given reviewer.
An active certification case is one that is part of a campaign that is in a review stage, and whose current stage number is the same as the owning campaign current stage number.
The ".." symbol denotes "owning campaign".
PrismConstants.T_PARENT is the QName for ".." path segment.
Following example uses @ symbol to dereference linkRef to ShadowType in user object. This allows e.g. filtering users that have projection on specified resource. Please note, that @ has limitation towards general (any object type) usage and will work with statically defined types like ObjectType, FocusType, ShadowType.
UserType: linkRef/@/resourceRef/@/name contains 'CSV' (norm).