Skip to end of metadata
Go to start of metadata

This page describes Query API as of midPoint 3.4. Query API for versions up to 3.3 (although maybe a bit out of date) can be found here and here.

Objects, queries and filters

midPoint works with tree-shaped data structures, just like hierarchical databases do. These structures are called prism objects.They are stored in various places, namely:

PlaceObjectsResponsible module
midPoint repositoryalmost all midPoint objectsrepository
remote resourcesresource objects (accounts, groups, ...)provisioning + connectors
embedded workflow enginework items, process instancesworkflow

Each of these objects is conceptually a tree. Tree nodes are called prism items: containers, references and properties, while each item has one or more values: container values, reference values or property values. (See the page on Prism Data Structures). Some items are strictly single-valued, others may contain more values. The resource objects are currently a bit simplified: they consist of one single-valued container having a set of non-structured properties and their values - called attributes.

In the following we will discuss primarily queries targeted at repository objects. These notes apply to resource and workflow objects appropriately.

A query can be:

  1. an object query, i.e. the one that is targeted at one or more object classes. Examples of queries are e.g.
    1. "give me all users with a given name",
    2. "give me all the objects with a given name",
    3. "give me all focal objects with a given assignment" etc.
  2. a container value query. Such queries are targeted at container values. These are being introduced in the context of special objects, like certification campaigns and lookup tables, or even in cases when no parent object exists, like work items. Examples of queries:
    1. "give me all certification cases related to this reviewer, irrespective to in which campaign they are contained",
    2. "give me all rows for this particular lookup table that start with this prefix",
    3. "give me all work items allocated to particular user".
    Container value queries cannot be implemented using object queries, as it is not reasonable to fetch whole objects (campaigns, lookup tables) because they can be simply too large to be efficiently processed. And, in some cases, "owning" objects simply do not exist.

Each query consists of a filter and a paging instruction. The former says which records (objects, container values) are to be retrieved. The latter says how they are to be sorted and which ones have to be selected (e.g. those with numbers 100-199). Both of these components (filter, paging instruction) are optional.

Filter classification

A filter is either a primitive or complex one.

Primitive filters

Primitive filters are these that do not contain other filters. They are of two categories:

Trivial filters

FilterDescription
None filterPasses no values, i.e. always evaluates to "false".
All filterPasses all values, i.e. always evaluates to "true".
Undefined filter

Treated like nonexistent or invisible filter. For all fiters F1 and F2 the following holds:

F1 && Undefined = F1

F2 || Undefined = F2

Value filters

These filters decide on value(s) of a given property, reference or container.

Generally, they are characterized by:

  • a (left-side) item path, pointing to a property or a reference,
  • a (right-side) constant value(s) or item path, pointing to value(s) to be used in the comparison,
  • optionally a matching rule.
FilterApplicable left-side items for repo queriesApplicable left-side items for resource queriesApplicable right-side constant valuesApplicable right-side path-pointed valuesDescriptionComments
Equal filter propertypropertysingle-valued (see note 2), nullablesingle, nullable (see note 1)

For non-null filter value: Accepts if one of property values is the same as filter value.

For null filter value: Accepts if property has no values. This option is not available for resource objects, due to ConnId limitations.

 
Greater, Less filterpropertypropertysingle, non-nullsingletonAccepts if one of property values is greater/greater-or-equal/less/less-or-equal in comparison to the filter value. For null-valued singleton items always returns false (TODO verify that). 
Substring filterpropertypropertysingle, non-null-Accepts if the filter value is a substring of one of the property values (optionally specifying if the property value should start or end with the filter value). 
Ref filterreference-single or multivalued (since 3.6), nullable-

For non-null filter values: Accepts if one of the reference values match the filter value (or one of filter values, if there are more than one).

For null filter values: Accepts if reference is empty.

"Matches" means that:

  1. OIDs match,
  2. relations match: null is equivalent to org:default. So if you want to match any relations, use PrismConstants.Q_ANY.
  3. referenced types matches: but here null in filter means "any type".
Org filter(applicable to object as a whole)-single, non-null (or null with isRoot flag)-Accepts if the object is direct child or any descendant (this is configurable) of the referenced org. Alternatively, passes if the object is the root of the tree. As of 3.7.1 it can check the relation as well (see a note below).Although technically not a Value filter, this filter can be seen as a special case of Ref filter using parentOrgRef as the item to be tested, and with some advanced options (scope, isRoot).
InOid filter(applicable to object/container value as a whole)-multivalued, non-null-Accepts if object OID (or ID for container values) is among filter values.Similar; see note 4.

Notes:

  1. Resource and workflow filters do not support items on the right side of an operator. Only constant values may be present there.
  2. We should probably introduce special kind of Equal filter (named e.g. In filter) to implement the following comparisons:
    1. Equal filter: multivalued left-hand side (LHS) vs. single-valued right-hand-side (RHS)
    2. In filter: multivalued LHS vs. multivalued RHS ("non empty set intersection" semantics)
  3. Interestingly enough, there is an InFilter available now. It is implemented only when searching in the provisioning module. This filter is mapped to ICF EqualsFilter that provides set equality test. (So the filter name does not match its function.) It should probably be removed.
  4. Question is if we should treat querying by ID/OID in the same way as querying by property, i.e. via Equal filter (and, maybe, Greater filter and the like), where ID/OID would be treated as special kind of property. This would eliminate the need for InOid filter; but it might require deeper changes (e.g. there is no itemDefinition for ID/OID, etc). So, at least for midPoint 3.4, querying by ID/OID is done via InOid filter, not Equal filter.
  5. Ref filter and Org filter can specify a relation to be looked for. It is specified as a relation on the reference value passed to the filter. However, for historical reasons, the null relation value is treated differently:
    1. For Ref filter, null relation means default relation. If you need to check for any relation, you have to provide a value of q:any there.
    2. For Org filter, null relation means any relation. Of course, q:any can be used as well (and is recommended for clarity).
  6. The Org filter relation is supported only for the "directChildOf" and "childOf" queries. It is silently ignored for "parentOf" queries. It is interpreted as a relation of the last (lowest) reference in the path, i.e. if we are looking for a user that is a child of org O1 with the relation of manager, we are looking for a user that is a manager of an org O2, which is either O1 itself or is any of its descendants.

Complex filters

Complex filters do contain other filters. They are:

FilterDescription
And, Or, NotBasic logical filters.
Type (type T, filter F)Accepts iff the object is of type T and filter F passes.
Exists (item I, filter F)Accepts iff there exists a value v of item I so that F(v) passes. This is useful e.g. to find an assignment with a given tenantRef and orgRef.

Logical filters

And, Or and Not filters are quite self explanatory.

Type filter

An example: Imagine that the original query asked for an ObjectType. Then it is possible to set up Type filter with type=UserType, filter=(name equals "xyz") to find only users with the name of "xyz":

Example

Exists filter

First of all, how should be individual value filters evaluated?

For example,

  • equal(name, 'xyz')

means "the value of object's name is xyz". Simple enough.

In a similar way,

  • ref(assignment/tenantRef, oid1)

means "there is an assignment with a tenantRef pointing to oid1".

But what about this?

  • and(ref(assignment/tenantRef, oid1), ref(assignment/orgRef, oid2))

This one could be interpreted in two ways:

  1. There should be an assignment $a that has $a/tenantRef = oid1 and $a/orgRef = oid2.
  2. There should be assignments $a1, $a2 (potentially being the same) such that $a1/tenantRef = oid1 and $a2/orgRef = oid2.

Up to and including midPoint 3.3.1, the query is interpreted in the first way (one assignment satisfying both conditions).

But the interpretation should be following:

  • Each condition is interpreted separately.
  • So ref(assignment/tenantRef, oid1) should be read as "There is an assignment/tenantRef that points to oid1".
  • Therefore, the above complex filter should be interpreted in the second way: There should be assignments $a1, $a2 (potentially being the same) such that $a1/tenantRef = oid1 and $a2/orgRef = oid2.

If it's necessary to say that one particular value of an item (presumably container) satisfies a complex filter, we use Exists filter.

The above complex filter - if needed to be interpreted in the first way - should be written like this:

  • exists ( assignment , and ( ref (tenantRef, oid1), ref (orgRef, oid2) ) )

Written in XML:

This feature is a part of midPoint 3.4 and above.

Differences in filter interpretation

There are actually four "query engines" that interpret filters and queries:

NameDescriptionData types
repositoryInterprets queries issued against repository objects.almost all, except the ones described below
provisioning (connectors)Interprets queries issued against resource objects, i.e. objects that reside on particular resources (AD, LDAP, CSV, ...).ShadowType (some parts of them)
workflow engineInterprets queries issued against work items, because they are kept in Activiti repository.WorkItemType
in-memory evaluatorInterprets queries/filters issued against objects already loaded into memory. Typically used for authorization evaluation.all

These engines differ in capabilities and supported options. Due to historical reasons they might even interpret some filters in a slightly different way; this is unwanted and will be eventually fixed when discovered.

Let us summarize main differences here. Note that "ok" means "fully supported". "N/A" means "not applicable", i.e. not supported at all. The state is current as of midPoint 3.7.1.

FilterRepositoryProvisioning (connectors)WorkflowIn-memory
EqualokRight-side items are not supported. The null right side constant is not supported ( MID-1460 - ICF support for "is null" or "not present" filters Open ).Right-side items are not supported. Only externalId item can be queried.Right-side items are not supported.
Greater, LessokN/AN/AN/A
SubstringokOnly contains mode is supported; startsWith and endsWith ones are not.N/Aok
RefokN/AOnly assigneeRef and candidateRef items can be queried.ok; Additionally, there are two parameters driving the behavior of filters with null oid and targetType: oidNullAsAny and targetTypeNullAsAny. These are not honored by other interpreters yet.
OrgokN/AN/AN/A
InOidokN/AN/Aok
And, Or, NotokokOnly And connective (i.e. conjunction) is supported.ok
TypeokN/AN/Asupported but not much tested
ExistsokN/AN/Aok

General constraint for provisioning queries: It is not possible to mix both on-resource and repository items in one query, e.g. to query for both c:attributes/ri:something and c:intent.

For authoritative information, see FilterInterpreter and related classes (provisioning); and WorkItemProvider class (workflows).

Creating filters

Filters can be created using Java API (traditional or fluent one) or via XML.

The following samples are taken from TestQueryConvertor class. XML versions are in files named test*.xml in this directory.

Primitive filters

AllFilter

XML
Traditional Java API
Fluent Java API

Note that QueryBuilder can return either whole query when .build() is used, or just a filter - with .buildFilter().

None and Undefined filters are created in a similar way.

Just for completeness, the whole query looks like this:

XML

and the corresponding Fluent Java API call is:

Fluent Java API

To be concise, we'll show only filters (no wrapping queries) in the following examples.

Value filters

EqualFilter

XML
Traditional Java API

Fluent Java API:

Fluent Java API

Another example (we'll show only XML and fluent Java API from this point on):

XML
Fluent Java API

Comparing item to another item:

XML
Fluent Java API

Comparisons

XML
Fluent Java API

Or a more complex example:

XML
Fluent Java API

Substring filter

XML
Fluent Java API

Ref filter

"Canonical" form is the following:

XML

In Java:

Semantics of individual 'or'-conditions is:

  1. resourceRef should contain: target OID = 'oid1', relation = (empty), and the type of target object (stored in the resourceRef!) can be any
  2. resourceRef should contain: target OID = 'oid1', relation = (empty), type of target (stored in the resourceRef!) must be 'ResourceType'
  3. resourceRef should contain: target OID = 'oid1', relation = 'test', and type of target (stored in the resourceRef!) must be 'ResourceType'

Currently, the reference target type, if used, must match exactly. So e.g. if the references uses RoleType, and the filter asks for AbstractRoleType, the value would not match.

It is suggested to avoid querying for target object type, if possible.

XML can be written also in alternative ways:

Org filter

XML
XML
XML
Fluent Java API
Fluent Java API
Fluent Java API

InOid

XML
Fluent Java API

This one selects container values with ID 1, 2 or 3, having owner (object) with OID of "00000000-1111-2222-3333-777777777777".

XML
Fluent Java API

Logical filters

An artificial example:

XML
Fluent Java API

Type filter

XML
Fluent Java API

Exists filter

An example: Find all certification cases that have at least one missing response for a given reviewer.

So we are looking for a certification case, that has a decision D for which:

  1. D's reviewer is the given one,
  2. D's stage number is the same as case's stage number (because certification case contains decisions from all the stages),
  3. D's response is either null or 'noResponse'

It looks like this in XML:

XML

And in Java:

Expression filter

XML

This example returns all objects with a name starting with "C".

Date filtering

XML

This example returns all objects with extension attribute "EndDate" (type of XMLGregorianCalendar), which is set since 31 Decenber last year to 01 January of this year.

Paging

TODO

Special symbols in item paths ("..", "@", "#")

TODO

An example: Find all active certification cases for a given reviewer.

An active certification case is one that is part of a campaign that is in a review stage, and whose current stage number is the same as the owning campaign current stage number.

XML

The ".." symbol denotes "owning campaign".

PrismConstants.T_PARENT is the QName for ".." path segment.

Following example uses @ symbol to dereference linkRef to ShadowType in user object. This allows e.g. filtering users that have projection on specified resource. Please note, that @ has limitation towards general (any object type) usage and will work with statically defined types like ObjectType, FocusType, ShadowType. 

 

 

  • No labels