Open APIs

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

Hi,

As a contributor to current guidelines regarding pagination, few insights on the provided feedback:

• no full support should be mandatory; at the opposite supporting - only - relevant use-cases should be the promoted approach; from a practical and realistic concern no point in spending outstanding effort to support all possible permutations when those have no relevance in context; what is supported should however be clearly documented; all this is especially true - but not limited - to JSON Path
• based on above I agree on emphasizing on MAY vs. MUST
• there is already support for alias and mask in the guidelines
• escaping (_) the reserved keywords, such as pagination and filter, would allow collision with a secondary field with similar meaning, which doesn't seem a good practice and thus guideline to promote

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

@Christophe MICHEL, thank you for the insights!

• • escaping (_) the reserved keywords, such as pagination and filter, would allow collision with a secondary field with similar meaning, which doesn't seem a good practice and thus guideline to promote

That's a good point and I thought about it. Collisions could occur either way. But it's unusual to use this naming scheme in your public Api. If it's used internally, it's often used as an indicator of private fields in languages that do not support private keywords. E.g. older javascript. So the question is, is it more likely that someone wants to define the prefixed or non-prefixed variant in their model?
Or do you mean that it's a bad practice to have attribute named like this in the first place? Tbh. Personally I wouldn't use them, even if I could, but still like the indication of 'careful! this is something special and not a property filter'. Also allows query parser to translate everything without special pattern to a db selector.

• • there is already support for alias and mask in the guidelines

Can you point me to the chapters you're referring to? I made a quick search in the pdfs for both terms and did not find them. Nice to know that I missed something here and can read up on it

Hi,

As a contributor to current guidelines regarding pagination, few insights on the provided feedback:

• no full support should be mandatory; at the opposite supporting - only - relevant use-cases should be the promoted approach; from a practical and realistic concern no point in spending outstanding effort to support all possible permutations when those have no relevance in context; what is supported should however be clearly documented; all this is especially true - but not limited - to JSON Path
• based on above I agree on emphasizing on MAY vs. MUST
• there is already support for alias and mask in the guidelines
• escaping (_) the reserved keywords, such as pagination and filter, would allow collision with a secondary field with similar meaning, which doesn't seem a good practice and thus guideline to promote

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

My bad on alias and mask. I felt those were included in the guidelines but actually not as per 4.x versions. So I agree it is missing.

Here below is how it would look like.

In case of frequently used or recurring filters, it can be convenient to set one of the following:

• A fixed alias URI for the criteria
• A predefined mask or a set of masks that simplify the forging the request, rather than a detailed request

Example:

• Alias: catalog/item/bestselling
• Mask: catalog/item?sizerange=L, catalog/item?sizerange=XL

@Christophe MICHEL, thank you for the insights!

• • escaping (_) the reserved keywords, such as pagination and filter, would allow collision with a secondary field with similar meaning, which doesn't seem a good practice and thus guideline to promote

That's a good point and I thought about it. Collisions could occur either way. But it's unusual to use this naming scheme in your public Api. If it's used internally, it's often used as an indicator of private fields in languages that do not support private keywords. E.g. older javascript. So the question is, is it more likely that someone wants to define the prefixed or non-prefixed variant in their model?
Or do you mean that it's a bad practice to have attribute named like this in the first place? Tbh. Personally I wouldn't use them, even if I could, but still like the indication of 'careful! this is something special and not a property filter'. Also allows query parser to translate everything without special pattern to a db selector.

• • there is already support for alias and mask in the guidelines

Can you point me to the chapters you're referring to? I made a quick search in the pdfs for both terms and did not find them. Nice to know that I missed something here and can read up on it

Hi,

As a contributor to current guidelines regarding pagination, few insights on the provided feedback:

• no full support should be mandatory; at the opposite supporting - only - relevant use-cases should be the promoted approach; from a practical and realistic concern no point in spending outstanding effort to support all possible permutations when those have no relevance in context; what is supported should however be clearly documented; all this is especially true - but not limited - to JSON Path
• based on above I agree on emphasizing on MAY vs. MUST
• there is already support for alias and mask in the guidelines
• escaping (_) the reserved keywords, such as pagination and filter, would allow collision with a secondary field with similar meaning, which doesn't seem a good practice and thus guideline to promote

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

I would oppose any official endorsement of "virtual fields" (not that my opinion carries any weight). I am unsympathetic to these sorts of cheats intended to relieve developers from having to understand something somewhat complicated query. This is a single design time decision, we're not asking non-developers to see our understand it.

------------------------------
Vance Shipley
SigScale

Original Message:
Sent: Mar 19, 2025 13:23
From: Omar Sood
Subject: TMF630 4.3 Collection Filtering Suggestions

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

------------------------------
With kind regards
Omar Sood
conology
------------------------------

@Omar Sood, thanks for brining this topic. 

I can certify it is indeed a great pain to follow existing guidelines when it comes to filtering or partial attributes selection based on the content of the arrays (such as char-cs, permissions, etc.). Specifically with microservice architecture where each component is built by independent team using programming languages and databases of their choice, hence multiplying efforts to first construct (as a consumer) and then parse  (as a provider) complex JSONPath and construct Query out of it. 

Recent use-case I faced concerning TMF637:

GET /product/	?relatedParty.id=321&productSpecification.id=123	&filter=productCharacteristic[?(@.name=='Rufnummer' && @.value=='0176123456456')]	&fields=id,productCharacteristic[?(@.name=='Rufnummer' || @.name=='EVN')].name,productCharacteristic[?(@.name=='Rufnummer' || @.name=='EVN')].value[  {    "id": "003RO0000035WQgYAM",    "productCharacteristic": [      {        "name": "Rufnummer",        "value": "0176123456456"      },	  {        "name": "EVN",        "value": "Nein"      }    ]  }]

It is easier to support something like that

GET /product/	?relatedParty.id=321&productSpecification.id=123	&productCharacteristicRufnummer=0176123456456	&fields=id,productCharacteristicRufnummer

However, this works perfectly (intuitively) only with name-value pairs, like with various char-cs. There might be other conditions, such as action-function pair as part of privileges array defined in TMF672

I think best that can be done here is document that pattern as a concept with few examples and leave API provider to decide which attributes of the array are most meningful for filtering and display, build accordingly. 

@Christophe MICHEL, could you please ellaborate how MASK can be used in above mentioned examples? Did it make it into 763?

------------------------------
Denis Kozlov
Telefonica Germany GmbH & Co. OHG

Original Message:
Sent: Mar 20, 2025 05:04
From: Vance Shipley
Subject: TMF630 4.3 Collection Filtering Suggestions

I would oppose any official endorsement of "virtual fields" (not that my opinion carries any weight). I am unsympathetic to these sorts of cheats intended to relieve developers from having to understand something somewhat complicated query. This is a single design time decision, we're not asking non-developers to see our understand it.

------------------------------
Vance Shipley
SigScale

Original Message:
Sent: Mar 19, 2025 13:23
From: Omar Sood
Subject: TMF630 4.3 Collection Filtering Suggestions

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

------------------------------
With kind regards
Omar Sood
conology
------------------------------

@Omar Sood, thanks for brining this topic. 

I can certify it is indeed a great pain to follow existing guidelines when it comes to filtering or partial attributes selection based on the content of the arrays (such as char-cs, permissions, etc.). Specifically with microservice architecture where each component is built by independent team using programming languages and databases of their choice, hence multiplying efforts to first construct (as a consumer) and then parse  (as a provider) complex JSONPath and construct Query out of it. 

Recent use-case I faced concerning TMF637:

GET /product/	?relatedParty.id=321&productSpecification.id=123	&filter=productCharacteristic[?(@.name=='Rufnummer' && @.value=='0176123456456')]	&fields=id,productCharacteristic[?(@.name=='Rufnummer' || @.name=='EVN')].name,productCharacteristic[?(@.name=='Rufnummer' || @.name=='EVN')].value[  {    "id": "003RO0000035WQgYAM",    "productCharacteristic": [      {        "name": "Rufnummer",        "value": "0176123456456"      },	  {        "name": "EVN",        "value": "Nein"      }    ]  }]

It is easier to support something like that

GET /product/	?relatedParty.id=321&productSpecification.id=123	&productCharacteristicRufnummer=0176123456456	&fields=id,productCharacteristicRufnummer

However, this works perfectly (intuitively) only with name-value pairs, like with various char-cs. There might be other conditions, such as action-function pair as part of privileges array defined in TMF672

I think best that can be done here is document that pattern as a concept with few examples and leave API provider to decide which attributes of the array are most meningful for filtering and display, build accordingly. 

@Christophe MICHEL, could you please ellaborate how MASK can be used in above mentioned examples? Did it make it into 763?

------------------------------
Denis Kozlov
Telefonica Germany GmbH & Co. OHG

Original Message:
Sent: Mar 20, 2025 05:04
From: Vance Shipley
Subject: TMF630 4.3 Collection Filtering Suggestions

I would oppose any official endorsement of "virtual fields" (not that my opinion carries any weight). I am unsympathetic to these sorts of cheats intended to relieve developers from having to understand something somewhat complicated query. This is a single design time decision, we're not asking non-developers to see our understand it.

------------------------------
Vance Shipley
SigScale

Original Message:
Sent: Mar 19, 2025 13:23
From: Omar Sood
Subject: TMF630 4.3 Collection Filtering Suggestions

Hello everyone,

I'd like to discuss some feedback on TMF630, especially with respect to collection filtering, and hope to initiate a healthy discussion on the topic.

First, let me say that TMF630 is a really well-designed REST Api guideline applicable to more than just TMF APIs and a great source of proper REST design even outside the TMF context.

Collection filtering is defined by simple attribute query filters (Part 1) and by JSONPath filter mechanism (Part 6).

I'd like to recognize first, that the TMF Api design naturally arrives at requirements that fall into the latter path. Examples are search by externalId, search by characteristic, and search by related party.

At the same time, we need to be aware that - looking at questions in this forum and my personal experience in consulting - the JSONPath filtering is a large complexity for a lot of api implementers, especially for standard database solutions that a lot of developers choose. Therefore, a relevant amount of providers seem to choose to implement only a subset of the JSONPath specification, or implement variations based on the same idea - again based on personal experience and other forum posts.

I'd like to initiate a discussion about three possible improvements to TMF630.

Clearify and weaken TMF630 Part 6 collection filtering to emphasize implementation dependency

We introduce API standards and guides to clearly communicate expectations between client and server. The way Part 6 is written at the moment makes it ambiguous whether an implementing server is supposed to support the full JSONPath filter selector complexity or only a select subset. It doesn't use keywords like MAY or MUST. The way it is written, a client should expect full support. In reality due to underlying infrastructure, Api providers might want to support a subset, or even forbid theoretically supported queries due to their impact on the underlying infrastructure. The best example might be anything, that relates to structural search on the whole collection and includes $. An Api provider might also decide to cost-estimate the query and reject queries that look too expensive.

I think all these are totally valid variations that should be considered and allowed, if communicated appropriately. I think, here Part 6 could be improved to declare a clear stance on this topic.

If this is intended - as i would suggest - it should emphasize that supported queries are up to the discretion of the provider and should be clarified. Ideally, the guideline could define a minimal guaranteed feature set, and declare some framework on how to best communicate the available sub specification.

Clearify JSONPath filtering with respect to comparing fields defined in part 1

Part 1 introduces virtual properties like created.gte . While they are not required in JSONPath, there is not a definite reference to this in Part 6. While I'd say, this should not be part of filter selectors, I see it sometimes in the wild, again to ease some more complex query cases. Some clarifying note might be nice.

Virtual fields as a mid-complexity approach

A common emerging pattern for selected use cases is the introductions of what I call virtual fields as query parameter, such as product?characteristicFoo=bar, as a way to query a product that has the chracteristic with name foo and value bar. This is a common choice with more traditional db backends where the complexity of JSONPath doesn't justify the use cases. It also helps to expose only intended queries with optimized query strategies and applicable indexing.

To me, this is a totally valid approach that's worth including in the guideline to assist adoption in mid complexity cases. This could also benefit from some rules. I suggest allowing and including them, but prefixed with _, e.g. _foo to avoid clashes with actual properties. I'm aware that this may lead to poor Api design, if used extensively.

Prefix non-attribute-filter query parameters

In addition to the suggestion above, I'd like to throw my 50 cents in here, that it would be nice to also replace limit, offset, fields, filter with _limit, _offset, _fields, _filter, if a v6 is introduced. Imagine a Usage resources and one might already want to define a limit attribute, which would clash at the moment.

Same could be said for other current virtual fields like ._gte instead of .gte

See also

tmf630 part 6 jsonpath filters impractical by @Jan Cruz: I considered to comment there, but I think my feedback is a bit different.
tmf get query operations: One forum example api providers chose to implement virtual fields, before finding more complicated (better?) solutions
sigscale rest query scanner impl: One open sourced server implementation I stumbled across that chose not to implemented boolean operators. Again, I find this decision totally sane, but would like to see a clearer stance on expectations for both server and client.

Tagging the editor of both parts @Pierre Gauthier 

------------------------------
With kind regards
Omar Sood
conology
------------------------------