TM Forum Community

 View Only
  • 1.  TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 09, 2024 03:30
    Edited by Ilyas Premji Apr 10, 2024 03:52

    TMF630 Part 6 - TMF630 REST API Design Guidelines 4.2.0 - specifies the filter parameter with JsonPath value so that each json attribute can be query-able.

    Question: How practical is this to implement and is there an example in the wild?

    Context:  Consider millions of records from any popular database such as ElasticSearch (ES).  To keep things simple, data in ES has the exact same data model exposed to the client, so no transformations would be required when subset of data gets fetched.

    1. Client sends GET request with a JsonPath filter to Service.
    2. Service converts JsonPath filter to Database query language.
    3. Service returns Database results to Client.

    Regarding Step 2, is there an example in the wild irrespective of DB query language used?  Even in the simpler setup of consistent client and DB data models, manually parsing the JsonPath to the DB query language is already a complex task.  I say manually as there does not appear to be a library which handles this.  Imagine having to parse every combination of this RFC 9535: JSONPath: Query Expressions for JSON to the DB query language.

    Of course, the JsonPath transformation complexity only increases even more when the database does not store the data in the "native" format so every field would also require transformation.  So for example, field 'contact.email' might be stored in a completely separate path in the DB.  Or you may have one-to-many or many-to-many mappings.  Or JsonPath array positions may not make sense when data is stored in a different structure.  The dynamic translations of fields, paths, operators, '$', '[]', '*', '?' etc from one syntax to another seems too cumbersome and impractical to design and code.

    Back to the question, how practical is this to implement and is there an open source example out in the wild?  I feel until DBs widely support JsonPath as a query language so "Step 2" would be mostly passthrough, then TMF630 Part 6 is too impractical to implement, especially when there are hundreds of fields and complex transformations involved.


    #General



  • 2.  RE: TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 09, 2024 07:36

    Hi Jan

    You make a good point here, but at the end of the day it's an implementation decision for each API whether or not to support JSON Path filtering. It could be that for moderately sized storage (reference data or cached data for example), implementing JSON Path outside the persistence layer might be acceptable.

    And of course if you're lucky enough to have a persistence store that supports JSON Path natively, even better.

    The fallback is the vanilla filtering described in design guidelines part 1.

    Thanks for your contribution to the discussion.



    ------------------------------
    Jonathan Goldberg
    Amdocs Management Limited
    Any opinions and statements made by me on this forum are purely personal, and do not necessarily reflect the position of the TM Forum or my employer.
    ------------------------------



  • 3.  RE: TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 10, 2024 20:04

    Hi @Jonathan Goldberg, thanks for the response.  Tagging the authors as well @Florin Tene, @Pierre Gauthier to hopefully share their perspective on this.

    I think this puts the Part 6 Guideline in a precarious position.  It seems it can only be practically applied in niche scenarios, such as those Jonathan described (which I agree with).  But for the vast majority of times, data being "filtered" is taken from DBs (SQL, NoSQL), especially for your more important APIs.  So if you're using Relational, you can't support; and none of the NoSQL DBs support JsonPath as a search language either.  So if can only be practically applied to the likes of reference or cache data, then the Guideline can only apply to a very very small subset of use cases.

    Compare to other Guidelines such as "return 201/202 when creating entities", which would apply to the vast majority of cases; with maybe niche scenarios where it would not apply.  I feel that's how guidelines generally should be - applies to the vast majority of use cases (unless caveats explicitly stated); not the other way around, which seems to be the case for Part 6.  (We understand this above-the-norm complexity, but for others, could be seen as just another guideline to follow.)

    Part 6 seems more aspirational, rather than a guideline.  It does sound good in theory as JsonPath does unambiguously describe a json, but can unfortunately only be practically applied in a limited set of use cases.



    ------------------------------
    Jan
    ------------------------------



  • 4.  RE: TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 11, 2024 05:47

    @Jan Cruz declares TMF630 Part 6 useless because he/she fails to find a complete solution for JSONPath readily available. Never mind that Redis and other products do support it.

    At my organization we write software, which means implementing feature requirements in code. When our applications required advanced filtering requests we looked to TM Forum Open APIs for guidance. We first implemented a proprietary extension using a pattern proposed in AP-832. Later we were pleased to have TMF630 Part 6 provide a normative specification. We now implement ODA Components with powerfully expressive queries over standard Open APIs. It's far from useless, it's just apparently beyond your capability.



    ------------------------------
    Vance Shipley
    SigScale
    ------------------------------



  • 5.  RE: TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 11, 2024 21:58

    Hi @Vance Shipley

    "Redis and other products" - which products that can be the storage of choice for Trouble Tickets which was the example used in Part 6?  None within AWS.  Redis is more under the "cache data" scenario already acknowledged. 

    In https://engage.tmforum.org/discussion/tmf-get-query-operations you posted "You don't have to support every possible filter expression, just the ones you need.": I wholeheartedly agree with that caveat.  I guess Part 6 was misinterpreted?  It is not a recommendation to make every combination of fields and operations query-able as JsonPath can.  Rather it's a guideline on how to express required complex queries, not a push to handle ALL possible complex queries.  Embrace the 501s.

    "it's just apparently beyond your capability" - Ouch.:)  For what it's worth, I've implemented Part 6 translations well beyond "the ones you need" (every-attribute-every-operator-every-combination) with TMF used as justification for the requirement, being skeptical that these low level, multitude of translations cannot possibly be replicated by other TMF orgs with varying DBs and data models and support the full JsonPath syntax.



    ------------------------------
    Jan
    ------------------------------



  • 6.  RE: TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 12, 2024 10:54

    The statement "You don't have to support every possible filter expression, just the ones you need." leads to variations among implementations. An implementation that aspires to be reused across a diverse community would want to have coverage of "every possible filter expression". Variations like this across vendor implementations does not bode well for the standard influencing improved interoperability.



    ------------------------------
    Ben Eng
    Oracle Corporation
    ------------------------------



  • 7.  RE: TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 12, 2024 22:19

    What's your point @Ben Eng? Are you arguing that vendors must support arbitrarily complex JSON Path queries if they support it at all?  Or are you joining @Jan Cruz in arguing against the continued existence of the advanced query pattern in TMF630 Part 6? Either is nonsense to me.

    Here is a simple fact: without Part 6 you cannot query for items which have a specific Characteristic value. How do you build a useful ODA Component without the ability to query on Characteristics? My experience was that we could not. 



    ------------------------------
    Vance Shipley
    SigScale
    ------------------------------



  • 8.  RE: TMF630 Part 6 - JsonPath Filters - Impractical?

    TM Forum Member
    Posted Apr 13, 2024 10:40

    I am arguing that "vendors must support arbitrarily complex JSON Path queries if they support it at all" should be preferred. To the extent that this is not supported there may be no interoperability across implementations. If interoperability is the goal, the standard should be more precise and prescriptive. Of course, you are free to disagree if interoperability is not a priority, which may well be the case for a particular project. It's not something I feel strongly about, so I'm not going to argue for it further.



    ------------------------------
    Ben Eng
    Oracle Corporation
    ------------------------------