Open APIs

 View Only
  • 1.  Case Insensitive filtering

    Posted Nov 22, 2021 01:30
    Hello Everyone,

    According to a use-case, for example searching a name in customer management API, I could not find any case-sensitivity specification in definitions.
    in Page 28 , TMF630 Part 6 , I have seen a note as : "• JSONPath expressions, including property names and values, are case-sensitive."
    The problem is not even about partial searching, it is exactly when the names are written in wrong cases , 

    Let me know how to find an array of entities including names when data is stored as "Ebrahim,EBRAHIM,ebrahim and so on"

    Regards

    ------------------------------
    Ebrahim Mirzazadeh
    ParsPooyesh
    ------------------------------


  • 2.  RE: Case Insensitive filtering

    TM Forum Member
    Posted Nov 22, 2021 02:38
    Thanks Ebrahim for this catch.
    Very important that this gets resolved, as a direction to API users and implementers.
    Hopefully @Florin Tene, who wrote this chapter, can relate and give guidance.​

    ------------------------------
    Jonathan Goldberg
    Amdocs Management Limited
    Any opinions and statements made by me on this forum are purely personal, and do not necessarily reflect the position of the TM Forum or my employer.
    ------------------------------



  • 3.  RE: Case Insensitive filtering

    TM Forum Member
    Posted Nov 22, 2021 16:53

    Hi Ebrahim, 

          I can cover this topic from the JSON Path perspective, not necessarily from the Customer Management API, mainly because the JSONPath is a generic pattern applicable to all APIs (of course, depending on the implementation).

        On the JSONPath specification by Stefan Goessner (available here: https://goessner.net/articles/JsonPath/), there is no mention regarding the case sensitivity. As such, it is pretty much a library implementation decision on the underlying JSON libraries.

        However, I believe the major libraries by default do support case sensitivity on both attributes and their values.

        Now, the statement from the Design Guidelines does indeed cover both attributes and values, and that is due to the behavior from the major libraries (e.g., Jayway, Gatling, and Goessner's implementation).

    I think it is easier to go through this one via an example we consider.
    So let's assume we have the following resource (I will stick to one resource but is the same behavior if applicable across a collection of resources).

    Let's use the classic example from JSONPath specification:

    {

        "store": {

            "Book": [

                {

                    "category": "reference",

                    "author": "Nigel Rees",

                    "title": "Sayings of the Century",

                    "price": 8.95

                },

                {

                    "category": "fiction",

                    "author": "nigel rees",

                    "title": "Sword of Honour",

                    "price": 12.99

                },

                {

                    "category": "fiction",

                    "author": "John Xadi",

                    "title": "Moby Dick",

                    "isbn": "0-553-21311-3",

                    "price": 8.99

                },

                {

                    "category": "fiction",

                    "author": "J. R. R. Tolkien",

                    "title": "The Lord of the Rings",

                    "isbn": "0-395-19395-8",

                    "price": 22.99

                }

            ],

            "bicycle": {

                "color": "red",

                "price": 19.95

            }

        },

        "expensive": 10

    }

      Now, in the above example, we have:

    • one array (collection of books) called 'Book' - notice the capital B
    • in the array, we have two entries where the same author "Nigel Rees" is recorded in two different ways:
      • Nigel Rees (notice the capital N and R)
      • nigel rees (all lower case)

    So, for the first one, if we're covering the attribute case, we have the following expressions:

    • $.store.book - which return no result

    while 

    • $.store.Book - correctly identifies the "Book" array and returns the correct result

    [

       {

          "category" : "reference",

          "author" : "Nigel Rees",

          "title" : "Sayings of the Century",

          "price" : 8.95

       },

       {

          "category" : "fiction",

          "author" : "nigel rees",

          "title" : "Sword of Honour",

          "price" : 12.99

       },

       {

          "category" : "fiction",

          "author" : "Nigels Rees",

          "title" : "Moby Dick",

          "isbn" : "0-553-21311-3",

          "price" : 8.99

       },

       {

          "category" : "fiction",

          "author" : "J. R. R. Tolkien",

          "title" : "The Lord of the Rings",

          "isbn" : "0-395-19395-8",

          "price" : 22.99

       }

    ]

        Now, regarding the attribute value, we have the following expressions:

    • $.store.Book[?(@.author=='Nigel Rees')] which correctly identifies the value from "$['store']['Book'][0]"

    [

       {

          "category" : "reference",

          "author" : "Nigel Rees",

          "title" : "Sayings of the Century",

          "price" : 8.95

       }

    ]

    • $.store.Book[?(@.author=='nigel rees')] which correctly identifies the value from "$['store']['Book'][1]"

    [

       {

          "category" : "fiction",

          "author" : "nigel rees",

          "title" : "Sword of Honour",

          "price" : 12.99

       }

    ]

    So, the value is also case-sensitive.

    The JSONPath is case sensitive because the underlying JSON is also case sensitive, pretty much.

    It depends if you have native support in the backend for JSONPath or not.    

    Depending on your specific case (backend support, processing), there are other patterns usually, like UPPER(Oracle/PostgreSQL) or toUpper(Mongo). For example: postgresSQL(which supports natively jsonPath): 

    where lower(data::text)::jsonb @> lower('[{"city":"New York"}]')::jsonb  would do the trick.


    Now, if we're looking for a pure JSONPath answer, JSONPath supports regexp expressions (subject to the underlying JSONPath library).

    So, having the same example as above, the following query will select all the results where the author name is case insensitive:

    $.store.Book[?(@.author=~/.*Nigel Rees/i)]

    (Jayway library - Ruby regex operator)

    or 

    $.store.Book[?(/^.*Nigel Rees.*$/i.test(@.author))]

    (Goessner's library) 


    In conclusion, in my view, the statement that the JSONPath is case-sensitive is correct.
    If we're looking for case-insensitive option, there are alternative options available although the best one is subject to specific case/project restrictions. 



    ------------------------------
    Florin Tene
    CityFibre
    ------------------------------



  • 4.  RE: Case Insensitive filtering

    TM Forum Member
    Posted Nov 23, 2021 07:38
    Thanks Florin for this very full answer.
    To be honest, however, I think there is a case to be made for the attribute value matching, at least, to be case-insensitive. I can't imagine a normal business case where a case-sensitive search would be better; values are generally entered by end users, not developers.
    For the attribute name matching, it might be less critical, since the attribute names will generally be defined up-front by developers and used by developers and integrators.

    ------------------------------
    Jonathan Goldberg
    Amdocs Management Limited
    Any opinions and statements made by me on this forum are purely personal, and do not necessarily reflect the position of the TM Forum or my employer.
    ------------------------------



  • 5.  RE: Case Insensitive filtering

    TM Forum Member
    Posted Nov 23, 2021 07:51
    Hey Jonathan,

      Agree, however there are limitations driven by the underlying JSON in this case. 
      Case-insensitive search can be supported by either different approaches in the microservices implementations or supported natively by the backends.
      Also, the examples I gave above does cover case-insensitive search purely using JSON Path - regexp feature.

      I will add a JIRA ticket to enhance the DG - Part 6 to provide additional examples using the regexp feature to cover this cases. 

    Thank you,
    Florin

    ------------------------------
    Florin Tene
    CityFibre
    ------------------------------



  • 6.  RE: Case Insensitive filtering

    Posted Nov 29, 2021 03:11
    Edited by Ebrahim Mirzazadeh Nov 29, 2021 03:45

    Hi Florin,
    Before everything, I have to appreciate you putting time to describe both front and underlying aspects of the problem with a good example.
    As you may noticed, I was not focused on a specific API and the case insensitivity in all APIs was the target.
    After reading your reply, the only working approach at the moment looks as using regex in JSONPath, because the database patches affect the results globally. Someone may need the case-sensitive queries and patching in data layer may create further problems.
    Thinking about the problem looking internationally, convinced me that we may need even more customization to have correct results. As an example, we have trouble on difference between "Ya" or "Kaf" characters code in Arabic and Persian. I would investigate more in depth with cooperation of our backend developer and return back to you if there was any other idea or problem.

    Sample in "Ya" difference : 
    روحی - روحي 
    ابراهیم - ابراهيم

    Regards



    ------------------------------
    Ebrahim Mirzazadeh
    ParsPooyesh
    ------------------------------