Open APIs

Expand all | Collapse all

Returning large payloads or documents as API response

  • 1.  Returning large payloads or documents as API response

    TM Forum Member
    Posted 12 days ago
    Edited by Steve Ranford-Bragg 12 days ago
    Hi,

    We have several scenarios where we need to return large payloads and our Apigee platform has a recommended maximum payload of around 10Mb and if using async responses Kafka's maximum limit is even smaller I believe.

    Some examples are :

    1. Customers are fetching details of our physical network inventory which covers the entire UK and for metropolitan areas may create very large payloads which exceed 10Mb
    2. As the FTTP network is rolled out we want to send details of addresses which have been enabled which again is a large dataset
    3. Where paging is not currently possible as the response payload is not returned as an array, so the use of offset and limit in it current format would not work

    At the moment we have worked around this by planning to send a UUID in the event payload for a document in our content management platform which the partner can then fetch from but that means a non-standard integration and we are trying to work within the framework where possible.

    We could use a redirect to send the consumer to a URL which contains the content but that would seem to be only a solution where the file already exists due to acceptable response times if creating a large payload but also would either require reauthentication for a secure service which the system would not be expecting in the current set up or a pre-generated token which is not great security and also could expire and be inaccessible.

    I was thinking that a new header could be added to the current API standards, say "X-Response-Location', which then the consumer knows uses TMF667 Document Management and can negotiate the security if necessary and contains a document with one or more attachments. For async responses, a new field could be added to the standard entity which gives a URL to direct the consumer to the event payload using the document API rather than it being embedded in "event" object?

    These are only some initial thoughts but wondered if anyone else had found the same limitation and how they had solved it?

    ------------------------------
    Steve Ranford-Bragg
    BT Group plc
    ------------------------------


  • 2.  RE: Returning large payloads or documents as API response

    TM Forum Member
    Posted 12 days ago
    Hi Steve
    I would say that there are use cases where APIs (e.g. REST) are not fit-for-purposes.
    Mass extract scenarios are a good example, and more generally mass operations (e.g. update 300,000 subscriptions with a new price due to operational considerations).
    For extract, one approach would be to build a framework that accepts a query and streams the results of the query as Kafka messages, or as a bunch of files. You would need to provide tools for monitoring this, and also to consider possibly separating the query (search criteria) from the content extract.
    At any rate it's much more complex than one API call, even with the header enhancements you suggested.
    Hope it helps.

    ------------------------------
    Jonathan Goldberg
    Amdocs Management Limited
    Any opinions and statements made by me on this forum are purely personal, and do not necessarily reflect the position of the TM Forum or my employer.
    ------------------------------



  • 3.  RE: Returning large payloads or documents as API response

    TM Forum Member
    Posted 12 days ago
    Agreed but I was wondering whether there were any plans to add support to the standards? As you say there are ways to achieve it but there are bound to be scenarios for many members and a common way to do it could be useful to avoid having point specific implementations for different partners.

    The way we plan to it currently does as you suggest which separates query from how the content is fetched and the response is just a directive to where that can be fetched.

    ------------------------------
    Steve Ranford-Bragg
    BT Group plc
    ------------------------------



  • 4.  RE: Returning large payloads or documents as API response

    TM Forum Member
    Posted 11 days ago
    Hi Steve

    To be perfectly honest, I think we need to concentrate in the Open API program on basic functionality, we still have a lot to do there. Mass operations are definitely important to telco business, but the RoI of trying to standardize them seems to be a lot smaller, and the technical challenges much larger.

    Regarding the separation between query and content, I recommend that you consider the Keyset Pagination algorithm, for retrieving 100,000s of IDs by query from an even larger data set - it can be applied on SQL and over ElasticSearch (and probably on other styles of data store). See this link, for example: http://allyouneedisbackend.com/blog/2017/09/24/the-sql-i-love-part-1-scanning-large-table/

    Hope it helps

    ------------------------------
    Jonathan Goldberg
    Amdocs Management Limited
    Any opinions and statements made by me on this forum are purely personal, and do not necessarily reflect the position of the TM Forum or my employer.
    ------------------------------



  • 5.  RE: Returning large payloads or documents as API response

    TM Forum Member
    Posted 11 days ago
    Thanks, that's fair enough. If there had been any thought on the subject it would have been useful to see if we could head in the same direction but we have a working solution for now.

    ------------------------------
    Steve Ranford-Bragg
    BT Group plc
    ------------------------------