Hi,
We have several scenarios where we need to return large payloads and our Apigee platform has a recommended maximum payload of around 10Mb and if using async responses Kafka's maximum limit is even smaller I believe.
Some examples are :
1. Customers are fetching details of our physical network inventory which covers the entire UK and for metropolitan areas may create very large payloads which exceed 10Mb
2. As the FTTP network is rolled out we want to send details of addresses which have been enabled which again is a large dataset
3. Where paging is not currently possible as the response payload is not returned as an array, so the use of offset and limit in it current format would not work
At the moment we have worked around this by planning to send a UUID in the event payload for a document in our content management platform which the partner can then fetch from but that means a non-standard integration and we are trying to work within the framework where possible.
We could use a redirect to send the consumer to a URL which contains the content but that would seem to be only a solution where the file already exists due to acceptable response times if creating a large payload but also would either require reauthentication for a secure service which the system would not be expecting in the current set up or a pre-generated token which is not great security and also could expire and be inaccessible.
I was thinking that a new header could be added to the current API standards, say "X-Response-Location', which then the consumer knows uses TMF667 Document Management and can negotiate the security if necessary and contains a document with one or more attachments. For async responses, a new field could be added to the standard entity which gives a URL to direct the consumer to the event payload using the document API rather than it being embedded in "event" object?
These are only some initial thoughts but wondered if anyone else had found the same limitation and how they had solved it?
------------------------------
Steve Ranford-Bragg
BT Group plc
------------------------------