Note
- Using search_after requires multiple search requests with the same query and sort values. If a insert/delete of Devices occurs between these requests, the order of the search results may change, causing inconsistent results across pages.
- If there are more than 10K records, it's not possible to get search_after value for the last page. Hence, the last page link won't be available in Smart Search response.
- Sort by Device Id will be added by default to sort the search results whether any other sort criteria is provided or not.
Deep pagination query:
{
"search": {
"queryString": "id: *"
}
}
{
"response": {
"searchableProperties": {
"account (long[])": "",
.....
},
"queryTime": 342,
"totalCount": 134925,
"results": [
{
"device": {
"id": 9901,
.....
}
},
.....
{
"device": {
"id": 10000,
.....
}
}
],
"_links": [
{
"href": "https://connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=100&pgSize=100&sort=lastModifiedOn",
"rel": "self"
},
{
"href": "https://connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=99&pgSize=100&sort=lastModifiedOn",
"rel": "prev"
},
{
"href": "https://connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=1&pgSize=100&sort=lastModifiedOn",
"rel": "first"
},
{
"href": "https://connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=101&pgSize=100&sort=lastModifiedOn&searchAfter=1607158662540,10000",
"rel": "next"
}
]
}
}
Note
To go to next page (pg=101), client should wither click on the "next" link or copy & paste the "next" url as is to go to the next page. Don't modify the url.
Next page (pg=101) request url:
{
"search": {
"queryString": "id: *"
}
}
{
"response": {
"searchableProperties": {
"account (long[])": "",
.....
},
"queryTime": 342,
"totalCount": 134925,
"results": [
{
"device": {
"id": 10001,
.....
}
},
.....
{
"device": {
"id": 10100,
.....
}
}
],
"pageNumToPrevSearchAfterMappings": {
"101": "1607158662540,10000"
},
"_links": [
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=101&pgSize=100&sort=lastModifiedOn&searchAfter=1607158662540,10000",
"rel": "self"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=100&pgSize=100&sort=lastModifiedOn",
"rel": "prev"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=1&pgSize=100&sort=lastModifiedOn",
"rel": "first"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=102&pgSize=100&sort=lastModifiedOn&searchAfter=1607245752757,10100",
"rel": "next"
}
]
}
}
The response for pg=101. will contain "pageNumToPrevSearchAfterMappings" which represents page number to searchAfter mappings for the previous pages. Now, to go to next page, make sure to copy the "pageNumToPrevSearchAfterMappings" & add it to the search query in the request body (see below).
Next page (pg=102) request url:
{
"search": {
"queryString": "id: *",
"pageNumToPrevSearchAfterMappings": {
"101": "1607158662540,169870"
}
}
}
{
"response": {
"searchableProperties": {
"account (long[])": "",
.....
},
"queryTime": 342,
"totalCount": 134925,
"results": [
{
"device": {
"id": 10101,
.....
}
},
.....
{
"device": {
"id": 10200,
.....
}
}
],
"pageNumToPrevSearchAfterMappings": {
"101": "1607158662540,10000",
"102": "1607245752757,10100"
},
"_links": [
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=102&pgSize=100&sort=lastModifiedOn&searchAfter=1607245752757,10100",
"rel": "self"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=101&pgSize=100&sort=lastModifiedOn&searchAfter=1607158662540,10000",
"rel": "prev"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=1&pgSize=100&sort=lastModifiedOn",
"rel": "first"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=103&pgSize=100&sort=lastModifiedOn&searchAfter=1607504206932,10200",
"rel": "next"
}
]
}
}
Follow the above steps to traverse to subsequent pages.
When using sort by extensions fields like appId, firmwareVersion, fileVersions.version etc, if the field is null or missing for the last record of the page then searchAfter will have null values and shows invalid next page link and shows the same response in subsequent pages, so please make sure use the "missing" parameter inside sort, that can be set any value. ES will internally use this value for null values of that sort field and shows the proper response.
{
"search": {
"query": {
"query": {
"bool": {
"must": [],
"must_not": []
}
},
"sort": [
{
"extensions.appId": {
"order": "asc",
"missing" : "_missing"
}
},
{
"id": {
"order": "asc"
}
}
]
},
"accountId": 96461
}
}
{
"response": {
"searchableProperties": {
"account (long[])": "",
.....
},
"queryTime": 342,
"totalCount": 134925,
"results": [
{
"device": {
"id": 632031,
.....
}
},
.....
{
"device": {
"id": 632031,
.....
}
}
],
"pageNumToPrevSearchAfterMappings": {
"101": "_missing,631930",
"102": "_missing,632030"
},
"_links": [
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=102&pgSize=100&searchAfter=_missing%2C632030",
"rel": "self"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=101&pgSize=100&searchAfter=_missing%2C631930",
"rel": "prev"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=1&pgSize=100",
"rel": "first"
},
{
"href": "https://dev.connect.calamp.com/connect/services/devices/smartsearch?v=2.1&pg=103&pgSize=100&searchAfter=_missing%2C632130",
"rel": "next"
}
]
}
}